Error when parsing certain files for duplications

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: GROOVY-0.1
Fix Version/s: GROOVY-0.5
Component/s: Groovy
Labels:
None

Patch Submitted:

Yes
Number of attachments :
0

Description

The following error can happen. This seems to be fixed in the version 1.7 of groovy parser. CodeNarc is currently in 1.6

[ERROR]  Unexpected token when lexing file : /Users/holly/Downloads/Gosu-codenarc/src/main/groovy/org/codenarc/util/io/ClassPathResource.groovy
groovyjarjarantlr.TokenStreamRecognitionException: expecting anything but ''\n''; got it anyway
	at org.codehaus.groovy.antlr.parser.GroovyLexer.nextToken(GroovyLexer.java:696) [na:na]
	at org.sonar.plugins.groovy.foundation.GroovyCpdTokenizer.tokenize(GroovyCpdTokenizer.java:45) [na:na]
	at org.sonar.duplications.cpd.CPD.add(CPD.java:109) [sonar-duplications-2.2.jar:na]
	at org.sonar.duplications.cpd.CPD.add(CPD.java:74) [sonar-duplications-2.2.jar:na]
	at org.sonar.plugins.cpd.CpdSensor.configureCPD(CpdSensor.java:109) [sonar-cpd-plugin-2.2.jar:na]
	at org.sonar.plugins.cpd.CpdSensor.executeCPD(CpdSensor.java:88) [sonar-cpd-plugin-2.2.jar:na]
	at org.sonar.plugins.cpd.CpdSensor.analyse(CpdSensor.java:68) [sonar-cpd-plugin-2.2.jar:na]
	at org.sonar.batch.SensorsExecutor.execute(SensorsExecutor.java:58) [sonar-batch-2.2.jar:na]
	at org.sonar.batch.ProjectBatch.execute(ProjectBatch.java:53) [sonar-batch-2.2.jar:na]
	at org.sonar.batch.Batch.analyzeProject(Batch.java:112) [sonar-batch-2.2.jar:na]
	at org.sonar.batch.Batch.analyzeProjects(Batch.java:81) [sonar-batch-2.2.jar:na]
	at org.sonar.batch.Batch.execute(Batch.java:57) [sonar-batch-2.2.jar:na]
	at org.sonar.maven2.BatchMojo.executeBatch(BatchMojo.java:142) [sonar-core-maven-plugin-20100727133639.jar:na]
	at org.sonar.maven2.BatchMojo.execute(BatchMojo.java:133) [sonar-core-maven-plugin-20100727133639.jar:na]
	at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0]
	at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:82) [sonar-maven-plugin-1.0-beta-1.jar:na]
	at org.codehaus.mojo.sonar.Bootstraper.start(Bootstraper.java:65) [sonar-maven-plugin-1.0-beta-1.jar:na]
	at org.codehaus.mojo.sonar.SonarMojo.execute(SonarMojo.java:117) [sonar-maven-plugin-1.0-beta-1.jar:na]
	at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:678) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:553) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:523) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:371) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:268) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:181) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:356) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:137) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) [maven-2.2.0-uber.jar:2.2.0]
	at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:41) [maven-2.2.0-uber.jar:2.2.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0_20]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [na:1.6.0_20]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [na:1.6.0_20]
	at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_20]
	at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) [classworlds-1.1.jar:1.1]
	at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) [classworlds-1.1.jar:1.1]
	at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) [classworlds-1.1.jar:1.1]
	at org.codehaus.classworlds.Launcher.main(Launcher.java:375) [classworlds-1.1.jar:1.1]

Activity

Ascending order - Click to sort in descending order

Evgeny Mandrikov made changes - 05/May/11 10:09 AM

Field	Original Value	New Value
Description	The following error can happen. This seems to be fixed in the version 1.7 of groovy parser. CodeNarc is currently in 1.6 [ERROR] Unexpected token when lexing file : /Users/holly/Downloads/Gosu-codenarc/src/main/groovy/org/codenarc/util/io/ClassPathResource.groovy groovyjarjarantlr.TokenStreamRecognitionException: expecting anything but ''\n''; got it anyway at org.codehaus.groovy.antlr.parser.GroovyLexer.nextToken(GroovyLexer.java:696) [na:na] at org.sonar.plugins.groovy.foundation.GroovyCpdTokenizer.tokenize(GroovyCpdTokenizer.java:45) [na:na] at org.sonar.duplications.cpd.CPD.add(CPD.java:109) [sonar-duplications-2.2.jar:na] at org.sonar.duplications.cpd.CPD.add(CPD.java:74) [sonar-duplications-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.configureCPD(CpdSensor.java:109) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.executeCPD(CpdSensor.java:88) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.analyse(CpdSensor.java:68) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.batch.SensorsExecutor.execute(SensorsExecutor.java:58) [sonar-batch-2.2.jar:na] at org.sonar.batch.ProjectBatch.execute(ProjectBatch.java:53) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.analyzeProject(Batch.java:112) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.analyzeProjects(Batch.java:81) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.execute(Batch.java:57) [sonar-batch-2.2.jar:na] at org.sonar.maven2.BatchMojo.executeBatch(BatchMojo.java:142) [sonar-core-maven-plugin-20100727133639.jar:na] at org.sonar.maven2.BatchMojo.execute(BatchMojo.java:133) [sonar-core-maven-plugin-20100727133639.jar:na] at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0] at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:82) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.codehaus.mojo.sonar.Bootstraper.start(Bootstraper.java:65) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.codehaus.mojo.sonar.SonarMojo.execute(SonarMojo.java:117) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:678) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:553) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:523) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:371) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:268) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:181) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:356) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:137) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:41) [maven-2.2.0-uber.jar:2.2.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0_20] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [na:1.6.0_20] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [na:1.6.0_20] at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_20] at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.main(Launcher.java:375) [classworlds-1.1.jar:1.1]	The following error can happen. This seems to be fixed in the version 1.7 of groovy parser. CodeNarc is currently in 1.6 {noformat} [ERROR] Unexpected token when lexing file : /Users/holly/Downloads/Gosu-codenarc/src/main/groovy/org/codenarc/util/io/ClassPathResource.groovy groovyjarjarantlr.TokenStreamRecognitionException: expecting anything but ''\n''; got it anyway at org.codehaus.groovy.antlr.parser.GroovyLexer.nextToken(GroovyLexer.java:696) [na:na] at org.sonar.plugins.groovy.foundation.GroovyCpdTokenizer.tokenize(GroovyCpdTokenizer.java:45) [na:na] at org.sonar.duplications.cpd.CPD.add(CPD.java:109) [sonar-duplications-2.2.jar:na] at org.sonar.duplications.cpd.CPD.add(CPD.java:74) [sonar-duplications-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.configureCPD(CpdSensor.java:109) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.executeCPD(CpdSensor.java:88) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.plugins.cpd.CpdSensor.analyse(CpdSensor.java:68) [sonar-cpd-plugin-2.2.jar:na] at org.sonar.batch.SensorsExecutor.execute(SensorsExecutor.java:58) [sonar-batch-2.2.jar:na] at org.sonar.batch.ProjectBatch.execute(ProjectBatch.java:53) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.analyzeProject(Batch.java:112) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.analyzeProjects(Batch.java:81) [sonar-batch-2.2.jar:na] at org.sonar.batch.Batch.execute(Batch.java:57) [sonar-batch-2.2.jar:na] at org.sonar.maven2.BatchMojo.executeBatch(BatchMojo.java:142) [sonar-core-maven-plugin-20100727133639.jar:na] at org.sonar.maven2.BatchMojo.execute(BatchMojo.java:133) [sonar-core-maven-plugin-20100727133639.jar:na] at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0] at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:82) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.codehaus.mojo.sonar.Bootstraper.start(Bootstraper.java:65) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.codehaus.mojo.sonar.SonarMojo.execute(SonarMojo.java:117) [sonar-maven-plugin-1.0-beta-1.jar:na] at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:678) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:553) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:523) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:371) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:268) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:181) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:356) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:137) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) [maven-2.2.0-uber.jar:2.2.0] at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:41) [maven-2.2.0-uber.jar:2.2.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0_20] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [na:1.6.0_20] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [na:1.6.0_20] at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_20] at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) [classworlds-1.1.jar:1.1] at org.codehaus.classworlds.Launcher.main(Launcher.java:375) [classworlds-1.1.jar:1.1] {noformat}

Hide

Permalink

Martin Homik added a comment - 23/Sep/11 6:11 AM

Is there any explanation for the bug? I use Grails v1.3.7 with codenarc v0.15 and groovy-sonar-plugin v0.3. I get this exception which complains about the following characters: 0xA, ''\n'', and '$'. I suspect, this is an encoding problem (I use UTF-8 in my project). However, I don't know if it is possible to fix it outside the plugin. The bug occurs when CpdSensor analyses codenarc-report.xml.

Show

Martin Homik added a comment - 23/Sep/11 6:11 AM Is there any explanation for the bug? I use Grails v1.3.7 with codenarc v0.15 and groovy-sonar-plugin v0.3. I get this exception which complains about the following characters: 0xA, ''\n'', and '$'. I suspect, this is an encoding problem (I use UTF-8 in my project). However, I don't know if it is possible to fix it outside the plugin. The bug occurs when CpdSensor analyses codenarc-report.xml.

Hide

Permalink

Evgeny Mandrikov added a comment - 23/Sep/11 7:30 AM

Hi Martin,
CpdSensor doesn't perform analysis of codenarc-report.xml, instead it parses source code of a project, so problem comes from a parser, but what is weird is that exactly this parser is used to compile Groovy code...

Show

Evgeny Mandrikov added a comment - 23/Sep/11 7:30 AM Hi Martin, CpdSensor doesn't perform analysis of codenarc-report.xml, instead it parses source code of a project, so problem comes from a parser, but what is weird is that exactly this parser is used to compile Groovy code...

Hide

Permalink

Evgeny Mandrikov added a comment - 14/Dec/11 2:46 AM

What is strange is that current mechanism for lexical analysis throws an exception for following simple code:

class Greet {
  def name
  Greet(who) { name = who }
  def salute() { println "Hello $name!" }
}

And indirect usage of GroovyLexer via TokenStream (instead of direct) solves the problem:

TokenStream tokenStream = new GroovyLexer(reader).plumb();
Token token = tokenStream.nextToken();
while (token.getType() != Token.EOF_TYPE) {
  ...
  token = tokenStream.nextToken();
}

All this was tested with groovy-all-1.8.4.jar

Show

Evgeny Mandrikov added a comment - 14/Dec/11 2:46 AM What is strange is that current mechanism for lexical analysis throws an exception for following simple code: class Greet { def name Greet(who) { name = who } def salute() { println "Hello $name!" } } And indirect usage of GroovyLexer via TokenStream (instead of direct) solves the problem: TokenStream tokenStream = new GroovyLexer(reader).plumb(); Token token = tokenStream.nextToken(); while (token.getType() != Token.EOF_TYPE) { ... token = tokenStream.nextToken(); } All this was tested with groovy-all-1.8.4.jar

Evgeny Mandrikov made changes - 14/Dec/11 3:35 AM

Fix Version/s

GROOVY-0.5 [ 18225 ]

Evgeny Mandrikov made changes - 14/Dec/11 6:24 AM

Patch Submitted

Yes [ 10070 ]

Evgeny Mandrikov made changes - 14/Dec/11 2:20 PM

Assignee

Evgeny Mandrikov [ godin ]

Evgeny Mandrikov made changes - 14/Dec/11 2:20 PM

Priority

Major [ 3 ]

Critical [ 2 ]

Hide

Permalink

Evgeny Mandrikov added a comment - 14/Dec/11 3:48 PM

Patch applied in r4827 and according to integration test seems to work fine - code of CodeNarc 0.9 can be sucesfully parsed.

Show

Evgeny Mandrikov added a comment - 14/Dec/11 3:48 PM Patch applied in r4827 and according to integration test seems to work fine - code of CodeNarc 0.9 can be sucesfully parsed.

Evgeny Mandrikov made changes - 14/Dec/11 3:48 PM

Status	Open [ 1 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Hide

Permalink

Khayrat Glende added a comment - 16/Dec/11 8:18 AM

Great, I just came up to the same solution and was going to apply the patch too

Show

Khayrat Glende added a comment - 16/Dec/11 8:18 AM Great, I just came up to the same solution and was going to apply the patch too

Evgeny Mandrikov made changes - 04/Jan/12 2:11 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:

Evgeny Mandrikov

Reporter:

Olivier Gaudin

Vote (12)

Watch (6)

Dates

Created:

27/Jul/10 6:51 AM

Updated:

04/Jan/12 2:11 PM

Resolved:

14/Dec/11 3:48 PM