JRuby

__FILE__ corrupts cyrillic characters

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.6
  • Fix Version/s: JRuby 1.7.0.RC1
  • Component/s: Parser
  • Labels:
    None
  • Environment:
    Mac OS X 10.6.6, running JRuby 1.6 within Eclipse 3.6.1
  • Number of attachments :
    0

Description

We are executing scripts from Java with something very much like the following

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETON, LocalVariableBehavior.TRANSIENT);
EmbedEvalUnit unit = container.parse(PathType.ABSOLUTE, this._filename);
result = unit.run();

If I execute a script in a path like "/Users/klindsey/Documents/Александр/script.rb", a call to 'puts _FILE' in that script results in the following output: "/Users/klindsey/Documents/?????????/script.rb". However, putting $0 shows the correct value. Is there something that needs to be set when executing scripts to preserve the encoding in __FILE_?

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Christopher Williams added a comment -

The corresponding bug report from our Aptana Studio 3 user is here: https://aptana.lighthouseapp.com/projects/35272/tickets/1663-bug-with-loading-bundles

Show
Christopher Williams added a comment - The corresponding bug report from our Aptana Studio 3 user is here: https://aptana.lighthouseapp.com/projects/35272/tickets/1663-bug-with-loading-bundles
Hide
Permalink
Charles Oliver Nutter added a comment -

Hmm, I am so far unable to reproduce on a system that's UTF-8 by default:

~/projects/jruby ➔ cat Русский.rb 
puts __FILE__

~/projects/jruby ➔jrubyt Русский.rb 
Русский.rb

Going to need more information (user's system encoding, etc) to investigate this. Could it be Windows-specific?

Show
Charles Oliver Nutter added a comment - Hmm, I am so far unable to reproduce on a system that's UTF-8 by default: ~/projects/jruby ➔ cat Русский.rb puts __FILE__ ~/projects/jruby ➔jrubyt Русский.rb Русский.rb Going to need more information (user's system encoding, etc) to investigate this. Could it be Windows-specific?
Hide
Permalink
Charles Oliver Nutter added a comment -

Here's a gist of that session, since JIRA seems to mangle Cyrillic

https://gist.github.com/887907

Show
Charles Oliver Nutter added a comment - Here's a gist of that session, since JIRA seems to mangle Cyrillic https://gist.github.com/887907
Hide
Permalink
Kevin Lindsey added a comment -

Hi Charles,

We're running inside of Eclipse 3.6.1. The original user was running under Windows, but I've been able to reproduce the issue here on a Mac as well. I'm not sure what info you might need, so here's a system property dump from an Eclipse instance that exhibited this issue.

HTH,
Kevin

eclipse.home.location=file:/Applications/eclipse-3.6/
eclipse.launcher=/Applications/eclipse-3.6/Eclipse.app/Contents/MacOS/eclipse
eclipse.launcher.name=Eclipse
eclipse.p2.data.area=@config.dir/.p2
eclipse.product=com.aptana.rcp.product
eclipse.refreshBundles=true
eclipse.startTime=1301345179083
file.encoding=UTF-8
file.encoding.pkg=sun.io
file.separator=/
ftp.nonProxyHosts=local|*.local|169.254/16|*.169.254/16
gopherProxySet=false
java.awt.graphicsenv=apple.awt.CGraphicsEnvironment
java.awt.printerjob=apple.awt.CPrinterJob
java.class.path=/Applications/eclipse-3.6/plugins/org.eclipse.equinox.launcher_1.1.0.v20100507.jar
java.class.version=50.0
java.endorsed.dirs=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/endorsed
java.ext.dirs=/Library/Java/Extensions:/System/Library/Java/Extensions:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext
java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
java.io.tmpdir=/var/folders/Wj/WjUUfvqkENuvF3is0hSVw++++TI/-Tmp-/
java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
java.runtime.name=Java(TM) SE Runtime Environment
java.runtime.version=1.6.0_24-b07-334-10M3326
java.specification.name=Java Platform API Specification
java.specification.vendor=Sun Microsystems Inc.
java.specification.version=1.6
java.vendor=Apple Inc.
java.vendor.url=http://www.apple.com/
java.vendor.url.bug=http://bugreport.apple.com/
java.version=1.6.0_24
java.vm.info=mixed mode
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Sun Microsystems Inc.
java.vm.specification.version=1.0
java.vm.vendor=Apple Inc.
java.vm.version=19.1-b02-334
line.separator=

mrj.build=10M3326
mrj.version=1060.1.6.0_24-334
org.eclipse.debug.ui.breakpoints.toggleFactoriesUsed=false
org.eclipse.equinox.launcher.splash.location=/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp/splash.bmp
org.eclipse.equinox.simpleconfigurator.configUrl=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.equinox.simpleconfigurator/bundles.info
org.eclipse.swt.internal.carbon.smallFonts=
org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1,OSGi/Minimum-1.2,JRE-1.1,J2SE-1.2,J2SE-1.3,J2SE-1.4,J2SE-1.5,JavaSE-1.6
org.osgi.framework.language=en
org.osgi.framework.os.name=MacOSX
org.osgi.framework.os.version=10.6.7
org.osgi.framework.processor=x86-64
org.osgi.framework.system.packages=javax.accessibility,javax.activation,javax.activity,javax.annotation,javax.annotation.processing,javax.crypto,javax.crypto.interfaces,javax.crypto.spec,javax.imageio,javax.imageio.event,javax.imageio.metadata,javax.imageio.plugins.bmp,javax.imageio.plugins.jpeg,javax.imageio.spi,javax.imageio.stream,javax.jws,javax.jws.soap,javax.lang.model,javax.lang.model.element,javax.lang.model.type,javax.lang.model.util,javax.management,javax.management.loading,javax.management.modelmbean,javax.management.monitor,javax.management.openmbean,javax.management.relation,javax.management.remote,javax.management.remote.rmi,javax.management.timer,javax.naming,javax.naming.directory,javax.naming.event,javax.naming.ldap,javax.naming.spi,javax.net,javax.net.ssl,javax.print,javax.print.attribute,javax.print.attribute.standard,javax.print.event,javax.rmi,javax.rmi.CORBA,javax.rmi.ssl,javax.script,javax.security.auth,javax.security.auth.callback,javax.security.auth.kerberos,javax.security.auth.login,javax.security.auth.spi,javax.security.auth.x500,javax.security.cert,javax.security.sasl,javax.sound.midi,javax.sound.midi.spi,javax.sound.sampled,javax.sound.sampled.spi,javax.sql,javax.sql.rowset,javax.sql.rowset.serial,javax.sql.rowset.spi,javax.swing,javax.swing.border,javax.swing.colorchooser,javax.swing.event,javax.swing.filechooser,javax.swing.plaf,javax.swing.plaf.basic,javax.swing.plaf.metal,javax.swing.plaf.multi,javax.swing.plaf.synth,javax.swing.table,javax.swing.text,javax.swing.text.html,javax.swing.text.html.parser,javax.swing.text.rtf,javax.swing.tree,javax.swing.undo,javax.tools,javax.transaction,javax.transaction.xa,javax.xml,javax.xml.bind,javax.xml.bind.annotation,javax.xml.bind.annotation.adapters,javax.xml.bind.attachment,javax.xml.bind.helpers,javax.xml.bind.util,javax.xml.crypto,javax.xml.crypto.dom,javax.xml.crypto.dsig,javax.xml.crypto.dsig.dom,javax.xml.crypto.dsig.keyinfo,javax.xml.crypto.dsig.spec,javax.xml.datatype,javax.xml.namespace,javax.xml.parsers,javax.xml.soap,javax.xml.stream,javax.xml.stream.events,javax.xml.stream.util,javax.xml.transform,javax.xml.transform.dom,javax.xml.transform.sax,javax.xml.transform.stax,javax.xml.transform.stream,javax.xml.validation,javax.xml.ws,javax.xml.ws.handler,javax.xml.ws.handler.soap,javax.xml.ws.http,javax.xml.ws.soap,javax.xml.ws.spi,javax.xml.ws.wsaddressing,javax.xml.xpath,org.ietf.jgss,org.omg.CORBA,org.omg.CORBA_2_3,org.omg.CORBA_2_3.portable,org.omg.CORBA.DynAnyPackage,org.omg.CORBA.ORBPackage,org.omg.CORBA.portable,org.omg.CORBA.TypeCodePackage,org.omg.CosNaming,org.omg.CosNaming.NamingContextExtPackage,org.omg.CosNaming.NamingContextPackage,org.omg.Dynamic,org.omg.DynamicAny,org.omg.DynamicAny.DynAnyFactoryPackage,org.omg.DynamicAny.DynAnyPackage,org.omg.IOP,org.omg.IOP.CodecFactoryPackage,org.omg.IOP.CodecPackage,org.omg.Messaging,org.omg.PortableInterceptor,org.omg.PortableInterceptor.ORBInitInfoPackage,org.omg.PortableServer,org.omg.PortableServer.CurrentPackage,org.omg.PortableServer.POAManagerPackage,org.omg.PortableServer.POAPackage,org.omg.PortableServer.portable,org.omg.PortableServer.ServantLocatorPackage,org.omg.SendingContext,org.omg.stub.java.rmi,org.w3c.dom,org.w3c.dom.bootstrap,org.w3c.dom.css,org.w3c.dom.events,org.w3c.dom.html,org.w3c.dom.ls,org.w3c.dom.ranges,org.w3c.dom.stylesheets,org.w3c.dom.traversal,org.w3c.dom.views,org.w3c.dom.xpath,org.xml.sax,org.xml.sax.ext,org.xml.sax.helpers
org.osgi.framework.vendor=Eclipse
org.osgi.framework.version=1.5.0
org.osgi.supports.framework.extension=true
org.osgi.supports.framework.fragment=true
org.osgi.supports.framework.requirebundle=true
os.arch=x86_64
os.name=Mac OS X
os.version=10.6.7
osgi.arch=x86_64
osgi.bundles=reference:file:/Applications/eclipse-3.6/plugins/org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start
osgi.bundles.defaultStartLevel=4
osgi.bundlestore=/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.osgi/bundles
osgi.checkConfiguration=true
osgi.configuration.area=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/
osgi.configuration.cascaded=false
osgi.dev=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/dev.properties
osgi.framework=file:/Applications/eclipse-3.6/plugins/org.eclipse.osgi_3.6.1.R36x_v20100806.jar
osgi.framework.shape=jar
osgi.framework.version=3.6.1.R36x_v20100806
osgi.install.area=file:/Applications/eclipse-3.6/
osgi.instance.area=file:/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/
osgi.logfile=/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/.metadata/.log
osgi.manifest.cache=/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.osgi/manifests
osgi.nl=en_US
osgi.nl.user=en_US
osgi.os=macosx
osgi.splashLocation=/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp/splash.bmp
osgi.splashPath=file:/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp,file:/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp.studio
osgi.syspath=/Applications/eclipse-3.6/plugins
osgi.tracefile=/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/.metadata/trace.log
osgi.ws=cocoa
path.separator=:
socksNonProxyHosts=local|*.local|169.254/16|*.169.254/16
sun.arch.data.model=64
sun.awt.exception.handler=apple.awt.CToolkit$EventQueueExceptionHandler
sun.boot.class.path=/Applications/eclipse-3.6/plugins/org.eclipse.jdt.debug_3.6.1.v20100715_r361/jdi.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsfd.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Frameworks/JavaRuntimeSupport.framework/Resources/Java/JavaRuntimeSupport.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/ui.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/laf.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/sunrsasign.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsse.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jce.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/charsets.jar
sun.boot.library.path=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Libraries
sun.cpu.endian=little
sun.cpu.isalist=
sun.io.unicode.encoding=UnicodeLittle
sun.java.launcher=SUN_STANDARD
sun.jnu.encoding=MacRoman
sun.management.compiler=HotSpot 64-Bit Server Compiler
sun.os.patch.level=unknown
use.bundle.cache=false
user.country=US
user.dir=/Applications/eclipse-3.6/Eclipse.app/Contents/MacOS
user.home=/Users/klindsey
user.language=en
user.name=klindsey
user.timezone=America/Chicago
Show
Kevin Lindsey added a comment - Hi Charles, We're running inside of Eclipse 3.6.1. The original user was running under Windows, but I've been able to reproduce the issue here on a Mac as well. I'm not sure what info you might need, so here's a system property dump from an Eclipse instance that exhibited this issue. HTH, Kevin eclipse.home.location=file:/Applications/eclipse-3.6/ eclipse.launcher=/Applications/eclipse-3.6/Eclipse.app/Contents/MacOS/eclipse eclipse.launcher.name=Eclipse eclipse.p2.data.area=@config.dir/.p2 eclipse.product=com.aptana.rcp.product eclipse.refreshBundles=true eclipse.startTime=1301345179083 file.encoding=UTF-8 file.encoding.pkg=sun.io file.separator=/ ftp.nonProxyHosts=local|*.local|169.254/16|*.169.254/16 gopherProxySet=false java.awt.graphicsenv=apple.awt.CGraphicsEnvironment java.awt.printerjob=apple.awt.CPrinterJob java.class.path=/Applications/eclipse-3.6/plugins/org.eclipse.equinox.launcher_1.1.0.v20100507.jar java.class.version=50.0 java.endorsed.dirs=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/endorsed java.ext.dirs=/Library/Java/Extensions:/System/Library/Java/Extensions:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home java.io.tmpdir=/var/folders/Wj/WjUUfvqkENuvF3is0hSVw++++TI/-Tmp-/ java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java java.runtime.name=Java(TM) SE Runtime Environment java.runtime.version=1.6.0_24-b07-334-10M3326 java.specification.name=Java Platform API Specification java.specification.vendor=Sun Microsystems Inc. java.specification.version=1.6 java.vendor=Apple Inc. java.vendor.url=http://www.apple.com/ java.vendor.url.bug=http://bugreport.apple.com/ java.version=1.6.0_24 java.vm.info=mixed mode java.vm.name=Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name=Java Virtual Machine Specification java.vm.specification.vendor=Sun Microsystems Inc. java.vm.specification.version=1.0 java.vm.vendor=Apple Inc. java.vm.version=19.1-b02-334 line.separator= mrj.build=10M3326 mrj.version=1060.1.6.0_24-334 org.eclipse.debug.ui.breakpoints.toggleFactoriesUsed=false org.eclipse.equinox.launcher.splash.location=/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp/splash.bmp org.eclipse.equinox.simpleconfigurator.configUrl=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.equinox.simpleconfigurator/bundles.info org.eclipse.swt.internal.carbon.smallFonts= org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1,OSGi/Minimum-1.2,JRE-1.1,J2SE-1.2,J2SE-1.3,J2SE-1.4,J2SE-1.5,JavaSE-1.6 org.osgi.framework.language=en org.osgi.framework.os.name=MacOSX org.osgi.framework.os.version=10.6.7 org.osgi.framework.processor=x86-64 org.osgi.framework.system.packages=javax.accessibility,javax.activation,javax.activity,javax.annotation,javax.annotation.processing,javax.crypto,javax.crypto.interfaces,javax.crypto.spec,javax.imageio,javax.imageio.event,javax.imageio.metadata,javax.imageio.plugins.bmp,javax.imageio.plugins.jpeg,javax.imageio.spi,javax.imageio.stream,javax.jws,javax.jws.soap,javax.lang.model,javax.lang.model.element,javax.lang.model.type,javax.lang.model.util,javax.management,javax.management.loading,javax.management.modelmbean,javax.management.monitor,javax.management.openmbean,javax.management.relation,javax.management.remote,javax.management.remote.rmi,javax.management.timer,javax.naming,javax.naming.directory,javax.naming.event,javax.naming.ldap,javax.naming.spi,javax.net,javax.net.ssl,javax.print,javax.print.attribute,javax.print.attribute.standard,javax.print.event,javax.rmi,javax.rmi.CORBA,javax.rmi.ssl,javax.script,javax.security.auth,javax.security.auth.callback,javax.security.auth.kerberos,javax.security.auth.login,javax.security.auth.spi,javax.security.auth.x500,javax.security.cert,javax.security.sasl,javax.sound.midi,javax.sound.midi.spi,javax.sound.sampled,javax.sound.sampled.spi,javax.sql,javax.sql.rowset,javax.sql.rowset.serial,javax.sql.rowset.spi,javax.swing,javax.swing.border,javax.swing.colorchooser,javax.swing.event,javax.swing.filechooser,javax.swing.plaf,javax.swing.plaf.basic,javax.swing.plaf.metal,javax.swing.plaf.multi,javax.swing.plaf.synth,javax.swing.table,javax.swing.text,javax.swing.text.html,javax.swing.text.html.parser,javax.swing.text.rtf,javax.swing.tree,javax.swing.undo,javax.tools,javax.transaction,javax.transaction.xa,javax.xml,javax.xml.bind,javax.xml.bind.annotation,javax.xml.bind.annotation.adapters,javax.xml.bind.attachment,javax.xml.bind.helpers,javax.xml.bind.util,javax.xml.crypto,javax.xml.crypto.dom,javax.xml.crypto.dsig,javax.xml.crypto.dsig.dom,javax.xml.crypto.dsig.keyinfo,javax.xml.crypto.dsig.spec,javax.xml.datatype,javax.xml.namespace,javax.xml.parsers,javax.xml.soap,javax.xml.stream,javax.xml.stream.events,javax.xml.stream.util,javax.xml.transform,javax.xml.transform.dom,javax.xml.transform.sax,javax.xml.transform.stax,javax.xml.transform.stream,javax.xml.validation,javax.xml.ws,javax.xml.ws.handler,javax.xml.ws.handler.soap,javax.xml.ws.http,javax.xml.ws.soap,javax.xml.ws.spi,javax.xml.ws.wsaddressing,javax.xml.xpath,org.ietf.jgss,org.omg.CORBA,org.omg.CORBA_2_3,org.omg.CORBA_2_3.portable,org.omg.CORBA.DynAnyPackage,org.omg.CORBA.ORBPackage,org.omg.CORBA.portable,org.omg.CORBA.TypeCodePackage,org.omg.CosNaming,org.omg.CosNaming.NamingContextExtPackage,org.omg.CosNaming.NamingContextPackage,org.omg.Dynamic,org.omg.DynamicAny,org.omg.DynamicAny.DynAnyFactoryPackage,org.omg.DynamicAny.DynAnyPackage,org.omg.IOP,org.omg.IOP.CodecFactoryPackage,org.omg.IOP.CodecPackage,org.omg.Messaging,org.omg.PortableInterceptor,org.omg.PortableInterceptor.ORBInitInfoPackage,org.omg.PortableServer,org.omg.PortableServer.CurrentPackage,org.omg.PortableServer.POAManagerPackage,org.omg.PortableServer.POAPackage,org.omg.PortableServer.portable,org.omg.PortableServer.ServantLocatorPackage,org.omg.SendingContext,org.omg.stub.java.rmi,org.w3c.dom,org.w3c.dom.bootstrap,org.w3c.dom.css,org.w3c.dom.events,org.w3c.dom.html,org.w3c.dom.ls,org.w3c.dom.ranges,org.w3c.dom.stylesheets,org.w3c.dom.traversal,org.w3c.dom.views,org.w3c.dom.xpath,org.xml.sax,org.xml.sax.ext,org.xml.sax.helpers org.osgi.framework.vendor=Eclipse org.osgi.framework.version=1.5.0 org.osgi.supports.framework.extension=true org.osgi.supports.framework.fragment=true org.osgi.supports.framework.requirebundle=true os.arch=x86_64 os.name=Mac OS X os.version=10.6.7 osgi.arch=x86_64 osgi.bundles=reference:file:/Applications/eclipse-3.6/plugins/org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start osgi.bundles.defaultStartLevel=4 osgi.bundlestore=/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.osgi/bundles osgi.checkConfiguration=true osgi.configuration.area=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/ osgi.configuration.cascaded=false osgi.dev=file:/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/dev.properties osgi.framework=file:/Applications/eclipse-3.6/plugins/org.eclipse.osgi_3.6.1.R36x_v20100806.jar osgi.framework.shape=jar osgi.framework.version=3.6.1.R36x_v20100806 osgi.install.area=file:/Applications/eclipse-3.6/ osgi.instance.area=file:/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/ osgi.logfile=/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/.metadata/.log osgi.manifest.cache=/Users/klindsey/Documents/Workspaces/RED/.metadata/.plugins/org.eclipse.pde.core/Studio - Rubles/org.eclipse.osgi/manifests osgi.nl=en_US osgi.nl.user=en_US osgi.os=macosx osgi.splashLocation=/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp/splash.bmp osgi.splashPath=file:/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp,file:/Users/klindsey/Documents/Projects/red/plugins/com.aptana.rcp.studio osgi.syspath=/Applications/eclipse-3.6/plugins osgi.tracefile=/Users/klindsey/Documents/Workspaces/runtime-red-rubles.product/.metadata/trace.log osgi.ws=cocoa path.separator=: socksNonProxyHosts=local|*.local|169.254/16|*.169.254/16 sun.arch.data.model=64 sun.awt.exception.handler=apple.awt.CToolkit$EventQueueExceptionHandler sun.boot.class.path=/Applications/eclipse-3.6/plugins/org.eclipse.jdt.debug_3.6.1.v20100715_r361/jdi.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsfd.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Frameworks/JavaRuntimeSupport.framework/Resources/Java/JavaRuntimeSupport.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/ui.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/laf.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/sunrsasign.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsse.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jce.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/charsets.jar sun.boot.library.path=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Libraries sun.cpu.endian=little sun.cpu.isalist= sun.io.unicode.encoding=UnicodeLittle sun.java.launcher=SUN_STANDARD sun.jnu.encoding=MacRoman sun.management.compiler=HotSpot 64-Bit Server Compiler sun.os.patch.level=unknown use.bundle.cache=false user.country=US user.dir=/Applications/eclipse-3.6/Eclipse.app/Contents/MacOS user.home=/Users/klindsey user.language=en user.name=klindsey user.timezone=America/Chicago
Hide
Permalink
Charles Oliver Nutter added a comment -

After looking a bit, I suspect it's fine leading up to JRuby since $0 works. The string for $0 is from that input filename encoded to UTF-8 (unconditionally). For _FILE_, it looks like we're unconditionally encoding as ISO-8859-1, which probably explains the issue; I do not think Cyrillic characters can be encoded in ISO-8859-1.

I'll talk to Tom about this today and see what the proper remedy should be.

Show
Charles Oliver Nutter added a comment - After looking a bit, I suspect it's fine leading up to JRuby since $0 works. The string for $0 is from that input filename encoded to UTF-8 (unconditionally). For _ FILE _, it looks like we're unconditionally encoding as ISO-8859-1, which probably explains the issue; I do not think Cyrillic characters can be encoded in ISO-8859-1. I'll talk to Tom about this today and see what the proper remedy should be.
Hide
Permalink
Charles Oliver Nutter added a comment -

Marking a parser issue. The code in question is in ParserSupport.getttable, which returns a new FileNode like this:

return new FileNode(token.getPosition(), ByteList.create(token.getPosition().getFile()));

ByteList.create is defined thusly:

    public static ByteList create(CharSequence s) {
        return new ByteList(plain(s),false);
    }

    ...

    public static byte[] plain(CharSequence s) {
        if (s instanceof String) return encode(s, "ISO-8859-1");

        // Not a String...get it the slow way
        byte[] bytes = new byte[s.length()];
        for (int i = 0; i < bytes.length; i++) {
            bytes[i] = (byte) s.charAt(i);
        }
        return bytes;
    }

So _FILE_ is probably destroying any filenames that can't be represented in ISO-8859-1, since Java's charset logic will replace unencodable characters with something like '?'.

Show
Charles Oliver Nutter added a comment - Marking a parser issue. The code in question is in ParserSupport.getttable, which returns a new FileNode like this: return new FileNode(token.getPosition(), ByteList.create(token.getPosition().getFile())); ByteList.create is defined thusly: public static ByteList create(CharSequence s) { return new ByteList(plain(s),false); } ... public static byte[] plain(CharSequence s) { if (s instanceof String) return encode(s, "ISO-8859-1"); // Not a String...get it the slow way byte[] bytes = new byte[s.length()]; for (int i = 0; i < bytes.length; i++) { bytes[i] = (byte) s.charAt(i); } return bytes; } So _ FILE _ is probably destroying any filenames that can't be represented in ISO-8859-1, since Java's charset logic will replace unencodable characters with something like '?'.
Hide
Permalink
Thomas E Enebo added a comment -

commit 3dbefe3 should make the situation better, but _FILE_ will use Java's locale encoding which for most people will be correct. I am not sure if I should resolve this or not since I know _FILE_ is not going to be correct when filesystem does not match locale and is also not ASCII (assuming you use character encoding that accepts ascii chars – which most do).

Show
Thomas E Enebo added a comment - commit 3dbefe3 should make the situation better, but _ FILE _ will use Java's locale encoding which for most people will be correct. I am not sure if I should resolve this or not since I know _ FILE _ is not going to be correct when filesystem does not match locale and is also not ASCII (assuming you use character encoding that accepts ascii chars – which most do).
Hide
Permalink
Charles Oliver Nutter added a comment -

I'm going to optimistically say this is fixed, since we did a considerable amount of encoding work for JRuby 1.7. If this is still an issue, feel free to reopen with a reproduction.

Show
Charles Oliver Nutter added a comment - I'm going to optimistically say this is fixed, since we did a considerable amount of encoding work for JRuby 1.7. If this is still an issue, feel free to reopen with a reproduction.

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    Kevin Lindsey
Vote (1)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: