Encoding::UndefinedConversionError when dropping a mongo database with Mongoid 3

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: JRuby 1.7.0.pre2
Fix Version/s: JRuby 1.7.0.RC1
Component/s: Encoding
Labels:
None
Environment:
Mac OS X 10.7.4 Java 1.7.0_05

Number of attachments :
0

Description

jruby-1.7.0.preview2 :003 > ::Mongoid::Sessions.default.drop
Encoding::UndefinedConversionError: Input length = 1
	from org/jruby/RubyIO.java:1400:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:235:in `handle_socket_errors'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:146:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:169:in `with_connection'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:145:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:517:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:123:in `ensure_connected'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:516:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:531:in `logging'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:515:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:504:in `process'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:70:in `command'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:366:in `refresh'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/cluster.rb:112:in `refresh'
... 4 levels...
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:108:in `with_node'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:50:in `command'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:89:in `drop'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:305:in `with'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:88:in `drop'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:130:in `drop'
	from (irb):3:in `evaluate'
	from org/jruby/RubyKernel.java:1064:in `eval'
	from org/jruby/RubyKernel.java:1389:in `loop'
	from org/jruby/RubyKernel.java:1172:in `catch'
	from org/jruby/RubyKernel.java:1172:in `catch'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:47:in `start'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:8:in `start'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands.rb:41:in `(root)'
	from org/jruby/RubyKernel.java:1018:in `require'
	from script/rails:6:in `(root)'jruby-1.7.0.preview2 :004 > 

$ jruby -v
jruby 1.7.0.preview2 (1.9.3p203) 2012-08-30 b16f8c3 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_05-b06 [darwin-x86_64]

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 11:11 AM

This is using Mongoid 3.0.5/moped 1.2.1

Show

Tim Olsen added a comment - 30/Aug/12 11:11 AM This is using Mongoid 3.0.5/moped 1.2.1

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 11:12 AM

This error does not occur with jruby-1.6.7.2 or MRI Ruby 1.9.3p0

Show

Tim Olsen added a comment - 30/Aug/12 11:12 AM This error does not occur with jruby-1.6.7.2 or MRI Ruby 1.9.3p0

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 11:25 AM - edited

I observe an interesting behavioral difference here; sockets appear to default to ASCII-8BIT unless otherwise specified:

system ~/projects/jruby $ ruby-1.9.3 -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding]"
[#<Encoding:ASCII-8BIT>, nil]
[#<Encoding:ASCII-8BIT>, nil]

system ~/projects/jruby $ ruby-1.9.3 -rsocket -e "s = File.open('build.xml'); p [s.external_encoding, s.internal_encoding]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding]"
[#<Encoding:UTF-8>, nil]
[#<Encoding:ASCII-8BIT>, nil]

Show

Charles Oliver Nutter added a comment - 30/Aug/12 11:25 AM - edited I observe an interesting behavioral difference here; sockets appear to default to ASCII-8BIT unless otherwise specified: system ~/projects/jruby $ ruby-1.9.3 -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding]" [#<Encoding:ASCII-8BIT>, nil] [#<Encoding:ASCII-8BIT>, nil] system ~/projects/jruby $ ruby-1.9.3 -rsocket -e "s = File.open('build.xml'); p [s.external_encoding, s.internal_encoding]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding]" [#<Encoding:UTF-8>, nil] [#<Encoding:ASCII-8BIT>, nil]

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 11:37 AM

Ok, I found code in MRI's rsock_init_sock, called by all socket initializers, that sets ascii8bit_binmode unconditionally using rb_io_ascii8bit_binmode(sock);.

I'm making the same change in JRuby, which may very likely knock down a lot of encoding conversion issues in the socket subsystem.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 11:37 AM Ok, I found code in MRI's rsock_init_sock, called by all socket initializers, that sets ascii8bit_binmode unconditionally using rb_io_ascii8bit_binmode(sock);. I'm making the same change in JRuby, which may very likely knock down a lot of encoding conversion issues in the socket subsystem.

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 12:26 PM

So here's a fix I made on master, though it may or may not resolve problems for you:

commit be7c85ce8bf8703a7161713355c875b6dd6dcd4b
Author: Charles Oliver Nutter <headius@headius.com>
Date:   Thu Aug 30 12:01:54 2012 -0500

    Sockets default to binmode and ascii8bit external.

:100644 100644 4fed9a0... 096a649... M	src/org/jruby/ext/socket/RubyBasicSocket.java
:000000 100644 0000000... ecf93d4... A	test/externals/ruby1.9/excludes/TestSocket_TCPSocket.rb
:100644 100644 1433ac4... ad4036b... M	test/mri.1.9.index

Since you have the ability to reproduce this, can you give master a try?

Show

Charles Oliver Nutter added a comment - 30/Aug/12 12:26 PM So here's a fix I made on master, though it may or may not resolve problems for you: commit be7c85ce8bf8703a7161713355c875b6dd6dcd4b Author: Charles Oliver Nutter <headius@headius.com> Date: Thu Aug 30 12:01:54 2012 -0500 Sockets default to binmode and ascii8bit external. :100644 100644 4fed9a0... 096a649... M src/org/jruby/ext/socket/RubyBasicSocket.java :000000 100644 0000000... ecf93d4... A test/externals/ruby1.9/excludes/TestSocket_TCPSocket.rb :100644 100644 1433ac4... ad4036b... M test/mri.1.9.index Since you have the ability to reproduce this, can you give master a try?

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 12:34 PM

Interesting. If I break inside moped, I see a difference in the internal encoding and binmode.

jruby-head:

jruby-1.7.0.preview2 :002 > ::Mongoid::Sessions.default.drop
::Mongoid::Sessions.default.drop
/Users/tim/.rvm/gems/jruby-head/gems/ruby-debug-base-0.10.4-java/lib/ruby-debug-base.rb:215 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:230
handle_socket_errors { super }
(rdb:1) p [external_encoding, internal_encoding, binmode?]
p [external_encoding, internal_encoding, binmode?]
[#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, false]

ruby-1.9.3-p0:

1.9.3p0 :001 > ::Mongoid::Sessions.default.drop
::Mongoid::Sessions.default.drop
/Users/tim/.rvm/gems/ruby-1.9.3-p0/gems/moped-1.2.1/lib/moped/connection.rb:230
handle_socket_errors { super }
(rdb:1) p [external_encoding, internal_encoding, binmode?]
p [external_encoding, internal_encoding, binmode?]
[#<Encoding:ASCII-8BIT>, nil, true]
(rdb:1)

It appears that jruby defaults to binmode false and ruby 1.9.3-p0 defaults to binmode true. Calling set_encoding('binary') in jruby does not change the binmode (I'm not sure if it's supposed to).

$ jruby -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding, s.binmode?]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding, s.binmode?]"
[nil, nil, false]
[#<Encoding:ASCII-8BIT>, nil, false]
$ ruby -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding, s.binmode?]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding, s.binmode?]"
[#<Encoding:ASCII-8BIT>, nil, true]
[#<Encoding:ASCII-8BIT>, nil, true]

Show

Tim Olsen added a comment - 30/Aug/12 12:34 PM Interesting. If I break inside moped, I see a difference in the internal encoding and binmode. jruby-head: jruby-1.7.0.preview2 :002 > ::Mongoid::Sessions.default.drop ::Mongoid::Sessions.default.drop /Users/tim/.rvm/gems/jruby-head/gems/ruby-debug-base-0.10.4-java/lib/ruby-debug-base.rb:215 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:230 handle_socket_errors { super } (rdb:1) p [external_encoding, internal_encoding, binmode?] p [external_encoding, internal_encoding, binmode?] [#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, false] ruby-1.9.3-p0: 1.9.3p0 :001 > ::Mongoid::Sessions.default.drop ::Mongoid::Sessions.default.drop /Users/tim/.rvm/gems/ruby-1.9.3-p0/gems/moped-1.2.1/lib/moped/connection.rb:230 handle_socket_errors { super } (rdb:1) p [external_encoding, internal_encoding, binmode?] p [external_encoding, internal_encoding, binmode?] [#<Encoding:ASCII-8BIT>, nil, true] (rdb:1) It appears that jruby defaults to binmode false and ruby 1.9.3-p0 defaults to binmode true. Calling set_encoding('binary') in jruby does not change the binmode (I'm not sure if it's supposed to). $ jruby -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding, s.binmode?]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding, s.binmode?]" [nil, nil, false] [#<Encoding:ASCII-8BIT>, nil, false] $ ruby -rsocket -e "s = TCPSocket.new('google.com', 80); p [s.external_encoding, s.internal_encoding, s.binmode?]; s.set_encoding('binary'); p [s.external_encoding, s.internal_encoding, s.binmode?]" [#<Encoding:ASCII-8BIT>, nil, true] [#<Encoding:ASCII-8BIT>, nil, true]

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 12:37 PM

Our messages passed each other in flight. I'll try it now!

Show

Tim Olsen added a comment - 30/Aug/12 12:37 PM Our messages passed each other in flight. I'll try it now!

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 12:49 PM

Still failing:

$ jruby --version
jruby 1.7.0.preview2 (1.9.3p203) 2012-08-30 be7c85c on Java HotSpot(TM) 64-Bit Server VM 1.7.0_05-b06 [darwin-x86_64]
$ rails c
include_class is deprecated. Use java_import.
include_class is deprecated. Use java_import.
Loading development environment (Rails 3.2.8)
jruby-1.7.0.preview2 :001 > ::Mongoid::Sessions.default.drop
Encoding::UndefinedConversionError: Input length = 1
	from org/jruby/RubyIO.java:1400:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:235:in `handle_socket_errors'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:146:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:169:in `with_connection'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:145:in `write'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:517:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:123:in `ensure_connected'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:516:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:531:in `logging'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:515:in `flush'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:504:in `process'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:70:in `command'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:366:in `refresh'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/cluster.rb:112:in `refresh'
... 4 levels...
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:108:in `with_node'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:50:in `command'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:89:in `drop'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:305:in `with'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:88:in `drop'
	from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:130:in `drop'
	from (irb):1:in `evaluate'
	from org/jruby/RubyKernel.java:1064:in `eval'
	from org/jruby/RubyKernel.java:1389:in `loop'
	from org/jruby/RubyKernel.java:1172:in `catch'
	from org/jruby/RubyKernel.java:1172:in `catch'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:47:in `start'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:8:in `start'
	from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands.rb:41:in `(root)'
	from org/jruby/RubyKernel.java:1018:in `require'
	from script/rails:6:in `(root)'jruby-1.7.0.preview2 :002 >

binmode is now true inside moped, but internal_encoding is still UTF-8 (instead of nil as in ruby-1.9.3-p0):

jruby-1.7.0.preview2 :001 > ::Mongoid::Sessions.default.drop
/Users/tim/.rvm/gems/jruby-head/gems/ruby-debug-base-0.10.4-java/lib/ruby-debug-base.rb:215 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
/Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:230
handle_socket_errors { super }
(rdb:1) p [external_encoding, internal_encoding, binmode?]
p [external_encoding, internal_encoding, binmode?]
[#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, true]

Maybe it's related to the internal encoding being UTF-8 instead of nil?

Show

Tim Olsen added a comment - 30/Aug/12 12:49 PM Still failing: $ jruby --version jruby 1.7.0.preview2 (1.9.3p203) 2012-08-30 be7c85c on Java HotSpot(TM) 64-Bit Server VM 1.7.0_05-b06 [darwin-x86_64] $ rails c include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. Loading development environment (Rails 3.2.8) jruby-1.7.0.preview2 :001 > ::Mongoid::Sessions.default.drop Encoding::UndefinedConversionError: Input length = 1 from org/jruby/RubyIO.java:1400:in `write' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:235:in `handle_socket_errors' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:229:in `write' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:146:in `write' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:169:in `with_connection' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:145:in `write' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:517:in `flush' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:123:in `ensure_connected' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:516:in `flush' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:531:in `logging' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:515:in `flush' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:504:in `process' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:70:in `command' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/node.rb:366:in `refresh' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/cluster.rb:112:in `refresh' ... 4 levels... from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:108:in `with_node' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session/context.rb:50:in `command' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:89:in `drop' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:305:in `with' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/database.rb:88:in `drop' from /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/session.rb:130:in `drop' from (irb):1:in `evaluate' from org/jruby/RubyKernel.java:1064:in `eval' from org/jruby/RubyKernel.java:1389:in `loop' from org/jruby/RubyKernel.java:1172:in `catch' from org/jruby/RubyKernel.java:1172:in `catch' from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:47:in `start' from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands/console.rb:8:in `start' from /Users/tim/.rvm/gems/jruby-head/gems/railties-3.2.8/lib/rails/commands.rb:41:in `(root)' from org/jruby/RubyKernel.java:1018:in `require' from script/rails:6:in `(root)'jruby-1.7.0.preview2 :002 > binmode is now true inside moped, but internal_encoding is still UTF-8 (instead of nil as in ruby-1.9.3-p0): jruby-1.7.0.preview2 :001 > ::Mongoid::Sessions.default.drop /Users/tim/.rvm/gems/jruby-head/gems/ruby-debug-base-0.10.4-java/lib/ruby-debug-base.rb:215 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag /Users/tim/.rvm/gems/jruby-head/gems/moped-1.2.1/lib/moped/connection.rb:230 handle_socket_errors { super } (rdb:1) p [external_encoding, internal_encoding, binmode?] p [external_encoding, internal_encoding, binmode?] [#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, true] Maybe it's related to the internal encoding being UTF-8 instead of nil?

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 12:52 PM

That's very likely. I'm not sure where it's getting set to UTF-8 though; we might need to add some logging to the encoding logic in JRuby to see where/when it is happening.

You could probably try to trace through the code and see where it actually gets set to UTF-8, from the time it is created (in create_connection) through to the write that fails.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 12:52 PM That's very likely. I'm not sure where it's getting set to UTF-8 though; we might need to add some logging to the encoding logic in JRuby to see where/when it is happening. You could probably try to trace through the code and see where it actually gets set to UTF-8, from the time it is created (in create_connection) through to the write that fails.

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 1:30 PM

The internal encoding is being set to UTF-8 inside ::TCPSocket#initialize() which I am unable to step into.

It appears to be an interaction with Rails (3.2.8). I am able to reproduce the setting of internal_encoding to UTF-8 in a fresh Rails project without moped:

[tim@Tims-iMac rails]$ rvm use jruby-head
Using /Users/tim/.rvm/gems/jruby-head
Running /Users/tim/.rvm/hooks/after_use
[tim@Tims-iMac rails]$ ruby --version
jruby 1.7.0.preview2 (1.9.3p203) 2012-08-30 be7c85c on Java HotSpot(TM) 64-Bit Server VM 1.7.0_05-b06 [darwin-x86_64]
[tim@Tims-iMac rails]$ which rails
/Users/tim/.rvm/gems/jruby-head/bin/rails
[tim@Tims-iMac rails]$ rails new jruby-example
      create  
[ ELIDED ]
Using uglifier (1.2.7) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
[tim@Tims-iMac rails]$ cd jruby-example/
[tim@Tims-iMac jruby-example]$ rails c
Loading development environment (Rails 3.2.8)
jruby-1.7.0.preview2 :001 > require 'socket'
 => true 
jruby-1.7.0.preview2 :002 > s = TCPSocket.new('google.com', 80); p s.internal_encoding
#<Encoding:UTF-8>
 => #<Encoding:UTF-8> 
jruby-1.7.0.preview2 :003 > 
[tim@Tims-iMac jruby-example]$ cd ..
[tim@Tims-iMac rails]$ rvm use jruby-1.6.7.2
Using /Users/tim/.rvm/gems/jruby-1.6.7.2
Running /Users/tim/.rvm/hooks/after_use
[tim@Tims-iMac rails]$ ruby --version
jruby 1.6.7.2 (ruby-1.9.2-p312) (2012-08-23 26e08ba) (Java HotSpot(TM) 64-Bit Server VM 1.7.0_05) [darwin-x86_64-java]
[tim@Tims-iMac rails]$ which rails
/Users/tim/.rvm/gems/jruby-1.6.7.2/bin/rails
[tim@Tims-iMac rails]$ rails new jruby-1672-example
      create  
[ ELIDED ]
Installing uglifier (1.2.7) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
[tim@Tims-iMac rails]$ cd jruby-1672-example/
[tim@Tims-iMac jruby-1672-example]$ rails c
Loading development environment (Rails 3.2.8)
jruby-1.6.7.2 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding
nil
 => nil 
jruby-1.6.7.2 :002 > 
[tim@Tims-iMac jruby-1672-example]$ cd ..
[tim@Tims-iMac rails]$ rvm use ruby-1.9.3-p0
Using /Users/tim/.rvm/gems/ruby-1.9.3-p0
Running /Users/tim/.rvm/hooks/after_use
[tim@Tims-iMac rails]$ ruby --version
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.2.0]
[tim@Tims-iMac rails]$ which rails
/Users/tim/.rvm/gems/ruby-1.9.3-p0/bin/rails
[tim@Tims-iMac rails]$ rails new ruby-193-example
      create  
[ ELIDED ]
Installing uglifier (1.2.7) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
[tim@Tims-iMac rails]$ cd ruby-193-example/
[tim@Tims-iMac ruby-193-example]$ rails c
Loading development environment (Rails 3.2.8)
1.9.3p0 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding
nil
 => nil 
1.9.3p0 :002 > ^D
[tim@Tims-iMac ruby-193-example]$

Show

Tim Olsen added a comment - 30/Aug/12 1:30 PM The internal encoding is being set to UTF-8 inside ::TCPSocket#initialize() which I am unable to step into. It appears to be an interaction with Rails (3.2.8). I am able to reproduce the setting of internal_encoding to UTF-8 in a fresh Rails project without moped: [tim@Tims-iMac rails]$ rvm use jruby-head Using /Users/tim/.rvm/gems/jruby-head Running /Users/tim/.rvm/hooks/after_use [tim@Tims-iMac rails]$ ruby --version jruby 1.7.0.preview2 (1.9.3p203) 2012-08-30 be7c85c on Java HotSpot(TM) 64-Bit Server VM 1.7.0_05-b06 [darwin-x86_64] [tim@Tims-iMac rails]$ which rails /Users/tim/.rvm/gems/jruby-head/bin/rails [tim@Tims-iMac rails]$ rails new jruby-example create [ ELIDED ] Using uglifier (1.2.7) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed. [tim@Tims-iMac rails]$ cd jruby-example/ [tim@Tims-iMac jruby-example]$ rails c Loading development environment (Rails 3.2.8) jruby-1.7.0.preview2 :001 > require 'socket' => true jruby-1.7.0.preview2 :002 > s = TCPSocket.new('google.com', 80); p s.internal_encoding #<Encoding:UTF-8> => #<Encoding:UTF-8> jruby-1.7.0.preview2 :003 > [tim@Tims-iMac jruby-example]$ cd .. [tim@Tims-iMac rails]$ rvm use jruby-1.6.7.2 Using /Users/tim/.rvm/gems/jruby-1.6.7.2 Running /Users/tim/.rvm/hooks/after_use [tim@Tims-iMac rails]$ ruby --version jruby 1.6.7.2 (ruby-1.9.2-p312) (2012-08-23 26e08ba) (Java HotSpot(TM) 64-Bit Server VM 1.7.0_05) [darwin-x86_64-java] [tim@Tims-iMac rails]$ which rails /Users/tim/.rvm/gems/jruby-1.6.7.2/bin/rails [tim@Tims-iMac rails]$ rails new jruby-1672-example create [ ELIDED ] Installing uglifier (1.2.7) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed. [tim@Tims-iMac rails]$ cd jruby-1672-example/ [tim@Tims-iMac jruby-1672-example]$ rails c Loading development environment (Rails 3.2.8) jruby-1.6.7.2 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding nil => nil jruby-1.6.7.2 :002 > [tim@Tims-iMac jruby-1672-example]$ cd .. [tim@Tims-iMac rails]$ rvm use ruby-1.9.3-p0 Using /Users/tim/.rvm/gems/ruby-1.9.3-p0 Running /Users/tim/.rvm/hooks/after_use [tim@Tims-iMac rails]$ ruby --version ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.2.0] [tim@Tims-iMac rails]$ which rails /Users/tim/.rvm/gems/ruby-1.9.3-p0/bin/rails [tim@Tims-iMac rails]$ rails new ruby-193-example create [ ELIDED ] Installing uglifier (1.2.7) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed. [tim@Tims-iMac rails]$ cd ruby-193-example/ [tim@Tims-iMac ruby-193-example]$ rails c Loading development environment (Rails 3.2.8) 1.9.3p0 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding nil => nil 1.9.3p0 :002 > ^D [tim@Tims-iMac ruby-193-example]$

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 1:48 PM

Reproduced here. Damn ye, Rails!

system ~/projects/jruby/blah $ rails c
Loading development environment (Rails 3.2.8)
irb(main):001:0> require 'socket'
=> true
irb(main):002:0> s = TCPSocket.new('google.com', 80); p s.internal_encoding
#<Encoding:UTF-8>
=> #<Encoding:UTF-8>
irb(main):003:0>

Investigating.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 1:48 PM Reproduced here. Damn ye, Rails! system ~/projects/jruby/blah $ rails c Loading development environment (Rails 3.2.8) irb(main):001:0> require 'socket' => true irb(main):002:0> s = TCPSocket.new('google.com', 80); p s.internal_encoding #<Encoding:UTF-8> => #<Encoding:UTF-8> irb(main):003:0> Investigating.

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 2:00 PM

Ok, I think I have this now.

I didn't match rb_io_ascii8bit_binmode logic exactly, and was falling back on other encoding-setting logic that defaulted internal encoding to the global internal encoding. Duplicating the proper logic from MRI required explicitly setting "read" encoding to ASCII8BIT, "write" encoding to null, and binmode and !textmode. There were a few other bits in the MRI logic that don't correspond to JRuby data structures.

Here's the commit. I'm marking this resolved proactively, since within Rails sockets do now appear to be of the proper external+internal+flags.

commit 8e77e2b90865bc30cb277579822d7855d166cc82
Author: Charles Oliver Nutter <headius@headius.com>
Date:   Thu Aug 30 13:58:01 2012 -0500

    Better match rb_io_ascii8bit_binmode logic for new sockets.

:100644 100644 b66a25a... e5829cc... M	src/org/jruby/RubyIO.java
:100644 100644 096a649... 38df404... M	src/org/jruby/ext/socket/RubyBasicSocket.java
:100644 100644 639ce0c... 5dfc952... M	src/org/jruby/util/io/OpenFile.java

Show

Charles Oliver Nutter added a comment - 30/Aug/12 2:00 PM Ok, I think I have this now. I didn't match rb_io_ascii8bit_binmode logic exactly, and was falling back on other encoding-setting logic that defaulted internal encoding to the global internal encoding. Duplicating the proper logic from MRI required explicitly setting "read" encoding to ASCII8BIT, "write" encoding to null, and binmode and !textmode. There were a few other bits in the MRI logic that don't correspond to JRuby data structures. Here's the commit. I'm marking this resolved proactively, since within Rails sockets do now appear to be of the proper external+internal+flags. commit 8e77e2b90865bc30cb277579822d7855d166cc82 Author: Charles Oliver Nutter <headius@headius.com> Date: Thu Aug 30 13:58:01 2012 -0500 Better match rb_io_ascii8bit_binmode logic for new sockets. :100644 100644 b66a25a... e5829cc... M src/org/jruby/RubyIO.java :100644 100644 096a649... 38df404... M src/org/jruby/ext/socket/RubyBasicSocket.java :100644 100644 639ce0c... 5dfc952... M src/org/jruby/util/io/OpenFile.java

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 2:33 PM

Ok. internal_encoding is now set to nil upon socket initialization for me in rails, but mongoid is still failing to drop the database.

It appears that calling set_encoding('binary') is setting the internal_encoding back to UTF-8 in jruby-head (but not ruby 1.9.3-p0):

[tim@Tims-iMac jruby-example]$ rails c
Loading development environment (Rails 3.2.8)
jruby-1.7.0.preview2 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding; s.set_encoding('binary'); p s.internal_encoding
nil
#<Encoding:UTF-8>
 => #<Encoding:UTF-8> 
jruby-1.7.0.preview2 :002 > 

[tim@Tims-iMac jruby-example]$ cd ../ruby-193-example/
[tim@Tims-iMac ruby-193-example]$ rvm use ruby-1.9.3-p0
Using /Users/tim/.rvm/gems/ruby-1.9.3-p0
Running /Users/tim/.rvm/hooks/after_use
[tim@Tims-iMac ruby-193-example]$ rails c
Loading development environment (Rails 3.2.8)
1.9.3p0 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding; s.set_encoding('binary'); p s.internal_encoding
nil
nil
 => nil

We're getting closer

Show

Tim Olsen added a comment - 30/Aug/12 2:33 PM Ok. internal_encoding is now set to nil upon socket initialization for me in rails, but mongoid is still failing to drop the database. It appears that calling set_encoding('binary') is setting the internal_encoding back to UTF-8 in jruby-head (but not ruby 1.9.3-p0): [tim@Tims-iMac jruby-example]$ rails c Loading development environment (Rails 3.2.8) jruby-1.7.0.preview2 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding; s.set_encoding('binary'); p s.internal_encoding nil #<Encoding:UTF-8> => #<Encoding:UTF-8> jruby-1.7.0.preview2 :002 > [tim@Tims-iMac jruby-example]$ cd ../ruby-193-example/ [tim@Tims-iMac ruby-193-example]$ rvm use ruby-1.9.3-p0 Using /Users/tim/.rvm/gems/ruby-1.9.3-p0 Running /Users/tim/.rvm/hooks/after_use [tim@Tims-iMac ruby-193-example]$ rails c Loading development environment (Rails 3.2.8) 1.9.3p0 :001 > require 'socket'; s = TCPSocket.new('google.com', 80); p s.internal_encoding; s.set_encoding('binary'); p s.internal_encoding nil nil => nil We're getting closer

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 2:33 PM

reopening

Show

Tim Olsen added a comment - 30/Aug/12 2:33 PM reopening

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 2:42 PM

Grr...ok, I'll investigate that scenario too.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 2:42 PM Grr...ok, I'll investigate that scenario too.

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 3:05 PM

So the rule for setting up internal/external (or read/write, in whatever parlance gets used) is that if external is set to ascii (a la set_encoding('binary') or other paths) and internal is not specified, leave internal as null. Otherwise, set internal to default internal encoding.

That logic lives in rb_io_ext_int_to_encs, which has some MRI-specific logic, but I'm attempting to duplicate at least the internal == null && external != ascii check, which makes your latest example work too.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 3:05 PM So the rule for setting up internal/external (or read/write, in whatever parlance gets used) is that if external is set to ascii (a la set_encoding('binary') or other paths) and internal is not specified, leave internal as null. Otherwise, set internal to default internal encoding. That logic lives in rb_io_ext_int_to_encs, which has some MRI-specific logic, but I'm attempting to duplicate at least the internal == null && external != ascii check, which makes your latest example work too.

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 3:13 PM

Try again?

commit 290099647ac6a3ac1d7e6f50f5a820dbf2819be2
Author: Charles Oliver Nutter <headius@headius.com>
Date:   Thu Aug 30 15:12:08 2012 -0500

    Reinstate ext != ascii check for IO set_encoding logic. JRUBY-6871

:100644 100644 e5829cc... 7d64db1... M	src/org/jruby/RubyIO.java
:100644 100644 5f090da... 693f902... M	src/org/jruby/util/io/EncodingOption.java

At least we seem to be circling in on the right logic, and I'm not just hacking one-off fixes in.

Show

Charles Oliver Nutter added a comment - 30/Aug/12 3:13 PM Try again? commit 290099647ac6a3ac1d7e6f50f5a820dbf2819be2 Author: Charles Oliver Nutter <headius@headius.com> Date: Thu Aug 30 15:12:08 2012 -0500 Reinstate ext != ascii check for IO set_encoding logic. JRUBY-6871 :100644 100644 e5829cc... 7d64db1... M src/org/jruby/RubyIO.java :100644 100644 5f090da... 693f902... M src/org/jruby/util/io/EncodingOption.java At least we seem to be circling in on the right logic, and I'm not just hacking one-off fixes in.

Hide

Permalink

Tim Olsen added a comment - 30/Aug/12 3:31 PM

Fixed! Thank you for the quick turnaround!

Show

Tim Olsen added a comment - 30/Aug/12 3:31 PM Fixed! Thank you for the quick turnaround!

Hide

Permalink

Charles Oliver Nutter added a comment - 30/Aug/12 11:05 PM

Hooray!

Show

Charles Oliver Nutter added a comment - 30/Aug/12 11:05 PM Hooray!

People

Assignee:

Charles Oliver Nutter

Reporter:

Tim Olsen

Vote (0)

Watch (3)

Dates

Created:

30/Aug/12 11:10 AM

Updated:

30/Aug/12 11:05 PM

Resolved:

30/Aug/12 3:31 PM