Details
-
Type:
Bug
-
Status:
Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: JRuby-OSSL 0.7.3
-
Fix Version/s: JRuby 1.7.1
-
Component/s: OpenSSL, Ruby 1.9.2, Ruby 1.9.3
-
Labels:None
-
Environment:Hidevsrikarunyan@~$ jruby -v
jruby 1.6.0.RC2 (ruby 1.8.7 patchlevel 330) (2011-02-09 5434c72) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_22) [darwin-x86_64-java]
Error opening script file: /Users/vsrikarunyan/rubygems (No such file or directory)
vsrikarunyan@~$ jruby -S gem environment
RubyGems Environment:
- RUBYGEMS VERSION: 1.5.0
- RUBY VERSION: 1.8.7 (2011-02-09 patchlevel 330) [java]
- INSTALLATION DIRECTORY: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8
- RUBY EXECUTABLE: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/bin/jruby
- EXECUTABLE DIRECTORY: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/bin
- RUBYGEMS PLATFORMS:
- ruby
- universal-java-1.6
- GEM PATHS:
- /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8
- /opt/local/lib/ruby1.9/gems/1.9.1
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- "install" => "--no-rdoc --no-ri --env-shebang"
- "update" => "--no-rdoc --no-ri --env-shebang"
- REMOTE SOURCES:
- http://rubygems.org/
vsrikarunyan@~$ jruby -S gem list --local
*** LOCAL GEMS ***
abstract (1.0.0)
actionmailer (3.0.4)
actionpack (3.0.4)
activemodel (3.0.4)
activerecord (3.0.4)
activerecord-jdbc-adapter (1.1.1)
activerecord-jdbcmysql-adapter (1.1.1)
activeresource (3.0.4)
activesupport (3.0.4)
archive-tar-minitar (0.5.2)
arel (2.0.8)
bouncy-castle-java (1.5.0145.2)
builder (2.1.2)
bundler (1.0.10)
columnize (0.3.2)
erubis (2.6.6)
gem_plugin (0.2.3)
gemcutter (0.6.1)
i18n (0.5.0)
jdbc-mysql (5.1.13)
jruby-openssl (0.7.3)
json (1.5.1 java)
linecache19 (0.5.11)
mail (2.2.15)
mime-types (1.16)
minitest (1.6.0)
mongrel (1.1.5 java)
oauth (0.4.4)
polyglot (0.3.1)
rack (1.2.1)
rack-mount (0.6.13)
rack-test (0.5.7)
rails (3.0.4)
railties (3.0.4)
rake (0.8.7)
rdoc (2.5.8)
ruby-debug-base19 (0.11.24)
ruby-debug-ide (0.4.16)
ruby_core_source (0.1.4)
rubygems-update (1.5.2)
sources (0.0.1)
thor (0.14.6)
treetop (1.4.9)
tzinfo (0.3.24)Showvsrikarunyan@~$ jruby -v jruby 1.6.0.RC2 (ruby 1.8.7 patchlevel 330) (2011-02-09 5434c72) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_22) [darwin-x86_64-java] Error opening script file: /Users/vsrikarunyan/rubygems (No such file or directory) vsrikarunyan@~$ jruby -S gem environment RubyGems Environment: - RUBYGEMS VERSION: 1.5.0 - RUBY VERSION: 1.8.7 (2011-02-09 patchlevel 330) [java] - INSTALLATION DIRECTORY: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8 - RUBY EXECUTABLE: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/bin/jruby - EXECUTABLE DIRECTORY: /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/bin - RUBYGEMS PLATFORMS: - ruby - universal-java-1.6 - GEM PATHS: - /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8 - /opt/local/lib/ruby1.9/gems/1.9.1 - GEM CONFIGURATION: - :update_sources => true - :verbose => true - :benchmark => false - :backtrace => false - :bulk_threshold => 1000 - "install" => "--no-rdoc --no-ri --env-shebang" - "update" => "--no-rdoc --no-ri --env-shebang" - REMOTE SOURCES: - http://rubygems.org/ vsrikarunyan@~$ jruby -S gem list --local *** LOCAL GEMS *** abstract (1.0.0) actionmailer (3.0.4) actionpack (3.0.4) activemodel (3.0.4) activerecord (3.0.4) activerecord-jdbc-adapter (1.1.1) activerecord-jdbcmysql-adapter (1.1.1) activeresource (3.0.4) activesupport (3.0.4) archive-tar-minitar (0.5.2) arel (2.0.8) bouncy-castle-java (1.5.0145.2) builder (2.1.2) bundler (1.0.10) columnize (0.3.2) erubis (2.6.6) gem_plugin (0.2.3) gemcutter (0.6.1) i18n (0.5.0) jdbc-mysql (5.1.13) jruby-openssl (0.7.3) json (1.5.1 java) linecache19 (0.5.11) mail (2.2.15) mime-types (1.16) minitest (1.6.0) mongrel (1.1.5 java) oauth (0.4.4) polyglot (0.3.1) rack (1.2.1) rack-mount (0.6.13) rack-test (0.5.7) rails (3.0.4) railties (3.0.4) rake (0.8.7) rdoc (2.5.8) ruby-debug-base19 (0.11.24) ruby-debug-ide (0.4.16) ruby_core_source (0.1.4) rubygems-update (1.5.2) sources (0.0.1) thor (0.14.6) treetop (1.4.9) tzinfo (0.3.24)
-
Number of attachments :1
Description
/app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/bin/jruby --1.9 -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /Users/vsrikarunyan/workspace/mine/lab/linkedin.rb
/app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/protocol.rb:135:in `rbuf_fill': undefined method `read_nonblock' for #<OpenSSL::SSL::SSLSocket:0x246adb31> (NoMethodError)
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/protocol.rb:116:in `readuntil'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/protocol.rb:126:in `readline'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:2211:in `read_status_line'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:2200:in `read_new'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:1183:in `transport_request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:1169:in `request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:1162:in `request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:627:in `start'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/1.9/net/http.rb:1160:in `request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8/gems/oauth-0.4.4/lib/oauth/consumer.rb:164:in `request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8/gems/oauth-0.4.4/lib/oauth/consumer.rb:197:in `token_request'
from /app/jruby/JRuby.Framwork/Versions/1.6.0.RC2/lib/ruby/gems/1.8/gems/oauth-0.4.4/lib/oauth/consumer.rb:139:in `get_request_token'
from /Users/vsrikarunyan/workspace/mine/lab/linkedin.rb:21:in `(root)'
from org/jruby/RubyKernel.java:1073:in `load19'
from -e:1:in `(root)'
Process finished with exit code 1
-
- jruby-openssl-0.7.5.dev.gem
- 22/Nov/11 2:50 AM
- 666 kB
- Hiroshi Nakamura
Issue Links
- is duplicated by
-
JRUBY-6368
undefined method `read_nonblock' on jruby new app
-
-
JRUBY-5970
gem push in 1.9 mode exits with OpenSSL errors
-
-
JRUBY-6308
OpenSSL::SSL::SSLSocket#read_nonblock broken with bundler/gem plugins in Ruby 1.9 mode
-
-
JRUBY-6331
undefined method `read_nonblock' for #<OpenSSL::SSL::SSLSocket:0x771eeb>
-
Activity
I have created a git project with instructions on how to reproduce:
https://github.com/kbaum/undefined_method_read_nonblock
Let me know if you need anything else.
Thanks!
It's easy to reproduce, just fetch any https website under 1.9 mode. The error message isn't lying. There is no #read_nonblock on SSLSocket.
The 1.8 version of #rbuf_fill used #sysread, which is defined.
In addition to fetching https websites, I've also run into this issue when trying to send SSL email, e.g. via Gmail; it's a blocker in my current project which has a stand-alone mail queue.
It was a blocker for me too, so i monkey patched the one method to look like the ruby 1.8 version and it seems to work for me:
Same issue here. Thanks at @Karl Baum, your monkey patch seems to get me running.
This will be a bit bigger than just a one-off fix. We could add a temporary alias from new read methods to sysread, since in theory they should function the same other than buffering and blocking, but the IO bits of SSLSocket need a refactoring.
I checked on the methods we define and the ones defined in 1.9. Here's what we're missing:
rb_define_method(cSSLSocket, "connect_nonblock", ossl_ssl_connect_nonblock, 0);
rb_define_method(cSSLSocket, "accept_nonblock", ossl_ssl_accept_nonblock, 0);
rb_define_private_method(cSSLSocket, "sysread_nonblock", ossl_ssl_read_nonblock, -1);
rb_define_private_method(cSSLSocket, "syswrite_nonblock", ossl_ssl_write_nonblock, 1);
rb_define_method(cSSLSocket, "session_reused?", ossl_ssl_session_reused, 0);
rb_define_method(cSSLSocket, "session=", ossl_ssl_set_session, 1);
So basically we need nonblocking versions of connect, accept, sysread, and syswriter, though the latter two are not public (perhaps used from Ruby code?)
The odd thing here is that there is no read_nonblock, and if I check SSLSocket's methods under 1.9 it is indeed not defined:
~/projects $ ruby1.9.2 -ropenssl -e "p OpenSSL::SSL::SSLSocket.methods.include? 'read_nonblock'" false
So the question is why it's attempting to call read_nonblock against SSLSocket. I suspect there may be some coercion missing that wraps SSLSocket in an IO, which then delegates to sysread_nonblock and friends. Hmm.
Ok, I think I found it. read_nonblock and many other IOish methods are defined in the .rb files for OpenSSL. The buffering.rb file defines the Buffering module, which has all the methods in question build atop sysread/syswrite/sysread_nonblock/syswrite_nonblock.
So the simple answer is that we need to implement the missing nonblock methods above and then update the OpenSSL .rb files to 1.9's latest. The methods will exist in 1.8 mode, but there's really no way around that.
I'd like to help close this bug.
I have forked jruby-ossl and implemented sysread_nonblock and syswrite_nonblock. The code is available here: https://github.com/duncanmak/jruby-ossl
I ran Karl Baum's test case, and on my local machine, here's the output I see:
duncan@morpheme:~/git/undefined_method_read_nonblock (master)$ JRUBY_OPTS='--1.9' ./test.rb Net::POPAuthenticationError: -ERR authentication failed check_response_auth at /home/duncan/.rvm/rubies/jruby-1.6.4/lib/ruby/1.9/net/pop.rb:984
Which looks right to me.
The methods remaining are: connect_nonblock, accept_nonblock, session_reused? and session=.
connect_nonblock/accept_nonblock:
I have been going over the C implementation, I think what I need to do is to raise IO::WaitReadable/IO::WaitWritable instead of waitSelect(SelectionKey.OP_READ | SelectionKey.OP_WRITE); – I don't really know what I should do here, though.
session_reused?:
From the bit of research that I have done, it looks like SSL sessions are always reusable (unless they're closed?), so is it safe to just return true?
session=:
It doesn't look like the SSL implementation in Java allow for this sort of session swapping. Could I just throw UnsupportedOperationException here?
Any suggestions / code review will be greatly appreciated.
Thanks!
Guys, sorry for keeping this issue long time. Duncan, thanks for looking this issue. I hope I can help you to implement 1.9 support features.
At first, we should design how we implement -1.8/-1.9 supports in 1 gem. Updating lib/openssl/* with CRuby 1.9 lib would break --1.8 mode... It's best to integrate jruby-ossl into ext/openssl of JRuby (JRUBY-5035) but the branch is on hiatus, just since I don't have enough time...
> connect_nonblock/accept_nonblock: > I have been going over the C implementation, I think what I need to do is to raise > IO::WaitReadable/IO::WaitWritable instead of waitSelect(SelectionKey.OP_READ | SelectionKey.OP_WRITE); > – I don't really know what I should do here, though.
Basically true, and a little bit extra-handling for before STARTTLS. Can you check if test/openssl/test_pair.rb and test_ssl.rb of CRuby runs fine with your branch?
And I think I should fix several nonblocking IO of JRuby itself first. (Try test/ruby/test_io.rb of CRuby repo against JRuby --1.9 mode.)
> session_reused?: > From the bit of research that I have done, it looks like SSL sessions are always reusable (unless > they're closed?), so is it safe to just return true? > session=: > It doesn't look like the SSL implementation in Java allow for this sort of session swapping. Could > I just throw UnsupportedOperationException here?
Yeah, I don't think we can make SSLSession persistent in Java. Session cache is done by JCE in Java, so SSLSession (JRUBY-4371) would be a read only object.
Any suggestions how we go forward?
I tried test_pair.rb and test_ssl.rb:
duncan@morpheme:~/Downloads/ruby-1.9.2-p290/test/openssl$ jruby-git --1.9 test_ssl.rb -v
Loaded suite test_ssl
Started
OpenSSL::TestSSL#test_client_auth: 5.81 s: E
OpenSSL::TestSSL#test_client_session: 5.14 s: E
OpenSSL::TestSSL#test_connect_and_close: 5.14 s: E
OpenSSL::TestSSL#test_ctx_setup: 0.04 s: .
OpenSSL::TestSSL#test_not_started_session: 0.42 s: E
OpenSSL::TestSSL#test_parallel: 5.12 s: E
OpenSSL::TestSSL#test_post_connection_check: 5.10 s: E
OpenSSL::TestSSL#test_read_and_write: 5.12 s: E
OpenSSL::TestSSL#test_server_session: NameError: uninitialized constant OpenSSL::SSL::Session
const_missing at org/jruby/RubyModule.java:2569
session at /home/duncan/.rvm/gems/jruby-1.6.4/gems/jruby-openssl-0.7.5.dev/lib/openssl/ssl.rb:129
test_server_session at test_ssl.rb:499
call at org/jruby/RubyProc.java:270
call at org/jruby/RubyProc.java:258
server_loop at test_ssl.rb:80
call at org/jruby/RubyProc.java:270
call at org/jruby/RubyProc.java:224
duncan@morpheme:~/Downloads/ruby-1.9.2-p290/test/openssl$ jruby-git --1.9 test_pair.rb -v Loaded suite test_pair Started OpenSSL::TestEOF1#test_eof_0: 0.34 s: E OpenSSL::TestEOF1#test_eof_0_rw: 0.00 s: . OpenSSL::TestEOF1#test_eof_1: 0.10 s: E OpenSSL::TestEOF1#test_eof_2: 0.09 s: E OpenSSL::TestEOF1#test_eof_3: 0.09 s: E OpenSSL::TestEOF2#test_eof_0: 0.11 s: E OpenSSL::TestEOF2#test_eof_0_rw: 0.00 s: . OpenSSL::TestEOF2#test_eof_1: 0.10 s: E OpenSSL::TestEOF2#test_eof_2: 0.10 s: E OpenSSL::TestEOF2#test_eof_3: 0.10 s: E OpenSSL::TestPair#test_connect_accept_nonblock: 0.01 s: E OpenSSL::TestPair#test_getc: 0.10 s: E OpenSSL::TestPair#test_puts_empty: 0.12 s: E OpenSSL::TestPair#test_puts_meta: 0.11 s: E OpenSSL::TestPair#test_read_nonblock:
And it hangs on test_read_nonblock - so I guess there's more work to be done.
I made a few more changes, and now in my branch, I see 3 remaining errors when I run test_pair.rb from CRuby's test/openssl directory:
duncan@morpheme:~/Downloads/ruby-1.9.2-p290/test/openssl$ jruby --1.9 test_pair.rb -v
Loaded suite test_pair
Started
OpenSSL::TestEOF1#test_eof_0: 0.71 s: .
OpenSSL::TestEOF1#test_eof_0_rw: 0.00 s: .
OpenSSL::TestEOF1#test_eof_1: 0.87 s: .
OpenSSL::TestEOF1#test_eof_2: 0.11 s: .
OpenSSL::TestEOF1#test_eof_3: 0.11 s: .
OpenSSL::TestEOF2#test_eof_0: 0.46 s: .
OpenSSL::TestEOF2#test_eof_0_rw: 0.00 s: .
OpenSSL::TestEOF2#test_eof_1: 0.89 s: .
OpenSSL::TestEOF2#test_eof_2: 0.11 s: .
OpenSSL::TestEOF2#test_eof_3: 0.10 s: .
OpenSSL::TestPair#test_connect_accept_nonblock: 0.01 s: E
OpenSSL::TestPair#test_getc: 0.09 s: .
OpenSSL::TestPair#test_puts_empty: 0.10 s: .
OpenSSL::TestPair#test_puts_meta: 0.11 s: .
OpenSSL::TestPair#test_read_nonblock: 0.15 s: E
OpenSSL::TestPair#test_readall: 0.10 s: .
OpenSSL::TestPair#test_readline: 0.11 s: .
OpenSSL::TestPair#test_readpartial: 0.11 s: .
OpenSSL::TestPair#test_write_nonblock: 0.10 s: .
OpenSSL::TestPair#test_write_nonblock_with_buffered_data: 0.11 s: E
Finished in 4.397000 seconds.
1) Error:
test_connect_accept_nonblock(OpenSSL::TestPair):
NoMethodError: undefined method `connect_address' for #<TCPServer:0x111f2041>
test_pair.rb:200:in `__ensure__'
org/jruby/RubyBasicObject.java:1685:in `__send__'
2) Error:
test_read_nonblock(OpenSSL::TestPair):
OpenSSL::SSL::SSLError: read would raise
org/jruby/ext/openssl/SSLSocket.java:562:in `sysread_nonblock'
/home/duncan/.rvm/gems/jruby-1.6.4@ossl-dev/gems/jruby-openssl-0.7.5.dev/lib/openssl/buffering.rb:145:in `read_nonblock'
test_pair.rb:157:in `test_read_nonblock'
test_pair.rb:46:in `__ensure__'
test_pair.rb:45:in `__ensure__'
test_pair.rb:145:in `test_read_nonblock'
org/jruby/RubyBasicObject.java:1685:in `__send__'
3) Error:
test_write_nonblock_with_buffered_data(OpenSSL::TestPair):
OpenSSL::SSL::SSLError: write would raise
org/jruby/ext/openssl/SSLSocket.java:600:in `syswrite_nonblock'
/home/duncan/.rvm/gems/jruby-1.6.4@ossl-dev/gems/jruby-openssl-0.7.5.dev/lib/openssl/buffering.rb:292:in `write_nonblock'
test_pair.rb:185:in `test_write_nonblock_with_buffered_data'
test_pair.rb:46:in `__ensure__'
test_pair.rb:45:in `__ensure__'
test_pair.rb:183:in `test_write_nonblock_with_buffered_data'
org/jruby/RubyBasicObject.java:1685:in `__send__'
20 tests, 96 assertions, 0 failures, 3 errors, 0 skips
To fix the remaining two errors, we need to make IO.select do what SSLSocket.waitSelect does.
If I understand correctly, that means that all internal (not imported) IO must be unblocking as same as CRuby 1.9... But I think you're right. That's the way to go.
More people are running JRuby in 1.9 mode and are running into this issue. It would be nice to get this minimally working, at least.
Here's the branch Duncan fixes some issue for 1.9.
https://github.com/duncanmak/jruby-ossl/commits/implement-nonblock-with-dual-lib-support
The branch also includes my change for 1.8/1.9 double support in one gem: https://github.com/jruby/jruby-ossl/compare/310dd6311d...38d00c7cd7
To support all of OpenSSL + non-blocking issue, we need to modify IO layer of JRuby itself. But the Duncan's branch would be good to release as an initial try for 1.9 mode.
Nick, how do you think my 1.8/1.9 support part? If it's really the way to go, I'll merge Duncan's branch to master and push a new JRuby-OSSL gem.
NaHi, everything looks great. I had this issue on my agenda for this week already, so if you can make the merges and push a new gem that would be wonderful. Let me know if I can help in any way.
bump
Hiroshi, Nick, anything we can help with here? Currently a blocker for a few projects we're looking at with JRuby.
Hi, sorry that I keep you waiting.
Would you please try the attached gem built from Duncanmak's development branch? I've not yet tested this on other than Sun/Oracle's 6u29 though...
Yeah, this is bad. Blows up aws-sdk, which uses SSL to speak with Amazon REST APIs.
The referenced monkey patch appears to work; testing the attached gem will take a little longer.
Duncan's attached jruby-openssl 0.7.5.dev gem failed for me.
System: (Mac OSX Lion / JRuby 1.6.5 64bit).
Test - invoking Stripe's HTTPS-based RESTful payment service.
JRuby (ruby 1.9.2) The 0.7.5.dev gem leaves JRuby in a hung state with high CPU usage when attempting an SSL request.
JRuby (ruby 1.9.2) The 0.7.4 gem produces the "undefined method 'read_nonblock'" issue
JRuby (ruby 1.8.7) - The 0.7.5.dev gem works correctly.
JRuby (ruby 1.8.7) - The 0.7.4 gem works correctly
Native Ruby 1.9.2 executes the code successfully.
Let me know if there's anything I can do to help develop / test a fix on OSX.
I've pulled the "implement-nonblock-with-dual-lib-support" (0.7.5.dev) branch from github and run the tests using JRuby in 1.9 mode on my Mac (OSX Lion / JRuby 1.6.5 64bit / Java build 1.6.0_26-b03-383-11A511) The tests that cause the process to hang with high CPU are:
test_ssl.rb :
test_jruby_4826
test_integration.rb :
test_ca_path_name
test_ssl_verify
test_pathlen_does_not_appear
There's a mix of pass/fail/error for the other tests on the platform. I hope to get around to looking more closely at the results in another week or so.
igrigorik
{ ~/Desktop/undefined_method_read_nonblock } > ruby -vjruby 1.7.0.dev (ruby-1.9.3-p0) (2011-12-19 cdb3e75) (OpenJDK 64-Bit Server VM 1.7.0-u2-b21) [darwin-amd64-java]
igrigorik { ~/Desktop/undefined_method_read_nonblock }
> gem list jruby-openssl
jruby-openssl (0.7.5.dev, 0.7.4)
Trying to run a simple net/https connection, which hangs the process indefinitely: https://gist.github.com/d034dd170b02037c0a9f
Installed jruby-head is up to date as of yesterday.
Ben: Any luck from your side for this?
I applied an one-off fix to jruby-1_6 for this issue.
commit 5dbab19d45b89b959532684c71b98e73e254b39f
Author: Hiroshi Nakamura <nahi@ruby-lang.org>
Date: Tue Dec 27 00:36:58 2011 +0900
[1.9] One-off fix for read_nonblock CPU 100% issue
Replace "read_nonblock + IO.select timeout" in 1.9 stdlib with
"Timeout.timeout + sysread" in 1.8's one. With current JRuby,
IO.select([io], nil, nil, timeout) could return promptly even if io is
not readable. (I'm not sure the condition when it happens at this
moment.)
Because of this IO.select behavior, rbuf_fill in 1.9/net/protocol.rb
fails to busy loop of io.read_nonblock and IO.select. As a workaround,
this commit replaces rbuf_fill of 1.9 stdlib with 1.8's one which uses
Timeout + sysread instead.
sysread_nonblock + IO.select is introduced from perf concern. JRuby has
a better Timeout logic so this change should not affect users.
I apply this change only for jruby-1_6 branch. For 1.7, I'll try to
solve this issue by fixing nonblocking IO at master.
Can someone try again with jruby-1.6.6.dev at http://ci.jruby.org/job/jruby-dist-release/lastSuccessfulBuild/artifact/ ?
If you're using master, please apply the same patch for a workaround: https://github.com/jruby/jruby/commit/5dbab19d45b89b959532684c71b98e73e254b39f
Hiroshi's commit got reverted a few days later by
https://github.com/jruby/jruby/commit/be3f90ff7b48ad54662a20f8360ff37139ab75fb
so it is broken again
I cherry-picked NaHi's commit again to the 1.6 branch: e29ca705a4.
We will put JRuby 1.6.6 out next week, which includes a number of other 1.9 fixes. It would be great if we could get jruby-openssl out too with its 1.9 fixes.
Nahi: What can I do to help get jossl released?
Nahi - yeah, let me know also. I'm free this weekend, and would love to put in the finishing touches.
Hiro: Did this change make it into our Ruby stdlib fork? If not, it will get overwritten again.
Thanks Charles and Duncan. I report the status here.
I've already merged Duncan's fixes into master branch of jruby-ossl. New release of jruby-ossl gem includes both 1.8 and 1.9 libraries and part of features should work fine on 1.9 mode, too.
For read_nonblock issue, the fix is needed for jruby's lib/ruby/1.9/net/protocol.rb (5dbab19d45b89b959532684c71b98e73e254b39f for jruby master) for a while. Eventually I need to investigate non-blocking IO of JRuby IO subsystem, but it would need more time. For more details, please see the commit log of about workaround fix. Sorry I can't spent enough time to investigate this issue.
I didn't merge the fix to our Ruby stdlib fork, since I thought it should be one-off fix for jruby-1_6 (I don't want it affect to master.) Since it would need more time to fix, I'll update our stdlib fork later.
I was going to release 0.7.5 but I cannot share time for releasing. I'll push it in weekend. Here's History.txt for 0.7.5.
This release improved 1.9 mode support with help of Duncan Mak <duncan@earthaid.net>. Now jruby-ossl gem includes both 1.8 and 1.9 libraries and part of features should work fine on 1.9 mode, too. - JRUBY-6270: Wrong keyUsage check for SSL server - JRUBY-6260: OpenSSL::ASN1::Integer#value incompatibility - JRUBY-6044: Improve Ecrypted RSA/DSA pem support - JRUBY-5972: Allow to load/dump empty PKCS7 data - JRUBY-5833: Fix X509Name handling; X509Name RDN can include multiple elements - JRUBY-5362: Improved 1.9 support - JRUBY-4992: Warn if loaded by non JRuby interpreter
Duncan, 1.9 support is not yet finished. Would you please see TODO-1_9-support.txt in the repo and try to solve some issue? You can run 1.9 tests by; jruby --1.9 -S rake test
Please bear in mind that you need to apply 5dbab19d45b89b959532684c71b98e73e254b39f to jruby for read_nonblock issue.
Commenting so I get notified of updates. FWIW I'm using Karl's monkey-patch for the time being.
I re-cherry-picked (is that a word?) NaHi's commit to the master branch: 9db86c6.
FWIW I also pulled the change into our stdlib fork so it won't go away. jruby-ruby_1_9_3@a637894 and jruby-ruby_1_9_2@f844a14.
I have been looking into using syswrite_nonblock as well, and I don't think that this implementation is correct. It raises an error each time it is called, whose .inspect returns this: #<#<Class:0x2940943>: write would raise>
It looks like it is trying to raise an IO::WaitWritable each time it is run, but this is not correct either. It looks like this is trying to repeat the following C code:
if (errno == EWOULDBLOCK || errno == EAGAIN)
rb_mod_sys_fail(rb_mWaitWritable, "write would block");
It should only raise a WaitWritable (or WaitReadable for read_nonblock) in the case of EWOULDBLOCK/EAGAIN.
This patch is more along the lines of my understanding of what read/write_nonblock are supposed to do.
Looking more at CRuby's implementation, I am more convinced the current implementation is wrong. write_would_block is only called if ssl_get_error returns SSL_ERROR_WANT_WRITE and likewise for read_would_block.
Re-implemented read_nonblock and write_nonblock here: https://github.com/matthauck/jruby-ossl/commit/154d6b316867af142d7aedba997b286ecf5506e8
test_pair now succeeds all read_nonblock/write_nonblock tests.
jruby --1.9 ~/Ruby/ruby-1.9.2-p290/test/openssl/test_pair.rb -v
Loaded suite /Users/mhauck/Ruby/ruby-1.9.2-p290/test/openssl/test_pair
Started
OpenSSL::TestEOF1#test_eof_0: 0.41 s: .
OpenSSL::TestEOF1#test_eof_0_rw: 0.01 s: .
OpenSSL::TestEOF1#test_eof_1: 0.18 s: .
OpenSSL::TestEOF1#test_eof_2: 0.02 s: .
OpenSSL::TestEOF1#test_eof_3: 0.02 s: .
OpenSSL::TestEOF2#test_eof_0: 0.10 s: .
OpenSSL::TestEOF2#test_eof_0_rw: 0.00 s: .
OpenSSL::TestEOF2#test_eof_1: 0.18 s: .
OpenSSL::TestEOF2#test_eof_2: 0.02 s: .
OpenSSL::TestEOF2#test_eof_3: 0.02 s: .
OpenSSL::TestPair#test_connect_accept_nonblock: 0.01 s: E
OpenSSL::TestPair#test_getc: 0.02 s: .
OpenSSL::TestPair#test_puts_empty: 0.02 s: .
OpenSSL::TestPair#test_puts_meta: 0.02 s: .
OpenSSL::TestPair#test_read_nonblock: 0.03 s: .
OpenSSL::TestPair#test_readall: 0.02 s: .
OpenSSL::TestPair#test_readline: 0.02 s: .
OpenSSL::TestPair#test_readpartial: 0.02 s: .
OpenSSL::TestPair#test_write_nonblock: 0.03 s: .
OpenSSL::TestPair#test_write_nonblock_with_buffered_data: 0.02 s: .
Finished in 1.185000 seconds.
1) Error:
test_connect_accept_nonblock(OpenSSL::TestPair):
NoMethodError: undefined method `connect_address' for #<TCPServer:0x399b08cc>
/Users/mhauck/Ruby/ruby-1.9.2-p290/test/openssl/test_pair.rb:200:in `__ensure__'
org/jruby/RubyBasicObject.java:1688:in `__send__'
20 tests, 101 assertions, 0 failures, 1 errors, 0 skips
Thanks Matt, test result improvement sounds nice! I'm going to check your branch and merge next week. Sorry for the delay.
I just ran into undefined method write_nonblock when upgrading from jruby-openssl 0.7.4 to 0.7.5 when running jruby in 1.8 mode. I made an issue for it here: https://jira.codehaus.org/browse/JRUBY-6391. Feel free to mark it as a duplicate if you are already aware of it.
Thanks Matt for updated patch. https://github.com/jruby/jruby-ossl/pull/14 and https://github.com/matthauck/jruby-ossl/commit/154d6b316867af142d7aedba997b286ecf5506e8 is the latest patch, right?
I confirmed the fix, but having checked the patch, I'm thinking the fix should be done against jruby itself.
The original waitSelect() in openssl was a copy of RubyThread.select as I wrote in comment. It was a temporary fix for SSLSocket, until we refactor RubyIO for better non-blocking support. Anyway I need to investigate why waitSelect() was broken, because received a bug report that timeout does not work against SSLSocket on jruby 1.6.5 + jruby-openssl 0.7.5.
Do you think you can apply your patch against jruby and how it works? Or shall I try it?
Hmm, I'm not immediately sure how RubyThread and RubyIO would integrate together. I can can take a look tonight perhaps, but if you know what to do with it, then you can try it.
For blocking IO in Ruby, IO.select needs to change Thread state to "sleep" and take care interruption by other Thread (like timeout.)
For no-blocking IO, we can use selectNow() as you patched, but there are some common logic that should be shared with blocking IO (that's the reason why you change waitSelect(), not just using selectNow() in *_nonblock() methods I guess.)
I think it's a good chance to merge jruby-ossl's waitSelect() with RubyThread.select() again, on the basis of the fact that we have now a workaround (net/http modification.)
I'll try it by myself, too.
Bump! This is a big blocker for us.
I do not see undefined method, but instead always get error (read would raise), when I'm expected read would block. Karl's patch does not seem to help me with this case either.
Matt, I did not have success with your patch - resulted in some strange errors (happy to provide detail if needed).
Anything I can do to help?
Current status: net/protocol and dependents (e.g. net/http) should work now by rbuf_fill patch but SSLSocket#read_nonblock is still not yet implemented properly.
I wrote "I'll try" but I still cannot promise the date I can try because of my primary job.
Now JRuby-OSSL is under the process of merging back to JRuby (JRUBY-5035, master branch for 1.7) and it might help to investigate and fix the problem (non-blocking and SSLSocket) at master branch. I keep the "Assignee" of this ticket but feel free to take over. I think I can review fixes proposed.
I'll write again when I can try to investigate this issue again.
Any ETA for a fix? This is a big blocker for us at Travis CI atm.
FYI: This issue surfaces when using net-http-persistent.
The nonblock methods were added as of 1.7pre2, so I'm going to call this resolved. The provided test case (kbaum repository) appears to work properly now.
There may still be issues with the implementation of the various nonblocking methods, but since this method was about their lack of existence and inability to use a library that needs them, I think it's fine to resolve. New bugs should be filed as new issues.
I am skeptical that this has truly been implemented as it still uses the previous erroneous implementation. I am getting "Write would block" on every write_nonblock and "Read would block" on every read_nonblock call...
I would really like to re-open this bug rather than open a new one since it is the same issue. Raising an exception every time a method is called is no better than not defining the method to begin with... especially since it appears that in jruby 1.7 we are not able to replace jruby-openssl with our own forked version, a fork much easier to maintain than a whole jruby fork...
You're right...I am ashamed that we claimed it was working based on my half-assed attempt.
We can do this right, but it will take more time to understand the lifecycle of an SSL handshake and how it plays into nonblocking/partial IO. Reopening and marking for JRuby 1.7.1.
I have pushed a quick fix for this to the post17 branch that will go into 1.7.1. For a simple script (below) it works properly now. I'm going to proceed to running the openssl tests to work out more issues.
Commit:
commit b4af8a56aea8d3b8031bef7fdd364565e543f491
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Oct 24 13:12:45 2012 -0500
Fix read_nonblock to actually attempt the read, rather than raise.
:100644 100644 7c175fc... 47d3ec3... M src/org/jruby/ext/openssl/SSLSocket.java
Script:
require 'openssl' require 'socket' Thread.abort_on_exception = true server = TCPServer.new(10000) s = nil Thread.new { s = server.accept } c = TCPSocket.new('localhost', 10000) 1 until s c.puts('hello') puts s.gets # connection is established ss = OpenSSL::SSL::SSLSocket.new(s) cs = OpenSSL::SSL::SSLSocket.new(c) Thread.new do while true begin puts ss.read_nonblock(1000) rescue IO::WaitReadable puts "waiting for read" sleep 0.5 retry end end end loop do sleep 1 cs.write('hello') end
Ok, with the above commit and one additional, I'm able to run the read_nonblock test from test_ssl (along with a few others) and I'd really like to say we're good here now.
commit f3c7847e16f5ce026d96be4259cf4e41034c51bf
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Oct 24 14:51:28 2012 -0500
More fixes for SSLSocket.
* Treat TCPServer#shutdown as unsupported by raising ENOTCONN
* Unmask several test_ssl tests that pass now.
:100644 100644 de79e18... 9520b70... M src/org/jruby/ext/socket/RubyTCPServer.java
:100644 100644 55c1485... 34411ac... M test/externals/ruby1.9/excludes/OpenSSL/TestSSL.rb
commit b4af8a56aea8d3b8031bef7fdd364565e543f491
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Oct 24 13:12:45 2012 -0500
Fix read_nonblock to actually attempt the read, rather than raise.
:100644 100644 7c175fc... 47d3ec3... M src/org/jruby/ext/openssl/SSLSocket.java
What about for writing? Not sure if this got lost in the conversation, but cf. my previous suggested implementation: https://github.com/matthauck/jruby-ossl/commit/154d6b316867af142d7aedba997b286ecf5506e8
Matt: Your logic looks sound. I probably just missed it in previous conversations. I'll look at getting it (or a form of it) incorporated.
My earlier commits did half of what yours does, and then the following two commits incorporate your additional changes plus some tweaks to allow interrupting read/write threads.
commit d7deae6aa78797515bcc86223b8e0fa1a1537272
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Oct 24 19:44:03 2012 -0500
Tweaks to previous commit to support interrupting blocked threads.
:100644 100644 da88636... c095582... M src/org/jruby/ext/openssl/SSLSocket.java
commit 70921f420e6bb1baefbb32c2734eae108fc62e19
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Oct 24 19:24:39 2012 -0500
Incorporate Matt Hauck's SSLSocket nonblock read/write cleanups.
:100644 100644 ec64ed2... da88636... M src/org/jruby/ext/openssl/SSLSocket.java
Can you provide us with some simple cases where this is failing?