I tried different -Xms and -Xmx values, didn't seem to help.
C:\workspace\jruby_trunk\bin>gem install optiflag
Bulk updating Gem source index for: http://gems.rubyforge.org
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.jruby.util.ByteList.grow(ByteList.java:537)
at org.jruby.util.ByteList.append(ByteList.java:116)
at org.jruby.util.ByteList.append(ByteList.java:122)
at org.jvyamlb.ScannerImpl.update(ScannerImpl.java:330)
at org.jvyamlb.ScannerImpl.ensure(ScannerImpl.java:350)
at org.jvyamlb.ScannerImpl.scanLineBreak(ScannerImpl.java:581)
at org.jvyamlb.ScannerImpl.scanToNextToken(ScannerImpl.java:562)
at org.jvyamlb.ScannerImpl.fetchMoreTokens(ScannerImpl.java:488)
at org.jvyamlb.ScannerImpl.peekToken(ScannerImpl.java:411)
at org.jvyamlb.ParserImpl$ProductionEnvironment.produce(ParserImpl.java:355)
at org.jvyamlb.ParserImpl.parseStreamNext(ParserImpl.java:730)
at org.jvyamlb.ParserImpl.peekEvent(ParserImpl.java:681)
at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:161)
at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:167)
at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:167)
at org.jvyamlb.ComposerImpl.composeDocument(ComposerImpl.java:100)
at org.jvyamlb.ComposerImpl.getNode(ComposerImpl.java:76)
at org.jvyamlb.BaseConstructorImpl.getData(BaseConstructorImpl.java:97)
at org.jruby.RubyYAML.load(RubyYAML.java:223)
at org.jruby.RubyYAMLInvoker$load_FS1.call(Unknown Source)
at org.jruby.runtime.callback.FastInvocationCallback.execute(FastInvocationCallback.java:49)
at org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:70)
at org.jruby.runtime.CallAdapter$DefaultCallAdapter.call(CallAdapter.java:167)
at org.jruby.runtime.CallAdapter$DefaultCallAdapter.call(CallAdapter.java:108)
at org.jruby.evaluator.ASTInterpreter.callNode(ASTInterpreter.java:561)
at org.jruby.evaluator.ASTInterpreter.evalInternal(ASTInterpreter.java:208)
at org.jruby.evaluator.ASTInterpreter.orNode(ASTInterpreter.java:1335)
at org.jruby.evaluator.ASTInterpreter.evalInternal(ASTInterpreter.java:338)
at org.jruby.evaluator.ASTInterpreter.eval(ASTInterpreter.java:165)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:144)
at org.jruby.runtime.CallAdapter$DefaultCallAdapter.call(CallAdapter.java:167)
at org.jruby.runtime.CallAdapter$DefaultCallAdapter.call(CallAdapter.java:108)

Orpheus:~/workspace/jruby> head bin/gem
#!/usr/bin/env jruby -J-Xmx384M
#--
# Copyright 2006 by Chad Fowler, Rich Kilmer, Jim Weirich and others.
# All rights reserved.
# See LICENSE.txt for permissions.
#++


require 'rubygems'
Gem.manage_gems
Orpheus:~/workspace/jruby> bin/gem install optiflag
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed optiflag-0.6.5
Installing ri documentation for optiflag-0.6.5...
Installing RDoc documentation for optiflag-0.6.5...

The above memory settings are the default in our gem script. As far as I can see, there is nothing bad in the YAML spec for optiflag, since I can install it just fine. Could you try again with the default settings?

gem query --remote

• • • REMOTE GEMS ***
      BBulk updating Gem source index for: http://gems.rubyforge.org
      Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
      at org.jvyamlb.ParserImpl$25.produce(ParserImpl.java:488)
      at org.jvyamlb.ParserImpl.parseStreamNext(ParserImpl.java:823)
      at org.jvyamlb.ParserImpl.peekEvent(ParserImpl.java:774)
      at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:147)
      at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:167)
      at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:167)
      at org.jvyamlb.ComposerImpl.composeNode(ComposerImpl.java:167)
      at org.jvyamlb.ComposerImpl.composeDocument(ComposerImpl.java:100)
      at org.jvyamlb.ComposerImpl.getNode(ComposerImpl.java:76)
      at org.jvyamlb.BaseConstructorImpl.getData(BaseConstructorImpl.java:97)
      at org.jruby.RubyYAML.load(RubyYAML.java:221)
      at org.jruby.RubyYAMLInvokerSload1.call(Unknown Source)

I'm willing to believe that I have an environment anomaly, but I wonder what the
heck it could be? I'm glad that Pat Cheng observed the same problem. That
suggests that there is some difference between the modules or permissions
on a developer's system, as opposed to a "user's" system. Maybe you could try
it with a clean install of jruby binaries, rather than from a development environment?

Another highly likely suspect is an old Java class that somehow sneaked into the
binaries, so that your development classes are going to the wrong offsets in the
release versions. (I've seen that sort of thing before, although one would like to
imagine that it doesn't happen with interpreted classess...)

I just tried it again with a more recent trunk, and it works for me.

The version of JvYAML shipped in 1.0 takes much more memory, and I'm not sure it will be possible to backport those changes to the 1.0-branch.

b. I question the resolution.

The resolution is given as "unable to reproduce". Yet
Pat Cheng saw the same problem with the release version, and
as far as I can tell, everyone else who tried to reproduce
it was using the trunk, rather than the release.

To my mind, the inability to browse and load gems qualifies
as a "showstopper" bug. Version 1.0 shouldn't have gone out
with that problem, and a 1.0.x patch is needed to make it
into something usable. But Ola suggests that a back port to
the 1.0 branch isn't likely to be possible, which means that
branch is dead.

I thought about downloading the trunk, but I saw a ton of
bugs, and that gave me pause. So it seems that a stable
1.1 release is quite a ways off--which is a problem, because
I would be inclined to label it as an unresolved "Blocker"
in 1.0 and 1.0.1, and declare it "resolved in 1.1".

Eric, can you reproduce this?

Hence, "cannot reproduce". Eric, if you could try again against the 1.0 branch (ASAP, we're cutting a release within hours) and try to run this again, we would appreciate it. At the moment, it seems to work fine, albeit requiring a memory bump on one or two platforms.