JRuby

SystemStackError 'stack level too deep' exceptions in deployed GF Rails application

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Not A Bug
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: Rails WAR Deployment
  • Labels:
    None
  • Environment:
    JRuby 1.1.6 / Rails 2.0.2 / Glassfish v2ur2 / Java 6.0 (HotSpot)
  • Number of attachments :
    0

Description

Not sure where to post this, but thought I would try here, as I think this could be stemming from a larger runtime issue... would love to hear any feedback from the team...

We have a largish (60+ models, 300+ migrations) Rails 2.0.2 application that we've deployed on Glassfish v2 on JRuby 1.1.6. So far we haven't really hit any issues w/our move from MRI to JRuby – kudos! – except (seemingly) a chronic issue with some actions blowing the stack & popping SystemStackError 'stack level too deep' exceptions. In some instances, these actions just seem to hang the GF request, raising no runtime exception at all. We've tried changing our -Xss setting for our GF process to as high as 10m & can still reproduce this behavior. It seems almost all of these are POST/write-heavy actions rather than GET/read-only.

I should also mention that we are deploying a fairly non-trivial rails app – most requests can trigger several observer/AR callbacks often across a couple class hierarchies & we also do a fair bit of dynamic runtime extension (Module#extend on each controller instance) to mix-in behavior according to what kind of request we are serving. These things together could possibly blow the stack (esp if chained together by callbacks), but as I mentioned above, changes to stack size always seem to yield the same result. Also, we've never seen this on MRI – and we've been running this application in production for over a year now.

We are thinking of upgrading to 2.2 and seeing if we can repro but thought I would check here first to see if this could possibly be a symptom of a larger issue w/the runtime.

Here is a link to a some sample stack traces: http://pastie.org/345302

Our JVM options for our GF server are (as specified in the admin console):
-server -XX:+UseParallelOldGC -XX:ParallelGCThreads=4 -XX:+DisableExplicitGC -XX:NewRatio=2 -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m -Xss10m

And we are running a static pool of 2 runtimes.

Activity

Hide
Mike Wadhera added a comment -

Charles, went ahead & tried running under trunk with $DEBUG = true. At first I was getting LoadErrors when visiting pages, but it turned out that moving the $DEBUG declaration after the rails initializer block fixed it.

We are now getting stranger errors:
http://pastie.org/345908

It looks like the "too many arguments for format string" is a known issue with debug mode in 2.x rails:
http://dev.rubyonrails.org/ticket/8970

Any subsequent requests to actions that raise that ArgumentError will have LoadErrors (as illustrated in that paste).

Is there another way to trigger the full java stack dump without using $DEBUG?

Show
Mike Wadhera added a comment - Charles, went ahead & tried running under trunk with $DEBUG = true. At first I was getting LoadErrors when visiting pages, but it turned out that moving the $DEBUG declaration after the rails initializer block fixed it. We are now getting stranger errors: http://pastie.org/345908 It looks like the "too many arguments for format string" is a known issue with debug mode in 2.x rails: http://dev.rubyonrails.org/ticket/8970 Any subsequent requests to actions that raise that ArgumentError will have LoadErrors (as illustrated in that paste). Is there another way to trigger the full java stack dump without using $DEBUG?
Hide
Charles Oliver Nutter added a comment -

Mike: I believe those errors you see are not actually impacting execution; they're just extra (too much?) logging of load processes internal to JRuby. Have you tried investigating further, and if so have you made any progress tracking down the issue?

Show
Charles Oliver Nutter added a comment - Mike: I believe those errors you see are not actually impacting execution; they're just extra (too much?) logging of load processes internal to JRuby. Have you tried investigating further, and if so have you made any progress tracking down the issue?
Hide
Tiago Bastos added a comment -

I'm with the same error using glassfigh gem: http://pastie.org/365596

Show
Tiago Bastos added a comment - I'm with the same error using glassfigh gem: http://pastie.org/365596
Hide
Mike Wadhera added a comment -

Charles: The $DEBUG was impacting the rails runtime – see the ArgumentError stack dump pasted above. You ended up sending me a patch that dumped the entire java stack on overflows, which we used to diagnose the root problem without using $DEBUG. Maybe I should open a separate bug about using $DEBUG w/Rails...

After inspecting the unwound java stack, we found that this exception was caused by a circular YAML.dump call in the FeedTools gem. The solution was to remove the YAML.dump override from FT. Charles, you mentioned I opened a bug about this. That bug is here: http://jira.codehaus.org/browse/JRUBY-3269 Given that, you can go ahead and close this.

Show
Mike Wadhera added a comment - Charles: The $DEBUG was impacting the rails runtime – see the ArgumentError stack dump pasted above. You ended up sending me a patch that dumped the entire java stack on overflows, which we used to diagnose the root problem without using $DEBUG. Maybe I should open a separate bug about using $DEBUG w/Rails... After inspecting the unwound java stack, we found that this exception was caused by a circular YAML.dump call in the FeedTools gem. The solution was to remove the YAML.dump override from FT. Charles, you mentioned I opened a bug about this. That bug is here: http://jira.codehaus.org/browse/JRUBY-3269 Given that, you can go ahead and close this.
Hide
Hiro Asari added a comment -

Resolving based on Mike's comment on January 20.

Show
Hiro Asari added a comment - Resolving based on Mike's comment on January 20.

People

Vote (1)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: