castor

Castor has a memory leak which becomes apparent when unmarshalling large XML files.

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Not A Bug
  • Affects Version/s: 1.2
  • Fix Version/s: 1.3.1
  • Component/s: XML
  • Labels:
    None
  • Number of attachments :
    0

Description

Castor appears to have a memory leak when unmarshalling XML files. Large files cause OutOfMemoryErrors. This is a critical problem and renders Castor unusable for high volume data processing.

Turning on the debug for castor package, we were able to see the log messages from the Castor classes. Mostly the log messages were for adding descriptors using different methods (by package name, by element name...) to resolve the mapping, in case if a mapping for a node element is not found. This happens when the element name in the xml does not have a direct mapping in the mapping files.

For example: An XML element called person-info exists in the xml, but we don't have a class mapped directly to this element. Instead the location attribute is used to bind a different object to the XML element.

Also looked in to the XMLClassDescriptorResolverImpl Castor class where these messages are logged. These missing classes were added to an ArrayList and we witnessed it growing larger and larger over a period of time as the process parses more and more data.

We used JProbe to match the above behavior with the snapshot taken. The snapshot revealed the ArrayList object with a very large size in the heap memory.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Werner Guttmann added a comment -

Ed, a colleague of mine and myself fixed a problem closely related to this about 6 month ago. The required patch had been committed to SVN trunk at about the same time. In other words, Castor 1.3 should have a fix for that problem (or a very similar problem). If I remember correctly, the information about the fact that a ClassDescriptor for a given XML artefact was not defined by the means of a mapping or a generated class descriptor was not added to a cache (where it should have been).

Can I please ask you to try your test(s) against 1.3 and report back your findings.

In addition, what ArrayList object precisely was growing in size ad infinitum ?

Show
Werner Guttmann added a comment - Ed, a colleague of mine and myself fixed a problem closely related to this about 6 month ago. The required patch had been committed to SVN trunk at about the same time. In other words, Castor 1.3 should have a fix for that problem (or a very similar problem). If I remember correctly, the information about the fact that a ClassDescriptor for a given XML artefact was not defined by the means of a mapping or a generated class descriptor was not added to a cache (where it should have been). Can I please ask you to try your test(s) against 1.3 and report back your findings. In addition, what ArrayList object precisely was growing in size ad infinitum ?
Hide
Permalink
Werner Guttmann added a comment -

Ed, did switching to Castor make any difference ? If not, I'd like to know about this.

Having said that, are you in a position to attach an XML file that I can use to replay this problem (incl. mapping file, etc.) ?

Show
Werner Guttmann added a comment - Ed, did switching to Castor make any difference ? If not, I'd like to know about this. Having said that, are you in a position to attach an XML file that I can use to replay this problem (incl. mapping file, etc.) ?
Hide
Permalink
Ed Wallen added a comment -

Werner, our initial tests show that the problem may have been fixed with the 1.3 version of Castor. We are doing more extensive load testing now to make sure that we can no longer reproduce the problem. I'll let you know what we find. Thanks!

Show
Ed Wallen added a comment - Werner, our initial tests show that the problem may have been fixed with the 1.3 version of Castor. We are doing more extensive load testing now to make sure that we can no longer reproduce the problem. I'll let you know what we find. Thanks!
Hide
Permalink
Werner Guttmann added a comment -

Thanks, Ed, for the feedback. Just ping us whenever you've got questions. I do remember, though, that I did run into heavy performance problems closely related to big XML documents (500+ MB) right after 1.2 had been made available.

Show
Werner Guttmann added a comment - Thanks, Ed, for the feedback. Just ping us whenever you've got questions. I do remember, though, that I did run into heavy performance problems closely related to big XML documents (500+ MB) right after 1.2 had been made available.
Hide
Permalink
Ed Wallen added a comment -

Werner, the load testing cycles have completed and the bug that was there related to the memory leak has definitely been fixed. Thanks so much for your help!

Show
Ed Wallen added a comment - Werner, the load testing cycles have completed and the bug that was there related to the memory leak has definitely been fixed. Thanks so much for your help!
Hide
Permalink
Werner Guttmann added a comment -

Thanks, Ed.

Show
Werner Guttmann added a comment - Thanks, Ed.

People

  • Assignee:
    Werner Guttmann
    Reporter:
    Ed Wallen
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: