Digest::Base stores in memory all bytes passed to the digest, causing OutOfMemoryError

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.1.4
  • Fix Version/s: JRuby 1.2
  • Component/s: Extensions
  • Labels:
    None

Description

Digest::Base is currently implemented by appending to a StringBuffer.

This will cause a certain OutOfMemoryError if you try to digest a file which approaches half of your memory limit, which makes it unsuitable for the kind of task for which digests are commonly used (i.e. finding the digest of files for the sake of comparison.)

Instead of using an internal StringBuffer it should just update the MessageDigest directly.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Charles Oliver Nutter added a comment -

Can you come up with a patch for this? It's probably not hard, but we're pretty slammed with 1.1.5 preparations right now.

Show
Charles Oliver Nutter added a comment - Can you come up with a patch for this? It's probably not hard, but we're pretty slammed with 1.1.5 preparations right now.
Hide
Permalink
Trejkaz added a comment -

Time seems a bit tight right now for me. If I get some free time in the next week or so I'll take a shot at it.

Assuming there are specs already, anyway, as that's the only way I'll know how it's really supposed to work.

Show
Trejkaz added a comment - Time seems a bit tight right now for me. If I get some free time in the next week or so I'll take a shot at it. Assuming there are specs already, anyway, as that's the only way I'll know how it's really supposed to work.
Hide
Permalink
Charles Oliver Nutter added a comment -

Fixed in r9110.

It turned out that all the MessageDigest objects from JDK are cloneable; so instead of storing all data in-memory and re-digesting every time, we can continue to update the same digest and clone it immediately before producing intermediate results. This lets us produce intermediate results without finishing off the digest.

Performance is likely improved as well, since it won't have to digest all the data anew every time a digest is requested. I also made a few minor perf cleanups to eliminate extra back-and-forth to Java Strings.

Show
Charles Oliver Nutter added a comment - Fixed in r9110. It turned out that all the MessageDigest objects from JDK are cloneable; so instead of storing all data in-memory and re-digesting every time, we can continue to update the same digest and clone it immediately before producing intermediate results. This lets us produce intermediate results without finishing off the digest. Performance is likely improved as well, since it won't have to digest all the data anew every time a digest is requested. I also made a few minor perf cleanups to eliminate extra back-and-forth to Java Strings.

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    Trejkaz
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: