Archiva reaches fd limit and keeps complaining of 'too many open files'

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Duplicate
  • Affects Version/s: 1.3.1
  • Fix Version/s: None
  • Component/s: indexing
  • Labels:
    None
  • Environment:

Description

Let me start by saying that I looked at MRM-1097 but it seems to be a different problem because there they say that the accumulating file descriptors were sockets and here they are index files.

Being more specific, our Archiva is accumulating several open file descriptors to already deleted files. In less than one hour it can accumulate as much as 1000 such file descriptors, quickly reaching the process file descriptor limit which we already increaded from 4K to 12K in an attempt to delay the problem.

This are the kind of descriptors I'm talking about:

# lsof -p 13052|grep -F '.cfs (deleted)'|head
java    13052 root  188u   REG       8,49  6622810   7393646 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkvx.cfs (deleted)
java    13052 root  190u   REG       8,49  6622878   1572928 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw3.cfs (deleted)
java    13052 root  191u   REG       8,49  6622878   1572928 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw3.cfs (deleted)
java    13052 root  192u   REG       8,49  6622878   1572928 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw3.cfs (deleted)
java    13052 root  193u   REG       8,49  6622812   7393642 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkv7.cfs (deleted)
java    13052 root  194r   REG       8,49  6622878   1572928 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw3.cfs (deleted)
java    13052 root  195u   REG       8,49  6622878   1572918 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw5.cfs (deleted)
java    13052 root  197u   REG       8,49  6622813   7393638 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkv9.cfs (deleted)
java    13052 root  198u   REG       8,49  6622878   1572928 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkw3.cfs (deleted)
java    13052 root  200u   REG       8,49  6622812   7393630 /l/home3/archiva/dsso/data/repositories/internal/inhouse/snapshots/.indexer/_1tkvl.cfs (deleted)

# lsof -p 13052|grep -c -F '.cfs (deleted)'
817
  1. GZip Archive
    archiva-dsso.csv.gz
    03/Sep/10 9:36 AM
    0.6 kB
    Gustavo Chaves
  2. GZip Archive
    scan.log.gz
    02/Sep/10 2:04 PM
    8 kB
    Gustavo Chaves

Issue Links

duplicates

Bug - A problem which impairs or prevents the functions of the product. MRM-1097 Error 500 "too many open files"

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.
is related to

Bug - A problem which impairs or prevents the functions of the product. MRM-1097 Error 500 "too many open files"

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Brett Porter added a comment -

we're due to review the use of the indexer in the next release, so hopefully we can pick this up at the time.

Do you have some data on how many artifacts you are processing through the indexer and how often?

Show
Brett Porter added a comment - we're due to review the use of the indexer in the next release, so hopefully we can pick this up at the time. Do you have some data on how many artifacts you are processing through the indexer and how often?
Hide
Permalink
Andreyev Dias de Melo added a comment -

We have 5253 jars, 12752 poms. Repositories is re-scanned in 0 0 * * * or 0 0 0/2 * *.

Show
Andreyev Dias de Melo added a comment - We have 5253 jars, 12752 poms. Repositories is re-scanned in 0 0 * * * or 0 0 0/2 * *.
Hide
Permalink
Gustavo Chaves added a comment -

I'm attaching the scan.log.gz file which was extracted from the archiva logs with this command:

grep -A16 'Repository Dir ' archiva.log >scan.log

In it you can see how many times the scan process was performed in the last days.

Show
Gustavo Chaves added a comment - I'm attaching the scan.log.gz file which was extracted from the archiva logs with this command: 
grep -A16 'Repository Dir ' archiva.log >scan.log
In it you can see how many times the scan process was performed in the last days.
Hide
Permalink
Gustavo Chaves added a comment -

I'm attaching archiva-dsso.csv.gz. It contains a record of the number of those strange open file descriptors that I captured each 10 minutes since yesterday.

Show
Gustavo Chaves added a comment - I'm attaching archiva-dsso.csv.gz. It contains a record of the number of those strange open file descriptors that I captured each 10 minutes since yesterday.
Hide
Permalink
Andreyev Dias de Melo added a comment -

This problem looks like this http://www.mail-archive.com/java-user@lucene.apache.org/msg29216.html.

I'll try to upgrade only lucene's jar to an early version.

Show
Andreyev Dias de Melo added a comment - This problem looks like this http://www.mail-archive.com/java-user@lucene.apache.org/msg29216.html. I'll try to upgrade only lucene's jar to an early version.
Hide
Permalink
Brett Porter added a comment -

in the end I think we identified that the other problem was also related to the indexing, which should be fixed

Show
Brett Porter added a comment - in the end I think we identified that the other problem was also related to the indexing, which should be fixed

People

  • Assignee:
    Brett Porter
    Reporter:
    Gustavo Chaves
Vote (1)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: