Unable to lock JVM memory (ENOMEM)

Overview:

This article describes one of the errors you might notice in Upsource logs. 

Symptoms: 

Upsource continuously shows "Indexing in progress" for some (all) revisions, or Apache Cassandra database gets restarted on a regular basis. Going through Upsource logs you encounter the following error: 

Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.

or

..Too many open files..

Solution:

Insufficient resource limits may result in this kind of error. To mitigate this we recommend setting:

  • maximum open files to 100000
  • memory locking and address space limit to unlimited.
  • number of processes to 32768

You can do it by adding the following lines to the /etc/security/limits.conf file:

 

* - memlock unlimited

* - nofile 100000

* - nproc 32768

* - as unlimited

1 comment

For a machine that has more users, the entries should perhaps look more like:

 

upsource - memlock unlimited
upsource - nofile 65536
upsource - nproc 32768
upsource - as unlimited

 

Depending on how Upsource is started, it may then be necessary to actually raise these limit before starting, so the process does not inherit values from your shell. That can be done by adding ulimit -l unlimited and such to /etc/init.d/upsource.

Edited by Michael Piefel
0

Please sign in to leave a comment.

Have more questions?

Submit a request