Unable to lock JVM memory (ENOMEM)

Permanently deleted user

Updated January 13, 2021 17:21

Overview:

This article describes one of the errors you might notice in Upsource logs.

Symptoms:

Upsource continuously shows "Indexing in progress" for some (all) revisions, or Apache Cassandra database gets restarted on a regular basis. Going through Upsource logs you encounter the following error:

Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.

..Too many open files..

Solution:

Insufficient resource limits may result in this kind of error. To mitigate this we recommend setting:

maximum open files to 100000
memory locking and address space limit to unlimited.
number of processes to 32768

You can do it by adding the following lines to the /etc/security/limits.conf file:

* - memlock unlimited

* - nofile 100000

* - nproc 32768

* - as unlimited

2 comments

Michael Piefel

Updated April 21, 2016 07:13

For a machine that has more users, the entries should perhaps look more like:

upsource - memlock unlimited
upsource - nofile 65536
upsource - nproc 32768
upsource - as unlimited

Depending on how Upsource is started, it may then be necessary to actually raise these limit before starting, so the process does not inherit values from your shell. That can be done by adding ulimit -l unlimited and such to /etc/init.d/upsource.

Edited by Michael Piefel April 21, 2016 07:13

Zak

Updated January 13, 2021 17:21

That problem in docker isn't trivial. I did next after correct limits as described above. It worked.

In my case:

stop docker service
edit `/lib/systemd/system/docker.service`. Add line "LimitMEMLOCK=infinity" and apply that by `systemctl daemon-reload`

add to my docker-compose file ulimits configuration:

 ulimits:
   memlock:
     soft: -1
     hard: -1

Edited by Zak January 13, 2021 17:21

Please sign in to leave a comment.

Have more questions?

Submit a request