Starting Upsource 2.5 fails with Cassandra errors Follow
(See also https://youtrack.jetbrains.com/issue/UP-6002)
When starting upsource, is fails with Cassandra errors, see the log below. I checked if the contents of /etc/security/limits.conf were still as described in the docs. I also tried to increase JVM memory, as described in the known issues. All seems well, but I still get the same errors.
Note that the error says "FileNotFountException", but the mentioned file is actually on the file system. The FNFE is caused by "too many open files".
Has anyone encoutered the same or similar problems? How to work around this?
[Apache Cassandra Error] 2016-01-04T11:12:18,939 [[APP-WRAPPER] Proxy 1] WARN o.a.cassandra.service.StartupChecks - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
[Apache Cassandra Error] 2016-01-04T11:12:33,770 [SSTableBatchOpen:2] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2837-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:695) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:656) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:450) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:356) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:493) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] Caused by: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2837-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error] at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.(FileInputStream.java:138) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.(FileInputStream.java:93) ~[na:1.8.0_45]
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:94) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] ... 15 common frames omitted
[Apache Cassandra Error] 2016-01-04T11:12:33,772 [SSTableBatchOpen:2] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-907-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:695) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:656) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:450) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:356) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:493) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] Caused by: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-907-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error] at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.(FileInputStream.java:138) ~[na:1.8.0_45]
[Apache Cassandra Error] at java.io.FileInputStream.(FileInputStream.java:93) ~[na:1.8.0_45]
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:94) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] ... 15 common frames omitted
[Apache Cassandra Error] 2016-01-04T11:12:33,796 [SSTableBatchOpen:1] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2315-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]
############## 1474874 lines in between were skipped from the output, all messages including skipped ones might be found in log files ##############
[Apache Cassandra Error] at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87) [cassandra-service.jar:na]
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] java.lang.RuntimeException: Exit called with exit code 100
[Apache Cassandra Error] at com.jetbrains.cassandra.service.DieExitCodeSecurityManager.checkExit(DieExitCodeSecurityManager.java:20)
[Apache Cassandra Error] at java.lang.Runtime.exit(Runtime.java:107)
[Apache Cassandra Error] at java.lang.System.exit(System.java:971)
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109)
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:98)
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64)
[Apache Cassandra Error] at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:160)
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:362)
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:319)
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:505)
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:481)
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:346)
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.(Keyspace.java:274)
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:116)
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:93)
[Apache Cassandra Error] at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726)
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280)
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159)
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383)
[Apache Cassandra Error] at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87)
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83)
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81)
[Apache Cassandra Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[Apache Cassandra Error] at java.lang.Thread.run(Thread.java:745)
[Apache Cassandra Error] 2016-01-04T11:12:37,736 [[APP-WRAPPER] Proxy 1] ERROR c.j.c.s.DieExitCodeSecurityManager - Exit called with exit code 100
[Apache Cassandra Error] java.lang.RuntimeException: Exit called with exit code 100
[Apache Cassandra Error] at com.jetbrains.cassandra.service.DieExitCodeSecurityManager.checkExit(DieExitCodeSecurityManager.java:20) ~[cassandra-service.jar:na]
[Apache Cassandra Error] at java.lang.Runtime.exit(Runtime.java:107) [na:1.8.0_45]
[Apache Cassandra Error] at java.lang.System.exit(System.java:971) [na:1.8.0_45]
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:98) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:160) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:362) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:319) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:505) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:481) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:346) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.(Keyspace.java:274) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:116) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:93) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error] at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87) [cassandra-service.jar:na]
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error] at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
[Apache Cassandra Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Apache Cassandra failed to start
Please sign in to leave a comment.
A short update: I tried to set {{nofile}} in {{/etc/security/limits.conf}} to a rediculous high value. Now I don't get the "too many open files" error, but I get a time out instead:
Are there any other problems in the logs? (especially, please check logs/cassandra/*)?
Yes, we are having the same issue. We sent the logs to JetBrains and are waiting for them to get back to us.
Here's a tar.gz with the contents of the entire logs/cassandra directory
Attachment(s):
cassandralogs.tar.gz