1. Upsource Support | JetBrains
  2. Community
  3. General Discussion

Starting Upsource 2.5 fails with Cassandra errors

Avatar
Bart Kummel
Created

(See also https://youtrack.jetbrains.com/issue/UP-6002)

When starting upsource, is fails with Cassandra errors, see the log below. I checked if the contents of /etc/security/limits.conf were still as described in the docs. I also tried to increase JVM memory, as described in the known issues. All seems well, but I still get the same errors.

Note that the error says "FileNotFountException", but the mentioned file is actually on the file system. The FNFE is caused by "too many open files".

Has anyone encoutered the same or similar problems? How to work around this?

[Apache Cassandra Error] 2016-01-04T11:12:18,939 [[APP-WRAPPER] Proxy 1] WARN o.a.cassandra.service.StartupChecks - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
[Apache Cassandra Error] 2016-01-04T11:12:33,770 [SSTableBatchOpen:2] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2837-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:695) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:656) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:450) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:356) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:493) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] Caused by: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2837-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error]     at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.(FileInputStream.java:138) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.(FileInputStream.java:93) ~[na:1.8.0_45]
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:94) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     ... 15 common frames omitted
[Apache Cassandra Error] 2016-01-04T11:12:33,772 [SSTableBatchOpen:2] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-907-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:695) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:656) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:450) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:356) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:493) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] Caused by: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-907-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error]     at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.(FileInputStream.java:138) ~[na:1.8.0_45]
[Apache Cassandra Error]     at java.io.FileInputStream.(FileInputStream.java:93) ~[na:1.8.0_45]
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:94) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     ... 15 common frames omitted
[Apache Cassandra Error] 2016-01-04T11:12:33,796 [SSTableBatchOpen:1] ERROR o.a.c.c.DebuggableThreadPoolExecutor - Error in ThreadPoolExecutor
[Apache Cassandra Error] java.lang.RuntimeException: java.io.FileNotFoundException: /opt/Upsource/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-2315-big-CompressionInfo.db (Too many open files)
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:121) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.0.jar:2.2.0]

############## 1474874 lines in between were skipped from the output, all messages including skipped ones might be found in log files ##############
[Apache Cassandra Error]     at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87) [cassandra-service.jar:na]
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[Apache Cassandra Error] java.lang.RuntimeException: Exit called with exit code 100
[Apache Cassandra Error]     at com.jetbrains.cassandra.service.DieExitCodeSecurityManager.checkExit(DieExitCodeSecurityManager.java:20)
[Apache Cassandra Error]     at java.lang.Runtime.exit(Runtime.java:107)
[Apache Cassandra Error]     at java.lang.System.exit(System.java:971)
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109)
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:98)
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64)
[Apache Cassandra Error]     at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:160)
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:362)
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:319)
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:505)
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:481)
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:346)
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.(Keyspace.java:274)
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:116)
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:93)
[Apache Cassandra Error]     at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726)
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280)
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159)
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383)
[Apache Cassandra Error]     at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87)
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83)
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81)
[Apache Cassandra Error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[Apache Cassandra Error]     at java.lang.Thread.run(Thread.java:745)
[Apache Cassandra Error] 2016-01-04T11:12:37,736 [[APP-WRAPPER] Proxy 1] ERROR c.j.c.s.DieExitCodeSecurityManager - Exit called with exit code 100
[Apache Cassandra Error] java.lang.RuntimeException: Exit called with exit code 100
[Apache Cassandra Error]     at com.jetbrains.cassandra.service.DieExitCodeSecurityManager.checkExit(DieExitCodeSecurityManager.java:20) ~[cassandra-service.jar:na]
[Apache Cassandra Error]     at java.lang.Runtime.exit(Runtime.java:107) [na:1.8.0_45]
[Apache Cassandra Error]     at java.lang.System.exit(System.java:971) [na:1.8.0_45]
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:98) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:160) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:362) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:319) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:505) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:481) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:346) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.(Keyspace.java:274) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:116) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:93) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:726) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:280) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:159) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:383) [apache-cassandra-2.2.0.jar:2.2.0]
[Apache Cassandra Error]     at com.jetbrains.cassandra.service.CassandraServiceMain.start(CassandraServiceMain.java:87) [cassandra-service.jar:na]
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error]     at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81) [app-wrapper.jar:0.6.419]
[Apache Cassandra Error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
[Apache Cassandra Error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[Apache Cassandra Error]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Apache Cassandra failed to start
4 comments
Avatar
Bart Kummel
Created
Comment actions Permalink

A short update: I tried to set {{nofile}} in {{/etc/security/limits.conf}} to a rediculous high value. Now I don't get the "too many open files" error, but I get a time out instead:

com.jetbrains.service.util.StatusException: Timeout running process: bin/cassandra.sh start
    at com.jetbrains.service.util.cmd.CmdUtil.executeCommandWithExitCode(CmdUtil.java:107)
    at com.jetbrains.service.util.cmd.CmdUtil.executeCommandWithExitCode(CmdUtil.java:54)
    at com.jetbrains.bundle.services.impl.CliService.executeServiceCommandWithExitCode(CliService.java:197)
    at com.jetbrains.bundle.services.impl.CliService.executeServiceCommand(CliService.java:156)
    at com.jetbrains.bundle.services.impl.CliService.doStart(CliService.java:48)
    at com.jetbrains.bundle.services.impl.ServiceBase.start(ServiceBase.java:62)
    at com.jetbrains.bundle.Services.startService(Services.java:518)
    at com.jetbrains.bundle.Services.startAllServices(Services.java:490)
    at com.jetbrains.bundle.Services.start(Services.java:468)
    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:248)
    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:208)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.jetbrains.bundle.bootstrap.Bundle$EntryPointProvider$1.invoke(Bundle.java:108)
    at com.sun.proxy.$Proxy3.start(Unknown Source)
    at com.jetbrains.bundle.bootstrap.Bundle.start(Bundle.java:54)
    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83)
    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
0
Avatar
Evgeny Pasynkov
Created
Comment actions Permalink

Are there any other problems in the logs? (especially, please check logs/cassandra/*)?

0
Avatar
Karen Jones
Created
Comment actions Permalink

Yes, we are having the same issue. We sent the logs to JetBrains and are waiting for them to get back to us.

0
Avatar
Bart Kummel
Created
Comment actions Permalink

Here's a tar.gz with the contents of the entire logs/cassandra directory



Attachment(s):
cassandralogs.tar.gz
0

Please sign in to leave a comment.