Cassandra does not flush

After installation of Upsource (vers. 2017.2.2057) and setting up 5 Projects from our SVN our server (Windows) crashed after several hours. I don't know why, but that's not the topic here.

Normally after a restart cassandra should have read in all commit logs and everything should have been there as before, but there were 17GB of commit logs and cassandra was not able to read them into memory in time (the process got killed and restarted before it was finished I suppose). The only way to restart upsource was deleting all the commit logs - which worked, but all data from the last 24 hours was gone!!
I believe this 17GB were created when Upsource started scanning our Subversion repo.

After investigating a little how cassandra works, I found that by default it should flush the commitlogs every 10 seconds - but when I flushed manually with 'nodetool -p 10023 flush' the 17GB were flushed and after killing the server again all data was still there. So I suppose cassandra does not flush correctly by itself!

This behavior is reproducible - when I restart with a backup, the commit logs start to grow again after some time and they stay that big, which makes a clean restart of Upsource impossible!

Either flushing has to work correctly or you have to give Cassandra more time to read the Logs back when starting up.

BTW: To get nodetool running, I needed to copy cassandra.in.bat manually from the cassandra distro - that should be included in upsource for it to work!

2 comments
Comment actions Permalink

Started Cassandra manually now with this message:

[Apache Cassandra Error] 2017-08-10T11:23:21,418 [[APP-WRAPPER] Proxy 1] ERROR .a.c.utils.JVMStabilityInspector - Exiting due to error while processing commit log during initialization.
[Apache Cassandra Error] org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Could not read commit log descriptor in file E:\upsource-2017.2.2057\data\cassandra\commitlog\CommitLog-6-1502355443631.log
[Apache Cassandra Error] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:153) [apache-cassandra-3.10.jar:3.10]

So it's not taking too long but can't read some file. This happens EVERY time the machine crashes. Not really what commitlogs are for...

0
Comment actions Permalink

Hi Claudius,

Nodetool is bundled with Upsource and located here: .../upsource/apps/cassandra/bin/nodetool

As for your issue, please check that cassandra directories are not indexed by antivirus or any windows services. Looks like smth is locking out your commit logs.

0

Please sign in to leave a comment.