Startup fails due to Cassandra timeout

Hi,

I'm having some issues starting upsource on a headless Centos 6.6 box.
After unzipping the distribution, running 'upsource.sh start', and entering info like hostname and admin credentials, Upsource takes a very long time '* Starting Apache Cassandra'.

logs/upsource-stderr.log shows:

[Upsource Error] [2014-12-22 09:57:19,858]  ERROR - brains.bundle.util.BrowserUtil - Unable to open url in browser: Couldn't find web browser
[Upsource Error] log4j:WARN No appenders could be found for logger (com.jetbrains.bundle.starting_page.config.StartingPageBackendInitializer).
[Upsource Error] log4j:WARN Please initialize the log4j system properly.
[Upsource Error] log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[Upsource Error] [APP-WRAPPER] Failed to start the application: Failed to start Upsource 1.0: Timeout running process: bin/cassandra.sh start --debug
[Upsource Error] [APP-WRAPPER] com.jetbrains.launcher.StartupException: Failed to start Upsource 1.0: Timeout running process: bin/cassandra.sh start --debug
[Upsource Error]        at com.jetbrains.bundle.BundleMain.start(BundleMain.java:164)
[Upsource Error]        at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:79)
[Upsource Error]        at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:77)
[Upsource Error]        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[Upsource Error]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[Upsource Error]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Upsource Error]        at java.lang.Thread.run(Thread.java:745)
[Upsource Error] Caused by: com.jetbrains.service.util.StatusException: Timeout running process: bin/cassandra.sh start --debug
[Upsource Error]        at com.jetbrains.service.util.cmd.CmdUtil.executeCommandWithExitCode(CmdUtil.java:94)
[Upsource Error]        at com.jetbrains.bundle.services.impl.CliService.executeServiceCommandWithExitCode(CliService.java:158)
[Upsource Error]        at com.jetbrains.bundle.services.impl.CliService.executeServiceCommand(CliService.java:118)
[Upsource Error]        at com.jetbrains.bundle.services.impl.CliService.doStart(CliService.java:46)
[Upsource Error]        at com.jetbrains.bundle.services.impl.ServiceBase.start(ServiceBase.java:51)
[Upsource Error]        at com.jetbrains.bundle.Services.startService(Services.java:422)
[Upsource Error]        at com.jetbrains.bundle.Services.startAllServices(Services.java:400)
[Upsource Error]        at com.jetbrains.bundle.Services.start(Services.java:379)
[Upsource Error]        at com.jetbrains.bundle.BundleMain.start(BundleMain.java:188)
[Upsource Error]        at com.jetbrains.bundle.BundleMain.restartServices(BundleMain.java:247)
[Upsource Error]        at com.jetbrains.bundle.BundleMain.start(BundleMain.java:200)
[Upsource Error]        at com.jetbrains.bundle.BundleMain.start(BundleMain.java:158)
[Upsource Error]        ... 6 more
[Upsource Error] [2014-12-22 11:00:20,792]  ERROR - rvice.util.cmd.LogStreamThread - Error in reading stream
[Upsource Error] java.io.IOException: Stream closed
[Upsource Error]        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
[Upsource Error]        at java.io.BufferedInputStream.fill(BufferedInputStream.java:206)
[Upsource Error]        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
[Upsource Error]        at com.jetbrains.service.util.cmd.InterruptableInputStream.read(InterruptableInputStream.java:39)
[Upsource Error]        at java.io.InputStream.read(InputStream.java:170)
[Upsource Error]        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
[Upsource Error]        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
[Upsource Error]        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
[Upsource Error]        at java.io.InputStreamReader.read(InputStreamReader.java:184)
[Upsource Error]        at java.io.BufferedReader.fill(BufferedReader.java:154)
[Upsource Error]        at java.io.BufferedReader.readLine(BufferedReader.java:317)
[Upsource Error]        at java.io.BufferedReader.readLine(BufferedReader.java:382)
[Upsource Error]        at com.jetbrains.service.util.cmd.LogStreamThread.run(LogStreamThread.java:39)
[Upsource Error] [2014-12-22 11:00:20,794]  ERROR - rvice.util.cmd.LogStreamThread - Error in reading stream
[Upsource Error] java.io.IOException: Stream closed
[Upsource Error]        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
[Upsource Error]        at java.io.BufferedInputStream.fill(BufferedInputStream.java:206)
[Upsource Error]        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
[Upsource Error]        at com.jetbrains.service.util.cmd.InterruptableInputStream.read(InterruptableInputStream.java:39)
[Upsource Error]        at java.io.InputStream.read(InputStream.java:170)
[Upsource Error]        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
[Upsource Error]        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
[Upsource Error]        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
[Upsource Error]        at java.io.InputStreamReader.read(InputStreamReader.java:184)
[Upsource Error]        at java.io.BufferedReader.fill(BufferedReader.java:154)
[Upsource Error]        at java.io.BufferedReader.readLine(BufferedReader.java:317)
[Upsource Error]        at java.io.BufferedReader.readLine(BufferedReader.java:382)
[Upsource Error]        at com.jetbrains.service.util.cmd.LogStreamThread.run(LogStreamThread.java:39)


I can't find any errors in logs/cassandra, though a thread-dump has been written to that folder (threads-20141222110025791.txt), which seems suspicious to me.

It looks like eventually Cassandra is started though (upsource-stdout.log):

cassandra: Cassandra 2.1.1 startup completed. Redirected logging to log files in/opt/Upsource/logs/cassandra


Anyone knows what's going on here?
Let me know if I should provide more info/logs.


Thanks,
Guno

9 comments
Comment actions Permalink

Hi Guno,

It's difficult to say what really happen from the given output. Could you please zip and post all entire contents of the "logs" folder located in Upsource home directory?

Thanks in advance.

0
Comment actions Permalink

Attached logs.



Attachment(s):
logs.zip
0
Comment actions Permalink

Hi,

Thank you for the logs.
To successfully run in a headless mode, we require full OpenJDK to be installed.
Please refer to the issue: https://youtrack.jetbrains.com/issue/UP-2426

0
Comment actions Permalink

Ok, so I've now tried with openjdk:

> java -version:
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.2.el6_6-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

But it still stalls at 'Starting Apache Cassandra'.
Tried again after issuing 'bin/upsource.sh configure –launcher -J-Djava.awt.headless=true', but still the same. Is there something I'm missing?

Attaching new logs.



Attachment(s):
logs.zip
0
Comment actions Permalink

Hi Guno,

We've carefully checked your logs and didn't find any specific errors. You already specified your OS version and JDK version, but maybe there any other environment specific details?

Anyway we'll try to reproduce your issue in our labs, in meanwhile could you try to install Oracle Java and see if the issue persists?

0
Comment actions Permalink

Hi Artem,

The first set of logs I attached to this thread were from a regular Oracle JDK:

> java -version
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

As per Evgeny's suggestion I also tried OpenJDK (2nd set of logs):
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.2.el6_6-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

Both didn't work.

Other environment details:

  • CentOS release 6.6 (Final)
  • Headless
  • Kernel 2.6.32-358.18.1.el6.x86_64
  • Running behind proxy server, which is configured via environment variables http_proxy and https_proxy (I don't think I've explicitely specified the proxy server to Upsource/Java).
0
Comment actions Permalink

Guno,

Thanks for the details. We'll continue investigating the issue on our side. Also I've created the issue in our tracker - https://youtrack.jetbrains.com/issue/UP-2928 so you can see the the progress.

Thanks

0
Comment actions Permalink

Ok, thanks. I'll follow the Youtrack issue for updates.

0

Please sign in to leave a comment.