Upsource stops working after a while

Hi all,

our installation (winserver 2012 R2, 8 core, 8 gb, hyper-v virtual machine) stops working after a while. The UI is still responsive but new check-ins are no longer indexed.

Log excerpt from upsource-stderr just around the outage

[Upsource Error] [2014-12-18 13:07:56,162]   WARN - undle.proxy.jetty.ProxyServlet - Proxying /~socket.io/ failed: null
[Upsource Error] java.net.ConnectException: Connection refused: no further information
[Upsource Error]      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[Upsource Error]      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
[Upsource Error]      at org.eclipse.jetty.io.SelectorManager.finishConnect(SelectorManager.java:340)
[Upsource Error]      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processConnect(SelectorManager.java:671)
[Upsource Error]      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:640)
[Upsource Error]      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:607)
[Upsource Error]      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[Upsource Error]      at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[Upsource Error]      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
[Upsource Error]      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
[Upsource Error]      at java.lang.Thread.run(Thread.java:745)


Log excert from cassandra-stderr just around the outage

[Apache Cassandra Error] 13:07:27.239 [NonPeriodicTasks:1] ERROR o.a.c.io.sstable.SSTableDeletingTask - Unable to delete C:\Program Files\Upsource\data\cassandra\data\xxx\anchorscache-1375577085cb11e4b0eab1bba6e14afe\boss-anchorscache-ka-13-Data.db (it will be removed on server restart; we'll also retry after GC)
...
many more


Log excerpt from upsource-analyzer-stderr around the outage

[Upsource Analyzer Error] [2014-12-18 12:56:31,236] WARN  ing send processor-1 urce.messaging.impl.Connection - Error sending message '127.0.0.1:10041'->'127.0.0.1:10021' on upsource~monitoring
[Upsource Analyzer Error] java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
[Upsource Analyzer Error] durch den Hostcomputer abgebrochen
[Upsource Analyzer Error]      at sun.nio.ch.SocketDispatcher.write0(Native Method)
[Upsource Analyzer Error]      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
[Upsource Analyzer Error]      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
[Upsource Analyzer Error]      at sun.nio.ch.IOUtil.write(IOUtil.java:65)
[Upsource Analyzer Error]      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.SocketOutputStream$1.performIO(SocketOutputStream.java:49)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:110)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.SocketOutputStream.write(SocketOutputStream.java:109)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.SocketOutputStream.write(SocketOutputStream.java:81)
[Upsource Analyzer Error]      at java.io.OutputStream.write(OutputStream.java:75)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.Connection.writeTo(Connection.java:195)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.Connection.write(Connection.java:167)
[Upsource Analyzer Error]      at com.jetbrains.upsource.messaging.impl.MessagingImpl$8.run(MessagingImpl.java:381)
[Upsource Analyzer Error]      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[Upsource Analyzer Error]      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Upsource Analyzer Error]      at java.lang.Thread.run(Thread.java:745)
[Upsource Analyzer Error] [2014-12-18 12:56:33,861] WARN  PPER] Daemon Proxy 2 ckend.cli.multi.ProjectContext - Dispose of ProjectId{pmd} is started
[Upsource Analyzer Error] [2014-12-18 12:56:33,877] WARN  PPER] Daemon Proxy 2 ckend.cli.multi.ProjectContext - Dispose of ProjectId{boss} is started
================================================================
[Upsource Analyzer Error] [2014-12-18 13:08:15,022] WARN  S_CONTENTS_CLUSTER-2 r.core.impl.graph.GraphService - Building commits graph without FETCH service
[Upsource Analyzer Error] [2014-12-18 13:08:15,428] WARN   socket listener-1-1 e.messaging.impl.MessagingImpl - Message ordinal mismatch from 127.0.0.1:10021 (/127.0.0.1:59121). Expected 0, received 1
[Upsource Analyzer Error] [2014-12-18 13:08:31,662] WARN   socket listener-1-1 e.messaging.impl.MessagingImpl - Message ordinal mismatch from 127.0.0.1:10031 (/127.0.0.1:59246). Expected 0, received 1


Can anyone help? This currently prevents us from using upsource more extensivly as this outage occurs about every three hours and requires a restart of upsource.


EDIT:
It appears that the analyzer process stops responding after a while. I have set Xmx to 4096m and XX:MaxPermSize to 1024m. Lets see what happens.

Edited by Jan Harms

13 comments

Hi Jan,

Looks like you are using EAP version. Could you please download and install release one (can be found here https://www.jetbrains.com/upsource/download/)

Please let me know if my suggestion is not correct or release Upsource version doesn't help.

Thanks in advance.

0

Hey Artem,

thanks for taking the time to look at the logs. Here are the versions of the components.

Upsource: 1.0.12551
Hub: 0.8.125

The file "VERSION.txt" from the base folder contains: 1.0.12551

The file "version.properties" from the base folder contains:
#Bundle version properties
#Fri, 05 Dec 2014 22:06:09 +0300
bundle.build.number=0.3.134


Thank you.



Edit:

New error occuring

09:13:58: Internal error: Error while importing revision MAIN-{ revision: RevisionId{xxx, 10012} @ ~ 2014 Aug 8 17:46:03}-cluster=VCS_CONTENTS_CLUSTER (base: RevisionId{xxx, 10052}(Wed Aug 13 17:18:01 CEST 2014), AnalyzerState{code=15, [revision info imported] [files] [constituents] }). Writer to keyspace xxx is corrupted due to unrecoverable problem


Edited by Jan Harms

0

Jan,

Thanks for the info.

All entire contents from the "logs" folder (<upsource_home_directory>\logs) will be appreciated for the further analysis.

Looking forward to hearing from you.

0

I have sent an email with logs and a screenshot.

Thank you.

0

It appears this issues is also caused by the problems with the cassandra db.

0

That's correct, the main issue is related to Cassandra. As my colleague Evgeny mentioned, Cassandra doesn't work well under Windows with high load. So in your case we would recommend to install Upsource on unix like system.

We are investigating this issue with Cassandra and Windows in order to provide a support update, but so far no estimates yet.

0

I'm experiencing the same issue on Ubuntu 14.04. I'll post a new forum entry.

0

Hi,

I'm experiencing the same issue with Upsource 1.0.12566 running on CentOS 6.6, Java build 1.7.0_76-b13.

Regards,
Jānis

0

Hi Jānis,

We would need logs folder from Upsource directory in order to determine the root cause.

Please send it to upsource-support@jetbrains.com

Thansks in advance.

0

Hi,

I sent you all the requested information to the provided e-mail address. I hope it gave some answers.

Regards,
Jānis

0

Hi,

It seams that the problem realy is gone in version 2.0. Thank you.

Regards,
Jānis

0

Usource Stops Working. I have installed it on unix box. I am using it on latest java version 1.8

I am facing Usource Stops Working(upsource crashed daily) issue. I am using upsource version 2017.2.2057. Daily it work for 8 HRS duration beyond that it get crashed.

Post that we have to stop server manually and again start the server to up upsource system.

We tried by increasing max perm size to 1024(XX:MaxPermSize=1024), -Xmx=4096m in below path in file (upsource.jvmoptions.dist.bak)

Path: /upsource/conf/upsource.jvmoptions.dist.bak

 

What could be the probable reason? could someone help me to fix this issue?

0

Confronting Cassandra DB Connection Problem? Call Cassandra Technical Support to settle it rapidly

On the off chance that any clients endeavoring to associate with a hub, all of a sudden they got a message which demonstrates that "Unfit to interface with any server". Be that as it may they additionally give hostname yet at the same time they need to confront same issue. To settle out this issue pick Cognegic's Cassandra Database Consulting and Support or Cassandra Database Support to take care of your concern. We give proactive help 24*7 and furthermore give moment caution if any issue has been happened.

For More Info: https://cognegicsystems.com/

Contact Number: 1-800-450-8670

Email Address- info@cognegicsystems.com

Company’s Address- 507 Copper Square Drive Bethel Connecticut (USA) 06801

0

Please sign in to leave a comment.