Upsource database error

We are getting Database connection failure when i click to see reviews in Upsource. Cassandra error log is empty. Following is the Upsource-frontend error log.

[2018-06-27 13:33:59,159] ERROR r: update topology-1 essaging.impl.TopologyProvider - [127.0.0.1:10011]: failed to update messaging topology
java.lang.Error: An error occurred during flushing data to database
at com.jetbrains.upsource.db.impl.DatabaseFlushExecutorImpl.consume(DatabaseFlushExecutorImpl.java:192)
at com.jetbrains.upsource.db.impl.AbstractBufferingWriter.a(AbstractBufferingWriter.java:159)
at com.jetbrains.upsource.db.impl.AbstractBufferingWriter.flush(AbstractBufferingWriter.java:130)
at com.jetbrains.upsource.database.datastax.CqlDatabase.flush(CqlDatabase.kt:242)
at com.jetbrains.upsource.database.datastax.CqlDatabase.flush(CqlDatabase.kt:221)
at com.jetbrains.upsource.database.datastax.CqlDatabase$myDatabaseFlusher$1.flush(CqlDatabase.kt:65)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.putValue(DynamicBufferingTable.kt:188)
at com.jetbrains.upsource.backend.server.core.db.MessagingTopologyTable.updateNodeDescription(MessagingTopologyTable.java:44)
at com.jetbrains.upsource.messaging.impl.TopologyProvider$listenForChanges$1.run(TopologyProvider.kt:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.jetbrains.upsource.util.NamedDaemonThreadFactory.a(NamedDaemonThreadFactory.java:34)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: Writer to db 'UpsourceSystem' is corrupted due to unrecoverable problem.
at com.jetbrains.upsource.db.impl.DatabaseFlushExecutorImpl.consume(DatabaseFlushExecutorImpl.java:170)
... 16 more
Caused by: com.jetbrains.upsource.db.api.query.DatabaseConnectionError: Database connection failure
at com.jetbrains.upsource.database.datastax.ThroughDbMediator.submitWriteTask(ThroughDbMediator.kt:48)
at com.jetbrains.upsource.database.datastax.ThroughDbMediator$submitWriteTask$1.doResume(ThroughDbMediator.kt)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:60)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resumeWithException(CoroutineImpl.kt:59)
at kotlinx.coroutines.experimental.DispatchTask.run(CoroutineDispatcher.kt:122)
at kotlinx.coroutines.experimental.EventLoopBase$QueuedRunnableTask.run(EventLoop.kt:189)
at kotlinx.coroutines.experimental.EventLoopBase.processNextEvent(EventLoop.kt:129)
at kotlinx.coroutines.experimental.BlockingCoroutine.joinBlocking(Builders.kt:214)
at kotlinx.coroutines.experimental.BuildersKt.runBlocking(Builders.kt:152)
at kotlinx.coroutines.experimental.BuildersKt.runBlocking$default(Builders.kt:145)
at com.jetbrains.upsource.database.datastax.buffering.CqlBufferedQueue$StorageChunk.flush(CqlBufferedQueue.kt:55)
at com.jetbrains.upsource.db.impl.AbstractBufferingWriter.flushChunk(AbstractBufferingWriter.java:206)
at com.jetbrains.upsource.db.impl.DatabaseFlushExecutorImpl$Executor.a(DatabaseFlushExecutorImpl.java:249)
at __. db_UpsourceSystem .__.run(JavaGeneratorTemplate.java:43)
at com.jetbrains.upsource.db.impl.DatabaseFlushExecutorImpl$Executor.a(DatabaseFlushExecutorImpl.java:248)
at com.jetbrains.upsource.db.impl.DatabaseFlushExecutorImpl$Executor.run(DatabaseFlushExecutorImpl.java:231)
... 1 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:10030 (com.datastax.driver.core.exceptions.BusyPoolException: [/127.0.0.1] Pool is busy (no available connection and timed out after 5000 MILLISECONDS)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:230)
at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:50)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:301)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:368)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53)
at com.datastax.driver.core.HostConnectionPool$PendingBorrow$1.run(HostConnectionPool.java:694)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
... 1 more

3
4 comments
Avatar
Permanently deleted user

I'm experimenting the same error.

Any insight ?

0

Hi.

We have the same issue from time to time. The pod restart is a workaround.

0

It looks like someone does't close db connections)

0

plz try cmd to restart upsource:

```shell

sudo <path_to_upsource_root_path>/bin/upsource.sh restart

```

wait a menont .It works for me.

0

Please sign in to leave a comment.