Email notifications not being sent - "MailProcessor - Outgoing mail handler error"

Since a recent server reboot, we have not been receiving email alerts. Prior to this alerts were working fine.
The "test email" function works without issue from hub - but it is my understanding that this is not a good test...


looking at this log  /opt/upsource-server/upsource-3.5.3616/logs/upsource-frontend/warn.log - we can see the following strange errors occurring repeatedly (see log output below)

any ideas on what we can try to resolve this issue?

 

[2017-04-19 16:00:50,249] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 1. 
[2017-04-19 16:00:52,283] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 2.
[2017-04-19 16:00:56,317] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 3.
[2017-04-19 16:01:02,351] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 4.
[2017-04-19 16:01:02,386] ERROR going mail handler-1 ase.datastax.StatementRepeater - Unable to retry 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because maximum attempts number is reached (4).
[2017-04-19 16:01:02,386] ERROR going mail handler-1 kend.server.mail.MailProcessor - Outgoing mail handler error
com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at com.jetbrains.upsource.database.datastax.ReadExecutor.execute(ReadExecutor.java:39)
at com.jetbrains.upsource.database.datastax.ReadExecutor.lambda$runReadQuery$0(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:113)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$run$1(StatementRepeater.java:129)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.run(StatementRepeater.java:127)
at com.jetbrains.upsource.database.datastax.ReadExecutor.runReadQuery(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.ThroughDbMediator.runReadQuery(ThroughDbMediator.java:75)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.doIterateRows(CqlDynamicTable.java:162)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRows(CqlDynamicTable.java:119)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRow(CqlDynamicTable.java:114)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.b(DynamicBufferingTable.java:114)
at com.jetbrains.upsource.stats.StatsAccumulator.a(StatsAccumulator.java:41)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:49)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:40)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.iterateRow(DynamicBufferingTable.java:110)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:442)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.b(MailProcessor.java:392)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:149)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.jetbrains.upsource.util.NamedDaemonThreadFactory.a(NamedDaemonThreadFactory.java:34)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:95)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:128)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:174)
at com.datastax.driver.core.RequestHandler.access$2600(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:793)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1012)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:935)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:303)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
... 1 more
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:76)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:277)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:257)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 17 more
[2017-04-19 16:01:50,251] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 1.
[2017-04-19 16:01:52,286] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 2.
[2017-04-19 16:01:56,321] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 3.
[2017-04-19 16:02:02,364] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 4.
[2017-04-19 16:02:02,398] ERROR going mail handler-1 ase.datastax.StatementRepeater - Unable to retry 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because maximum attempts number is reached (4).
[2017-04-19 16:02:02,398] ERROR going mail handler-1 kend.server.mail.MailProcessor - Outgoing mail handler error
com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at com.jetbrains.upsource.database.datastax.ReadExecutor.execute(ReadExecutor.java:39)
at com.jetbrains.upsource.database.datastax.ReadExecutor.lambda$runReadQuery$0(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:113)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$run$1(StatementRepeater.java:129)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.run(StatementRepeater.java:127)
at com.jetbrains.upsource.database.datastax.ReadExecutor.runReadQuery(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.ThroughDbMediator.runReadQuery(ThroughDbMediator.java:75)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.doIterateRows(CqlDynamicTable.java:162)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRows(CqlDynamicTable.java:119)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRow(CqlDynamicTable.java:114)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.b(DynamicBufferingTable.java:114)
at com.jetbrains.upsource.stats.StatsAccumulator.a(StatsAccumulator.java:41)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:49)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:40)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.iterateRow(DynamicBufferingTable.java:110)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:442)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.b(MailProcessor.java:392)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:149)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.jetbrains.upsource.util.NamedDaemonThreadFactory.a(NamedDaemonThreadFactory.java:34)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:95)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:128)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:174)
at com.datastax.driver.core.RequestHandler.access$2600(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:793)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1012)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:935)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:303)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
... 1 more
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:76)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:277)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:257)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 17 more
[2017-04-19 16:02:50,260] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 1.
[2017-04-19 16:02:52,305] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 2.
[2017-04-19 16:02:56,338] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 3.
[2017-04-19 16:03:02,373] WARN going mail handler-1 ase.datastax.StatementRepeater - Retry to send 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because of 'Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)'. Retry #: 4.
[2017-04-19 16:03:02,409] ERROR going mail handler-1 ase.datastax.StatementRepeater - Unable to retry 'Bound statement: SELECT * FROM mailnotifications WHERE rowid IN ?;. Is idempotent.' request because maximum attempts number is reached (4).
[2017-04-19 16:03:02,409] ERROR going mail handler-1 kend.server.mail.MailProcessor - Outgoing mail handler error
com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at com.jetbrains.upsource.database.datastax.ReadExecutor.execute(ReadExecutor.java:39)
at com.jetbrains.upsource.database.datastax.ReadExecutor.lambda$runReadQuery$0(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:113)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$tryRetry$0(StatementRepeater.java:105)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.tryRetry(StatementRepeater.java:103)
at com.jetbrains.upsource.database.datastax.StatementRepeater.lambda$run$1(StatementRepeater.java:129)
at com.jetbrains.upsource.database.datastax.StatementRepeater.executeWithFailAction(StatementRepeater.java:115)
at com.jetbrains.upsource.database.datastax.StatementRepeater.run(StatementRepeater.java:127)
at com.jetbrains.upsource.database.datastax.ReadExecutor.runReadQuery(ReadExecutor.java:31)
at com.jetbrains.upsource.database.datastax.ThroughDbMediator.runReadQuery(ThroughDbMediator.java:75)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.doIterateRows(CqlDynamicTable.java:162)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRows(CqlDynamicTable.java:119)
at com.jetbrains.upsource.database.datastax.CqlDynamicTable.iterateRow(CqlDynamicTable.java:114)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.b(DynamicBufferingTable.java:114)
at com.jetbrains.upsource.stats.StatsAccumulator.a(StatsAccumulator.java:41)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:49)
at com.jetbrains.upsource.stats.StatsAccumulator.decorate(StatsAccumulator.java:40)
at com.jetbrains.upsource.db.impl.DynamicBufferingTable.iterateRow(DynamicBufferingTable.java:110)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:442)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.b(MailProcessor.java:392)
at com.jetbrains.upsource.backend.server.mail.MailProcessor.a(MailProcessor.java:149)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.jetbrains.upsource.util.NamedDaemonThreadFactory.a(NamedDaemonThreadFactory.java:34)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:95)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:128)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:174)
at com.datastax.driver.core.RequestHandler.access$2600(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:793)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1012)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:935)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:303)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
... 1 more
Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (1 responses were required but only 0 replica responded, 1 failed)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:76)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:277)
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:257)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
... 17 more

 

2 comments

Hey James,

Due to DB connection issue, the writer got corrupted. It may happen in some cases. The simplest way to fix it is to restart Upsource.

If the issue persists after Upsource restart, please log a support case and attach full Upsource logs.

0

i can confirm that a restart of the Upsource server resolved the issue

0

Please sign in to leave a comment.