7 comments
Comment actions Permalink

Also a comment for Brett-curtis. It was also working fine with the upgrade. When I upgraded the pod to latest version it picked up the old configuration and entered "upgrade" status. Then I was able to connect to the service via "kubectl proxy" and continue the upgrade process from there (Upsource printed in the log the URL I should connect to to continue the upgrade).

One small complaint- It would be nice if Upsource provides a working health check during the upgrade as well (it was not) - it was a small annoyance because I had to use the proxy and service was not available during the upgrade. I just created a feature request at upsource's youtrack https://youtrack.jetbrains.com/issue/UP-9231 to have a separate /healthz health check URL that wiill provide HTTP 200 response even diring the upgrade. This way I would not have to use kubectl proxy trick.

Please upvote if you feel that it might be useful :). 

3
Comment actions Permalink

Not yet, unfortunately, but we do plan to add those in some foreseeable future.

0
Comment actions Permalink

Thanks Artem.  

I suppose any docs for deploying dev tools (like TeamCity or Hub) are great as well.

 

0
Comment actions Permalink

I just wanted to share my experience here from successful Kubernetes installation of upsource so that others can fix their problems quickly.

Update! The 'SocketException' problem is solved as of 2017.2.2197 version of Upsource - see two comments below for an explanation. 

I managed to successfuly deploy Upsource using Kubernetes cluster (running on Google Container Engine). I followed the  https://www.jetbrains.com/help/upsource/docker-installation.html indeed and created our own custom helm charts for it. I only had one serious issues which delayed it (quite signidficantly because I did not have time to look at it in more details). The problem was that the installation failed while running the setup wizard:

I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] Error initializing messaging server [local service port: 80]
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	... 1 more 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:365) 
I  upsource-cluster-init: [Upsource Cluster Initialize Service Error] 	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:250) 
I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:980) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1258) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:554) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at sun.nio.ch.Net.bind(Net.java:425) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at sun.nio.ch.Net.bind(Net.java:433) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at sun.nio.ch.Net.bind0(Native Method) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] Caused by: java.net.SocketException: Permission denied I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at java.lang.Thread.run(Thread.java:748) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.launcher.AppProxy$6$1.call(AppProxy.java:97) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.launcher.AppProxy$6$1.call(AppProxy.java:99) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.ClusterInitService.start(ClusterInitService.java:39) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.lifetimes.Lifetime.nested(Lifetime.java:123) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.lifetimes.Lifetime.nested(Lifetime.java:115) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.lifetimes.Lifetime.a(Lifetime.java:123) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.ClusterInitService.a(ClusterInitService.java:42) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.backend.server.facade.UpsourceApplicationEnvironment.<init>(UpsourceApplicationEnvironment.java:100) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] at com.jetbrains.upsource.messaging.impl.MessagingImpl.<init>(MessagingImpl.java:96) I upsource-cluster-init: [Upsource Cluster Initialize Service Error] com.jetbrains.upsource.UpsourceException: Error initializing messaging server [local service port: 80]

It seems to me that for some reason the internal messaging server is trying to listen at port 80 which it cannot because in the default docker image, there is a USER command used and upsource runs as non-root user inside the docker container. While this is usual for "standard" runs, running programs inside Docker as root user is rather normal. There are some issues connected with mapping the users if you are mounting folders from the host, so I usually prefer to run everything as root user inside docker. This is the workaround I used for this problem. I created my own image based on the upsource one where I switch the user back to root, and change permissions of all the files to be owned by root just in case. I pushed it to our private registry and I pointed my helm chart to that private image. Then it started to work. 

Artem - maybe there is another way of fixing it for the future, but this way I could start using Upsource in our Kubernetes cluster as of today :). Please let me know if there is a better way.

 Here is the Dockerfile I used:

FROM jetbrains/upsource:2017.2.2057
USER root
RUN chown -fR root.root / || true
0
Comment actions Permalink

I've fully automated this process using the official image and I will share the code on a public repo shortly if folks are interested.. It does everything including ingress resources that use kube-lego for automatic ssl (Using Let's Encrypt) as well as external-dns (if you're using Cloud DNS). 

 

I think some things in the application design could be looked at to make this a bit easier, mainly with upgrades.. I don't want to create a backup in the UI and import for example. I'd much prefer to apply a new image and have it recognize my persistent disk and do whatever magic it needed to do. 

 

Nonetheless, good start and this should be good enough for me to provide to our development teams as an offering. 

0
Comment actions Permalink

Note. The installation problem is solved in the most recent version of Upsource ( 2017.2.2197 ) - The relevant issue is here: https://youtrack.jetbrains.com/issue/UP-9194 . It turned out that messaging server in upsource used UPSOURCE_SERVICE_HOST and UPSOURCE_SERVICE_PORT variables, and if you had your service defined in kubernetes with 'upsource' name (of course I did as probably almost everyone else woud), then kubernetes will create UPSOURCE_SERVICE_HOST and UPSOURCE_SERVICE_PORT variables (https://kubernetes.io/docs/concepts/containers/container-environment-variables/)  pointing to your service's host/port and the messaging server would pick those variables. 

Quite an interesting issue (I would not even call it a bug :). Jetbrains solved it in 2017.2.2197 and changed name of variables read by messaging server to UPSOURCE_SERVICE_MESSAGING_HOST UPSOURCE_SERVICE_MESSAGING_PORT respectively.  I updated to latest version and could get rid of our custom docker image (after changing ownership of configuration files in the mounted volume back to jetbrains user).

Good job Jetbrains! That was quick!

0
Comment actions Permalink

I came across https://github.com/egymgmbh/upsource-k8s looking for the same thing.

I'd be really happy to also find out how to achieve the same setup as https://www.jetbrains.com/help/upsource/setting-up-upsource-cluster.html, but inside Kubernetes.

Things that make this scary:

* No operational experience with Cassandra in our org whatsoever, no likelihood to change that, and Cassandra's reputation for being hard to operate.

* No mention within the Upsource documentation of how to monitor at an application level - by which I mean, how to export metrics into a monitoring product. In our case, that's prometheus.io.

* No guidelines (or better, turnkey config sample) for how to export logs into Logstash.

0

Please sign in to leave a comment.