Upgrading on GKE

The fact that you return a 503 on / after upgrade breaks various health checks in GKE. Is there some /path I can get a 200 from post upgrade prior to hitting the generated one and running through the config?

I'm unable to upgrade as a result since ingress won't forward any request as it thinks the app is not available. 

Thanks for any help

Comment actions Permalink

Hi Brett,

Could you please elaborate where/when 503 code is returned? 

Comment actions Permalink

Sure. The health check pulls down application http response code 503 when hitting / after upgrade. When kubernetes gets a 503 back the readiness probes fails because it thinks the application is in an unusable state and does not route traffic to the containers. 

Info on those: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

This is just the first layer of the problem. Even if those are removed the health checks created by ingress fail for the same reason. The first being a check at the kubernetes layer and second being a check at the load balancer layer. 

My current W/A to this issue is to remove the readiness probes and hit the url REALLY fast before the load balancer compute health checks fail :p Not really supportable.


Thanks a bunch

Comment actions Permalink

And to be more clear, I'm upgrading from 

  • Build 2017.2.2057


  • Build 2017.2.2307
After upgrade it generates a unique URL to initiate the backend upgrade process I guess. 

Please sign in to leave a comment.