Home on Galust Arakelyan

Backing up a Kubernetes cluster whose nodes can't reach anything

Sat, 24 Oct 2026 00:00:00 +0000

In October 2026 our cloud provider lost two of its three availability zones within a few days. The production cluster was still there, and its API server answered over VPN, but the surviving worker nodes had no internet access: the NAT gateway lived in a dead zone. The provider’s object storage was read-only.

We needed a full backup of the cluster before deciding where to restore it. The usual way is to install Velero into the cluster, but that was impossible here: nodes couldn’t pull the Velero image, and they couldn’t reach any object storage to write to.

About

Mon, 01 Jan 0001 00:00:00 +0000

I’m Galust, a DevOps engineer. Right now I run production Kubernetes for a fintech lending group: managed Kubernetes in the cloud, VMware and our own hardware, plus all the stateful pieces that come with money — PostgreSQL, RabbitMQ, Redis, Redpanda and Vault.

The part of the job I value most is what happens when things break. In October 2026 our cloud provider lost two of its three availability zones within a few days. Production went down, the surviving worker nodes had no internet access, and everything had to be pulled out and started somewhere else. I backed up the clusters with Velero running outside the cluster, saved the images and the state of Vault, Redis, RabbitMQ and Redpanda, and then restored them on new Kubernetes in our office within a day. The backup part is described in this post.

CV (print)

Mon, 01 Jan 0001 00:00:00 +0000