Use rsync with K8s for fast development

If you are actively developing an application for k8s, it’s often necessary to test your code directly on a Kubernetes platform.

Over the last few years I’ve seen a lot of developers waiting on docker commands and K8s deployments to complete, just to check minor code changes… Painful. Let’s stop the pain.

The Value of a Fast Edit-Compile-Test Cycle.

Don’t underestimate the value of a fast edit-compile-test (ECT) cycle. For people who actually write code (seems like there are less and less these days) - there are two big flavors of coding behavior. We all fall somewhere in the spectrum of these extremes:

• The meticulous, careful artist. This person thinks through their code constantly - but they infrequently compile. When they do build - it will generally work wtih a few minor issues. They move slower and more carefully than others. Their edit-compile-test cycle is very infrequent. I’ve seen people write hours of code before actually even compiling it.
• The drunk, Lego-block engineer. This person is slapping together code super fast, not really knowing what will work and what won’t. They move quick but perhaps a bit sloppy. Their edit-compile-test cycle is very frequent. Sometimes every few lines.

Which is more efficient? Who knows. One thing is for sure. The more you lean towards the latter the more important a fast edge-compile-test cycle.

The challenge of ECT cycle on Kubernetes

If your code requires you to test on a real Kubernetes cluster (not something running on your personal dev machine - like using Minikube or similar projects) - then you should be aware of some techniques for quick ECT cycles.

And the truth is your code will require you to test on a real cluster at some point - i.e. a cluster that is very close to your production cluster. So in reality, these techniques apply to any development style.

Update Binaries not Containers for Fast ECT

Normally we will build a container image for a K8s deployment. But container builds are slow, and even a super-fast K8s cluster, running on AWS or GKE will not be instantaneous in deployment. Additionally continuous building and deployment of entire containers:

• Can increase the cost of your development cluster
• May create side effects in your K8s cluster (such as pod eviction for lack of resources in other workloads)
• Will eat up diskspace and create a huge amount of docker images on whatever machine does the builds.
• Container build are slow. And if you insist on build on an arch different than your target architecture (i.e. an Apple Silicon laptop to an amd64 cluster) your build will be painfully slow.

Instead, we can simply update the binary in a running container if do a small amount of prep.

Third Party Tools for Fast ECT

There are a number of open-source and paid tools to speed up the K8s ECT cycle. For instance, Telepresence from Ambassador Labs and mirrord which is MIT licensed. Both are fine options if you need more complex capabilities such as running an interactive debugger.

That being said, most of the time this stuff is overkill. The simplest thing to do is to just quickly update the binary and restart your process. And the tools to do this can easily be placed in your container.

Rsync to the Rescue

In this example, I’ll show you how to use rsync and fswatch to constantly monitor your code tree and then update your project in your pod on K8s. The beauty of this is rsync and fswatch are free and work on MacOS, Windows WSL or Linux, and this technique doesn’t have any weird side effects or caveats that might affect testing.

Most of the work is a one-time effort. You will end with a development and production container.

• The development container will have all the tools needed to run rsync, do a build (if desired - see notes at end) and deploy so that K8s won’t kill your container when your process dies.
• The production container is untouched.

Example of using Rsync and Fswatch

Prereqs

In this example, we will reference our github.com/IzumaNetworks/trivial-golang-k8s-deployment repo. All code in this example is there.

Pre-requirements:

0. A working k8s cluster
1. Clone the example repo github.com/IzumaNetworks/trivial-golang-k8s-deployment or just deploy with the publicly available deployment.yaml files to your cluster.
2. The tool watch-krsync.sh is available here github.com/IzumaNetworks/k8s-rsync Clone this repo also.
3. Install Fswatch:

Mac:

# MacPorts
$ port install fswatch

# Homebrew
$ brew install fswatch

Debians:

sudo apt-get install fswatch

Other installs: More details here

Make a Development Container

We need a version of our container which contains rsync.

For the trivial-golang-k8s-deployment) project, the original Dockerfile was:

FROM golang:1.20
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0

ADD app /app
ENV CGO_ENABLED=0
RUN cd /app && go build -o app .
WORKDIR /app
ENTRYPOINT [ "/app/app" ]

our new one adds a few lines to install rsync

FROM golang:1.20
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example DEBUG"
LABEL org.opencontainers.image.licenses=Apache-2.0

# golang container is based on Debian
RUN DEBIAN_FRONTEND=noninteractive apt-get update
# install rsync
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install rsync
RUN DEBIAN_FRONTEND=noninteractive apt-get -y clean

ADD app /app
ADD loop.sh /
ENV CGO_ENABLED=0
RUN cd /app && go build -o app .
WORKDIR /app
# just start a dummy loop to keep the container up
ENTRYPOINT [ "/loop.sh" ]

Also our ENTRYPOINT is now ./loop.sh

Loop.sh is a simple bash script that just keeps the container up. This way we can restart our service without redeploying a container.

#!/bin/bash

COUNT=0

while true
do
	echo "debug loop: ${COUNT} seconds"
	sleep 5
	COUNT=$((COUNT + 5))
done

Make a Development Deployment YAML

And we need a new deployment config.

Our original example deployment is deployment.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: trivial-golang-k8s-deployment
  labels:
    app: trivial-golang-k8s-deployment
spec:
  containers:
  - name: trivial-golang-k8s-deployment
    image: ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /hello
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
  imagePullSecrets:
    - name: regcred

The new debug deployment YAML is debug-deployment.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: trivial-golang-k8s-deployment
  labels:
    app: trivial-golang-k8s-deployment
spec:
  containers:
  - name: trivial-golang-k8s-deployment-debug
    image: ghcr.io/izumanetworks/trivial-golang-k8s-deployment-debug:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
  imagePullSecrets:
    - name: regcred

Notice it removes the liveness probe and uses the debug container image.

Deploy the example container

kubectl apply -f debug-deployment.yaml

Run watch-krsync.sh

Run watch-krsync.sh

% ../k8s-rsync/watch-krsync.sh trivial-golang-k8s-deployment ./app:/app
initial sync....
LOCAL: ./app REMOTE: /app
~/trivial-golang-k8s-deployment ~/trivial-golang-k8s-deployment
~/trivial-golang-k8s-deployment
Checking ./app:/app (~/trivial-golang-k8s-deployment/app) ./app
building file list ... done
./
app
build.sh
go.mod
hello.go

sent 6292937 bytes  received 15624 bytes  2523424.40 bytes/sec
total size is 6469281  speedup is 1.03
watching....  ./app

The script will immediately sync your local files to the pod. Your local files always take precedence over any different file on the pod.

Now, as you edit, your changed source files will automatically update in the container.

Edit, Compile & Test

• Edit your files.
• They will automatically sync to the container as you change them.
• Build and test in the container:

Shell into the pod in another terminal:

kubectl exec --stdin --tty trivial-golang-k8s-deployment -- /bin/bash

root@trivial-golang-k8s-deployment:/app# ./build.sh
root@trivial-golang-k8s-deployment:/app# ./app
Hello world 1. Start.
...

Note: You may also be testing locally - that’s fine. If you do your binary may update locally and that will be copied (delta’s only) to the container. Depending on your local dev machine’s architecture, the binary might be incompatible. So it is best to run the build in the container.

Further Refinement

This example shows a simple Golang environment. Your own use case may require more tooling: libraries and compilers needed for your code base to run a full build.

• A second option is to build locally and let rsync move that build to the container, just doing the testing in the container. The caveat here is the build process locally must be compatible with that inside the container.
• A third option would be to do the build in a pre-built docker container - placing the build output on the local file system outside the local container. watch-krsync.sh would then pick up the binary file change and move it to the container.