How To Use Github Container Registry and Gitlab Container Registry

Build, Push and Pull a Container from Github to K8s

How to use Gitlab Container Registry or Github GHCR as a container registry for K8s

  |  

0 Comments k8s Kubernetes k8s-dev

Docker registries and Kubernetes

To deploy a container on k8s you will need your container image available on a publicly available container registry server.

Below we will show you how to use either Github’s GHCR service or Gitlab’s Container Registry service to store and deploy container’s to any Kubernetes cluster.

What is a container registry?

A container registry, sometimes called a Docker registry, is a service that holds container images, such as those made with docker and other tools, and allows k8s to pull them with a well known API.

The well known API is now referred to as the OCI Distribution Specification. It was previously referred to simply as Docker Registry, with the API being called Docker Registry HTTP API v2. The definitive Registry implementation is now called simply Distribution - and it has been donated to CNCF. The implementation is available on github: Distribution

Regsitry Services

Running Distribution on your own requires setup, is another service you must maintain, and involves various security implementations. There are a variety of products, many of which you may already have at your disposal, that implement the OCI Distribution API - aka. provide a container registry service.

If hosting your own OCI Distribution service is required - such as in an on-premise or isolated installation, then look at these projects:

  • Harbor is a well-known, actively developed, fully featured container registry
  • Neuxs is another open-source product that has a commercially available option as well.
  • The Distribtution service itself from CNCF. It is a bit more complicated and less feature-rich than the ones above.

Choosing a service

Chances are you / your organization already pay for Github or Gitlab. For this reason, using their services is worth a look since they will effectively be free. Also, GHCR is free for public images.

If you deploy on AWS or Google Cloud already, look at their corresponding container services as well. These services can make secrets management on their platforms easier, since any container pull done by k8s on their platform will automatically authenticate with their service. That being said their services are not free. Additionally pulling from their container services from the outside can be more complicated. In large organizations, the complicated permissions on these services can take even more time to iron out.

Recommendation: Just use Github or Gitlab registries

To make developer and dev ops flow simple, and not have to involve dev ops too significantly in the developer workflow - we generally recommend using Github or Gitlab services.

Tutorial: Push & Deploy a container from Github GHCR

In this example, we are going to build a container and push it to Github GHCR. Then we will deploy it on k8s.

In our example we are going to reference a trivial Go service: trivial-golang-k8s-deployment but any container image will apply.

Build and tag your container image

Build your docker image and tag it. Replace trivial-golang-k8s-deployment with whatever you will call your container image.

docker build --tag trivial-golang-k8s-deployment .

Push to GHCR

First you will need to login to GHCR from docker.

As noted by Github in their docs, you must use a Personal Access Token (classic) in order to login to GHCR. You can generate a token by going here once logged into Github:

https://github.com/settings/tokens/new

Check the write:packages capability for your token. This will automatically select the repo capbabilities (and all sub capabilities)

Once generated, store your token somewhere secure. Here we always use a password manager like KeePassXC.

NOTE for Enterprises
Some Github orgs will require you to click the Configure SSO option to activate the token.

Login to GHCR: Docker login with token

Login to ghcr.io

docker login ghcr.io

Username: The email address you use for Github

Password: The token you generated above

Look for…

Login Succeeded

Tag & Push

Now we need to push our image to ghcr.io.

A word on registry paths with Github:

By convention, you should follow this pattern:

ghcr.io/NAMESPACE/IMAGE_NAME:TAG

Where NAMESPACE is the org or personal account. So in our example repo at github.com/IzumaNetworks/trivial-golang-k8s-deployment NAMESPACE is izumanetworks (capitals are not allowed)

The IMAGE_NAME:TAG can be anything. However, by convention, the IMAGE_NAME would typically match or be relatable to the repo name. TAG is always the version number or a development status such as latest or dev or alpha

In order to push our image we need to tag it using a path to ghcr.io. Above, using our example repo, we built and tagged it as trivial-golang-k8s-deployment. It is currently stored in the Docker image cache locally.

TIP
You can see the local images in the cache by running docker images

Let’s tag it and then push to ghcr.io:

docker tag trivial-golang-k8s-deployment ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest
docker push ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest

Deployment on K8s from ghcr.io

K8s needs a secret stored in a specific way in order to pull from a private registry. If your image is publicly available, you can pull it without a secret.

Creating a read-only K8s token for GHCR

While we could use the same token above for pulling the repo, its more secure to use a new token which only has read access to the registry.

Follow similar procedures as above…

Create a k8s secret from the token

There are a number of ways to create registry tokens - look at the k8s docs for all the options.

A simple way is to run:

kubectl create secret docker-registry regcred --docker-server=ghcr.io --docker-username=GITHUB_EMAIL --docker-password=TOKEN --docker-email=GITHUB_EMAIL

where TOKEN is the read-only token we just made and GITHUB_EMAIL is the email you use to sign into Github:

Output:

secret/regcred created

Success. Now let’s check the secret.

Verify our registry secret: Peek at dockerconfigjson

The command above creates a k8s secret of type: kubernetes.io/dockerconfigjson

FYI: the secret is called dockerconfigjson because normally the docker stores credentials in ~/.docker/config.json

A special base64 encoded of .dockerconfigjson - is in the secret. This contains the auth information for the registry

kubectl get secret regcred --output=yaml

Outputs something like this:

apiVersion: v1
data:
  .dockerconfigjson: eyJasdasdasdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...=
kind: Secret
metadata:
  creationTimestamp: "2023-11-29T05:28:23Z"
  name: regcred
  namespace: default
  resourceVersion: "5422565"
  uid: fcfbde08-9699-4e9e-b13a-d033e2b37bc1
type: kubernetes.io/dockerconfigjson

The value in .dockerconfigjson is a base64 encoded. Let’s look at it:

kubectl get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode

will yield a result similar to:

{"auths":{"ghcr.io":{"username":"[email protected]","password":"ghp_xxxxxxxxxxxxxxxxxxx","email":"[email protected]","auth":"sadasjdhXsdskdjASZQkjsadjksjdaskdjaksdjaksdjaskdjaskdj=="}}}%

Verify your fields look correct.

Create a Deployment using our registry secret

Create a new YAML file - deployment.yaml (or use the one in our example repo). As before - trivial-golang-k8s-deployment could be replaced with your own pod name.

apiVersion: v1
kind: Pod
metadata:
  name: trivial-golang-k8s-deployment
  labels:
    app: trivial-golang-k8s-deployment
spec:
  containers:
  - name: trivial-golang-k8s-deployment
    image: ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
  imagePullSecrets:
    - name: regcred

Notice the imagePullSecrets uses the regcred secret. Also - we use imagePullPolicy: Always to ensure in our testing we actually do a pull.

Deploy it:

kubectl apply -f deployment.yaml

and then…

kubectl describe pod trivial-golang-k8s-deployment

should reveal the pull successfully happening.

Output:

...
  Normal  Scheduled  8s    default-scheduler  Successfully assigned default/trivial-golang-k8s-deployment to gke-test-cluster-default-pool-da432508-ftsl
  Normal  Pulling    8s    kubelet            Pulling image "ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest"
  Normal  Pulled     7s    kubelet            Successfully pulled image "ghcr.io/izumanetworks/trivial-golang-k8s-deployment:latest" in 327.13398ms (327.148349ms including waiting)
  Normal  Created    7s    kubelet            Created container trivial-golang-k8s-deployment
  Normal  Started    7s    kubelet            Started container trivial-golang-k8s-deployment

There it is. You can now pull images from GHCR into your k8s cluster.

TIP

While it may not be plainly obvious, regcred is not a special name. The registry credential secrets are just any other secret object. You can name your registry secrets anything, and you can use multiple secrets if you have different secure registries.

Also bear in mind the same rules apply as for other secrets. Secrets aren’t shared across namespaces for instance.

Tutorial: Push & Deploy a container From Gitlab Container Registry

In this example, we are going to build a container and push it to Gitlab Container Registry. Then we will deploy it on k8s. The process is very similar to Github above.

Push to Gitlab

You will need to login to Gitlab Container Regsitry from docker.

As noted by Gitlab in their docs, you can use a Personal Access Token, Deploy Token and a few other methods for login. We are going to just use a Personal Access Token:

Go to https://gitlab.com/-/profile/personal_access_tokens

Click Add New Token

Name your token, and choose read_registry and write_registry for permissions. Store the token somewhere secure.

Login to Gitlab Container Registry

docker login registry.gitlab.com

Username: The username you have in Gitlab (if you click your profile pic you will see @username)

Password: The token you generated above

Look for…

Login Succeeded

Registry Password Security

While the docker login command offers the -u and -p options - you should never use these. This will put your token on the command history of your shell. Best practice is to login with a method that does not make a record of the password (token).

There are a number of ways to login to an OCI Registry programmatically. For instance the --password-stdin option.

Example: echo $MYTOKEN | docker login registry.gitlab.com -u myusername --password-stdin

Tag & Push

Like Github ghcr.io, Gitlab’s registry has a convention for image paths:

registry.gitlab.com/GROUP/PROJECT/IMAGE_NAME:TAG

Where GROUP is the Gitlab group the project is in. So for our example, the repo https://gitlab.com/izumanet/trivial-golang-k8s-deployment will be used - izumanet is the group. Project is trivial-golang-k8s-deployment

The IMAGE_NAME can be anything. TAG is typically the version number or a development status such as latest or dev or alpha

We need to tag our container image for Gitlab. Using our example repo, we built above and tagged as trivial-golang-k8s-deployment:

docker tag trivial-golang-k8s-deployment registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest
docker push registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest

Because Gitlab requires the project name to be in the path, I just chose to call the image app to simplify things. You could call it whatever you want.

Deployment on K8s from Gitlab Registry

Now let’s create a k8s secret for authentication to Gitlab’s registry. Process is the same as Github.

Creating a read-only K8s token for deployment from Gitlab

We could use the same token above for pulling the repo, but let’s create a read-only token for security reasons. Other tokens like deploy tokens are an option on Gitlab.

Follow similar procedures as above…

Create a k8s secret from the token

We will follow the same procedures we did for Github - we are going to call this secret regcredgitlab:

kubectl create secret docker-registry regcredgitlab --docker-server=registry.gitlab.com --docker-username=GITLAB_USERNAME --docker-password=TOKEN --docker-email=GITLAB_EMAIL

where TOKEN is the read-only token we just made and GITHUB_EMAIL is the email you use to sign into Github:

Output:

secret/regcredgitlab created

Success.

Registry Password Security

The k8s official examples require you to type your token on the command line. Here is how to enter your token without it going into the command history:

kubectl create secret docker-registry regcredgitlab --docker-server=registry.gitlab.com --docker-username=GITLAB_USERNAME --docker-password=`read -r pass; echo $pass` --docker-email=GITLAB_EMAIL

Type in your token right afterwards then ENTER.

Some versions of read support the -s switch - this will also prevent what you type from echoing to the terminal.

If you want to verify the secret you can use the same procedures as above for Github.

Create a Deployment using our secret for Gitlab

The process for deployment mirrors the process with Github. Notice our imagePullScrets is now regcredgitlab

apiVersion: v1
kind: Pod
metadata:
  name: trivial-golang-k8s-deployment
  labels:
    app: trivial-golang-k8s-deployment
spec:
  containers:
  - name: trivial-golang-k8s-deployment
    image: registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
  imagePullSecrets:
    - name: regcredgitlab

Again, we use imagePullPolicy: Always to ensure in our testing we actually do a pull.

Deploy it:

kubectl apply -f deployment-gitlab.yaml

and then…

kubectl describe pod trivial-golang-k8s-deployment

should reveal the pull successfully happening.

Output:

...
  Normal  Scheduled  6s    default-scheduler  Successfully assigned default/trivial-golang-k8s-deployment to gke-test-cluster-default-pool-da432508-ftsl
  Normal  Pulling    6s    kubelet            Pulling image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest"
  Normal  Pulled     5s    kubelet            Successfully pulled image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest" in 450.879137ms (450.901273ms including waiting)
  Normal  Created    5s    kubelet            Created container trivial-golang-k8s-deployment
  Normal  Started    5s    kubelet            Started container trivial-golang-k8s-deployment

You can now pull images from Gitlab into your k8s cluster.

General Troubleshooting

Running kubectl get pods after deployment should reveal a Running pod. Instead if you see a ErrImgPull or ImagePullBackOff there is problem with your secret or image path.

Starting by running:

kubectl describe pod trivial-golang-k8s-deployment

Output might look like:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  5s    default-scheduler  Successfully assigned default/trivial-golang-k8s-deployment to gke-test-cluster-default-pool-da432508-ftsl
  Normal   Pulling    4s    kubelet            Pulling image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest"
  Warning  Failed     4s    kubelet            Failed to pull image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest": failed to resolve reference "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://gitlab.com/jwt/auth?scope=repository%3Aizumanet%2Ftrivial-golang-k8s-deployment%2Fapp%3Apull&service=container_registry: 401 Unauthorized
  Warning  Failed     4s    kubelet            Error: ErrImagePull
  Normal   BackOff    3s    kubelet            Back-off pulling image "registry.gitlab.com/izumanet/trivial-golang-k8s-deployment/app:latest"
  Warning  Failed     3s    kubelet            Error: ImagePullBackOff

Notice the 401 Unauthorized - definitely a credential problem or a bad image path of some kind.

Verify your token works and your image path is correct

If your secret in k8s is not working, then test the token outside k8s as first step:

  • use docker login with the same secret you want K8s to use.
  • If you have already logged into a registry, you may need to remove the cached credential: remove the appropriate key / value pair in the ~/.docker/config.json or the equivalent file on your system.
  • If docker login is successful, try doing a pull of the image.
  • If all this works, then something is wrong with your K8s secret config itself, not the token.

A failed docker login will usually reveal your token or path issue:

Error response from daemon: Get "https://registry.gitlab.com/v2/": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/user/profile/account/two_factor_authentication#troubleshooting

CI / CD considerations

The above examples are good for deployment tasks on almost any K8s cluster, including Izuma Networks own Izuma Edge nodes.

However, if you are developing a CI/CD pipeline, you will want to use Github Actions or Gitlab Pipelines. These can build, upload your package to the registry, and trigger a k8s deployment all automatically if designed correctly.

For a later series of posts…