Gitlab CI/CD and Helm + file -

A short post about how I use GitLab CI/CD to automagically deploy to Kubernetes using Helm and Helmfile .

First things first, this is for a setup that isn’t completely CI/CD, in some situation we can’t simply continuously update the running app to latest. It would however still be nice to keep it managed and tracked via version control, this is where Helmfiles come in.

Helm(file)?

Helm is a package manager for Kubernetes, Helmfile is for managing your Helm (chart) deployments via version control. Helm creates charts that keep track of how and what to do for a deployment. This is based on images from a Docker registry. These charts can be published in multiple ways, for example, the two biggest can be found here . Helmfile(’s) lists these charts, from multiple repositories, with additional settings/configuration.

Helmfile

The Helmfile will be in the root of our git repos and will look something like this:

---
environments:
  default:
    values:
      - environment/default/values.yaml
    secrets:
      - environment/default/secrets.yaml
    missingFileHandler: warn
---
repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com
  - name: bitnami
    url: https://charts.bitnami.com/bitnami
  - name: myrepo
    url: git+https://{{ env "HELM_REPO_USER" | default "helm" }}:{{ requiredEnv "HELM_REPO_PASS" }}@gitlab.com/patvdleer/helm-repo@/charts

helmDefaults:
  # no verify due to unable to sign https://github.com/helm/helm/issues/2843#issuecomment-449047847
  verify: false
  wait: true
  timeout: 720
  recreatePods: false
  force: true

templates:
  default: &default
    chart: myrepo/{{`{{ .Release.Name }}`}}
    namespace: {{ .Namespace }}
    missingFileHandler: Debug
    wait: true
    values:
      - environment/{{`{{ .Environment.Name }}`}}/{{`{{ .Release.Name }}`}}.yaml
    secrets:
      - environment/{{`{{ .Environment.Name }}`}}/{{`{{ .Release.Name }}`}}-secrets.yaml

releases:
{{ if env "DEPLOY_NGINX" | default false }}
  - name: nginx-ingress
    <<: *default
    chart: stable/nginx-ingress
    version: ^1.26.2
    namespace: ingress-nginx
    set:
      - name: rbac.create
        value: true
      - name: controller.image.pullPolicy
        value: "Always"
      - name: controller.stats.enabled
        value: true
      - name: controller.service.type
        value: "ClusterIP"
      - name: controller.kind
        value: "DaemonSet"
      - name: controller.daemonset.useHostPort
        value: true
{{ end }}
  - name: mysql
    version: 1.3.1
    chart: stable/mysql
    values:
      - MYSQL_ROOT_PASSWORD: super_super_secret!!!
    set:
      - name: readinessProbe.initialDelaySeconds
        value: 120
      - name: readinessProbe.timeoutSeconds
        value: 5
  - name: stable/mongo
    <<: *default
    version: 7.2.9
  - name: redis
    <<: *default
    version: 10.2.1
    set:
      - name: usePassword
        value: true
  - name: fluentd
    <<: *default
    values:
      - environment/fluentd.yaml
    chart: stable/fluentd
    version: 2.1.3

For more examples see https://github.com/roboll/helmfile/tree/master/examples .

Don’t forget to set the HELM_REPO_PASS as a environment variables (in GitLab) so Helm(file) can access your private chart repo.

Dockerfile

Why Docker? I needed a bunch of tools that aren’t available in other images (yet?) so I built it myself.

I need the following:

Go
Helm + plugins
- helm-git (to use a git repo as a chart repo)
- helm-diff
- helm-secrets
Helmfile
kubectl
SOPS (depends of helm-secrets)

FROM golang:1.13-buster
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get install -y locales \
    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8

# SOPS
ENV SOPS_VERSION 3.5.0
RUN DPKG_ARCH=$(dpkg --print-architecture); \
    wget -q https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops_${SOPS_VERSION}_amd64.deb; \
    apt install ./sops_${SOPS_VERSION}_amd64.deb; \
    rm sops_${SOPS_VERSION}_amd64.deb

# HELM
ENV HELM_VERSION v3.0.1
RUN wget -q https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz; \
    tar -zxvf helm-${HELM_VERSION}-linux-amd64.tar.gz; \
    rm helm-${HELM_VERSION}-linux-amd64.tar.gz; \
    chmod +x linux-amd64/helm; \
    mv linux-amd64/helm /usr/bin/helm

# HELMFILE
ENV HELMFILE_VERSION v0.94.1
RUN wget -q https://github.com/roboll/helmfile/releases/download/${HELMFILE_VERSION}/helmfile_linux_amd64 -O /usr/bin/helmfile; \
    chmod +x /usr/bin/helmfile

# HELM PLUGINS
RUN helm plugin install https://github.com/aslafy-z/helm-git
RUN helm plugin install https://github.com/databus23/helm-diff
RUN helm plugin install https://github.com/futuresimple/helm-secrets

# kubectl
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl; \
    chmod +x ./kubectl; \
    mv ./kubectl /usr/local/bin/kubectl

Pipeline yaml

First lets build the image we’ll use later on to actually deploy, as listed below, there is an only/changes/Dockerfile in here. The reason is simple, the image isn’t affected by the content of the repository, if the Helmfile changes it won’t affect the image so there is no reason to rebuild it.

image: docker:git

services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay

stages:
  - build
  - lint
  - deploy
  - cleanup

build-image:
  stage: build
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME" .
    - docker push "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME"
    - docker push "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:latest"
  only:
    changes:
      - Dockerfile

Lint

The next step is linting, this is the first step where we use the above built image. This works because the image is run with the repository as context, i.e. I can run the script and it will be run against my Helmfile.

Yes this can be done automagically via git hook pre-commit or pre-push but I don’t always have everything installed on my local workstation.

lint:
  image: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
  stage: lint
  script:
    - helmfile lint

Deploy && Cleanup

Once I know the helmfile is valid I proceed to the next step and deploying it to Kubernetes. This step is split up into two parts, one for branches other than master and master itself. The difference is the subdomain and the cleanup step.

deploy_review:
  image: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
  stage: deploy
  script:
    - helmfile --namespace $KUBE_NAMESPACE sync
  # when: manual
  variables:
    BASE_DOMAIN: $CI_COMMIT_REF_NAME.0x43.nl
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: http://$CI_COMMIT_REF_NAME.0x43.nl
    on_stop: stop_review
  only:
    - branches
  except:
    - master

stop_review:
  image: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
  stage: cleanup
  when: manual
  script:
    - echo "Stopping" # script is required
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop

deploy_master:
  stage: deploy
  script:
    - helmfile --namespace $KUBE_NAMESPACE sync
  variables:
    BASE_DOMAIN: 0x43.nl
  environment:
    name: production
    url: http://www.0x43.nl
  when: manual
  only:
    - master

The rest should be pretty straight forward, Kubernetes config and setup can all be managed via GitLab.

Gitlab CI/CD and Helm + file

Using CI/CD, Helm and Helmfile to deploy