Gitlab CI/CD and PyPi -

A short post about how I use GitLab CI/CD to automagically test and deploy to PyPi .

Logically the yml consists of 3 parts which coincides with the stage, each stage contains jobs;

test
test-deploy
deploy

If one of the jobs fail, the pipeline is stopped and the deployment is canceled. This should prevent me from releasing faulty code.

In practice we split up the repeating parts to keep the yml DRY . In yml we can do .test: &test_template which means we use the content of .test as a variable, a copy paste if you will, within the yml. Later on, the paste part, we call on the test_template name and “paste” it into another section, optionally overriding parts. To do this we use <<: *test_template were it should be “pasted”.

Side note: properties within a yml are order based, if you use <<: *test_template first your variable, stage for example will override the value of stage from the test_template. If you declare your stage first the template will override it. Take a look at deploy_template and deploy:test: for an example.

stages:
  - test
  - test-deploy
  - deploy

cache:
  paths:
    - ~/.cache/pip/

.test: &test_template
  stage: test
  before_script:
    - pip install -r requirements.txt
  script:
    - python -m unittest discover

test:PY3.6:
  <<: *test_template
  image: python:3.6

test:PY3.7:
  <<: *test_template
  image: python:3.7

test:PY3.8:
  <<: *test_template
  image: python:3.8

test:coverage:
  stage: test
  image: python:3.7
  before_script:
    - pip install -r requirements.txt
    - pip install coverage
  script:
    - coverage run --omit="/*venv*,/*tests*" -m unittest discover
    - coverage report -m

test:lint:
  stage: test
  image: python:3.7
  before_script:
    - pip install pylint
    - pip install -r requirements.txt
  script:
    - pylint --max-line-length=120 --exit-zero solc

.deploy: &deploy_template
  stage: deploy
  image: python:3.7
  before_script:
    - pip install -r requirements.txt
    - pip install setuptools wheel twine
    - python setup.py sdist bdist_wheel

deploy:test:
  <<: *deploy_template
  stage: test-deploy
  variables:
    TWINE_USERNAME: $TEST_USERNAME
    TWINE_PASSWORD: $TEST_PASSWORD
  script:
    - twine upload --repository-url https://test.pypi.org/legacy/  dist/*
  only:
    - tags

deploy:live:
  <<: *deploy_template
  variables:
    TWINE_USERNAME: $LIVE_USERNAME
    TWINE_PASSWORD: $LIVE_PASSWORD
  script:
    - twine upload dist/*
  only:
    - tags

Pipeline

The above yml will result in a pipeline as displayed below, the 3 stages clearly split into columns. The testing of the deployment and the actual deployment are only ran on tags.

GitLab pipeline

Keeping secrets

To publish to PyPi you need an account with username and password but you do not want these somewhere hardcoded and embeded in git for everyone to see. To handle these variable we get them, on run time, from the environment, in this case via the variables option from GitLab.

Note: We marked the passwords as masked to prevent them from showing up in the CI/CD pipeline/jobs logs.

GitLab variables

Coverage

In one of the jobs I check the coverage of the code, to get GitLab to check for the coverage we need to provide a regex to parse the output of the jobs. We do not need to set a specific job to parse, as far as I know they are all parsed. The regex I use, ^TOTAL\s+\d+\s+\d+\s+((?:\d+\%)|(?:\d+.\d+\%))$ is used for Python unittests.

GitLab test coverage parsing

Gitlab CI/CD and PyPi

Using CI/CD to publish

Gitlab CI/CD and PyPi

Using CI/CD to publish

Pipeline

Keeping secrets

Coverage

See also