gitlab

Reusable job attributes in other jobs in GitLab CI/CD with !reference

Michael Friedrich

Sep 17, 2021 • 4 min read

Today I wrote some Python code and figured that I do not have linting available with pyflakes in CI/CD. Been reading this great blog post and built a short snippet:

stages:
- lint  
- test

lint-python:
  stage: lint
  image: python:latest
  script:
    - pip install -r python/requirements.txt
    - pip install pyflakes  
    - pyflakes python/

As a developer, you spot repeating blocks

Installing the requirements with pip, and pyflakes could potentially be used by more jobs.

"Let's keep using best practices with re-usability" - my brain is at preparing a Pipeline Efficiency workshop for the Open Source Automation Days.

Easy peasy, let's use a job template to keep the Python requirements in a central place, and use extends to import it into the lint-python job. Everything done in the pipeline editor with live linting.

Pipeline runs, and fails. HUH?

Problem: Extending job templates and overrides

After staring at the third pipeline retry, I decided to take a step back. I went into the CI/CD pipeline editor, and instead of viewing my code, I switched into the merged YAML tab.

My mistake was to assume that the imported script section from .python-req gets automatically merged with the script section in the lint-python job. That's not the case - as with other DSLs, an object attribute is overridden when specified in a later scope. I developed a DSL some years ago myself where this discussion also came up, now I remember.

.python-req:
  script:
    - pip install -r python/requirements.txt
    - pip install pyflakes

lint-python:
  extends: .python-req
  stage: lint
  image: python:latest
  script:
    - pyflakes python/

gets merged to

".python-req":
  script:
  - pip install -r python/requirements.txt
  - pip install pyflakes
lint-python:
  script:
  - pyflakes python/
  extends: ".python-req"
  stage: lint
  image: python:latest

I am overriding the script section after importing the job template .python-req. Amazing. Not.

Is there a way to modify the script array after importing a job template, or at least keep the job template and import its attributes key?

Solution: !reference job attributes in other jobs

I remembered reviewing an upcoming GitLab 14.3 feature with importing rules into a job and merging array lists. Hmmmmm. Let's open the documentation and learn more about !reference.

Let's modify the script section in lint-python to !reference the .python-req job template's script section, as in - !reference [.python-req, script]

.python-req:
  script:
    - pip install -r python/requirements.txt
    - pip install pyflakes

lint-python:
  extends: .python-req
  stage: lint
  image: python:latest
  script:
    - !reference [.python-req, script]
    - pyflakes python/

The pipeline editor renders !reference as unknown tag, opened an issue. The merged YAML view also needs a better representation of the merged script array, opened an issue.

Now the pipeline installs the requirements and pyflakes, and errors out because my code is not good enough. Amazing. Yes :)

You can find the MR with the CI/CD configuration here:

and the fixed code at

In the future, programming mistakes are detected earlier. Users who peek into the examples in the API playground project will benefit too. I use it often to answer questions in the GitLab community forum :)

Key learnings

Use the GitLab CI/CD Pipeline Editor to write configuration with live linting
Take advantage of verifying extends and !reference with the merged YAML tab.
Use !reference to reuse job attributes in other job scopes

The docs also say that !reference works across include 'd files. This is beyond amazing!

Use the !reference custom YAML tag to select keyword configuration from other job sections and reuse it in the current section. Unlike YAML anchors, you can use !reference tags to reuse configuration from included configuration files as well.