Reusable job attributes in other jobs in GitLab CI/CD with !reference
Today I wrote some Python code and figured that I do not have linting available with pyflakes in CI/CD. Been reading this great blog post and built a short snippet:
stages: - lint - test lint-python: stage: lint image: python:latest script: - pip install -r python/requirements.txt - pip install pyflakes - pyflakes python/
As a developer, you spot repeating blocks
Installing the requirements with pip, and pyflakes could potentially be used by more jobs.
"Let's keep using best practices with re-usability" - my brain is at preparing a Pipeline Efficiency workshop for the Open Source Automation Days.
Easy peasy, let's use a job template to keep the Python requirements in a central place, and use
extends to import it into the
lint-python job. Everything done in the pipeline editor with live linting.
Pipeline runs, and fails. HUH?
Problem: Extending job templates and overrides
After staring at the third pipeline retry, I decided to take a step back. I went into the CI/CD pipeline editor, and instead of viewing my code, I switched into the merged YAML tab.
My mistake was to assume that the imported
script section from
.python-req gets automatically merged with the
script section in the
lint-python job. That's not the case - as with other DSLs, an object attribute is overridden when specified in a later scope. I developed a DSL some years ago myself where this discussion also came up, now I remember.
.python-req: script: - pip install -r python/requirements.txt - pip install pyflakes lint-python: extends: .python-req stage: lint image: python:latest script: - pyflakes python/
gets merged to
".python-req": script: - pip install -r python/requirements.txt - pip install pyflakes lint-python: script: - pyflakes python/ extends: ".python-req" stage: lint image: python:latest
I am overriding the
script section after importing the job template
.python-req. Amazing. Not.
Is there a way to modify the script array after importing a job template, or at least keep the job template and import its attributes key?
Solution: !reference job attributes in other jobs
I remembered reviewing an upcoming GitLab 14.3 feature with importing rules into a job and merging array lists. Hmmmmm. Let's open the documentation and learn more about !reference.
Let's modify the
script section in
.python-req job template's
script section, as in
- !reference [.python-req, script]
.python-req: script: - pip install -r python/requirements.txt - pip install pyflakes lint-python: extends: .python-req stage: lint image: python:latest script: - !reference [.python-req, script] - pyflakes python/
The pipeline editor renders
!reference as unknown tag, opened an issue. The merged YAML view also needs a better representation of the merged
script array, opened an issue.
Now the pipeline installs the requirements and pyflakes, and errors out because my code is not good enough. Amazing. Yes :)
You can find the MR with the CI/CD configuration here:
and the fixed code at
In the future, programming mistakes are detected earlier. Users who peek into the examples in the API playground project will benefit too. I use it often to answer questions in the GitLab community forum :)
- Use the GitLab CI/CD Pipeline Editor to write configuration with live linting
- Take advantage of verifying
!referencewith the merged YAML tab.
!referenceto reuse job attributes in other job scopes
The docs also say that
!reference works across
include 'd files. This is beyond amazing!
!referencecustom YAML tag to select keyword configuration from other job sections and reuse it in the current section. Unlike YAML anchors, you can use
!referencetags to reuse configuration from included configuration files as well.