Efficient DevSecOps Pipelines in a Cloud Native World

Getting started with CI/CD has become more easy with the DevOps platform era. GitLab provides a new pipeline editor to help with onboarding, and allow to prevent common configuration mistakes.

Pipelines and their definitions are always changing, and a work in progress similar to source code development. It is easy to create a long lasting pipeline which blocks reviews from fast feedback, or waste ten thousands of dollars of cloud compute resources.

The overall question is - how to create efficient DevSecOps pipelines, and make use of all available options: Configuration, resources, CI/CD infrastructure, Security and Observability.

Based on the talk I had created after creating the CI/CD Pipeline Efficiency docs after a CI monitoring webinar ...

... I have now built a 10+ hours workshop for Open Source Automation Days 2021 with old and new best practices for everyone to learn and benefit. I've added new topics focussing on auto-scaling and cloud native deployments, whilst enriching the configuration examples with many hidden gems.

Yesterday, a discussion about a great blog post "Let's make faster GitLab CI/CD pipelines" on Hacker News inspired me to share the workshop and now blog about it. You can find all workshops in the GitLab Developer Evangelism handbook.

You can learn async at your own pace, with the slides providing the exercises and the solutions in a workshop project. Note that there is one exercise which needs preparations for auto-scaling which is described in the slides.

The following topics will be practiced:

Introduction: CI/CD meets Dev, Sec and Ops
CI/CD: Terminology and first steps
Pipeline editor
Analyse & Identify
Learn using the GitLab CI Pipeline Exporter to monitor the exercise project throughout the workshop.
Efficiency actions
Config Efficiency: CI/CD Variables in variables, job templates (YAML anchors, extends), includes (local, remote), rules and conditions (if, dynamic variables, conditional includes), !reference tags (script, rules), maintain own CI/CD templates (include templates, override config values), parent-child pipelines, multi project pipelines, better error messages to fix failures fast
Resource Use Efficiency: Identification, max pipeline duration analysis, fail fast with stages grouping, fail fast with async needs, analyse blocking stages pipeline (solution with needs), matrix builds for parallel execution (pratice: combine matrix and extends, combine matrix and !reference), extends merge strategies (with and without !reference)
CI/CD Infrastructure Efficiency: Optimization ideas, custom build images, optimize builds with C++ as example, GitLab runner resource analysis (sharing, tags, external dependencies, Kubernetes), local runner exercise, resource groups, storage usage analysis, caching (Python dependency exercise, including when:always on failed jobs)
Auto-scaling: Overview, AWS auto-scaling with GitLab Runner with Terraform, insights into Spot Runners on AWS Graviton
Group discussion
Deployment Strategies: IaC, GitOps, Terraform, Kubernetes, registries
Security: Secrets in CI/CD variables, Hashicorp Vault, secrets scanning, vulnerability scanning
Observability: CI/CD Runner monitoring, SLOs, quality gates, CI/CD Tracing
More efficiency ideas: Auto DevOps, Fast vs Resources, Conclusion and tips

Take your time with learning, the workshop has a lot of topics inside. Let me know how it goes, and join https://forum.gitlab.com/ for questions and feedback (please tag @dnsmichi). 🤗

2021-10-04 OSAD: Efficient DevSecOps Pipelines in a Cloud Native World Workshop

Efficient DevSecOps Pipelines in a Cloud Native World workshop Michael Friedrich Senior Developer Evangelist Organisation: https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/issues/4358 Repositories with exercises and solutions: https://gitlab.com/gitlab-de/workshop...

Google Docs

Everyone Can Contribute - learned a new gem from this workshop? Blog about it and share on social tagging @dnsmichi @gitlab! 💡

Blog post also published on https://dev.to/dnsmichi/efficient-devsecops-pipelines-in-a-cloud-native-world-free-workshop-hmk