GitLab CI Cheat-Sheet
This is a cheat-sheet that I've put together that summarizes some of the most useful commands that I personally use the most when I'm working with GitLab CI.
GitLab CI is a powerful tool that helps you implement Continuous Delivery. It
uses .gitlab-ci.yml file as it's main configuration file. This file can be
split into multiple configuration files, but the commands are still the same.
This file defines the whole continuous delivery and continuous integration (CI/CD) pipeline which you can then use enforce quality control on source code changes automatically.
In this article we will focus on this configuration file.
Configuration Sections
stages: Defines the stages in your pipeline (e.g., build, test, deploy).image: Specifies the Docker image to use for the job environment.variables: Defines variables globally for the pipeline.cache: Configures caching between jobs to speed up pipeline execution.before_script: Commands that run before each job's script.script: The main set of commands executed by the job.after_script: Commands that run after each job's script.only/except: Controls when jobs should be executed based on branches or tags.
Here is an example of a gitlab pipeline configuration:
stages:
- build
- test
- deploy
variables:
PROJECT_NAME: "project-id"
build:
stage: build
image: ubuntu:latest
script:
- echo "Building the application..."
- make
test:
stage: test
image: python:3.9
script:
- echo "Running tests..."
- pytest
deploy:
stage: deploy
image: alpine:latest
script:
- echo "Deploying application to production..."
only:
- mainGitlab will execute all steps in each stage and then move on to the next stage. Through options you can override this behavior and build in any order (and specify dependencies between stages as well).
The full reference of gitlab ci file can be found here (opens in a new tab).
Pipeline Configuration
- Define Stages:
stages: [build, test, deploy]- Outlines the primary stages of the CI/CD pipeline. - Use Docker Image:
image: docker:19.03.12- Specifies the Docker image to use for all jobs. - Specify a Service:
services: [docker:dind]- Uses Docker-in-Docker as a service for building containers. - Dependencies: Use
dependenciesto control which artifacts are passed between jobs. - Environment: Use
environmentto define the environment to which the job deploys artifacts. - Include: Use
includeto include external YAML files for reusable configurations.
Job Definitions
- Run Only on Main Branch:
only: [main]- Ensures the job runs only on the main branch. - Cache Dependencies:
cache: { paths: [node_modules/] }- Caches directories between jobs to improve speed. - Define Artifacts:
artifacts: { paths: [build/] }- Specifies files or directories to be saved as artifacts. - Parallel Execution:
parallel: 4- Runs the job in four parallel instances. - Job Artifacts: Use
artifactsto specify files and directories that are created by a job and need to be saved or used by another job.
Script Execution
- Custom Script Execution:
script: echo "Hello, World!"- Executes a custom script command. - Before and After Scripts:
before_script: [echo "Setup"]- Runs commands before the job's script.
Conditional Execution
- Conditional Job Execution:
rules: [{ if: '$CI_COMMIT_BRANCH == "main"', when: always }]- Executes a job based on a specific condition. - Skip Job Unless Manual Trigger:
rules: [{ if: '$CI_PIPELINE_SOURCE == "push"', when: never }]- Prevents job execution unless manually triggered.
Environment and Deployment
- Set Deployment Environment:
environment: production- Marks the job as deploying to the production environment. - Dynamic Environment Names:
environment: { name: review/$CI_COMMIT_REF_NAME }- Uses dynamic names for environments.
Advanced Features
- Trigger Another Project:
trigger: { project: my-group/my-project, branch: main }- Triggers a pipeline in another project. - Include External YAML:
include: 'https://example.com/gitlab-ci.yml'- Includes an external CI/CD configuration. - Use Variables in Scripts:
script: echo "Deploying version $VERSION"- Utilizes predefined or custom variables in scripts. - Manual Job Trigger:
when: manual- Requires manual intervention to trigger the job.
Optimization and Efficiency
- Artifact Expiry:
artifacts: { expire_in: 1 week }- Sets artifacts to expire and be deleted after a week. - Cache Key Customization:
cache: { key: "$CI_COMMIT_REF_SLUG", paths: [cache/] }- Customizes the cache key for uniqueness. - Specific Changes Trigger:
only: { changes: [README.md] }- Runs the job only if specific files are changed.
Include
Use include to import shared CI/CD configurations, keeping your
.gitlab-ci.yml clean and DRY (Don’t Repeat Yourself). This is particularly
useful for large projects or when maintaining common standards across multiple
projects.
include:
- project: 'my-group/my-project'
file: '/templates/.common-ci-template.yml'Multi Project Pipelines
trigger: Triggers a pipeline in another project. Useful for orchestrating larger workflows across multiple projects.
deploy_downstream_job:
stage: deploy
script:
- echo "Triggering downstream project pipeline"
trigger:
project: my-group/my-downstream-project
branch: mainDynamic Child Pipelines
trigger:include: Dynamically generates and includes pipeline configuration from a script output. Enables dynamic generation of pipeline configurations.
create_child_pipeline:
stage: build
script: echo "Generating child pipeline configuration..."
artifacts:
paths:
- dynamic-config.yml
trigger:
include: dynamic-config.yml
strategy: dependEnvironments
Deployment environments make gitlab automatically link to latest deployment from the merge request making it easy to review changes.
environment: Define deployment environments. Supports dynamic environments using variables for branch-specific deployments.
deploy_to_production:
stage: deploy
script: echo "Deploying application..."
environment:
name: production
url: https://production.example.com
only:
- mainRules for jobs
Rules determine when a job runs. Rules are more powerful than gitlab only
keyword.
rules: Defines conditions under which a job will be included or excluded from the pipeline, replacing the only/except syntax for more complex logic.
test:
stage: test
script: echo "Running tests..."
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: manualRules can also include a job only if some files have been changed:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- changes:
- Dockerfile
- "/src/**/*"Another kind of conditional job execution:
deploy_production:
stage: deploy
script:
- make deploy
when: manual
only:
# Only release branches
- /^release-.*$/Artifacts and caching
Artifacts are your primary way to reuse produced binaries between jobs. Use artifacts for compilation output and use cache for files that need to be downloaded but do not change very often.
build:
stage: build
script:
- make build
artifacts:
paths:
- build/
cache:
paths:
- .cache/
test:
stage: test
script:
- make test
dependencies:
- build
cache:
paths:
- .cache/Note that by default artifacts are saved only on success. If you want to save
them regardless of pipeline status you should specify when: always under
artifacts key.
If your jobs need to download a lot of dependencies (like pip or npm packages)
then caching will help you reuse these artifacts across repeated pipeline runs.
Use $CI_COMMIT_REF_SLUG to tag the cache using the branch name.
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm/
policy: pull-pushParallel jobs
Gitlab can create jobs based on a parameter matrix. This is useful when you want to execute many different combinations of builds and do not want to define exponentially large number of pipelines.
test:
stage: test
script:
- make test
parallel:
matrix:
- VAR1: foo
VAR2: ["variation1", "variation2"]
- VAR1: bar
VAR2: ["variation3", "variation4"]This will automatically create jobs test:foo:variation1, test:foo:variation2,
test: bar:variation3 and test:bar:variation4.
Another form of parallel job execution is using parallel is with a number.
build:
stage: build
script: build.sh --output build-$CI_NODE_INDEX.zip
parallel: 4
artifacts:
paths:
- build-$CI_NODE_INDEX.zipWhen using parallel jobs to generate artifacts, make sure each job produces a
uniquely named artifact to prevent conflicts. You can use environment variables
like $CI_JOB_ID or $CI_NODE_INDEX to differentiate them. $CI_NODE_TOTAL
will be equal to 4 in the above configuration.
Custom Docker Images
I think the most powerful aspect of gitlab CI is the ability to run everything inside docker. We will not go into details of creating docker images in this article but in most cases you want to use your own docker image for jobs. The job will run inside the image and this way you can have a predictable starting point every single time. All build tools should be already included in the image.
build:
stage: build
image: my-custom-image:latest
script:
- build scriptDocker Services
Use docker services when you want to run your code against a server. This can even be used to end-to-end test a docker image that is built in the same pipeline.
Gitlab runner will start the services as separate docker containers that are part of the same network as the job.
test:
stage: test
services:
- name: postgres:latest
alias: db
variables:
DATABASE_URL: "postgresql://postgres:@db:5432/my_database"
script:
- ./test.shalias is the name of the service machine so you can use it in place of a
hostname when you need to communicate with the service.
Dynamic Environments
- Dynamic Environments: Dynamically create environments for each branch or merge request, allowing stakeholders to review changes in a live environment.
review:
stage: deploy
script: deploy_to_review_app.sh
environment:
name: review/$CI_COMMIT_REF_NAME
url: https://$CI_COMMIT_REF_NAME.review.example.com
on_stop: stop_reviewCleanup
Implement an after_script section in your .gitlab-ci.yml file. This ensures
that cleanup commands run after the job's script section completes, regardless
of whether the job succeeds or fails.
test:
script:
- execute_tests.sh
after_script:
- cleanup_resources.shThis is particularly useful for releasing resources, deleting temporary files, or shutting down services that were started during the job.
Environment Cleanup
For jobs that deploy to dynamic environments, use the on_stop keyword to
specify a job that runs to clean up the environment when it's stopped manually
from the GitLab UI.
deploy_review_app:
environment:
name: review/$CI_COMMIT_REF_NAME
on_stop: stop_review_app
stop_review_app:
stage: cleanup
script:
- echo "Cleaning up review app resources..."
when: manual
environment:
name: review/$CI_COMMIT_REF_NAME
action: stopThis method is particularly effective for cleaning up environments used for review apps or temporary deployments.
Conditional Cleanup
GitLab CI/CD variables like $CI_JOB_STATUS can be used to conditionally
execute cleanup steps based on the outcome of previous jobs or stages.
after_script:
- if [ "$CI_JOB_STATUS" = "failed" ]; then echo "Running cleanup after failure"; fiOr use custom checks
cleanup:
script:
- if ./check_resource_status.sh; then ./cleanup.sh; fiFor critical cleanup tasks, consider using the retry keyword to attempt the
cleanup multiple times in case of failures. You can also set allow_failure: true to prevent the cleanup job from causing the entire pipeline to fail if it
can't complete successfully.
cleanup_resources:
script:
- robust_cleanup_command
allow_failure: true
retry:
max: 3
when: on_failureDocker and Services for Test Environments
Use Docker containers to create consistent, isolated environments for your simulations and system tests. This ensures tests run in a controlled environment, mirroring production settings.
test:
stage: test
image: docker:19.03.12
services:
- postgres:12
script:
- execute_system_tests.shUse artifacts for test results and debugging
Configure jobs to generate and store artifacts like logs, screenshots, or reports from your simulation or system tests. This is invaluable for debugging and historical analysis.
simulation_test:
stage: test
script: run_simulation.sh
artifacts:
paths:
- logs/
- screenshots/
expire_in: 1 weekUse retry for flaky tests
For tests that are known to be flaky and might fail intermittently due to
external dependencies, use the retry keyword to automatically re-run failed
tests.
flaky_test:
stage: test
script: possibly_flaky_test.sh
retry: 2Schedule Full System Tests
Schedule full system tests to run during off-peak hours. This allows for thorough testing without impacting the development workflow.
stages:
- simulation
nightly_simulation_test:
stage: simulation
script: run_full_simulation_suite.sh
only:
- schedulesClear Pipeline Stages
- Logical Grouping: Organize your pipeline into clear, logical stages such
as
build,test,deploy, etc. This structure enhances readability and understanding of the pipeline flow.
stages:
- build
- test
- deployUse dependencies
Use the dependencies keyword to explicitly specify which artifacts a job
should receive from previous stages. This minimizes unnecessary data transfer
and speeds up job execution.
deploy:
stage: deploy
dependencies:
- build
script:
- deploy_script.shUse needs
The needs keyword allows you to create non-linear pipelines (DAG), enabling
jobs to start as soon as their dependencies are completed rather than waiting
for the entire previous stage to finish. This can significantly reduce pipeline
execution times.
unit_test:
stage: test
script: run_unit_tests.sh
integration_test:
stage: test
script: run_integration_tests.sh
needs: ["unit_test"]Use Anchors and Aliases
Avoid repetition by using YAML anchors (&) and aliases (*) to reuse
configurations in your GitLab CI file. This is particularly useful for repeated
script blocks or environment variables.
.build_template: &build_definition
script:
- echo "Building project"
- build_command
build_job:
<<: *build_definition
stage: buildUse extends
Use the extends keyword to inherit from one or more templates. This allows for
more flexibility compared to anchors and aliases, especially for overriding
parts of the configuration.
.template:
script:
- setup_command
job1:
extends: .template
script:
- additional_command # This will be executed after `setup_command`.Use variables for Global and Local Scopes
Define global variables at the top of your .gitlab-ci.yml for use across all
jobs, and override or extend them in individual jobs as needed.
variables:
GLOBAL_VAR: "global_value"
job1:
variables:
LOCAL_VAR: "local_value"
script:
- echo $GLOBAL_VAR
- echo $LOCAL_VARStore Secrets in Variables
Use CI/CD variables for sensitive data like tokens or passwords. Define them in
the GitLab UI to keep them out of your .gitlab-ci.yml.
deploy:
stage: deploy
script:
- deploy_command --token $DEPLOY_TOKENUse comments in YAML
Use comments (#) liberally to explain complex configurations, the purpose of
jobs, or why specific settings are used. This aids in maintainability.
# This job builds the project
build:
stage: build
script:
- echo "Building"
# Temporary workaround for XYZ issue
- build_workaround_command