GitLab CI Cheat-Sheet

This is a cheat-sheet that I’ve put together that summarizes some of the most useful commands that I personally use the most when I’m working with GitLab CI.

GitLab CI is a powerful tool that helps you implement Continuous Delivery. It uses .gitlab-ci.yml file as it’s main configuration file. This file can be split into multiple configuration files, but the commands are still the same.

This file defines the whole continuous delivery and continuous integration (CI/CD) pipeline which you can then use enforce quality control on source code changes automatically.

In this article we will focus on this configuration file.

Configuration Sections

stages: Defines the stages in your pipeline (e.g., build, test, deploy).
image: Specifies the Docker image to use for the job environment.
variables: Defines variables globally for the pipeline.
cache: Configures caching between jobs to speed up pipeline execution.
before_script: Commands that run before each job’s script.
script: The main set of commands executed by the job.
after_script: Commands that run after each job’s script.
only/except: Controls when jobs should be executed based on branches or tags.

Here is an example of a gitlab pipeline configuration:


    stages:
      - build
      - test
      - deploy
 
    variables:
      PROJECT_NAME: "project-id"
 
    build:
      stage: build
      image: ubuntu:latest
      script:
        - echo "Building the application..."
        - make
 
    test:
      stage: test
      image: python:3.9
      script:
        - echo "Running tests..."
        - pytest
 
    deploy:
      stage: deploy
      image: alpine:latest
      script:
        - echo "Deploying application to production..."
      only:
        - main

Gitlab will execute all steps in each stage and then move on to the next stage. Through options you can override this behavior and build in any order (and specify dependencies between stages as well).

The full reference of gitlab ci file can be found here .

Pipeline Configuration

Define Stages: stages: [build, test, deploy] - Outlines the primary stages of the CI/CD pipeline.
Use Docker Image: image: docker:19.03.12 - Specifies the Docker image to use for all jobs.
Specify a Service: services: [docker:dind] - Uses Docker-in-Docker as a service for building containers.
Dependencies: Use dependencies to control which artifacts are passed between jobs.
Environment: Use environment to define the environment to which the job deploys artifacts.
Include: Use include to include external YAML files for reusable configurations.

Job Definitions

Run Only on Main Branch: only: [main] - Ensures the job runs only on the main branch.
Cache Dependencies: cache: { paths: [node_modules/] } - Caches directories between jobs to improve speed.
Define Artifacts: artifacts: { paths: [build/] } - Specifies files or directories to be saved as artifacts.
Parallel Execution: parallel: 4 - Runs the job in four parallel instances.
Job Artifacts: Use artifacts to specify files and directories that are created by a job and need to be saved or used by another job.

Script Execution

Custom Script Execution: script: echo "Hello, World!" - Executes a custom script command.
Before and After Scripts: before_script: [echo "Setup"] - Runs commands before the job’s script.

Conditional Execution

Conditional Job Execution: rules: [{ if: '$CI_COMMIT_BRANCH == "main"', when: always }] - Executes a job based on a specific condition.
Skip Job Unless Manual Trigger: rules: [{ if: '$CI_PIPELINE_SOURCE == "push"', when: never }] - Prevents job execution unless manually triggered.

Environment and Deployment

Set Deployment Environment: environment: production - Marks the job as deploying to the production environment.
Dynamic Environment Names: environment: { name: review/$CI_COMMIT_REF_NAME } - Uses dynamic names for environments.

Advanced Features

Trigger Another Project: trigger: { project: my-group/my-project, branch: main } - Triggers a pipeline in another project.
Include External YAML: include: 'https://example.com/gitlab-ci.yml' - Includes an external CI/CD configuration.
Use Variables in Scripts: script: echo "Deploying version $VERSION" - Utilizes predefined or custom variables in scripts.
Manual Job Trigger: when: manual - Requires manual intervention to trigger the job.

Optimization and Efficiency

Artifact Expiry: artifacts: { expire_in: 1 week } - Sets artifacts to expire and be deleted after a week.
Cache Key Customization: cache: { key: "$CI_COMMIT_REF_SLUG", paths: [cache/] } - Customizes the cache key for uniqueness.
Specific Changes Trigger: only: { changes: [README.md] } - Runs the job only if specific files are changed.

Include

Use include to import shared CI/CD configurations, keeping your .gitlab-ci.yml clean and DRY (Don’t Repeat Yourself). This is particularly useful for large projects or when maintaining common standards across multiple projects.


    include:
      - project: 'my-group/my-project'
        file: '/templates/.common-ci-template.yml'

Multi Project Pipelines

trigger: Triggers a pipeline in another project. Useful for orchestrating larger workflows across multiple projects.


    deploy_downstream_job:
      stage: deploy
      script:
        - echo "Triggering downstream project pipeline"
      trigger:
        project: my-group/my-downstream-project
        branch: main

Dynamic Child Pipelines

trigger:include: Dynamically generates and includes pipeline configuration from a script output. Enables dynamic generation of pipeline configurations.


    create_child_pipeline:
      stage: build
      script: echo "Generating child pipeline configuration..."
      artifacts:
        paths:
          - dynamic-config.yml
      trigger:
        include: dynamic-config.yml
        strategy: depend

Environments

Deployment environments make gitlab automatically link to latest deployment from the merge request making it easy to review changes.

environment: Define deployment environments. Supports dynamic environments using variables for branch-specific deployments.


    deploy_to_production:
      stage: deploy
      script: echo "Deploying application..."
      environment:
        name: production
        url: https://production.example.com
      only:
        - main

Rules for jobs

Rules determine when a job runs. Rules are more powerful than gitlab only keyword.

rules: Defines conditions under which a job will be included or excluded from the pipeline, replacing the only/except syntax for more complex logic.


    test:
      stage: test
      script: echo "Running tests..."
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
          when: always
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          when: manual

Rules can also include a job only if some files have been changed:


    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - changes:
            - Dockerfile
            - "/src/**/*"

Another kind of conditional job execution:


    deploy_production:
      stage: deploy
      script:
        - make deploy
      when: manual
      only:
        # Only release branches
        - /^release-.*$/

Artifacts and caching

Artifacts are your primary way to reuse produced binaries between jobs. Use artifacts for compilation output and use cache for files that need to be downloaded but do not change very often.


    build:
      stage: build
      script:
        - make build
      artifacts:
        paths:
          - build/
      cache:
        paths:
          - .cache/
 
    test:
      stage: test
      script:
        - make test
      dependencies:
        - build
      cache:
        paths:
          - .cache/

Note that by default artifacts are saved only on success. If you want to save them regardless of pipeline status you should specify when: always under artifacts key.

If your jobs need to download a lot of dependencies (like pip or npm packages) then caching will help you reuse these artifacts across repeated pipeline runs. Use $CI_COMMIT_REF_SLUG to tag the cache using the branch name.


    cache:
      key: ${CI_COMMIT_REF_SLUG}
      paths:
        - .npm/
      policy: pull-push

Parallel jobs

Gitlab can create jobs based on a parameter matrix. This is useful when you want to execute many different combinations of builds and do not want to define exponentially large number of pipelines.


    test:
      stage: test
      script:
        - make test
      parallel:
        matrix:
          - VAR1: foo
            VAR2: ["variation1", "variation2"]
          - VAR1: bar
            VAR2: ["variation3", "variation4"]

This will automatically create jobs test:foo:variation1, test:foo:variation2, test: bar:variation3 and test:bar:variation4.

Another form of parallel job execution is using parallel is with a number.


    build:
      stage: build
      script: build.sh --output build-$CI_NODE_INDEX.zip
      parallel: 4
      artifacts:
        paths:
          - build-$CI_NODE_INDEX.zip

When using parallel jobs to generate artifacts, make sure each job produces a uniquely named artifact to prevent conflicts. You can use environment variables like $CI_JOB_ID or $CI_NODE_INDEX to differentiate them. $CI_NODE_TOTAL will be equal to 4 in the above configuration.

Custom Docker Images

I think the most powerful aspect of gitlab CI is the ability to run everything inside docker. We will not go into details of creating docker images in this article but in most cases you want to use your own docker image for jobs. The job will run inside the image and this way you can have a predictable starting point every single time. All build tools should be already included in the image.


    build:
      stage: build
      image: my-custom-image:latest
      script:
        - build script

Docker Services

Use docker services when you want to run your code against a server. This can even be used to end-to-end test a docker image that is built in the same pipeline.

Gitlab runner will start the services as separate docker containers that are part of the same network as the job.


    test:
      stage: test
      services:
        - name: postgres:latest
          alias: db
      variables:
        DATABASE_URL: "postgresql://postgres:@db:5432/my_database"
      script:
        - ./test.sh

alias is the name of the service machine so you can use it in place of a hostname when you need to communicate with the service.

Dynamic Environments

Dynamic Environments: Dynamically create environments for each branch or merge request, allowing stakeholders to review changes in a live environment.


    review:
      stage: deploy
      script: deploy_to_review_app.sh
      environment:
        name: review/$CI_COMMIT_REF_NAME
        url: https://$CI_COMMIT_REF_NAME.review.example.com
        on_stop: stop_review

Cleanup

Implement an after_script section in your .gitlab-ci.yml file. This ensures that cleanup commands run after the job’s script section completes, regardless of whether the job succeeds or fails.


    test:
      script:
        - execute_tests.sh
      after_script:
        - cleanup_resources.sh

This is particularly useful for releasing resources, deleting temporary files, or shutting down services that were started during the job.

Environment Cleanup

For jobs that deploy to dynamic environments, use the on_stop keyword to specify a job that runs to clean up the environment when it’s stopped manually from the GitLab UI.


    deploy_review_app:
      environment:
        name: review/$CI_COMMIT_REF_NAME
        on_stop: stop_review_app
 
    stop_review_app:
      stage: cleanup
      script:
        - echo "Cleaning up review app resources..."
      when: manual
      environment:
        name: review/$CI_COMMIT_REF_NAME
        action: stop

This method is particularly effective for cleaning up environments used for review apps or temporary deployments.

Conditional Cleanup

GitLab CI/CD variables like $CI_JOB_STATUS can be used to conditionally execute cleanup steps based on the outcome of previous jobs or stages.


    after_script:
      - if [ "$CI_JOB_STATUS" = "failed" ]; then echo "Running cleanup after failure"; fi

Or use custom checks


cleanup:
  script:
    - if ./check_resource_status.sh; then ./cleanup.sh; fi

For critical cleanup tasks, consider using the retry keyword to attempt the cleanup multiple times in case of failures. You can also set allow_failure: true to prevent the cleanup job from causing the entire pipeline to fail if it can’t complete successfully.


    cleanup_resources:
      script:
        - robust_cleanup_command
      allow_failure: true
      retry:
        max: 3
        when: on_failure

Docker and Services for Test Environments

Use Docker containers to create consistent, isolated environments for your simulations and system tests. This ensures tests run in a controlled environment, mirroring production settings.


    test:
      stage: test
      image: docker:19.03.12
      services:
        - postgres:12
      script:
        - execute_system_tests.sh

Use artifacts for test results and debugging

Configure jobs to generate and store artifacts like logs, screenshots, or reports from your simulation or system tests. This is invaluable for debugging and historical analysis.


    simulation_test:
      stage: test
      script: run_simulation.sh
      artifacts:
        paths:
          - logs/
          - screenshots/
        expire_in: 1 week

Use `retry` for flaky tests

For tests that are known to be flaky and might fail intermittently due to external dependencies, use the retry keyword to automatically re-run failed tests.


    flaky_test:
      stage: test
      script: possibly_flaky_test.sh
      retry: 2

Schedule Full System Tests

Schedule full system tests to run during off-peak hours. This allows for thorough testing without impacting the development workflow.


    stages:
      - simulation
 
    nightly_simulation_test:
      stage: simulation
      script: run_full_simulation_suite.sh
      only:
        - schedules

Clear Pipeline Stages

Logical Grouping: Organize your pipeline into clear, logical stages such as build, test, deploy, etc. This structure enhances readability and understanding of the pipeline flow.


stages:
  - build
  - test
  - deploy

Use `dependencies`

Use the dependencies keyword to explicitly specify which artifacts a job should receive from previous stages. This minimizes unnecessary data transfer and speeds up job execution.


deploy:
  stage: deploy
  dependencies:
    - build
  script:
    - deploy_script.sh

Use `needs`

The needs keyword allows you to create non-linear pipelines (DAG), enabling jobs to start as soon as their dependencies are completed rather than waiting for the entire previous stage to finish. This can significantly reduce pipeline execution times.


    unit_test:
      stage: test
      script: run_unit_tests.sh
 
    integration_test:
      stage: test
      script: run_integration_tests.sh
      needs: ["unit_test"]

Use Anchors and Aliases

Avoid repetition by using YAML anchors (&) and aliases (*) to reuse configurations in your GitLab CI file. This is particularly useful for repeated script blocks or environment variables.


    .build_template: &build_definition
      script:
        - echo "Building project"
        - build_command
 
    build_job:
      <<: *build_definition
      stage: build

Use `extends`

Use the extends keyword to inherit from one or more templates. This allows for more flexibility compared to anchors and aliases, especially for overriding parts of the configuration.


    .template:
      script:
        - setup_command
 
    job1:
      extends: .template
      script:
        - additional_command # This will be executed after `setup_command`.

Use `variables` for Global and Local Scopes

Define global variables at the top of your .gitlab-ci.yml for use across all jobs, and override or extend them in individual jobs as needed.


    variables:
      GLOBAL_VAR: "global_value"
 
    job1:
      variables:
        LOCAL_VAR: "local_value"
      script:
        - echo $GLOBAL_VAR
        - echo $LOCAL_VAR

Store Secrets in Variables

Use CI/CD variables for sensitive data like tokens or passwords. Define them in the GitLab UI to keep them out of your .gitlab-ci.yml.


    deploy:
      stage: deploy
      script:
        - deploy_command --token $DEPLOY_TOKEN

Use comments in YAML

Use comments (#) liberally to explain complex configurations, the purpose of jobs, or why specific settings are used. This aids in maintainability.


# This job builds the project
build:
  stage: build
  script:
    - echo "Building"
    # Temporary workaround for XYZ issue
    - build_workaround_command