GitLab CI/CD and components: a story of reusability

Building and rebuilding CI/CD pipelines…

#

Continuous Integration and Continuous Deployment (CI/CD) has become an essential concept for creating containerised images, deploying infrastructure code, and verifying vulnerabilities in your tooling.

If you have ever maintained different types of pipelines of varying complexity, you know how much code from these automated pipelines can be duplicated across different code repositories, whether they are application or infrastructure repositories.

Even worse, how can organisations ensure CI/CD pipelines follow company-wide standards, including syntax validation steps, vulnerability scanning, or code coverage verification? Not so straightforward!

Yet since version 17, GitLab introduced a new concept available across all editions: CI/CD components. This paradigm finally brings reusability, versioning, and native job sharing between GitLab projects.

I have been using this concept, finding it extremely crucial, especially for establishing a standard and ensuring the implementation of a common baseline accessible to everyone.

This article explores this concept in depth, demonstrating how to create a suite of components for OpenTofu, the Terraform fork.

A component? What for?

#

A GitLab component is a portion of YAML code stored in a Git repository that can be called within a CI/CD pipeline from the .gitlab-ci.yml file.

Here’s an example:

File format.yml

spec:
  inputs:
    stage:
      default: lint
[...]
---
tofu-fmt:
  stage: $[[ inputs.stage ]]
  script: tofu fmt -check -recursive -write=false -diff
[...]

The details will be covered later, particularly around structure, but this component allows calling a tofu-fmt job directly within a CI/CD pipeline.

The advantage? Avoiding code duplication whilst versioning and making ready-to-use components available to other teams in your organisation or the community.

Obviously, not all pipeline steps are meant to be used as components. The goal is to identify those you want to use most, particularly based on a theme or a specific tool, to build a standardised library.

For example, components might organised in groupes under OpenTofu, Terraform, Docker, Podman, or as a set of components for infrastructure as code, security checks, or containerisation.

I’m quite keen on breaking things down as much as possible to create components that are scalable yet maintainable. Never forget they will need updating once their implementation is complete.

One structure to rule them all!

#

As mentioned above, a component lives within a code repository that must meet certain requirements regarding its directory structure.

Here’s an example with the opentofu Git repository:

├── templates/
│   └── format.yml
├── LICENSE.md
├── README.md
└── .gitlab-ci.yml

This contains:

• A README.md to explain the different components detailed, potentially including configurable variables, how to call them with examples, etc.;

• A LICENSE.md file to assign a licence to this component, particularly for public components. MIT or Apache 2.0 licences are often preferred;

• The .gitlab-ci.yml file can execute tests to validate component usage, run a test suite, but most importantly create a release based on a tag to publish a new version;

• A templates folder containing components as .yml files. Subdirectories can be created, but these must contain a template.yml file to be properly exploited as a component.

Finally, the repository name and its structure shouldn’t be neglected, especially for managing permissions and avoiding confusion!

Publishing to share better

#

Another important step is publishing your components. Although not mandatory, this step is crucial to ensure they are accessible within your organization or available for public release.

Following DevOps principles, adding a release job within the .gitlab-ci.yml file of the repository in question enables publishing everything within GitLab’s CI/CD Catalogue.

This catalogue, as its name suggests, lists all components available in your GitLab instance based on user permissions.

Be aware though: additional steps must be completed with the Owner role to authorise your Git repository to publish to the global catalogue.

To publish within GitLab’s catalogue, here’s an example of an automated publication that can be done with this snippet of code:

.gitlab-ci.yml

stages:
  - release

create-release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:v0.24.0
  script: echo "Creating release $CI_COMMIT_TAG"
  release:
    tag_name: $CI_COMMIT_TAG
    description: "Release $CI_COMMIT_TAG of components in $CI_PROJECT_PATH"
  rules:
    - if: $CI_PIPELINE_SOURCE != "push"
      when: never
    - if: $CI_COMMIT_TAG

This will generate a release based on a tag defined within the repository.

Versioning is a key step. GitLab recommends following semantic versioning. Nevertheless, calling a component remains possible in different ways:

• Through the SHA of a commit (e.g. 1f4d90e308bf4cf9abe17060fdd59bbd71f720bf);
• A tag (e.g. 1.1.0)
• A branch name (e.g. main)
• And finally, the ~latest keyword for retrieving the most recent version of a component.

The SHA is recommended, although not easily readable. The best option remains the tag but requires proper versioning while respecting immutability.

Putting it to use

#

Using your component relies on dedicated syntax within a YAML file, typically .gitlab-ci.yml:

include:
  - component: ${CI_SERVER_FQDN}/cicd-catalog/opentofu/[email protected]
    inputs:
      image_tag: "v1.10.6"

Through the include block, in list form, components can be enumerated with their versions @1.1.0. As you can see, the templates folder does not need to be specified in the component path, only the name of the YAML file or folder containing it.

The inputs block references the variables defined in the component’s spec. In this example, image_tag sets the image version for OpenTofu.

A few good practices

#

To conclude this largely theoretical section, here are some principles for making your components robust and adaptable:

• Improve your component through iteration: avoid creating 40 inputs to make your component fully configurable. Focus on its value using a few key variables. You can always improve it over time.

• Manage version compatibility: your component will be evolving over time, so maintaining an exhaustive changelog from version to version is necessary, covering future deprecations or feature additions;

• Validate your component: add test jobs in the .gitlab-ci.yml of the project grouping your components to ensure the template works in different configurations, particularly the script section when updating an image;

• Favour official GitLab variables: avoid hard-coding URLs or paths. Use predefined variables within GitLab CI/CD instead. Here are some examples:

  • $CI_SERVER_FQDN: The domain name of the GitLab instance;
  • $CI_API_V4_URL: The root URL of the GitLab API;
  • $CI_PROJECT_DIR: The directory where your project gets cloned during CI/CD execution.

• Create exhaustive documentation: detail the commands, expected variables, and optional behaviours in the README.md. Don’t hesitate to add different use cases to facilitate integration of your component with advanced parameterisation.

The moment of truth

#

It’s time to create our OpenTofu components!

To keep things simple, here’s a suggestion to create one component per GitLab job:

• format: checks code indentation with the fmt command;
• security: executes security analysis with checkov;
• lint: analyses best practices with tflint;
• plan: runs a tofu plan, storing the plan for later use, without forgetting a caching mechanism for the .terraform folder and .terraform.lock.hcl file;
• apply: performs a tofu apply with the plan as a parameter.

Not bad for a start, right?

You can retrieve the code directly through my GitLab repository, giving you an idea of the work required:

As you can see, the templates folder is quite populated with as many YAML files as there are components described above.

As an example, let’s explore the plan component, a rather comprehensive one. The variables, through the spec block, allow customising the job according to needs:

spec:
  inputs:
    job_name:
      default: plan
      description: "Name of the job"
    stage:
      default: plan
      description: "Pipeline stage where the job will run"
    image:
      default: ghcr.io/opentofu/opentofu
      description: "Container image to use for OpenTofu"
    version:
      default: "1.10.6"
      description: "OpenTofu version to use"
    working_directory:
      default: "."
      description: "Directory containing OpenTofu files"
    plan_file:
      default: "tfplan"
      description: "Name of the plan file to generate"
[...]

Personally, I find it important to give users the ability to name the job through a job_name variable to avoid conflicts with other components or existing jobs, but above all to avoid using the extends keyword to rename it.

Remember not to forget the --- to separate the job code from the defined variables.

On the job side, the template remains fairly easy to read, so there’s no need to overcomplicate things… The aim is to allow users to generate a file containing the plan of resources to create, update, or delete following the tofu plan command, with optional parameters: backend_config, var_file, and var.

"$[[ inputs.job_name ]]":
  stage: $[[ inputs.stage ]]
  image:
    name: $[[ inputs.image ]]:$[[ inputs.version ]]
    entrypoint: [""]
  script:
    - cd $[[ inputs.working_directory ]]
    - |
      INIT_ARGS=""
      if [[ -n "$[[ inputs.backend_config ]]" ]]; then
        IFS=',' read -ra CONFIGS <<< "$[[ inputs.backend_config ]]"
        for config in "${CONFIGS[@]}"; do
          INIT_ARGS="$INIT_ARGS -backend-config=$config"
        done
      fi
[...]

Variables are identified and replaced using the $[[ inputs.version ]] syntax.

If you’d like to test these components, here’s an example of a .gitlab-ci.yml file that might inspire more than a few:

include:
  # Format
  - component: $CI_SERVER_FQDN/filador-public/cicd-library/opentofu/[email protected]
    inputs:
      stage: analyze
      working_directory: "./infra"
  # Security
  - component: $CI_SERVER_FQDN/filador-public/cicd-library/opentofu/[email protected]
    inputs:
      stage: analyze
      working_directory: "./infra"
  # Lint
  - component: $CI_SERVER_FQDN/filador-public/cicd-library/opentofu/[email protected]
    inputs:
      stage: analyze
      working_directory: "./infra"
  # Plan
  - component: $CI_SERVER_FQDN/filador-public/cicd-library/opentofu/[email protected]
    inputs:
      working_directory: "./infra"
  # Apply
  - component: $CI_SERVER_FQDN/filador-public/cicd-library/opentofu/[email protected]
    inputs:
      working_directory: "./infra"

stages:
  - analyze
  - plan
  - apply

plan:
  needs:
    - format
    - security
    - lint

apply:
  needs:
    - plan

The inputs block provides the ability to override default values. In my case, the OpenTofu code sits in an infra folder, and I’ve grouped the format, security, and lint steps into the analyze stage.

As you might have guessed, it’s like a kind of puzzle where you import the components that make sense for the CI/CD pipeline you’re building. The modular aspect is an essential strength of this approach.

One final personal preference: I prefer managing dependencies with the needs keyword outside the components to maintain readability and understand dependencies very easily when reading this file.

A world of components

#

Reinventing the wheel indefinitely might not be your best option!

Indeed, on GitLab.com, the community already provides a rich component catalogue. You’re free to use them, contribute, fork them, or adapt them to your needs.

The time saving is considerable, not to mention these get improved and tested by a large community with a wide range of options.

Of course, the example above is not as rich as the OpenTofu component maintained by GitLab, but you get the idea.

So, useful?

#

Once you start using this concept, it’s hard doing it the old way!

Whether those provided by the community or your own components, there’s something for everyone.

This approach clearly falls within the realm of Platform Engineering, providing a catalogue of standardised, ready-to-use components that meet the company’s quality and security requirements.

Up to you now!