Writing scripts for continous integration/delivery (CI/CD) pipelines is often done as quick as possible in order to get the pipeline green (i.e. working as expected). Often though, the good engineering practices we put into our applications are neglected for the CI/CD pipelines. When projects and code bases evolve over time this can easily become an error-prone maintenance nightmare that no one dares to touch. This blog post will show some ideas you can use to significantly improve the maintainability and robustness of any CI/CD pipeline setup.

Do One Thing and Do It Well

CI/CD scripts tend to grow over time as additional functionality and automation is being added. Instead of interleaving the different pipeline parts or putting them into a single big file, we can instead put a single pipeline job to it’s own file following the single-responsibility principle. As an example, we could come up with the following scripts:

• ci/build.sh: A script to build the source code to a deployable artifact.
• ci/deploy.sh: A script to do the actual deployment of the built artifact to a specific environment.
• ci/export-translations.sh: Reads all translation keys from the source code and creates them in the translation management tool if they don’t yet exist.
• ci/rollout-infrastructure.sh: Creates and updates any necessary infrastructure using an infrastructure-as-code approach, e.g. by leveraging Terraform.

From the file names it is now already pretty clear what parts of the workflow have been automated. This helps new team members just as well as it will help your future self returning to an old project. Each of the scripts should be no longer than 200 lines of code, so that they are easy to understand, follow along and maintain.

Handle All Edge Cases!

Handling edge cases is very important, as it will tremendously increase the robustness of your scripts. There is nothing worse than a pipeline doing awkward things as assumptions have changed, but the script did not notice. Depending on the scripting language the first thing to do is to set the script execution to be as strict as possible.

For a BASH script this would mean exiting when using undefined variables (-u) and exiting the script itself in case any command fails (-e):

set -eu

While this already gives us a great deal of more safety, we now need to verify all assumptions we might have for the script. That especially includes the existence of required environment variables, files, folders or connections. If the preconditions are not met, the script should output a descriptive error message of what is missing, why the information is required and how to resolve it. Let’s see what this could look like:

# first check whether the environment variable has been properly set
if [ -z "${CF_API}" ]; then
    echo 'Please provide the Cloud Foundry API endpoint as environment variable $CF_API'
    exit 1
fi
# then use it in your script
cf api "${CF_API}"

Adding robust is not limited only to checking variables, but you should always check exit codes or verify the output of the commands you called, so that you can be sure that if a command succeeds in your script, it really has succeeded.

Spicing up With Colors and Emojis

Scripts often produce lots of output and sometimes it can get really tricky to find the information you need to find within those hundreds of lines of log messages. You can improve the output of your scripts by adding colors and emojis to relevant parts of log messages. A good way to do so, is to structure your script into different phases and adding a colorful message for each phase. For example:

RED='\033[0;31m'
GREEN='\033[0;32m'
BLUE='\033[0;34m'
NC='\033[0m'
function info() { echo "${BLUE}${@}${NC}"; }
function error() { echo "${RED}${@}${NC}"; }
function success() { echo "${GREEN}${@}${NC}"; }

info "🔐 Logging in to Cloud Foundry"
cf login ...
info "🚀 Pushing app"
cf push ...
info "💬 Sending Slack notification for successful deployment"
curl ...

Defining helper functions for meaningful messages makes sure you actually use them. It’s also a good idea to not only structure the log messages using this technique, but also to emphasize the most critical message in case of an error. Speaking about errors, writing great error messages is really worth the time. Consider the following messages:

1. Couldn’t read the input file
2. Couldn’t parse the input file at ./inputs/run-01/sample-data06.txt
3. Couldn’t parse the input file at ./inputs/run-01/sample-data06.txt. Did you forget to init your Git submodules?

I’m pretty sure that error message (3) is the most helpful. Why? It contains the fault (couldn’t parse) along with its cause (problem with the specific file) and offers a potential solution that is likely the underlying root cause.

Sticking to One Language

I’m sure you’ve heard that you should pick the right tool for the right job. And as we have different programming and scripting languages, some may be better suited for one case whereas others make it easier to deal with something else. When it comes to your CI/CD scripts do yourself a favor and stick to one, at most two languages. Adding more languages has several drawbacks:

• The scripts start to work and behave differently, producing different output
• Changing the pipeline requires more skills and knowledge
• The WTF metric for your code review will be insanely high, as no one understands what happens
• You need all that tooling in your local setup as well as in containers

Please don’t make anyone setup Ruby, Python, Node, Go, Bash and Perl to run all of the scripts.

Consistency is Key

When there are multiple projects or repositories that need to be built and deployed, it becomes increasingly important to align the scripts for all these repositories. Consistency in how CI/CD is done helps share the knowledge across the team and ensures that all CI/CD pipelines operate at a similar level of engineering quality. Consistency can mean a lot of things, but to get you started here are some ideas:

• A directory structure for pipelines and infrastructure scripts (e.g. /ci and /infra in every repository)
• Naming conventions so that it is easy to tell which artifact belongs to which project/repository
• Sticking to infrastructure as code, where no pipeline or infrastructure changes are done manually

Print Versions, All the Time

When hunting bugs, misbehavior’s and discrepancies between your local setup and what the CI/CD pipeline does, it is inevitable to know what versions of which tool run where. That’s why it is super useful to just simply print all the version info you can get of all the tools you use right at the beginning of your CI/CD script. Especially as these versions might change over time being able to lookup which version has been used for which pipeline run is a must have. It’s as simple as you can imagine, and you can’t work without doing this any longer:

git --version
java -version
node --version
terraform --version

Your Turn

Now it’s your turn. Go improve your CI/CD scripts and use some proper engineering principles to ensure your pipelines are of high quality. Do you have more suggestions on how to properly write CI/CD scripts? Share your thoughts in the comments below!