YAML tips and tricks for Concourse CI
Have you ever had to copy-paste big chunks of YAML code while writing your Concourse CI pipelines? We’ll cover the tips and tricks that we use at BeTomorrow to reduce some of the boilerplate.
Concourse CI is a continuous integration tool. You can learn more about it at https://concourse-ci.org/. Even if you don’t have any notions about Concourse CI, there are still some YAML tricks in this article that can come in handy in any context.
Remember that settings files are YAML too
If your settings file is written in this fashion:
And used like this in your pipeline file:
You can write it like this instead:
You can then access those settings in your pipeline like this:
This helps categorize settings and makes the pipeline more readable. backend.git and backend.artifact.s3 are YAML objects.
One of the drawbacks here is that sensitive information (the private key and the AWS key & secret) are in the settings file, which is not advisable. Indeed, all these settings can be retrieved by anyone who has access to the Concourse server. We’ll see how to aggregate different YAML objects in the chapter “Use merging”.
Use anchors to stay DRY
Anchors define YAML references that can be used elsewhere.
Anchors are defined with the & character and referenced with *. For example, we can change this:
Into this:
You will be able to re-use the slack_failure_notification anchor for any other job now.
Use merging
You can use the << key to merge YAML objects.
foo: &foo
a: 10
b: 20bar:
<<: *foo
# => { a: 10, b: 20 }
You can also merge an array of objects. Only the first definition for a given key will be used. It cannot be redefined by following objects in the array.
foo: &foo
a: 10
b: 20bar: &bar
b: 30
c: 40foobar:
<<: [ *foo, *bar ]
# => { a: 10, b: 20, c: 40 }
Following the previous example, we’ll set an anchor for the terraform AWS key and merge it into the params object:
This pipeline is exactly the same as before, just less dense.
Beware, you must use the "<<” key when merging a settings variable to avoid a cryptic error, e.g. "<<": ((some.setting)).
Using settings stored in a vault
In the first part, we pointed out that it is not advisable to store sensitive information in the settings.yml file. This information should be stored in a vault. Once stored in a vault, we can use merging to recreate the resources:
Write strings over multiple lines
If you want to break down a long string over multiple lines, there are two ways to do so.
You can use the | character to use the literal style, which uses each new line as a literal newline in the string and adds an extra newline character at the end.
foo: |
a
very
long
string
# => "a\nvery\nlong\nstring\n"Or you can use the > character to use the fold style, which removes single new lines but keeps a new line character at the end.
foo: >
a
very
long
string
# => "a very long string\n"Handle different environments
In our examples, we have a terraform-backend-dev job, which means we have probably duplicated the job with a -prod and maybe a -demo . All the tricks above help with duplicating these jobs as they can re-use some common parts.
However, it can sometimes be best to parametrize the pipeline into accepting different environments. For example, we could have two separate pipelines, terraform-dev and terraform-prod, instead of having two separate jobs.
We need an extra settings-{env}.yml file for each environment to do this.
We can create each pipeline with its specific settings by providing multiple configuration files:
$ fly -t example set-pipeline -n \
-p terraform-dev \
-c pipeline.yml \
-l settings.yml \
-l settings-dev.ymlBeware though, as with everything, this can provide a solution to a specific problem and should not be used to answer a different one. Duplicating a pipeline should only be done after thorough reflection. Furthermore, it might be worth considering if having jobs with a -{env} prefix is really justified.
Final words
Concourse CI is a great tool but it does have some drawbacks. We have used it extensively at BeTomorrow and code duplication is maybe the number one problem for us.
I will be glad if any of these tips help you and I would greatly appreciate if you could add some of your own or explain how you handle different environments.
Thanks for reading!
Additional links
Wow, those were some great tips, but can you give me an example of when to use Concourse CI? Of course:
What books do I recommend? I’m glad you asked 🤓:
Why are value objects so great? For many reasons:
Resources
Liked what you read? Please click the 👏 below so others can find it!
