Management of environment files using CircleCI contexts

Everyone knows that pushing sensitive data to git repositories early or late may lead to security issues. Usually to avoid such situations all the files which may contain any sensitive information, developers add to .gitignore files which tells git that it should skip those files from adding to the repository and all the more pushing them to the remote one. There are many types of files which usually don’t add to repositories. Depending on used technologies they can b .env files which contain environment variables, various configuration secret files, certificates, private keys etc.

Actually if you are managing just a single monolithic app most likely you will not encounter any problem during maintenance of it. The management of environment files becomes quite complex when you have a microservice architecture with lots of services and have deployed them to multiple environments such as testing, staging, production. In this article I have described a mechanism which allows to facilitate management of .env files by using CircleCI contexts and a little bit scripting.

What are CircleCI contexts ?

The official website of CircleCI gives the following definition of contexts:

Contexts provide a mechanism for securing and sharing environment variables across projects. The environment variables are defined as name/value pairs and are injected at runtime.

So to be brief, contexts allow us to define environment variables through the CircleCI dashboard and use them during job runs. It’s possible to create multiple contexts with different names and include various environment variables into it.

it must be said that CircleCI doesn’t group variables by contexts during job runs instead overrides earlier contexts with the same variable names. And this is one of the problems which will be resolved during implementation of the current mechanism.

Predefining contexts

To be simple let’s assume there is an api service which requires a .env file to be in the root directory of it. Also let’s assume there are testing and staging environments where the mentioned application is running.

So there are two environments which means it’s required to have two .env files defined. By navigating to the contexts section of the organization settings in CircleCI dashboard, you need to create two different contexts for each .env file with appropriate names.

Let’s name them this way:

Later by adding new variables into each of these contexts, they will automatically appear into the appropriate environment file.

Creating a mapping file

Let’s create a folder named scripts in the .circleci directory of the project. In the new created scripts folder create a mappings.txt file with the following content:

            .env=API_USERS_
.env.test=API_USERS_TEST_

Left side of each line in the file above is an environment file path and on the right side is defined a prefix which later in CircleCI contexts will be added to variable names.

Adding variables to contexts

As we already have created the mappings file it’s time to add environment variables to the contexts. Names of variables should comply with the following structure:

{service_prefix}_{variable_name}

where service_prefix is a prefix defined in the mappings.txt for the current environment and variable_name is an actual environment variable name which will be included in .env file.

After adding variables to the contexts, they should look like this:

Creating create/copy shell scripts

All the variables defined in CircleCI contexts are stored as environment variables in the machines where jobs are running.

Purpose of creating shell scripts is to generate .env files in those machines and move them to external machines(servers) if needed.

Existence of the environment file for testing environment (.env.test) can be necessary in the current executer machine, for running automation tests. The environment file intended for staging is necessary for copying it from the executor machine to staging server.

Let’s create a script called generate_env_files.sh in .circleci/scripts directory. Here is the code example which should contain the script:

mappings=()
while IFS= read -r line; do
 mappings+=("$line")
done < mappings.txt

for value in "${mappings[@]}" ; do
 prefix=${value#*=};
 env_path=${value%=*};
 rm -f "$env_path" && for l in $(printenv | grep ^"$prefix"); do echo ${l#$prefix} >> "$env_path"; done
done

The script above does the following:

So after running the script above .env and .env.test files with environment variables will be created in the root directory of the project in the executer machine. Now tests can be run as .env.test file already exists in the root directory of the app.

The next step is moving environment files to the external servers. Let’s create a file called copy_env_files.sh in .circleci/scripts directory and place there the following code snippet:

mappings=()
while IFS= read -r line; do
 mappings+=("$line")
done < mappings.txt

for value in "${mappings[@]}" ; do
 env_path=${value%=*};
 scp $env_path xx.xx.xx.xx:/var/www/api-users/$env_path
done

The script above does the following:

NOTE: xx.xx.xx.xx should be replaced with the real ip address of the machine

CircleCI configuration file

Here is a sample CircleCI configuration which runs tests and deploy changes by automatically generating environment files.

version: 2
jobs:
 deploy:
   steps:
     - checkout
     - add_ssh_keys:
         fingerprints:
           - "c1:37:bf:df:a7:a1:04:7c:be:d1:46:83:9a:bf:76:73"
     - run:
         name: Add to known_hosts
         command: "xx.xx.xx.xx >> ~/.ssh/known_hosts"
     - run:
         name: Create .env files
         command: ./.circleci/scripts/generate-env-files.sh
     - run:
         name: Copy .env files
         command: ./.circleci/scripts/copy-env-files.sh
     - run:
         name: Deploy api-users
         command: |
           ssh xx.xx.xx.xx \
           "cd /var/www/api-users && \
           git pull && \
           npm run start"
 test:
   steps:
     - checkout
     - run:
         name: Create .env files
         command: ./.circleci/scripts/generate-env-files.sh
     - run:
         name: Run tests
         command: npm run test
workflows:
 version: 2
 test_and_deploy:
   jobs:
     - test:
         context:
           - testing-api-users
     - deploy:
         context:
           - staging-api-users
         requires:
           - test
         filters:
           branches:
             only: master

There are two jobs defined in the yaml file above, test and deploy. Let’s go through the steps of each job and give a brief explanation of each one.

test

deploy

Also, please take a look at 40 and 43 lines where contexts for each environment are defined.

Conclusion

Everything described in this article is a working example of the real application. We applied this method in one of our projects with microservices architecture.

Thank you for reading !