Our applications run in various environments (local, on-demand, staging, production, etc.), and we are used to decoupling their configuration from the code itself. Some parts of the configurations are considered secrets - values we like to keep private as they usually allow consuming and accessing the information on different components of our system. Once we add more parts to our environment, we face a new challenge: how will we distribute, manage and rotate secrets to avoid unintended exposures?
In this series, we will discuss two techniques to consume secrets in our applications and suggest practical integrations with popular secrets managers such as HashiCorp Vault, AWS Secrets Manager, AWS SSM Parameter Store, and GCP Secret Manager.
A note about secret distribution and management: in this series of articles we will not discuss secrets management and distribution techniques. Examples might include referrals to some secrets managers, but you should read the relevant documentation of your chosen solution to ensure you are using the vendor’s best practices when applying it.
secret: something kept from the knowledge of others or shared only confidentially with a few
In an engineering context, we see secrets as values that should be kept confidential and used to authenticate with a different system. Values such as passwords, API keys, and certificates are usually considered secrets.
We expect secrets to be set during an integration (e.g., adding a connection to a database or consuming information from a 3rd party API) and rotated frequently to avoid unintended long exposure. These tasks are usually managed by the integrator, who is the administrator or the engineer responsible for the integration, but the secrets themselves are consumed by our application regularly (each startup / each request).
Applications have two options to consume secrets:
As each technique has its pros and cons, we will describe each and compare them.
Secrets managers have APIs and usually well-documented SDKs. As developers, we are used to consuming APIs, and it makes sense to initialize our application by pulling the latest secrets directly from the managers.
As an example, this snippet uses boto3 to retrieve a secret from AWS SSM:
ssm_client = boto.client('ssm')
db_password_param = ssm.get_parameter(Name='/Prod/Db/Password', WithDecryption=True)
db_password = db_password_param['Parameter']['Value']
Although the above snippet is quite simple, it lacks essential elements such as error handling and configurable secret names (Prod vs. Dev …).
Getting back to The Twelve-Factor App Config section, we can treat secrets as a config value and let the infrastructure handle their source. Whether by a configuration file or an environment, you can supply that config as a different part of your application.
In Kubernetes, there’s a basic object named Secret which can be referenced by a directory or environment variables.
The following snippet uses a Secret to configure a MySQL database’s root password and share it with our “application”:
- name: MYSQL_ROOT_PASSWORD
- containerPort: 3306
- name: alpine
command: [/bin/sh, -c]
- printenv | grep MYSQL_
- name: MYSQL_PASSWORD
Regardless of which option you choose, there are still a few important things to keep in mind:
Generally, I would suggest always passing the configuration to the application rather than pulling it from the code. It enables greater flexibility when running the application in different environments (production, development, and local) and allows the developers to focus on using the value rather than fetching it.
It also allows the infrastructure to grow independently - changing the secret location in the secrets manager, managing permissions, and even changing the secrets manager solution without requiring any applicative changes.
Have any questions or comments about this post? Maybe you have a similar project or an extension to this one that you'd like to showcase? Join the Velocity Discord server to ask away, or just stop by to talk K8s development with the community.