Our applications run in various environments (local, on-demand, staging, production, etc.), and we are used to decoupling their configuration from the code itself. Some parts of the configurations are considered secrets - values we like to keep private as they usually allow consuming and accessing the information on different components of our system. Once we add more parts to our environment, we face a new challenge: how will we distribute, manage and rotate secrets to avoid unintended exposures?
In this series, we will discuss two techniques to consume secrets in our applications and suggest practical integrations with popular secrets managers such as HashiCorp Vault, AWS Secrets Manager, AWS SSM Parameter Store, and GCP Secret Manager.
A note about secret distribution and management: in this series of articles we will not discuss secrets management and distribution techniques. Examples might include referrals to some secrets managers, but you should read the relevant documentation of your chosen solution to ensure you are using the vendor’s best practices when applying it.
What is a Secret anyway?
secret: something kept from the knowledge of others or shared only confidentially with a few
In an engineering context, we see secrets as values that should be kept confidential and used to authenticate with a different system. Values such as passwords, API keys, and certificates are usually considered secrets.
We expect secrets to be set during an integration (e.g., adding a connection to a database or consuming information from a 3rd party API) and rotated frequently to avoid unintended long exposure. These tasks are usually managed by the integrator, who is the administrator or the engineer responsible for the integration, but the secrets themselves are consumed by our application regularly (each startup / each request).
Applications have two options to consume secrets:
- Directly from the secrets manager (via code in the app)
- As a configuration provided by the infrastructure (via configuration file/environment variables)
As each technique has its pros and cons, we will describe each and compare them.
Applications pull secrets
Secrets managers have APIs and usually well-documented SDKs. As developers, we are used to consuming APIs, and it makes sense to initialize our application by pulling the latest secrets directly from the managers.
As an example, this snippet uses boto3 to retrieve a secret from AWS SSM:
ssm_client = boto.client('ssm')
db_password_param = ssm.get_parameter(Name='/Prod/Db/Password', WithDecryption=True)
db_password = db_password_param['Parameter']['Value']
Although the above snippet is quite simple, it lacks essential elements such as error handling and configurable secret names (Prod vs. Dev …).
When should I use this option?
- You don’t control who runs your application, and you want to ensure only the application can see the secret.
- You have an advanced use case that requires you to fetch secrets from different dynamic paths.
- You are ok with writing implementation-specific code, as your choice of a secrets manager will not be changing anytime soon.
When should I NOT use this option?
- You don’t want additional dependency in the code - this introduces a new code chunk which increases the complexity, requires proper error handling, and increases the bundle size.
- Permissions - you can’t ensure the app itself will have the right permissions to access the secrets manager (e.g., cloud credentials when using AWS/GCP secrets managers).
- Local development issues - running the application on your local machine will require the proper credentials to access the secrets manager.
Infrastructure provides secrets
Getting back to The Twelve-Factor App Config section, we can treat secrets as a config value and let the infrastructure handle their source. Whether by a configuration file or an environment, you can supply that config as a different part of your application.
In Kubernetes, there’s a basic object named Secret which can be referenced by a directory or environment variables.
The following snippet uses a Secret to configure a MySQL database’s root password and share it with our “application”:
- name: MYSQL_ROOT_PASSWORD
- containerPort: 3306
- name: alpine
command: [/bin/sh, -c]
- printenv | grep MYSQL_
- name: MYSQL_PASSWORD
When should I use this option?
- You don’t want to couple your code with the secrets manager’s API or SDK.
- You run your application in different environments - passing different values is easy through an environment variable.
- You manage the infrastructure differently from your application and don’t want to depend on its changes, such as the secrets manager solution or the secrets page in that secrets manager.
When should I NOT use this option?
- In Kubernetes, Secrets are stored in plain text in etcd (you should read the Information Security for Secrets.) If you don’t define good RBAC around them, other users of your cluster might consume them without your knowledge.
- Secrets distribution - in the previous option every user was using the same secrets manager and could access its values. This option requires you to pass the value around (especially when running locally.)
Considerations for either approach
Regardless of which option you choose, there are still a few important things to keep in mind:
- A secret is a concept. In our application, they are assigned to variables and might appear in log messages or transmitted in plain transport. It’s your job to make sure they are kept safe.
- We are not looking to hide the secret in our application’s memory space. If someone gains access to the process' memory or gains permission to run code inside our application, they will probably be visible to this secret. (e.g., Heartbleed Bug)
When to use each option
Generally, I would suggest always passing the configuration to the application rather than pulling it from the code. It enables greater flexibility when running the application in different environments (production, development, and local) and allows the developers to focus on using the value rather than fetching it.
It also allows the infrastructure to grow independently - changing the secret location in the secrets manager, managing permissions, and even changing the secrets manager solution without requiring any applicative changes.
In the next part of this series, we will explore different secrets managers and how to use them in Kubernetes:
- Injecting secrets from HashiCorp Vault (Basic and via the Agent Injector)
- Using secrets-init to load cloud secrets manager’s values
- Syncing Kubernetes Secrets with the Secrets Store CSI Driver