It’s pretty common to need Chef to be able to store secrets such as database passwords or API keys. The easiest thing to do is to store them in a data bag, but that’s open for the world to see. Encrypted data bags are nice but key management is a gigantic pain. A better solution is Chef Vault, which encrypts the data bag’s secret once for each client (a client being a Chef node or administrative user)
At the same time your organization likely has a need to keep secret data for applications, too. One could store these secrets in the same place as the Chef secrets but if you don’t like having Chef manage application configuration files then you’re out of luck. HashiCorp Vault is one solution here that we’ve used successfully.
With HashiCorp Vault, each client (or groups of clients) has a token that gives them access to certain secrets, dictated by a policy. So you can say that a certain token can only read user accounts and give that to your application or Chef. But how do you keep that token a secret?
I’ll also add that the management of HashiCorp Vault is nicer than that of Chef-Vault. That is, making changes to the secrets is a bit nicer because there’s a well defined API that directly manipulates the secrets, rather than having to use the Chef-Vault layer of getting the private key for the encrypted data bag and making changes to JSON. Furthermore this lets us store some of our secrets in the same place that applications are looking, which can be beneficial.
In this example, I have a Chef recipe with a custom resource to create users in a proprietary application. I want to store the user information in HashiCorp vault because the management of the users will be easier for the operations team, and it will also allow other applications to access the same secrets. The basic premise here is that the data will go in HashiCorp Vault and the token to access the HashiCorp Vault will be stored in Chef’s Vault.
The first thing to do is set up your secrets in HashiCorp Vault. We’ll want to create a policy that only allows read access in to the part of the Vault that Chef will read from. Add this to
1 2 3
Create the policy:
Create a token that uses that policy. Note that the token must be renewable, as we’re going to have Chef renew it each time. Otherwise it’ll stop working after a month.
1 2 3 4 5 6 7 8
That value beginning with
ba85 is the token that Chef will use to talk to the Vault. With your root token you can add your first secret:
At this point we have a user in the HashiCorp Vault and a token that will let Chef read it. Test for yourself with
vault auth and
Now it’s time to get Chef to store and read that key. Store the token in some JSON such as
And create a secret that’s accessible to the servers and any people needed:
This creates a secret in a data bag called
myapp_credentials in an item called
vault_token. The secret itself is a piece of JSON with a key of
token and a value of the token itself. The secret is only accessible by
sean (me) and
server1.example.com. If you later want to add a new server or user to manage it, you need to run
Which will encrypt the data bag secret with something that only
server2.example.com can decrypt.
I won’t get into all the details of Chef Vault other than to refer you to this helpful article.
Now, let’s get Chef to read that secret within the recipe! Most things in these examples are static strings to make it easier to read. In a real recipe you’d likely move them to attributes.
chef-vault into your recipe. Begin by referencing it in
And in the recipe, include the recipe and use the
chef_vault_item helper to read the secret:
1 2 3 4 5
Now that we have the token to the HashiCorp Vault, we can access the secrets using the Vault Rubygem.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
The testing story is fairly straightforward. If you’re using the
chef_vault_item as opposed to directly through
ChefVault::Item, then it’ll automatically fall back to using unencrypted data bags which are easily mockable. Similarly, HashiCorp Vault can be mocked or pointed to a test instance.
This seems to give a good balance of security and convenience. We manage the Chef specific secrets in the Chef Vault, and use the HashiCorp vault for things that are more general. And the pattern is simple enough to be used in other places.