The Problem

It’s pretty common to need Chef to be able to store secrets such as database passwords or API keys. The easiest thing to do is to store them in a data bag, but that’s open for the world to see. Encrypted data bags are nice but key management is a gigantic pain. A better solution is Chef Vault, which encrypts the data bag’s secret once for each client (a client being a Chef node or administrative user)

At the same time your organization likely has a need to keep secret data for applications, too. One could store these secrets in the same place as the Chef secrets but if you don’t like having Chef manage application configuration files then you’re out of luck. HashiCorp Vault is one solution here that we’ve used successfully.

With HashiCorp Vault, each client (or groups of clients) has a token that gives them access to certain secrets, dictated by a policy. So you can say that a certain token can only read user accounts and give that to your application or Chef. But how do you keep that token a secret?

I’ll also add that the management of HashiCorp Vault is nicer than that of Chef-Vault. That is, making changes to the secrets is a bit nicer because there’s a well defined API that directly manipulates the secrets, rather than having to use the Chef-Vault layer of getting the private key for the encrypted data bag and making changes to JSON. Furthermore this lets us store some of our secrets in the same place that applications are looking, which can be beneficial.

In this example, I have a Chef recipe with a custom resource to create users in a proprietary application. I want to store the user information in HashiCorp vault because the management of the users will be easier for the operations team, and it will also allow other applications to access the same secrets. The basic premise here is that the data will go in HashiCorp Vault and the token to access the HashiCorp Vault will be stored in Chef’s Vault.

The Code

The first thing to do is set up your secrets in HashiCorp Vault. We’ll want to create a policy that only allows read access in to the part of the Vault that Chef will read from. Add this to myapp.hcl

1
2
3

path "secret/myapp/*" {
  policy = "read"
}

Create the policy:

1
2

[root@vault ~]# vault policy-write myapp myapp.hcl
Policy 'myapp' written.

Create a token that uses that policy. Note that the token must be renewable, as we’re going to have Chef renew it each time. Otherwise it’ll stop working after a month.

1
2
3
4
5
6
7
8

[root@vault ~]# vault token-create -policy=myapp -renewable=true
Key               Value
---               -----
token             ba85411e-ab76-0c0f-c0b8-e26ce294ae0d
token_accessor
token_duration    720h0m0s
token_renewable   true
token_policies    [myapp]

That value beginning with ba85 is the token that Chef will use to talk to the Vault. With your root token you can add your first secret:

1

vault write secret/myapp/testuser password=abc123 path=/tmp

At this point we have a user in the HashiCorp Vault and a token that will let Chef read it. Test for yourself with vault auth and vault read!

Now it’s time to get Chef to store and read that key. Store the token in some JSON such as secret.json.

1

{ "token": "ba85411e-ab76-0c0f-c0b8-e26ce294ae0d"}

And create a secret that’s accessible to the servers and any people needed:

1

knife vault create myapp_credentials vault_token -A sean,server1.example.com -M client -J ./secret.json

This creates a secret in a data bag called myapp_credentials in an item called vault_token. The secret itself is a piece of JSON with a key of token and a value of the token itself. The secret is only accessible by sean (me) and server1.example.com. If you later want to add a new server or user to manage it, you need to run

1

knife vault update myapp_credentials vault_token -A server2.example.com

Which will encrypt the data bag secret with something that only server2.example.com can decrypt.

I won’t get into all the details of Chef Vault other than to refer you to this helpful article.

Now, let’s get Chef to read that secret within the recipe! Most things in these examples are static strings to make it easier to read. In a real recipe you’d likely move them to attributes.

First, get chef-vault into your recipe. Begin by referencing it in metadata.rb

1

depends 'chef-vault'

And in the recipe, include the recipe and use the chef_vault_item helper to read the secret:

1
2
3
4
5

include_recipe 'chef-vault'

# The key to unlock the HashiCorp vault is in Chef
bag = chef_vault_item('myapp', 'vault_token')
vault_token = bag['token']

Now that we have the token to the HashiCorp Vault, we can access the secrets using the Vault Rubygem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

vault = Vault::Client.new(address: 'https://vault.example.com:8200')
vault.token = vault_token
# Renew the lease while we're here otherwise it'll eventually expire and be useless.
vault.auth_token.renew_self 3600 * 24 * 30

# Take a listing of the secrets in the path we chose earlier
vault.logical.list('/secret/myapp/').each do |name|
  # Extract each individual secret into a hash
  user_data = vault.logical.read("/secret/myapp/#{name}")
  # Apply the custom resource using parts of the secret
  myapp_user name.downcase do
    unix_user name.downcase
    password user_data.data[:password] # This is the password in the vault
    path user_data.data[:path] # This is the path from the vault
  end
end

The testing story is fairly straightforward. If you’re using the chef_vault_item as opposed to directly through ChefVault::Item, then it’ll automatically fall back to using unencrypted data bags which are easily mockable. Similarly, HashiCorp Vault can be mocked or pointed to a test instance.

This seems to give a good balance of security and convenience. We manage the Chef specific secrets in the Chef Vault, and use the HashiCorp vault for things that are more general. And the pattern is simple enough to be used in other places.