Migrating Secrets between Hashicorp Vault Instances

Hashicorp Vault is a popular secret management tool from Hashicorp that allows us to store, access, and manage our secrets securely. We recently decided to move our Vault instance to Kubernetes and thus we needed a way to migrate all our existing secrets to the new instance. To achieve this, I created a Python script that scrapes the old vault instance and then adds those secrets to the new Vault instance retaining the directory structure of the secrets.

Hashicorp Vault is much more than a simple secret store, it's powerful, flexible, and can perform a variety of operations depending on the usecase. It all starts with Secrets Engines; a secret engine is like a plugin that allows us to perform a specific type of operation. Vault comes with tons of Secret Engines each with its own specific purpose. For example:

• KV: A generic Key-Value store used to store arbitrary secrets within the configured physical storage for Vault
• Cloud: Secrets Engines such as AWS, Azure, Google Cloud, etc. can dynamically generate secrets based on the IAM policies and can integrate with IDPs
• Database: Generate database credentials dynamically based on the configured roles. Most of the commonly used DBs are supported

Vault enables the secrets engines at a designated path, and any incoming request with a matching route prefix is automatically directed to that secrets engine by the router. To the end user, the secrets engine presents itself as a virtual filesystem that supports CRUD operations.

Note: We will be using the hvac python client to interact with the Vault. You can use pip to install it on your system if it is already not there.

1. First, we will identify the list of secret engines which are enabled on a vault and add them to a list

import hvac

# get list of all the secret engines in a vault
def listVaultSecretEngines(vault):
    _temp = []
    backends = vault.sys.list_mounted_secrets_engines()['data'].keys()
    for key in backends:
        _temp.append(key.rstrip('/'))

    return _temp

2. Once we have the list of all the secret engines in a vault, we will iterate over this and get all the secrets inside them. We will use a dictionary to store these in a key-value pair format (secret_engine : secret_value). Now, one of the points to consider is that the secrets engine is similar to a virtual filesystem, therefore, the secrets can be inside a directory as well. To accommodate this, the function listAllVaultSecrets checks if the item is a secret (directories end with a slash), then recursively calls itself till it gets a secret and then adds the secret to a list

# This function fetches all the secrets in a particular secret_engine
def listAllVaultSecrets(vault, secret_engine, path="/"):
    list_response = listSecrets(vault, secret_engine, path)

    if list_response is None:
        return
    else:
        keys = list_response['data']['keys']
        for key in keys:
            if key[-1] == "/":
                listAllVaultSecrets(vault, secret_engine, path + key)
            else:
                addKeyToVaultDict(secret_engine, path + key)

# This function returns a dictionary that contains a list of secret_engines
# and their respective secrets
def getVaultSecrets(vault):
    secret_engines = []

    # List All Secret Engines in the Vault
    secret_engines = listVaultSecretEngines(vault)

    for x in range(len(secret_engines)):
        listAllVaultSecrets(vault, secret_engines[x], "/")

    return vault_secrets

3. Another challenge here is that python dictionaries do not allow duplicate keys. Therefore, the below function creates a list inside the dictionary to store multiple values under a key

vault_secrets = {}
# This function creates/updates a dictionary of lists to bypass duplicate 
# keys problem
def addKeyToVaultDict(key, value):
    if key not in vault_secrets:
        vault_secrets[key] = value
    elif isinstance(vault_secrets[key], list):
        vault_secrets[key].append(value)
    else:
        vault_secrets[key] = [vault_secrets[key], value]

4. Now, that we have a list of secret engines with their respective secrets. We will create a function to read the actual value of the secret

# This function returns the value of a secret from Vault
def readSecret(vault, secret_engine, secret):
    read_response = vault.secrets.kv.v2.read_secret_version(
        path=secret,
        mount_point=secret_engine,
        raise_on_deleted_version=True
    )
    secret_value = read_response['data']['data']
    return secret_value

5. Next step, we will create a function to write secrets in the vault. As the secrets are inside a secret engine, we will check if the respective secret engine exists, if not we will create one. However, if the secret engine exists, we will directly create the secret

# Create a secret engine in a vault
def createSecretEngine(vault, path):
    vault.sys.enable_secrets_engine(
        backend_type='kv',
        version=2,
        path=path + "/"
    )

# Create a secret in the Vault
def writeSecret(vault, secret_engine, secretPath, secret):
    try:
        # check if secret engine exists, if not create first
        secretEngines = listVaultSecretEngines(vault)

        if secret_engine not in secretEngines:
            createSecretEngine(vault, secret_engine)
            print("New Secret Engine created - ", secret_engine)

        vault.secrets.kv.create_or_update_secret(
            path=secretPath,
            mount_point=secret_engine,
            secret=secret
        )
    except Exception as e:
        print("Some error occurred in writing secret", e)

Note: In my case, we were only using ‘kv’ secret engines, that’s why I have hard-coded the backend_type as ‘kv’. If there are multiple types of secret engines in use, we can store the type and other properties of the secret engine while reading them and then use them to create the secret engines in the new vault.

6. At this stage, we have all the components of the script, we just need two more things. First, we need to create two vault clients -

old_vault_client = hvac.Client(
    url='https://oldvault.company.com',
    token='xxxxxxxx'
)

new_vault_client = hvac.Client(
    url='https://newvault.company.com',
    token='xxxxxxxxx'
)

Second, we need to create a function that calls the other functions we have created in a particular manner to tie all these things together

# Migrate secrets from one vault instance to another
def migrateVaultSecrets(old_vault, new_vault):
    try:
        secret_list = getVaultSecrets(old_vault)
        for key in secret_list:
            for secretpath in secret_list[key]:
                secret_value = readSecret(old_vault, key, secretpath)
                writeSecret(new_vault, key, secretpath, secret_value)
        print("Secrets migrated successfully")
    except Exception as e:
        print("Some Error occured in migrating secrets", e)

And Voila! That finishes our script that can migrate all the secrets from one instance of Hashicorp Vault to another. Hopefully, this is helpful for you. All of the code samples for this exercise are available on GitHub for reference.

References:

• https://hvac.readthedocs.io/en/stable/overview.html
• https://developer.hashicorp.com/vault/docs