Best Practices for Using HashiCorp Terraform with HashiCorp Vault

Introduction

In this section, the speaker introduces herself and the purpose of the webinar. She also provides an overview of what will be covered in the presentation.

• Becca Petra, a software engineer on the Vault team, is presenting.

• The purpose of the webinar is to demonstrate best practices for using Terraform and Vault together.

• The presentation will cover how to use Terraform to spin up a real Vault cluster backed by Console and running in AWS.

• The sections that will be covered are: Getting Started Using Terraform, Recommended Architecture, CD in Vault with Threats, and Consuming Those Secrets.

Webinar Logistics

In this section, the speaker provides information about logistics related to the webinar.

• The webinar is being recorded and will be available post-processing within a day or two.

• There is a questions box where attendees can submit their questions during the presentation.

• The last 15 minutes of the webinar will be dedicated to Q&A.

Speaker Introduction

In this section, Becca Petra introduces herself and her role on the Vault team.

• Becca Petra is a software engineer on the Vault team who has been with HashiCorp since February of last year.

• She maintains the Terraform Vault provider which allows her to think about how Terraform and Vault work together.

• Her main focus as a vault engineer is security.

Best Practices for Using Terraform and Vault Together

In this section, Becca Petra covers best practices for using Terraform and Vault together.

• Terraform is a cross-platform tool for writing infrastructure code used to create resources quickly.

• Vault is a cross-platform Swiss Army knife for protecting against data leaks.

• Console is used by Vault as a database for storing secrets in an encrypted format and is also a service mesh tool.

• AWS is short for Amazon Web Services, which will be used to spin up the cluster today.

• The presentation will cover how to use Terraform to spin up a real Vault cluster backed by Console and running in AWS in only five minutes.

• The sections that will be covered are: Getting Started Using Terraform, Recommended Architecture, CD in Vault with Threats, and Consuming Those Secrets.

Getting Started Using Terraform

In this section, Becca Petra covers how to get started using Terraform.

• A pre-created Terraform config is available for attendees to use.

• Becca Petra will begin spinning up the recommended architecture and let it run over the course of five minutes while explaining it.

• Once the architecture is set up, she will continue setting it up on the command line.

Vocabulary Drive-by

In this section, Becca Petra provides definitions for key terms related to Terraform and Vault.

• Terraform is a cross-platform tool for writing infrastructure code used to create resources quickly.

• Vault is a cross-platform Swiss Army knife for protecting against data leaks.

• Console is used by Vault as a database for storing secrets in an encrypted format and is also a service mesh tool.

• AWS is short for Amazon Web Services, which will be used to spin up the cluster today.

Setting up Terraform with AWS S3 Remote State Backend

In this section, the speaker explains how to set up Terraform with AWS S3 remote state backend.

Cloning Pre-made Code from Github

• Clone pre-made code from a public repository on Github.

• Change directory into the best practices architecture section.

Using AWS S3 Remote State Backend

• Use AWS S3 remote state backend to protect sensitive data in the terraform state file.

• Add it to terraform config before running it.

• Use steep file encryption for terraform when there are many secrets in your config or state.

• Rotate keys and secrets periodically as recommended by AWS.

Running Terraform Apply

• Run terraform apply after initializing terraform and pulling down all necessary plugins for terraform.

• Choose US East 1 as the region for spinning up resources in AWS environment.

• Confirm that you want to spin up resources by typing "yes".

Recommended Architecture

In this section, the speaker explains what Terraform is doing while spinning up a single data center for Vault.

Spinning Up Data Center for Vault

• Spin up a single data center for Vault cluster consisting of seven total instances.

• Three of them are vault servers and three are console servers protected by Bastion host in front of them.

• They aren't directly exposed to the internet.

Vault Secrets Management

In this section, the speaker discusses how secrets are managed in transit and at rest using Vault. They recommend watching a production hardening guide after setting up instances to complete further steps such as trimming off core dumps, enabling auditing, and disabling the shell command history.

Managing Secrets with Terraform and Vault

• Vault is the only place where secrets and sensitive data should be managed.

• Members of your team using Terraform to spin up additional resources should authenticate with Vault using an easy-off method like Github.

• People on your team using Terraform should use it with their individual vault token and with no secrets placed directly in the config.

Backing Up Encrypted Vault Data

• Console's snapshotting mechanism can be used to save and restore encrypted vault data that has been placed in it.

• This allows you to save and restore vault secrets while continuing to keep them in a safe and encrypted format.

AWS Console Overview

In this section, the speaker takes a look inside the AWS console after finishing spinning up instances for best practices architecture.

Best Practices Architecture

• The best practices architecture includes seven instances - a bastion host, three console nodes, and three vault nodes.

• The instance type used is t2 small but m5 larges are recommended for real production environments.

• Instances are not all in the same availability zone for redundancy purposes.

Terraform Configuration

• The speaker returns to the terminal to check on Terraform configuration.

Setting up Vault

In this section, the speaker sets up a Vault instance and unseals it with keys.

Initializing Vault

• By default, when you spin up a Vault instance, it comes up in a sealed state to prevent unauthorized access.

• To unseal the instance, you need to use specific keys that are distributed among three people.

• The speaker demonstrates how to initialize and unseal the Vault instance using live commands.

• The process involves running vault operator and vault operator seal with three different keys.

Seeding Vault with Secrets

In this section, the speaker explains how to seed secrets into the newly created Vault instance.

Adding Secrets to Vault

• Before adding secrets, the speaker gives an overview of what they are doing and why they are doing it.

• They then demonstrate how to add secrets by creating a new secret engine mount and writing secrets to it using live commands.

• The process involves running vault secrets enable, vault write, and vault read commands.

Dynamic Secrets

• The speaker introduces dynamic secrets as a way of generating temporary credentials for applications or services.

• They demonstrate how to create a dynamic secret backend for AWS IAM credentials using live commands.

• The process involves running vault secrets enable -path=aws -plugin-name=aws-secret-backend-plugin, configuring AWS credentials, and generating dynamic credentials using vault read aws/creds/my-role.

Conclusion

In this section, the speaker concludes their presentation on setting up and seeding a Vault instance.

Key Takeaways

• The speaker emphasizes the importance of securing sensitive data with tools like HashiCorp's Vault.

• They highlight some key features of Vault such as dynamic secrets and secret engines.

• They encourage viewers to explore Vault further and experiment with it in their own environments.

Final Thoughts

• The speaker thanks the audience for watching and encourages them to reach out if they have any questions or feedback.

• They provide some resources for learning more about Vault, including documentation and community forums.

Setting up Vault Tokens

In this section, the speaker explains how to set up a vault token and export it as an environment variable. The token is used to access secrets from Vault in Terraform.

Exporting Vault Token

• To access secrets from Vault in Terraform, a vault token must be exported as an environment variable.

• The token should not be put into the Terraform config file.

• When running terraform apply, it will pull AWS keys or database passwords from Vault and put them into the state file.

• These secrets are short-lived and expire in five minutes.

Benefits of Using Vault

In this section, the speaker explains why using Vault is beneficial for DevOps teams.

Dynamic Credentials

• AWS keys and other secrets can be dynamically created on-the-fly every time Terraform is used.

• This means that zero secrets live in the Terraform config file, making it more secure.

• Centralizing all of this in Vault creates an audit log of every secret being accessed by whom and when.

Protection Against Insider Attacks

• Short-lived credentials protect against insider attacks since they are unique to each person and only valid for a limited amount of time.

• By removing someone's access from Github organization, their access to all systems is automatically removed.

Protection Against Accidental Leaks

• Even if a DevOps member doesn't encrypt their Terraform state file, any leaked secrets would have already expired.

Configuring Vault with Command-Line Interface

In this section, the speaker explains how to configure Vault using command-line interface (CLI).

Enabling Github Auth Method

• Enable Github auth method so that DevOps team members can get their vault tokens using their Github identities.

• Members must be part of the organization to access secrets.

Using UI

• Vault has a user interface that is available in the open-source version.

• However, today's configuration will be done using CLI since the quick setup cluster is designed to be accessed from a bastion host.

Setting up Vault for DevOps Team Members

In this section, the speaker explains how to set up Vault for DevOps team members. The process involves enabling Github authentication and creating an AWS key and secret for use with Terraform.

Creating a Policy

• Create a policy that allows anyone with it applied to read creds from the path of AWS creds devrel.

• This is where Vault will dynamically create a new AWS key and secret that expires in only five minutes.

Enabling Github Authentication

• Enable the Github auth method.

• Allow people at the Hashi for organization to use it.

• Set TTL to 24 hours.

• Apply dev policy to members of the vault team in Github who authenticate using Github.

Enabling AWS Secrets Engine

• Enable the AWS secrets engine.

• Use access key and secret variables when talking to AWS.

• Set credentials to expire after five minutes.

• Grant privileges inside of AWS when issuing credentials.

Consuming Secrets in Terraform Config Files

In this section, the speaker explains how DevOps team members can consume secrets in their Terraform config files using Vault.

Retrieving Secrets from Vault

• Every secret in the Terraform config file can be retrieved from Vault.

• Vault can create unique short-lived secrets for Active Directory, all cloud services, databases, Google Cloud Nomad, SSH gene, and more being released every day.

Using Terraform with Vault

• If you already use Vault but not Terraform, this is how to get started while maintaining your security story.

• If you use Terraform but not Vault, consider using it as a way to strongly protect secrets in your Terraform config and state.

• Vault excels at protecting secrets, and together with Terraform, they're like peanut butter and jelly.

Pulling Secrets from Vault

• This method showcases Terraform's ability to pull secrets from nearly anywhere.

• If configured earlier, Terraform can also pull secrets straight out of Vault.

Using AWS Remote State with Encrypted S3 Bucket

The speaker recommends using the AWS remote state instead of long-lived secrets in the Terraform configure state. This can be done by pointing it at an encrypted S3 bucket.

AWS Remote State

• Use the AWS remote state instead of long-lived secrets in the Terraform configure state.

• Point it at an encrypted S3 bucket for added security.

Workflow of Engineer Using Vault

The speaker explains how an engineer uses Vault to securely provide short-lived dynamic secrets to their DevOps team.

Engineer Workflow

• Engineer gets a Vault token using their Github identity.

• Token is exported so it doesn't live in the Terraform config file.

• When running terraform apply, Terraform pulls short-lived secrets from Vault using config files.

• Secrets are rendered harmless in five minutes, making this process secure and simple.

Summary and References

The speaker summarizes what was covered in the webinar and provides references for further reading.

Summary

• Three topics were covered - spinning up a best practices vault cluster, securing it, and using it to provide short-lived dynamic secrets to your DevOps team.

• No long-lived secrets should live in your Terraform configure state files. Secrets are where Vault shines, and it's available in our open-source version.

References

• Vault Reference Architecture - talks through what kind of Amazon EC2 instances you might want to use if you use this approach and the pros and cons of various things

• Best Practices Vault Cluster on AWS - pre-made terraform templates that can be used

• Production Hardening Guide - for locking down the vault instances

• Example of how to pull AWS keys from Vault - put up by a colleague named Roger Berlin

Using Azure Secrets Engine with Terraform

The speaker explains how the Azure secrets engine can be used with Terraform.

Azure Secrets Engine

• The second circular config file shown in the webinar can pull from anywhere in Vault that supports read access.

• If the Azure secrets engine has a read for its credentials, then out of the box, you can use that same config to pull from it.

Terraform and Internet Access

The speaker answers a question about whether internet access is needed for Terraform to pull stuff.

Internet Access

• In this example, internet access is needed because Terraform is using Amazon Web Services which is hosted on the cloud.

• However, if you weren't using AWS, you could certainly use Terraform without internet access.

Best Practices for Deploying a Vault in Production

The speaker discusses the best practices for deploying a vault in a production environment.

Using Best Practices to Deploy a Vault

• Use M5 large instance sizes.

• Make edits to the instance sizes as needed.

Understanding the Auto Unseal Feature

The speaker explains how the auto unseal feature works and how to configure it.

Configuring Auto Unseal

• Go into premade terraform config files and change them to use an AWS key.

• Spin up the vault and initialize it with initial unsealed keys and vault token.

• Update the config to use auto unseal method.

• Do a migration to use auto and seal keys.

Application Specific Terraform Skirts

The speaker answers whether application-specific terraform skirts should be kept in a separate repo or in the application-specific repo.

Keeping Terraform Skirts

• Cannot make recommendations because not on tariff one team.

• Will research and respond later.

Expiration of Master Key Passwords for Databases

The speaker discusses whether master key passwords for databases expire after five minutes, and what happens if resources do not allow you to change passwords.

Expiration of Master Key Passwords

• Database secrets engine lets you configure how long you want it to last.

• If state file is leaked, scope of damages is limited.

• Use dynamic secrets so that they expire quickly.

Running Terraform Without Access to Cloud Provider

The speaker answers whether it is possible to run Terraform without access to a cloud provider.

Running Terraform

• Yes, it is possible to run vaults on anything.

• Use Terraform to spin up resources locally on your machine.

Using Multi-Cloud with Vault

The speaker discusses the possibility of using multi-cloud for Patrick's with three vault servers, one on each cloud provider. They discuss the challenges of networking them together and recommend having a bastion in front of them.

Using Role-Based Access to Switch AWS Accounts

• The AWS auth engine allows for role-based access where an EC2 instance is launched into a role and uses its credentials to authenticate to Vault.

• Another option is assuming a role through Vault, which provides flexibility.

• Secrets can be passed as environmental variables, such as the Vault token.

• These are captured inside the Terraform state file and should be set up to be secure.

Executing Vault Login Command

The speaker explains how to execute the vault login command and SSH into the bastion.

SSHing into Bastion and Vault Instance

• In this example, you would need to SSH into the bastion first before accessing the vault instance since it was newly spun up.

• In practice, developers on the AWS VPN can set environment variables for the vault address and hook in they obtain every day from their local computer without needing to SSH into anything.

Console Installation with Vault Cluster

The speaker answers questions about console installation along with managing a vault cluster.

Installing Console Alongside Vault Cluster

• In this example, three console nodes were spun up separately that three vault servers are talking to.

• Console is available separately from Vault and is open-source in this example.

Managing AWS Resources with Vault

The speaker discusses how to manage AWS resources with Vault and whether the user who creates or manages them can be seen in CloudTrail.

Managing AWS Resources with Vault

• AWS keys are generated from Vault through an API call that is captured by CloudTrail.

• The vault user appears to create the resources, but the person behind the vault can be found in the audit logs.

Terraform Version and Vault Enterprise

The speaker answers general questions about Terraform version and whether the repo uses install Vault OSS or Vol Enterprise.

Terraform Version and Vault Installation

• The repository uses install Vault OSS, not Vol Enterprise.

• The current version of Terraform is 0.11.8.