Container Image Signing with AWS Signer and Amazon EKS

Introduction to Container Image Signing

In this section, the host introduces the topic of container image signing and welcomes the guests from AWS who will be discussing container image signing in the context of Amazon EKS with Signer.

Importance of Container Image Signing

• Container image signing is compared to sealing a package when shopping on Amazon, ensuring authenticity and confidence in the contents.

• It provides security throughout the software supply lifecycle, from development to deployment on an Amazon EKS cluster.

Guests Introduction

• Jeremy Cowan, from Amazon EKS, will demonstrate how to use Open Policy Agent (OPA) with Ratify for verifying image signatures.

• Milan Gold Kern, a senior developer at AWS Signer, will discuss container image signing using Notary and AWS Signer.

Context and Relevance of Container Image Signing

• Container images can be sourced from various places, including public registries like ECR Public or private registries.

• The goal is to ensure authenticity and integrity by verifying that images are unmodified and from trusted sources.

• Authenticity is achieved through signing, providing confidence in the source or author of the image.

• Revocation control allows invalidation of signed images or identities used for signing.

Components and Groundwork

This section discusses the components involved in container image signing and lays the groundwork for further discussion.

Understanding Container Image Signing

• Container image signing ensures authenticity and integrity throughout different stages such as importing base images or deploying them to Kubernetes environments.

• Authenticity verifies that images are from expected sources or vendors/authors.

• Revocation control allows invalidating previously signed images or identities used for signing.

Advantages of Flow Demo

This section highlights the advantages of using the demonstrated flow for container image signing.

Advantages of the Demonstrated Flow

• Authenticity provides confidence in the signer and ensures that contents match expectations.

• Revocation control allows invalidation of signatures and artifacts, providing flexibility and security.

The transcript does not provide further sections or timestamps related to the content.

Working with Notation CLI and AWS CLI

In this section, the speaker discusses the tools required for signing images and authenticating against AWS services. They mention working with the Notation CLI, which is part of the Notary project, to sign images. Additionally, they highlight the need to use the AWS CLI to authenticate against AWS services like Signer and ECR (Elastic Container Registry).

Local Machine Setup

• The speaker mentions working with the Notation CLI as part of the Notary project.

• The Notation CLI helps in signing images.

• The AWS CLI is needed for authentication against AWS services like Signer and ECR.

Image Signing Process

• A base image is assumed to be hosted in a registry (such as ECR).

• However, this image is currently unsigned.

• The goal is to have a running container image in an EKS cluster that has been verified and signed.

Components in Kubernetes Cluster

• The speaker mentions having a Kubernetes cluster running on Amazon Elastic Kubernetes Service (EKS).

• Two components mentioned are Ratify and OPA (Open Policy Agent), which help with verifying and signing container images.

Demo: Signing Images using Signer and ECR

In this section, a demo is presented on how to sign images using Signer and ECR. The process starts from the developer's local machine.

Starting the Image Signing Process

• To start the process of signing an image, notation needs to be installed on the local machine.

• Notation implements specifications from the Notary project for container image signing.

• AWS CLI commands will be used during the demo.

Creating a Signing Profile

• The first step is to go to AWS Signer and create a signing profile.

• A signing profile represents the signing identity and simplifies the management of keys and certificates.

• AWS Signer handles PKI management, including key rotation.

Getting a Signed Image in ECR

• Once a signing profile is created, notation can be used to sign an image.

• The notation sign command communicates with ECR to get the image digest and generates a signature using the signer plugin.

• The generated signature is then pushed back to ECR, where it persists alongside the signed image.

Conclusion

In this section, final remarks are made before jumping into the demo.

Final Remarks

• The speaker highlights that signatures exist and persist alongside images in ECR.

• Collaboration between internal teams at AWS and open-source projects like Notary B2 was required for this feature.

• The overall flow has been simplified for demonstration purposes.

Overview of Signature Verification and Revocation

In this section, the speaker discusses the process of verifying signatures and what happens when signatures are revoked. They also mention the ability to perform these operations locally.

Signature Verification and Revocation

• Signatures can be verified to ensure their authenticity.

• If a signature is revoked, it means that it is no longer considered valid.

• There is a way to perform signature verification and revocation operations locally.

Creating a Signing Profile in AWS Console

The speaker demonstrates how to create a signing profile using the AWS console.

Creating a Signing Profile

• Use the AWS console to create a signing profile.

• Select the signing platform, such as containers.

• Use default settings for creating the profile.

No Dealing with Keys or Certificates in AWS Signer

The speaker explains that when creating a signing profile in AWS Signer, there is no need to handle keys, certificates, or PKI (Public Key Infrastructure).

No Dealing with Keys or Certificates

• When creating a signing profile in AWS Signer, there is no need to handle keys, certificates, or PKI.

• All key management and certificate handling are managed by AWS Signer.

Copying the Signing Profile

The speaker copies the created signing profile for future use.

Copying the Signing Profile

• After creating a signing profile, copy its details for later use.

Checking Notation Plugin Installation

The speaker checks if the notation plugin is installed correctly.

Checking Notation Plugin Installation

• Use the notation plugin list command to verify if the notation plugin is installed.

• Confirm that the AWS signer plugin for notation is installed.

Validity Period of Signatures

The speaker answers a question about why signatures have a validity period.

Validity Period of Signatures

• Signatures can have a validity period to limit their usage.

• This is useful in cases where you want to prevent the use of stale artifacts.

• The default validity period is around 11 years, but it can be customized for specific use cases.

Signing an Image with Notation CLI

The speaker demonstrates how to sign an image using the Notation CLI.

Signing an Image with Notation CLI

• Use the notation sign command to sign an image.

• Specify the image URL or digest URL for signing.

• Use the AWS signer plugin for actual signing operations.

Describing and Inspecting Signatures

The speaker describes and inspects signatures associated with an image.

Describing and Inspecting Signatures

• Use the AWS CLI command get signing profile to describe a signing profile created through the console.

• Use the notation inspect command to check if there are any signatures associated with an image before signing it.

Signing an Image with AWS CLI and Notation Plugin

The speaker signs an image using both AWS CLI and the Notation plugin.

Signing an Image with AWS CLI and Notation Plugin

• Use AWS CLI's notation sign command along with the AWS signer plugin for actual signing operations.

• Specify the ID of the key (signing profile) to be used for signing.

Integration of AWS CLI and Notation CLI

The speaker discusses the integration of AWS CLI and Notation CLI for container image signing workflows.

Integration of AWS CLI and Notation CLI

• AWS CLI is primarily used for setup operations, such as creating and managing profiles and jobs.

• Notation CLI is mainly used for signing and verification workflows.

• The core signing and verification workflows are available through notation, while other AWS operations are performed using the AWS CLI.

Signature Generation in Console and Signing Job Resource

The speaker demonstrates how signatures are generated in the console and introduces the concept of a signing job resource in AWS Signer.

Signature Generation in Console and Signing Job Resource

• Signatures are generated alongside the original image stored in the registry.

• In AWS Signer, there is a resource called a "signing job" associated with signature generation.

Signing Profile and Signing Job

This section discusses the resources involved in signing, including the signing profile and signing job.

Resources for Signing

• A signing profile is equivalent to a key used for signing operations.

• A signing job is used to track the progress of a signature.

Kicking off Signing Job from CLI

This section explains how to initiate a signing job using the CLI.

Initiating Signing Job

• The signing job was initiated from the CLI, specifically from the notation CLI.

• The notation CLI uses a plugin to communicate with AWS Signer and complete the signing process.

Viewing Signature and Original Image

This section demonstrates viewing the signature and original image after completing the signing process.

Viewing Signature and Image

• The signature and original image can be seen after completing the signing process.

• Notation pulls all associated signatures for an image.

• Attributes such as the signing profile, profile version, and metadata are displayed.

Inspecting Signatures

This section explains how to inspect signatures associated with an image.

Inspecting Signatures

• The location inspect command lists all signatures associated with an image.

• Each signature includes information such as the signing profile, profile version, and metadata.

Trusting Container Images in Notation

This section discusses trusting container images in Notation and configuring trust policies.

Trusting Container Images

• By default, Notation does not trust any images; explicit configuration is required.

• Trust policies need to be set up in order to specify which identities or profiles are trusted for specific images.

• It is recommended to have a set of trusted sources for pulling container images, both for internal and public images.

Importance of Trusted Sources for Container Images

This section emphasizes the importance of trusting sources for container images.

Importance of Trusted Sources

• It is crucial to have trusted sources for container images to ensure compatibility and security.

• In Kubernetes, it is not advisable to trust any random container image.

• Private registries are often used for pulling internal images, which can be further secured by signing them with an enterprise key.

• Public images should ideally be signed by the entity that created them.

Configuring Trust Policies in Notation

This section explains how to configure trust policies in Notation.

Configuring Trust Policies

• Notation has trust policies specifically designed for container use cases.

• A sample policy can be used to verify an image based on a registry namespace prefix.

• Different verification levels, such as strict, can be set.

• The trustor refers to the AWS Signer's trust or certificate.

• Specific signing profiles can be specified as trusted identities.

Verifying Images with Notation

This section demonstrates how to verify container images using Notation.

Verifying Images

• The notation verify command is used to verify an image reference.

• Notation automatically uses the appropriate plugin for signing verification.

• Multiple plugins can be used, and notation provides commands to simplify their usage.

Importing Trust Policies in Notation

This section explains how to import trust policies in Notation.

Importing Trust Policies

• Trust policies can be imported into Notation using the notation policy import command.

• The policy file specifies which identities or profiles are trusted for specific registry scopes.

• Registry scopes can be wildcarded or specified as an array for multiple scopes.

Trusting Specific Signing Profiles

This section discusses trusting specific signing profiles in Notation.

Trusting Signing Profiles

• By importing trust policies, specific signing profiles can be trusted for signing images.

• The imported policy specifies the trusted identities or profiles.

• Notation now trusts the specified signing profile to sign images with the corresponding registry scope.

The transcript is already in English.

Configuring Notation with Trust Policy and Image Verification

In this section, the speaker explains how to configure Notation with a trust policy and perform image verification using AWS ECR (Elastic Container Registry). The process involves requesting signatures associated with an image from ECR, verifying those signatures, and performing a revocation check using AWS Signer. The speaker also discusses customization options for air-gapped environments.

Configuring Notation with Trust Policy

• To begin, configure Notation with a trust policy.

• Use the notation verify image reference command to request all signatures associated with an image from ECR.

• Notation will verify the signatures and perform a revocation check using AWS Signer.

• Customization options are available in the notation documentation, such as using a permissive policy that only logs revocation checks.

Handling Air-Gapped Environments

• For air-gapped environments where revocation checks cannot be performed externally, exceptions can be set up for specific images.

• Different policy levels can be applied to different sets of images to support incremental adoption of signing.

Verifying Images in Kubernetes Cluster Deployment

This section focuses on verifying images in the software supply chain when deploying them into an Amazon EKS (Elastic Kubernetes Service) cluster. The speaker discusses integrating dynamic admission controllers like Opa Gatekeeper or Caverno to enforce policies during object instantiation within the cluster.

Automating Verification in Kubernetes Cluster Deployment Workflow

• When deploying applications or pods into an Amazon EKS cluster, it is important to automate the verification step.

• Users can use their local CLI or CI/CD pipelines to deploy applications into the cluster.

• The second step is to run a dynamic admission controller like Opa Gatekeeper or Caverno in the cluster.

• These controllers verify that objects instantiated within the cluster adhere to configured policies.

Using Dynamic Admission Controllers for Verification

• Opa Gatekeeper, for example, can call external services like Ratify to verify image signatures using Notary APIs and AWS Signer.

• The result of the verification is returned to the admission controller, which determines whether to admit or reject the object (e.g., deployment).

• Dynamic admission controllers ensure that images are validated and meet policy requirements before being deployed in the Kubernetes cluster.

The transcript provided does not cover all sections of the video.

[t=0:33:44s] Signer and Image Signing

In this section, the speaker discusses the use of AWS Signer to sign images in one region and deploy them to multiple regions without any additional steps. The speaker also mentions that image verification can be done in any AWS region.

Signing and Verification with AWS Signer

• With AWS Signer, you can sign an image in one region and push it to ECR.

• When deploying to multiple regions, there is no need for any additional steps.

• You can verify the signed image in any AWS region.

[t=0:34:26s] Demonstrating Signed Image Deployment

The speaker demonstrates how to deploy a signed image into a cluster using AWS Signer.

Steps for Deploying a Signed Image

• The speaker shows an already signed image using notation LS.

• The image has a signature associated with it.

• Additional information about the image signature can be obtained using notation inspect.

• The deployment of the signed image is done into the cluster where Opa Gatekeeper and Ratify are running.

• The container starts running successfully, indicating that the signed image deployment is successful.

[t=0:36:20s] Deployment with Unsigned Image

The speaker attempts to deploy an unsigned image and encounters an issue during the process.

Attempting Deployment with Unsigned Image

• An attempt is made to create a deployment called "signed" using an unsigned image from ECR public registry.

• There seems to be an issue during this deployment process.

• Live debugging is performed by Jeremy to identify and resolve the issue.

• It appears that there might be a problem with either Notation or Opa's connection to Ratify service.

• After restarting Ratify service, the deployment should result in an error message indicating refusal due to an unsigned image.

[t=0:39:21s] Importance of Good Mechanisms in Image Validation

The speaker emphasizes the importance of relying on good mechanisms rather than just good intentions when it comes to image validation and verification.

Relying on Good Mechanisms for Image Validation

• While developers may have good intentions, it is crucial to have good mechanisms in place for image validation.

• Tools like Opa Gatekeeper and Ratify are used to validate and verify images automatically.

• Operations teams set up these mechanisms so that developers don't need to be aware of the validation process.

• By using these mechanisms, clusters can ensure that images are signed, secure, and verified without relying solely on developer intentions.

Timestamps are provided where available.

Deployment Failure and Pod Rejection

The speaker discusses a failed deployment for a pod and explains that the pods were rejected by Opa (Open Policy Agent). They demonstrate how to check for rejected pods.

Failed Deployment and Pod Rejection

• The speaker mentions that the create operation failed for a pod within the deployment.

• They explain that when checking for pods, none are found with the specified label because they were rejected by Opa.

• The deployment was successfully instantiated in the cluster, but none of the pods associated with it were created.

Creating a Naked Pod without Deployment

The speaker demonstrates creating a naked pod (without a deployment) using the kubectl run command. They show an error message from Opa Gatekeeper indicating that the pod was rejected due to non-compliance with policies.

Creating a Naked Pod

• The speaker uses the kubectl run command to create a naked pod without a deployment.

• An error message from Opa Gatekeeper is displayed, stating that the pod was rejected due to non-compliance with policies.

• The rejection reason mentioned is "signature verification failed for all signatures" because the image used was not signed.

Customizing Trust Policies for Dev Environments

The speaker addresses customization of trust policies based on different environments. They discuss allowing unsigned pods in development environments while enforcing stricter policies in staging and production environments.

Customizing Trust Policies

• It is suggested that in development environments, less strict policies can be applied to allow unsigned pods.

• A comment from Charles in the chat emphasizes developers' need to deploy unsigned pods during development.

• Jeremy explains that trust policy settings can be configured and exceptions can be specified.

• A trust policy example for Ratify is shown, which includes registry scopes, signature verification level (e.g., strict, permissive, audit), and trusted identities or signing profiles.

Revoking Image Signatures

The speaker discusses the process of revoking image signatures when vulnerabilities are discovered. They demonstrate how to revoke an image signature using the Notary CLI and browser interface.

Revoking Image Signatures

• The speaker explains a scenario where a vulnerability is found in an image and its signature needs to be revoked.

• They mention running notation inspect to obtain the job ID associated with the image's signature.

• Using the browser interface, they navigate to the signing jobs section and select the corresponding job to revoke the signature.

• The revocation process is demonstrated.

Impact on Running Pods after Signature Revocation

The speaker addresses a question regarding running pods after revoking an image's signature. They explain that Kubernetes does not automatically shut down running pods when a signature is revoked.

Impact on Running Pods

• A question is raised about whether all running pods need to be torn down after revoking an image's signature.

• Jeremy confirms that Kubernetes does not automatically shut down running pods even if their signatures have been revoked.

• He mentions that Opa acts on API calls before they are emitted and verifies or mutates requests based on policies set up.

• Depending on application requirements, there may be a trade-off between availability and security posture when dealing with insecure images.

Integrating Notation Library with Kubernetes

The speaker discusses integrating Notation Library (a Golang library) with Kubernetes for additional functionality beyond using Notary CLI.

Integrating Notation Library

• The speaker mentions that besides using the Notary CLI, there is also a Notation Library available in Golang.

• This library provides the same set of functionality as the CLI and can be integrated with Kubernetes for added capabilities.

Signature Scaling and Availability

In this section, the discussion revolves around the scalability of signatures and its impact on availability. The alternative approach of running in permissive mode to avoid availability issues is also mentioned.

Signature Scalability and Availability

• Signatures may not be able to scale, leading to potential availability issues.

• Running in permissive mode can be an alternative approach to maintain availability even if revocation impacts occur.

Deleting a Pod with Revoked Signature

This section focuses on deleting a pod that was deployed with a container image whose signature was revoked. The process of deletion is demonstrated.

Deleting the Pod

• The pod that was deployed with a revoked signature is deleted.

• Confirmation of the deletion is expected shortly.

• After deletion, attention will shift towards examining the deployment.

Impact of Revoked Signature on Pod Deletion

This section discusses the impact of a revoked signature on pod deletion. The delay in the delete operation is acknowledged, followed by an explanation of how deployments are managed by replica sets.

Delayed Delete Operation

• There is some surprise at the time taken for the delete operation but it's attributed to typical live demo scenarios.

• Despite deleting the pod, there is no need for recreation as deployments are designed to heal automatically.

• One desired pod becomes unavailable due to revocation, resulting in a failed creation attempt.

OPA Gatekeeper and AWS Signer API Interaction

This section highlights how OPA Gatekeeper and AWS Signer API interact when dealing with revoked signatures. It emphasizes Jeremy's dual role as an image publisher and application operator.

Revoked Signature Verification Process

• As expected, when encountering a revoked signature, OPA Gatekeeper and AWS Signer API collaborate.

• The AWS Signer API confirms that the signature is no longer valid, leading to the revocation detection.

• Jeremy's dual role as an image publisher and application operator is emphasized.

Signature Verification Levels and Exceptions

This section discusses different signature verification levels and the possibility of using exceptions. It highlights the flexibility in configuring policies based on specific requirements.

Signature Verification Levels and Exceptions

• Different signature verification levels can be configured, allowing for flexibility in policy implementation.

• Exceptions can be used to handle scenarios where certain components within a deployment are not signed.

• The ability to configure multiple levels of verification is beneficial when dealing with deployments consisting of multiple container images.

Offloading Image Verification to CI/CD Pipeline

This section explores the option of offloading image verification to a CI/CD pipeline instead of relying solely on EKS and admission controllers.

Offloading Verification to CI/CD Pipeline

• Offloading image verification to a CI/CD pipeline is feasible based on workflow requirements.

• Implementing checks earlier in the software supply chain lifecycle allows for repeated verifications at each step.

• The same workflow can be applied throughout the CI/CD process, ensuring comprehensive verification before deployment.

Control Plane Deployment with Kubernetes

This section emphasizes the importance of good mechanisms and automation in utilizing Kubernetes as a control plane for deployments. It highlights various implementation approaches while acknowledging that the demonstrated flow aligns with common practices.

Utilizing Kubernetes as Control Plane

• Good mechanisms and automation enable Kubernetes to serve as a control plane for deployments.

• Various implementation approaches exist, but the demonstrated flow aligns with common practices.

• Multiple ways are available to implement container image signing using AWS Signer and Amazon EKS.

Flexibility of AWS Signer for Revocation

This section focuses on the flexibility provided by AWS Signer for revocation. It highlights the ability to revoke both signing profiles and individual signatures, allowing targeted revocations.

Flexibility in Revocation

• AWS Signer offers flexibility in revocation by allowing the revocation of signing profiles and individual signatures.

• Targeted revocations can be performed to minimize the blast radius.

• The key takeaway is understanding the importance of signing, incorporating it into application flows, and leveraging AWS Signer and notation for easy adoption.

Getting Involved with Open Source Project

In this final section, viewers are encouraged to get involved with the open-source project related to signer. Ways to stay updated with the latest updates are also mentioned.

Involvement with Open Source Project

• Viewers are invited to participate in the open-source project related to signer.

• Staying up-to-date with the latest updates can be done by following along with a blog that provides hands-on steps using notation CLI and AWS Signer.

Contributing to the Notary Project

In this section, the speaker discusses how anyone can contribute to the Notary Project, an open-source project. They encourage individuals to share their experiences with adoption and provide feedback on new feature requests.

Getting Involved

• The Notary Project is an open-source project that welcomes contributions from everyone.

• Individuals are encouraged to visit the notaryproject.dev website for documentation and information.

• The GitHub repository "notary project" is where contributors can find more details and get involved.

Thanking Guests and Encouraging Feedback

The speaker expresses gratitude towards Mill and Jeremy for their demo and overview of the Notary Project. They also emphasize the importance of customer feedback and invite viewers to share their thoughts through various channels.

Acknowledging Guests

• The speaker thanks Mill and Jeremy for their informative demo and overview of the Notary Project.

Importance of Feedback

• AWS values customer feedback and actively listens to users' needs.

• Viewers are encouraged to provide comments, suggestions, or questions through YouTube, LinkedIn, GitHub, or Twitter.

• Feedback plays a crucial role in shaping future developments based on customer requirements.