Implement automatic key rotation on Azure DevOps service connections

… and stop using long secret lifetimes

Introduction

More and more organizations are seeing the benefits of defining cloud infrastructure as code and are leveraging automated pipelines for their cloud deployments.

Azure DevOps is a popular choice since it comes with a variety or useful features that encompass the whole agile workflow. From maintaining your sprints and backlog to storing your code inside a Git repository and to test and deploy your code to your favorite cloud platform of choice.

I’m not only seeing Microsoft Azure customers embrace these capabilities but AWS and Google Cloud customers as well.

For the purpose of this blog and the solution provided, we’ll be focusing on Microsoft Azure Resource Manager (ARM) for now.

Within Azure DevOps you’ll be using an ARM “service connections” to connect and authenticate to Azure subscriptions and resources. These connections generally make use of an “App registration” created inside Azure Active Directory. Next, this “application” is granted to the appropriate resources in Azure to make sure Azure DevOps can make the appropriate changes from its pipelines. And lastly the application ID and secret are configured in Azure DevOps as a service connection which will be used pipelines to authenticate.

When generating a secret for an application in Azure Active Directory through the Azure portal you have several choices for its lifetime:

And here’s where a problem lies. This secret will be used as a password.
So even if you choose a lifetime of either 1 or 2 years, this still is a major security risk. Unfortunately way to often I’m seeing customers even choosing to let this secret never expire at all!

The problem

You might be thinking : “Well I’ve stored the application’s secret inside the Azure DevOps service connection on behalf of my DevOps teams, and they won’t be able to read it from its configuration.”

And while that is technically true, also consider that every time you run a pipeline (that’s using that service connection) those credentials are stored on the agent processing your pipeline jobs. And although you won’t be able to just simply put that password on screen by adding a PowerShell script or CLI tasks to the pipeline, it’s still actually quite easy to retrieve that password if you know a certain trick. Your pipeline output will mask those specific passwords by displaying ***, but by encoding secure strings with Base64 (or convert them to hexadecimal values) they will show up in clear text and you can decode them afterwards.

example that shows Azure DevOps obfuscating your secrets but no more once you encode them to Base64.

If you want to try it out yourself please import the following pipeline:

Pascal Naber also wrote an excellent blog about this. I’d suggest you check this out as well. So you simply cannot rely on keeping these keys a secret!

And once members of those DevOps teams are getting their hands on that password you’ll never know what will happen with it next. They might want to use it as a handy backdoor to access Azure resources with a higher level of permissions than their own account has. Or what if somebody leaves the company? Even though their own account might be disabled they would still be able to access your Azure infrastructure indefinitely through that app registration. Because let’s be honest; who’s actually recycling app registrations regularly in a secure way?

You might be thinking those secrets are safely stored inside Azure DevOps and inside your Azure Key Vaults. But see who’s laughing last once your devs use a simple trick to pull that secret.

Key rotation to the rescue

Just as with your employee accounts you can shorten the window of a password being abused by rotating it regularly. But unfortunately I’m not seeing many companies actually doing this for their “service accounts”. It might seem like a hassle to change the password in places where they’re used. And more often than not I’m seeing customers struggle to even know where these app registrations were actually used for in the first place! So they just keep the passwords in place, afraid they might break things when they’re changed.

Rotating secrets like these will only work well if you make the secret rotation part of the whole DevOps pipeline. And to do so you want to make sure the teams create a sense of ownership so they will be the ones responsible for keeping these connections secure.

When a pipeline is created to daily rotate these secrets, your security risk will be much much lower since nobody actually knows the exact key. And even if they do, it will expire very soon. Most importantly, if somebody leaves the organization they won’t be able to leak the password to a possible adversary. Albeit intentionally or nonintentional.

No matter how hard they’ll press you, you won’t be able to tell them where the m̶o̶n̶e̶y̶ password is

Azure DevOps implementation

The suggested solution below consists of the following components/steps:

1. The main component is a YAML pipeline which consists of a schedule (in cron-format), several customer-specific parameters and a job which refers to a YAML “template” containing the actual steps to perform.
   This “template” is used within a YAML pipeline to keep the amount of lines and repeatable code as low as possible. Especially once you apply this solution to a DevOps project with a large amount of service connections this might become less convenient.
2. Within the pipeline template a PowerShell script is triggered which will perform the actual key rotation on the app registration and will update the Azure DevOps service connection. More details will follow below.
3. Several different Microsoft APIs are used to retrieve the app registration details, create a new secret and update the Azure DevOps service connection.

Only these three files are needed to accomplish our current goal

Prerequisites

Before you can start implementing the automatic key rotation you’ll need to make sure some permissions and authorizations are properly configured:

• The app registration used within the Azure DevOps service connection needs to have some specific authorizations: It needs to become owner of itself to be able to generate new secrets. And a specific Graph permissions (Application.ReadWrite.OwnedBy) should be granted otherwise the pipeline will not be able to read details from applications it owns. And lastly, an admin consent should be performed.
• Within Azure DevOps the built-in “Build Service” user should become a member of the “Endpoint Administrators” group. Otherwise it won’t be able to make alterations to the service connections.

Without proper authorizations you shall not pass!

Configure app registration permissions

For setting up the app registration settings you can do this manually or use this handy this PowerShell script:

Do note that you won’t be able to add an app registration as an owner through the Azure Portal! This can only be done with through the APIs.

To use this script simply run it and enter in your tenantId and the applicationId for this particular app registration. Next you’ll be asked to perform a device authentication with your browser and the required permissions will be added to the app registration:

Once you’re done it should look something like this:

Configure Azure DevOps permissions

As stated above within Azure DevOps the built-in “Build Service” user should become a member of the “Endpoint Administrators” group for it to be able to make alterations to the service connections.

The name of the “Build Service” account starts with your project name:

Setting up the pipeline

A copy of all source code listed below can be found on my Github page here.

Note: the only changes you should make is the serviceConnectionName variable.

If you need to apply automatic key rotation to more than one service connection, you can add as many variables for them that you’d like. Note that you have to duplicate the code block in lines 24–30 for every additional service connection as well.

How the PowerShell script works

Here’s an overview of what steps are taken within the script to get a better understanding of how it works.

1. Right at the start the lifetime for newly generated secrets is defined by a parameter. In this case it is set to 15 days. So within two weeks this app registration will expire, but if this pipeline will run every day you will never reach that limit.
2. Several variables like accesstoken, tenantID, project URI and app registrations-specifics are set based on environmental variables on the agent.
3. OAUTH2 Authentication is used to perform the authentication against the Microsoft Active Directory tenant with the application ID and secret.
4. The script will check if it can find the app registration with a Graph API lookup.
5. Next, the service connection is queried through Azure DevOps serviceendpoint API. If multiple service connections are found using the same app registration, or if no service connections were found at all, the script will exit and an error with details will be shown.
6. A new secret is generated on the app registration using the Microsoft Graph API with the lifetime defined at the start. The build URL from this pipeline run is added as a description.
7. Within Azure DevOps the service connection is updated with the new secret accordingly.
8. Lastly, the script will look for old secrets the app registration might be having. These will be removed. The script will ignore the just recently generated secret as well as one additional existing secret based on its startDateTime. So only the two most recent secrets are kept.

The result

If everything goes according to plan after several runs of your pipeline your app registration should look something like this:

In this example the pipeline was ran two times in a row therefore the expiration dates on both secrets are the same. All older secrets are automatically removed.

Once your pipeline has successfully run for the first time, you might as well go ahead and delete your initial secret (the one with the long lifetime) because you’re not using it anymore. Eventually this secret will be automatically removed by the script the third time it runs because it will only keep the two most recent secrets.

And there you have it! A fully automated rotation of secrets without any human intervention. Nobody needs to know any secrets and even if somebody pulls the secret from a pipeline, it will expire very soon.
Much much sooner that what I’m currently seeing at most organizations anyway. 😉

I’d like to express a BIG thank you to my dear colleague Dennis van den Akker whom without, this technical solution was not achieved and this blog wasn’t made possible. Thanks buddy! 👊🏻

— Koos

Keys? In a safer future we don’t need keys!