Secure your Logstash connections to Microsoft Sentinel

by implementing automatic key rotation with "Rot8r"

UPDATE

I’ve recently worked on the source code for the Microsoft Sentinel / Log Analytics output plugin (together with a fellow-MVP) and we’ve implemented Managed Identity support. This completely eliminates the use of secrets altogether and also makes this key rotation mechanism obsolete.

I’d recommend not using application IDs and secrets any longer, and make use of Managed Identities instead.

Read more about this update, and how to use it here.

Introduction

People who work with me, and/or read my articles here on Medium, already know this: I’m a big fan of Elastic’s Logstash. I regularly called it a "Swiss Army knife" due to its versatility and capabilities, and already wrote several articles about implementing Logstash as a log collector for Microsoft Sentinel. After recent updates, both to log ingestion methods for Sentinel and the newly updated output plugin, I also wrote about ingesting DCR-based custom logs with Logstash.

But one thing always kept nagging me: I really didn't like those (plain text!!1) passwords in the pipeline configuration files! 😳

Many companies struggle to implement automatic key rotation because most of their Logstash instances reside in on-premises infrastructure, where Infrastructure-as-Code principles don't exist. Therefore rotating and updating these secrets was all done manually, or worse; not at all! It was like the Wild West!

The land of the free… On-premises infrastructure deployments sometimes feel like a thing of the past.

So, I wanted to come up with an easy solution which could be implemented to automate this process. No matter where Logstash resides, how it was deployed in the first place nor what operating system it was running.

Please let me introduce: Rot8r 🤖

Authentication overview

A quick recap; Logstash will authenticate to a Data Collection Rule via a Data Collection Endpoint with an applicationId and secret.

General overview of a syslog forwarder for Microsoft Sentinel based on Logstash and DCR-based tables

Within Logstash this "pipeline" is configured in a configuration file. These are stored by default in /etc/logstash/conf.d/*.conf. Here's an example:

input {
     syslog {
         port => 5514
    }
}

output {
    microsoft-sentinel-logstash-output-plugin {
        client_app_Id => "< applicationId >"
        client_app_secret => "myapplicationsecret"
        tenant_id => "< tenantId >"
        data_collection_endpoint => "https://< DCE URI >.westeurope-1.ingest.monitor.azure.com"
        dcr_immutable_id => "dcr-< immutableId >"
        dcr_stream_name => "Custom-< tableName_CL >"
    }
}

If you're new to DCR-based custom logs and/or Logstash, please read my two earlier articles about these subjects here and here.

If a new secret is created, somebody needs to go into the configuration file and update the secret there. And that's where my problem lies…

Key rotation for Azure AD applications

My idea was that the application(s) we create for Logstash, could benefit from the same key rotation approach I used for Azure DevOps service connections in the past.

Was this two years ago already?! Time flies…

This method requires the application to be an owner of itself, so it was able to create new secrets and remove old ones by performing specific requests to the Azure AD Graph API. More on this later…

Instead of updating Azure DevOps connection credentials, the script should now update the secret in the Logstash configuration file.

PowerShell on Linux

You might be wondering: "Wait, how are you going to run a PowerShell script on a Linux machine? An operating system frequently used for running Logstash"

Well, since PowerShell went open-source as "PowerShell Core" version 6 in 2016, we also gained the capabilities of running PowerShell cross-platform on Linux and MacOS for example.

PowerShell core is my default shell on MacOS when opening up Terminal

I used Ubuntu for this example and it was very straightforward:

# Update the list of packages
sudo apt-get update
# Install pre-requisite packages.
sudo apt-get install -y wget apt-transport-https software-properties-common
# Download the Microsoft repository GPG keys
wget -q "https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb"
# Register the Microsoft repository GPG keys
sudo dpkg -i packages-microsoft-prod.deb
# Update the list of packages after we added packages.microsoft.com
sudo apt-get update
# Install PowerShell
sudo apt-get install -y powershell
# Start PowerShell
pwsh

Microsoft describes how to install PowerShell on all different Linux distributions in their documentation.

Logstash keystore (optional)

And what about those passwords being stored as plain text in those configuration files? Well, it turns out Elastic has you covered! You can create a Logstash keystore on the local machine to securely store key pair values in.

The Logstash configuration will then only show key references instead of plain text passwords:

input {
     syslog {
         port => 5514
    }
}

output {
    microsoft-sentinel-logstash-output-plugin {
        client_app_Id => "${app_id}"
        client_app_secret => "${app_secret}"
        tenant_id => "${tenant_id}"
        data_collection_endpoint => "https://< DCE URI >.westeurope-1.ingest.monitor.azure.com"
        dcr_immutable_id => "dcr-< immutableId >"
        dcr_stream_name => "Custom-< tableName_CL >"
    }
}

The local keystore can be password protected (optional, but recommended) which is stored in an environment variable only readable to the user account which needs access to the keystore.

set +o history
export LOGSTASH_KEYSTORE_PASS=mykeystorepassword
set -o history

By using set +o history we omit the password being stored in shell's history

Create the keystore

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create

Add key

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add app_secret

List all keys

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

Remove key

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash remove app_secret

By adding -E to sudo the current environment variables stay accessible. Otherwise the keystore process would not be able to retrieve the password from the environment variable set earlier. For a production environment it's HIGHLY advised to perform these steps without sudo, and use a dedicated user responsible for key rotation with appropriate permissions to required files.

For more information about Logstash keystore, please check out Elastic's documentation about this subject.

Preparations

Before we can start using Rot8r, we need to make sure our Azure AD application is prepared and ready so that it's able to update its own keys.

Application permission requirements

• Application needs to be owner of it’s own application.
• Application requires the following Microsoft.Graph permission: Application.ReadWrite.OwnedBy.

The main problem is that the Azure portal will not allow us to add the application as an owner for itself. But luckily the Graph API allows us to work around this.

Therefore I've created a PowerShell script Add-AppOwner.ps1 to help you with these steps. This script needs to be run one time only to setup the proper permissions for this application.

Both scripts mentioned in this article can be found in my Github repository.

To run this script you need to following permissions:

• Global Administrator, or have all of the following permissions:
  - Application.ReadWrite.All
  - Directory.Read.All
  - AppRoleAssignment.ReadWrite.All

The script Add-AppOwner.ps1 will:

• Ask to perform an interactive device login. (the validation code will be copied to clipboard for your convenience 😉)
• Validate the applicationId
• Retrieve the current owner(s)
• Add the application delegation for Microsoft.Graph permission Application.ReadWrite.OwnedBy
• Perform the admin consent for the application delegation Microsoft.Graph permission Application.ReadWrite.OwnedBy
• Add the service principal to the application as owner

Add-AppOwner.ps1 in action, taking care of all preparations for you

After running Add-AppOwner.ps1 your application in the Azure portal should look something like this:

The application is now owner of itself

Graph permissions are added, including admin consent

Create first secret

There's one last thing to do and that's creating the very first secret. Once Rot8r is scheduled to run regularly, this secret will automatically be "rotated" away and removed. But make sure to copy the password somewhere for your first Rot8r run. After this first run the secret will no longer be needed.

By selecting "Custom" you can use a very short lifespan, which I would highly recommend

Rot8r.ps1

Ok, now it's time to look at the main attraction here! Here's a general overview of what steps Rot8r will perform:

Rot8r process overview

1. After Rot8r is started (i.e. by a scheduled task or cron job) it will authenticate towards Azure Active Directory using the same credentials as Logstash uses to ingest logs into Sentinel. The secret PowerShell uses to authenticate is pulled from a secure .cred file. More on this below…
2. It wil generate an additional secret with a limited lifecycle (default of 31 days)
3. The newly created secret is encrypted and stored again in the secure .cred file, overwriting the previous one.
4. Depending on the parameters provided either one of two things will happen next:
   a) Logstash configuration file (path provided as parameter) is read, and the existing secret is replaced with the newly generated one.
   b) A Logstash keystore key (provided as parameter) is updated with the new value of the newly created secret.
5. All secrets, minus the two most recent ones, are deleted.
6. Logstash system service is restarted to immediately make sure changes are in effect and using the latest configuration change.

PowerShell .cred file

The latest secret is always stored into a .cred file for the next run of Rot8r. This file contains the System.Security.SecureString PowerShell uses to encrypt passwords.

To read more about SecureString and how secure these are, please see Microsoft's documentation about this subject.

Upon first run this file doesn't exist yet, therefore Rot8r will ask you to provide a secret (you've created this during preparation steps above) so that it can create its first .cred file:

Rot8r asking for a secret. This is a one-time-only event.

Rot8r.ps1 can be found in my Github repository

Parameters

Rot8r's behavior can be modified with a couple of parameters:

• secretAddDays
  The number of days the new application secret will be valid. Default is for 31 days.
• tenantId
The Tenant ID of the Azure Active Directory in which the application resides.
• applicationId
The applicationId of the application on which the secret needs to be rotated.
• logstashConfigLocation
Path to logstash pipeline configuration file i.e. '/etc/logstash/conf.d/syslog-to-sentinel.conf'
• logstashKeystoreKey
Name of the key in the keystore container the app secret referenced inside the Logstash configuration file.
• printOutput
Add '-printOutput' for easier troubleshooting external Logstash-specific commands, like updating keystore key and restarting service.

Example with config file

sudo pwsh -File logstash-rot8r.ps1 -tenantId <tenant_id> --applicationId <app_id> -logstashConfigLocation /etc/logstash/conf.d/syslog-to-sentinel.conf

Where -logstashConfigLocation refers to the physical location of the Logstash pipeline configuration you wish to apply key rotation on.

Example with keystore

sudo pwsh -File logstash-rot8r.ps1 -tenantId <tenant_id> --applicationId <app_id> -logstashKeystoreKey sentinel_app_id

Where -logstashKeystoreKey refers to a key saved in the Logstash keystore.

It’s important to note that you either provide a logstashConfigLocation OR a logstashKeystoreKey and not both!

As mentioned above, the Logstash keystore might rely on an environment variable containing the keystore’s master password. (depending on your setup) This variable and the physical location of the keystore file(s) should be accessible by the user running ‘Rot8r’. If updating the keystore key fails, you might want to consider adding the printOutput switch so that these steps are no longer verbose, and the results can be viewed in the console.

sudo pwsh -File logstash-rot8r.ps1 -tenantId <tenant_id> --applicationId <app_id> -logstashKeystoreKey sentinel_app_id -printOutput

Rot8r in action, performing both updates to Azure AD as well as to Logstashes keystore

Conclusion

With Rot8r you can shorten your lifespan of your secrets drastically. You can even consider running this daily and keep the lifespan of your secrets as short as a few days!

When using this in conjunction with Logstash keystore, the secret is no longer visible in plain text anywhere on the server. And because the rotation task is fully automated, nobody needs to even know what the current key value is!

There's obviously still a risk that the user running the process, or the entire machine, could be compromised. Resulting in a way in to decrypt the .cred file and gain access to the account. But since you'll only be using this account for publishing logs to one or more Data Collection Rules, the impact can be considered very very small. And don't forget how much better this compared to where we came from…

I wouldn't call myself an expert on PowerShell nor am I an advanced Linux sysadmin. So if you have feedback on any of my approaches above, please let me know! Also never hesitate to fork my repository and submit a pull request. They always make me smile because I learn from them and it will help out others using these tools. 👌🏻

I hope you like this tool and it will make your environment safer as well.

If you have any follow-up questions, please reach out to me!

— Koos