Logstash goes passwordless!
By authentication with Managed Identities
TTL;DR Together with my dear friend and fellow-MVP Pouyan Khabazi, I've spend a day and a night (there might've been some pizza involved 🍕) in a separate room with our notifications turned off. We dug into Microsoft's source code for their Logstash output plugin, and came up with a solution for integrating Managed Identities. No more plain-text passwords! Nor the need for setting up key rotation mechanisms. Finally!
Introduction
A short while ago Pouyan and me got talking about the key rotation mechanism (r0t8r) I've built for Logstash earlier. We both still didn't really like the fact that we still needed to create an application registration and secrets in order for Logstash to be able to authenticate to Azure.
Bruno Gabrielli at Microsoft had recently shown in a blogpost that Azure Arc-enabled Servers should be able to authenticate to the Azure Monitor log ingestion API. So why wasn't Logstash able to leverage this authentication method?
While Microsoft did transition from workspace ID and key authentication to the newer DCR-based ingestion API a while back, there haven't been much updates on this plugin ever since. And authenticating with Managed Identities was never implemented.
But luckily for us the plugin is open-source. So, Pouyan and I decided to join forces and take a closer look into the source code to see if we could find any way to add a new authentication method. And we succeeded! 💪🏻
Our pull request is currently under review and waiting for approval.
If you don't want to wait you can download the updated plug-in from the repository and install it manually. Instructions on this are further down in this article.
Why was this so important to us?
A common challenge for developers and sysadmins is the management of secrets and certificates used for authentications between services.
Sure, with Azure Key Vault and key rotation mechanisms you can achieve a secure solution. But it requires a lot more overhead and complexity. Lastly, Logstash unfortunately stores your application secret in plain text in a configuration file, or you should create a Logstash keystore which adds again another layer of complexity.
Managed Identities eliminate the need for managing these credentials altogether!
- No need for managing credentials. Credentials aren’t even accessible to anybody.
- System-assigned Managed Identities are tied to a specific Azure resource. If the resource is removed, so is the corresponding identity. You don't need to worry about lifecycle management.
- Using Managed Identities is free.
There are two types of managed identities:
- System-assigned — Some Azure resources, such as Logic Apps, App Services and Virtual Machines allow you to enable a Managed Identity directly on the resource.
- User-assigned — You can also create a Managed Identity and assign it to one or more Azure resources.
System-assigned Managed Identities
When enabled on an Azure resource:
- A service principal of a special type is created in Microsoft Entra ID for the identity. The service principal is tied to the lifecycle of that Azure resource. When the Azure resource is deleted, Azure automatically deletes the corresponding service principal for you.
- By design, only that Azure resource can use this identity to request tokens from Microsoft Entra ID.
- You authorize the managed identity to have access to one or more services in a similar fashion as you would with app registrations.
- The name of the system-assigned service principal is always the same as the name of the Azure resource it is created for.
User-assigned Managed Identities
Needs to be created manually.
- A service principal of a special type is created in Microsoft Entra ID for the identity. This service principal is a separate entity and needs to be managed separately from the resources that use it.
- A single user-assigned identity can be used by multiple resources.
- Authorizing a user-assigned Managed Identity works the same as with a system-assigned variant.
Learning Ruby
WWith a "little bit" of help by GitHub Copilot, we managed to learn more about Ruby, the open-source programming language used by Elastic to built Logstash. And we quickly found the logic which was responsible for retrieving a bearer token from Microsoft's Graph API with the use of the application registration and secret. Next, we had to figure out how we could replace this bearer token with the one from our Managed Identity.
Retrieving a bearer token
Logic for requesting a token is done in logAnalyticsAadTokenProvider.rb and replacing it with the bearer token from the Managed Identity seemed pretty straightforward. But since we'd be requiring a different request URI, header and logging output, we decided to create a copy named logAnalyticsMiTokenProvider.rb and stored the different requirements there:
For some reason the API's for requesting Managed Identity bearer tokens provides the expires_in property (depicting the amount of seconds before token expiration) as a string instead of an integer! This caused the token lifetime calculation and automatic renewal to stop working. So, we had to force the response into an integer ourselves:
Next, we needed to add a new boolean parameter managed_identity in microsoft-sentinel-log-analytics-logstash-output-plugin.rb to use in pipeline configuration files:
Validating the Logstash configuration file is done by logstashLoganalyticsConfiguration.rb. Therefore we had to add an if statement checking if the managed_identity boolean was used or not. Because client_app_id, client_secret and tenant_id will then no longer be used:
Using Azure Arc connected machine agent
To get a bearer token on an Azure Arc connected machine the process is a bit different. As Bruno Gabrielli at Microsoft pointed out we first had to communicate with the local azcmagent service to find the path to a .KEY file for authenticating with the azcmagent. Next, we can use the contents of this .KEY to retrieve the bearer token by communicating with agent again:
CHALLENGE_TOKEN_PATH=$(curl -s -D - -H Metadata:true "http://127.0.0.1:40342/metadata/identity/oauth2/token?api-version=2019-11-01&resource=https%3A%2F%2Fmanagement.azure.com" | grep Www-Authenticate | cut -d "=" -f 2 | tr -d "[:cntrl:]")
CHALLENGE_TOKEN=$(cat $CHALLENGE_TOKEN_PATH)
if [ $? -ne 0 ]; then
echo "Could not retrieve challenge token, double check that this command is run with root privileges."
else
curl -s -H Metadata:true -H "Authorization: Basic $CHALLENGE_TOKEN" "http://127.0.0.1:40342/metadata/identity/oauth2/token?api-version=2019-11-01&resource=https%3A%2F%2Fmanagement.azure.com"
fi
Because of the different approach required compared to requesting a token on an Azure Virtual Machine, we decided to put this logic into a separate file as well named logAnalyticsArcTokenProvider.rb.
IMPORTANT
Only users which are a member of the
himdsgroup are allowed to retrieve the contents of the.KEYfile. If you want to run Logstash as a system service (and you probably will) you'll need to add thelogstashuser to this group by running:
sudo usermod -a -G himds logstash
Detecting Azure Arc
With a new function added to logAnalyticsClient.rb we check if the azcmagent is available or not. This determines if the machine is running in or outside of Azure and we can request the token appropriately.
logAnalyticsMiTokenProvider.rb or logAnalyticsArcTokenProvider.rb will be handling the token requestInstallation
Once Microsoft approves our pull request the updated plugin should automatically become available for automatic installation. Logstash plugins are available in self-contained packages called gems and hosted on RubyGems.org.
sudo /usr/share/logstash/bin/logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-pluginMore information about working with Logstash plugins can be found here.
If you're already running the plugin, you can check its version by running:
sudo /usr/share/logstash/bin/logstash-plugin list --verbose microsoft-sentinel-log-analytics-logstash-output-plugin
Make sure you're using the correct plugin name! Microsoft did deprecate earlier plugins a couple of time now. Plugins like:
microsoft-sentinel-logstash-output-plugin,microsoft-logstash-output-azure-loganalyticsandazure_loganalyticsare all deprecated!The latest version is named:
microsoft-sentinel-log-analytics-logstash-output-plugin(say thay ten times fast)
Manual installation
If you can't wait for Microsoft to merge our pull request, or your server does not has a connection to the Internet, you can also perform an offline installation.
- Download the plugin files from the GitHub repository and store the folder in
~/microsoft-sentinel-log-analytics-logstash-output-pluginfor example. - Add the path to Logstash's Gemfile:
sudo nano /usr/share/logstash/Gemfileand scroll all the way down and add:gem "microsoft-sentinel-log-analytics-logstash-output-plugin", :path => "/home/<USERNAME>/microsoft-sentinel-log-analytics-logstash-output-plugin/":
3. Install the plugin by running: sudo /usr/share/logstash/bin/logstash-plugin install --no-verify:
Thank you Akintola ADJIBAO for providing us this guide!
4. Update/create your Logstash pipeline configuration specifying a Managed Identity for its authentication method:
output {
microsoft-sentinel-log-analytics-logstash-output-plugin {
managed_identity => true
data_collection_endpoint => "https://< DCE URI >.ingest.monitor.azure.com"
dcr_immutable_id => "dcr-< IMMUTABLE ID >"
dcr_stream_name => "Custom-< TABLE NAME >"
}
}Conclusion
Both Pouyan and me had a blast during our little "hackathon"! We learned a lot about Ruby and requesting tokens via Azure Arc. GitHub Copilot was a real lifesaver here!
We hope everything was clearly explained, and it sparked enough interest get rid of application registrations and secrets!
If you have any questions don’t hesitate to reach out to either of us! You can find Pouyan here on Medium as well or on X. Also make sure to listen to his latest podcasts about Defender for Cloud and DevSecOps on the TalkingSecurity Podcast! 👌🏻🎙️
You can follow me Medium or keep an eye on my X and LinkedIn feeds to get notified about new articles here on Medium.
We still wouldn’t call ourselves experts on Ruby. So, if you have feedback on our approaches, please let us know! Also don't hesitate to fork the Microsoft repository and submit your own pull request!
We hope you liked this solution as much as we do, and it will help make your environment a little safer as well!
— Koos & Pouyan
