On June 14, 2022 Splunk Inc. has informed the larger Splunk users community that there is a pretty scary new exploit that could allow Splunk Deployment servers to be hijacked using one of it's deployment clients.
Why is this important? The Splunk Deployment Server is traditionally used to manage all clients in a normal splunk deployment. Meaning that every single asset(workstation or server) that has a Universal Forwarder(Splunk's collection agent) installed will be connected to one or more Deployment Servers for updates and general management. Administrators can deploy updates to all Splunk Universal Forwarders(UFs) or any other Splunk clients checking in( Heavy Forwarders, Search Heads, Indexers, Cluster Master, etc..) through packages maintained via Splunk's server classes method. These packages are in ASCII format and can be splunk specific configuration updates via text files. The packages can also be scripts(bash, python, powershell, etc) that execute locally on each server.
This means that if your DS is compromised, essentially you could have your entire environment compromised. This also means that if executed appropriately, a bad actor can schedule(logic bomb) or run a job that could own you entire infrastructure if timed accurately. So think about your Domain Controllers, Active Directory Servers, critical Web Servers, custom Application Servers or Databases containing invaluable records of all your customers/cients.
How attack works?
Currently the exact details of the attack is unknown and Splunk has not published(wisely) the exact method to take advantage of this exploit. The mechanism in which the attack works is essentially remote code execution via reverse hijacking of the Deployment Server(DS). As mentioned above, typically the DS controls any clients checking in for updated packages/bundles. However, this vulnerability allows an an actor to publish custom bundles from an individual universal forwarder(only) to all other clients checking in on the Deployment Server. So this means a UF can own a Heavy Forwarder, Search Head, Indexer, Cluster Master, Deployer and your entire Splunk Infrastructure if you have them as a client(not best practice) to the Deployment Server.
Who is affected?
Only Splunk Customers who utilize the Deployment Server functionality are affected by CVE-2022-32158. If you are unsure if your Deployment Server is configured to manage agents, you can check on your DS in the GUI by going to Settings ---> Forwarder Management (see below). You can also use a combination of searches from either your DS(REST based--| rest splunk_server=local /services/deployment/server/clients) or Splunk SPL of internal from your Search Heads or Monitoring Console to view clients checking in(index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host).
If GUI is disabled on your Deployment Server in your env, on the command line(CLI), you can check in $SPLUNK_HOME/etc/deployment-apps location to see if you have any apps that are actively being deployed. Admins can also run the following command on the DS $SPLUNK_HOME/bin/splunk list deploy-clients to see which clients are checking in. You can also tail the $SPLUNK_HOME/var/log/splunk/splunkd.log and look for active connections with DC component.
How to Stop the Attack?
Splunk's primary recommendation is to upgrade the the Splunk Deployment Server to Version 9.0. While this presents other challenges for some customers like application package approvals, possible vulnerability to future unknown version exploits, and possible bugs; this is the best option at this time :-|....
As far as we are aware, there are currently no plans to retrofit the current Splunk 9.0 version patch to older Splunk versions(pre 9.x) but we will keep our ears to the ground and update when informed.
OpenPath Recommendations
If the scenario permits and your Deployment Server is a stand alone machine, upgrade!
If upgrading to Splunk 9.0 is absolutely out of the question,
Customers can create monitors of what apps are being deployed from the Deployment Server.
Customers can utilize Splunk's Blacklist client feature to manage what apps are deployed to what servers. While this is not meant for that functionality, by blacklisting certain servers within server classes, customers maybe able to control what apps are deployed to what servers.
If your Deployment Server is serving multiple Splunk functions(i.e License Master, Cluster Master, Monitoring Console), please consider splitting the roles to a dedicated server, meaning move the DS to another server or one of the other functions to another server. Then upgrade to Splunk 9.0 for the DS.
Depending on the Splunk version your other components are, you could break Splunk if you upgrade to version 9.0 and your Search Head, Indexers, or Forwarders aren't compatible with version 9.0. Please refer to the Splunk Compatibility Matrix