Managing Splunk Using GitHub

There are times where a Splunk Team may want to manage their Splunk Deployment using GitHub or some version control tool for scalability, security, CICD, and other Redundancy purposes. Whatever your reasons for using a SVN tool, there are some really good benefits for source controlling your changes through GitHub. Below is a quick guide on how to integrate the two applications.

We are using a Search Head Cluster as an example but you can of course use any Splunk Component(distributed or standalone) you choose.

Here are the high level steps.

1. Create a Dedicated Repo in GitHub

2. Create Auth/Personal Tokens on GitHub(Click Here)

   1. If using a dedicated service account to push changes, ensure account has been created.

3. Install Git Utility on the Deployer

4. Ensure Splunk Admins or System User(Step #2) have their Auth tokens configured on the Deployer (Click Here)

5. Create a dedicated staging area on the Deployer for your GitHub changes

6. Create an rsync/scp cron script that will grab you user private objects & app specific changes from the SH Captain and merge them to the deployer's dedicated location in Step #5

7. Run Git Commands to Push Changes(Pull Request) from Deployer to GitHub

8. Approve or Deny Pull Requests based on Company Policies

9. Merge Changes to GitHub Repo

1. Create a Dedicated Repo in GitHub ( https://docs.github.com/en/get-started/quickstart/create-a-repo )

2. Create Auth/Personal Tokens on GitHub(Click Here)

   1. It is recommended to use authorization tokens vs password as GitHub is attempting to increase security and reduce password related breaches. GitHub automatically cleans out unused auth tokens older than a year. There is a feature to set token expiration for shorter time frames to adhere to stricter agency/company mandates.

   2. Follow the instructions from the link, be sure to set caching if not against your security policies.

   3. Additionally tokens are only allowed with https, so be prepared to validate SSL certs when making merge requests.

3. Install Git Utility on the Deployer

   1. You can use you normal installer utility typically to install git(yum, apt-get...etc)

   2. If no internet connectivity, you have to use scp, satellite repo, file server or a proxy server to install git(Installing Git).

4. Ensure Splunk Admins or System User(Step #2) have their Auth tokens configured on the Deployer (Click Here)

   1. Modify Git Config files and ensure token has been cached on the deployer.

5. Create a dedicated staging area on the Deployer for your GitHub changes

   1. Create something like (/opt/git, /apps/git, /svn/repo)

6. Create Copy Script to Pull Configs from SHC to Deployer's dedicated staging area in #5

   1. Create a script that will pull $SPLUNK_HOME/etc/users and $SPLUNK_HOME/etc/apps from either the Captain(if working properly or any of the search head nodes). The Search Head Captain is ideal as it will be aware of all changes on the Cluster prior to the other nodes.

7. Run Git Commands to push configs to your GitHub Repo(This can be automated if using a service account)

   1. Make sure to create a new branch each push

   2. Ensure you have proper process to validate EVERY PR. Even if it's automated. You don't want to push trash in your repo.

8. Validate PR and Merge

   1. As mentioned in step #7, please validate all PRs before merging to your master branch.

9. Merge Changes

   1. Merge changes and repeat! 😁