Google Cloud Platform & Splunk Integration
In this blog today we’re gonna show how to integrate GCP (Google Cloud Platform) data into splunk using “Splunk add-on for google cloud platform”.
There are generally five inputs available within this app :
1. Cloud Pub/Sub 2. Cloud Monitoring 3. Cloud Billing 4. Cloud Storage Bucket 5. Resource Metadata
Now we will see in this blog how we can ingest these logs within Splunk to analyze respective GCP servers.
Prerequisites:
1. Need to have a valid Google Cloud account. 2. You must have admin or sc_admin privilege. 3. Need to be “Owner” or “Editor” of the projects available in GCP to push data from GCP to Splunk. 4. Need to have access to GCP IAM.
So let’s start the proceeding step-wise.
Step-1: Set up a Google cloud service account
Click this below link to open the “Service account page”.
https://console.cloud.google.com/iam-admin/serviceaccounts
Note: Don’t forget to register with your Gmail/mail, which has access to the GCP
Then choose the Project, which you want to monitor.
Then Click on “Create Service Account”
Name: <give a name to your service account>
Now click on “ADD KEY” and choose “Create New Key”
Then choose JSON and Create.
As soon as you click on create it will download a JSON file, and open that file it will be like this.
It is a JSON file with all required credentials like “private_key”, “client_id”, “client_email” etc.
Step-2: Create a PUB/SUB Subscription
In the search bar available in “GCP”, enter “pub/sub” and choose “PUB/SUB Subscription”
Click on “Create Topic” and create a topic called “abc”
As soon as you create the topic “subscriptions” will be created with the same name.
Step 3: Install “Splunk Add-on for Google Cloud Platform” and configure
Now log in to your Splunk instance with your credentials.
Click on “Find More Apps”
Search for “Splunk Add-on for Google Cloud Platform” and Click on install
Login with your Splunkbase credentials and install.
After installing restart your Splunk instance.
Now launch the app and go to the ”Configuration” tab, click on “Add Credentials”
Name : <choose any name> Google Service Account Credentials : <copy the JSON KEY from the file you have download previously for the Service account, and paste it here>
Click on Add
Now navigate towards the “Input” tab to create a new input and choose “Cloud Pub/Sub”
Name : <Choose name of your Input> Credentials : <Select from dropdown list, which just conferred in “Configuration” tab> Project : <Choose from the dropdown, automatically extracted from JSON key file> Pub/Sub Subscription : <Choose the subscriptions name> Index : <Choose the index, for indexing the data in Splunk>
Click on Add
Now go to the Search tab, and search
index=main sourcetype="google:gcp:pubsub:message"
NOTE: pub/sub message is data related to messages published from the cloud platform.
There are 5 inputs available, you can configure one by one from the same credentials.
The procedure of adding input is the same for all. Like this also we have added a few inputs.
We have configured “Resource Metadata” also, check below.
index=main sourcetype="google:gcp:resource:metadata"
And after adding those inputs for the demo we found 34 sources, related to network speed, Disk usage, cpu usage, data based on region and many more.
In this way, you also can explore “GCP” data with Splunk.
Happy Splunking!!
