Google Cloud Platform & Splunk Integration

Google Cloud Platform & Splunk Integration

In this blog today we’re gonna show how to integrate GCP (Google Cloud Platform) data into splunk using “Splunk add-on for google cloud platform”.

There are generally five inputs available within this app :

1. Cloud Pub/Sub
2. Cloud Monitoring
3. Cloud Billing
4. Cloud Storage Bucket
5. Resource Metadata

Now we will see in this blog how we can ingest these logs within Splunk to analyze respective GCP servers.

1. Need to have a valid Google Cloud account.
2. You must have admin or sc_admin privilege.
3. Need to be “Owner” or “Editor” of the projects available in GCP to push data 
from GCP to Splunk.
4. Need to have access to GCP IAM.

So let’s start the proceeding step-wise.

Step-1: Set up a Google cloud service account
Click this below link to open the “Service account page”.

Note: Don’t forget to register with your Gmail/mail, which has access to the GCP

Then choose the Project, which you want to monitor.

Then Click on “Create Service Account

Name: <give a name to your service account>

Now click on “ADD KEY” and choose “Create New Key
Then choose JSON and Create.

As soon as you click on create it will download a JSON file, and open that file it will be like this.

You can also know about :  Sending Data from Database To Splunk Using DB Connect ( DBX - Part 2)

It is a JSON file with all required credentials like “private_key”, “client_id”, “client_email” etc.

Step-2: Create a PUB/SUB Subscription
In the search bar available in “GCP”, enter “pub/sub” and choose “PUB/SUB Subscription

Click on “Create Topic” and create a topic called “abc

As soon as you create the topic “subscriptions” will be created with the same name.

Step 3: Install “Splunk Add-on for Google Cloud Platform” and configure

Now log in to your Splunk instance with your credentials.

Click on “Find More Apps

Search for “Splunk Add-on for Google Cloud Platform” and Click on install

Login with your Splunkbase credentials and install.

After installing restart your Splunk instance.
Now launch the app and go to the ”Configuration” tab, click on Add Credentials

Name : <choose any name>
Google Service Account Credentials : <copy the JSON KEY from the file you have download previously for the Service account, and paste it here>

Click on Add

Now navigate towards the “Input” tab to create a new input and choose “Cloud Pub/Sub

Name : <Choose name of your Input>
Credentials : <Select from dropdown list, which just conferred in “Configuration” tab>
Project : <Choose from the dropdown, automatically extracted from JSON key file>
Pub/Sub Subscription : <Choose the subscriptions name>
Index : <Choose the index, for indexing the data in Splunk>

Click on Add

Now go to the Search tab, and search

index=main sourcetype="google:gcp:pubsub:message"

NOTE: pub/sub message is data related to messages published from the cloud platform.
There are 5 inputs available, you can configure one by one from the same credentials.
The procedure of adding input is the same for all. Like this also we have added a few inputs.

You can also know about :  Send Splunk Alert To A Slack Channel

We have configured “Resource Metadata” also, check below.

index=main sourcetype="google:gcp:resource:metadata"

And after adding those inputs for the demo we found 34 sources, related to network speed, Disk usage, cpu usage, data based on region and many more.

In this way, you also can explore “GCP” data with Splunk.

Happy Splunking!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.