AWS S3 and Splunk Integration

AWS S3 and Splunk Integration

Welcome back guys , to one more amusing post on our blog. Today we are going to show you how to ingest data from an AWS S3 bucket to Splunk.

Step- 1 : Check the s3 bucket(s) you want to integrate.

For demonstration purpose we have created an AWS S3 bucket by the name s3-bucket-splunk, we have to integrate this bucket with our splunk so that any data stored in this bucket is ingested in Splunk.

AWS1

Step-2 : Checking/Adding data to the bucket.

We have uploaded a file named test-log to our s3 bucket which consists some linux secure file logs.

AWS2

Step -3 : Create an user (in case you don’t have one) having privilege to read the contents of the bucket(s).

We have created a user called test-user and attached the AdministratorAccess policy.

AWS3

Step-4: Generate a Key id and secret key for that user.

Step-5:  Download the splunk add-on for amazon web services , you can use the link below

https://splunkbase.splunk.com/app/1876/

AWS4

Step-6: On the Splunk instance go to Manage Apps >> Install App from file and upload the add-on you just downloaded, once the installation is complete you need to restart your Splunk.

Step-7: On the add-on interface navigate to Configurations  >> Account and click on the Add button.

You will get a pop-up form as shown below.

You can also know about :  O365 Integration With Splunk

AWS5

Name : Provide the user for this account.

In our case – test-user

Key ID: Provide the Key ID for this user.

Security Key: Provide the security Key for this user.

Region Category:   Select the region , Global by default.

Once done submit by clicking on the Add button.

AWS6

Step-8: Navigate to the Inputs page, Click on Create new input >> S3 Access logs, under Input Type select Generic S3.

AWS7

Name: Provide a name for this input.

AWS Account: Select the AWS Account with the right access.

Assume Role: If you want to assume any role for the user, optional.

S3 Bucket: Select the S3 bucket available for this user.

S3 Key Prefix:  Provide the s3 key prefix, if required, optional.

Start Date/Time: The timestamp from where you want to ingest the data.

End Date/Time: The timestamp at which you want to stop ingesting the data.

Index: Select the index where you want to store the incoming data.

Depending on your requirement you can set the polling interval, the frequency at which splunk will fetch the data from aws bucket.

Once done, Click on the save button.

Step-9:  Verify the data  in your Splunk.

AWS8

Congratulations!! You have successfully completed the Integration.

For more posts like this keep following us.

You can also know about :  Index Time Field Extraction in SPLUNK

Happy Splunking!!

3 comments

  1. Is there a more specific permission sets? I don’t want to give out AdministratorAccess for production account.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.