AWS S3 and Splunk Integration
Welcome back guys , to one more amusing post on our blog. Today we are going to show you how to ingest data from an AWS S3 bucket to Splunk.
Step- 1 : Check the s3 bucket(s) you want to integrate.
For demonstration purpose we have created an AWS S3 bucket by the name s3-bucket-splunk, we have to integrate this bucket with our splunk so that any data stored in this bucket is ingested in Splunk.
Step-2 : Checking/Adding data to the bucket.
We have uploaded a file named test-log to our s3 bucket which consists some linux secure file logs.
Step -3 : Create an user (in case you don’t have one) having privilege to read the contents of the bucket(s).
We have created a user called test-user and attached the AdministratorAccess policy.
Step-4: Generate a Key id and secret key for that user.
Step-5: Download the splunk add-on for amazon web services , you can use the link below
https://splunkbase.splunk.com/app/1876/
Step-6: On the Splunk instance go to Manage Apps >> Install App from file and upload the add-on you just downloaded, once the installation is complete you need to restart your Splunk.
Step-7: On the add-on interface navigate to Configurations >> Account and click on the Add button.
You will get a pop-up form as shown below.
Name : Provide the user for this account. In our case – test-user Key ID: Provide the Key ID for this user. Security Key: Provide the security Key for this user. Region Category: Select the region , Global by default.
Once done submit by clicking on the Add button.
Step-8: Navigate to the Inputs page, Click on Create new input >> S3 Access logs, under Input Type select Generic S3.
Name: Provide a name for this input. AWS Account: Select the AWS Account with the right access. Assume Role: If you want to assume any role for the user, optional. S3 Bucket: Select the S3 bucket available for this user. S3 Key Prefix: Provide the s3 key prefix, if required, optional. Start Date/Time: The timestamp from where you want to ingest the data. End Date/Time: The timestamp at which you want to stop ingesting the data. Index: Select the index where you want to store the incoming data.
Depending on your requirement you can set the polling interval, the frequency at which splunk will fetch the data from aws bucket.
Once done, Click on the save button.
Step-9: Verify the data in your Splunk.
Congratulations!! You have successfully completed the Integration.
For more posts like this keep following us.
Happy Splunking!!
is there any documentation for clustered environment as well ?
Soon, it is coming, stay tuned !!
can you share the link for cluster environment as well ? how to pass the data from S3 bucket to splunk using forwarders.
Is there a more specific permission sets? I don’t want to give out AdministratorAccess for production account.