How To Index The Last Line Of A Log File In Splunk
Hi everyone!!
We have come with a new and interesting topic related to the Splunk Administration that is “How To Index The Last Line Of A Log File In Splunk”. We often see that multi-line events come from the application server side. But most of the case some portion of those log are not so important. Let’s say the last line is carrying all the information about the log and you want to ingest only the last line of those log files. Did you ever think about this?
So, in this blog we are going to discuss about that thing. We will show you how to take only the last line of a multi-line event. We will be using some vm’s in the GCP. Please find the screen shot for this demo.
Step 1: Create an index in the Indexer.
Login to the indexer and then go to Setting >> Indexes.
You can see the “New Index” option on the top right corner. Click on that.
We have given the index name as “test”. After that click on Save to save the changes.
Once verify by putting the index name in the search bar.
Step 2: Write props.conf in the Heavy Forwarder ( HF )
Login to the CLI of the Heavy Forwarder ( HF ) to write the props.conf.
You can find the props.conf in below path.
$SPLUNK_HOME/etc/system/local
Run “ls” command after going to “$SPLUNK_HOME/etc/system/local” path you will be able to see the props.conf. Open props.conf to edit the file.
Inside the props.conf we have written like below.
[sedlog] SHOULD_LINEMERGE=true SEDCMD-aaa= s/(?msi).*\n(.*)/\1/g
Here “sedlog” is the sourcetype name. We have used the SEDCMD attribute for masking. Actually we are masking all the lines except but the last line. “aaa” is the sed class name. We have written a sed expression after the equal symbol. Basically we are matching multi-line up to the last line using [(?msi).*]. Then we have used “\n” for the new line. Also we have put “.*” inside the first parenthesis. Also we have taken the reference using “\1” for the contents what ever it is coming under “.*” , we have mentioned inside the first parenthesis. After that just save the file.
Restart Heavy Forwarder. Go to the below path
$SPLUNK_HOME/bin
And then run the following command to restart HF.
# ./splunk restart
Step 3: Write inputs.conf in the Universal Forwarder ( UF )
Login to the CLI of the Universal Forwarder ( UF ) to write the inputs.conf.
You can find the inputs.conf in below path.
$SPLUNK_HOME/etc/system/local
Run “ls” command after going to “$SPLUNK_HOME/etc/system/local” path you will be able to see the inputs.conf. Open inputs.conf to edit the file.
Inside the inputs.conf write like below.
[monitor:///tmp/testlog/*.txt] index=test sourcetype=sedlog
Here we are monitoring all the files with “.txt” extension under the “/tmp/testlog” directory.
We have mentioned the index name as “test” which we have already created in our indexer and also we have mentioned the sourcetype name as “sedlog” for which we have already written the props.conf in the HF. After that save this file.
Restart Universal Forwarder(UF). Go to the below path
$SPLUNK_HOME/bin
And then run the following command to restart UF.
# ./splunk restart
After restarting the Splunk Universal Forwarder
Go to the following path
/tmp/testlog
Try to create some file with “.txt” extension.
Step 4: Check on the Search Head ( SH ).
On the search head search by index=test and sourcetype=sedlog
You can see only the last lines of the each log files are being ingested.
Now what will happen if log rotate occurs in the directory. We have removed the older file called a.txt and again created a file with the same name.
Note: For the existing log files like b.txt, c.txt if any new contents comes in then also it will take last line from the delta.
The configuration applied on the new logs also. The last line of the new log rotated file will be ingested in Splunk.
I hope you have understood “How To Index The Last Line Of A Log File In Splunk” clearly.
Happy Splunking !!!
