Find Out The Errors Occurring 2 Or More Times Consecutively

Find Out The Errors Occurring 2 Or More Times Consecutively

Today we are back with another trick, which is How can we find out errors occurring two or more consecutive times.
Let’s take an example, we have an index called “test_index”, there we are indexing data related to errors occurring into the servers based on time in the source type “testlog_new”. Take a look.

index="test_index" sourcetype="testlog_new"
| table error _time

1

2

Now here our main concern is to find those errors only which are taking place more than one time consecutively.
In that above table error “Loyality host offline for Server 5” and “Loyality host offline for Server 2” have taken place more than one time consecutively that we will consider but “Authentication error for Server 1” we will not consider, because it just occurred once. But if you see later one again “Authentication error for Server 1” occurred three times consecutively there we will consider it.

Method 1:

index="test_index" sourcetype="testlog_new"
| table error _time
| streamstats current=f window=1 max(error) as Previous_error
| eval isConsecutive = if (error == Previous_error, 1, 0)
| streamstats count as count by error reset_before=(isConsecutive==0)
| streamstats count(eval(isConsecutive==0)) as transaction_id
| stats max(count) as Consecutive by error, transaction_id
| sort transaction_id
| fields - transaction_id
| where Consecutive > 1

3

Result:

4
Explanation:
We took the data from index “test_index” and sourcetype “testlog_new”. After that using table command, we took two fields “error” and “_time” in tabular view. Then using the “streamstats” command we listed all the previous errors with respect to the “error” field. Then using the “eval” command we created a field called “isConsecutive”, which is based on the condition, if the error is the same as “Previous_error” then it will return “1” or else “0”. After that, we created another field called  “count” using the “streamstats” command with respect to the “error” field, i.e. it will provide a consecutive count for the same error and count will reset on each unique error. After that, we created a field called “transaction_id” which will do the same as the previous step but here count will not reset on each unique error, it will hold the previous count record. In the next step using the “stats” command we took all maximum count respect to “error” and “transaction_id”. Then we sorted the “transaction_id” field and excluded the “transaction_id” field using the “fields” command. Finally, we filtered where “Consecutive” is greater than 1.

You can also know about :  How to Add Custom Emoji in Splunk Dashboard

This how you will get the list of error which has occurred more than once.

Method 2:

index="test_index" sourcetype="testlog_new"
| table error _time
| streamstats count as count by error reset_on_change=true
| stats max(count) as Consecutive_Count by error
| where Consecutive_Count >= 2

5

Result:

6
Explanation:
We took the data from index “test_index” and sourcetype “testlog_new”. After that using table command, we took two fields “error” and “_time” in tabular view. Then using “streamstats” command we took counts with respect to the “error” field, where reset_on_change=true i.e. count will reset on each change in “error” otherwise it will continue counting with the same consecutive error. In the next step using the “stats” command, we took all maximum count respect to “error”. Finally, we filtered where “Consecutive_Count” is greater than equal to 2.

This is all about “Find Out The Errors Occurring 2 Or More Times Consecutively”.

Happy Splunking!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.