Hello Guys, Let’s talk about some SPL ( Splunk Search Processing Language ) tricks to find out how one can schedule the report if the data is not coming to the index in the last 7 days i.e., list down the name of the index(es) for which there is no ingestion in the last 7 days.
Example 1:
| rest /services/data/indexes count=0
| stats max(maxTime) as updated by title
| eval Data7daysbefore=relative_time(now(),"-7d@d")
| eval HRFofData7daysbefore=strftime(Data7daysbefore,"%c")
| eval Epochupdated=strptime(updated,"%Y-%m-%dT%H:%M:%S+%z")
| eval Result=if(Epochupdated < Data7daysbefore,"Data not coming in the past 7 Days","Data is available")
| fields title,updated,Result
| search NOT Result="Data is available"
| rename title as Index
Now, you can set up an alert giving a condition as “No Of Results > 0“, if this condition matches, an alert will be triggered.
Example 2:
Now, let’s say the requirement is after the third alert is triggered, we want to disable the alert because we don’t want these many alerts to be there in the mail and also it implies that no one has taken any action against the “no data ingestion”.
| rest /services/data/indexes count=0
| stats max(maxTime) as updated by title
| eval Data7daysbefore=relative_time(now(),"-7d@d")
| eval HRFofData7daysbefore=strftime(Data7daysbefore,"%c")
| eval Epochupdated=strptime(updated,"%Y-%m-%dT%H:%M:%S+%z")
| eval Result=if(Epochupdated < Data7daysbefore,"Data not coming in the past 7 Days","Data is available")
| fields title,updated,Result
| search NOT Result="Data is available"
| rename title as Index
| appendcols
[| rest services/alerts/fired_alerts
| fields eai:acl.owner,title,triggered_alert_count splunk_server
| search title="Checking"
| fields triggered_alert_count ]
Now, create an alert and give the condition as “search triggered_alert_count < 3” in the “Custom” Trigger Conditions
Note:- The alert will be triggered 3 times, after that there will not be any notification sent to the recipient
Example 3:
There is one more way of achieving the same thing using “tstats” command in Splunk but it is slower than the “rest” command in Splunk. I would prefer to go with the “rest” command if the data volume is more.
Make sure you select the right time from the “time range picker” dropdown. I have chosen “All time” to show all indexes
| tstats latest(_time) as LT where index=* OR index=_* by index
| eval Epoch1daybefore=relative_time(now(),"-1d")
| eval Result=if(LT < Epoch1daybefore,"Data is not available","Data is available")
| search NOT Result="Data is available"
| eval HRLT=strftime(LT,"%c"),HR24hours=strftime(Epoch1daybefore,"%c")
| rename index as Index
| fields Index,HRLT,Result
I hope you all enjoyed the above queries and looking for something similar to this. Stay tuned.
Happy Splunking !!
