Real-time Vs Historical searches & Reports

Splunk supports real-time and historical both types of searches, and we in this post are going to throw some light upon these splunk search Time-frames. 

About real-time searches and reports

When you go for real-time searches and reports, you can search events before they get indexed  into Splunk and preview reports as the events pour in.

When using real-time searches and reports,

• You have an option to design alerts based on real-time searches that keep running continuously in the background. These real-time alerts can provide timelier notifications than alerts that are based on scheduled reports.
• You can even use real-time search results and reports in dashboards.

NOTE: Increased number of concurrent real-time searches can greatly affect the indexing performance of your splunk instance(s). To overcome this limitation & negative performance impact on the indexer, you can enable indexed real-time searches. Splunk by default, allows only users with the Admin role to run and save real-time searches.

How Real-time search works?

Splunk Real-time searches scan incoming events for indexing. The scan looks for events that contain index-time fields that indicate the event could be a match for your search.

The number of matching events can fluctuate up or down over time as the search discovers matching events at a faster or slower rate. Since, when the real-time search runs, splunk to find actual events periodically evaluates your search criteria within the sliding time range window that you have defined for the search.

Here is an example of a real-time search with a one minute time range window for your reference. At the point that the following screenshot was taken, the search had scanned a total of 298 events since it was launched. The matching event count of 218 represents the number of events matching the search criteria that were identified in the past minute. 

clearly, you can see the newest events appear on the right-hand side of the timeline. As time passes, the events move left until the events move off the left-hand side, disappearing from the time range window entirely. A real-time search should continue running until you or another user stops the search or deletes the search job. The real-time search should not “time out” for any other reason.

NOTE: Splunk Real-time searches have the advantage of all search functionalities, including advanced ones like transactions,lookups and so on. There are also some search commands that are to be used specifically in conjunction with real-time searches, like  streamstats.

Indexed real-time search

As mentioned earlier real-times searches can have a negative impact on performance. A solution to this is to enable indexed real-time search, which runs the searches like historical searches, but also continually updates the search with new events as the events appear on disk.

CAUTION: Use Indexed real-time search only when you don’t need up-to-the-second accuracy.

• Indexed real-time search can only be enabled by users with file system accesses, such as system administrators.

The sync delay lag time

Always remember, that the results returned by an indexed real-time search is going to lag behind a real-time search. Built into indexed real-time searches is a sync (synchronizing) delay. The sync delay is a precaution that ensures none of your data is missed.

Reasons data doesn’t  appear on disk in the order they are indexed:

• splunk uses multiple threads for indexing simultaneously
• sync delay ordering on your operating system

An indexed real-time remembers the latest indexed event that is returned for the current iteration of the time range window. That event is used as the start point for the next iteration of the time range window. In case sync delay is not imposed, some of the events before the latest event might not be searchable yet, and can be missed due to the continuously shifting time frame.

You can control the number of seconds of sync delay lag time with the setting indexed_realtime_disk_sync_delay = <int> in ‘limits.conf’. By default, this delay is set to 60 seconds.

About historical searches and reports

A Historical search has a distinct time range, such as the past hour, the previous day, or a time period between two dates. Historical searches are used to review data in the past, but can be set to review events with future-dated timestamps (depends upon the data in your index).

That’s all, thanks for reading.
Hope!! you enjoyed this post, leave a comment if you have any questions, further suggestions.

Happy Splunking!!