Index time Vs Search time Processing

Index time Vs Search time Processing

Splunk Enterprise terms “index time” and “search time” distinguish between the ways of processing that occur during indexing and when search operations are being performed.

Index time: It is the time period from when Splunk receives new data to when the data is written to a Splunk index. Inbetween this time, the data is parsed into segments and events, default fields and timestamps are extracted, and transforms are applied.

Search time: It takes place when you search through data, Splunk creates those fields when compiling search results and does not store them in the index. When you are running a search, several operations are performed by Splunk to derive various knowledge objects and apply them to the events returned by the search. These knowledge objects include extracted fields, calculated fields etc.

Knowing about these terms becomes more important while administering Splunk Enterprise. For instance, assume that you are planning to use custom meta-data such as host and source type, then you should define that meta-data before you start indexing, so that the indexing process can tag related events with them. Once indexing has been done, you cannot change the meta-data assignments.

If you wish to apply your custom meta-data to the already indexed data, you can choose either to re-index the data, in order to apply the custom meta-data to the existing data, as well as to new data, or as an alternative, you can manage the issue at search time by tagging the events with alternate values.

Index time and search time Extraction

When Splunk is indexing data, it parses the data stream into a series of events, as a part of the processing, it also adds a number of fields to the event data. These fields comprise of the default fields that it adds automatically and any custom fields that you had specified.

The process of adding fields to events is known as field extraction. There are two types of field extraction:

  • Index-time field extraction: These fields are stored in the index and become part of the event data.
  • Search-time field extraction: This takes place when you search through your indexed data. Splunk creates those fields when compiling search results and does not store them in the index.

There are two types of indexed fields:

  • Default fields: These are the ones which Splunk automatically adds to each event.
  • Custom fields: The ones which you have specified.

NOTE: While working with fields, consider that most machine data either are not structured or have a structure that changes constantly. For this type of data, use search-time field extraction for maximum flexibility and reliability. Search-time field extractions can be easily modified even after you have defined it. The general rule as recommended by Splunk, it is better to perform most knowledge-building activities, such as field extraction, at search time. Index-time custom field extractions can cost performance at both index time and search time. Whenever a new field is added to the number of fields extracted during indexing, the indexing process slows, also search operations on that index become slower, due to the increased additional fields. To avoid such performance issues consider relying upon search-time field extractions whenever possible.

Types of field extraction

Splunk offers three field extraction types, namely inline, transform, and automatic key-value.

                        Extraction Type             Configuration location
                     Inline extractions
Inline extractions have 
EXTRACT- configurations in 
props.conf stanzas.
                Transform extractions
These extractions have REPORT-
 name configurations defined
in props.conf stanzas.
Their props.conf configurations 
must reference field transform
stanzas in transforms.conf.
        Automatic key-value extractions
Automatic key-value extractions
need to be configured in 
props.conf stanzas where KV_MODE
is set to a valid value except
none.

Below is the list of processes carried out at Index time and  Search time by Splunk.

Index-time Processes

These processes are executed between the point when the data is consumed from the source and the point when it is written to the disk(on indexer).

Following are the processes that occur during Index time:

>>Meta-data/default field extraction (such as host, source, source type,
and timestamp)

>>Static or dynamic host assignment for specific inputs
>>Default host assignment overrides
>>Source type customization
>>Custom index-time field extraction
>>Structured data field extraction
>>Event timestamping
>>Event line-breaking
>>Event segmentation (also occurs at search time)

Search-time Processes

Search-time processing happens while a search operation is going on, as events are gathered by the search.

Following are the processes that occur at search time:

>>Event segmentation (also occurs at index time)
>>Event type matching
>>Search-time field extraction (automatic and custom field extractions,
including multivalue fields and calculated fields)

>>Field aliasing
>>Addition of fields from lookups
>>Source type renaming
>>Tagging

Hope! you had a good time reading this post.
Thanks,
Happy Splunking!!
Advertisements

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.