INGEST_EVAL

INGEST_EVAL

Hi guys,

We all know that at the time of indexing when the data is getting  stored  into indexers , Splunk software parses the data stream into a series of events. Now ,we can perform different actions on those events. For parsing and filtering we use two configuration files i.e. props.conf and transforms.conf  in Heavy Forwarders.

You  all know about  eval command .But you can also perform eval command just by an attribute named INGEST_EVAL at index time.

Below is the sample data on which we are going to perform the parsing,

TODAY I AM GOING TO SHOW YOU
HOW TO USE THE ATTRIBUTE
INGEST_EVAL

Step:1

First,you have to go to the  location where you want save the sample data and there you have to create a file where you want to save your data.

Here,I have created one file called data.txt in /tmp location.You can use any other location or any existing file for storing you data.

Step:2

In the next step we will configure inputs.conf .You will find the inputs.conf  in the following path

$SPLUNK_HOME$/etc/system/local.

 Now,here I will give the absolute path of  sample.txt ,  index name and mention the metadata(host,source,sourcetype)[but it is not mandatory to define metadata].

Here,I have specified index =example

Step:3

Now we will configure the props.conf in . You can find the props.conf in following path

 $SPLUNK_HOME$/etc/system/local
[sample]
TRANSFORMS-abc=text

As you can see ,I have specified here the sourcetype  sample in stanza here in the props.conf.

Now, the second attribute is TRANSFORMS-abc=text(the general format is TRANSFORMS-<class_name>-<unique_stanza_name>. So, here the mentioned class name is abc(you can give any string) and the unique_stanza_name is text(you can give any string).Now, the stanza_name you have to specify in transforms.conf. Lets see how in the next step.

Step:4

Here the transforms.conf will be configured . You can find the transforms.conf in following path,

$SPLUNK_HOME$/etc/system/local

Example:1

[text]
INGEST_EVAL = len=length(_raw)

First,I have mentioned here the transformation name in stanza I.e. [text]

Now, we all know that in search time ‘| eval’ command can use ‘length’ function to get the length of the event.

But now if I want get the same thing  in index time I will use the second attribute

I.e. INGEST_EVAL = len=length(_raw)

So,one new field will get indexed named ‘len’ where the length of ‘_raw’ will get stored.

Step:5

Now, I will configure fields.conf. You can find the fields.conf in following path,

$SPLUNK_HOME$/etc/system/local
[len]
INDEXED=true

In fields.conf we will give the field which I want to get indexed in stanza.

The second attribute INDEXED=true  will index the field.

Step:6

After configuring configuration files you always should restart splunk in HF,so that all the changes will be will be updated.

Step:7

After restarting splunk you just have to go to location of sample.txt and the use the command [vi data.txt]

and write the sample data into it.

Step:8

As,you can see when index=”example” is written in the search box  one field is created named .

Now, you can see in “len” field the length of the sample data  is stored i.e. 69.

Step:1 to Step:3 are same as before.

Step:4

Here the transforms.conf will be configured. You can find the transforms.conf in following path,

$SPLUNK_HOME$/etc/system/local

Example:2

[text]
INGEST_EVAL = low=lower(_raw)

First, I have mentioned here the transformation name in stanza I.e. [text]

Now, we all know that in search time ‘| eval’ command can use ‘lower’ function to make upper case strings in lower case strings  in  the events.

But now if I want get the same thing  in index time I will use the second attribute

I.e. INGEST_EVAL = low=lower(_raw)

So, one new field will get indexed named ‘low’ where the events of  ‘_raw’ will be lower cased

Step 5:

Now, I will configure fields.conf. You can find the fields.conf in following path,

$SPLUNK_HOME$/etc/system/local
[low]

INDEXED=true

In fields.conf we will give the field which I want to get indexed in stanza.

The second attribute INDEXED=true  will index the field.

Step:6 and Step:7 will be same as before.

Step:8

As,you can see when index=”example” is written in the search box  one field is created named “low”.

As, you can see the filed low is created in which the sample data gets stored in lower case.

Step:1 to Step:3  will be same as before.

Step:4

Here the transforms.conf will be configured. You can find the transforms.conf in following path,

$SPLUNK_HOME$/etc/system/local

Example:3

[text]

INGEST_EVAL = list=if(length(_raw)>10,"nullqueue"," ")

First, I have mentioned here the transformation name in stanza i.e. [text]

Now, we all know that in search time ‘| eval’ command can use ‘if’ function

So,the if function also can be used with INGEST_EVAL at index time.

So, one new field will get indexed named ‘list’.Now,if the length of the raw data will be greater than 10 then the value “nullqueue” will get stored in “list” field else null  value will get stored in “list” field.

Step:5 will be same as before.

Step:6

Now, I will configure fields.conf. You can find the fields.conf in following path,

$SPLUNK_HOME$/etc/system/local
[queue]
INDEXED=true

In fields.conf we will give the field which I want to get indexed in stanza.

The second attribute INDEXED=true  will index the field.

Step:6 and Step:7 will be same as before.

Step:8

As,you can see when index=”example” is written in the search box  one field is created named “list”.

So,you can see as the length of the sample data is greater than 10 that’s why nullqueue is stored in “list” field. You can use all functions of eval command with INGEST_EVAL attribute.

Hope, this has helped you in achieving the below requirement without fail:

INGEST_EVAL

Happy Splunking  !!

Advertisements

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.