INGEST_EVAL
Hi guys,
We all know that at the time of indexing when the data is getting stored into indexers , Splunk software parses the data stream into a series of events. Now ,we can perform different actions on those events. For parsing and filtering we use two configuration files i.e. props.conf and transforms.conf in Heavy Forwarders.
You all know about eval command .But you can also perform eval command just by an attribute named INGEST_EVAL at index time.
Below is the sample data on which we are going to perform the parsing,
TODAY I AM GOING TO SHOW YOU HOW TO USE THE ATTRIBUTE INGEST_EVAL
Step:1
First,you have to go to the location where you want save the sample data and there you have to create a file where you want to save your data.
Here,I have created one file called data.txt in /tmp location.You can use any other location or any existing file for storing you data.
Step:2
In the next step we will configure inputs.conf .You will find the inputs.conf in the following path
$SPLUNK_HOME$/etc/system/local.
Now,here I will give the absolute path of sample.txt , index name and mention the metadata(host,source,sourcetype)[but it is not mandatory to define metadata].
Here,I have specified index =example
Step:3
Now we will configure the props.conf in . You can find the props.conf in following path
$SPLUNK_HOME$/etc/system/local
[sample] TRANSFORMS-abc=text
As you can see ,I have specified here the sourcetype sample in stanza here in the props.conf.
Now, the second attribute is TRANSFORMS-abc=text(the general format is TRANSFORMS-<class_name>-<unique_stanza_name>. So, here the mentioned class name is abc(you can give any string) and the unique_stanza_name is text(you can give any string).Now, the stanza_name you have to specify in transforms.conf. Lets see how in the next step.
Step:4
Here the transforms.conf will be configured . You can find the transforms.conf in following path,
$SPLUNK_HOME$/etc/system/local
Example:1
[text] INGEST_EVAL = len=length(_raw)
First,I have mentioned here the transformation name in stanza I.e. [text]
Now, we all know that in search time ‘| eval’ command can use ‘length’ function to get the length of the event.
But now if I want get the same thing in index time I will use the second attribute
I.e. INGEST_EVAL = len=length(_raw)
So,one new field will get indexed named ‘len’ where the length of ‘_raw’ will get stored.
Step:5
Now, I will configure fields.conf. You can find the fields.conf in following path,
$SPLUNK_HOME$/etc/system/local
[len] INDEXED=true
In fields.conf we will give the field which I want to get indexed in stanza.
The second attribute INDEXED=true will index the field.
Step:6
After configuring configuration files you always should restart splunk in HF,so that all the changes will be will be updated.
Step:7
After restarting splunk you just have to go to location of sample.txt and the use the command [vi data.txt]
and write the sample data into it.
Step:8
As,you can see when index=”example” is written in the search box one field is created named .
Now, you can see in “len” field the length of the sample data is stored i.e. 69.
Step:1 to Step:3 are same as before.
Step:4
Here the transforms.conf will be configured. You can find the transforms.conf in following path,
$SPLUNK_HOME$/etc/system/local
Example:2
[text] INGEST_EVAL = low=lower(_raw)
First, I have mentioned here the transformation name in stanza I.e. [text]
Now, we all know that in search time ‘| eval’ command can use ‘lower’ function to make upper case strings in lower case strings in the events.
But now if I want get the same thing in index time I will use the second attribute
I.e. INGEST_EVAL = low=lower(_raw)
So, one new field will get indexed named ‘low’ where the events of ‘_raw’ will be lower cased
Step 5:
Now, I will configure fields.conf. You can find the fields.conf in following path,
$SPLUNK_HOME$/etc/system/local
[low] INDEXED=true
In fields.conf we will give the field which I want to get indexed in stanza.
The second attribute INDEXED=true will index the field.
Step:6 and Step:7 will be same as before.
Step:8
As,you can see when index=”example” is written in the search box one field is created named “low”.
As, you can see the filed low is created in which the sample data gets stored in lower case.
Step:1 to Step:3 will be same as before.
Step:4
Here the transforms.conf will be configured. You can find the transforms.conf in following path,
$SPLUNK_HOME$/etc/system/local
Example:3
[text] INGEST_EVAL = list=if(length(_raw)>10,"nullqueue"," ")
First, I have mentioned here the transformation name in stanza i.e. [text]
Now, we all know that in search time ‘| eval’ command can use ‘if’ function
So,the if function also can be used with INGEST_EVAL at index time.
So, one new field will get indexed named ‘list’.Now,if the length of the raw data will be greater than 10 then the value “nullqueue” will get stored in “list” field else null value will get stored in “list” field.
Step:5 will be same as before.
Step:6
Now, I will configure fields.conf. You can find the fields.conf in following path,
$SPLUNK_HOME$/etc/system/local
[queue] INDEXED=true
In fields.conf we will give the field which I want to get indexed in stanza.
The second attribute INDEXED=true will index the field.
Step:6 and Step:7 will be same as before.
Step:8
As,you can see when index=”example” is written in the search box one field is created named “list”.
So,you can see as the length of the sample data is greater than 10 that’s why nullqueue is stored in “list” field. You can use all functions of eval command with INGEST_EVAL attribute.
Hope, this has helped you in achieving the below requirement without fail:
INGEST_EVAL
Happy Splunking !!
