Bucket Rolling Criteria In Splunk

Bucket Rolling Criteria In Splunk

Hi guys !!

We have been telling/discussing one thing very repeatedly “data is getting indexed in the indexer” OR “lets fetch the data from this index” OR “Why my data is taking too much time to fetch” OR “Lets create the index in the indexer to index the data coming from the Application servers”,etc.

But have we ever thought of knowing the real concept lying behind the scene ? How data gets indexed ? What happens when data reaches to the Indexers ? That is the reason today we have come up with a new topic in Splunk called “BUCKET”.

Bucket is usually an unit of directory structure in the file system which is created by itself at the time of indexing .When new data comes from the application servers it gets stored/indexed in the Indexer in the form of the Bucket. Basically there are 4 bucket stages in Splunk which are as follows :

1. Hot 
2. Warm 
3. Cold
4. Frozen

Buckets are stored in “$SPLUNK_HOME/var/lib/splunk” directory in the indexer component of Splunk. It gets created as soon as data gets indexed in the indexer.

Today we will show you the bucket rolling criteria in Splunk.

HOT BUCKET :

While indexing the data, buckets get created. It is called HOT state means data stored in HOT bucket. It is writable as well as readable at the same time. The data which is currently written to the indexer will get stored in the HOT Bucket and at the same time it can be fetched through the Search Head if any end-users are trying to access the data stored in it.

PATH : $SPLUNK_HOME/var/lib/splunk/<index_name>/db/

ROLLING CRITERIA ( HOT TO WARM ) :

Rolling criteria(s) from HOT bucket to WARM bucket are listed below : 

. When Splunk gets restarted
. When HOT bucket is full
    ( Maximum size of the data 10 GB for 64-bit system ) and 
    ( 750 MB for 32-bit system )
. After a certain period of time(maxHotSpanSecs = 90 days in secs)
. When maximum HOT buckets limit cross (maxHotBuckets = 3/index)
. When HOT bucket has not received data for a long time.

WARM BUCKET :

Once data reaches to the WARM bucket, it is read only. Active incoming data will not get written to the WARM bucket. It is created in the same directory as HOT Bucket

By Default the no. of WARM bucket is 0
When data rolls up from HOT to WARM, WARM bucket gets created
We can create maximum of 300 WARM buckets

PATH : $SPLUNK_HOME/var/lib/splunk/<index_name>/db/

ROLLING CRITERIA ( WARM TO COLD ) :

Rolling criteria(s) from WARM bucket to COLD bucket are listed bellow : 

. If warm bucket(s) cross the maximum storage limit (Default 500000 MB)
. If default max no. of warm bucket increases (Max 300)

COLD BUCKET :

Once data reaches to the COLD bucket, it is also READ-ONLY like WARM bucket. Since this bucket is very less accessible it can be kept in a low cost disk storage OR in a storage with very less expensive disk unlike HOT and WARM bucket.

After rolling up data from WARM bucket, it comes to COLD bucket.

PATH : $SPLUNK_HOME/var/lib/splunk/<index_name>/colddb/

ROLLING CRITERIA ( COLD TO FROZEN ) :

Rolling criteria(s) from COLD bucket to FROZEN bucket are listed bellow : 

. If default max size of the cold bucket(s) cross (Max default size 500000 MB)
. If frozenTimePeriodInSecs crosses ( Default 6 years)

FROZEN BUCKET :

After rolling up data from COLD bucket, it comes to FROZEN bucket where data gets deleted  by default. If we want we can archive the data in a specific location.

Once data is archived , we can bring it back to the THAWED bucket and rebuild it for re-indexing which does not effect the license at all. Under the directory “$SPLUNK_HOME/var/lib/splunk” we will not have any bucket called “FROZEN” because by default data comes to the FROZEN bucket will be deleted.

Hope you have understood the simple and very important concept of Bucket Rolling Criteria In Splunk.

Happy Splunking !!