Re-index your data into Splunk

Sometimes, due to some unavoidable reasons data loss may occur while indexing or partial indexing may take place, then you might want to re-index all your data again into Splunk. Following are the techniques to re-index your data: 

If you want to clean your existing data from any of your index before going for re-indexing process, use the commands below.

 i)  # cd $SPLUNK_HOME/bin 
ii) # ./splunk stop 
iii) # ./splunk clean eventdata -index  <index-name>
( to clean All Indexes, just drop off -index <index-name> )
iv) # ./splunk start

PROCESS 1: Remove/delete the fishbucket sub-directory which should re-index  all your data in all of your indexes.

CAUTION : Deleting the fishbucket sub-directory will re-index data coming into all your indexes from that splunk forwarder/instance, thus may severely impact your license usage.

To delete/remove the fishbucket:

1. Move to the directory   /opt/splunk/var/lib/splunk  (on the instance forwarding data)

2. Delete/Remove the sub-directory fishbucket 

See the pictures below for further reference,

–> index contents before deleting the fishbucket

–> deleting/removing the fish bucket

i)  #cd $SPLUNK_HOME/var/lib/splunk 
ii) #rm -rf fishbucket                                                   

–> restart your splunk instance ($SPLUNK_HOME/bin/splunk restart)

Now, as soon as your files are updated on the application server, the whole contents of your files will be re-indexed into splunk in their corresponding indexes.

PROCESS 2: Re-index data without deleting the fishbucket/re-index contents of any specific file 

There may be situations, when you only want to re-index the data for a particular file, then you can use the command given below to reset btprobe (run the command on the splunk instance forwarding data)

btprobe: It queries the fishbucket for checkpoints stored by monitor inputs. Any changes you make to the fishbucket using btprobe takes effect only after a restart. 

CAUTION: You must stop your splunk  instance before using btprobe.

i)  #cd $SPLUNK_HOME/bin 
ii) #./splunk stop 
iii)#./splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db 
--file <File Path> --reset 
iv) #./splunk start

Below are the screenshots for your reference (using the same index for better understanding ) ….

–> index contents before resetting btprobe

–> resetting the btprobe

–> This should re-index the contents of your file 

PROCESS 3: Re-index your file contents based on timestamp for which data has not been indexed in splunk

Suppose due to some reason data coming from the file went missing/didn’t get indexed for the timestamps 09/29/18 5:05:XX  to 09/29/18 10:12:XX , but after that the indexing process is working normally, then

1) Copy the file contents which haven’t been indexed ( from timestamp 09/29/18        5:05:XX  to 09/29/18 10:12:XX ) to a temporary file, say tmp_file.txt

2) Create a new input stanza in “inputs.conf ” for tmp_file.txt

 [monitor://absolute path of tmp_file.txt] 
 index =  your_index_name 
 sourcetype = source_type_name

3)   Restart your splunk instance

Congrats!!  NOW, You have the data indexed that was missing from splunk previously.

NOTE: For cases, when you don’t want to Re-index the contents of some specific file(s), even when you delete the fishbucket, visit our post followTail attribute in splunk

Thanks For Reading!! More contents coming soon, subscribe if you haven’t yet, to keep yourself updated with the latest posts on this blog!!!

Happy Splunking!!