Spread our blog

Timestamp recognition of dates with two-digit years fails beginning January 1, 2020 in Splunk

Disclaimer: This is our attempt to consolidate the whole release note from Splunk to make it simpler, in case, you are interested to learn about this issue in detail, please have a look at the official release notes here.

About the issue 

Starting from 1st Jan, 2020, all the un-patched Splunk  instances will be unable to recognize timestamps from events where the date contains a two-digit year. It is also known as 2020 Timestamp Issue in Splunk.

Also Starting from 13th September, 2020 at 12:26:39 PM UTC, all the un-patched Splunk instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.

Note: This issue appears when you have configured Splunk to determine
      timestamps automatically, and can result in problems related to
      Splunk functionality that depend on the event time stamps such 
      as incorrect search results, data retention etc. Also the data 
      that has been indexed with a wrong timestamp can’t be corrected
      the only workaround is to re-ingest this data.

The Splunk platform input processor uses a file called datetime.xml to help the processor correctly determine timestamps based on incoming data. The file uses regular expressions to extract many different types of dates and timestamps from incoming data..

We have Splunk 7.2.0 (unpatched, obviously), uploaded a file named sample_events.txt which has some sample events (copied from Splunk release notes) for test.

Screenshot (21)

As you can observe in the  screenshot, Splunk is unable to recognize the timestamp for the 2nd and  3rd event, also displays a warning symbol on the left hand side.

So, how to fix this!!

Splunk offers the below four solutions to this problem for on-premises deployments of Splunk. You can choose to proceed with any one of the below.

  • Download and deploy an app to temporarily replace the defective datetime.xml 
    with the fixed file (temporary solution, suitable for 
    large deployments)
  • Download an updated version of datetime.xml and apply it to
    each of your Splunk platform instances
  • Upgrade Splunk platform instances to a version with an updated
    version of datetime.xml
  • Make modifications to the existing datetime.xml file on your
    Splunk platform instances

Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file

This solution should be considered for companies with large deployments.

You do not need to stop the Splunk platform before you deploy the apps, but you must restart each instance that receives the apps.

To ensure that you have no problems with universal forwarders and timestamp recognition, deploy this fix on all universal forwarders, even those that do not specifically suffer from this problem.

  1. Download the xml fix app archive
    (MD5 hash: 2e6b7520fa1379d72ac80ca21a54d45a) from 
    splunk.com.
  2. Unarchive the .tgz file to a location that is accessible
    from all of your Splunk platform instances.
  3. Open the README file and follow the instructions to deploy
    one of the apps to each Splunk platform instance.
Note: This should be treated as a temporary solution only, try
      to upgrade to the patched version as early as possible.

Download an updated version of datetime.xml and apply it to each of your Splunk platform instances

This option is suitable for customers who cannot upgrade right away to a version of the Splunk platform with the fixed file, or who run an unsupported version that is lower than 6.6.x.

  1. Download the zip timestamp recognition ZIP file
    (MD5 hash: 00dfc319e89001fa16d6725dbf042234) from 
     splunk.com.
  2. Unarchive the ZIP file to a location that is accessible
    from all of your Splunk platform instances.
  3. On each Splunk platform instance, do the following:
    a) Copy the updated datetime.xml from the location where
       you downloaded it to the $SPLUNK_HOME/etc directory on
       the Splunk platform instance. Ensure that the updated
       file overwrites the existing file.
    b) Confirm that the new datetime.xml has been written
        to the $SPLUNK_HOME/etc
    c) Restart the Splunk platform. Your Splunk platform
       instance is now patched.

Upgrade Splunk platform instances to a version with an updated version of datetime.xml

You can fix this issue by installing/upgrading with the Splunk package that contains  the updated/fixed datetime.xml file. Below is the table for your reference from Splunk.

Major version Minor version with patched file Released?
6.6 6.6.12.1 Yes
7.0 7.0.13.1
(versions 7.0.12 and 7.0.13 are Splunk Cloud-only releases)
Yes
7.1 7.1.10 Yes
7.2 7.2.9.1 Yes
7.3 7.3.3 Yes
8.0 8.0.1 Yes

 Make modifications to the existing datetime.xml file on your Splunk instances

You can edit the datetime.xml file present at $SPLUNK_HOME/etc directiry on your Splunk instances.

Search and replace the string shown in the table from Splunk below:

Search for this string Replace with this string
<text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text> <text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text>
<text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text> <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
<text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text> <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text>
<text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text> <text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>

Save and close the file, then restart your Splunk Instance.

NOTE: Please be careful while editing this file, as this 
      file is responsible for all the timestamp extractions
      that Splunk does automatically, even a minor mistake
      can mess up event time-stamping. 

 Validate timestamp extraction after an update

After you have implemented any one of the above four solutions, the issue should be fixed, we followed the second solution to replace the datetime.xml file on my Splunk instance.

Let’s check whether it worked for me or not, I uploaded the same sample_events.txt file to our Splunk, and the picture below depicts how it went ,

Screenshot (22)

  • Splunk was able to recognize the timestamp from first two events, for the third event to be recognized, since it has a future timestamp we need to use a attribute MAX_DAYS_HENCE, the value of which we have set to 500.
  • Got the file integrity error on my Splunk web, since we have replaced a file that ships with Splunk, yes, we are talking about datetime.xml.
Note: You are likely to get file integrity errors, if you chose
to replace/edit the datetime.xml file, After you upgrade 
to a version that has the fixed datetime.xml, this message
should no longer appear on your Splunk instance. 


Hope you have understood how to solve the timestamp issue in Splunk.

Happy Splunking !!

You can also know about :  BREAK_ONLY_BEFORE
What’s your Reaction?
+1
+1
+1
+1
+1
+1
+1

Spread our blog

LEAVE A REPLY

Please enter your comment!
Please enter your name here