Timestamp recognition of dates with two-digit years fails beginning January 1, 2020 in Splunk
Disclaimer: This is our attempt to consolidate the whole release note from Splunk to make it simpler, in case, you are interested to learn about this issue in detail, please have a look at the official release notes here.
About the issue
Starting from 1st Jan, 2020, all the un-patched Splunk instances will be unable to recognize timestamps from events where the date contains a two-digit year. It is also known as 2020 Timestamp Issue in Splunk.
Also Starting from 13th September, 2020 at 12:26:39 PM UTC, all the un-patched Splunk instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.
Note: This issue appears when you have configured Splunk to determine
timestamps automatically, and can result in problems related to
Splunk functionality that depend on the event time stamps such
as incorrect search results, data retention etc. Also the data
that has been indexed with a wrong timestamp can’t be corrected
the only workaround is to re-ingest this data.
The Splunk platform input processor uses a file called datetime.xml to help the processor correctly determine timestamps based on incoming data. The file uses regular expressions to extract many different types of dates and timestamps from incoming data..
We have Splunk 7.2.0 (unpatched, obviously), uploaded a file named sample_events.txt which has some sample events (copied from Splunk release notes) for test.
As you can observe in the screenshot, Splunk is unable to recognize the timestamp for the 2nd and 3rd event, also displays a warning symbol on the left hand side.
So, how to fix this!!
Splunk offers the below four solutions to this problem for on-premises deployments of Splunk. You can choose to proceed with any one of the below.
-
Download and deploy an app to temporarily replace the defective datetime.xml with the fixed file (temporary solution, suitable for large deployments) -
Download an updated version of datetime.xml and apply it to each of your Splunk platform instances -
Upgrade Splunk platform instances to a version with an updated version of datetime.xml -
Make modifications to the existing datetime.xml file on your Splunk platform instances
Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file
This solution should be considered for companies with large deployments.
You do not need to stop the Splunk platform before you deploy the apps, but you must restart each instance that receives the apps.
To ensure that you have no problems with universal forwarders and timestamp recognition, deploy this fix on all universal forwarders, even those that do not specifically suffer from this problem.
-
Download the xml fix app archive (MD5 hash: 2e6b7520fa1379d72ac80ca21a54d45a) from splunk.com. -
Unarchive the .tgz file to a location that is accessible from all of your Splunk platform instances. -
Open the README file and follow the instructions to deploy one of the apps to each Splunk platform instance.
Note: This should be treated as a temporary solution only, try
to upgrade to the patched version as early as possible.
Download an updated version of datetime.xml and apply it to each of your Splunk platform instances
This option is suitable for customers who cannot upgrade right away to a version of the Splunk platform with the fixed file, or who run an unsupported version that is lower than 6.6.x.
-
Download the zip timestamp recognition ZIP file (MD5 hash: 00dfc319e89001fa16d6725dbf042234) from splunk.com. -
Unarchive the ZIP file to a location that is accessible from all of your Splunk platform instances. -
On each Splunk platform instance, do the following: a) Copy the updated datetime.xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file. b) Confirm that the new datetime.xml has been written to the $SPLUNK_HOME/etc c) Restart the Splunk platform. Your Splunk platform instance is now patched.
Upgrade Splunk platform instances to a version with an updated version of datetime.xml
You can fix this issue by installing/upgrading with the Splunk package that contains the updated/fixed datetime.xml file. Below is the table for your reference from Splunk.
| Major version | Minor version with patched file | Released? |
| 6.6 | 6.6.12.1 | Yes |
| 7.0 | 7.0.13.1 (versions 7.0.12 and 7.0.13 are Splunk Cloud-only releases) |
Yes |
| 7.1 | 7.1.10 | Yes |
| 7.2 | 7.2.9.1 | Yes |
| 7.3 | 7.3.3 | Yes |
| 8.0 | 8.0.1 | Yes |
Make modifications to the existing datetime.xml file on your Splunk instances
You can edit the datetime.xml file present at $SPLUNK_HOME/etc directiry on your Splunk instances.
Search and replace the string shown in the table from Splunk below:
| Search for this string | Replace with this string |
| <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text> | <text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text> |
| <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text> | <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text> |
| <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text> | <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text> |
| <text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text> | <text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text> |
Save and close the file, then restart your Splunk Instance.
NOTE: Please be careful while editing this file, as this
file is responsible for all the timestamp extractions
that Splunk does automatically, even a minor mistake
can mess up event time-stamping.
Validate timestamp extraction after an update
After you have implemented any one of the above four solutions, the issue should be fixed, we followed the second solution to replace the datetime.xml file on my Splunk instance.
Let’s check whether it worked for me or not, I uploaded the same sample_events.txt file to our Splunk, and the picture below depicts how it went ,
- Splunk was able to recognize the timestamp from first two events, for the third event to be recognized, since it has a future timestamp we need to use a attribute MAX_DAYS_HENCE, the value of which we have set to 500.
- Got the file integrity error on my Splunk web, since we have replaced a file that ships with Splunk, yes, we are talking about datetime.xml.
Note: You are likely to get file integrity errors, if you chose
to replace/edit the datetime.xml file, After you upgrade
to a version that has the fixed datetime.xml, this message
should no longer appear on your Splunk instance.
Hope you have understood how to solve the timestamp issue in Splunk.
Happy Splunking !!
