Important Announcement: Splunk Timestamp
recognition of dates with two-digit years fails
beginning January 1,2020

Splunk partner logo

26/11/19 | Grace Maher - Certified Splunk Consultant

The Problem:

As of January 1st, 2020, the following instances that remain unpatched will be unable to recognise timestamps that contain a two digit year, this means that data that has a two digit year in its timestamp will be indexed with the incorrect time:

  • Splunk Cloud
  • Splunk Enterprise Indexer
  • Splunk Enterprise Heavy Forwarder
  • Splunk Light
  • Splunk Universal Forwarders (Under Certain Conditions*)
  • Splunk Enterprise Search Heads (That forward their internal logs)
  • Splunk Enterprise Search Heads (That forward their internal logs)

Further to this, as of September 13th, 2020, the same unpatched instances as above will also be unable to recognise timestamps from events with dates that are based on Unix time.

* Splunk Universal Forwarders may be affected if they process data inputs prior to forwarding with the force_local_processing = true setting in props.conf

How is this caused?

The Splunk platform has an input processor which uses a file called datetime.xml to assist the processor in correctly determining the timestamps based on the incoming data. This file uses regular expressions to extract many different types of dates and timestamps from the incoming data.

If you are using an unpatched affected instance, the file supports the extraction of two-digit years of “19”, which is up to and including December 31st, 2019. Therefore on January 1st, 2020, these unpatched instances will mistakenly treat incoming data has having an invalid timestamp year, and could either add timestamps using the current year, or misinterpret the date incorrectly and add a misinterpreted date and time.

So what is the impact?

This issue affects all Splunk Cloud, Splunk Enterprise Indexers & Heavy Forwarders, and Splunk light instances no matter what OS they are deployed on.

It does not affect Universal (previously called light) Forwarders.

The problem will appear when you have configured the input source to automatically determine the timestamp, and will result in one of the following problems:

  • Incorrect timestamping of incoming data
  • Incorrect rollover of data buckets due to the incorrect timestamp
  • Incorrect retention of data overall due to the incorrect timestamp
  • Incorrect search results due to data ingested with the incorrect timestamp.
Resolution Options

Upgrade Universal Forwarders with Deployment App:

If you wish to upgrade your universal forwarders using a deployment server app, please see our guide here and app download.

This app can be deployed via the deployment server to your Universal Forwarders, or via a direct app install to a forwarder.

However it is important to note that when they are upgraded to a fixed version (Versions 7.1.10, 7.2.9.1, 7.3.3, or 8.0.01) you will need to remove this app from them.

It is imperative that you do this to avoid any further date/time stamping and parsing issues.

Unfortunately, there is no way to correct the timestamp of the data after it has been indexed. If you have indexed data into an unpatched affected Splunk instance, you must patch the instance and then re-ingest the data for the timestamps to be correct.

There are three main options for resolving this problem:

1. Download an updated version of the datetime.xml and apply it to your Splunk Instance:

Splunk has provided an updated datetime.xml file to download and replace your current. This option is the preferred path for customers who cannot upgrade right away to a version of the Splunk Platform with the fixed file.

After you download the file, you must apply it directly over the existing datetime.xml file using the following procedure. You must apply the updated file to all affected on-premises Splunk platform instances prior to January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  • Download the datetime.zip timestamp recognition ZIP file from splunk.com.
  • Unarchive the ZIP file to a location that is accessible from all of your Splunk platform instances.
  • On each Splunk platform instance, do the following:
  • Stop the Splunk platform.
  • Using your operating system file management utilities, copy the updated datetime.xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file.
  • Confirm that the new datetime.xml has been written to the $SPLUNK_HOME/etc directory.
  • Restart the Splunk platform. Your Splunk platform instance is now patched.

2. Make Modifications to existing datetime.xml on your Splunk platform instance:

If you feel comfortable with making changes to the datetime.xml file directly, you can do so without upgrading Splunk. If you are a Splunk Cloud customer that uses a heavy forwarder to send data to your Splunk Cloud instance, you can use this procedure to update those heavy forwarders. 

Use caution when making changes to the $SPLUNK_HOME/etc/datetime.xml file. Outside of fixing this specific issue, never make changes to this file. Typos and incompatible characters in the file can cause direct, lasting negative effects on timestamp recognition and data ingestion. If you are not comfortable with making these changes, consider one of the previous options for fixing this problem, or contact Splunk Support for assistance.

You must complete these changes on all affected Splunk platform instances by January 1, 2020, or you will experience timestamp recognition problems from that point forward.

  • Stop the Splunk platform.
  • Using either a shell prompt or your operating system file management utilities, go to the $SPLUNK_HOME/etc directory in your Splunk Enterprise installation.
  • Open datetime.xml for editing with a text editor.
  • Search for and replace the following strings, according to this table:

Search for this string

Replace with this string

 <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text> <text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text>
 <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text> <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
<text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text><text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text>
<text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text> <text><![CDATA[((?<=^|[\s#,”=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
  • Save the file and close it. If the editor asks if you want to overwrite the file, answer yes.
  • Confirm that the datetime.xml file has been updated.
  • Restart the Splunk platform. Your Splunk platform instance is now patched.

3. Upgrade to a Splunk version with an updated datetime.xml:

Splunk is releasing updated versions of Splunk Cloud, Splunk Enterprise and Splunk Light that contain an updated datetime.xml.

The below table lists the versions that include the fixed file and provides links on how to upgrade Splunk Enterprise and Light. 

Simply apply the fixed version directly over the existing install to patch it.

Major version

Minor version with patched file

Splunk platform upgrade instructions

6.6

6.6.12.1

 

7.0

7.0.13.1

(versions 7.0.12 and 7.0.13 are Splunk Cloud-only releases)

 

7.1

7.1.10

 

7.2

7.2.9.1

 

7.3

7.3.3 (released)

Splunk EnterpriseSplunk Light

8.0

8.0.1

 

Join our webinar or contact our support team for more information.

 We’ll be hosting a webinar on 4th December to provide further details. Alternatively, you can drop us a line if you have any questions.

Scroll to Top