This presentation was created out outline the Splunk timestamp issue, the root cause, impact and discuss potential solutions. ISSUE Beginning on January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. Beginning on September 13, 2020 at 12:26:39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data. The write up is on our Tech Blog: https://medium.com/adarma-tech-blog/splunk-timestamp-issue-4ba1087735c7

1. ©2019 ADARMA. ALL RIGHTS RESERVED

2. These are Adarma’s conclusions based upon the most recent official information from
Splunk andwhere noted our own observations.
Disclaimer
Werecommendreading in full theSplunk documentationon the issue:
SplunkDocs: https://docs.splunk.com/Documentation/Splunk/8.0.0/ReleaseNotes/FixDatetimexml2020
If in doubt reachout toAdarma’sor Splunk’sdedicatedProfessionalServicesfor guidance andhelp.

3. Agenda
3
- Introduction
- Problem Statement
- RootCause
- Impact & Checking Data Sources
- Recommended Solution(LongTerm)
- Tactical Solutions (ShortTerm)
- Q&A
Objective: Summarise the recent discovery of a Splunk configuration issue, outline its
cause, impact and steps to minimise its impact to your environment.

4. Q&A (Written Questions Only)
4

5. c
5
Beginningon January1, 2020, un-patched Splunk platform instances will be unable torecognize
timestamps from events where thedate contains a two-digityear.
Beginningon September 13, 2020 at 12:26:39 PM CoordinatedUniversal Time (UTC),un-patched
Splunk platform instances will be unable to recognize timestamps from events with dates that are
based on Unix time, due to incorrect parsing of timestamp data.
Problem Statement
Source: SplunkDocs

6. c
6
TheSplunk platforminput processoruses a filecalleddatetime.xmlto helpthe processor correctlydetermine
timestamps basedon incoming data.The file uses regularexpressionsto extractmany differenttypes ofdates and
timestamps from incomingdata. Example:
Root Cause
$SPLUNK_HOME/etc/datetime.xml
Before After
<text><![CDATA[(20dd|19dd|[901]
d(?!d))]]></text>
<text><![CDATA[(20dd|19dd|[9012]
d(?!d))]]></text>
<text><![CDATA[(?:^|source::).*?(?<!
d|d.|-
)(?:20)?([901]d)(0d|1[012])([012]
d|3[01])(?!d|-| {2,})]]></text>
<text><![CDATA[(?:^|source::).*?(?<!
d|d.|-
)(?:20)?([9012]d)(0d|1[012])([012]
d|3[01])(?!d|-| {2,})]]></text>
Source: SplunkDocs

7. c
7
The issueappearswhen youhave configured the inputsource to automaticallydetermine timestamps, andcan result in one or more of the
followingproblems:
• Incorrecttimestampingof incoming data
• Incorrectrolloverof data buckets due to the incorrecttimestamping
• Incorrectretention of data overall
• Incorrectsearchresults due to data ingested with incorrecttimestamps
There is nomethod to correct the timestamps after the Splunkplatform hasingested the data when the problem starts.If you ingest datawith an
un-patched Splunkplatforminstancebeginning onJanuary 1, 2020, you must patchthe instanceandre-ingest that datafor its timestampsto be
correct.
Impact
Source: SplunkDocs

8. Impacted Splunk Software
8
•SplunkUniversal
Forwarders
•When they have been configured to
process structured data, such as CSV,
XML, and JSON files, using
the INDEXED_EXTRACTIONS setting in
props.conf
•When they have been configured to
process data locally, using
the force_local_processing setting
in props.conf
•When they have been configured with
a monitor input, and that input
subsequently encounters an unknown
file type
•Splunk Cloud
•Splunk Light
•Splunk Enterprise Indexers (Clustered or Not)
•Heavy Forwarders or IDMs
•Search Heads (Clustered or Not)
•Search Head Deployers
•Deployment Servers
•Cluster Masters
•License Masters
Source: SplunkDocs

9. c
9
index=_internal earliest=-48h@h group=tcpin_connections sourcetype=splunkd
component=Metrics
[search index=_internal earliest=-48h@h sourcetype=splunkd
component=Metrics name=structuredparsing (processor=linebreaker OR
processor=aggregator OR processor=regexreplacement)
| stats count by host
| fields - count
| rename host AS hostname]
| stats avg(tcp_KBps) AS avg_kbps first(version) AS version first(os) AS os
first(fwdType) AS fwd_type first(guid) AS guid by hostname
| eval avg_kbps=round(avg_kbps,2)
| sort - avg_kbps
Should indicate if Universal Forwarders are impacted (requires internal metrics).
Gratitude to:SplunkProfessionalServices
Checking UF Impact (SPL)

10. c
10
find . -name props.conf | xargs grep
DATETIME_CONFIG | grep -v -e CURRENT -e NONE
Should show impacted sourcetypes using ‘auto timestamp recognition’.
Gratitudeto:SvetlanafromSplunkSupport
Checking Impact (Linux Find)

11. c
11
•Upgrade Splunk platform instances to a version with an
updated version of datetime.xml
Upgrade Splunk Software
•Always ensure the following are set in props.conf
•TIME_PREFIX, TIME_FORMAT,
MAX_TIMESTAMP_LOOKAHEAD, SHOULD_LINEMERGE,
LINE_BREAKER, TRUNCATE
Always Configure the Magic 6 in props.conf
Recommended Solution(Long Term)
Source: SplunkDocs &AdarmaTechBlog

12. c
12
• Adding this to EVERY impacted sourcetype configuration will force Splunk to use this
configuration and not automatically try and detect the format via datetime.xml
Add ‘TIME_FORMAT’ config to props.conf
• Download and deploy an app that temporarily replaces the defective datetime.xml file
with the fixed file.
• Download an updated version of datetime.xml and apply it to each of your Splunk
platform instances (with file integrity issues)
Replace datetime.xml or temporarilyreplaceit via anApp
• Make modifications to the existing datetime.xml file on your Splunk platform instances
Modify datetime.xml directly(with file integrityissues)
Tactical Solutions (Short Term)
Source: SplunkDocs &AdarmaTechBlog

13. c
13
Q&A

14. Thank You!
Twitter: @cyberharibu
Email: harry.mclaren@adarma.com
Contact Adarma: enquiries@adarma.com
Blog:medium.com/adarma-tech-blog

15. ©2019 ADARMA. ALL RIGHTS RESERVED
YOUR TRUSTED SECURITY PARTNER