esdateutil 1.0.1

Elasticsearch datemath and dateformat parsing library. Zero dependencies

esdateutil

Provides utilities for handling dates like how Elasticsearch does.

In particular:

• Datemath parsing and evaluation (ES Datemath Docs)
• Datetime string format parsing (ES Dateformat Docs)

The goals of this project are:

• Be as close to Elasticsearch behaviour as Python makes sensible.
• Require no runtime dependencies.
• Customizability; most functionality should be parameterizable.

Installation

esdateutil is on PyPI and can be downloaded via pip: pip install esdateutil

You can also just copy and use the bits you need. Because there are no dependencies and not a lot of code the whole library is just 2 files and an __init__ totalling O(100s) LOC. Output for 1.0.0 from scc:

$ ssc esdateutil
────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code
────────────────────────────────────────────────────────────────────
Python                       3       533       92        59      382
────────────────────────────────────────────────────────────────────

Usage

Basic Usage

>>> from datetime import datetime
>>> datetime.now() # now is as below for all examples
datetime.datetime(2024, 9, 24, 8, 36, 17, 503027)

>>> from esdateutil import datemath, dateformat

>>> df = dateformat.DateFormat() # defaults to strict_date_optional_time||epoch_millis
>>> df.parse("2024-09-24T08:36Z") # strict_date_optional_time
datetime.datetime(2024, 9, 24, 08, 36, tzinfo=datetime.timezone.utc)
>>> df.parse("1727163377503") # epoch_millis
datetime.datetime(2024, 9, 24, 8, 36, 17, 503000)

>>> dm = datemath.DateMath()
>>> dm.eval("now-5m/h") # now minus 5 minutes rounded to the hour
datetime.datetime(2024, 9, 24, 8, 0)
>>> dm.eval("2024-09-24||-5m/h") # absolute time minus 5 minutes rounded to the hour
datetime.datetime(2024, 9, 23, 23, 0)

Configuration Options

Dateformat

DateFormat objects are for parsing strings representing datetimes, with support for Elasticsearch built-in formats and format syntax. They take any number of date format strings and on calling DateMath.parse will try each of them sequentially against the string being parsed, returning the first that matches. If the string doesn't match any of the given dateformats, it returns a ValueError containing each dateformat's failure reason.

fmt & separator

The fmt argument to DateFormat can be a string separated by separator (default "||") or a list containing strings and functions.

fmt as a string behaves the same as Elasticsearch format strings taking the same built-in format names except that custom formats use Python strptime instead of Java DateTimeFormatter formatted strings.

esdateutil doesn't support the full list of built-in formats in the Elastic documentation. To see the list of supported built-in formats:

>>> from esdateutil import dateformat
>>> print(list(dateformat.DATE_FORMATS.keys()))
...

As of writing, esdateutil supports:

• strict_date
• date
• strict_date_optional_time
• date_optional_time
• strict_date_optional_time_nanos (*)
• epoch_millis
• epoch_second

*: Only supports up to microsecond precision. See "Differences from Elasticsearch" below

By default the DateFormat object uses the format strict_date_optional_time||epoch_millis, which is what Elasticsearch uses.

You can also configure the separator, e.g.:

>>> from esdateutil import dateformat
>>> parser = dateformat.DateFormat("strict_date, epoch_seconds", separator=", ")
>>> parser.parse("1729972832")

tzinfo

tzinfo can be set to a Python timezone instance which will be set on any returned datetime instances from DateMath.parse. If the string being parsed contains timezone information and the format sets that, this is used instead of the tzinfo value. Format functions are responsible for this behaviour, so a custom format can be used to alter this.

Datemath

DateMath objects can be used to evaluate or parse datemath expressions. Simple as! You can configure most of the things that you might want to configure, per the below instance options.

tzinfo

A timezone instance to add to evaluated datetimes. If the anchor parser sets the tzinfo, that will be prefered. Defaults to None.

separator

Separator between an explicit anchor and the datemath expression. Defaults to "||". e.g.

>>> from esdateutil import datemath
>>> dm = datemath.DateMath(separator=":")
>>> dm.eval("2024-12-24:+1d")

now_str

What to use as the anchor that gets the current datetime. Defaults to "now".

>>> from esdateutil import datemath
>>> dm = datemath.DateMath(now_str="sofort")
>>> dm.eval("sofort+1d")

now_fn

Function to use to get the current datetime. Takes an argument tz which is given as the arg tzinfo. Defaults to lambda tz: datetime.now(tz).

date_fn

Function to use to parse non-now datetime anchors in DateMath.eval and DateMath.parse. Defaults to the equivalent of dateformat.DateFormat.parse, i.e. ES strict_date_optional_time||epoch_millis.

units_delta

Dictionary that maps datemath unit chars to functions to add and subtract that amount of time. Defaults to datemath.UNITS_DELTA_DEFAULT - see datemath.py for details.

units_round

Dictionary that maps datemath unit chars to fucntions to round to that duration of time. Defaults to datemath.UNITS_ROUND_DOWN - see datemath.py for details. datemath also provides UNITS_ROUND_UP_MICROS and UNITS_ROUND_UP_MILLIS for use with this argument, to round up to the highest time point in the given duration.

Debug Logging

Debug logging can be set the same way as any Python lib using the logging stdlib library:

import logging

import esdateutil

# To set globally:
logging.basicConfig(level=logging.DEBUG)

# Else to set for just the library logger:
handler = logging.StreamHandler()
esdateutil_logger = logging.getLogger("esdateutil")
esdateutil_logger.setLevel(logging.DEBUG)
esdateutil_logger.addHandler(handler)

Development

The development scripts under scripts/ are used to build and test the library. They rely on pyenv and pyenv-virtualenv to work. Because testing is done on 3.3 through 3.12+, you will need to build or download an old version of openssl to build python 3.3 via pyenv. Read more about using pyenv and pyenv-virtualenv.

The builds and tests are very simple, so feel free to do your own thing so long as it works.

To run the test suite:

$ eval "$(pyenv init -)" # See https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv
$ pyenv install <py_ver>... # install all versions of python you need to test
$ scripts/tests.sh <py_ver>... # Run tests for given python versions
$ scripts/tests.sh # Run for all targeted python versions

Testing has only been performed on Linux. I expect everything to work on MacOS and Windows. :v)

Differences from Elasticsearch

One of the consequences of using Python's built-in datetime objects and functions by default is that they can behave very differently from version to version and from Elasticsearch defaults. Below are some of the most important differences in functionality to be aware of.

• The default time resolution in Elasticsearch is milliseconds, whereas in Python datetime it is microseconds. This shouldn't be important unless you are using datemath.UNITS_ROUND_UP_MICROS or another custom datemath round implementation. UNITS_ROUND_UP_MILLIS is provided as an alternative.
• Elasticsearch has optional support for nanosecond precision - because Python datetimes use microsecond precision, we cannot support this completely. This impacts dateformat strict_date_option_time_nanos, which can still be used for microsecond precision instead of millis precision.
• For custom dateformat strings we use strptime as a backup instead of Java's time format strings.

Alternatives

python-datemath

There is another Python project python-datemath for parsing datemath expressions. This projects has different goals to esdateutil, the main difference between them is that python-datemath parses a custom datemath variant, whereas esdateutil.datemath adheres strictly to the Elasticsearch datemath syntax. This means that although the syntax overlaps they will accept and reject different strings.

In most cases, this probably doesn't matter. See the table below for a specific feature difference breakdown.

Difference	esdateutil.datemath	python-datemath
Syntax	Accepts and rejects same syntax as Elasticsearch. Unit chars are configurable.	Allows additional uppercase unit chars (Y for year, W for week, D for day, S for second), allows long-form units (e.g. seconds, days), allows fractional durations (e.g. +1.2d), does not allow missing number (e.g. +y vs +1y), treats expressions without anchors as having now (e.g. +2d is equivalent to now+2d)
Date String Support	Accepts the equivalent of strict_date_optional_time||epoch_millis by default. Date parser can be overwritten by a user function.	Accepts epoch seconds or all formats supported by arrow.get by default.
Date Types	Uses Python's built-in datetime, timedelta, and timezone types for all date operations.	Uses arrow's Arrow type for all operations. This can be converted to a datetime.
Dependencies	0 runtime dependencies. 5 build dependencies (pyenv, pyenv-virtualenv, build, setuptools, wheel).	4 runtime dependenices, including transitive dependencies: arrow --> python-dateutil --> six + types-python-dateutil. 47 build dependencies.
Version Support	Supports Python 3.3+	Supports Python 3.8+ with arrow 1.0.3+. Previous versions support 2.7 and 3.x
Performance	Processes 1 million datemath strings in 11.39s On My Machine(TM). See profiling/ for details and to reproduce.	Processes 1 million datemath strings in 103.39s On My Machine(TM). See profiling/ for details and to reproduce.
Type Hints	No type hints.	Strict type checking with inline types.
Timezones	Returns tz-unaware datetimes by default, unless tzinfo is given as an argument or timezone details are in a datestring.	Assumes datetimes are UTC by default, unless tz is provided in a date string or given as a string argument.
Options	https://git.sr.ht/~murr/esdateutil/tree/master/item/esdateutil/datemath.py#L79	https://github.com/nickmaccarthy/python-datemath/blob/master/datemath/helpers.py#L85
Licence	MIT	Apache 2.0