Skip to main content

Fast import of Windows EventLogs(.evtx) into Elasticsearch.

Project description

evtx2es

MIT License PyPI version Python Versions pytest

evtx2es logo

Fast import of Windows EventLogs(.evtx) into Elasticsearch.

Life is too short and there is not enough time to process huge Windows EventLogs with pure-Python software.
evtx2es uses Rust library pyevtx-rs, so it runs much faster than traditional software.

Usage

evtx2es can be executed from the command line or incorporated into a Python script.

$ evtx2es /path/to/your/file.evtx
from evtx2es import evtx2es

if __name__ == '__main__':
  filepath = '/path/to/your/file.evtx'
  evtx2es(filepath)

Arguments

evtx2es supports simultaneous import of multiple files.

$ evtx2es file1.evtx file2.evtx file3.evtx

Additionally, it also allows for recursive import under the specified directory.

$ tree .
evtxfiles/
  ├── file1.evtx
  ├── file2.evtx
  ├── file3.evtx
  └── subdirectory/
    ├── file4.evtx
    └── subsubdirectory/
      ├── file5.evtx
      └── file6.evtx

$ evtx2es /evtxfiles/ # The Path is recursively expanded to file1~6.evtx.

Options

--version, -v

--help, -h

--quiet, -q
  Flag to suppress standard output
  (default: False)

--multiprocess, -m:
  Enable multiprocessing for faster execution
  (default: False)

--size:
  Chunk size for processing (default: 500)

--host:
  ElasticSearch host address (default: localhost)

--port:
  ElasticSearch port number (default: 9200)

--index:
  Destination index name for importing (default: evtx2es)

--scheme:
  Protocol scheme to use (http or https) (default: http)

--pipeline:
  Elasticsearch Ingest Pipeline to use (default: )

--datasetdate:
  Date of the latest record in the dataset, extracted from TimeCreated field (MM/DD/YYYY.HH:MM:SS) (default: 0)

--login:
  The login to use if Elastic Security is enabled (default: )

--pwd:
  The password associated with the provided login (default: )

Examples

When using from the commandline interface:

$ evtx2es /path/to/your/file.evtx --host=localhost --port=9200 --index=foobar --size=500

When using from the python-script:

if __name__ == '__main__':
    evtx2es('/path/to/your/file.evtx', host=localhost, port=9200, index='foobar', size=500)

With credentials for Elastic Security:

$ evtx2es /path/to/your/file.evtx --host=localhost --port=9200 --index=foobar --login=elastic --pwd=******

Note: The current version does not verify the certificate.

Appendix

Evtx2json

An additional feature: :sushi: :sushi: :sushi:

Convert Windows Event Logs to a JSON file.

$ evtx2json /path/to/your/file.evtx /path/to/output/target.json

Convert Windows Event Logs to a Python List[dict] object.

from evtx2es import evtx2json

if __name__ == '__main__':
  filepath = '/path/to/your/file.evtx'
  result: List[dict] = evtx2json(filepath)

Output Format Example

Using the sample evtx file of JPCERT/CC:LogonTracer as an example.

[
  {
    "event_record_id": 227559,
    "timestamp": "2016-10-06 01:50:49.420927 UTC",
    "winlog": {
      "channel": "Security",
      "computer_name": "WIN-WFBHIBE5GXZ.example.co.jp",
      "event_id": 4624,
      "opcode": 0,
      "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
      "provider_name": "Microsoft-Windows-Security-Auditing",
      "record_id": 227559,
      "task": 12544,
      "version": 0,
      "process": {
        "pid": 572,
        "thread_id": 1244
      },
      "event_data": {
        "AuthenticationPackageName": "Kerberos",
        "IpAddress": "192.168.16.102",
        "IpPort": "49220",
        "KeyLength": 0,
        "LmPackageName": "-",
        "LogonGuid": "F4DC1C19-0544-BC52-0900-DFC19752C3C6",
        "LogonProcessName": "Kerberos",
        "LogonType": 3,
        "ProcessId": 0,
        "ProcessName": "-",
        "SubjectDomainName": "-",
        "SubjectLogonId": "0x0",
        "SubjectUserName": "-",
        "SubjectUserSid": "S-1-0-0",
        "TargetDomainName": "EXAMPLE",
        "TargetLogonId": "0x1fa0869",
        "TargetUserName": "WIN7_64JP_02$",
        "TargetUserSid": "S-1-5-21-1524084746-3249201829-3114449661-1107",
        "TransmittedServices": "-",
        "WorkstationName": "",
        "Status": null
      }
    },
    "log": {
      "file": {
        "name": "sample/Security.evtx"
      }
    },
    "event": {
      "code": 4624,
      "created": "2016-10-06T01:50:49.420927Z"
    },
    "@timestamp": "2016-10-06T01:50:49.420927Z"
  },
  ...
]

Performance Evaluations

evtx2es was evaluated using the sample evtx file of JPCERT/CC:LogonTracer (about 30MB binary data).

$ time evtx2es ./Security.evtx
> 6.25 user 0.13 system 0:14.08 elapsed 45%CPU

See Qiita for more information.

Running Environment

OS: Ubuntu 18.04
CPU: Intel Core i5-6500
RAM: DDR4 32GB

ElasticSearch 7.4 was running on the Docker version(Official Image).
https://hub.docker.com/_/elasticsearch

Installation

from PyPI

$ pip install evtx2es

from GitHub Releases

The version compiled into a binary using Nuitka is also available for use.

$ chmod +x ./ntfsdump
$ ./ntfsdump {{options...}}
> ntfsdump.exe {{options...}}

Contributing

The source code for evtx2es is hosted at GitHub, and you may download, fork, and review it from this repository(https://github.com/sumeshi/evtx2es). Please report issues and feature requests. :sushi: :sushi: :sushi:

License

evtx2es is released under the MIT License.

Powered by following libraries:

Inspired by EvtxtoElk.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

evtx2es-1.6.1.tar.gz (12.0 kB view details)

Uploaded Source

Built Distribution

evtx2es-1.6.1-py3-none-any.whl (13.4 kB view details)

Uploaded Python 3

File details

Details for the file evtx2es-1.6.1.tar.gz.

File metadata

  • Download URL: evtx2es-1.6.1.tar.gz
  • Upload date:
  • Size: 12.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.11.9 Linux/6.5.0-1021-azure

File hashes

Hashes for evtx2es-1.6.1.tar.gz
Algorithm Hash digest
SHA256 ca489319831ccb74cb1ad7bd1243be7369ad0ecf5fd3e208ad988d9b56d48d22
MD5 ce63e42dce49e24d35e95d08760b072b
BLAKE2b-256 0a1307d6f5b4b9a1909aa751e5bbcaad93c9505469a4083d4a0806a1eb379b84

See more details on using hashes here.

File details

Details for the file evtx2es-1.6.1-py3-none-any.whl.

File metadata

  • Download URL: evtx2es-1.6.1-py3-none-any.whl
  • Upload date:
  • Size: 13.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.11.9 Linux/6.5.0-1021-azure

File hashes

Hashes for evtx2es-1.6.1-py3-none-any.whl
Algorithm Hash digest
SHA256 80dace56110c3075596844efbded33d18abb13526a33aa9acf47b2ea6718d0b6
MD5 28576d56680b70c6aec35b8099c4b0dc
BLAKE2b-256 596c244461341ff4e2af34075b9babd3ebab195badb3fdee1f0e3f077007fc1d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page