Skip to main content

Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy

Project description

splitlog

Hadoop Yarn application logs aggregate all container logs of a Yarn application into a single file. This makes it very difficult to use Unix command line tools to analyze these logs: Grep will search over all containers and context provided for hits often does not include Yarn container name or host name. splitlog splits a combined logfile for all containers of an application into a file system hierarchy suitable for further analysis:

out
└── hadoopnode
    ├── container_1671326373437_0001_01_000001
    │   ├── directory.info
    │   ├── launch_container.sh
    │   ├── prelaunch.err
    │   ├── prelaunch.out
    │   ├── stderr
    │   ├── stdout
    │   └── syslog
    ├── container_1671326373437_0001_01_000002
    │   ├── directory.info
    │   ├── launch_container.sh
    │   ├── prelaunch.err
    │   ├── prelaunch.out
    │   ├── stderr
    │   ├── stdout
    │   └── syslog
    └── container_1671326373437_0001_01_000003
        ├── directory.info
        ├── launch_container.sh
        ├── prelaunch.err
        ├── prelaunch.out
        ├── stderr
        ├── stdout
        └── syslog

4 directories, 21 files

Installation

Python 3.9+ must be available. Installation via pipx:

pipx install splitlog

How to use

Read logs from standard input:

yarn logs -applicationId application_1582815261257_232080 | splitlog

Read logs from file application_1582815261257_232080.log:

splitlog -i application_1582815261257_232080.log

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

splitlog-4.0.1.tar.gz (8.1 kB view details)

Uploaded Source

Built Distribution

splitlog-4.0.1-py3-none-any.whl (8.8 kB view details)

Uploaded Python 3

File details

Details for the file splitlog-4.0.1.tar.gz.

File metadata

  • Download URL: splitlog-4.0.1.tar.gz
  • Upload date:
  • Size: 8.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for splitlog-4.0.1.tar.gz
Algorithm Hash digest
SHA256 917899bfc190427c4f3f9b2243cdce7bb324f750c6680668505134d9f580d1ce
MD5 87ebbd0e0d4f33389a7f648f218ce6b6
BLAKE2b-256 5e9f6ac39e46d3c0fea0b44fd3739e2c87e9f0aeba868476006dc14e276abba8

See more details on using hashes here.

Provenance

The following attestation bundles were made for splitlog-4.0.1.tar.gz:

Publisher: release.yml on splitlog/splitlog

Attestations:

File details

Details for the file splitlog-4.0.1-py3-none-any.whl.

File metadata

  • Download URL: splitlog-4.0.1-py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for splitlog-4.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 1e8e50da8631a1a77d5d29ef7a52b0094f6435f4d0bae66c4b1a142a475dbfc2
MD5 b8657714cd9a44c5906efc146f90c41d
BLAKE2b-256 d0d0447b1c4fb3cc89893988bdb94c6aa9cb2a780952e87f8f6570ed8aaeaa07

See more details on using hashes here.

Provenance

The following attestation bundles were made for splitlog-4.0.1-py3-none-any.whl:

Publisher: release.yml on splitlog/splitlog

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page