Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy
Project description
splitlog
Hadoop Yarn application logs aggregate all container logs of a Yarn application into a single file. This makes it very
difficult to use Unix command line tools to analyze these logs: Grep will search over all containers and context
provided for hits often does not include Yarn container name or host name. splitlog
splits a combined logfile for all
containers of an application into a file system hierarchy suitable for further analysis:
out
└── hadoopnode
├── container_1671326373437_0001_01_000001
│ ├── directory.info
│ ├── launch_container.sh
│ ├── prelaunch.err
│ ├── prelaunch.out
│ ├── stderr
│ ├── stdout
│ └── syslog
├── container_1671326373437_0001_01_000002
│ ├── directory.info
│ ├── launch_container.sh
│ ├── prelaunch.err
│ ├── prelaunch.out
│ ├── stderr
│ ├── stdout
│ └── syslog
└── container_1671326373437_0001_01_000003
├── directory.info
├── launch_container.sh
├── prelaunch.err
├── prelaunch.out
├── stderr
├── stdout
└── syslog
4 directories, 21 files
Installation
Python 3.9+ must be available. Installation via pipx:
pipx install splitlog
How to use
Read logs from standard input:
yarn logs -applicationId application_1582815261257_232080 | splitlog
Read logs from file application_1582815261257_232080.log
:
splitlog -i application_1582815261257_232080.log
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file splitlog-4.0.1.tar.gz
.
File metadata
- Download URL: splitlog-4.0.1.tar.gz
- Upload date:
- Size: 8.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 917899bfc190427c4f3f9b2243cdce7bb324f750c6680668505134d9f580d1ce |
|
MD5 | 87ebbd0e0d4f33389a7f648f218ce6b6 |
|
BLAKE2b-256 | 5e9f6ac39e46d3c0fea0b44fd3739e2c87e9f0aeba868476006dc14e276abba8 |
Provenance
The following attestation bundles were made for splitlog-4.0.1.tar.gz
:
Publisher:
release.yml
on splitlog/splitlog
-
Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
splitlog-4.0.1.tar.gz
- Subject digest:
917899bfc190427c4f3f9b2243cdce7bb324f750c6680668505134d9f580d1ce
- Sigstore transparency entry: 144028376
- Sigstore integration time:
- Predicate type:
File details
Details for the file splitlog-4.0.1-py3-none-any.whl
.
File metadata
- Download URL: splitlog-4.0.1-py3-none-any.whl
- Upload date:
- Size: 8.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e8e50da8631a1a77d5d29ef7a52b0094f6435f4d0bae66c4b1a142a475dbfc2 |
|
MD5 | b8657714cd9a44c5906efc146f90c41d |
|
BLAKE2b-256 | d0d0447b1c4fb3cc89893988bdb94c6aa9cb2a780952e87f8f6570ed8aaeaa07 |
Provenance
The following attestation bundles were made for splitlog-4.0.1-py3-none-any.whl
:
Publisher:
release.yml
on splitlog/splitlog
-
Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
splitlog-4.0.1-py3-none-any.whl
- Subject digest:
1e8e50da8631a1a77d5d29ef7a52b0094f6435f4d0bae66c4b1a142a475dbfc2
- Sigstore transparency entry: 144028377
- Sigstore integration time:
- Predicate type: