README
Parse and slice hadoop logs
Yarn RM
Dataset
from khadoop.yarn import logrm
Parse all files that look like a regular Ressource Manager log with default name.
logrm.FILEPATTERN is a unix-like pattern file to help glob them.
parsed = []
for filelog in LOGFOLDER.glob(logrm.FILEPATTERN):
print(filelog)
parsed += logrm.process(filelog.open())
logrm.process will parse each line and produce a list of dict with sensible information
each dict look like :
{
'accepted_to_running': 6, # nb sec between ACCEPT to RUNNING
'id_application': 'application_1596547077642_6854',
'accept_to_running_ts':'2020-08-06 14:59:59,119' # timestamp set for log line 'FROM accepted to RUNNING'
}
the accepted_to_running represent here the number between these two timestamps on yarn aggregated RM log:
2020-08-06 14:59:52,756 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(779)) - application_1596547077642_6854 State change from SUBMITTED to ACCEPTED
...
2020-08-06 14:59:59,119 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(779)) - application_1596547077642_6854 State change from ACCEPTED to RUNNING
Related
Setup dev
Env variables:
HIVESERVER_TEST= #raw hiveserver log file
YARNLOG #folder with RM logs
1 maintainer
Release files for khadoop 1.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| khadoop-1.4.0.tar.gz | 9.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| khadoop-1.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size:19.1 kB
Release files / khadoop-1.4.0.tar.gz
| Download URL | khadoop-1.4.0.tar.gz |
|---|---|
| Size | 9.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49d7eeb7c5bc433efabed99e9f43a2199e210282c58c6a181ebf8375b512c174
|
|
BLAKE2b-256 checksum How to use checksums |
c95896b0573bd1db118e538b2f719ad0ef5b726155a465b2a3a834eb2526fed0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.4 CPython/3.8.5 Linux/4.19.128-microsoft-standard
|
Release files / khadoop-1.4.0-py3-none-any.whl
| Download URL | khadoop-1.4.0-py3-none-any.whl |
|---|---|
| Size | 10.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
443478183abac2e3e8f501cb7e3963e3905e5316547a07dd0b255bed49f92ba8
|
|
BLAKE2b-256 checksum How to use checksums |
915e263f55205147d721e60a45e867f9764fe1db8ed39bd670f5190174434259
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.4 CPython/3.8.5 Linux/4.19.128-microsoft-standard
|
