Skip to main content

pyjq

A simple Python package to Query Json Data.

Features

  • Supports pure json files
  • Supports multiple json objects in a file, delimited by newlines (/n)
  • Supports gzipped files
  • Supports customizabile filters
  • Supports pure datetime range filters

Todo

The filters could be extended easily, adopting Python3 stdlib operator. See pyjq.PyJQ.filter to extend ops mapping.

Installation

pip install pyjq-ng

Example data

See example/alerts.json. pyjq works on lines by lines (splitted by \n). It have been used for Wazuh alert json files and Django dumps.

pyjq -j examples/django_dump.json -limit 2 -filter 'fields__original_url == https://google.com'
pyjq -j examples/django_dump.json -limit 2 -filter 'model == urlshortener.urlshortener'

Usage

'agent__name' it's an example of the namespace used by pyjq to access to nested childs. It other word it means json['agent']['name']. It haven't limits on number of nested elements.

Apply some custom filters with AND and OR operators on Wazuh Alert file

pyjq -j ../Scaricati/alerts.json -filter 'agent__ip == 172.16.16.102 and agent__name == telegram-gw or agent__ip == 172.16.16.108'

Contains operator

pyjq -j ../Scaricati/alerts.json -filter 'rule__description in iptables and agent__name == dev-bastion'

Convert a specified filed to a pure datetime object and filter in a specified range

pyjq -j ../Scaricati/alerts.json -start_datetime 2020-04-06T10:22:00 -end_datetime 2020-04-06T13:22:00 -datetime_field timestamp

Realtime reading, it will only takes the latter entries, delimited by newline \n

pyjq -j /var/ossec/logs/alerts/alerts.json -datetime_field timestamp -realtime

Use a gzipped json file directly

pyjq -j ../Scaricati/alerts.json.gzip

Limit results to 2

pyjq -j ../Scaricati/alerts.json  -limit 2

Realtime monitoring of a specific entity

pyjq -j /var/ossec/logs/alerts/alerts.json -realtime -filter 'agent__name == tinyurl and rule__level == 3'

Custom callback, usefull for bot integration and other pub/sub APIs

python3 pyjq -j examples/alerts.json -realtime -filter 'agent__name == tinyurl and rule__description in ssh' -callback 'examples.callback.things'

Reading from stdin

cat examples/alerts.json | python3 ./pyjq -filter 'rule__level > 3'

# continous processing
tail -f  /tmp/alerts.json | python3 ./pyjq -filter 'location != osquery'

Author

Giuseppe De Marco giuseppe.demarco@unical.it

Credits

Wazuh SIEM group @GarrLab

Metadata

Release files for pyjq-ng 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyjq-ng 0.8.0
File Size Uploaded Provenance Actions
pyjq-ng-0.8.0.tar.gz 5.2 kB Without provenance Details for pyjq-ng-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyjq-ng 0.8.0
File Interpreter ABI Platform Actions
pyjq_ng-0.8.0-py3-none-any.whl Python 3 none any Details for pyjq_ng-0.8.0-py3-none-any.whl

Total release size: 11.7 kB

Release files / pyjq-ng-0.8.0.tar.gz

Download URL pyjq-ng-0.8.0.tar.gz
Size 5.2 kB
Tags Source
SHA-256 checksum
How to use checksums
d059465e5d9ae351262acd26b459f99a3dffa79fd6c63f5fc4da1637f3032308
BLAKE2b-256 checksum
How to use checksums
4af711ca7bb88d333343f0a49aed774f6acbe459d2b8a82d9fc16cfe7fe0de32
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.4

Release files / pyjq_ng-0.8.0-py3-none-any.whl

Download URL pyjq_ng-0.8.0-py3-none-any.whl
Size 6.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1a4f1c60f3b820841dfd88ca710c0727c06b53618bfa740ec2ea2934cf8d1a69
BLAKE2b-256 checksum
How to use checksums
ddcecd045e9625477d1cf9d30d56a38529a8deee7b5d414c5481d55102a6c81f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.4

Security for pyjq-ng 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page