Filter JSON and JSON Lines data with Python syntax.
Project description
jello
Filter JSON and JSON Lines data with Python syntax
jello
is similar to jq
in that it processes JSON and JSON Lines data except jello
uses standard python dict and list syntax.
JSON or JSON Lines can be piped into jello
(JSON Lines are automatically slurped into a list of dictionaries) and are available as the variable _
. Assign the output the the variable r
to print as JSON or simple lines.
For more information on the motivations for this project, see my blog post.
Install
pip3 install --upgrade jello
Usage
<JSON Data> | jello [OPTIONS] query
query
can be most any valid python code as long as the result is assigned to r
. _
is the sanitized JSON from STDIN presented as a python dict or list of dicts. For example:
$ cat data.json | jello 'r = _["key"]'
Options
-c
compact print JSON output instead of pretty printing-i
initialize environment with a custom config file-l
lines output (suitable for bash array assignment)-r
raw output of selected keys (no quotes)-n
print selected null values-h
help-v
version info
Note: The
lines()
convenience function has been deprecated and will be removed in a future version. Use the-l
option instead to generate output suitable for assignment to a bash variable or array. Use of thelines()
function will generate a warning message toSTDERR
.
Custom Configuration File
You can use the -i
option to initialize the jello
environment with your own configuration file. The configuration file accepts valid python code and can be as simple as adding import
statements for your favorite libraries.
The filename must be .jelloconf.py
and must be located in the proper directory based on the OS platform:
- Linux:
~/
- Windows:
%appdata%/
To simply import a module (e.g. glom
) your .jelloconf.py
file would look like this:
from glom import *
Then you could use glom
in your jello
filters:
$ jc -a | jello -i 'r = glom(_, "parsers.25.name")'
"lsblk"
Alternatively, if you wanted to initialize your jello
environment to substitute glom
syntax for _
your .jelloconf.py
file could look like this:
def _(q, data=_):
import glom
return glom.glom(data, q)
Then you could use the following syntax to filter the JSON data:
$ jc -a | jello -i 'r = _("parsers.6.compatible")'
[
"linux",
"darwin",
"cygwin",
"win32",
"aix",
"freebsd"
]
Examples:
lambda functions and math
$ echo '{"t1":-30, "t2":-20, "t3":-10, "t4":0}' | jello '\
keys = _.keys()
vals = _.values()
cel = list(map(lambda x: (float(5)/9)*(x-32), vals))
r = dict(zip(keys, cel))'
{
"t1": -34.44444444444444,
"t2": -28.88888888888889,
"t3": -23.333333333333336,
"t4": -17.77777777777778
}
$ jc -a | jello 'r = len([entry for entry in _["parsers"] if "darwin" in entry["compatible"]])'
32
for loops
Output as JSON array
jc -a | jello '\
r = []
for entry in _["parsers"]:
if "darwin" in entry["compatible"]:
r.append(entry["name"])'
[
"airport",
"airport_s",
"arp",
"crontab",
"crontab_u",
...
]
Output as bash array
jc -a | jello -rl '\
r = []
for entry in _["parsers"]:
if "darwin" in entry["compatible"]:
r.append(entry["name"])'
airport
airport_s
arp
crontab
crontab_u
...
List and Dictionary Comprehension
Output as JSON array
$ jc -a | jello 'r = [entry["name"] for entry in _["parsers"] if "darwin" in entry["compatible"]]'
[
"airport",
"airport_s",
"arp",
"crontab",
"crontab_u",
...
]
Output as bash array
$ jc -a | jello -rl 'r = [entry["name"] for entry in _["parsers"] if "darwin" in entry["compatible"]]'
airport
airport_s
arp
crontab
crontab_u
...
Environment Variables
$ echo '{"login_name": "joeuser"}' | jello '\
r = True if os.getenv("LOGNAME") == _["login_name"] else False'
true
Using 3rd Party Libraries
You can import and use your favorite libraries to manipulate the data. For example, using glom
:
$ jc -a | jello '\
from glom import *
r = glom(_, ("parsers", ["name"]))'
[
"airport",
"airport_s",
"arp",
"blkid",
"crontab",
"crontab_u",
"csv",
...
]
Complex JSON Manipulation
The data from this example comes from https://programminghistorian.org/assets/jq_twitter.json
Under Grouping and Counting, Matthew describes an advanced jq
filter against a sample Twitter dataset that includes JSON Lines data. There he describes the following query:
“We can now create a table of users. Let’s create a table with columns for the user id, user name, followers count, and a column of their tweet ids separated by a semicolon.”
https://programminghistorian.org/en/lessons/json-and-jq
Here is a simple solution using jello
:
$ cat jq_twitter.json | jello -l '\
user_ids = set()
r = []
for tweet in _:
user_ids.add(tweet["user"]["id"])
for user in user_ids:
user_profile = {}
tweet_ids = []
for tweet in _:
if tweet["user"]["id"] == user:
user_profile.update({
"user_id": user,
"user_name": tweet["user"]["screen_name"],
"user_followers": tweet["user"]["followers_count"]})
tweet_ids.append(str(tweet["id"]))
user_profile["tweet_ids"] = ";".join(tweet_ids)
r.append(user_profile)'
...
{"user_id": 2696111005, "user_name": "EGEVER142", "user_followers": 1433, "tweet_ids": "619172303654518784"}
{"user_id": 42226593, "user_name": "shirleycolleen", "user_followers": 2114, "tweet_ids": "619172281294655488;619172179960328192"}
{"user_id": 106948003, "user_name": "MrKneeGrow", "user_followers": 172, "tweet_ids": "501064228627705857"}
{"user_id": 18270633, "user_name": "ahhthatswhy", "user_followers": 559, "tweet_ids": "501064204661850113"}
{"user_id": 14331818, "user_name": "edsu", "user_followers": 4220, "tweet_ids": "615973042443956225;618602288781860864"}
{"user_id": 2569107372, "user_name": "SlavinOleg", "user_followers": 35, "tweet_ids": "501064198973960192;501064202794971136;501064214467731457;501064215759568897;501064220121632768"}
{"user_id": 22668719, "user_name": "nodehyena", "user_followers": 294, "tweet_ids": "501064222772445187"}
...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.