A powerful tool for pretty-printing, querying and conversion JSON documents.
Project description
𝑱𝒔𝒐𝒏𝑭𝒎𝒕
jsonfmt (JSON Formatter) is a simple yet powerful JSON processing tool.
As we all know, Python has a built-in tool for formatting JSON data: python -m json.tool. However, its functionality is too simple, so jsonfmt extends it with many practical features:
🎨 It can not only print JSON data in a pretty way,
🔄 But also convert JSON, TOML, XML and YAML data formats to each other,
🔎 And even extract content from JSON data using JMESPATH or JSONPATH.
🧐 You can even use jsonfmt to compare differences between two JSON or other formatted data.
Quick Start
Installation
$ pip install jsonfmt
Usage
- Process data from files.
$ jf [options] [data_files ...]
- Process data from
stdin.
$ echo '{"hello": "world"}' | jf [options]
Positional Arguments
files: The data files to process, supporting JSON / TOML / XML / YAML formats.
Options
-h: Show this help documentation and exit.-C: CopyMode, which will copy the processing result to the clipboard.-d: DiffMode, which compares the difference between the two input data.-D DIFFTOOL: DifftoolMode, similar to "DiffMode". You can specify a tool to perform diff comparisons.-o: OverviewMode, which can display an overview of the structure of the data, helping to quickly understand larger data.-O: OverwriteMode, which will overwrite the original file with the formated text.-c: Suppress all whitespace separation (most compact), only valid for JSON.-e: Escape all characters to ASCII codes.-f: The format to output (default: same as input data format, options:json/toml/xml/yaml).-i: Number of spaces for indentation (default: 2, range: 0~8, set to 't' to use Tab as indentation).-l: Query language for extracting data (default: auto-detect, options: jmespath / jsonpath).-p QUERYPATH: JMESPath or JSONPath query path.-s: Sort the output of dictionaries alphabetically by key.--set 'foo.k1=v1;k2[i]=v2': Key-value pairs to add or modify (separated by ";").--pop 'k1;foo.k2;k3[i]': Key-value pairs to delete (separated by ";").-v: Show the version.
User Guide
In order to demonstrate the features of jsonfmt, we need to first create a test data and save it to the file example.json. The file contents are as follows:
{
"name": "Bob",
"age": 23,
"gender": "纯爷们",
"money": 3.1415926,
"actions": [
{
"name": "eating",
"calorie": 1294.9,
"date": "2021-03-02"
},
{
"name": "sporting",
"calorie": -2375,
"date": "2023-04-27"
},
{
"name": "sleeping",
"calorie": -420.5,
"date": "2023-05-15"
}
]
}
Then, convert this data to TOML, XML and YAML formats, and save them as example.toml, example.xml and example.yaml respectively.
These data files can be found in the test folder of the source code:
test/
|- example.json
|- example.toml
|- example.xml
|- example.yaml
1. Pretty Print JSON Data
Syntax Highlighting and Indentation
The default working mode of jsonfmt is to format the data and print it with syntax highlighting.
The option -i specifies the number of spaces for indentation. By default, it is 2 spaces, and the number of spaces allowed is between 0 and 8. If you want to use the tab as indentation, set it to t.
The -s option is used to sort the dictionary alphabetically by key.
If there are some non-ASCII characters in the JSON data, you can use -e to escape them.
$ jf -s -i 4 test/example.json
Output:
{
"actions": [
{
"calorie": 1294.9,
"date": "2021-03-02",
"name": "eating"
},
{
"calorie": -2375,
"date": "2023-04-27",
"name": "sporting"
},
{
"calorie": -420.5,
"date": "2023-05-15",
"name": "sleeping"
}
],
"age": 23,
"gender": "纯爷们",
"money": 3.1415926,
"name": "Bob"
}
Read JSON from Pipeline
Sometimes the data to be processed comes from the output of other commands. Just use the pipe character | to connect the two commands and then take it from the stdin.
$ curl -s https://jsonplaceholder.typicode.com/posts/1 | jf -i 4
Output:
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nrep..."
}
2. Minimize the JSON data
The -c option used to suppress all whitespace and newlines to compact the JSON data into a single line.
$ echo '{
"name": "alex",
"age": 21,
"items": [
"pen",
"phone"
]
}' | jf -c
Output:
{"name":"alex","age":21,"items":["pen","phone"]}
3. Extract Partial Content from JSON Data
jsonfmt uses both JMESPath and JSONPath as its query languages.
JMESPath (JSON Meta Language for Expression Path) is a query language introduced by AWS for processing JSON data. Among the many JSON query languages, JMESPath seems to be the most widely-used, fastest-growing, and highest-rated. Its syntax is more concise and more universal than jq, and its functionality is more powerful and feature-rich than JSONPath. Therefore, I prefer to use it as the primary JSON query language.
JMESPath can elegantly use simple syntax to extract part of the content from JSON data, and also compose the filtered data into a new object or array. The official JMESPath tutorial is here.
JMESPath Examples
-
Extract the first item of
actionsfrom example.json:$ jf -p 'actions[0]' test/example.json
Output:
{ "name": "eating", "calorie": 1294.9, "date": "2021-03-02" }
-
Filter all items with
calorie < 0fromactions.# Here, `0` means 0 is a number $ jf -p 'actions[?calorie<`0`]' test/example.json
Output:
[ { "name": "sporting", "calorie": -2375, "date": "2023-04-27" }, { "name": "sleeping", "calorie": -420.5, "date": "2023-05-15" } ]
-
Show all keys and the length of
actions.$ jf -p '{all_keys:keys(@), actions_len:length(actions)}' test/example.json
Output:
{ "all_keys": [ "name", "age", "gender", "money", "actions" ], "actions_len": 3 }
-
Sort the items in
actionsby theircalorievalue, and define the result as a new dictionary.$ jf -p 'sort_by(actions, &calorie)[].{foo: name, bar:calorie}' test/example.json
Output:
[ { "foo": "sporting", "bar": -2375 }, { "foo": "sleeping", "bar": -420.5 }, { "foo": "eating", "bar": 1294.9 } ]
JSONPath Examples
JSONPath was inspired by the design of XPath. Therefore, it can precisely locate any element in the JSON document through path expressions, similar to XPath, enabling efficient retrieval, filtering, and operation of complex nested data.
Unlike the tag hierarchical structure of XML, JSONPath specially handles JSON key-value pairs and arrays, allowing users to conveniently access multi-level object properties, iterate over objects and arrays, and filter data based on conditions.
Some queries that are difficult to handle with JMESPath can be easily achieved with JSONPath.
-
Filter all
namefields using relative paths:# Use -l to specify the query language as JSONPath $ jf -l jsonpath -p '$..name' test/example.json
Output:
[ "Bob", "eating", "sporting", "sleeping" ]
Querying TOML, XML and YAML
One of the powerful features of jsonfmt is that you can process TOML, XML and YAML in exactly the same way as JSON, and freely convert the result format. You can even process these four formats simultaneously in one command.
-
Read data from a toml file and output in YAML format
$ jf -p '{all_keys:keys(@), actions_len:length(actions)}' test/example.toml -f yaml
Output:
all_keys: - name - age - gender - money - actions actions_len: 3
-
Process three formats at once
$ jf -p 'actions[0]' test/example.json test/example.toml test/example.yaml
Output:
1. test/example.json { "name": "eating", "calorie": 1294.9, "date": "2021-03-02" } 2. test/example.toml name = "eating" calorie = 1294.9 date = "2021-03-02" 3. test/example.xml <?xml version="1.0" ?> <root> <name>eating</name> <calorie>1294.9</calorie> <date>2021-03-02</date> </root> 4. test/example.yaml name: eating calorie: 1294.9 date: '2021-03-02'
4. Format Conversion
jsonfmt supports processing JSON, TOML, XML and YAML formats. Each format can be converted to other formats by specifying the "-f" option.
-
nullis not supported in TOML. Therefore, allnullvalues will be deleted when converting from other formats to TOML. -
XML does not support multi-dimensional arrays. Therefore, if the original data contains multi-dimensional arrays, a wrong data will be generated during the conversion to XML format.
Example 1. JSON to YAML
$ jf test/example.json -f yaml
Output:
name: Bob
age: 23
gender: 纯爷们
money: 3.1415926
actions:
- name: eating
calorie: 1294.9
date: '2021-03-02'
- name: sporting
calorie: -2375
date: '2023-04-27'
- name: sleeping
calorie: -420.5
date: '2023-05-15'
Example 2. TOML to XML
$ jf test/example.toml -f xml
Output:
<?xml version="1.0" ?>
<root>
<name>Bob</name>
<age>23</age>
<gender>纯爷们</gender>
<money>3.1415926</money>
<actions>
<name>eating</name>
<calorie>1294.9</calorie>
<date>2021-03-02</date>
</actions>
<actions>
<name>sporting</name>
<calorie>-2375</calorie>
<date>2023-04-27</date>
</actions>
<actions>
<name>sleeping</name>
<calorie>-420.5</calorie>
<date>2023-05-15</date>
</actions>
</root>
5. Diff Comparison
In development, we often need to compare differences between some data or configurations. For example, compare the return results of an API when passing in different parameters, or compare the differences between system configuration files in different formats by operations personnel.
jsonfmt supports various diff-tools by default, such as diff, vimdiff, git, code, kdiff3, meld, and also supports WinMerge and fc on Windows, and other tools can also be supported through the -D option.
By default, jsonfmt will first check if git is installed on the computer. If git is available, jsonfmt will call git config --global diff.tool to read the configured diff-tool. If it's not set, it will use the default diff-tool of git for processing. If git is not available, it will search in the order of code, kdiff3, meld, vimdiff, diff, WinMerge, fc. If no available diff-tool is found, jsonfmt will exit with an error.
In DiffMode, jsonfmt will first format the data to be compared (at this time, the -s option will be automatically enabled), and save the result to a temporary file, and then call the specified tool for diff comparison.
Example 1: Compare two JSON files
$ jf -d test/example.json test/another.json
Output:
--- /tmp/.../jf-jjn86s7r_example.json 2024-03-23 18:22:00
+++ /tmp/.../jf-vik3bqsu_another.json 2024-03-23 18:22:00
@@ -3,21 +3,16 @@
{
"calorie": 1294.9,
"date": "2021-03-02",
- "name": "eating"
+ "name": "thinking"
},
{
- "calorie": -2375,
- "date": "2023-04-27",
- "name": "sporting"
- },
- {
"calorie": -420.5,
"date": "2023-05-15",
"name": "sleeping"
}
],
"age": 23,
- "gender": "纯爷们",
+ "gender": "male",
"money": 3.1415926,
- "name": "Bob"
+ "name": "Tom"
}
Example 2: Specify diff-tool with -D
The -D DIFFTOOL option can specify a diff comparison tool. As long as its command format matches command [options] file1 file2, it doesn't matter whether it's in jsonfmt's default supported tool list or not.
$ jf -D sdiff test/example.json test/another.json
Output:
{ {
"actions": [ "actions": [
{ {
"calorie": 1294.9, "calorie": 1294.9,
"date": "2021-03-02", "date": "2021-03-02",
"name": "eating" | "name": "thinking"
}, },
{ {
"calorie": -2375, <
"date": "2023-04-27", <
"name": "sporting" <
}, <
{ <
"calorie": -420.5, "calorie": -420.5,
"date": "2023-05-15", "date": "2023-05-15",
"name": "sleeping" "name": "sleeping"
} }
], ],
"age": 23, "age": 23,
"gender": "纯爷们", | "gender": "male",
"money": 3.1415926, "money": 3.1415926,
"name": "Bob" | "name": "Tom"
} }
Example 3: Specify options for the selected tool
If you need to pass parameters to the diff-tool, you can use -D 'DIFFTOOL OPTIONS'.
$ jf -D 'diff --ignore-case --color=always' test/example.json test/another.json
Output:
6c6
< "name": "eating"
---
> "name": "thinking"
9,13d8
< "calorie": -2375,
< "date": "2023-04-27",
< "name": "sporting"
< },
< {
20c15
< "gender": "纯爷们",
---
> "gender": "male",
22c17
< "name": "Bob"
---
> "name": "Tom"
Example 4: Compare data in different formats
For data from different sources, their formats, indentation, and key order may be different. In this case, you can use -i and -f together for diff comparison.
$ jf -d -i 4 -f toml test/example.toml test/another.json
Output:
--- /var/.../jf-qw9vm33n_example.toml 2024-03-23 18:29:17
+++ /var/.../jf-dqb_fl4x_another.json 2024-03-23 18:29:17
@@ -1,18 +1,13 @@
age = 23
-gender = "纯爷们"
+gender = "male"
money = 3.1415926
-name = "Bob"
+name = "Tom"
[[actions]]
calorie = 1294.9
date = "2021-03-02"
-name = "eating"
+name = "thinking"
[[actions]]
-calorie = -2375
-date = "2023-04-27"
-name = "sporting"
-
-[[actions]]
calorie = -420.5
date = "2023-05-15"
name = "sleeping"
6. Handle Large JSON Data Conveniently
Very often, JSON data from program interfaces is very large, which makes it difficult for us to read, debug, and process. jsonfmt provides four ways to handle large JSON data:
- Use JMESPath or JSONPath to read part of the content (already covered in the previous section)
- Use pager mode to view larger JSON data
- Show an overview of large JSON data
- Copy the processing result to the clipboard
Use pager mode to view larger JSON data
Pager mode is similar to the more command. When the JSON data is too large to be fully displayed in the window area, jsonfmt will automatically display the result in pager mode.
The operations in pager mode are the same as the more command:
| Key | Description |
|---|---|
| j | Move forward one line |
| k | Move backward one line |
| f or ctrl+f | Move forward one page |
| b or ctrl+b | Move backward one page |
| g | Jump to the top of the page |
| G | Jump to the bottom of the page |
| / | Search mode |
| q | Exit pager mode |
The return value of this API below is a large JSON data, you can paste this command into the terminal to try the pager mode:
$ curl -s https://jsonplaceholder.typicode.com/users | jf
Show an overview of large JSON data
Sometimes we only want to see an overview of the JSON data without caring about the details. In this case, you can use the -o option. It will clear the sublists in the JSON, and replace the strings to "..." to show the overview.
If the root node of the JSON data is a list, only its first child element will be preserved in the overview.
$ jf -o test/example.json
Output:
{
"name": "...",
"age": 23,
"gender": "...",
"money": 3.1415926,
"actions": []
}
Copy the processing result to the clipboard
If you want to paste the processed result into a file, but the output printed in the terminal exceeds one page, it may be difficult to copy. In this case, you can use the -C option to automatically copy the result to the clipboard.
$ jf -C test/example.json
After completing the above operation, you can use ctrl+v or cmd+v to paste the result into other documents.
When processing multiple targets at the same time, such as: jf -C file1 file2 file3 ..., jsonfmt will copy the processing results of all files to the clipboard, with two newline characters '\n\n' separating multiple results.
7. Modify Values in Input Data
When you need to change some content in the input document, use the --set and --pop options.
The format is --set 'key=value'. If you need to modify multiple values, you can separate them with ;, like this: --set 'k1=v1;k2=v2'. If the key-value pair does not exist, it will be added.
For items in a list, use key[i] or key.i to specify. If the index is greater or equal to the number of elements, the value will be appended.
Add key-value pairs
# Add country = China, and append an item to actions
$ jf --set 'country=China; actions[3]={"name": "drinking"}' test/example.json
Output:
{
"name": "Bob",
"age": 23,
"gender": "纯爷们",
"money": 3.1415926,
"actions": [
{
"name": "eating",
"calorie": 1294.9,
"date": "2021-03-02"
},
{
"name": "sporting",
"calorie": -2375,
"date": "2023-04-27"
},
{
"name": "sleeping",
"calorie": -420.5,
"date": "2023-05-15"
},
{
"name": "drinking"
}
],
"country": "China"
}
Modify values
# Modify money and actions[1]["name"]
$ jf --set 'money=1000; actions[1].name=swim' test/example.json
Output:
{
"name": "Bob",
"age": 23,
"gender": "纯爷们",
"money": 1000,
"actions": [
{
"name": "eating",
"calorie": 1294.9,
"date": "2021-03-02"
},
{
"name": "swim",
"calorie": -2375,
"date": "2023-04-27"
},
{
"name": "sleeping",
"calorie": -420.5,
"date": "2023-05-15"
}
]
}
Delete key-value pairs
# Delete gender and actions[1]
$ jf --pop 'gender; actions[1]' test/example.json
Output:
{
"name": "Bob",
"age": 23,
"money": 3.1415926,
"actions": [
{
"name": "eating",
"calorie": 1294.9,
"date": "2021-03-02"
},
{
"name": "sleeping",
"calorie": -420.5,
"date": "2023-05-15"
}
]
}
Of course, you can also use --set and --pop at the same time:
jf --set 'skills=["Django","Flask"];money=1000' --pop 'gender;actions[1]' test/example.json
8. Output to File
jsonfmt does not provide a dedicated option to write the processing result to a file. Because you can easily handle this by using the terminal's redirection symbol >, which is supported on both Linux and Windows.
$ jf -si 4 test/example.json > formatted.json
If you need to overwrite the processed result to the original file, you can use the -O option:
# Sort by object keys, set indentation to 4 spaces, set the name value to Alex, and write the final result to the original file
$ jf -s -i 4 --set 'name=Alex' -O test/example.json
TODO
- Add URL support to directly compare data from two APIs
- Add INI format support
- Add merge mode to combine multiple JSON or other formatted data into one
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
