qic 1.1.2

data(JSON/XML/YAML) command line query tools with interactive mode.

qic - Query In Console

JSON/YAML/XML comand line query tool with support of interactive mode.

By design, it only process a few common situations and there's a lot compromise which may need some attention. It's not intending to embed a fully working syntax/sementic engine and only provide a few "sugar" features. For too complicated situation, other than introducing very long single line expression, please try code blocks where you can use any python syntax.

For detail, look into the Wiki

---

Installation

it's available from pip.

pip install qic --upgrade

you can run it as,

python -mqic

or just,

qic

note, if you install it as non-root user, it's placed in ~/.local/bin folder, so either refer the full path or add to your PATH.

note 2, by default, qic read output from pipe line or a file specified by -f, so, use qic -h for help info. If you just type qic followed by enter, you will find it "hang" there:)

[~]$ sudo python3 -mpip install qic --upgrade
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Collecting qic
  Using cached qic-1.0.9.tar.gz (14 kB)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/site-packages (from qic) (0.2.5)
Requirement already satisfied: pyyaml in /usr/local/lib64/python3.9/site-packages (from qic) (5.4.1)
Requirement already satisfied: xmltodict in /usr/local/lib/python3.9/site-packages (from qic) (0.12.0)
Requirement already satisfied: dicttoxml in /usr/local/lib/python3.9/site-packages (from qic) (1.7.4)
Requirement already satisfied: pygments in /usr/local/lib/python3.9/site-packages (from qic) (2.9.0)
Requirement already satisfied: prompt_toolkit in /usr/local/lib/python3.9/site-packages (from qic) (3.0.19)
Using legacy 'setup.py install' for qic, since package 'wheel' is not installed.
Installing collected packages: qic
    Running setup.py install for qic ... done
Successfully installed qic-1.0.9
[~]$ 
[~]$ 
[~]$ which qic
/usr/local/bin/qic
[~]$ 
[~]$ qic -h
usage: qic [-h] [-f INFILE] [-t SRCTYPE] [-i INDENT] [-l ROWS] [-m MODULES] [-K KEYS_INCLUDED] [-E KEYS_EXCLUDED] [-F] [-s] [-I] [-p] [-c]
           [-C] [-X]
           [code]

positional arguments:
  code                  code to compile. may be a file.

optional arguments:
  -h, --help            show this help message and exit
  -f INFILE, --infile INFILE
                        input file
  -t SRCTYPE, --srctype SRCTYPE
                        JSON,YAML or XML
  -i INDENT, --indent INDENT
                        how many spaces for indent. default 4.
  -l ROWS, --rows ROWS  use this to shrink each list included. useful for DS including too many records.
  -m MODULES, --modules MODULES
                        import modules.
  -K KEYS_INCLUDED, --keys KEYS_INCLUDED
                        only keep these keys.
  -E KEYS_EXCLUDED, --nokeys KEYS_EXCLUDED
                        these keys should be excluded.
  -F, --functionize     wrap code into an internal function
  -s, --rawstr          output raw stings for easy grep
  -I, --interactive     interactive mode
  -p, --plain           force no color code
  -c, --compact         dump data structure in compact mode
  -C, --keepcolor       do not remove ANSI color code from input stream.
  -X, --debug           debug mode

Input Stream

please be noted, due to the lack of colored text support from markdown language, the terminal output is much less impressive than in real world. good thing is that there's still minimum support of font styles which make it different than plain console output. so if not extremely necessary, I will just paste what I got from "copy as HTML" than capturing a lot screenshots.

here, we have a test json file as below,

[{
  "_id": {
    "$oid": "5968dd23fc13ae04d9000001"
  },
  "product_name": "sildenafil citrate",
  "supplier": "Wisozk Inc",
  "quantity": 261,
  "unit_cost": "$10.47"
}, {
  "_id": {
    "$oid": "5968dd23fc13ae04d9000002"
  },
  "product_name": "Mountain Juniperus ashei",
  "supplier": "Keebler-Hilpert",
  "quantity": 292,
  "unit_cost": "$8.74"
}, {
  "_id": {
    "$oid": "5968dd23fc13ae04d9000003"
  },
  "product_name": "Dextromathorphan HBr",
  "supplier": "Schmitt-Weissnat",
  "quantity": 211,
  "unit_cost": "$20.53"
}]

we can feed qic in a few ways,

1. through a pipe
2. specified by -f
3. type or paste ad-hoc text

[yx@mtp qic]$cat test/s1.json  | qic 
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]

[yx@mtp qic]$
[yx@mtp qic]$qic -f test/s1.json 
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]

in below example, JSON is pasted directly, then followed by a CTL+D to end the input.
we can notice the first part is of "plain format" while the QIC output has some style -- it's actually much more beautiful in a terminal supporting ansi colors.

[yx@mtp qic]$qic
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]

[yx@mtp qic]$

Access

Qic is made by python and it supports python syntax naturally.

_ means doc root

[yx@mtp qic]$qic -f test/s1.json "_"
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]

Access List

[yx@mtp qic]$qic -f test/s1.json "_[0]"
{
  "_id": {
    "$oid": "5968dd23fc13ae04d9000001"
  },
  "product_name": "sildenafil citrate",
  "supplier": "Wisozk Inc",
  "quantity": 261,
  "unit_cost": "$10.47"
}

[yx@mtp qic]$qic -f test/s1.json "_[0:2]"
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  }
]

Access DICT

use python "['key']" syntax

[yx@mtp qic]$qic -f test/s1.json "_[0]['supplier']"
Wisozk Inc

[yx@mtp qic]$qic -f test/s1.json "_[0]['unit_cost']"
$10.47

use .

[yx@mtp qic]$qic -f test/s1.json "_[0].supplier"
Wisozk Inc

[yx@mtp qic]$qic -f test/s1.json "_[0].unit_cost"
$10.47

what if we want multiple keys?

1. use "{}" key select syntax we can see with -X option, qic actually has translated the brief syntax into standard python way.

[yx@mtp qic]$qic -f test/s1.json "_[0].{unit_cost,quantity}"
{
  "unit_cost": "$10.47",
  "quantity": 261
}

[yx@mtp qic]$qic -f test/s1.json "_[0].{unit_cost,quantity}" -X
# run : 
{'unit_cost':_[0]['unit_cost'],'quantity':_[0]['quantity']}
{
  "unit_cost": "$10.47",
  "quantity": 261
}

2. use -K keys including options.

[yx@mtp qic]$qic -f test/s1.json "_[0]" -K unit_cost,quantity
{
  "quantity": 261,
  "unit_cost": "$10.47"
}

List Expansion

Let's check python list comprehension first,

[yx@mtp qic]$qic -f test/s1.json "[i['quantity'] for i in _]"
[
  261,
  292,
  211
]

[yx@mtp qic]$qic -f test/s1.json "[{'quantity':i['quantity'],'supplier':i['supplier']} for i in _]"
[
  {
    "quantity": 261,
    "supplier": "Wisozk Inc"
  },
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  },
  {
    "quantity": 211,
    "supplier": "Schmitt-Weissnat"
  }
]

use [] to save some typing

[yx@mtp qic]$qic -f test/s1.json "_[].quantity"
[
  261,
  292,
  211
]

[yx@mtp qic]$qic -f test/s1.json "_[].{quantity,supplier}"
[
  {
    "quantity": 261,
    "supplier": "Wisozk Inc"
  },
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  },
  {
    "quantity": 211,
    "supplier": "Schmitt-Weissnat"
  }
]

run in debug mode to check expanded code

[yx@mtp qic]$qic -f test/s1.json "_[].quantity" -X
# run : 
[ _hcm['quantity'] for _hcm in _ ]
[
  261,
  292,
  211
]

[yx@mtp qic]$
[yx@mtp qic]$qic -f test/s1.json "_[].{quantity,supplier}" -X
# run : 
[ {'quantity':_oeu['quantity'],'supplier':_oeu['supplier']} for _oeu in _ ]
[
  {
    "quantity": 261,
    "supplier": "Wisozk Inc"
  },
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  },
  {
    "quantity": 211,
    "supplier": "Schmitt-Weissnat"
  }
]

note, slice is also supported

[yx@mtp qic]$qic -f test/s1.json "_[:2].{quantity,supplier}" -X
# run : 
[ {'quantity':_jgr['quantity'],'supplier':_jgr['supplier']} for _jgr in _[:2] ]
[
  {
    "quantity": 261,
    "supplier": "Wisozk Inc"
  },
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  }
]

[yx@mtp qic]$
[yx@mtp qic]$qic -f test/s1.json "_[1:].{quantity,supplier}" -X
# run : 
[ {'quantity':_chd['quantity'],'supplier':_chd['supplier']} for _chd in _[1:] ]
[
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  },
  {
    "quantity": 211,
    "supplier": "Schmitt-Weissnat"
  }
]

[yx@mtp qic]$
[yx@mtp qic]$qic -f test/s1.json "_[:].{quantity,supplier}" -X
# run : 
[ {'quantity':_wax['quantity'],'supplier':_wax['supplier']} for _wax in _ ]
[
  {
    "quantity": 261,
    "supplier": "Wisozk Inc"
  },
  {
    "quantity": 292,
    "supplier": "Keebler-Hilpert"
  },
  {
    "quantity": 211,
    "supplier": "Schmitt-Weissnat"
  }
]

Interactive mode

-I will bring you to interactive mode.

interactive mode will read stdin so you can not pipe json content into QiC, use -f instead.

the Input Json will be parsed and all the keys are used for word completion.

you may have noticed, in dot(.) syntax, you can ignore the case since QiC will try to find the right one for you. Only in the case you have defined some different keys only diff from each other with case -- specify the right one.

in below case, unfortunately the color is missing in markdown. when typed "", a few internal helper functions starts with "" were prompted.

yx@mtp qic]$qic -f test/s1.json -I
[qic] $ _
          _          
          _fl        
          _flatlist  
          _id        
          _j         
          _l         
          _l2pt      

here, when typed "q", "quantity" was promoted.

[qic] $ _[0].q
               quantity  


in case you want to write some complicated logic with multiple lines, use "'''" to start and end your block.

[qic] $
[qic] $ '''
[qic] $ sum=0
[qic] $ for i in _ :
[qic] $     cost = float(i.unit_cost.replace("$",""))
[qic] $     sum += int(i.quantity) * cost
[qic] $ print("total = ", sum)
[qic] $ '''
total =  9616.58
[qic] $
[qic] $

---

quit

just type "\q"

test json file

[yx@mtp qic]$qic -f test/s1.json 
[
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000001"
    },
    "product_name": "sildenafil citrate",
    "supplier": "Wisozk Inc",
    "quantity": 261,
    "unit_cost": "$10.47"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000002"
    },
    "product_name": "Mountain Juniperus ashei",
    "supplier": "Keebler-Hilpert",
    "quantity": 292,
    "unit_cost": "$8.74"
  },
  {
    "_id": {
      "$oid": "5968dd23fc13ae04d9000003"
    },
    "product_name": "Dextromathorphan HBr",
    "supplier": "Schmitt-Weissnat",
    "quantity": 211,
    "unit_cost": "$20.53"
  }
]