Skip to main content


Project description



Project owner: Jay Young


pip install jlpipe


full commands list:

  • dynamojson: convert json from dynamodb format to normal
  • jsonkey: return json keys of each line
  • jsonallkeys: show statistics of keys of json lines
  • jsonselect: select fields from json
  • json2csv: convert json to csv
  • json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
  • parquet2json: convert parquet file to json format
  • jsondecompress: decompress gzip compressed field
  • arrayunpack: similiar to postgres unnest, unnest a array to lines
  • pgcopy: similiar to pgfutter, but it use plain text to copy
  • parallel_split: similiar to parallel --pipe, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent

$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent

More usage reference to the command help text

command --help

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jlpipe-1.1.0.tar.gz (7.8 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page