Skip to main content

jlpipe

Project description

jlpipe

Introduction

Project owner: Jay Young dev@yjmade.net

Installation

pip install jlpipe

Usage

full commands list:

  • dynamojson: convert json from dynamodb format to normal
  • jsonkey: return json keys of each line
  • jsonallkeys: show statistics of keys of json lines
  • jsonselect: select fields from json
  • json2csv: convert json to csv
  • json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
  • parquet2json: convert parquet file to json format
  • jsondecompress: decompress gzip compressed field
  • arrayunpack: similiar to postgres unnest, unnest a array to lines
  • pgcopy: similiar to pgfutter, but it use plain text to copy
  • parallel_split: similiar to parallel --pipe, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent

$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent

More usage reference to the command help text

command --help

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jlpipe-1.1.0.tar.gz (7.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page