Skip to main content

jlpipe

Project description

jlpipe

Introduction

Project owner: Jay Young dev@yjmade.net

Installation

pip install jlpipe

Usage

full commands list:

  • dynamojson: convert json from dynamodb format to normal
  • jsonkey: return json keys of each line
  • jsonallkeys: show statistics of keys of json lines
  • jsonselect: select fields from json
  • json2csv: convert json to csv
  • json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
  • parquet2json: convert parquet file to json format
  • jsondecompress: decompress gzip compressed field
  • arrayunpack: similiar to postgres unnest, unnest a array to lines
  • pgcopy: similiar to pgfutter, but it use plain text to copy
  • parallel_split: similiar to parallel --pipe, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent

$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent

More usage reference to the command help text

command --help

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for jlpipe, version 1.1.0
Filename, size File type Python version Upload date Hashes
Filename, size jlpipe-1.1.0.tar.gz (7.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page