Skip to main content

jlpipe

Project description

jlpipe

Introduction

Project owner: Jay Young dev@yjmade.net

Installation

pip install jlpipe

Usage

full commands list:

  • dynamojson: convert json from dynamodb format to normal
  • jsonkey: return json keys of each line
  • jsonallkeys: show statistics of keys of json lines
  • jsonselect: select fields from json
  • json2csv: convert json to csv
  • json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
  • parquet2json: convert parquet file to json format
  • jsondecompress: decompress gzip compressed field
  • arrayunpack: similiar to postgres unnest, unnest a array to lines
  • pgcopy: similiar to pgfutter, but it use plain text to copy
  • parallel_split: similiar to parallel --pipe, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent

$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent

More usage reference to the command help text

command --help

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jlpipe-1.0.0.tar.gz (6.6 kB view details)

Uploaded Source

File details

Details for the file jlpipe-1.0.0.tar.gz.

File metadata

  • Download URL: jlpipe-1.0.0.tar.gz
  • Upload date:
  • Size: 6.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for jlpipe-1.0.0.tar.gz
Algorithm Hash digest
SHA256 99044cfe4c5da8d6ca5e9b5e713756674a6f85364552e5aa6b1e57c12172a371
MD5 e5f7ecc335eb061b062070efceb99f15
BLAKE2b-256 713a6db0581c065e89a8555bb7b385724d4a8c085719caa4ff16cf288e9963bc

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page