Skip to main content

jlpipe

Project description

jlpipe

Introduction

Project owner: Jay Young dev@yjmade.net

Installation

pip install jlpipe

Usage

full commands list:

  • dynamojson: convert json from dynamodb format to normal
  • jsonkey: return json keys of each line
  • jsonallkeys: show statistics of keys of json lines
  • jsonselect: select fields from json
  • json2csv: convert json to csv
  • json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
  • parquet2json: convert parquet file to json format
  • jsondecompress: decompress gzip compressed field
  • arrayunpack: similiar to postgres unnest, unnest a array to lines
  • pgcopy: similiar to pgfutter, but it use plain text to copy
  • parallel_split: similiar to parallel --pipe, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent

$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent

More usage reference to the command help text

command --help

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jlpipe-1.1.0.tar.gz (7.8 kB view details)

Uploaded Source

File details

Details for the file jlpipe-1.1.0.tar.gz.

File metadata

  • Download URL: jlpipe-1.1.0.tar.gz
  • Upload date:
  • Size: 7.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for jlpipe-1.1.0.tar.gz
Algorithm Hash digest
SHA256 f349fb14f590b88644d301a4ef6a28459604812fe656784ad546c35d657ab96f
MD5 1cd003cf446ff93701ea675b37bfd62b
BLAKE2b-256 a8dda574e7daa42abc67c147d276dbdc46035e5b6b4e9b0da8ff11576e1faf55

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page