jlpipe
Project description
jlpipe
Introduction
Project owner: Jay Young dev@yjmade.net
Installation
pip install jlpipe
Usage
full commands list:
- dynamojson: convert json from dynamodb format to normal
- jsonkey: return json keys of each line
- jsonallkeys: show statistics of keys of json lines
- jsonselect: select fields from json
- json2csv: convert json to csv
- json2pgtext: convert json to postgres plain text then can use pgcopy to import to database
- parquet2json: convert parquet file to json format
- jsondecompress: decompress gzip compressed field
- arrayunpack: similiar to postgres unnest, unnest a array to lines
- pgcopy: similiar to pgfutter, but it use plain text to copy
- parallel_split: similiar to
parallel --pipe
, start multiple worker ahead, and cycle each line of input and route to each worker, then forword workers output to its stdout, not garentee order.
$ cat *.json|parallel_split dynamojson|jsonselect data=. patent_id description=patent_description|jsondecompress description|json2pgtext -a|pgcopg all_patent
$ cat *.json|jsonallkeys -l 10000 -r >/dev/null
$ ls *.parquet|parquet2json|json2pgtext|pgcopg all_patent
More usage reference to the command help text
command --help
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
jlpipe-1.1.0.tar.gz
(7.8 kB
view details)
File details
Details for the file jlpipe-1.1.0.tar.gz
.
File metadata
- Download URL: jlpipe-1.1.0.tar.gz
- Upload date:
- Size: 7.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f349fb14f590b88644d301a4ef6a28459604812fe656784ad546c35d657ab96f |
|
MD5 | 1cd003cf446ff93701ea675b37bfd62b |
|
BLAKE2b-256 | a8dda574e7daa42abc67c147d276dbdc46035e5b6b4e9b0da8ff11576e1faf55 |