CLI for fast, flexbile concatenation of tabular data using polars.
Project description
joinem provides a CLI for fast, flexbile concatenation of tabular data using polars
- Free software: MIT license
- Repository: https://github.com/mmore500/joinem
- Documentation: https://github.com/mmore500/joinem/blob/master/README.md
Install
python3 -m pip install joinem
Features
- Lazily streams I/O to expeditiously handle numerous large files.
- Supports CSV and parquet input files.
- Due to current polars limitations, JSON and feather files are not supported.
- Input formats may be mixed.
- Supports output to CSV, JSON, parquet, and feather file types.
- Allows mismatched columns and/or empty data files with
--how diagonal
and--how diagonal_relaxed
. - Provides a progress bar with
--progress
. - Add programatically-generated columns to output.
Example Usage
Pass input filenames via stdin, one filename per line.
find path/to/*.parquet path/to/*.csv | python3 -m joinem out.parquet
Output file type is inferred from the extension of the output file name. Supported output types are feather, JSON, parquet, and csv.
find -name '*.parquet' | python3 -m joinem out.json
Use --progress
to show a progress bar.
ls -1 path/{*.csv,*.pqt} | python3 -m joinem out.csv --progress
If file columns may mismatch, use --how diagonal
.
find path/to/ -name '*.csv' | python3 -m joinem out.csv --how diagonal
If some files may be empty, use --how diagonal_relaxed
.
To run via Singularity/Apptainer,
ls -1 *.csv | singularity run docker://ghcr.io/mmore500/joinem out.feather
Add literal value column to output.
ls -1 *.csv | python3 -m joinem out.csv --with-column 'pl.lit(2).alias("two")'
Alias an existing column in the output.
ls -1 *.csv | python3 -m joinem out.csv --with-column 'pl.col("a").alias("a2")'
Apply regex on source datafile paths to create new column in output.
ls -1 path/to/*.csv | python3 -m joinem out.csv \
--with-column 'pl.lit(filepath).str.replace(r".*?([^/]*)\.csv", r"${1}").alias("filename stem")'
Read data from stdin and write data to stdout.
cat foo.csv | python3 -m joinem "/dev/stdout" --stdin --output-filetype csv --input-filetype csv
API
usage: __main__.py [-h] [--version] [--progress]
[--how {vertical,horizontal,diagonal,diagonal_relaxed}]
output_file
Concatenate CSV and/or parquet tabular data files.
positional arguments:
output_file Output file name
options:
-h, --help show this help message and exit
--version show program's version number and exit
--progress Show progress bar
--stdin Read data from stdin
--with-column WITH_COLUMNS
Expression to be evaluated to add a column, as access to
each datafile's filepath as `filepath` and polars as
`pl`. Example:
'pl.lit(filepath).str.replace(r".*/(.*)\.csv", r"${1}")
.alias("filename stem")'
--how {vertical,horizontal,diagonal,diagonal_relaxed}
How to concatenate frames. See <https://docs.pola.rs/py-
polars/html/reference/api/polars.concat.html> for more information.
--input-filetype INPUT_FILETYPE
Filetype of input. Otherwise, inferred.
Example: csv, parquet, json, feather
--output-filetype OUTPUT_FILETYPE
Filetype of output. Otherwise, inferred.
Example: csv, parquet
Provide input filenames via stdin. Example: find path/to/ -name '*.csv' | python3 -m joinem
out.csv
Citing
If joinem contributes to a scholarly work, please cite it as
Matthew Andres Moreno. (2024). mmore500/joinem. Zenodo. https://doi.org/10.5281/zenodo.10701182
@software{moreno2024joinem,
author = {Matthew Andres Moreno},
title = {mmore500/joinem},
month = feb,
year = 2024,
publisher = {Zenodo},
doi = {10.5281/zenodo.10701182},
url = {https://doi.org/10.5281/zenodo.10701182}
}
And don't forget to leave a star on GitHub!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file joinem-0.3.2.tar.gz
.
File metadata
- Download URL: joinem-0.3.2.tar.gz
- Upload date:
- Size: 6.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb8b639bdc365eee4394efd5835298abfda1ed92146eca12433dcbea5a660787 |
|
MD5 | 7ca745af1fb909f287a3ddb7358c5382 |
|
BLAKE2b-256 | 3127d8e010ff2ecb5329dbf35c2cc3a102156f842409fcbd7da322fac3802fba |
File details
Details for the file joinem-0.3.2-py2.py3-none-any.whl
.
File metadata
- Download URL: joinem-0.3.2-py2.py3-none-any.whl
- Upload date:
- Size: 6.3 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a9454d1b70efe6f6fccfa6552dc14a309a58b828cffd3c34dd50adc9a3d31b7 |
|
MD5 | d69724e23162a370a3010f60ad10f85c |
|
BLAKE2b-256 | 20b85fd859b14cebd08e5bd92f73bbb45c12dba159bc070fe576250752c386b3 |