A small library for taking the transpose of arbitrarily large .csvs
Project description
bigcsv: A small Python library to manipulate large csv files that can't fit in memory.
Transposition
bigcsv
allows for easy calculation of csv transposes, even when the csv is much too large to fit in memory.
Converting to h5ad
If data is purely numeric, it is much more efficient to store in in h5ad
(readable by AnnData
), which uses the amazing HDF5 format under-the-hood.
Installation
To install, run pip install bigcsv
How to use
All operations are method of the BigCSV
class, which contains metadata information used to do all calculations.
from bigcsv import BigCSV
obj = BigCSV(
file='massive_dataset.csv',
chunksize=400, # Number of rows to read in at each iteration
# leave as default
# insep=',',
# outsep=',',
# chunksize=400,
# save_chunks=False,
# quiet=False,
)
obj.to_h5ad(outfile='converted.h5ad')
# Or maybe we want to keep as csv, but transpose it (in the case of non-numerical data)
obj.transpose(outfile='dataset_T.csv')
Documentation
bigcsv.BigCSV
Class containing methods for manipulating csvs.
Parameters:
file
: Path to input file
outfile
: Path to output file (transposed input file)
insep=','
: Input separator for delimited file, by default is ,
outsep=','
: Output separator for delimited file (in the case of csv --> csv operations)
chunksize=400
: Number of lines per iteration
save_chunks
: To save intermediate chunks or not
chunkfolder=None
: Optional, Path to chunkfolder
quiet=False
: Boolean indicating whether to print progress or not
bigcsv.BigCSV.transpose_csv
Parameters:
outfile=None
: Ouput file to write to, or if specified in initialization, writes to that file name
bigcsv.BigCSV.to_h5ad
Parameters
outfile=None
: Ouput file to write to, or if specified in initialization, writes to that file name
sparsify: bool=False
: Sparsify rows in h5 matrix
compression: str='infer'
: Compression format of input csv, if compressed. Probably just leave to infer unless the filename is weird.
lines: int=None
: Number of lines in the file. If you know a priori, this saves some time. Also cannot be calculated for compressed files.
dtype: Any=None
: dtype of entries of input matrix
index_col: str=None
: Column of input csv to use as index, if any.
index: bool=True
: Save index when converting to h5ad.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bigcsv-1.0.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39b7d45571e6d3df8dd669209eb7c2d356f64b21bd51c1a436c642da77061f9e |
|
MD5 | a461804caeaaee6768e2703c8c21c358 |
|
BLAKE2b-256 | 446dfb8169f902f13ce0b1b9aa4e3a55fbe20867b774c764fa818c67ce282c84 |