Pea Kina aka 'Giant Panda'
Wrapper around pandas library, which detects separator, encoding
and type of the file. It allows to get a group of files with a matching pattern (python or glob regex).
It can read both local and remote files (HTTP/HTTPS, FTP/FTPS/SFTP or S3/S3N/S3A).
The supported file types are csv, excel, json, parquet and xml.
:information_source: If the desired type is not yet supported, feel free to open an issue or to directly open a PR with the code !
Please, read the documentation for more information
Installation
pip install peakina
Usage
Considering a file file.csv
a;b
0;0
0;1
Just type
>>> import peakina as pk
>>> pk.read_pandas('file.csv')
a b
0 0 0
1 0 1
Or files on a FTPS server:
- my_data_2015.csv
- my_data_2016.csv
- my_data_2017.csv
- my_data_2018.csv
You can just type
>>> pk.read_pandas('ftps://<path>/my_data_\\d{4}\\.csv$', match='regex', dtype={'a': 'str'})
a b __filename__
0 '0' 0 'my_data_2015.csv'
1 '0' 1 'my_data_2015.csv'
2 '1' 0 'my_data_2016.csv'
3 '1' 1 'my_data_2016.csv'
4 '3' 0 'my_data_2017.csv'
5 '3' 1 'my_data_2017.csv'
6 '4' 0 'my_data_2018.csv'
7 '4' 1 'my_data_2018.csv'
Using cache
You may want to keep the last result in cache, to avoid downloading and extracting the file if it didn't change:
>>> from peakina.cache import Cache
>>> cache = Cache.get_cache('memory') # in-memory cache
>>> df = pk.read_pandas('file.csv', expire=3600, cache=cache)
In this example, the resulting dataframe will be fetched from the cache, unless file.csv modification time has changed on disk, or unless the cache is older than 1 hour.
For persistent caching, use: cache = Cache.get_cache('hdf', cache_dir='/tmp')
Use only downloading feature
If you just want to download a file, without converting it to a pandas dataframe:
>>> uri = 'https://i.imgur.com/V9x88.jpg'
>>> f = pk.fetch(uri)
>>> f.get_str_mtime()
'2012-11-04T17:27:14Z'
>>> with f.open() as stream:
... print('Image size:', len(stream.read()), 'bytes')
...
Image size: 60284 bytes
Installation on macOS M1 chipset
install everything
brew install hdf5 snappy
HDF5_DIR="/opt/homebrew/Cellar/hdf5/1.12.1/" CPPFLAGS="-I/opt/homebrew/Cellar/snappy/1.1.9/include -L/opt/homebrew/Cellar/snappy/1.1.9/lib" poetry install
For more details, here is what is needed:
install pytables
brew install hdf5
HDF5_DIR="/opt/homebrew/Cellar/hdf5/1.12.1/" poetry run pip install tables
install python-snappy
brew install snappy
CPPFLAGS="-I/opt/homebrew/Cellar/snappy/1.1.9/include -L/opt/homebrew/Cellar/snappy/1.1.9/lib" poetry run pip install python-snappy
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file peakina-0.9.9.tar.gz.
File metadata
- Download URL: peakina-0.9.9.tar.gz
- Upload date:
- Size: 21.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/2.4.1 CPython/3.10.20 Linux/6.17.0-1020-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
221c01610b078d74e5338ac95e79e1d84d124defbafdd705401f549d13773062
|
|
| MD5 |
51cc20e6a6ea138914785b2776550ae1
|
|
| BLAKE2b-256 |
bb9333859b3540113804b316923e75c6c46ee7399e1643f2b54c633410182bf7
|
File details
Details for the file peakina-0.9.9-py3-none-any.whl.
File metadata
- Download URL: peakina-0.9.9-py3-none-any.whl
- Upload date:
- Size: 27.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/2.4.1 CPython/3.10.20 Linux/6.17.0-1020-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
96217744d92f992961f4d4320960e14216242366466ca11a520dd4a29b8fde8e
|
|
| MD5 |
cbb651fafb2f65467b9f99b61bf0e00e
|
|
| BLAKE2b-256 |
2b58c4f4b83f78496d236cd84b36ce001595d8eba28594b86d9bd739cbb5ac74
|
