Skip to main content

A fast and memory efficient way to load large CSV files (Timeseries data) into Pandas

Project description

fast_csv_loader.py

The csv_loader function efficiently loads a partial portion of a large CSV file containing time-series data into a pandas DataFrame.

The function allows:

  • Loading the last N lines from the end of the file.
  • Loading the last N lines from a specific date.

It can load any type of time-series (both timezone aware and Naive) and daily or intraday data.

It is useful for loading large datasets that may not fit entirely into memory. It also improves program execution time, when iterating or loading a large number of CSV files.

Supports Python >= 3.8

Install

pip install fast-csv-loader

Documentation

https://bennythadikaran.github.io/fast_csv_loader/

Performance

Loading a portion of a large file is significantly faster than loading the entire file in memory. Files used in the test were not particularly large. You may need to tweak the chunk_size parameter for your use case.

It is slower for smaller files or if you're loading nearly the entire portion of the file.

I chose a 6Kb chunk size based on testing with my specific requirements. Your requirements may differ.

csv_loader vs pandas.read_csv

Execution time - Last 160 lines

Execution time - Last 160 lines upto 1st Jan 2023

To run this performance test.

py tests/run.py

At the minimum, the CSV file must contain a Date and another column with newline chars at the end to correctly parse and load.

Date,Price\n
2023-12-01,200\n

Unit Test

To run the test:

py tests/test_csv_loader.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fast_csv_loader-1.0.1.tar.gz (17.1 kB view details)

Uploaded Source

Built Distribution

fast_csv_loader-1.0.1-py3-none-any.whl (17.0 kB view details)

Uploaded Python 3

File details

Details for the file fast_csv_loader-1.0.1.tar.gz.

File metadata

  • Download URL: fast_csv_loader-1.0.1.tar.gz
  • Upload date:
  • Size: 17.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for fast_csv_loader-1.0.1.tar.gz
Algorithm Hash digest
SHA256 c35473284a2f7cec65987ac3cd6dc0ebeddccafd56fe2542869f835e855e9544
MD5 d92187a414f13ee9fa386ef92f0e00ae
BLAKE2b-256 a6a4fcc30670f110db96a9f08cff704fc92aa81ac1bb732a382b113746a50910

See more details on using hashes here.

Provenance

The following attestation bundles were made for fast_csv_loader-1.0.1.tar.gz:

Publisher: publish-to-pypi.yml on BennyThadikaran/fast_csv_loader

Attestations:

File details

Details for the file fast_csv_loader-1.0.1-py3-none-any.whl.

File metadata

File hashes

Hashes for fast_csv_loader-1.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 63f7937319bc4d6b5c8e910f4fa48b78ea1dba4e0c71b929f5fcc7f4ca02f54b
MD5 b275ead97b7d2b7618da23ad617a0709
BLAKE2b-256 17faa842996e8fd633d710ca33673985216c1d345c2af1cf5a719fae258cb8ee

See more details on using hashes here.

Provenance

The following attestation bundles were made for fast_csv_loader-1.0.1-py3-none-any.whl:

Publisher: publish-to-pypi.yml on BennyThadikaran/fast_csv_loader

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page