Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

Estimate the distributions of line lengths of files.

Project description

Qudth randomly samples the lines within a large file and calculates statistics about each line. For example, in a 10-gigabyte text file, you might want to know how long a typical line is.

Line lengths

It would be very convenient if line length is what you are interested in, as that is the only thing we implement right now.

$ qudth qudth/cli.py -n 5 --bins 8

▁ ▁ ▂ ▁ ▁ ▁ ▃ ▃
01     52     59
Lengths of 5 lines in qudth/cli.py
(simple random sample with replacement)

Benchmarking

wc -l is equivalent to qudth’s line length estimation, but qudth’s sampling makes it much faster on large files. big-file.csv is 1 gigabyte in size.

_:~ t$ time qudth big-file.csv > /dev/null

real    0m0.287s
user    0m0.161s
sys     0m0.032s
_:~ t$ time wc -l big-file.csv > /dev/null

real    0m2.515s
user    0m1.475s
sys     0m0.440s

Future work

A more standard thing would perhaps be something that emitted a random sample to stdout. It could support different sampling strategies perhaps.

Project details


Release history Release notifications

This version
History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
qudth-0.0.3.tar.gz (2.7 kB) Copy SHA256 hash SHA256 Source None Aug 15, 2015

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page