This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Estimate the distributions of line lengths of files.

Project Description

Qudth randomly samples the lines within a large file and calculates statistics about each line. For example, in a 10-gigabyte text file, you might want to know how long a typical line is.

Line lengths

It would be very convenient if line length is what you are interested in, as that is the only thing we implement right now.

$ qudth qudth/cli.py -n 5 --bins 8

▁ ▁ ▂ ▁ ▁ ▁ ▃ ▃
01     52     59
Lengths of 5 lines in qudth/cli.py
(simple random sample with replacement)

Benchmarking

wc -l is equivalent to qudth’s line length estimation, but qudth’s sampling makes it much faster on large files. big-file.csv is 1 gigabyte in size.

_:~ t$ time qudth big-file.csv > /dev/null

real    0m0.287s
user    0m0.161s
sys     0m0.032s
_:~ t$ time wc -l big-file.csv > /dev/null

real    0m2.515s
user    0m1.475s
sys     0m0.440s

Future work

A more standard thing would perhaps be something that emitted a random sample to stdout. It could support different sampling strategies perhaps.

Release History

Release History

This version
History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
qudth-0.0.3.tar.gz (2.7 kB) Copy SHA256 Checksum SHA256 Source Aug 15, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting