This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Sample lines from a file.

Project Description

Sample lines from a file that has already been written.

Install

Install like so.

pip install sample-lines

How to

See the help for documentation.

sample-lines -h
usage: Randomly select lines from a file. [-h] [--sample-size N]
                                          [--method {simple-random,systematic}]
                                          [--repeat REPEAT]
                                          file

positional arguments:
  file

optional arguments:
  -h, --help            show this help message and exit
  --sample-size N, -n N
                        Number of lines to emit
  --method {simple-random,systematic}, -m {simple-random,systematic}
                        Sampling method
  --repeat REPEAT, -r REPEAT
                        Number of repetitions for systematic sampling

Samples are with replacement and weighted by line length. The probability of selecting a line is proportional the length of the previous line. This allows us to sample very quickly, but it makes this approach appropriate only if your file has reasonably consistent line lengths or at least if there is no periodic variation in line length.

How fast

Consider this 1-gigabyte CSV file.

$ wc big-file.csv
 2388430 27673790 1071895374 big-file.csv

Running wc took three seconds.

time wc big-file.csv
 2388430 27673790 1071895374 big-file.csv

real    0m3.789s
user    0m3.560s
sys     0m0.190s

Here’s how long it takes to parse the whole file.

$ time python3 -c 'for line in open("big-file.csv"): pass'

real    0m2.892s
user    0m2.641s
sys     0m0.245s

sample-lines is much faster. Here’s a simple random sample of 40 lines,

$ time sample-lines -n 40 -m simple-random big-file.csv > /dev/null

real    0m0.136s
user    0m0.113s
sys     0m0.018s

a systematic sample of 40 lines,

$ time sample-lines -n 40 -m systematic -r 4 big-file.csv > /dev/null

real    0m0.148s
user    0m0.122s
sys     0m0.019s

and repeated systematic sample, with 4 repeats and 10 lines each, for a total of 40 lines.

$ time sample-lines -n 10 -m systematic -r 4 big-file.csv > /dev/null

real    0m0.175s
user    0m0.140s
sys     0m0.025s

Most of the time in the above examples was spent loading Python and the various modules; printing the help takes almost as long as running the sample.

$ time sample-lines -h > /dev/null

real    0m0.157s
user    0m0.129s
sys     0m0.021s

So even a pretty big sample is still fast to run.

$ time sample-lines -n 2000 -m systematic -r 50 big-file.csv > /dev/null

real    0m2.695s
user    0m2.435s
sys     0m0.231s

Alternatives

Use sample if you want to sample from a stream.

Release History

Release History

This version
History Node

0.0.4

History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
sample-lines-0.0.4.tar.gz (2.6 kB) Copy SHA256 Checksum SHA256 Source Aug 15, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting