Skip to main content

Create random samples from CSV file

Project description

## csvsample

``csvsample`` extracts some rows from CSV file to create randomly sampled CSV.

### Features

* The size of original file does not need to be specified beforehand.
It means that the sampling process can be applied to data stream with
unknown size such as system logs, no matter how large the amount of data
* All methods accepts optional ``seed`` value. The same data set with the
same sampling rate always yields exactly the same result, which is good
for reproducibility.

### Install

You can install ``csvsample`` via ``pip``:

pip install csvsample

### API

``csvsample.sample()`` is the main API:

csvsample.sample(lines, sampling_method, **kwargs)

``lines`` can be any ``iterable`` containing valid CSV rows including header

``sampling_method`` should be one of followings:

* ``random``
* ``hash``
* ``reservoir``

``random`` sampling method performs random sampling using pseudo random number

import csvsample

with open('input.csv', 'r') as i:
with open('output.csv', 'w') as o:
o.writelines(csvsample.sample(i, 'random', sample_rate=0.1))

``hash`` sampling method performs hash-based sampling using extremely-fast hash

Let's say that instead of saving all users' log, you want to randomly select
10% of users and only save logs of those selected users. Simple random sampling
won't work. You can use hash-based sampling. "Consistent" nature of the
algorithm guarantees that any user ID selected once will always be selected

sampled = csvsample.sample(lines, 'hash', sample_size=0.1, col='user_id')

``reservoir`` sampling method performs reservoir sampling. Let's say that you
have an URL of 100GB csv file. Since you don't have enough disk space, you just
want to save small portion of sample which is representative and unbiased.

[reservoir sampling]( method
allows you to acquire random sample without saving entire data first:

sampled = csvsample.sample(lines, 'reservoir', sample_size=1000)

Now ``sampled`` variable contains exactly 1,000 randomly selected lines.

### Helpers

There are some convenience helpers:

* ``csvsample.sample_url(url, sampling_method, **kwargs)`` read CSV from given
``url``. You can specify character set encoding via ``encoding`` keyword
argument. (default: ``utf-8``)

``csvsample.sample()`` and other helpers return a generator containing sampled
CSV rows including header row. The generator contains special function
``to_buf()`` which converts itself into ``io.StringIO`` instance so that you can
pass the sampled CSV to other libraries such as Pandas:

import csvsample
import pandas as pd

sampled = csvsample.sample_url(url, 'random', sample_rate=0.1)
df = pd.read_csv(sampled.to_buf())

### Command-line interface

``csvsample`` also provides command-line interface.

Following URL contains a CSV file from [DataHub](

> curl -sL | head

A number of rows including header is 18019:

> curl -sL | wc -l

Let's make 10% of random sample:

> curl -sL | csvsample random 0.1 > sample.csv

> wc -l sample.csv
1777 sample.csv

> head -n 5 sample.csv

You may use reservoir sampling method to obtain exact number of rows:

> curl -sL | csvsample reservoir 100 > sample.csv

> wc -l sample.csv
100 sample.csv

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for csvsample, version 0.1.4
Filename, size File type Python version Upload date Hashes
Filename, size csvsample-0.1.4-py3-none-any.whl (5.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size csvsample-0.1.4.tar.gz (5.4 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page