csample 0.2.2

Hash-based sampling library for Python

csample: Hash-based sampling library for Python

Hash-based sampling is a filtering method that tries to approximate random sampling by using a hash function as a selection criterion.

Following list describes some features of the method:

• Since there are no randomness involved at all, the same data set with the same sampling rate (and also with the same salt value) always yields exactly the same result.

• The size of population doesn’t need to be specified beforehand. It means that the sampling process can be applied to data stream with unknown size such as system logs.

Applications

Here are some real and hypothetical applications:

• [RFC5475] Sampling and Filtering Techniques for IP Packet Selection is a well-known application.

• Online streaming algorithm to select 10% of users for A/B testing. “Consistent” nature of the algorithm guarantees that any user ID selected once will always be selected again. There’s no need to maintain a list of selected user IDs.

Usage

Two sampling functions are provided for a convenience.

sample_line() accepts iterable type containing strs:

data = [
    'alan',
    'brad',
    'cate',
    'david',
]
samples = csample.sample_line(data, 0.5)

sample_tuple() expects tuples instead of strs as a content of iterable. The third argument 0 indicates a column index:

data = [
    ('alan', 10, 5),
    ('brad', 53, 7),
    ('cate', 12, 6),
    ('david', 26, 5),
]
samples = csample.sample_tuple(data, 0.5, 0)

In both cases, the function returns immediately with sampled iterable.

Read the full documentation.

Command-line interface

csample also provides command-line interface.

Following command prints 50% sample from 100 integers:

> seq 100 | csample -r 0.5

To see more options use --help command-line argument:

> csample --help

Hash functions

To obtain fairly random/unbiased sample, it is critical to use suitable hash function.

There could be many criteria such as avalanche effect. For those who are interested, see link below:

• Empirical Evaluation of Hash Functions for Multipoint Measurements

csample currently supports xxhash and spooky.

Installation

Installing csample is easy:

pip install csample

or download the source and run:

python setup.py install