partd

Minimal key-value byte storage with appendable values

Partd stores key-value pairs. The values are raw bytes. We append onto existing values.

Partd is useful for shuffling operations.

API

Create a Partd:

>>> import partd
>>> p = partd.File('/path/to/new/dataset/'')

Append key-byte pairs to dataset:

>>> p.append({'x': b'Hello ', 'y': b'123'})
>>> p.append({'x': b'world!', 'y': b'456'})

Get all bytes associated to a set of keys:

>>> p.get(['y', 'x'])
[b'123456', b'Hello world!']

Idempotently set single key-value pair (no append, no update):
```
>>> p.iset('z', b'metadata'])
```

Destroy partd dataset:
```
>>> p.drop()
```

That’s it.

There is no in-memory state.

Implementations

The reference implementation uses file-based locks. This works surprisingly well as long as you don’t do many small writes.

If you do many small writes then you probably want to cache in memory. Fortunately you can buffer one partd with another, writing only large chunks as necessary when spaces runs low:

>>> p = Buffer(Dict(), File(), available_memory=2e9)  # 2GB memory buffer

You might also want to have many distributed process write to a single partd consistently. This can be done with a server

Server Process:

>>> local_partd = File()
>>> p = Server(local_partd, address='ipc://server')

Worker processes:

>>> p = Client('ipc://server')
>>> p.append(...)

Encodings and Compression

Once we can robustly and efficiently append bytes we move on to encoding various things as bytes either with serialization systems like Pickle or MSGPack or with compression routines like zlib, snappy, or blosc. In principle we want to compose all of these choice together

Write policy: Dict, File, Buffer, Client
Encoding: Pickle, Numpy, Pandas, …
Compression: Blosc, Snappy, …

Partd objects compose by nesting for example here we make a server backed by a buffered dict/file combination with a client using snappy compressed pickle data:

>>> server = Server(Buffer(Dict(), File(), available_memory=2e0))

>>> client = Pickle(Snappy(Client(server.address)))

And here a partd that writes pickle encoded BZ2 compressed bytes directly to disk:

>>> p = Pickle(BZ2(File('foo')))

Metadata

Release files for partd 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for partd 0.3.0
File	Size	Uploaded
partd-0.3.0.tar.gz	14.8 kB	Jun 12, 2015	Details

Release files / partd-0.3.0.tar.gz

Download URL	partd-0.3.0.tar.gz
Size	14.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`41ab11dae0eec877864da36be1df8fc55966c9aadb508a83e229efae33f2b541`
BLAKE2b-256 checksum How to use checksums	`b402d409b2d98760951be548f785b9cacd827c635bddddc2b8df4b6e1aa8721b`
Upload date	Jun 12, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No