Skip to main content

No project description provided

Project description


Big data, small databases.

Big data is really just lots and lots of little data.

If you split a large dataset into lots of small SQL databases sharded on a well-chosen key, they can work in concert to create a database system that can query very large datasets.

Worst-case Performance is important

A key insight when working with large datasets is that with monolithic big data tools’ performance is largely tied to having a full dataset completely loaded and working in a production-quality cluster.

Trough is designed to have very predictable performance characteristics: simply determine your sharding key, determine your largest shard, load it into a sqlite database locally, and you already know your worst-case performance scenario.

Designed to leverage storage, not RAM

Rather than having huge CPU and memory requirements to deliver performant queries over large datasets, Trough relies on flat sqlite files, which are easily distributed to a cluster and queried against.

Reliable parts, reliable whole

Each piece of technology in the stack was carefully selected and load tested to ensure that your data stays reliably up and reliably queryable. The code is small enough for one programmer to audit.

Ease of installation

One of the worst parts of setting up a big data system generally is getting setting sensible defaults and deploying it to staging and production environments. Trough has been designed to require as little configuration as possible.

An example ansible deployment specification has been removed from the trough repo but can be found at It is designed for a cluster Ubuntu 16.04 Xenial nodes.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Trough-0.2.0.tar.gz (34.5 kB view hashes)

Uploaded source

Built Distribution

Trough-0.2.0-py3-none-any.whl (39.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page