Skip to main content

No project description provided

Project description

https://travis-ci.org/internetarchive/trough.svg?branch=master

Trough

Big data, small databases.

Big data is really just lots and lots of little data.

If you split a large dataset into lots of small SQL databases sharded on a well-chosen key, they can work in concert to create a database system that can query very large datasets.

Worst-case Performance is important

A key insight when working with large datasets is that with monolithic big data tools’ performance is largely tied to having a full dataset completely loaded and working in a production-quality cluster.

Trough is designed to have very predictable performance characteristics: simply determine your sharding key, determine your largest shard, load it into a sqlite database locally, and you already know your worst-case performance scenario.

Designed to leverage storage, not RAM

Rather than having huge CPU and memory requirements to deliver performant queries over large datasets, Trough relies on flat sqlite files, which are easily distributed to a cluster and queried against.

Reliable parts, reliable whole

Each piece of technology in the stack was carefully selected and load tested to ensure that your data stays reliably up and reliably queryable. The code is small enough for one programmer to audit.

Ease of installation

One of the worst parts of setting up a big data system generally is getting setting sensible defaults and deploying it to staging and production environments. Trough has been designed to require as little configuration as possible.

An example ansible deployment specification has been removed from the trough repo but can be found at https://github.com/internetarchive/trough/tree/cc32d3771a7/ansible. It is designed for a cluster Ubuntu 16.04 Xenial nodes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Trough, version 0.1.4
Filename, size File type Python version Upload date Hashes
Filename, size Trough-0.1.4.tar.gz (31.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page