We're making managing a big data project as simple as a "git push". Think of us as "Heroku for Big Data"
With triv.io you can transform massive amounts of poorly structured data on a regular basis. Regardless of the size and whether your data exists in log files, multiple databases or even at the end of a web API we help you store, transform and share your data with others.
Install triv.io and login
$ pip install trivio
$ trivio login
Triv.io uses github for authentication. You'll need a github
username and password to proceed. Authentication is done directly
with the github servers, we never see or store your user name and
password. Don't believe us? Have a look at the trivio client source.
Create your first project
$ trivio create https://github.com/trivio/hello_world
Creating project hello_world
You do not have write access to this project
would you like to fork it?[Yn]: Y
forking project to https://github.com/username/hello_world
Your project is ready for use
Try it out
~ $ cd hello_world
hello_world $ trivio simulate word_counts
Make some changes
Using your favorite editor let's make word_counts use a better tokenizer
hello_world $ ls
hello_world $ vi word_counts.pyj
Change the file to look like this
def map(doc, params, tokenizer = re.compile('[^A-Z0-9_.]+', flags=re.I)):
for word in tokenizer.split(doc['payload']):
yield word, 1
Now run simulate again to verify the results
hello_world $ trivio simulate word_counts
Push your changes
Once you've tweaked your algorithm you need to push those changes to trivio
hello_world $ trivio push "Used a better tokenizer"
changing to srobertson
[master 37200ed] pushing
1 files changed, 1 insertions(+), 1 deletions(-)
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 301 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
4ebf177..37200ed master -> master
Successfully pushed your changes to trivio.
Your project is paused use 'trivio step' or 'trivio resume' to build using
Compute a single segment
By default, all projects start off "paused", this prevents them from running up your compute bills until you're satisfied your code works the way you expect it to. You've already used the `trivio simulate` method which samples your data and returns you results. The next step is to build a segment ("a portion of your table"see triv.io data model for more details)
$ trivio step word_counts
Building next segment 2009-9-17 00:00:00
Notice that this command returns immediately, building a segment can be quite involved. You may be producing terabytes of data which you wouldn't want to download to your workstation.
Monitoring Build Progress
$ trivio watch
Table Queued Built Failed
.../common-crawl/parsed-output 0 1 0
Next build: Never (project paused)
This will monitor your progress of your project. When the watch command spits out Next build Never. Hit `ctrl-c` to get back to your command line. As noted your project is paused so it won't build anything else until you either tell trivio to build a single segment with the step command or tell trivio to unpause your project. We'll get to unpausing in a moment
Now that trivio has had a chance to build part of your table we can query it for data. Tri.vio uses a sql like syntax for querying
$ trivio query "select * from word_counts"
| key | value |
| Zachary | 22731 |
| Alfred | 20477 |
| Gregory | 17179 |
| Ned | 16860 |
| Ulrich | 15300 |
| Thomas | 14995 |
Put Triv.io on Autopilot
Once you're happy with your project, you can tell triv.io to keep your tables up to date automatically based on the schedules you defined in your scripts.
$ trivio resume
Auto building all tables, use 'trivio watch' to monitor progress
Where to go from here
That's the basic workflow
* [Trivio data model](http://docs.triv.io/)
* [Talking to new storages](http://docs.triv.io/)
* [Interpreting mime types](http://docs.triv.io/)
* Check out our [blog](http://docs.triv.io/) for cool projects, tips and tricks
[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/trivio/trivio.client/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
TODO: Brief introduction on what you do with files - including link to relevant help section.