Skip to main content

Lightweight, fast and scalable text corpus library.

Project description

Welcome to Corpora!
===================
*Corpora* is a lightweight, fast and scalable corpus library able to store a collection of raw text documents with additional key-value headers. It uses Berkeley DB (bsddb3 module) for index managing what guarantee speed and bullet-proof. Text storage model is based on chunked flat, human readable text files. This architecture can easily scale up to millions documents, hundred of gigabytes collections.

Corpora module provides four main features:
* create a new corpus,
* append documents to a corpus,
* random access to any document in a corpus using it's unique ``id``,
* sequential access to document collection (generator over collection).

Key-Value document headers supports storing any kind of objects seriazable with yaml_. Corpora supports only append & read-only philosophy, for more information please read section :doc:`motivation`.

.. _yaml: http://www.yaml.org/

Quickstart
----------
Installation:
::

> sudo pip install corpora

Basic usage:


>>> from corpora import Corpus
>>> Corpus.create('/tmp/test_corpus')
>>> c = Corpus('/tmp/test_corpus')
>>> c.add('First document', 1)
>>> c.add('Second document', 2)
>>> c.save_indexes()
>>> len(c)
2
>>> c[1]
({'id': 1}, u'First document')
>>> c[2]
({'id': 2}, u'Second document')
>>> for t in c:
... print t
...
({'id': 1}, u'First document')
({'id': 2}, u'Second document')

Project details


Release history Release notifications | RSS feed

This version

1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Corpora-1.0.tar.gz (5.1 kB view details)

Uploaded Source

File details

Details for the file Corpora-1.0.tar.gz.

File metadata

  • Download URL: Corpora-1.0.tar.gz
  • Upload date:
  • Size: 5.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for Corpora-1.0.tar.gz
Algorithm Hash digest
SHA256 208a68da259c6c5ccd36a85d83c5cfdf43f1fb25aa77d21b7a2b0c6bfa1cd1db
MD5 02781c45591ff458819e13120394828d
BLAKE2b-256 6cf5998ee3d19c64e42a5a3839858ede61ccd504c13f24fbe3bf48ddb6fd3592

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page