Skip to main content

Monty, Mongo tinified. MongoDB implemented in Python !

Project description

drawing

Python package Version PyPi downloads

Monty, Mongo tinified. MongoDB implemented in Python!

Inspired by TinyDB and it's extension TinyMongo

What is it?

A pure Python-implemented database that looks and works like MongoDB.

>>> from montydb import MontyClient

>>> col = MontyClient(":memory:").db.test
>>> col.insert_many( [{"stock": "A", "qty": 6}, {"stock": "A", "qty": 2}] )
>>> cur = col.find( {"stock": "A", "qty": {"$gt": 4}} )
>>> next(cur)
{'_id': ObjectId('5ad34e537e8dd45d9c61a456'), 'stock': 'A', 'qty': 6}

Most of the CRUD operators have been implemented. You can visit issue #14 to see the full list.

This project is tested against:

  • MongoDB: 3.6, 4.0, 4.2 (4.4 on the way💦)
  • Python: 2.7, 3.6, 3.7, 3.8, 3.9

Install

pip install montydb
  • optional, to use real bson in operation (pymongo will be installed) For minimum requirements, montydb ships with it's own fork of ObjectId in montydb.types, so you may ignore this option if ObjectId is all you need from bson

    pip install montydb[bson]
    
  • optional, to use lightning memory-mapped db as storage engine

    pip install montydb[lmdb]
    

Storage

🦄 Available storage engines:

  • in-memory
  • flat-file
  • sqlite
  • lmdb (lightning memory-mapped db)

Depending on which one you use, you may have to configure the storage engine before you start.

⚠️

The configuration process only required on repository creation or modification. And, one repository (the parent level of databases) can only assign one storage engine.

To configure a storage, see flat-file storage for example:

from montydb import set_storage, MontyClient


set_storage(
    # general settings

    repository="/db/repo",  # dir path for database to live on disk, default is {cwd}
    storage="flatfile",     # storage name, default "flatfile"
    mongo_version="4.0",    # try matching behavior with this mongodb version
    use_bson=False,         # default None, and will import pymongo's bson if None or True

    # any other kwargs are storage engine settings.

    cache_modified=10,       # the only setting that flat-file have
)

# ready to go

Once that done, there should be a file named monty.storage.cfg saved in your db repository path. It would be /db/repo for the above examples.

Configuration

Now let's moving on to each storage engine's config settings.

🌟 In-Memory

memory storage does not need nor have any configuration, nothing saved to disk.

from montydb import MontyClient


client = MontyClient(":memory:")

# ready to go

🔰 Flat-File

flatfile is the default on-disk storage engine.

from montydb import set_storage, MontyClient


set_storage("/db/repo", cache_modified=5)  # optional step
client = MontyClient("/db/repo")  # use current working dir if no path given

# ready to go

FlatFile config:

[flatfile]
cache_modified: 0  # how many document CRUD cached before flush to disk.

💎 SQLite

sqlite is NOT the default on-disk storage, need configuration first before getting client.

Pre-existing sqlite storage file which saved by montydb<=1.3.0 is not read/writeable after montydb==2.0.0.

from montydb import set_storage, MontyClient


set_storage("/db/repo", storage="sqlite")  # required, to set sqlite as engine
client = MontyClient("/db/repo")

# ready to go

SQLite config:

[sqlite]
journal_mode: WAL

SQLite write concern:

client = MontyClient("/db/repo",
                     synchronous=1,
                     automatic_index=False,
                     busy_timeout=5000)

🚀 LMDB (Lightning Memory-Mapped Database)

lightning is NOT the default on-disk storage, need configuration first before get client.

Newly implemented.

from montydb import set_storage, MontyClient


set_storage("/db/repo", storage="lightning")  # required, to set lightning as engine
client = MontyClient("/db/repo")

# ready to go

LMDB config:

[lightning]
map_size: 10485760  # Maximum size database may grow to.

URI

Optionally, You could prefix the repository path with montydb URI scheme.

client = MontyClient("montydb:///db/repo")

Utilities

Pymongo bson may required.

  • montyimport

    Imports content from an Extended JSON file into a MontyCollection instance. The JSON file could be generated from montyexport or mongoexport.

    from montydb import open_repo, utils
    
    
    with open_repo("foo/bar"):
        utils.montyimport("db", "col", "/path/dump.json")
    
  • montyexport

    Produces a JSON export of data stored in a MontyCollection instance. The JSON file could be loaded by montyimport or mongoimport.

    from montydb import open_repo, utils
    
    
    with open_repo("foo/bar"):
        utils.montyexport("db", "col", "/data/dump.json")
    
  • montyrestore

    Loads a binary database dump into a MontyCollection instance. The BSON file could be generated from montydump or mongodump.

    from montydb import open_repo, utils
    
    
    with open_repo("foo/bar"):
        utils.montyrestore("db", "col", "/path/dump.bson")
    
  • montydump

    Creates a binary export from a MontyCollection instance. The BSON file could be loaded by montyrestore or mongorestore.

    from montydb import open_repo, utils
    
    
    with open_repo("foo/bar"):
        utils.montydump("db", "col", "/data/dump.bson")
    
  • MongoQueryRecorder

    Record MongoDB query results in a period of time. Requires to access database profiler.

    This works via filtering the database profile data and reproduce the queries of find and distinct commands.

    from pymongo import MongoClient
    from montydb.utils import MongoQueryRecorder
    
    client = MongoClient()
    recorder = MongoQueryRecorder(client["mydb"])
    recorder.start()
    
    # Make some queries or run the App...
    recorder.stop()
    recorder.extract()
    {<collection_1>: [<doc_1>, <doc_2>, ...], ...}
    
  • MontyList

    Experimental, a subclass of list, combined the common CRUD methods from Mongo's Collection and Cursor.

    from montydb.utils import MontyList
    
    mtl = MontyList([1, 2, {"a": 1}, {"a": 5}, {"a": 8}])
    mtl.find({"a": {"$gt": 3}})
    MontyList([{'a': 5}, {'a': 8}])
    

Why did I make this?

Mainly for personal skill practicing and fun.

I work in the VFX industry and some of my production needs (mostly edge-case) requires to run in a limited environment (e.g. outsourced render farms), which may have problem to run or connect a MongoDB instance. And I found this project really helps.


This project is supported by JetBrains

drawing    drawing

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

montydb-2.3.10.tar.gz (63.3 kB view details)

Uploaded Source

Built Distribution

montydb-2.3.10-py2.py3-none-any.whl (73.0 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file montydb-2.3.10.tar.gz.

File metadata

  • Download URL: montydb-2.3.10.tar.gz
  • Upload date:
  • Size: 63.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.5

File hashes

Hashes for montydb-2.3.10.tar.gz
Algorithm Hash digest
SHA256 c537056c8ec07ae36e387a69eec7733e75fe28b336470526f541a492f9aaf764
MD5 99abbdf30b018d08d93907fd1f5cbe17
BLAKE2b-256 c03dd75dc95fcf06e668977ae32661dd3814cb49982445f85c294bf49064da58

See more details on using hashes here.

Provenance

File details

Details for the file montydb-2.3.10-py2.py3-none-any.whl.

File metadata

  • Download URL: montydb-2.3.10-py2.py3-none-any.whl
  • Upload date:
  • Size: 73.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.5

File hashes

Hashes for montydb-2.3.10-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 b30d6c64dab90556d6866de3501c390dc158715c0ee3218fda7f867ef83fc133
MD5 2585fb197a6e055f9f6dd43e2a63c688
BLAKE2b-256 1434b80700dbb7cc572c40d5a82c4759013d17cfc968e4b6a3168fe729cfdade

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page