Skip to main content

A lightweight Pythonic OR Mapper for MongoDB.

Project description

MongoBase Logo

MongoBase is a Python package that provides high-level features:

  • Lightweight OR Mapper (ORM) for MongoDB
  • Simple DataBase Model structure definition with automatic type checking
  • High-level automatic text search indexes generation from multiple keys

Dependencies

  • pymongo_ 3.7+

More About MongoBase

Component Description
mongobase an high-level interface with model definition system from ModelBase and many database operations
modelbase an OR Mapper class with automatic type checking according to the defined structure (MongoBase subclass)

Philosophies on MongoBase are

  • enable to use MongoDB on python easily and programatically safely
  • cleary viewable everything about the data model just by a quick looking over the model definition
  • easy to learn how to use. for instance, method names correspond to MongoDB to be able to use them as if on the mongoclient.
  • high performance. it uses the latest connection pool mechanism so that efficiently use client objects

Basic Interfaces

Model Definition

Here is the sample definition of a model.

class Bird(MongoBase):
    __collection__ = 'birds'
    __structure__ = {
        '_id': ObjectId,
        'name': str,
        'age': int,
        'is_able_to_fly': bool,
        'created': dt.datetime,
        'updated': dt.datetime
    }
    __required_fields__ = ['_id', 'name']
    __default_values__ = {
        '_id': ObjectId(),
        'is_able_to_fly': False,
        'created': dt.datetime.now(dt.timezone.utc),
        'updated': dt.datetime.now(dt.timezone.utc)
    }
    __validators__ = {
        'name': validate_length(0, 1000),
    }
    __search_text_keys__ = ['name'] 
    __search_text_index_type__ = 'bigram'
    __indexes__ = [
        [('item_name', ASCENDING),],
    ]

The core model structure is defined as __structure__ by a dictionaly. It is possible to cleary find out how the document structure is. Other components of the model definition is:

Component Description
__collection__ the collection name of the document. (required)
__structure__ the core definition of the model. the type is automatically checked everytime when it is written on the db. the key _id is required. (required)
__required_fields__ required properties. (optional)
__default_values__ set default values for properties. (optional)
__validators__ validator methods automatically check the value when the document is written on the db. (optional)
__search_text_keys__ multiple keys can be set for the search text index. automatically written as the search_text property. (optional)
__search_text_index_type__ bigram: value of search_text is set as bigram strings. morpheme: the string in search_text is parsed to morphemes (optional)
__search_text_weight_type__ uniform: each string has the same weight. weighted: enable to set weights as [('key1', 3), ('key2', 1)] (optional)
__indexes__ indexes can be set. .createIndex() method creates the indexes on the db. (optional)

Now the basic usages are introduced.

Insert & Update

>> chicken = Bird({'_id': ObjectId(), 'name': 'chicken', 'age': 3})
>> chicken.save()
{'_id': ObjectId('5c80f4fa16fa0d6c102cd2a6'),
 'name': 'chicken',
 'age': 3,
 'is_able_to_fly': False,
 'created': datetime.datetime(2019, 3, 7, 10, 39, 54, 643685, tzinfo=datetime.timezone.utc),
 'updated': datetime.datetime(2019, 3, 7, 10, 39, 54, 643690, tzinfo=datetime.timezone.utc)}
>> chicken.is_able_to_fly = True
>> chicken.update()

Find

>>> Bird.findOne({'name': 'mother chicken'})
{'_id': ObjectId('5c79166716fa0d215968d3ba'),
 'name': 'mother chicken',
 'age': 63,
 'is_able_to_fly': False,
 'created': datetime.datetime(2019, 3, 1, 11, 20, 21, 306000),
 'updated': datetime.datetime(2019, 3, 1, 11, 20, 21, 306000)}
>>> mother_chicken.remove()
1
>>> all_chickens = Bird.find({'name': 'chicken'}, sort=[('_id', ASCENDING)])
list of mongobase instances are returned.
>>> len(all_chickens)
18
>>> Bird.count()
201

Bulk Operations

  • bulk_insert
>>> many_pigeon = []
>>> for i in range(10000):
>>>     many_pigeon += [Bird({'_id': ObjectId(), 'name': f'pigeon', 'age': i})]
>>> Bird.bulk_insert(many_pigeon)
10000
  • bulk_update
>>> updates = []
>>> for pigeon in many_pigeon:
>>>    pigeon.age *= 3
>>>    updates += [pigeon]
>>> Bird.bulk_update(updates)
10000

Contextual Database

with db_context(db_uri='localhost', db_name='test') as db:
    flamingo = Bird({'_id': ObjectId(), 'name': 'flamingo', 'age': 20})
    flamingo.save(db=db)

    flamingo.age = 23
    flamingo = flamingo.update(db=db)
    flamingo = Bird.findAndUpdateById(flamingo._id, {'age': 24}, db=db)

    n_flamingo = Bird.count({'name': 'flamingo'}, db=db)

Bird.count({'name': 'flamingo'})

Multi Processing

def breed(tasks):
    db = Bird._db()  # create a MongoDB Client for the forked process
    for i in range(len(tasks)):
        sparrow = Bird({'_id': ObjectId(), 'name': f'sparrow', 'age': 0})
        sparrow.save(db=db)

tasks = [[f'task {i}' for i in range(N_BATCH)] for j in range(N_PROCESS)
process_pool = multiprocessing.Pool(N_PROCESS)
process_pool.map(breed, tasks)

MongoBase has Many Other Features

If you'd like to know other features, please check the file mongobase.py.

DB Settings

simply write to mongobase/config.py

MONGO_DB_URI = "101.21.434.121"
MONGO_DB_URI_TEST = "localhost"
MONGO_DB_NAME = "zoo"
MONGO_DB_NAME_TEST = "zoo-test"
MONGO_DB_CONNECT_TIMEOUT_MS = 3000
MONGO_DB_SERVER_SELECTION_TIMEOUT_MS = 3000
MONGO_DB_SOCKET_TIMEOUT_MS = 300000
MONGO_DB_SOCKET_KEEP_ALIVE = True
MONGO_DB_MAX_IDLE_TIME_MS = 40000
MONGO_DB_MAX_POOL_SIZE = 200
MONGO_DB_MIN_POOL_SIZE = 10
MONGO_DB_WAIT_QUEUE_MULTIPLE = 12
MONGO_DB_WAIT_QUEUE_TIMEOUT_MS = 100

Getting Started

If you start MongoBase, there is a tutorial jupyter notebook here.
Highly recommend to check it. https://github.com/kazukiotsuka/mongobase/blob/master/tutorial/MongoBase_starting_guide.ipynb

Release and Contributing

Many methods are the wrapper of pymongo.
There are a lot of features that this library is covering.
Would appreciate if you add their methods anytime.

version 0.3.0

New features
  • bulk_insert()
  • bulk_update()
  • performance improvement with ConnectionPool (single MongoClient for each process)
  • MongoBase_start_guide.ipynb
  • contextual db client mode by with db_context() as db
  • code efficiency improvement
  • abolished insert_if_not_exists parameter for save(), update()
  • changed some method names (e.g. remove -> delete)
  • using pymongo > 3.5 methods (e.g. insert_one())
  • enhance documents

version 0.2.0

New features
  • MongoBase and ModelBase class are separated
  • enable to use MongoClient instance dynamically
  • some useful mongodb operations are added

version 0.1.0

New features
  • The initial implementation
  • automatic type checking mechanism
  • basic mongodb operations

License

MongoBase is MIT-style licensed, as found in the LICENSE file.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for mongobase, version 0.3.1a0
Filename, size File type Python version Upload date Hashes
Filename, size mongobase-0.3.1a0-py3-none-any.whl (14.5 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size mongobase-0.3.1a0.tar.gz (15.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page