Skip to main content

A NoSQL Database Implementation using Amazon S3

Project description

S3NoSQLDB

A NoSQL Database Implementation using Amazon S3 for storage.

Features

  • Create and manage collections in S3 buckets
  • Batch document insertion with automatic ID generation
  • Concurrent processing using ThreadPoolExecutor
  • Query documents using custom filter functions
  • Parquet file format with gzip compression

Installation

pip install s3nosqldb

Quick Start

from s3nosqldb import S3NoSQLDB

# Initialize the database
db = S3NoSQLDB(
    bucket_name="your-bucket-name",
    base_path="optional/path",
    batch_size=50000,
    max_workers=4
)

# Create a collection
db.create_collection("users", id_field="user_id", auto_generate_id=True)

# Insert documents
users = [
    {"name": "John Doe", "age": 30},
    {"name": "Jane Smith", "age": 25}
]
db.insert_batch("users", users)

# Query documents
def age_filter(doc):
    return doc["age"] > 25

results = db.query("users", filter_fn=age_filter)

Development

  1. Clone the repository:
git clone https://github.com/yourusername/s3nosqldb.git
cd s3nosqldb
  1. Create a virtual environment and install dependencies:
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e .[dev]
  1. Run tests:
pytest

License

This project is licensed under the MIT License - see the LICENSE file for details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3nosqldb-0.1.0.tar.gz (5.6 kB view hashes)

Uploaded Source

Built Distribution

s3nosqldb-0.1.0-py3-none-any.whl (5.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page