Skip to main content

Storage Module for Ellar

Project description

Ellar Logo

Test Coverage PyPI version PyPI version PyPI version

Introduction

The EllarStorage Module enriches your Ellar application with robust support for managing both cloud and local file storage. Leveraging the capabilities of the Apache libcloud package to simplify file storage operations within your Ellar-powered projects.

Installation

$(venv) pip install ellar-storage

This library drew inspiration from sqlalchemy-file.

Usage

To integrate EllarStorage into your project, follow the standard Ellar project structure and then configure the module as follows:

StorageModule

Similar to other Ellar modules, the StorageModule can be configured directly where it's used or through the application configuration.

StorageModule.setup

You can set up the StorageModule using the setup method. Here's a quick example:

import os
from pathlib import Path
from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider

BASE_DIRS = Path(__file__).parent

@Module(modules=[
    StorageModule.setup(
        files={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        images={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        documents={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        default="files"
    )
])
class ApplicationModule(ModuleBase):
    pass

In this example, after application initialization, folders for files, images, and documents will be created in the specified directory. Each folder is configured to be managed by a local storage driver. You can explore other supported storage drivers.

StorageModule.register_setup

Alternatively, you can move the storage configuration to the application config:

## project_name/root_module.py

from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule

@Module(modules=[StorageModule.register_setup()])
class ApplicationModule(ModuleBase):
    pass

Then, in config.py, you can define the storage configurations:

import os
from pathlib import Path
from ellar.core.conf import ConfigDefaultTypesMixin
from ellar_storage import get_driver, Provider

BASE_DIRS = Path(__file__).parent

class DevelopmentConfig(ConfigDefaultTypesMixin):
    DEBUG = True

    STORAGE_CONFIG = dict(
        storages=dict(
            files={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            },
            images={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            },
            documents={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            }
        ),
        default="files"
    )

StorageController

StorageModule also registers StorageController which is useful when retrieving saved files. This can be disabled by setting disable_storage_controller to True.

Also, StorageController is not protected and will be accessible to the public. However, it can be protected by simply applying @Guard or @Authorize decorator.

Retrieving Saved Data

By using request.url_for, we can generate a download link for the file we wish to retrieve For example:

from ellar.common import Inject, post
from ellar.core import Request

@post('/get-books')
def get_book_by_id(self, req: Request, book_id, session: Inject[Session]):
    book = session.execute(
        select(Book).where(Book.title == "Pointless Meetings")
    ).scalar_one()
    
    return {
      "title": book.title,
      "cover": req.url_for("storage:download", path="{storage_name}/{file_name}"),
      "thumbnail": req.url_for("storage:download", path=book.thumbnail.path)
    }

With req.url_for("storage:download", path="{storage_name}/{file_name}"), we are able to create a download link to retrieve saved files.

StorageService

At the end of the StorageModule setup, StorageService is registered into the Ellar DI system. Here's a quick example of how to use it:

## project_name/server.py

import os
from ellar.app import AppFactory
from ellar.common import datastructures, constants
from ellar.core import LazyModuleImport as lazyLoad
from ellar_storage import StorageService

application = AppFactory.create_from_app_module(
    lazyLoad("project_name.root_module:ApplicationModule"),
    config_module=os.environ.get(
        constants.ELLAR_CONFIG_MODULE, "carapp.config:DevelopmentConfig"
    ),
)

storage_service: StorageService = application.injector.get(StorageService)
# Example: save a file in the 'files' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'files' folder", name="file.txt"), upload_storage='files')
# Example: save a file in the 'images' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'images' folder", name="image.txt"), upload_storage='images')
# Example: save a file in the 'documents' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'documents' folder", name="docs.txt"), upload_storage='documents')

StorageService in Route Functions

You can inject StorageService into your controllers or route functions. For instance:

In Controller:

from ellar.common import ControllerBase, Controller
from ellar_storage import StorageService

@Controller()
class FileManagerController(ControllerBase):
    def __init__(self, storage_service: StorageService):
        self._storage_service = storage_service

In Route Function:

from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService

@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
    pass

See Sample Project

Some Quick Cloud Setup

Google Cloud Storage

  • For a service Account

    from ellar.common import Module
    from ellar.core import ModuleBase
    from ellar_storage import StorageModule, get_driver, Provider
    
    @Module(modules=[
        StorageModule.setup(
            files={
                # For a service Account
                "driver": get_driver(Provider.GOOGLE_STORAGE),
                "options": {
                    "key": "client_email",
                    "secret": "private_key",
                    "...": "..."
                },
            },
        )
    ])
    class ApplicationModule(ModuleBase):
        pass
    
  • Installed Application

    from ellar.common import Module
    from ellar.core import ModuleBase
    from ellar_storage import StorageModule, get_driver, Provider
    
    @Module(modules=[
        StorageModule.setup(
            files={
                # For a service Account
                "driver": get_driver(Provider.GOOGLE_STORAGE),
                "options": {
                    "key": "client_id",
                    "secret": "client_secret",
                    "...": "..."
                },
            },
        )
    ])
    class ApplicationModule(ModuleBase):
        pass
    
  • GCE instance

    from ellar.common import Module
    from ellar.core import ModuleBase
    from ellar_storage import StorageModule, get_driver, Provider
    
    @Module(modules=[
        StorageModule.setup(
            files={
                # For a service Account
                "driver": get_driver(Provider.GOOGLE_STORAGE),
                "options": {
                    "key": "GOOG0123456789ABCXYZ",
                    "secret": "key_secret",
                    "...": "..."
                },
            },
        )
    ])
    class ApplicationModule(ModuleBase):
        pass
    

See GCS

AWS S3

from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider

@Module(modules=[
    StorageModule.setup(
        files={
            "driver": get_driver(Provider.S3),
            "options": {
                "key": "api key",
                "secret": "api secret key"
            },
        },
    )
])
class ApplicationModule(ModuleBase):
    pass

Specifying canned ACL when uploading an object

If you want to specify custom ACL when uploading an object, you can do so by passing extra argument with the acl attribute to the save or save_content methods.

Valid values for this attribute are:

  • private (default)
  • public-read
  • public-read-write
  • authenticated-read
  • bucket-owner-read
  • bucket-owner-full-control

For example

from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService

@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
    extra = {"content_type": "application/octet-stream", "acl": "public-read"}
    stored_file = storage_service.save(file=file, extra=extra)
    
    return {"message": f"{stored_file.filename} saved"}

API Reference

StorageService

  • save(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Saves a file from an UploadFile object.
  • save_async(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Asynchronously saves a file from an UploadFile object.
  • save_content(self, **kwargs) -> StoredFile: Saves a file from content/bytes or through a file path.
  • save_content_async(self, **kwargs) -> StoredFile: Asynchronously saves a file from content/bytes or through a file path.
  • get(self, path: str) -> StoredFile: Retrieves a saved file if the specified path exists. The path can be in the format container/filename.extension or filename.extension.
  • get_async(self, path: str) -> StoredFile: Asynchronously retrieves a saved file if the specified path exists.
  • delete(self, path: str) -> bool: Deletes a saved file if the specified path exists.
  • delete_async(self, path: str) -> bool: Asynchronously deletes a saved file if the specified path exists.
  • get_container(self, name: Optional[str] = None) -> Container: Gets a libcloud.storage.base.Container instance for a configured storage setup.

StoredFile

StoredFile is a file-like object returned from saving and retrieving files. It extends some libcloud Object methods and has a reference to the libcloud Object retrieved from the storage container.

Key attributes include:

  • name: File name
  • size: File size
  • filename: File name
  • content_type: File content type
  • object: libcloud Object reference
  • read(self, n: int = -1, chunk_size: Optional[int] = None) -> bytes: Reads file content
  • get_cdn_url(self) -> Optional[str]: Gets file CDN URL
  • as_stream(self, chunk_size: Optional[int] = None) -> Iterator[bytes]: Creates a file stream
  • delete(self) -> bool: Deletes the file from the container

License

Ellar is MIT licensed.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ellar_storage-0.1.7.tar.gz (12.6 kB view details)

Uploaded Source

Built Distribution

ellar_storage-0.1.7-py3-none-any.whl (12.3 kB view details)

Uploaded Python 3

File details

Details for the file ellar_storage-0.1.7.tar.gz.

File metadata

  • Download URL: ellar_storage-0.1.7.tar.gz
  • Upload date:
  • Size: 12.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.5

File hashes

Hashes for ellar_storage-0.1.7.tar.gz
Algorithm Hash digest
SHA256 31bead64753fb59623a4e2b72103ed7a56a92a9a3390bc0378113679831d0769
MD5 8a465942d4a30c6868dcbcc1d4d30aa4
BLAKE2b-256 b2f5158e7893926b1091923b015ec1fa6bd6117a93f34b5418065198fa7bb88c

See more details on using hashes here.

File details

Details for the file ellar_storage-0.1.7-py3-none-any.whl.

File metadata

File hashes

Hashes for ellar_storage-0.1.7-py3-none-any.whl
Algorithm Hash digest
SHA256 6d734adfee80a3f84e37091e3901328e39342ac3e3665fdf026ef9ee055c0dbc
MD5 d8cb48c1b513ba6188a329be8fa9be5f
BLAKE2b-256 dabfb4dd2ea16f22c5857a1282a41380a1779e3203ac46d006d1eb76010ecd22

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page