ellar-storage 0.1.5

Storage Module for Ellar

Introduction

The EllarStorage Module enriches your Ellar application with robust support for managing both cloud and local file storage. Leveraging the capabilities of the Apache libcloud package to simplify file storage operations within your Ellar-powered projects.

Installation

$(venv) pip install ellar-storage

This library drew inspiration from sqlalchemy-file.

Usage

To integrate EllarStorage into your project, follow the standard Ellar project structure and then configure the module as follows:

StorageModule

Similar to other Ellar modules, the StorageModule can be configured directly where it's used or through the application configuration.

StorageModule.setup

You can set up the StorageModule using the setup method. Here's a quick example:

import os
from pathlib import Path
from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider

BASE_DIRS = Path(__file__).parent

@Module(modules=[
    StorageModule.setup(
        files={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        images={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        documents={
            "driver": get_driver(Provider.LOCAL),
            "options": {"key": os.path.join(BASE_DIRS, "media")},
        },
        default="files"
    )
])
class ApplicationModule(ModuleBase):
    pass

In this example, after application initialization, folders for files, images, and documents will be created in the specified directory. Each folder is configured to be managed by a local storage driver. You can explore other supported storage drivers.

StorageModule.register_setup

Alternatively, you can move the storage configuration to the application config:

## project_name/root_module.py

from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule

@Module(modules=[StorageModule.register_setup()])
class ApplicationModule(ModuleBase):
    pass

Then, in config.py, you can define the storage configurations:

import os
from pathlib import Path
from ellar.core.conf import ConfigDefaultTypesMixin
from ellar_storage import get_driver, Provider

BASE_DIRS = Path(__file__).parent

class DevelopmentConfig(ConfigDefaultTypesMixin):
    DEBUG = True

    STORAGE_CONFIG = dict(
        storages=dict(
            files={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            },
            images={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            },
            documents={
                "driver": get_driver(Provider.LOCAL),
                "options": {"key": os.path.join(BASE_DIRS, "media")},
            }
        ),
        default="files"
    )

StorageController

StorageModule also registers StorageController which is useful when retrieving saved files. This can be disabled by setting disable_storage_controller to True.

Also, StorageController is not protected and will be accessible to the public. However, it can be protected by simply applying @Guard or @Authorize decorator.

Retrieving Saved Data

By using request.url_for, we can generate a download link for the file we wish to retrieve For example:

from ellar.common import Inject, post
from ellar.core import Request

@post('/get-books')
def get_book_by_id(self, req: Request, book_id, session: Inject[Session]):
    book = session.execute(
        select(Book).where(Book.title == "Pointless Meetings")
    ).scalar_one()
    
    return {
      "title": book.title,
      "cover": req.url_for("storage:download", path="{storage_name}/{file_name}"),
      "thumbnail": req.url_for("storage:download", path=book.thumbnail.path)
    }

With req.url_for("storage:download", path="{storage_name}/{file_name}"), we are able to create a download link to retrieve saved files.

StorageService

At the end of the StorageModule setup, StorageService is registered into the Ellar DI system. Here's a quick example of how to use it:

## project_name/server.py

import os
from ellar.app import AppFactory
from ellar.common import datastructures, constants
from ellar.core import LazyModuleImport as lazyLoad
from ellar_storage import StorageService

application = AppFactory.create_from_app_module(
    lazyLoad("project_name.root_module:ApplicationModule"),
    config_module=os.environ.get(
        constants.ELLAR_CONFIG_MODULE, "carapp.config:DevelopmentConfig"
    ),
)

storage_service: StorageService = application.injector.get(StorageService)
# Example: save a file in the 'files' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'files' folder", name="file.txt"), upload_storage='files')
# Example: save a file in the 'images' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'images' folder", name="image.txt"), upload_storage='images')
# Example: save a file in the 'documents' folder
storage_service.save(
    file=datastructures.ContentFile(b"We can now save files in the 'documents' folder", name="docs.txt"), upload_storage='documents')

StorageService in Route Functions

You can inject StorageService into your controllers or route functions. For instance:

In Controller:

from ellar.common import ControllerBase, Controller
from ellar_storage import StorageService

@Controller()
class FileManagerController(ControllerBase):
    def __init__(self, storage_service: StorageService):
        self._storage_service = storage_service

In Route Function:

from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService

@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
    pass

See Sample Project

Some Quick Cloud Setup

Google Cloud Storage

• For a service Account

  from ellar.common import Module
  from ellar.core import ModuleBase
  from ellar_storage import StorageModule, get_driver, Provider
  
  @Module(modules=[
      StorageModule.setup(
          files={
              # For a service Account
              "driver": get_driver(Provider.GOOGLE_STORAGE),
              "options": {
                  "key": "client_email",
                  "secret": "private_key",
                  "...": "..."
              },
          },
      )
  ])
  class ApplicationModule(ModuleBase):
      pass
  

• Installed Application

  from ellar.common import Module
  from ellar.core import ModuleBase
  from ellar_storage import StorageModule, get_driver, Provider
  
  @Module(modules=[
      StorageModule.setup(
          files={
              # For a service Account
              "driver": get_driver(Provider.GOOGLE_STORAGE),
              "options": {
                  "key": "client_id",
                  "secret": "client_secret",
                  "...": "..."
              },
          },
      )
  ])
  class ApplicationModule(ModuleBase):
      pass
  

• GCE instance

  from ellar.common import Module
  from ellar.core import ModuleBase
  from ellar_storage import StorageModule, get_driver, Provider
  
  @Module(modules=[
      StorageModule.setup(
          files={
              # For a service Account
              "driver": get_driver(Provider.GOOGLE_STORAGE),
              "options": {
                  "key": "GOOG0123456789ABCXYZ",
                  "secret": "key_secret",
                  "...": "..."
              },
          },
      )
  ])
  class ApplicationModule(ModuleBase):
      pass
  

See GCS

AWS S3

from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider

@Module(modules=[
    StorageModule.setup(
        files={
            "driver": get_driver(Provider.S3),
            "options": {
                "key": "api key",
                "secret": "api secret key"
            },
        },
    )
])
class ApplicationModule(ModuleBase):
    pass

Specifying canned ACL when uploading an object

If you want to specify custom ACL when uploading an object, you can do so by passing extra argument with the acl attribute to the save or save_content methods.

Valid values for this attribute are:

• private (default)
• public-read
• public-read-write
• authenticated-read
• bucket-owner-read
• bucket-owner-full-control

For example

from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService

@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
    extra = {"content_type": "application/octet-stream", "acl": "public-read"}
    stored_file = storage_service.save(file=file, extra=extra)
    
    return {"message": f"{stored_file.filename} saved"}

API Reference

StorageService

• save(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Saves a file from an UploadFile object.
• save_async(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Asynchronously saves a file from an UploadFile object.
• save_content(self, **kwargs) -> StoredFile: Saves a file from content/bytes or through a file path.
• save_content_async(self, **kwargs) -> StoredFile: Asynchronously saves a file from content/bytes or through a file path.
• get(self, path: str) -> StoredFile: Retrieves a saved file if the specified path exists. The path can be in the format container/filename.extension or filename.extension.
• get_async(self, path: str) -> StoredFile: Asynchronously retrieves a saved file if the specified path exists.
• delete(self, path: str) -> bool: Deletes a saved file if the specified path exists.
• delete_async(self, path: str) -> bool: Asynchronously deletes a saved file if the specified path exists.
• get_container(self, name: Optional[str] = None) -> Container: Gets a libcloud.storage.base.Container instance for a configured storage setup.

StoredFile

StoredFile is a file-like object returned from saving and retrieving files. It extends some libcloud Object methods and has a reference to the libcloud Object retrieved from the storage container.

Key attributes include:

• name: File name
• size: File size
• filename: File name
• content_type: File content type
• object: libcloud Object reference
• read(self, n: int = -1, chunk_size: Optional[int] = None) -> bytes: Reads file content
• get_cdn_url(self) -> Optional[str]: Gets file CDN URL
• as_stream(self, chunk_size: Optional[int] = None) -> Iterator[bytes]: Creates a file stream
• delete(self) -> bool: Deletes the file from the container

License

Ellar is MIT licensed.