Storage Module for Ellar
Project description
Introduction
The EllarStorage Module enriches your Ellar application with robust support for managing both cloud and
local file storage.
Leveraging the capabilities of the Apache libcloud
package
to simplify file storage operations within your Ellar-powered projects.
Installation
$(venv) pip install ellar-storage
This library drew inspiration from sqlalchemy-file.
Usage
To integrate EllarStorage into your project, follow the standard Ellar project structure and then configure the module as follows:
StorageModule
Similar to other Ellar modules, the StorageModule
can be configured directly where it's used or through the application configuration.
StorageModule.setup
You can set up the StorageModule
using the setup
method. Here's a quick example:
import os
from pathlib import Path
from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider
BASE_DIRS = Path(__file__).parent
@Module(modules=[
StorageModule.setup(
files={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
images={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
documents={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
default="files"
)
])
class ApplicationModule(ModuleBase):
pass
In this example, after application initialization, folders for files
, images
, and documents
will be created in the specified directory. Each folder is configured to be managed by a local storage driver. You can explore other supported storage drivers.
StorageModule.register_setup
Alternatively, you can move the storage configuration to the application config:
## project_name/root_module.py
from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule
@Module(modules=[StorageModule.register_setup()])
class ApplicationModule(ModuleBase):
pass
Then, in config.py
, you can define the storage configurations:
import os
from pathlib import Path
from ellar.core.conf import ConfigDefaultTypesMixin
from ellar_storage import get_driver, Provider
BASE_DIRS = Path(__file__).parent
class DevelopmentConfig(ConfigDefaultTypesMixin):
DEBUG = True
STORAGE_CONFIG = dict(
storages=dict(
files={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
images={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
documents={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
}
),
default="files"
)
StorageController
StorageModule
also registers StorageController
which is useful when retrieving saved files.
This can be disabled by setting disable_storage_controller
to True
.
Also, StorageController
is not protected and will be accessible to the public.
However, it can be protected by simply applying @Guard
or @Authorize
decorator.
Retrieving Saved Data
By using request.url_for
, we can generate a download link for the file we wish to retrieve
For example:
from ellar.common import Inject, post
from ellar.core import Request
@post('/get-books')
def get_book_by_id(self, req: Request, book_id, session: Inject[Session]):
book = session.execute(
select(Book).where(Book.title == "Pointless Meetings")
).scalar_one()
return {
"title": book.title,
"cover": req.url_for("storage:download", path="{storage_name}/{file_name}"),
"thumbnail": req.url_for("storage:download", path=book.thumbnail.path)
}
With req.url_for("storage:download", path="{storage_name}/{file_name}")
,
we are able to create a download link to retrieve saved files.
StorageService
At the end of the StorageModule
setup, StorageService
is registered into the Ellar DI system. Here's a quick example of how to use it:
## project_name/server.py
import os
from ellar.app import AppFactory
from ellar.common import datastructures, constants
from ellar.core import LazyModuleImport as lazyLoad
from ellar_storage import StorageService
application = AppFactory.create_from_app_module(
lazyLoad("project_name.root_module:ApplicationModule"),
config_module=os.environ.get(
constants.ELLAR_CONFIG_MODULE, "carapp.config:DevelopmentConfig"
),
)
storage_service: StorageService = application.injector.get(StorageService)
# Example: save a file in the 'files' folder
storage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'files' folder", name="file.txt"), upload_storage='files')
# Example: save a file in the 'images' folder
storage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'images' folder", name="image.txt"), upload_storage='images')
# Example: save a file in the 'documents' folder
storage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'documents' folder", name="docs.txt"), upload_storage='documents')
StorageService in Route Functions
You can inject StorageService
into your controllers or route functions. For instance:
In Controller:
from ellar.common import ControllerBase, Controller
from ellar_storage import StorageService
@Controller()
class FileManagerController(ControllerBase):
def __init__(self, storage_service: StorageService):
self._storage_service = storage_service
In Route Function:
from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService
@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
pass
See Sample Project
Some Quick Cloud Setup
Google Cloud Storage
-
For a service Account
from ellar.common import Module from ellar.core import ModuleBase from ellar_storage import StorageModule, get_driver, Provider @Module(modules=[ StorageModule.setup( files={ # For a service Account "driver": get_driver(Provider.GOOGLE_STORAGE), "options": { "key": "client_email", "secret": "private_key", "...": "..." }, }, ) ]) class ApplicationModule(ModuleBase): pass
-
Installed Application
from ellar.common import Module from ellar.core import ModuleBase from ellar_storage import StorageModule, get_driver, Provider @Module(modules=[ StorageModule.setup( files={ # For a service Account "driver": get_driver(Provider.GOOGLE_STORAGE), "options": { "key": "client_id", "secret": "client_secret", "...": "..." }, }, ) ]) class ApplicationModule(ModuleBase): pass
-
GCE instance
from ellar.common import Module from ellar.core import ModuleBase from ellar_storage import StorageModule, get_driver, Provider @Module(modules=[ StorageModule.setup( files={ # For a service Account "driver": get_driver(Provider.GOOGLE_STORAGE), "options": { "key": "GOOG0123456789ABCXYZ", "secret": "key_secret", "...": "..." }, }, ) ]) class ApplicationModule(ModuleBase): pass
See GCS
AWS S3
from ellar.common import Module
from ellar.core import ModuleBase
from ellar_storage import StorageModule, get_driver, Provider
@Module(modules=[
StorageModule.setup(
files={
"driver": get_driver(Provider.S3),
"options": {
"key": "api key",
"secret": "api secret key"
},
},
)
])
class ApplicationModule(ModuleBase):
pass
Specifying canned ACL when uploading an object
If you want to specify custom ACL when uploading an object,
you can do so by passing extra
argument with the acl attribute to the save
or save_content
methods.
Valid values for this attribute are:
- private (default)
- public-read
- public-read-write
- authenticated-read
- bucket-owner-read
- bucket-owner-full-control
For example
from ellar.common import UploadFile, Inject, post
from ellar_storage import StorageService
@post('/upload')
def upload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
extra = {"content_type": "application/octet-stream", "acl": "public-read"}
stored_file = storage_service.save(file=file, extra=extra)
return {"message": f"{stored_file.filename} saved"}
API Reference
StorageService
- save(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Saves a file from an
UploadFile
object. - save_async(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Asynchronously saves a file from an
UploadFile
object. - save_content(self, **kwargs) -> StoredFile: Saves a file from content/bytes or through a file path.
- save_content_async(self, **kwargs) -> StoredFile: Asynchronously saves a file from content/bytes or through a file path.
- get(self, path: str) -> StoredFile: Retrieves a saved file if the specified
path
exists. Thepath
can be in the formatcontainer/filename.extension
orfilename.extension
. - get_async(self, path: str) -> StoredFile: Asynchronously retrieves a saved file if the specified
path
exists. - delete(self, path: str) -> bool: Deletes a saved file if the specified
path
exists. - delete_async(self, path: str) -> bool: Asynchronously deletes a saved file if the specified
path
exists. - get_container(self, name: Optional[str] = None) -> Container: Gets a
libcloud.storage.base.Container
instance for a configured storage setup.
StoredFile
StoredFile
is a file-like object returned from saving and retrieving files.
It extends some libcloud
Object methods and has a reference to the
libcloud
Object retrieved from the storage container.
Key attributes include:
- name: File name
- size: File size
- filename: File name
- content_type: File content type
- object:
libcloud
Object reference - read(self, n: int = -1, chunk_size: Optional[int] = None) -> bytes: Reads file content
- get_cdn_url(self) -> Optional[str]: Gets file CDN URL
- as_stream(self, chunk_size: Optional[int] = None) -> Iterator[bytes]: Creates a file stream
- delete(self) -> bool: Deletes the file from the container
License
Ellar is MIT licensed.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file ellar_storage-0.1.7.tar.gz
.
File metadata
- Download URL: ellar_storage-0.1.7.tar.gz
- Upload date:
- Size: 12.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31bead64753fb59623a4e2b72103ed7a56a92a9a3390bc0378113679831d0769 |
|
MD5 | 8a465942d4a30c6868dcbcc1d4d30aa4 |
|
BLAKE2b-256 | b2f5158e7893926b1091923b015ec1fa6bd6117a93f34b5418065198fa7bb88c |
File details
Details for the file ellar_storage-0.1.7-py3-none-any.whl
.
File metadata
- Download URL: ellar_storage-0.1.7-py3-none-any.whl
- Upload date:
- Size: 12.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d734adfee80a3f84e37091e3901328e39342ac3e3665fdf026ef9ee055c0dbc |
|
MD5 | d8cb48c1b513ba6188a329be8fa9be5f |
|
BLAKE2b-256 | dabfb4dd2ea16f22c5857a1282a41380a1779e3203ac46d006d1eb76010ecd22 |