Persistent cache for aiohttp requests
Project description
aiohttp-client-cache
aiohttp-client-cache is an async persistent cache for aiohttp client requests.
See full documentation at https://aiohttp-client-cache.readthedocs.io
Development Status
This is an early work in progress!
The current state is a working drop-in replacement (or mixin) for aiohttp.ClientSession
, with
multiple asynchronous cache backends.
Breaking changes should be expected until a 1.0
release.
Installation
Requires python 3.7+
Install the latest stable version with pip:
pip install aiohttp-client-cache
Note: You will need additional dependencies depending on which backend you want to use; See Cache Backends section below for details. To install with extra dependencies for all supported backends:
pip install aiohttp-client-cache[backends]
See Contributing for setup info for local development.
Usage Examples
See the examples folder for more detailed usage examples.
Here is a simple example using an endpoint that takes 1 second to fetch. After the first request, subsequent requests to the same URL will return near-instantly; so, fetching it 10 times will only take ~1 second instead of 10.
from aiohttp_client_cache import CachedSession, SQLiteBackend
async with CachedSession(cache=SQLiteBackend()) as session:
for i in range(10):
await session.get('http://httpbin.org/delay/1')
Here is an example with more customized behavior:
cache = SQLiteBackend(
cache_name='~/.cache/aiohttp-requests.db', # For SQLite, this will be used as the filename
expire_after=24, # By default, cached responses expire in a day
expire_after_urls={'*.site.com/static': 24*7}, # Requests with this pattern will expire in a week
ignored_params=['auth_token'], # Ignore this param when caching responses
)
async with CachedSession(cache=cache) as session:
await session.get('https://site.com/index.html') # Expires in a day
await session.get('https://img.site.com/static/a27bf6.jpg') # Expires in a week
See CacheBackend documentation for more usage details.
aiohttp-client-cache
can also be used as a mixin, if you happen have other mixin classes that you
want to combine with it:
from aiohttp import ClientSession
from aiohttp_client_cache import CacheMixin
class CustomSession(CacheMixin, CustomMixin, ClientSession):
pass
Cache Backends
Several backends are available. If one isn't specified, a non-persistent in-memory cache will be used.
SQLiteBackend
: Uses a SQLite database (requires aiosqlite)RedisBackend
: Uses a Redis cache (requires redis-py)MongoDBBackend
: Uses a MongoDB database (requires motor)
Incomplete:
DynamoDBBackend
: Uses a Amazon DynamoDB database (requires boto3)GridFSBackend
: Uses a MongoDB GridFS database, which enables storage of documents greater than 16MB (requires pymongo)
You can also provide your own backend by subclassing aiohttp_client_cache.backends.BaseCache
.
Expiration
If you are using the expire_after
parameter, expired responses are removed from the storage the
next time the same request is made. If you want to manually purge all expired items, you can use
CachedSession.delete_expired_responses
. Example:
session = CachedSession(expire_after=3) # Cached responses expire after 3 hours
await session.remove_expired_responses() # Remove any responses over 3 hours old
Conditional Caching
Caching behavior can be customized by defining various conditions:
- Response status codes
- Request HTTP methods
- Request headers
- Specific request parameters
- Custom filter function
See CacheBackend docs for details.
Related Projects
Other python cache projects you may want to check out:
- aiohttp-cache: A server-side async HTTP cache for the
aiohttp
web server - diskcache: A general-purpose (not HTTP-specific) file-based cache built on SQLite
- aiocache: General-purpose (not HTTP-specific) async cache backends
- requests-cache An HTTP cache for
requests
; also served as a starting point for makingaiohttp-client-cache
- CacheControl: An HTTP cache for
requests
that caches according to uses HTTP headers and status codes
Credits
Thanks to Roman Haritonov and
contributors
for the original requests-cache
!
This project is licensed under the MIT license, with the exception of portions of
storage backend code
adapted from requests-cache
, which is licensed under the BSD license
(copy included).
History
0.3.0 (TBD)
- Add support for setting different expiration times based on URL patterns
- Add support for serializing/deserializing
ClientSession.links
- Add case-insensitive response headers for compatibility with aiohttp.ClientResponse.headers
0.2.0 (2021-02-28)
- Refactor SQLite backend to use
aiosqlite
for async cache operations - Refactor MongoDB backend to use
motor
for async cache operations - Refactor Redis backend to use
aiosqlite
for async cache operations - Add integration tests and
docker-compose
for local test servers
0.1.0 (2020-11-14)
- Initial PyPI release
- First pass at a general refactor and conversion from
requests
toaiohttp
- Basic features are functional, but some backends do not actually operate asynchronously
requests-cache
See requests-cache
development history
for details on prior changes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aiohttp-client-cache-0.2.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a280ae6092adf5b1eba3974d4e87f5749bd5b0a113b97fbfde5e2dd617c41a42 |
|
MD5 | 0122671261cc8c9be98e7da96cefff65 |
|
BLAKE2b-256 | 6e2ae29b0feedf4be21c1ddacb709bf986b98efb506b7c4a2a2f233092131f44 |
Hashes for aiohttp_client_cache-0.2.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2bc22ccc1bfd82be21c3b774a526d912c87c69bdaee99bceb12d3333b19bb0f3 |
|
MD5 | 06082fe88c8b7b0ab46793eefea0576e |
|
BLAKE2b-256 | 8502d90701325dab6c9f0e129e686eddb8a8901fc5f2c3964932b176be12bb5c |