Skip to main content

Optimized collectstatic for S3ManifestStaticStorage

Project description


Optimized collectstatic for S3ManifestStaticStorage


  1. Install the package
pip install s3manifestcollectstatic
  1. Add s3manifestcollectstatic to INSTALLED_APPS


collectstatic can take a long time. When used with storages.backends.s3boto3.S3ManifestStaticStorage, collectstatic uploads the files twice, once without the hash at the end of the file name, and once with the hash. Also, it doesn't use multiple threads to upload to s3.

s3manifestcollectstatic uploads the files only once, uses threads to speed things up, and doesn't upload the files that are already on S3.

collectstatic: (Around 20 minutes)

$ time ./ collectstatic --noinput

604 static files copied, 646 post-processed.
./ collectstatic --noinput  29,94s user 2,27s system 2% cpu 20:25,06 total

s3manifestcollectstatic: (Around 30 seconds)

$ time ./ s3manifestcollectstatic

604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.
Start the upload of 604 files
Uploading the manifest
./ s3manifestcollectstatic  10,95s user 1,92s system 49% cpu 26,269 total

If you want to reupload the files use -f:

./ s3manifestcollectstatic -f

You can limit the maximum number of workers using -w:

./ s3manifestcollectstatic -w 4

Tested with:

  • Python 3.9, Django 3.2, django-storages 1.11
  • Python 3.11, Django 4.2, django-storages 1.13

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3manifestcollectstatic-0.3.2.tar.gz (3.6 kB view hashes)

Uploaded Source

Built Distribution

s3manifestcollectstatic-0.3.2-py3-none-any.whl (4.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page