Skip to main content

Speed up shutil.copyfile by using sendfile system call

Project description

Latest version Tests status Coverage Supported Python versions

pyfastcopy is a simple Python module that monkey patches the shutil.copyfile function of Python standard library to internally use the sendfile system call.

It can provide important performance improvements for large file copy (typically 30-40%). See the performance section for some numbers.

Because shutil.copyfile is used by other fonctions in the shutil module, the following functions also automatically benefit from the performance boost:

  • shutil.copy

  • shutil.copy2

  • shutil.copytree

If sendfile is not available on your system or fails, the classic, slower chunk file copy is used, so there is no downside to using this module.

For more information, see my proposed patch for Python.

Performance

Tests were done copying files (source and destination) on a tmpfs filesystem, so that no slowdown related to hard drive or SSD storage occurs. Test files were generated with pseudo random data using frandom.

See benchmark.py for details about the test procedure and how the following graphs were generated.

Python 3.4: graph1 graph2 graph3

Python 3.6: graph4 graph5 graph6

These tests show a 30-50% performance improvement of ``shutil.copyfile`` compared to stock Python.

Usage

Just import the module:

import pyfastcopy

And then use the patched shutil.copyfile as usual:

shutil.copyfile(src, dst)

The ``sendfile`` system call does not exist on Windows, so importing this module will have no effect.

Installation

From PyPI (with PIP)

  1. If you don’t already have it, install pip for Python 3 (not needed if you are using Python >= 3.4)

  2. Install pyfastcopy: pip3 install pyfastcopy

  3. Enjoy fast copy when using shutil.copyfile, shutil.copy, shutil.copy2 or shutil.copytree

From source

  1. If you don’t already have it, install setuptools for Python 3

  2. Clone this repository: git clone https://github.com/desbma/pyfastcopy

  3. Install pyfastcopy: python3 setup.py install

  4. Enjoy fast copy when using shutil.copyfile, shutil.copy, shutil.copy2 or shutil.copytree

License

Python Software Foundation License

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyfastcopy-1.0.3.tar.gz (4.4 kB view details)

Uploaded Source

File details

Details for the file pyfastcopy-1.0.3.tar.gz.

File metadata

  • Download URL: pyfastcopy-1.0.3.tar.gz
  • Upload date:
  • Size: 4.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for pyfastcopy-1.0.3.tar.gz
Algorithm Hash digest
SHA256 ed4620f1087a8949888973e315d3d59fbe9b8cc4ca5df553d76d2f21d2748999
MD5 21a4a7fc94f94015c0bec1e3c7898954
BLAKE2b-256 4380535d6b3de415e26d0a1cb774c6895dd07aa5986d2f8bde200393bd916790

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page