speedcopy

Patched python shutil.copyfile using native call CopyFile2 on windows to accelerate transfer on windows shares. On Linux, it issues special ioctl command CIFS_IOC_COPYCHUNK_FILE to enable server-side copy.

This works only when both source and destination files are on same SMB1(CIFS)/2/3 filesystem.

See https://wiki.samba.org/index.php/Server-Side_Copy

Installation

Add speedcopy to PYTHONPATH or:

pip install speedcopy

Usage

If you want to monkeypatch shutil.copyfile() then:

import shutil
import speedcopy

speedcopy.patch_copyfile()

# your code ...
shutil.copyfile(src, dst)

This will make last call to use speedcopy.

Direct use:

import speedcopy

# some code ...

speedcopy.copyfile(src, dst)

There is also debug mode enabled by setting speedcopy.SPEEDCOPY_DEBUG = True. This will print more information during runtime.

Releasing

Releases are automated by the 🚀 Release workflow (.github/workflows/pythonpublish.yml):

1. Run the workflow from the Actions tab on the branch to release from (usually develop). Pick the version part to bump (patch, minor, major, none) and the stage (final, alpha, beta, rc, or stable to drop a pre-release suffix). Examples from 2.2.1a1: none + stable gives 2.2.1, patch + alpha gives 2.2.2a1.
2. The workflow runs lint and tests, bumps the version in pyproject.toml, uv.lock and speedcopy/version.py, commits, and pushes tag vX.Y.Z.
3. Distributions are built, checked, published to TestPyPI and then PyPI (Trusted Publishing with attestations; the pypi environment needs approval), and attached to a GitHub Release with generated notes.

Pushing a v* tag by hand also runs the pipeline, but the tag must match the version in pyproject.toml.

Benchmark

You can run benchmark using benchmark.py script. It will run copy operations with different file sizes and print the results in a table format.

Usage

Benchmark can run in two modes: multithreaded and single-threaded. In multithreaded mode, it will run multiple copy operations in parallel using multiple workers. In single-threaded mode, it will run copy operations sequentially.

Arguments:

python benchmark.py PATH [--sizes-mb SIZES_MB] [--repeats REPEATS] [--copies-per-worker COPIES_PER_WORKER] [--workers WORKERS]

• PATH: Path to the directory where the benchmark files will be created and copied. This should be a path on an SMB/CIFS share for accurate results.
• --sizes-mb: Comma-separated list of file sizes in MB to test (default: 1,2,4,8,16,32).
• --repeats: Number of times to repeat each copy operation (default: 3).
• --copies-per-worker: Number of copy operations each worker should perform in multithreaded mode (default: 2).
• --workers: Number of worker threads to use in multithreaded mode (default: 4).

If workers is not set or set to 1, it will run in single-threaded mode.

Windows

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

overall gain was 5.41x

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

overall gain was 7.27x

Linux

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

maOS

Based on the measured values, there is no significant gain on macOS. The gain is around 1.05x in multithreaded mode and around 1.5x in single-threaded mode, which is not significant enough. It is possible that the file server wasn't configured to support server-side copy for macOS (on samba, you need to have specific options). Even though I've tested the configuration, and it should be working, it's possible that there is some issue with the setup.

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

Note that Windows, Linux and macOS timings do not correlate, it is taken from different systems. Also note that these figures are not taken from production grade hardware and setup and can be completely off at other places.