speedcopy

Patched python shutil.copyfile using native call CopyFile2 on windows to accelerate transfer on windows shares. On Linux, it issues special ioctl command CIFS_IOC_COPYCHUNK_FILE to enable server-side copy.

This works only when both source and destination files are on same SMB1(CIFS)/2/3 filesystem.

See https://wiki.samba.org/index.php/Server-Side_Copy

Installation

Add speedcopy to PYTHONPATH or:

pip install speedcopy

Usage

If you want to monkeypatch shutil.copyfile() then:

import shutil
import speedcopy

speedcopy.patch_copyfile()

# your code ...
shutil.copyfile(src, dst)

This will make last call to use speedcopy.

Direct use:

import speedcopy

# some code ...

speedcopy.copyfile(src, dst)

There is also debug mode enabled by setting speedcopy.SPEEDCOPY_DEBUG = True. This will print more information during runtime.

Benchmark

You can run benchmark using benchmark.py script. It will run copy operations with different file sizes and print the results in a table format.

Usage

Benchmark can run in two modes: multithreaded and single-threaded. In multithreaded mode, it will run multiple copy operations in parallel using multiple workers. In single-threaded mode, it will run copy operations sequentially.

Arguments:

python benchmark.py PATH [--sizes-mb SIZES_MB] [--repeats REPEATS] [--copies-per-worker COPIES_PER_WORKER] [--workers WORKERS]

• PATH: Path to the directory where the benchmark files will be created and copied. This should be a path on an SMB/CIFS share for accurate results.
• --sizes-mb: Comma-separated list of file sizes in MB to test (default: 1,2,4,8,16,32).
• --repeats: Number of times to repeat each copy operation (default: 3).
• --copies-per-worker: Number of copy operations each worker should perform in multithreaded mode (default: 2).
• --workers: Number of worker threads to use in multithreaded mode (default: 4).

If workers is not set or set to 1, it will run in single-threaded mode.

Windows

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

overall gain was 5.41x

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

overall gain was 7.27x

Linux

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

maOS

Based on the measured values, there is no significant gain on macOS. The gain is around 1.05x in multithreaded mode and around 1.5x in single-threaded mode, which is not significant enough. It is possible that the file server wasn't configured to support server-side copy for macOS (on samba, you need to have specific options). Even though I've tested the configuration, and it should be working, it's possible that there is some issue with the setup.

running with --sizes-mb 1,2,4,8,16,32 --repeats 3 --copies-per-worker 2 --workers 4

Multithreaded mode

running with --sizes-mb 1,2,4,8,16,32 --repeats 3

Single-threaded mode

Note that Windows, Linux and macOS timings do not correlate, it is taken from different systems. Also note that these figures are not taken from production grade hardware and setup and can be completely off at other places.