๐ Faster ModernGL Buffer inter process data transfers
Project description
[!IMPORTANT] Also check out ShaderFlow, where TurboPipe shines! ๐
๐ฅ Description
TurboPipe speeds up sending raw bytes from
moderngl.Buffer
objects primarily toFFmpeg
subprocess
The optimizations involved are:
- Zero-copy: Avoid unnecessary memory copies or allocation (intermediate
buffer.read()
) - C++: The core of TurboPipe is written in C++ for speed, efficiency and low-level control
- Chunks: Write in chunks of 4096 bytes (RAM page size), so the hardware is happy
- Threaded:
- Doesn't block Python code execution, allows to render next frame
- Decouples the main thread from the I/O thread for performance
โ
Don't worry, there's proper safety in place. TurboPipe will block Python if a memory address is already queued for writing, and guarantees order of writes per file-descriptor. Just call .sync()
when done ๐
๐ฆ Installation
It couldn't be easier! Just install in your package manager:
pip install turbopipe
poetry add turbopipe
pdm add turbopipe
rye add turbopipe
๐ Usage
See also the Examples folder for more controlled usage, and ShaderFlow usage of it!
import subprocess
import moderngl
import turbopipe
# Create ModernGL objects
ctx = moderngl.create_standalone_context()
buffer = ctx.buffer(reserve=1920*1080*3)
# Make sure resolution, pixel format matches!
ffmpeg = subprocess.Popen(
'ffmpeg -f rawvideo -pix_fmt rgb24 -s 1920x1080 -i - -f null -'.split(),
stdin=subprocess.PIPE
)
# Rendering loop of yours
for _ in range(100):
turbopipe.pipe(buffer, ffmpeg.stdin.fileno())
# Finalize writing
turbo.sync()
ffmpeg.stdin.close()
ffmpeg.wait()
โญ๏ธ Benchmarks
[!NOTE] The tests conditions are as follows:
- The tests are the average of 3 runs to ensure consistency, with 3 GB of the same data being piped
- The data is a random noise per-buffer between 128-135. So, multi-buffers runs are a noise video
- All resolutions are wide-screen (16:9) and have 3 components (RGB) with 3 bytes per pixel (SDR)
- Multi-buffer cycles through a list of buffer (eg. 1, 2, 3, 1, 2, 3... for 3-buffers)
- All FFmpeg outputs are scrapped with
-f null -
to avoid any disk I/O bottlenecks- The
gain
column is the percentage increase over the standard method- When
x264
is Null, no encoding took place (passthrough)- The test cases emoji signifies:
- ๐ข: Standard
ffmpeg.stdin.write(buffer.read())
on just the main thread, pure Python- ๐: Threaded
ffmpeg.stdin.write(buffer.read())
with a queue (similar to turbopipe)- ๐: The magic of
turbopipe.pipe(buffer, ffmpeg.stdin.fileno())
Also see
benchmark.py
for the implementation
โ Check out benchmarks in a couple of systems below:
Desktop โข (AMD Ryzen 9 5900x) โข (NVIDIA RTX 3060 12 GB) โข (DDR4 2x32 GB 3200 MT/s) โข (Arch Linux)
720p | x264 | Buffers | Framerate | Bandwidth | Gain |
---|---|---|---|---|---|
๐ข | Null | 1 | 882 fps | 2.44 GB/s | |
๐ | Null | 1 | 793 fps | 2.19 GB/s | -10.04% |
๐ | Null | 1 | 1911 fps | 5.28 GB/s | 116.70% |
๐ข | Null | 4 | 818 fps | 2.26 GB/s | |
๐ | Null | 4 | 684 fps | 1.89 GB/s | -16.35% |
๐ | Null | 4 | 1494 fps | 4.13 GB/s | 82.73% |
๐ข | ultrafast | 4 | 664 fps | 1.84 GB/s | |
๐ | ultrafast | 4 | 635 fps | 1.76 GB/s | -4.33% |
๐ | ultrafast | 4 | 869 fps | 2.40 GB/s | 31.00% |
๐ข | slow | 4 | 204 fps | 0.57 GB/s | |
๐ | slow | 4 | 205 fps | 0.57 GB/s | 0.58% |
๐ | slow | 4 | 208 fps | 0.58 GB/s | 2.22% |
1080p | x264 | Buffers | Framerate | Bandwidth | Gain |
---|---|---|---|---|---|
๐ข | Null | 1 | 385 fps | 2.40 GB/s | |
๐ | Null | 1 | 369 fps | 2.30 GB/s | -3.91% |
๐ | Null | 1 | 641 fps | 3.99 GB/s | 66.54% |
๐ข | Null | 4 | 387 fps | 2.41 GB/s | |
๐ | Null | 4 | 359 fps | 2.23 GB/s | -7.21% |
๐ | Null | 4 | 632 fps | 3.93 GB/s | 63.40% |
๐ข | ultrafast | 4 | 272 fps | 1.70 GB/s | |
๐ | ultrafast | 4 | 266 fps | 1.66 GB/s | -2.14% |
๐ | ultrafast | 4 | 405 fps | 2.53 GB/s | 49.24% |
๐ข | slow | 4 | 117 fps | 0.73 GB/s | |
๐ | slow | 4 | 122 fps | 0.76 GB/s | 4.43% |
๐ | slow | 4 | 124 fps | 0.77 GB/s | 6.48% |
1440p | x264 | Buffers | Framerate | Bandwidth | Gain |
---|---|---|---|---|---|
๐ข | Null | 1 | 204 fps | 2.26 GB/s | |
๐ | Null | 1 | 241 fps | 2.67 GB/s | 18.49% |
๐ | Null | 1 | 297 fps | 3.29 GB/s | 45.67% |
๐ข | Null | 4 | 230 fps | 2.54 GB/s | |
๐ | Null | 4 | 235 fps | 2.61 GB/s | 2.52% |
๐ | Null | 4 | 411 fps | 4.55 GB/s | 78.97% |
๐ข | ultrafast | 4 | 146 fps | 1.62 GB/s | |
๐ | ultrafast | 4 | 153 fps | 1.70 GB/s | 5.21% |
๐ | ultrafast | 4 | 216 fps | 2.39 GB/s | 47.96% |
๐ข | slow | 4 | 73 fps | 0.82 GB/s | |
๐ | slow | 4 | 78 fps | 0.86 GB/s | 7.06% |
๐ | slow | 4 | 79 fps | 0.88 GB/s | 9.27% |
2160p | x264 | Buffers | Framerate | Bandwidth | Gain |
---|---|---|---|---|---|
๐ข | Null | 1 | 81 fps | 2.03 GB/s | |
๐ | Null | 1 | 107 fps | 2.67 GB/s | 32.26% |
๐ | Null | 1 | 213 fps | 5.31 GB/s | 163.47% |
๐ข | Null | 4 | 87 fps | 2.18 GB/s | |
๐ | Null | 4 | 109 fps | 2.72 GB/s | 25.43% |
๐ | Null | 4 | 212 fps | 5.28 GB/s | 143.72% |
๐ข | ultrafast | 4 | 59 fps | 1.48 GB/s | |
๐ | ultrafast | 4 | 67 fps | 1.68 GB/s | 14.46% |
๐ | ultrafast | 4 | 95 fps | 2.39 GB/s | 62.66% |
๐ข | slow | 4 | 37 fps | 0.94 GB/s | |
๐ | slow | 4 | 43 fps | 1.07 GB/s | 16.22% |
๐ | slow | 4 | 44 fps | 1.11 GB/s | 20.65% |
Desktop โข (AMD Ryzen 9 5900x) โข (NVIDIA RTX 3060 12 GB) โข (DDR4 2x32 GB 3200 MT/s) โข (Windows 11)
๐ Conclusion
TurboPipe significantly increases the feeding speed of FFmpeg with data, especially at higher resolutions. However, if there's few CPU compute available, or the video is too hard to encode (slow preset), the gains are insignificant over the other methods (bottleneck). Multi-buffering didn't prove to have an advantage, debugging shows that TurboPipe C++ is often starved of data to write (as the file stream is buffered on the OS most likely), and the context switching, or cache misses, might be the cause of the slowdown.
Interestingly, due either Linux's scheduler on AMD Ryzen CPUs, or their operating philosophy, it was experimentally seen that Ryzen's frenetic thread switching degrades a bit the single thread performance, which can be "fixed" with prepending the command with taskset --cpu 0,2
(not recommended at all), comparatively speaking to Windows performance on the same system (Linux ๐ = Windows ๐ข). This can also be due the topology of tested CPUs having more than one Core Complex Die (CCD). Intel CPUs seem to stick to the same thread for longer, which makes the Python threaded method an unecessary overhead.
Personal experience
On realistically loads, like ShaderFlow's default lightweight shader export, TurboPipe increases rendering speed from 1080p260 to 1080p330 on my system, with mid 80% CPU usage than low 60%s. For DepthFlow's default depth video export, no gains are seen, as the CPU is almost saturated encoding at 1080p130.
๐ Future work
- Add support for NumPy arrays, memoryviews, and byte-like objects
- Improve the thread synchronization and/or use a ThreadPool
- Maybe use
mmap
instead of chunks writing - Test on MacOS ๐
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file turbopipe-1.0.4.tar.gz
.
File metadata
- Download URL: turbopipe-1.0.4.tar.gz
- Upload date:
- Size: 99.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49bc432274b9516b70f3a60a5dfab37047f743dea1fcc64ca383831bd97c7bc1 |
|
MD5 | 64a47beaab27c02740fcc45465b5cc57 |
|
BLAKE2b-256 | 8eee794f06737a214c363a1d44812b574547ba6d4fb0cd393b789693c752e042 |
File details
Details for the file turbopipe-1.0.4-cp312-cp312-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp312-cp312-win_amd64.whl
- Upload date:
- Size: 22.4 kB
- Tags: CPython 3.12, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5836b992167082b17497cfb7d72f086fdb305399e0c3138fcdad794b15ca456c |
|
MD5 | e9300a303e4899f6eadd48224b4176ee |
|
BLAKE2b-256 | c521440663d6bd609c3ac08238b9864bfd69afc639349d3768eca071fceb1401 |
File details
Details for the file turbopipe-1.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 33.4 kB
- Tags: CPython 3.12, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a95d7f0ff0aee0792816098ae17edddeea1ef701f5ba865e332739af1551c83a |
|
MD5 | f10704bdb0575b13d217cbf683f69e11 |
|
BLAKE2b-256 | 7559ff0436978324faac68eb9cedab519a46e2073f94cc1da60acad4f10c4205 |
File details
Details for the file turbopipe-1.0.4-cp312-cp312-macosx_11_0_arm64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp312-cp312-macosx_11_0_arm64.whl
- Upload date:
- Size: 30.0 kB
- Tags: CPython 3.12, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc6085bc0bddb91b52b45b67efe586e9ee96d27ddd3e435eb2d8ece0f7f6633c |
|
MD5 | bada4bc0cf947db09649d6c1046fdc35 |
|
BLAKE2b-256 | 13df5a710c192cce8a91109dcc08de5c851053fdbebc0e7a711995c0c3d2206c |
File details
Details for the file turbopipe-1.0.4-cp311-cp311-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp311-cp311-win_amd64.whl
- Upload date:
- Size: 22.4 kB
- Tags: CPython 3.11, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09765b40efd6ab42456d93fb6b3df17b9af70ce6cd59821542c3bd1c3508089a |
|
MD5 | 8fe3f1a42e6d843525b90ddf8a67a0d8 |
|
BLAKE2b-256 | 7a236cc8e57c485238921bf0535845fe17b49fc06fe18cf6918bed385dd20d28 |
File details
Details for the file turbopipe-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 33.5 kB
- Tags: CPython 3.11, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab809ae492d35ff235dc2b6a93939b3cc7ef7d5d0ba4bfb720eca3f595104697 |
|
MD5 | 088b84d547cf923b94b798d9950ca3aa |
|
BLAKE2b-256 | f15f293d415116c29057e38145c270d76f013a32081128285c18849ee3d40771 |
File details
Details for the file turbopipe-1.0.4-cp311-cp311-macosx_11_0_arm64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp311-cp311-macosx_11_0_arm64.whl
- Upload date:
- Size: 30.0 kB
- Tags: CPython 3.11, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a9ff8094e79574ed514562e460ca11b896e697c1fd26f2693746c192820d62d |
|
MD5 | ffd735b9bb5a30a6160d51586332341f |
|
BLAKE2b-256 | 61f10ba782f0de3cfd226f54621859c13b1367dd96193aff06e1e08615d4c081 |
File details
Details for the file turbopipe-1.0.4-cp310-cp310-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp310-cp310-win_amd64.whl
- Upload date:
- Size: 22.5 kB
- Tags: CPython 3.10, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c589326dffc4f07ce029184faa78c91daee559ba74c9477f8b01622564cd5864 |
|
MD5 | 76996fe41c8e791cc1fb282d65d6f328 |
|
BLAKE2b-256 | 09b800b5bbdcac05358d53fee39d86d411b4fa922634f7bb1569dc42ca38ef12 |
File details
Details for the file turbopipe-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 33.5 kB
- Tags: CPython 3.10, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b02b5284f9a77118be96581161ad46e71314430e489f1a04e3865c180f847376 |
|
MD5 | 6c77911d0300549796e97b94aa0b99da |
|
BLAKE2b-256 | ed92d7c477cc0f6651433340315ae0962f76cdec230a481e6d70b78f8ffcc5e1 |
File details
Details for the file turbopipe-1.0.4-cp310-cp310-macosx_11_0_arm64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp310-cp310-macosx_11_0_arm64.whl
- Upload date:
- Size: 30.0 kB
- Tags: CPython 3.10, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 468830834a6d49be5c8b1b3a3e7c631e1e2df42d65eadd20d53d2935609c1633 |
|
MD5 | bdf3bd6b8f154722dfb2b541685fa9e3 |
|
BLAKE2b-256 | f05e9168835271c438e8b4ca4cbd258b5d77276392bca9f8ddfce2289049c6cc |
File details
Details for the file turbopipe-1.0.4-cp39-cp39-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp39-cp39-win_amd64.whl
- Upload date:
- Size: 22.5 kB
- Tags: CPython 3.9, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2f3ec6bda377c949fc24743016e83420fa3c2b80b22457a9f8bae7c99a860a9 |
|
MD5 | e43bca6f0cded34843897927ddefcf0f |
|
BLAKE2b-256 | 1e2108e410a6786e1f5bb07e8e86e2fdec813fe94b6529caad84b4866e686a80 |
File details
Details for the file turbopipe-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 33.5 kB
- Tags: CPython 3.9, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff3505d150bd8c00bcb925a6b1b0501011ee7372d4f21d96fa5a4b4a2f45b0e8 |
|
MD5 | 5f742221dfa30a8c23565e14cfdf4b7b |
|
BLAKE2b-256 | ff5d2a2a1a78715a140f1032f24cb1b02827782e79d4cf71e187a0205e14bb19 |
File details
Details for the file turbopipe-1.0.4-cp39-cp39-macosx_11_0_arm64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp39-cp39-macosx_11_0_arm64.whl
- Upload date:
- Size: 30.0 kB
- Tags: CPython 3.9, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a765094989e678f1117a8c80abb06e10ae2dced58d2520574b72dd18ac97ae3 |
|
MD5 | 883d4176944f06316273d377aeb345d1 |
|
BLAKE2b-256 | 72925a9f6dd051c58f513956627120930a61562c39e0fa66a8bad0574d3c96be |
File details
Details for the file turbopipe-1.0.4-cp38-cp38-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp38-cp38-win_amd64.whl
- Upload date:
- Size: 22.5 kB
- Tags: CPython 3.8, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c117efaed5600819e970e3615459472615597e8d2699b950408e4ef2ba97775d |
|
MD5 | 56fd3d617b4e5b2e1eadd4bea6bd8055 |
|
BLAKE2b-256 | decb835b3e50fcbb9f49ffa1126178b55dea4e38f02b46acaccd6cc96580868c |
File details
Details for the file turbopipe-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 28.9 kB
- Tags: CPython 3.8, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6fb73fac9e08d233f61f6be04f8b0aa418fdf4492548865c7babf01678a4c43 |
|
MD5 | 44ecccc1bf406b0aaa59d3ebe07010cd |
|
BLAKE2b-256 | 1c3b5fa8772fe29ca69883a8fba30a3b46d503a09cdbba7a8ea7076b4a07ac9a |
File details
Details for the file turbopipe-1.0.4-cp38-cp38-macosx_11_0_arm64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp38-cp38-macosx_11_0_arm64.whl
- Upload date:
- Size: 30.1 kB
- Tags: CPython 3.8, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4e314de341cfaf9ef44b3fce30f1cb82d411dab72aadabfe4bbe33ecb4f2ac6 |
|
MD5 | 816e81b754b9c4b27edbd4dd2a892f9f |
|
BLAKE2b-256 | c30dd453bbb3b01b923b90cf8e127e8810874174b1f6893c1f2c15000257640c |
File details
Details for the file turbopipe-1.0.4-cp37-cp37m-win_amd64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 22.5 kB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0df71aaba14ca0f6c1f2810a27c7179020101fe2a8e711f332a57acc5f906bb9 |
|
MD5 | 476e0022d1499b98607984a1079b3a40 |
|
BLAKE2b-256 | 57e35704e837fd367d69c320e4b8c9758d43c75af8a5102abcca35b811db1838 |
File details
Details for the file turbopipe-1.0.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: turbopipe-1.0.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 28.9 kB
- Tags: CPython 3.7m, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0035ee7e8e49e5c44b1e9e80ac4725cf1330d9ba8e21a08c5a48fe176a2fb69 |
|
MD5 | 10697ab082d1092664137260c0e60609 |
|
BLAKE2b-256 | caf949c0197acac0e17e4bbc9078eb428c2698b4724e492db3b9d2bccb1676be |