Skip to main content

๐ŸŒ€ Faster ModernGL Buffer inter process data transfers

Project description

[!IMPORTANT] Also check out ShaderFlow, where TurboPipe shines! ๐Ÿ˜‰

TurboPipe


Faster ModernGL inter-process data transfers

๐Ÿ”ฅ Description

TurboPipe speeds up sending raw bytes from moderngl.Buffer objects primarily to FFmpeg subprocess

The optimizations involved are:

  • Zero-copy: Avoid unnecessary memory copies or allocation (intermediate buffer.read())
  • C++: The core of TurboPipe is written in C++ for speed, efficiency and low-level control
  • Chunks: Write in chunks of 4096 bytes (RAM page size), so the hardware is happy
  • Threaded:
    • Doesn't block Python code execution, allows to render next frame
    • Decouples the main thread from the I/O thread for performance

โœ… Don't worry, there's proper safety in place. TurboPipe will block Python if a memory address is already queued for writing, and guarantees order of writes per file-descriptor. Just call .sync() when done ๐Ÿ˜‰


๐Ÿ“ฆ Installation

It couldn't be easier! Just install in your package manager:

pip install turbopipe
poetry add turbopipe
pdm add turbopipe
rye add turbopipe

๐Ÿš€ Usage

See also the Examples folder for more controlled usage, and ShaderFlow usage of it!

import subprocess
import moderngl
import turbopipe

# Create ModernGL objects
ctx = moderngl.create_standalone_context()
buffer = ctx.buffer(reserve=1920*1080*3)

# Make sure resolution, pixel format matches!
ffmpeg = subprocess.Popen(
    'ffmpeg -f rawvideo -pix_fmt rgb24 -s 1920x1080 -i - -f null -'.split(),
    stdin=subprocess.PIPE
)

# Rendering loop of yours
for _ in range(100):
    turbopipe.pipe(buffer, ffmpeg.stdin.fileno())

# Finalize writing
turbo.sync()
ffmpeg.stdin.close()
ffmpeg.wait()

โญ๏ธ Benchmarks

[!NOTE] The tests conditions are as follows:

  • The tests are the average of 3 runs to ensure consistency, with 3 GB of the same data being piped
  • The data is a random noise per-buffer between 128-135. So, multi-buffers runs are a noise video
  • All resolutions are wide-screen (16:9) and have 3 components (RGB) with 3 bytes per pixel (SDR)
  • Multi-buffer cycles through a list of buffer (eg. 1, 2, 3, 1, 2, 3... for 3-buffers)
  • All FFmpeg outputs are scrapped with -f null - to avoid any disk I/O bottlenecks
  • The gain column is the percentage increase over the standard method
  • When x264 is Null, no encoding took place (passthrough)
  • The test cases emoji signifies:
    • ๐Ÿข: Standard ffmpeg.stdin.write(buffer.read()) on just the main thread, pure Python
    • ๐Ÿš€: Threaded ffmpeg.stdin.write(buffer.read()) with a queue (similar to turbopipe)
    • ๐ŸŒ€: The magic of turbopipe.pipe(buffer, ffmpeg.stdin.fileno())

Also see benchmark.py for the implementation

โœ… Check out benchmarks in a couple of systems below:

Desktop โ€ข (AMD Ryzen 9 5900x) โ€ข (NVIDIA RTX 3060 12 GB) โ€ข (DDR4 2x32 GB 3200 MT/s) โ€ข (Arch Linux)
720p x264 Buffers Framerate Bandwidth Gain
๐Ÿข Null 1 882 fps 2.44 GB/s
๐Ÿš€ Null 1 793 fps 2.19 GB/s -10.04%
๐ŸŒ€ Null 1 1911 fps 5.28 GB/s 116.70%
๐Ÿข Null 4 818 fps 2.26 GB/s
๐Ÿš€ Null 4 684 fps 1.89 GB/s -16.35%
๐ŸŒ€ Null 4 1494 fps 4.13 GB/s 82.73%
๐Ÿข ultrafast 4 664 fps 1.84 GB/s
๐Ÿš€ ultrafast 4 635 fps 1.76 GB/s -4.33%
๐ŸŒ€ ultrafast 4 869 fps 2.40 GB/s 31.00%
๐Ÿข slow 4 204 fps 0.57 GB/s
๐Ÿš€ slow 4 205 fps 0.57 GB/s 0.58%
๐ŸŒ€ slow 4 208 fps 0.58 GB/s 2.22%
1080p x264 Buffers Framerate Bandwidth Gain
๐Ÿข Null 1 385 fps 2.40 GB/s
๐Ÿš€ Null 1 369 fps 2.30 GB/s -3.91%
๐ŸŒ€ Null 1 641 fps 3.99 GB/s 66.54%
๐Ÿข Null 4 387 fps 2.41 GB/s
๐Ÿš€ Null 4 359 fps 2.23 GB/s -7.21%
๐ŸŒ€ Null 4 632 fps 3.93 GB/s 63.40%
๐Ÿข ultrafast 4 272 fps 1.70 GB/s
๐Ÿš€ ultrafast 4 266 fps 1.66 GB/s -2.14%
๐ŸŒ€ ultrafast 4 405 fps 2.53 GB/s 49.24%
๐Ÿข slow 4 117 fps 0.73 GB/s
๐Ÿš€ slow 4 122 fps 0.76 GB/s 4.43%
๐ŸŒ€ slow 4 124 fps 0.77 GB/s 6.48%
1440p x264 Buffers Framerate Bandwidth Gain
๐Ÿข Null 1 204 fps 2.26 GB/s
๐Ÿš€ Null 1 241 fps 2.67 GB/s 18.49%
๐ŸŒ€ Null 1 297 fps 3.29 GB/s 45.67%
๐Ÿข Null 4 230 fps 2.54 GB/s
๐Ÿš€ Null 4 235 fps 2.61 GB/s 2.52%
๐ŸŒ€ Null 4 411 fps 4.55 GB/s 78.97%
๐Ÿข ultrafast 4 146 fps 1.62 GB/s
๐Ÿš€ ultrafast 4 153 fps 1.70 GB/s 5.21%
๐ŸŒ€ ultrafast 4 216 fps 2.39 GB/s 47.96%
๐Ÿข slow 4 73 fps 0.82 GB/s
๐Ÿš€ slow 4 78 fps 0.86 GB/s 7.06%
๐ŸŒ€ slow 4 79 fps 0.88 GB/s 9.27%
2160p x264 Buffers Framerate Bandwidth Gain
๐Ÿข Null 1 81 fps 2.03 GB/s
๐Ÿš€ Null 1 107 fps 2.67 GB/s 32.26%
๐ŸŒ€ Null 1 213 fps 5.31 GB/s 163.47%
๐Ÿข Null 4 87 fps 2.18 GB/s
๐Ÿš€ Null 4 109 fps 2.72 GB/s 25.43%
๐ŸŒ€ Null 4 212 fps 5.28 GB/s 143.72%
๐Ÿข ultrafast 4 59 fps 1.48 GB/s
๐Ÿš€ ultrafast 4 67 fps 1.68 GB/s 14.46%
๐ŸŒ€ ultrafast 4 95 fps 2.39 GB/s 62.66%
๐Ÿข slow 4 37 fps 0.94 GB/s
๐Ÿš€ slow 4 43 fps 1.07 GB/s 16.22%
๐ŸŒ€ slow 4 44 fps 1.11 GB/s 20.65%
Desktop โ€ข (AMD Ryzen 9 5900x) โ€ข (NVIDIA RTX 3060 12 GB) โ€ข (DDR4 2x32 GB 3200 MT/s) โ€ข (Windows 11)

๐ŸŒ€ Conclusion

TurboPipe significantly increases the feeding speed of FFmpeg with data, especially at higher resolutions. However, if there's few CPU compute available, or the video is too hard to encode (slow preset), the gains are insignificant over the other methods (bottleneck). Multi-buffering didn't prove to have an advantage, debugging shows that TurboPipe C++ is often starved of data to write (as the file stream is buffered on the OS most likely), and the context switching, or cache misses, might be the cause of the slowdown.

Interestingly, due either Linux's scheduler on AMD Ryzen CPUs, or their operating philosophy, it was experimentally seen that Ryzen's frenetic thread switching degrades a bit the single thread performance, which can be "fixed" with prepending the command with taskset --cpu 0,2 (not recommended at all), comparatively speaking to Windows performance on the same system (Linux ๐Ÿš€ = Windows ๐Ÿข). This can also be due the topology of tested CPUs having more than one Core Complex Die (CCD). Intel CPUs seem to stick to the same thread for longer, which makes the Python threaded method an unecessary overhead.

Personal experience

On realistically loads, like ShaderFlow's default lightweight shader export, TurboPipe increases rendering speed from 1080p260 to 1080p330 on my system, with mid 80% CPU usage than low 60%s. For DepthFlow's default depth video export, no gains are seen, as the CPU is almost saturated encoding at 1080p130.


๐Ÿ“š Future work

  • Add support for NumPy arrays, memoryviews, and byte-like objects
  • Improve the thread synchronization and/or use a ThreadPool
  • Maybe use mmap instead of chunks writing
  • Test on MacOS ๐Ÿ™ˆ

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

turbopipe-1.0.4.tar.gz (99.9 kB view details)

Uploaded Source

Built Distributions

turbopipe-1.0.4-cp312-cp312-win_amd64.whl (22.4 kB view details)

Uploaded CPython 3.12 Windows x86-64

turbopipe-1.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.4 kB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

turbopipe-1.0.4-cp312-cp312-macosx_11_0_arm64.whl (30.0 kB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

turbopipe-1.0.4-cp311-cp311-win_amd64.whl (22.4 kB view details)

Uploaded CPython 3.11 Windows x86-64

turbopipe-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.5 kB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

turbopipe-1.0.4-cp311-cp311-macosx_11_0_arm64.whl (30.0 kB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

turbopipe-1.0.4-cp310-cp310-win_amd64.whl (22.5 kB view details)

Uploaded CPython 3.10 Windows x86-64

turbopipe-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.5 kB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

turbopipe-1.0.4-cp310-cp310-macosx_11_0_arm64.whl (30.0 kB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

turbopipe-1.0.4-cp39-cp39-win_amd64.whl (22.5 kB view details)

Uploaded CPython 3.9 Windows x86-64

turbopipe-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.5 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

turbopipe-1.0.4-cp39-cp39-macosx_11_0_arm64.whl (30.0 kB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

turbopipe-1.0.4-cp38-cp38-win_amd64.whl (22.5 kB view details)

Uploaded CPython 3.8 Windows x86-64

turbopipe-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.9 kB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

turbopipe-1.0.4-cp38-cp38-macosx_11_0_arm64.whl (30.1 kB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

turbopipe-1.0.4-cp37-cp37m-win_amd64.whl (22.5 kB view details)

Uploaded CPython 3.7m Windows x86-64

turbopipe-1.0.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.9 kB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

File details

Details for the file turbopipe-1.0.4.tar.gz.

File metadata

  • Download URL: turbopipe-1.0.4.tar.gz
  • Upload date:
  • Size: 99.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for turbopipe-1.0.4.tar.gz
Algorithm Hash digest
SHA256 49bc432274b9516b70f3a60a5dfab37047f743dea1fcc64ca383831bd97c7bc1
MD5 64a47beaab27c02740fcc45465b5cc57
BLAKE2b-256 8eee794f06737a214c363a1d44812b574547ba6d4fb0cd393b789693c752e042

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp312-cp312-win_amd64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 5836b992167082b17497cfb7d72f086fdb305399e0c3138fcdad794b15ca456c
MD5 e9300a303e4899f6eadd48224b4176ee
BLAKE2b-256 c521440663d6bd609c3ac08238b9864bfd69afc639349d3768eca071fceb1401

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a95d7f0ff0aee0792816098ae17edddeea1ef701f5ba865e332739af1551c83a
MD5 f10704bdb0575b13d217cbf683f69e11
BLAKE2b-256 7559ff0436978324faac68eb9cedab519a46e2073f94cc1da60acad4f10c4205

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 fc6085bc0bddb91b52b45b67efe586e9ee96d27ddd3e435eb2d8ece0f7f6633c
MD5 bada4bc0cf947db09649d6c1046fdc35
BLAKE2b-256 13df5a710c192cce8a91109dcc08de5c851053fdbebc0e7a711995c0c3d2206c

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 09765b40efd6ab42456d93fb6b3df17b9af70ce6cd59821542c3bd1c3508089a
MD5 8fe3f1a42e6d843525b90ddf8a67a0d8
BLAKE2b-256 7a236cc8e57c485238921bf0535845fe17b49fc06fe18cf6918bed385dd20d28

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ab809ae492d35ff235dc2b6a93939b3cc7ef7d5d0ba4bfb720eca3f595104697
MD5 088b84d547cf923b94b798d9950ca3aa
BLAKE2b-256 f15f293d415116c29057e38145c270d76f013a32081128285c18849ee3d40771

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3a9ff8094e79574ed514562e460ca11b896e697c1fd26f2693746c192820d62d
MD5 ffd735b9bb5a30a6160d51586332341f
BLAKE2b-256 61f10ba782f0de3cfd226f54621859c13b1367dd96193aff06e1e08615d4c081

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 c589326dffc4f07ce029184faa78c91daee559ba74c9477f8b01622564cd5864
MD5 76996fe41c8e791cc1fb282d65d6f328
BLAKE2b-256 09b800b5bbdcac05358d53fee39d86d411b4fa922634f7bb1569dc42ca38ef12

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b02b5284f9a77118be96581161ad46e71314430e489f1a04e3865c180f847376
MD5 6c77911d0300549796e97b94aa0b99da
BLAKE2b-256 ed92d7c477cc0f6651433340315ae0962f76cdec230a481e6d70b78f8ffcc5e1

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 468830834a6d49be5c8b1b3a3e7c631e1e2df42d65eadd20d53d2935609c1633
MD5 bdf3bd6b8f154722dfb2b541685fa9e3
BLAKE2b-256 f05e9168835271c438e8b4ca4cbd258b5d77276392bca9f8ddfce2289049c6cc

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: turbopipe-1.0.4-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 22.5 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for turbopipe-1.0.4-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 f2f3ec6bda377c949fc24743016e83420fa3c2b80b22457a9f8bae7c99a860a9
MD5 e43bca6f0cded34843897927ddefcf0f
BLAKE2b-256 1e2108e410a6786e1f5bb07e8e86e2fdec813fe94b6529caad84b4866e686a80

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ff3505d150bd8c00bcb925a6b1b0501011ee7372d4f21d96fa5a4b4a2f45b0e8
MD5 5f742221dfa30a8c23565e14cfdf4b7b
BLAKE2b-256 ff5d2a2a1a78715a140f1032f24cb1b02827782e79d4cf71e187a0205e14bb19

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1a765094989e678f1117a8c80abb06e10ae2dced58d2520574b72dd18ac97ae3
MD5 883d4176944f06316273d377aeb345d1
BLAKE2b-256 72925a9f6dd051c58f513956627120930a61562c39e0fa66a8bad0574d3c96be

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: turbopipe-1.0.4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 22.5 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for turbopipe-1.0.4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 c117efaed5600819e970e3615459472615597e8d2699b950408e4ef2ba97775d
MD5 56fd3d617b4e5b2e1eadd4bea6bd8055
BLAKE2b-256 decb835b3e50fcbb9f49ffa1126178b55dea4e38f02b46acaccd6cc96580868c

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c6fb73fac9e08d233f61f6be04f8b0aa418fdf4492548865c7babf01678a4c43
MD5 44ecccc1bf406b0aaa59d3ebe07010cd
BLAKE2b-256 1c3b5fa8772fe29ca69883a8fba30a3b46d503a09cdbba7a8ea7076b4a07ac9a

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d4e314de341cfaf9ef44b3fce30f1cb82d411dab72aadabfe4bbe33ecb4f2ac6
MD5 816e81b754b9c4b27edbd4dd2a892f9f
BLAKE2b-256 c30dd453bbb3b01b923b90cf8e127e8810874174b1f6893c1f2c15000257640c

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: turbopipe-1.0.4-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 22.5 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for turbopipe-1.0.4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 0df71aaba14ca0f6c1f2810a27c7179020101fe2a8e711f332a57acc5f906bb9
MD5 476e0022d1499b98607984a1079b3a40
BLAKE2b-256 57e35704e837fd367d69c320e4b8c9758d43c75af8a5102abcca35b811db1838

See more details on using hashes here.

File details

Details for the file turbopipe-1.0.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for turbopipe-1.0.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b0035ee7e8e49e5c44b1e9e80ac4725cf1330d9ba8e21a08c5a48fe176a2fb69
MD5 10697ab082d1092664137260c0e60609
BLAKE2b-256 caf949c0197acac0e17e4bbc9078eb428c2698b4724e492db3b9d2bccb1676be

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page