OmniStream

中文 · English

A single-binary, streaming file browser and previewer — point it at any local directory or S3-compatible object storage (MinIO / OSS / Ceph / R2 / …) and it instantly exposes them as a browsable, previewable HTTP service. The backend is built on axum + tokio + aws-sdk-s3, with one StorageBackend trait abstracting over every supported backend; a React SPA is bundled in, so opening http://<host>:<port>/ lets you walk directories, lazy-load thumbnails, and preview files in place. Preview supports:

• Images — png / jpg / gif / webp / avif / bmp / svg / ico
• Video — mp4 / webm / mov / mkv / m4v / ogv, with Range-based seeking; optional user-triggered FFmpeg compatibility playback for browser-unsupported codecs
• Text / code — syntax highlighting by extension: json / yaml / toml / md / rs / ts / py / go / sql / shell / proto, and many more
• Tabular data — Parquet (pure-JS decode via hyparquet, with an embedded DuckDB SQL query tab) / CSV / TSV; also supports Rows View card rendering (see docs/parquet_rows_view_user_guide.md)
• 3D models — glb / gltf / obj / ply / stl / pcd / fbx with orbit, grid, and axis controls (FBX external textures must be embedded; GLTF sidecars are loaded from the same storage)
• NumPy arrays — npy / npz metadata and sampled (N, 3) / (N, 6) point-cloud rendering
• Safetensors weights — header-only Tensor key / shape / dtype listing with stored-element counts and storage-size estimates; large weight data is never downloaded for the preview
• Anything else — generic fallback: icon + metadata + the browser's built-in viewer

Previewing files on S3 / S3-compatible storage requires the configured access key to hold both s3:GetObject (preview / download / HEAD) and s3:ListBucket (directory browsing / thumbnail listing). Missing either yields a 403 on the corresponding action. If you omit s3.bucket to use multi-bucket mode (see below), the credentials must additionally hold s3:ListAllMyBuckets so the root listing can enumerate every visible bucket. Write operations (/api/convert converting to Parquet) additionally require s3:PutObject. The local filesystem backend has no such requirement, but is restricted to the directory configured as local.root_path.

1. Install

For uv, see the official documentation.

For source builds, frontend/backend development, or release instructions, see docs/development_guide.md and docs/how_to_release.md.

2. Configuration

If $OMNI_CONFIG is set, OmniStream uses that exact path without falling back, even when the file does not exist. Otherwise, it uses the first existing file from this lookup order:

1. $XDG_CONFIG_HOME/omni-stream/config.toml
2. directories::ProjectDirs platform default (macOS: ~/Library/Application Support/omni-stream/; Linux: ~/.config/omni-stream/)
3. ./config.toml (current directory)

config.example.toml in the repo root works as a template. A minimal config:

[server]
host = "127.0.0.1"
port = 28080

[[storages]]
name = "local-data"
type = "local"
active = true
local = { root_path = "/var/lib/omni-stream" }

Or S3 / S3-compatible (MinIO / OSS):

[[storages]]
name = "production-s3"
type = "s3"
active = true
s3 = { endpoint = "http://minio.local:9000", bucket = "data", access_key = "...", secret_key = "..." }

The local backend supports follow_symlinks (default true; when false, symlinks appear as entries but reads return Forbidden, and that storage refuses all DuckDB/SQL operations):

[[storages]]
name = "local-data"
type = "local"
active = true
local = { root_path = "/var/lib/omni-stream", follow_symlinks = false }  # default true

s3.region defaults to us-east-1, which is fine for MinIO / LocalStack and AWS us-east-1 buckets — leave it out by default. Only set it when: (1) the target AWS bucket lives outside us-east-1, since SigV4 has to use the bucket's actual region (otherwise AWS returns AuthorizationHeaderMalformed); or (2) the S3-compatible gateway validates the region strictly (most don't). s3.force_path_style defaults to true (required for MinIO and most self-hosted gateways); set it to false for virtual-host-style gateways (some AOSS/OSS).

s3.bucket is optional. Omit it (or set it to "*") to enable multi-bucket mode: the storage root performs ListBuckets, and every bucket the credentials can see appears as a top-level directory; navigating into one drills down with the usual prefix listing. The credentials must hold the s3:ListAllMyBuckets IAM permission. Example:

[[storages]]
name = "all-prod-s3"
type = "s3"
s3 = { endpoint = "http://minio.local:9000", access_key = "...", secret_key = "..." }

Multiple [[storages]] entries can coexist; on startup the one with active = true wins, and if none is active the first entry is used. The frontend also lets you switch between them at runtime.

Environment overrides (prefix OMNI_, separator _):

Authentication (optional)

By default the API is completely open — only suitable for trusted LAN environments. To enable Bearer token auth, add to your config:

[auth]
enabled = true
token = "any-long-random-string"

Or rely entirely on environment variables (keep the secret out of the config file):

OMNI_AUTH_ENABLED=true OMNI_AUTH_TOKEN=$(openssl rand -hex 32) ./omni-stream

Once enabled, the default behavior splits read and write access (public_read = true, the default):

• Browse / preview / download (/api/list, /api/stat, /api/proxy, /api/transcode, /api/thumb, /raw) stay public — no token required.
• Write operations (/api/convert for Parquet conversion) always require Authorization: Bearer <token>.
• SQL queries (/api/query) are in the read group and require no token by default; however the endpoint only activates when auth.enabled = true (it never runs on a fully open API).
• Frontend: a 401 on a write triggers a token-entry dialog; the token is stored in localStorage and the request is automatically retried. The toolbar also has an Auth Token button to pre-enter the token before any write.

To lock down reads as well (every API request requires the token), set public_read = false:

[auth]
enabled = true
token = "any-long-random-string"
public_read = false   # every request requires the token

Note: the /raw file mount relies on Bearer-header auth. Browser navigation and automatic subresource loads cannot attach that header; programmatic fetch / XHR can only do so when code explicitly sets Authorization. Therefore browser-opened /raw dashboards are not practically usable under full lockdown (public_read = false). The default public_read = true keeps /raw accessible.

The embedded SPA (/, /assets/*) is always open. TLS is out of scope — put nginx / caddy in front for HTTPS.

Thumbnails (optional)

Disabled by default; when enabled, the grid view uses WebP thumbnails instead of full-resolution originals, significantly reducing bandwidth:

[thumbnails]
enabled = true
# cache_path = "~/.cache/omni-stream/thumbs"  # default; override to set a custom cache dir
# quality = 70        # WebP quality 1-100, default 70
# max_cache_bytes = 1073741824  # default 1 GiB

For the full set of options see the [thumbnails] section in config.example.toml.

Compatible video playback

Browsers cannot decode every codec that can be stored in an MP4 or MOV container. OmniStream can offer user-triggered compatible playback when [transcoding] enabled = true and FFmpeg with the libx264 and aac encoders is available. The capability is disabled by default because it consumes CPU and temporary disk. If the probe fails, startup continues with a warning and the video error explains that server-side compatible playback is unavailable.

[transcoding]
enabled = true # opt in; default is false
# ffmpeg_path = "ffmpeg"
# max_concurrent = 1
# timeout_secs = 1800
# max_source_bytes = 2147483648  # default 2 GiB
# max_width = 1920
# max_height = 1080
# max_video_bitrate_kbps = 4000

The server reads through StorageBackend, pipes the source to FFmpeg, and streams fragmented H.264/AAC MP4 from stdout. It does not create or retain a transcoded output file and never buffers the complete video in memory. Inputs whose metadata requires seeking may use FFmpeg's temporary input cache in the system temp directory; it is removed when FFmpeg exits. Sources larger than max_source_bytes (or without a known length) are rejected before FFmpeg starts; set the limit according to available temporary disk.

Transcoding never starts automatically: the user must click Try compatible playback after native decoding fails. Set transcoding.enabled = false to disable the capability (it is already off by default). The default single-process limit rejects extra work with HTTP 429, each process has a timeout and one encoder thread, and disconnecting the browser stops it. Live output does not support seeking. Because encoding consumes CPU, expose it only on a suitably sized host or behind trusted access controls. With auth disabled, or with auth.public_read = true, the read endpoint is public; internet-facing deployments should put it behind proxy authentication or rate limiting before enabling it.

For every option see the [transcoding] section in config.example.toml.

SQL queries and format conversion (optional — requires duckdb build)

If the binary was built with --features duckdb (PyPI wheels include it; cargo install does not by default) and auth.enabled = true, the [sql] section lets you tune DuckDB behavior:

[sql]
# enabled = true            # default; kill-switch independent of the build feature
# memory_limit = "512MB"    # default; DuckDB memory limit per query connection
# threads = 1               # default; DuckDB threads per query connection; higher improves throughput but raises peak memory
# query_timeout_secs = 300    # default (5 min); interactive SELECT interrupted past this, returns 408
# convert_timeout_secs = 1800 # default (30 min); background JSONL/CSV→Parquet conversion timeout
# max_rows = 10000            # default; results beyond this are truncated (truncated = true)

For prerequisites and full usage (Parquet SQL tab, format conversion button), see docs/edit_features_guide.md.

Config CLI

If you'd rather not copy config.example.toml by hand, the binary ships three config subcommands:

# List every candidate location in priority order, and mark which one the
# loader will pick.
omni-stream config list

# Lay down the bundled config.example.toml at one of the candidate paths
# (interactive selection, with a "custom path" option).
omni-stream config init

# Parse + validate. Surfaces missing fields, wrong types, or an empty
# `storages` list. Without an argument it checks the active path; pass a
# path to validate it directly.
omni-stream config check
omni-stream config check ./my-config.toml

The template config init writes is the verbatim config.example.toml from the repo, embedded into the binary — no external file required.

3. Run

If you installed via cargo install, omni-stream is already on $PATH:

# Serve one local directory with zero configuration
omni-stream serve ./data

# Same, but listen on a different port
omni-stream serve ./data -p 9000

# Use the config.toml found by the §2 lookup order
omni-stream

# Or point at a specific one
OMNI_CONFIG=/etc/omni-stream/config.toml omni-stream

# Or override just the port
OMNI_SERVER_PORT=8081 omni-stream

# Turn on request logging while debugging
RUST_LOG=info,tower_http=debug omni-stream

omni-stream serve <location> binds 127.0.0.1:28080 and uses a read-only local backend with the built-in defaults. -p / --port <PORT> is the only override; it does not load config.toml or OMNI_* configuration variables. Compatible playback is disabled for this zero-config command; use a config file with [transcoding] enabled = true when you want FFmpeg playback. Use the existing omni-stream form when you need a non-loopback host, authentication, or other storage backends.

Tarballs from GitHub Releases don't add themselves to $PATH, so either run ./omni-stream from the extracted directory or move it somewhere like /usr/local/bin/.

After startup, opening http://<host>:<port>/ in a browser lands you on the embedded SPA. Ctrl-C / SIGTERM triggers a graceful shutdown (axum::serve + with_graceful_shutdown).

4. HTTP API

The bundled SPA uses these endpoints; they are also available to curl and other clients:

• GET /api/server / GET /api/storages — server info and storage list
• GET /api/list?prefix=&page_token=&skip_pages= — browse a directory
• GET /api/stat/{*key} — fetch file metadata
• GET /api/proxy/{*key} — stream a file with transparent Range support
• GET /api/transcode/{*key} — optional user-triggered H.264/AAC compatibility stream
• GET /api/thumb/{*key} — on-demand WebP thumbnail
• POST /api/query — DuckDB read-only SQL (requires the duckdb feature)
• POST /api/convert — JSONL / NDJSON / TSV / CSV → Parquet conversion
• GET /raw/{storage} / GET /raw/{storage}/{*path} — navigable file mount
• Embedded SPA fallback — non-API paths fall back to index.html

For endpoint prerequisites and full SQL / conversion usage, see docs/edit_features_guide.md.

5. HTTP Error Semantics

Most error responses are {"error": "...", "message": "..."} JSON. Denied object reads use the structured ObjectReadForbidden shape: {"error": "...", "code": "OBJECT_READ_FORBIDDEN", "message": "...", "hint": "..."}.