SCIVEO

Sciveo is a Python toolkit and CLI for ML/AI workflows, engineering automation, experiment tracking, machine monitoring, local operations, network/industrial IO, media processing, AI agents, local storage, and production ML inference.

It is meant for practical systems where models, data, devices, services, and operators meet: train or evaluate ML models, collect measurements, process media, monitor machines, communicate with edge or plant equipment, run agents, and keep the operational tooling close to the Python code that uses it.

The package is intentionally broad. The CLI is the easiest entry point for operations and lab work, while the Python modules expose the same building blocks for notebooks, services, experiments, and pipelines.

ML/AI and Engineering Focus

Sciveo is built around the common shape of applied ML, AI, and engineering work:

• define and run parameterized experiments;
• sample parameter spaces and record configurations;
• attach datasets, metrics, scores, plots, media, and generated artifacts;
• monitor machines that produce data or run services;
• communicate with lab, plant, edge, and networked devices;
• capture and process image/video/audio data;
• run local or remote ML inference close to the data;
• coordinate AI agents and local tools;
• synchronize results to an API-backed project when remote tracking is needed.

The operations modules are part of the same picture. Monitoring, network IO, local storage, media workers, and admin tooling are there because real ML and engineering deployments often include cameras, sensors, edge boxes, industrial controllers, GPU machines, local disks, APIs, and long-running background jobs.

Main Capabilities

• Monitoring and watchdogs: host metrics, GPU/server telemetry where available, plant/Modbus monitoring, local watchdog actions, and service installation helpers.
• Experiments and project helpers: project runs, parameter sampling, datasets, scores, plots, metadata, and optional remote synchronization.
• Network and industrial IO: network discovery, SSH inventory/execution, Modbus TCP/RTU reads and writes, SNMP, MQTT, HTTP helpers, and protocol emulators.
• Admin and diagnostics: first-boot edge administration, Wi-Fi access point provisioning, service checks, doctor reports, and fleet inventory.
• Media capture and processing: RTSP/NVR capture, screen/camera capture, local media processing, ML media processors, and queue/API-backed media workers.
• AI agent console: interactive and one-shot agent workflows with hosted providers, local tools, image inputs, and optional local Hugging Face/GGUF runtimes.
• Encrypted chat rooms: shared-token chat rooms for lightweight operator and agent coordination.
• Local S3-compatible storage: a boto3-compatible object store backed by mounted local paths.
• Certification readiness tests: local, framework-neutral self-assessment, evidence-window scoring, and remediation suggestions, starting with an unofficial SOC 2-style security profile.
• SCIVEYOLO inference: Sciveo-owned SCIVEYOLO inference artifacts and runtime for production object detection without requiring Ultralytics at runtime.
• Supporting API, DB, web, content, and tools modules: shared building blocks used by experiments, services, and pipeline jobs.

Typical Workflows

Sciveo is useful when a project needs more than one isolated script. The common pattern is a Python workflow that touches models, measurements, local services, devices, and result tracking.

ML/AI Workflow

Use Sciveo to keep model work reproducible and close to the runtime environment:

import sciveo

def train_one_run():
    with sciveo.open() as experiment:
        learning_rate = experiment.config.learning_rate
        batch_size = experiment.config.batch_size

        metrics = {
            "loss": 0.18,
            "accuracy": 0.94,
        }

        experiment.log({"learning_rate": learning_rate, "batch_size": batch_size})
        experiment.eval("loss", metrics["loss"])
        experiment.eval("accuracy", metrics["accuracy"])
        experiment.score(metrics["accuracy"])

sciveo.start(
    project="vision-model-eval",
    function=train_one_run,
    configuration={
        "learning_rate": {"values": [0.001, 0.0005]},
        "batch_size": {"values": [8, 16]},
    },
    remote=False,
    count=4,
)

Engineering and Edge Workflow

Use the CLI to inspect and operate machines that produce data or run inference:

sciveo doctor --render json
sciveo scan --net 192.168.10.0/24 --health
sciveo fleet --net 192.168.10.0/24 --users operator,admin -i ~/.ssh/id_ed25519 --render json
sciveo monitor --period 60
sciveo watchdog --src memory --threshold 90 --period 10 --execute "systemctl restart app"

Media and Inference Workflow

Use media commands for local processing, and .sciveyolo artifacts for production object detection without requiring Ultralytics at runtime:

sciveo rtsp --url rtsp://camera/stream --output-path ./clip.mp4
sciveo media-run --input-path ./clip.mp4 --output-path ./media-out --render text
sciveo media-run --mode ml --processor image-object-detection --input-path ./frame.jpg --output-path ./detections

from sciveo.ml.images.sciveyolo import SCIVEYOLO

model = SCIVEYOLO("detector.sciveyolo")
results = model.predict("frame.jpg")

Device and Protocol Workflow

Use network and industrial IO helpers directly from the CLI during bring-up, testing, and diagnostics:

sciveo read --proto modbus --transport tcp --host 192.168.0.10 --port 502 --id 1 --address 30001 --kind input --type RAW --count 2
sciveo write --proto mqtt --host broker.local --topic plant/cmd --payload '{"limit":80}' --qos 1 --retain
sciveo emulate --server http --host 0.0.0.0 --port 8080 --data-json '{"status":{"ok":true}}'

Installation

Base install:

pip install sciveo

Useful extras:

pip install "sciveo[mon]"          # host monitoring helpers
pip install "sciveo[net]"          # network, Modbus, SNMP, MQTT helpers
pip install "sciveo[agent]"        # hosted AI agent providers
pip install "sciveo[agent-local]"  # local HF/GGUF runtime support
pip install "sciveo[media]"        # media capture/processing
pip install "sciveo[media-ml]"     # ML model/runtime dependencies
pip install "sciveo[server]"       # API/server helpers
pip install "sciveo[web]"          # Django/web helpers
pip install "sciveo[all]"          # core operations extras

media-ml intentionally does not install Ultralytics. For image/video ML workflows, install both media extras so OpenCV and model runtimes are present:

pip install "sciveo[media,media-ml]"

Sciveo SCIVEYOLO runtime loads .sciveyolo artifacts with PyTorch or ONNX Runtime depending on the artifact engine.

CLI Overview

Show CLI help and version:

sciveo --help
sciveo --version
sciveo help --json
sciveo help monitor
sciveo help monitor --json

Current top-level CLI commands:

init
monitor
watchdog
scan
ssh
read
write
emulate
admin
doctor
fleet
extensions
certification
nvr
rtsp
capture
media-server
media-run
agent
chat
storage
predictors-server

The sciveo help --json manifest is the machine-readable source for command summaries, usage forms, options, notes, and examples.

Certification Readiness Tests

See CERTIFICATION.md for the platform architecture, verification strategy, implemented scope, and phased roadmap.

Generate an editable SOC 2-style test input, complete its answers and evidence metadata, then evaluate it locally:

sciveo certification --action template --framework soc2 --output-path soc2-test.json
sciveo certification --action example --scenario software-company --output-path software-company.json
sciveo certification --action evaluate --framework soc2 --input-path soc2-test.json
sciveo certification --action evaluate --framework soc2 --input-path soc2-test.json --render json --output-path soc2-report.json

The initial profile is a Sciveo-authored general security checklist. It scores the supplied status and evidence metadata, highlights critical gaps, checks a configurable observation target (90 days by default), and returns concrete suggestions. Reports also contain structured control-test results, a ranked remediation plan, evidence-reference classifications, domain scores, and per-system gap summaries. The summary is an explanatory roll-up; per-check system reference coverage can cap scores. A reference does not mean the control is supported, and Sciveo does not open or verify referenced evidence.

This feature is an unofficial readiness self-assessment. It is not a SOC 2 examination, audit, certification, CPA opinion, or guarantee of an audit result; it does not reproduce the AICPA Trust Services Criteria. An independent licensed CPA firm determines the actual examination scope, observation period, testing, exceptions, and report. Use sciveo certification --help for the CLI contract.

The generated template is the versioned machine-readable input contract. Answers use implemented, partial, not_implemented, not_assessed, or not_applicable; the last status requires a text rationale, and every proposed exclusion remains scored until an independent review process exists. Evidence is a list of non-empty text references or objects containing name, reference, path, url, or id, plus optional source, system_ids, and an ISO observed_at date. source alone is not a concrete reference. Dated evidence outside the configured period is classified and ignored for that period. Answers may also set applicable_system_ids, gap_system_ids, applicability_rationale, remediation_action, owner, and target_date. Omitting a declared system requires a rationale and remains an explicit scope review item; critical scope exclusions prevent the strong readiness band. Targets must be integer values from 1–366 days. An explicitly supplied --observation-days overrides the input target; otherwise the input value wins, with 90 days as the fallback.

Complete Hybrid Software Company Example

Generate the bundled fictional assessment and evaluate it without contacting any external service:

sciveo certification --action example \
  --scenario software-company \
  --output-path software-company.json

sciveo certification --action evaluate \
  --input-path software-company.json

sciveo certification --action evaluate \
  --input-path software-company.json \
  --render json \
  --output-path software-company-report.json

The example fixes a 90-day window from 2026-01-01 through 2026-03-31 so its inputs and score remain reproducible. It models the following system boundary:

The synthetic answers produce a stable 60.0% developing readiness result, seven critical gaps, and ten ranked remediation tasks: Git and AWS practices are comparatively mature, while shared local accounts, edge firmware governance, centralized edge/NVR logging, NVR restoration tests, and device-vendor lifecycle reviews create remediation work. This is an example of how the scoring behaves, not a conclusion about any real company.

The example builder is frozen local data. Shared Sciveo imports may load global configuration, but certification does not authenticate to services or use, transmit, or copy credential values. It never invokes Git, connects to AWS, opens SSH sessions, contacts self-hosted servers, or talks to edge devices, cameras, or NVRs. Replace the synthetic inventory, answers, evidence IDs, owners, dates, and gaps with your own metadata before using the evaluator for internal readiness planning.

Configuration

Initialize local configuration:

sciveo init

The default local configuration lives under ~/.sciveo/. API credentials can also be supplied through environment variables where supported, for example:

export SCIVEO_SECRET_ACCESS_KEY="..."

Monitoring

Start host monitoring:

sciveo monitor --period 60

Monitor the root filesystem plus additional mounted paths or block devices. --paths accepts mount directories and always includes /. --disks accepts filesystem-bearing block devices or partitions and resolves each one to its current mountpoint:

sciveo monitor --period 60 \
  --paths '["/mnt/data-1"]' \
  --disks '["/dev/sdb1","/dev/sdc1"]'

Root capacity remains under DISK; requested mount paths are reported under PATHS, and requested block devices are reported under DISKS with their resolved device, mountpoint, filesystem, capacity, and mount state. Sciveo does not implicitly expand a whole disk such as /dev/sdb to /dev/sdb1; name the filesystem-bearing partition explicitly. Stable /dev/disk/by-uuid/... aliases are preferred when device names may change across boots.

Install monitoring as a service:

sciveo monitor --period 60 \
  --paths '["/mnt/data-1"]' \
  --disks '["/dev/sdb1"]' \
  --install

Run --install as the account that should run the monitor. Sciveo asks for sudo only while copying the systemd units and running systemctl; the installed service keeps the invoking account as its runtime user.

Every --install rewrites both sciveo-monitor.service and sciveo-monitor.timer, reloads systemd, and restarts the service so updated monitor arguments take effect immediately. The timer starts the continuous service once after each boot; the service itself continues sampling at --period intervals.

Inspect the installed units and follow monitor logs:

systemctl status sciveo-monitor.timer sciveo-monitor.service
systemctl list-timers --all sciveo-monitor.timer
journalctl -u sciveo-monitor.service -f

Write samples to a local path:

sciveo monitor --period 120 --output-path ./monitor.json

Start non-blocking monitoring from Python:

import sciveo

sciveo.monitor(period=120, block=False)

Plant/industrial monitoring uses the plant source:

sciveo monitor --src plant --host 192.168.1.50 --port 502 --period 60 --serial plant-1

Watchdogs

Watchdogs run local checks and either execute a configured command or use the built-in internet repair path when a condition remains unhealthy.

sciveo watchdog --src memory --threshold 90 --period 10 --execute "systemctl restart app"
sciveo watchdog --src disk --input-path /data --threshold 80 --period 600 --execute "find /data/tmp -type f -mtime +1 -delete"
sciveo watchdog --src network --targets '["1.1.1.1:443","8.8.8.8:53"]' --threshold 3 --period 30 --execute "echo network outage"

The network/internet watchdog performs concurrent deadline-bounded TCP checks, records resolution/bind/connect results, and applies at-least-one, all, majority, or count:N quorum. With --execute, it runs the configured shell command after --threshold consecutive failures and repeats the command on every failed period until a healthy check resets the counter. scan --health adds read-only gateway/DNS diagnostics, while monitor HTTP probes record endpoint health without repairs.

Use built-in internet recovery without running the whole watchdog as root:

sciveo watchdog --net --repair --period 30 --threshold 3

On Linux systemd machines, install the same configuration as a supervised boot service. Run this command from the non-root account that should run the watchdog; Sciveo asks for sudo only while installing root-owned files and using systemctl:

sciveo watchdog --net --repair --period 30 --threshold 3 --install
sciveo watchdog --net --repair --preflight
systemctl status sciveo-network-watchdog.service
journalctl -u sciveo-network-watchdog.service -f

--install is idempotent: every invocation stops and replaces the managed service and its exact configuration. It preserves the network repair targets, quorum, timing, platform, interface or gateway list, and log-level options in the unit's ExecStart. The service runs as the invoking non-root user with Restart=always; it does not run the watchdog process as root. Every service start first runs a read-only preflight that validates the unattended helper, exact service-unit allowlist, detected platform, selected interface, network manager, and that manager's required core helper actions. Remove all managed watchdog files and persistent incident state with:

sciveo watchdog --uninstall

Installation places a self-contained root-owned helper at /usr/local/libexec/sciveo-network-repair and grants the service user passwordless sudo access only to that exact executable. The helper accepts a fixed action enum, validates physical interface names and canonical NetworkManager UUIDs, uses fixed absolute system-program paths, and rejects arbitrary commands. Privileged actions have a fixed 30-second execution deadline. The optional service and gateway allowlists are accepted only as regular, root-owned files that are neither group- nor world-writable, and their exact contents are checked by preflight. The helper's DNS configuration repair refuses to overwrite a regular /etc/resolv.conf or a working symlink. Optional VPN/API service restarts are limited to canonical .service units in that allowlist.

Bare --repair also implies the network source, but spelling the mode as --net --repair is preferred because it makes the watchdog purpose explicit.

Repair mode defaults to 1.1.1.1:443, softel.bg:443, and sciveo.com:443 with majority policy. Additional VPN or API endpoints supplied through --targets are added to those defaults and deduplicated by host and port. Use --policy all when every configured endpoint must participate in the health decision.

For Linux edge machines with redundant uplinks, --gateways selects Internet route failover instead of single-interface repair:

sciveo watchdog --net --repair --install \
  --gateways '[
    {"interface":"eth0","priority":10},
    {"interface":"wwan0","priority":20,"probe":"on-select"}
  ]'

Lower priority numbers are preferred. The preferred gateway always uses continuous probing. The active gateway is checked every period; when a backup is active, the preferred gateway is also checked and restored immediately when its target quorum succeeds. Other backups default to on-select: they are not polled during normal operation and are probed only once when considered during an outage. A failed on-select route is not probed again during that same incident; a healthy active check, an external route change, or a successful switch starts a new incident. Set an unmetered backup to "probe":"continuous" only when checking it every period is acceptable. Configuration identifies each route only by its physical interface; Sciveo discovers its current IPv4 gateway from that interface's main-table default route whenever it probes or switches, so DHCP and mobile network gateway changes do not require reinstalling the service.

Gateway mode is deliberately narrow and managed-service-only: configure it with --install. It supports IPv4 on Linux, at least two unique physical interfaces with discoverable gateways, and at least one numeric IPv4 target. It requires the at-least-one target policy and limits timeout to 10 seconds. An explicit gateway-mode --targets list replaces the defaults; health succeeds when any configured TCP endpoint succeeds, and fails only when all endpoints fail. Hostnames and IPv4 literals are supported, but one numeric IPv4 endpoint is required so failover does not depend entirely on DNS. --interface cannot be combined with --gateways.

Standby TCP checks and route selection use the short-lived restricted helper, not extra capabilities on the long-running watchdog. The helper accepts only interface and host/port combinations from the exact root-owned configuration. It derives the gateway from the current ordinary interface route, with an exact Sciveo-owned standby as the temporary fallback if a manager removes that route. Before a standby TCP connection, it also proves that the exact target route resolves through that discovered gateway and interface. A switch adds a protocol-tagged metric-0 main-table override for the selected interface and then verifies the owned overlay. Sciveo also creates and restores only its protocol-tagged high-metric standby routes for configured interfaces, so manager removal of an ordinary default route does not prevent probing or failback. It never flushes or changes unrelated routes. Numeric endpoints are tried first and probing stops after the first success; hostname endpoints are then resolved to IPv4 and their TCP connection is still forced through the exact candidate route. If one candidate cannot be activated or fails its full active check, the same watchdog cycle continues with the next priority.

Before changing an interface, repair mode snapshots the owning manager, link/carrier, usable addresses, default route, gateway-neighbor state, and DNS, then separately probes numeric underlay anchors (1.0.0.1:443, 8.8.8.8:53, and [2606:4700:4700::1111]:443) through the selected physical interface/source. The at-least-one default keeps IPv4-only and IPv6-only links valid while providing dual-stack evidence when both are available. It repairs the lowest failed layer. If link, address, route, and gateway are healthy while the public underlay is unavailable, the incident is classified as upstream and local networking is left intact.

An isolated VPN/API endpoint is never mapped to an arbitrary process. To restart a known local service after proving that the Internet underlay is healthy, provide an exact target mapping at installation:

sciveo watchdog --net --repair \
  --targets "vpn.example.com:443" --policy all \
  --repair-services '{"vpn.example.com:443":["wg-quick@plant.service"]}' \
  --install

Only those installed units can pass the privilege helper. Sciveo watchdog, Sciveo Admin, SSH, hostapd/dnsmasq, and the network-manager/resolver control units are protected and cannot be mapped. A remote API outage with no explicit local mapping is reported and safely skipped.

--platform auto detects Linux or macOS; on Linux it separately detects per-interface ownership by NetworkManager, systemd-networkd, or dhcpcd. Explicit overrides include linux, ubuntu, debian, raspbian, raspberry-pi, rpi, macos, mac, and darwin, so Intel and Raspberry Pi Ubuntu machines use the same manager-aware Linux repair backend. At startup, repair mode logs the requested and detected platform at INFO level. On a multi-homed edge device, --interface eth0 is the explicit safe physical-uplink override; otherwise Sciveo selects the default-route physical interface and avoids tunnel devices.

The network --threshold, --recovery-checks, and --max-disruptive-repairs must be positive integers. --period, --timeout, and --repair-cooldown must be finite positive values, and the initial repair cooldown cannot exceed the 900-second backoff cap. --underlay-targets accepts numeric IP targets only; --underlay-policy controls their quorum.

Built-in repair uses structured commands, never flushes routes, never reboots, and never guesses which arbitrary VPN/API services to restart. Gateway mode may select only its explicit root-allowlisted routes. The managed service sends privileged actions only through its installed strict helper; operators do not need to grant direct sudo rights to nmcli, networkctl, dhcpcd, systemctl, or a shell. Its incident layer, cooldown, and disruptive circuit survive process restarts; stable healthy checks clear them. Watchdog repairs and Sciveo Admin network changes share one mutation lock. Admin opens a bounded maintenance lease before its first network mutation and keeps repairs inhibited until confirmation or lease expiry, including after a partially failed Admin apply.

This recovery path deliberately does not guess new static addressing, unknown Wi-Fi/VPN credentials, router configuration, or arbitrary services. Physical NIC/cable/router failures and provider outages require a redundant uplink or out-of-band management; a local watchdog also cannot prove inbound Internet reachability without an external heartbeat.

Diagnostics and Fleet

Local diagnostic report:

sciveo doctor
sciveo doctor --render json
sciveo doctor --logs --output-path /tmp/sciveo-doctor.json

Fleet inventory over SSH:

sciveo fleet --host operator@edge.local -i ~/.ssh/id_ed25519
sciveo fleet --net 192.168.10.0/24 --users operator,admin -i ~/.ssh/id_ed25519 --render json

JSON Data to HTML/CSS

sciveo.ui.JsonHtmlView creates an HTML/CSS string from JSON data. It is independent of terminal rendering, so the same output can be embedded in a web page, saved as HTML, or passed to sciveo.ui.term.HtmlView.

from sciveo.ui import JsonHtmlView, TableColumn
from sciveo.ui.term import HtmlView

workers = [
  {"host": "gpu-01", "state": "ready", "fps": 104.6},
  {"host": "gpu-02", "state": "busy", "fps": 89.2},
]
view = JsonHtmlView(
  title="Workers",
  columns=["host", "state", TableColumn("fps", "FPS", formatter="{:.1f}".format)],
  css="""
.sciveo-json-title { color: navy; font-weight: bold; }
.sciveo-json-table, .sciveo-json-cell { border-color: gray; }
""",
)
html = view.render(workers)
HtmlView(width=80).print(html)

render(data) accepts decoded values. A nonempty object is one table row; arrays of objects infer the union of keys in first-seen order. Sparse fields show -, while explicit nulls show null; nested cell values use compact JSON. Columns can select/reorder fields, rename headers, format values, and configure alignment. Numeric columns align right when all present non-null values are numeric; booleans remain left aligned. Indexed sequence rows require explicit indexed columns to become a table. Scalars, mixed/nested arrays, and empty JSON values use a readable preformatted JSON representation.

render_json(text) parses serialized JSON before rendering. Strings passed to render() are literal JSON string values, not HTML or automatically parsed JSON. All data, titles, column names, and formatter results are escaped as text.

The result is an HTML fragment with an embedded default stylesheet. css appends ordinary CSS overrides after the defaults. Use include_css=False when the page already supplies the stylesheet; stylesheet() returns the CSS separately. The generated classes are sciveo-json, sciveo-json-title, sciveo-json-table, sciveo-json-cell, sciveo-json-value, sciveo-json-empty, and the alignment classes sciveo-json-left, sciveo-json-right, and sciveo-json-center. Default text is black and tables have a complete cell grid. Simple selectors and supported terminal CSS properties can style both HTML and terminal output; per-cell colors and browser-only layout properties remain subject to the HtmlView subset described below.

The shared sciveo.ui.tables.TableData and TableColumn also serve terminal TableView. Existing sciveo.ui.term.TableColumn imports remain compatible. HTML generation uses column keys, titles, alignment, and formatters; max_width/overflow apply to direct terminal TableView rendering, while HtmlView(width=...) controls the terminal width of generated HTML. Importing these generic UI components needs only the standard library and does not load terminal components or optional dependencies.

Terminal UI Components

sciveo.ui.term provides reusable terminal components for Sciveo and third-party Python applications. Tables, menus, progress, status, panels, trees, summaries, rules, document and HTML/CSS formatting, and terminal helpers share the same stream and text handling. The core needs only the standard library. ImageView loads Pillow only when rendering an image; document code highlighting optionally uses Pygments and falls back to plain code when it is unavailable.

Tables and selection menus

from sciveo.ui.term import SelectMenu, TableColumn, TableView

workers = [
  {"host": "worker-1", "state": "ready", "latency_ms": 12.4},
  {"host": "worker-2", "state": "busy", "latency_ms": 28.7},
]
table = TableView(title="Workers", columns=[
  "host",
  "state",
  TableColumn("latency_ms", title="Latency (ms)", formatter="{:.1f}".format),
])
table.print(workers)
text = table.render(workers)  # Return a string without writing output.

menu = SelectMenu("Select a worker", label=lambda worker: worker["host"])
selected = menu.select(workers)  # Returns the original worker dictionary.

TableView accepts JSON text, one dictionary as one row, or a list of dictionaries. Omitted columns use the union of keys in first-seen order. Nested objects and arrays use compact JSON; missing fields show -, while explicit None shows null. Sequence rows require indexed columns, for example TableView(columns=[TableColumn(0, "Host"), TableColumn(1, "State")]).

Configure title, total width, and style="auto"|"unicode"|"ascii"|"plain" on the view. Columns accept align="auto"|"left"|"right"|"center", max_width, a callable formatter, and overflow="wrap"|"ellipsis"; automatic alignment places numbers on the right. Width defaults to the destination terminal width on a TTY and is unlimited for redirected tables. A width too narrow to fit the columns raises ValueError.

Framed tables show a complete cell grid by default. row_separators=True draws horizontal lines between records, keeping wrapped lines inside the same cell; column_separators=True draws vertical lines between columns. Disable either independently. The outer frame and header divider remain visible. style="plain" keeps its compact, unframed layout and ignores these two frame options.

Table text, headers, titles, and borders default to normal black. text_color sets the foreground; header_color and border_color inherit it unless supplied. Use TerminalColors constants or numeric SGR parameters such as "34" (blue). text_color=None uses the terminal's existing foreground. The existing color option still controls whether ANSI styling is automatic, forced, or disabled; redirected output and NO_COLOR remain plain by default.

from sciveo.ui.term import TableView, TerminalColors

TableView().print(workers)  # Full grid; normal black styling on a capable terminal.
TableView(
  row_separators=False,
  column_separators=True,
  text_color=TerminalColors.BLUE,
  header_color=TerminalColors.BOLD + TerminalColors.BLUE,
  border_color=TerminalColors.GRAY,
).print(workers)

SelectMenu uses POSIX arrow keys or j/k, and accepts a number followed by Enter. Other environments use a numbered line prompt. default is a zero-based index accepted by Enter; labels can come from a callable or an item's label field. The menu defaults to stderr and input to stdin, with explicit stream and input_stream overrides. Cancellation raises KeyboardInterrupt; exhausted input raises EOFError. Terminal settings and the cursor are restored on exit.

Progress and status

import sys
from sciveo.ui.term import LiveDisplay, ProgressBar, StatusIndicator

with ProgressBar(total=len(workers), label="Workers", unit="hosts") as progress:
  for worker in workers:
    progress.update(label=worker["host"], refresh=False)
    progress.advance()

with StatusIndicator("Preparing report") as status:
  status.update("Formatting rows")

StatusIndicator("One worker is unavailable", state="warning").print()

bar = ProgressBar(total=10, label="Files", unit="files")
bar.update(completed=3, refresh=False)
snapshot = bar.render(width=80)  # No output; useful for composing a larger view.

with LiveDisplay(stream=sys.stderr) as display:
  display.update("Workers: 2\nReady: 1", force=True)

ProgressBar reports completed/total counts, throughput, elapsed time, and ETA. total=None means unknown work; zero means empty work. Use update(completed=...) for an absolute count or advance(amount) for an increment. finish() preserves the actual count, including partial work. StatusIndicator supports running, info, success, warning, and error states; tick() advances its activity frame. Its context manager finishes with success on normal exit and error when an exception propagates.

These components do not start background workers. Updates come from the caller; use a context manager or explicit start()/finish() for a live lifecycle. render() returns a snapshot, while update() and advance() write unless refresh=False is supplied. StatusIndicator.print() writes a single status line. ProgressBar has no print() method; use print(bar.render()) for a plain snapshot. All live components default to stderr, leaving stdout available for application results.

LiveDisplay owns a redraw region and restores the cursor on completion or failure. Its default refresh limit is min_interval=0.1 seconds on a capable terminal. Redirected or non-cursor streams receive plain newline snapshots at log_interval=5.0 seconds, without cursor controls. force=True bypasses the interval; final output is always forced. Configure intervals and stream on ProgressBar, StatusIndicator, or LiveDisplay as needed.

Panels, trees, summaries, and rules

from sciveo.ui.term import KeyValueView, PanelView, RuleView, TreeNode, TreeView

PanelView(title="Workers", width=60).print("Two workers configured.\nOne is ready.")
PanelView(title="Configuration").print({"parallel": 8, "enabled": True})

tree = TreeNode("Cluster", [
  TreeNode("worker-1", [TreeNode("ready"), TreeNode("12.4 ms")]),
  TreeNode("worker-2", [TreeNode("busy")]),
])
TreeView(width=60).print(tree)
TreeView(title="Settings").print({"network": {"parallel": 8}, "ports": [22, 443]})

RuleView(title="Summary", width=60).print()
KeyValueView().print({"Ready": 1, "Busy": 1})
KeyValueView(inline=True).print({"Ready": 1, "Busy": 1})

PanelView renders text or formatted JSON with a title, configurable horizontal padding, and wrapping. Its width includes padding and borders. TreeView accepts a TreeNode, an ordered list of nodes, or decoded JSON values. Strings are literal labels; use json.loads() to convert serialized JSON first. Trees mark path cycles and stop descendants at max_depth (default 32), while shared nodes in independent branches remain visible. Panel and tree widths use the destination terminal on a TTY and natural width when redirected. Impossible widths raise ValueError instead of overflowing the configured layout.

KeyValueView takes mappings or ordered key/value pairs and supports aligned labels or an inline summary. separator, gap, and width configure its layout. RuleView prints a titled or untitled separator. Panels and trees support auto, unicode, ascii, and plain styles; rules support the first three.

Containers, positioned pages, and dialogs

PanelContainer stacks text and other static renderers inside a bordered area. Unlike the text/JSON PanelView, it preserves each child's colors and layout. Bind a component's render arguments with add(view, *args, **kwargs); ordinary strings are treated as plain text. Components receive the inner viewport size through their destination stream. Nested containers work the same way.

from sciveo.ui.term import ButtonBar, PanelContainer, TableView, TerminalPage

workers = [{"host": "gpu-01", "state": "ready"}, {"host": "gpu-02", "state": "busy"}]
panel = PanelContainer(title="Workers", width=44, height=15)
panel.add(TableView(), workers)
panel.add(ButtonBar(), ["Refresh", "Close"], selected=0)

page = TerminalPage(width=80, height=24)
page.add(panel, x=4, y=2, width=44, height=15)
page.print()  # A complete, printable page; render() returns the same content.
# page.draw() paints at the terminal origin, preserving the caller's cursor.

TerminalPage uses zero-based cell coordinates. Set x=None and/or y=None to center a component on that axis. Per-child width/height limit its area; later children paint over earlier ones. A page is bounded by its configured size and the destination viewport. draw(stream=..., x=..., y=...) paints that page at a terminal position. It preserves cells outside the page rectangle and the caller's cursor, and reserves the terminal's final column to prevent wrapping. Redirected drawing produces plain text without cursor controls.

Container width and height are hard limits including borders and horizontal padding. gap controls blank lines between children. Omitted height fits the content up to the available rows. Overflow scrolls inside the frame: the title and borders stay fixed, with a scrollbar and visible row/column ranges when needed. Set scrollbar=False to hide these indicators. Impossible layouts raise ValueError. Partial wide characters are blanked at clipping boundaries.

from sciveo.ui.term import PanelContainer, TableView

records = [{"id": index, "description": "A long description that extends beyond the visible panel width"} for index in range(100)]
panel = PanelContainer(title="Records", width=48, height=12, content_width=80)
panel.add(TableView(width=80), records)
panel.print()
panel.scroll(dy=8, dx=12).print()  # Move the content; keep the 48 × 12 frame.
# panel.show() opens an interactive panel: arrows scroll, Escape closes.

content_width specifies a wider virtual area for children when horizontal scrolling is useful. Otherwise text and width-aware renderers wrap to the inner width; any wider rendered output is still horizontally scrollable. scroll(dx, dy) uses relative cell offsets; scroll_to(x=..., y=...) uses absolute offsets. Offsets clamp to the available content, including after resizing or content changes. After rendering, scroll_limits reports maximum (x, y) offsets and content_area describes the visible body rectangle.

panel.show(x=..., y=..., area=TerminalArea(...)) provides keyboard scrolling through PanelDialog: arrows or h/j/k/l scroll one cell, PageUp/PageDown or Space scroll vertically by a page, Home returns to the origin, and End goes to the last row. Enter, Escape or q close; Ctrl-C and EOF propagate. The panel retains its offsets and is returned to the caller. Noninteractive output prints one bounded snapshot without waiting for input. Calling show() operates the container's scroll position, not the actions of static child buttons or menus.

Styles are auto, unicode, ascii, or plain, with the usual color option. Raw strings discard terminal controls; rendered children retain SGR styles only. Custom child components implement render(..., stream=None). Live components can contribute snapshots with panel.add(progress.render(width=40)); containers do not run child input loops or background workers.

ButtonBar and MenuView are reusable static widgets. Their selected argument is a zero-based index. ButtonBar supports left/center/right alignment and a configurable gap. MenuView keeps the selected item visible in a bounded list. Use the dialog classes for keyboard interaction:

from sciveo.ui.term import ConfirmDialog, SelectionDialog, TerminalArea

confirmed = ConfirmDialog("Start processing these files?", title="Start job").ask()

network = SelectionDialog(
  title="Select a network", width=48, height=10,
  area=TerminalArea(x=4, y=2, width=60, height=16),
).select(["Ethernet", "Wi-Fi", "VPN"], clear_on_exit=True)

Dialogs center within the terminal or the supplied TerminalArea; their own x/y options give offsets inside that area. render() produces a static preview, suitable for adding to a page. SelectionDialog.render(choices) accepts the same records and label callback as SelectMenu, and select() returns the original item. Use arrows/j/k or a number followed by Enter. Escape/q cancel selection with KeyboardInterrupt; EOF raises EOFError.

ConfirmDialog.ask() returns True for OK and False for Cancel or Escape/q. Cancel is the default; set default=True to change it. Customize ok_label and cancel_label as needed. Tab, Shift-Tab and left/right arrows switch buttons; Enter or Space selects. Ctrl-C and EOF propagate. Terminal input settings and cursor visibility are restored on completion, cancellation and errors. Long confirmation messages and selection subtitles can be read with PageUp and PageDown while the choice controls stay fixed. A message range indicator appears when space permits; an area too small for both message and controls is rejected.

Interactive dialogs redraw within their rectangle; size changes are handled on the next key event. clear_on_exit=True erases that rectangle, without recovering the old contents underneath it. A caller managing a page can call page.draw() after closing a dialog. Confirmation enables this cleanup by default; selection uses the existing SelectMenu default of leaving the final panel visible. Noninteractive streams and unsupported input terminals use the existing numbered prompt, showing every choice without cursor positioning.

Documents and image previews

from sciveo.ui.term import DocumentView, ImageView

document = "# Result\n\n**Ready** for inspection.\n\n```json\n{\"workers\": 2}\n```"
DocumentView().print(document)

# Accepts encoded bytes, a local path, a readable file, or a Pillow image.
ImageView(max_width=48, max_height=20).print("sample.jpg", name="Sample")

DocumentView formats Markdown headings, emphasis, lists, quotes, rules, and fenced code. It preserves code line structure and does not reflow documents. It does not require an agent or a model provider. ImageView produces RGB cells when color is enabled and an ASCII luminance preview otherwise. Image loading is local; it does not fetch URLs. width, max_width, and max_height bound the preview grid. Pillow is needed only for image rendering, and no new UI extra is required for the other components.

Simple HTML/CSS rendering

HtmlView renders HTML text to the terminal using the same table, panel, rule, style, and text components. It uses only the standard library.

from sciveo.ui.term import HtmlView

HtmlView(width=72).print("""
<style>
  h1 { color: navy; }
  .summary { border: 1px solid gray; padding: 1; }
  .number { text-align: right; }
</style>
<h1>Workers</h1>
<div class="summary"><strong>Ready</strong> for processing.</div>
<table>
  <tr><th>Host</th><th>State</th><th class="number">FPS</th></tr>
  <tr><td>gpu-01</td><td>ready</td><td>104</td></tr>
  <tr><td>gpu-02</td><td>busy</td><td>89</td></tr>
</table>
""")

Use render(html, stream=...) to return a string or print(html, stream=...) to write it. width defaults to the destination terminal width, or 80 columns when redirected. style accepts auto, unicode, ascii, and plain. color follows the other components; text_color defaults to black and accepts TerminalColors constants or numeric SGR parameters.

Supported markup includes headings, paragraphs, divisions, line breaks, rules, inline emphasis, preformatted text/code, ordered/unordered lists, blockquotes, and tables. HTML entities are decoded and unknown elements retain readable children. Image elements show their alternative text without loading images. Head, script, style, and template contents are hidden, with CSS collected from style elements.

CSS supports simple tag, .class, #id, and * selectors, comma groups, and inline style attributes. Supported properties include inherited color, font-weight, font-style, text-decoration, and text-align; display:none; block margin, padding, and their side-specific variants; and simple borders. Spacing uses nonnegative integers or ch/px, interpreted as terminal cells (horizontal) or rows (vertical), capped at 64 per side. CSS colors accept common names and #rgb/#rrggbb. Table colors apply to the whole table, and the first cell in each column supplies its alignment.

This is a small terminal layout subset: per-cell styling, rowspan/colspan, CSS combinators, !important, background colors, sizing, and positioning are unsupported. It does not execute JavaScript or load external resources.

Stream helpers and portability

import sys
from sciveo.ui.term import ColorPalette, TerminalColors, TerminalScreen, TerminalStyle
from sciveo.ui.term.terminal import TerminalText

style = TerminalStyle(stream=sys.stdout)
palette = ColorPalette(colors=[TerminalColors.CYAN, TerminalColors.GREEN, TerminalColors.MAGENTA])
print(style.paint("worker-1", palette.color("worker-1")))
print(style.paint("Ready", TerminalColors.BOLD + TerminalColors.GREEN))
safe_label = TerminalText.clip(style.text("A long label"), 10)

screen = TerminalScreen(stream=sys.stdout)
screen.clear_line()  # No cursor sequence is emitted to a redirected stream.

Static views expose render(data, stream=...) without writes and print(data, stream=...) with stdout resolved at call time; RuleView takes an optional title instead of data. Pass the intended destination to render() so layout and color match where the returned text will be written. TerminalStyle provides safe text and SGR styling, TerminalCapabilities detects stream support and dimensions, and TerminalText measures, clips, pads, and wraps text. ColorPalette.color(value) selects a stable color for a label; pass its result or a TerminalColors constant to TerminalStyle.paint().

color=None detects terminal support and respects NO_COLOR; color=False disables color, and color=True explicitly forces styling. Live redirected snapshots remain plain. Automatic decorations use Unicode on a capable terminal and ASCII otherwise; a known incompatible destination encoding also falls back to ASCII for explicit Unicode styles. Input terminal controls and embedded escape sequences are removed before styling. Layout accounts for combining marks and East Asian wide characters. TerminalScreen provides clear(), clear_line(), and clear_previous_lines(count); clearing a non-cursor stream writes a newline, while line-clearing operations are no-ops.

StreamRegion tracks text written incrementally by a caller. Call start(), then track(text) after each write; erase() clears the tracked region only when cursor position, terminal size, and visible rows still make that safe. It does not write or sanitize the input itself; use TerminalStyle.text() before writing untrusted text. This supports replacing streamed output with a final formatted view without coupling the application to an agent renderer.

The existing module renderers now compose the generic components:

Domain adapters still own their report fields, severity meanings, sorting, message content, and callbacks. The extraction centralizes reusable display behavior; it does not replace every domain output string. Existing JSON report contracts remain unchanged. Compatibility imports such as sciveo.network.ssh.TerminalCapabilities, sciveo.chat.ui.ChatColors, and the document renderer names in sciveo.agents.render remain available.

Network and Industrial IO

Network scans:

sciveo scan
sciveo scan --net 192.168.0.0/24 --port 22 --timeout 0.5
sciveo scan --health
sciveo scan --health --l2
sciveo scan --host 192.168.0.10 --health --ports '[22,80,443,502,554,161]'

SSH scan and command execution:

sciveo ssh --net 192.168.0.0/24 --users operator,admin -i ~/.ssh/id_ed25519 --execute '["hostname","uptime"]'
sciveo ssh --net 192.168.0.0/24 --execute hostname --sort execute --order ASC --render online

Modbus reads and writes:

sciveo read --proto modbus --transport tcp --host 192.168.0.10 --port 502 --id 1 --address 30001 --kind input --type RAW --count 2
sciveo write --proto modbus --transport tcp --host 192.168.0.10 --id 1 --reg '[40010,"U16",1,1]' --value 123
sciveo read --proto modbus --action scan --net 192.168.0.0/24 --render text

Modbus data representations:

Unsuffixed multi-register types use big-endian word order. This preserves the existing behavior; use an explicit _LE suffix for little-endian word order. Modbus fixes the byte order inside each 16-bit register, so the endian suffixes on U16 and S16 are accepted for consistency but produce the same value. Compatibility aliases are also accepted: I16/INT16, I32/INT32, I64/INT64, UINT8, UINT16, UINT32, UINT64, F32, F32_BE, F32_LE, and the older FLOAT, FLOAT_BE, and FLOAT_LE names.

For a multi-value write, pass a JSON value list. A single non-RAW type is repeated for every value, or --type may be a JSON list with one type per value. Each U8 is zero-extended into its own register by default. With --pack-u8, adjacent U8 values pack into the high and low bytes of one register; they must occur in aligned pairs so wider values always begin on a register boundary.

sciveo write --proto modbus --transport tcp --host 192.168.0.10 --id 247 --address 40313 --type U8 --value '[26,8,26,9,31,5]' --pack-u8 --function-code 10H --zero-based
sciveo write --proto modbus --transport tcp --host 192.168.0.10 --id 1 --address 40010 --type '["U8","U8","U16","FLOAT32_LE"]' --value '[1,2,300,12.5]' --function-code 10H

For example, read or write a 4-byte float using explicit word order:

sciveo read --proto modbus --transport tcp --host 192.168.0.10 --id 1 --address 30001 --kind input --type FLOAT32_LE
sciveo write --proto modbus --transport tcp --host 192.168.0.10 --id 1 --address 40010 --type FLOAT32_BE --value 12.5

Serial Modbus:

sciveo read --proto modbus --transport serial --serial-port /dev/ttyUSB0 --baudrate 9600 --bytesize 8 --parity N --stopbits 1 --id 1 --reg '[5039,"U16",0.1,1]'

SNMP, MQTT, and HTTP helpers:

sciveo read --proto snmp --host 192.168.0.1 --oid 1.3.6.1.2.1.1.1.0
sciveo read --proto snmp --host 192.168.0.1 --action walk --oid 1.3.6.1.2.1.1
sciveo write --proto mqtt --host broker.local --topic plant/cmd --payload '{"limit":80}' --qos 1 --retain
sciveo read --proto http --url http://192.168.0.10/status
sciveo write --proto http --url http://192.168.0.10/api/control --value '{"enabled":true}'

Protocol emulators:

sciveo emulate --server modbus --profile custom --host 0.0.0.0 --port 1502 --data-json '{"device_id":7,"holding":{"40010":123}}'
sciveo emulate --server snmp --host 0.0.0.0 --port 1161 --data-json '{"oids":{"1.3.6.1.2.1.1.1.0":"lab-agent"}}'
sciveo emulate --server mqtt --host 0.0.0.0 --port 1883 --data-json '{"retained":{"plant/power":1234}}'
sciveo emulate --server http --host 0.0.0.0 --port 8080 --data-json '{"status":{"ok":true}}'

Admin UI

The admin command is for first-boot and field administration of an edge machine:

sciveo admin
sciveo admin --web --wifi-ap --net 10.137.19.0/24 --ap-ssid sciveo-setup --ap-password CHANGE_ME --admin-auth none
sudo sciveo admin --install --web --wifi-ap --net 10.137.19.0/24 --ap-ssid sciveo-setup --ap-password CHANGE_ME --admin-auth none

The admin UI covers dashboard diagnostics, Ethernet and Wi-Fi setup, service state, pending configuration, and installed service management. Pending configuration is stored under ~/.sciveo/admin/.

VS Code Extension

The Python package can install the bundled VS Code extension asset:

sciveo extensions --install --vscode
sciveo extensions --reinstall --vscode
sciveo extensions --uninstall --vscode

If the editor CLI is not on PATH, set:

export SCIVEO_VSCODE_CLI="/path/to/code"

Extension config is stored under ~/.sciveo/extensions/vscode/.

Encrypted Chat Rooms

Start a room:

sciveo chat --serve ops-room --host 0.0.0.0 --port 8090 --max-clients 5

Join from another terminal:

sciveo chat --client 'sciveo-chat-v1....' --url ws://HOST:8090/ws/chat --name operator-a

Messages use encrypted AES-GCM envelopes, while connection setup uses a shared-token HMAC proof. Display names are decorative; anyone with the shared token is trusted as a room participant.

Persist and reload encrypted room history:

sciveo chat --serve ops-room --output-path ./chat-history.jsonl
sciveo chat --serve ops-room --input-path ./chat-history.jsonl --output-path ./chat-history.jsonl

Suppress decrypted message bodies in server logs:

sciveo chat --serve ops-room --silent

TLS:

sciveo chat --serve ops-room --tls-cert ./chat.crt --tls-key ./chat.key
sciveo chat --client 'sciveo-chat-v1....' --url wss://HOST:8090/ws/chat --name operator-a --tls-no-verify

Agent Console

Interactive agent console:

sciveo agent --provider auto
sciveo agent --profile coder --provider auto

One-shot prompt:

sciveo agent --provider auto --prompt "List the repo root and summarize the important files"

Profiles can be loaded with --profile NAME or --config PATH. Predefined profiles include coding, review, testing, research, and operations-oriented specializations.

Local runtime examples:

sciveo agent --action pull --model org/model-name --alias local-agent-model
sciveo agent --action run --model local-agent-model --host 127.0.0.1 --port 8910 --device mps --context 8192
sciveo agent --provider hf --prompt "Say hello in one sentence"

Agent orchestration uses the chat transport:

sciveo agent --action orchestrate --serve design-room --prompt "Review this module" --agents researcher,coder,reviewer,tester
sciveo agent --action orchestrate --serve design-room --prompt "Plan the next pass" --agent-write-policy discussion-only

Media Capture and Processing

Capture and stream helpers:

sciveo nvr --input-path cams.json
sciveo rtsp --url rtsp://camera/stream --output-path ./clip.mp4
sciveo capture --output-path ./screen.mp4 --fps 10

Run the queue/API-backed media worker:

sciveo media-server
sciveo media-run --mode worker

Run local/offline media processing:

sciveo media-run --input-path ./media --output-path ./media-out --render text
sciveo media-run --processor image-resize --input-path ./input.jpg --output-path ./image-out
sciveo media-run --config pipeline.json --input-path ./input.mp4 --output-path ./video-out

Run local ML media processing:

sciveo media-run --mode ml --processor image-to-text --input-path ./input.jpg --output-path ./ml-out

Local mode does not require API, queue, S3, or cloud credentials. Worker mode is for API/queue/storage-backed jobs.

SCIVEYOLO Object Detection

Sciveo provides a SCIVEYOLO inference runtime in sciveo.ml.images.sciveyolo.

Goals:

• load .sciveyolo artifacts without Ultralytics installed;
• use a Sciveo-owned PyTorch SCIVEYOLO graph for engine="pt";
• preserve a familiar SCIVEYOLO(...).predict(...) style API;
• support SCIVEYOLO model sizes n, s, m, l, and x through one generic scale-parameterized network;
• keep build-time conversion separate from production runtime.

Basic use:

from sciveo.ml.images.sciveyolo import SCIVEYOLO

model = SCIVEYOLO("model.sciveyolo")
results = model.predict("image.jpg", conf=0.25, iou=0.7)

Build a Sciveo artifact from a PyTorch checkpoint:

from sciveo.ml.images.sciveyolo import SCIVEYOLO

SCIVEYOLO.build("detector.pt", "detector.sciveyolo")

The default build engine is pt, which stores a state dict and metadata inside the .sciveyolo file. The artifact metadata includes engine, runtime, architecture, scale, model_size, variant, class names, image size, and source hash.

Auto-build sidecar behavior:

model = SCIVEYOLO("detector.pt")

When given a .pt path, Sciveo first looks for a same-name .sciveyolo sidecar next to it. If present, that artifact is loaded. If absent, Sciveo attempts to build it with engine="pt" and then loads the generated artifact.

Build an ONNX artifact only when explicitly wanted:

SCIVEYOLO.build("detector.pt", "detector-onnx.sciveyolo", engine="onnx")

Build a TensorRT-ready ONNX artifact when running on NVIDIA CUDA hosts:

SCIVEYOLO.build("detector.pt", "detector-trt.sciveyolo", engine="trt")

This creates an ONNX payload marked for TensorRT; it does not contain a compiled TensorRT engine. New TRT artifacts also embed a native PT fallback, including its verified state and architecture metadata. Loading an artifact never starts a TensorRT build by default. Sciveo tries every compatible cached TensorRT profile for the incoming tensor shape, then uses the configured fallback. The default is pt; onnx and error are explicit alternatives:

model = SCIVEYOLO("detector-trt.sciveyolo", device="cuda:0", trt_fallback="pt")
model = SCIVEYOLO("detector-trt.sciveyolo", device="cuda:0", trt_fallback="onnx")
model = SCIVEYOLO("detector-trt.sciveyolo", device="cuda:0", trt_fallback="error")

For an older TRT artifact without an embedded PT state, point to a prebuilt PT artifact. A raw checkpoint is intentionally rejected because fallback setup must not trigger conversion:

model = SCIVEYOLO(
    "detector-trt.sciveyolo",
    device="cuda:0",
    trt_fallback="pt",
    trt_fallback_model="detector-pt.sciveyolo",
)

The equivalent environment options are SCIVEO_SCIVEYOLO_TRT_FALLBACK and SCIVEO_SCIVEYOLO_TRT_FALLBACK_PATH.

TensorRT compilation must be explicitly allowed and should run outside the production workers. Build the observed batch/image profiles in a one-off process:

builder = SCIVEYOLO(
    "detector-trt.sciveyolo",
    device="cuda:0",
    engine="trt",
    allow_trt_build=True,
)
builder.compile_trt_profiles([
    {"batch": 16, "imgsz": [320, 640]},
    {"batch": 32, "imgsz": [320, 640]},
])

Each shape gets an independent context under profiles/, with a verified manifest recording the exact NCHW profile. A cross-process lock prevents concurrent builders from writing the same cache. Production processes omit allow_trt_build; engine="trt" by itself is always cache-only. They can load multiple validated contexts, choose one per batch, and apply the fallback for missing or rejected profiles. Partial, legacy engine-only, and invalid contexts are never rebuilt implicitly.

allow_trt_build=True grants permission to compile_trt_profiles() but does not make predict() compile. Ad-hoc build-on-prediction behavior requires the additional explicit trt_build_on_predict=True option (or SCIVEO_SCIVEYOLO_TRT_BUILD_ON_PREDICT=1) and is not intended for production workers.

Inspect runtime coverage and rank profiles worth compiling:

statistics = model.runtime_statistics()
print(statistics["trt"]["profile_hits"])
print(statistics["fallback"]["profile_candidates"])

suggested_shapes = [item["shape"] for item in statistics["fallback"]["profile_candidates"][:3]]
# Run this only in the separate builder process created above.
builder.compile_trt_profiles(suggested_shapes)

Statistics include TRT hit batches/images, runtime rejections, fallback batches/images and rates, reasons, per-shape counts, and loaded profile paths. Pass reset=True to return the current snapshot and reset the counters.

Build-time conversion may use a separate environment with export tooling. The production runtime only needs the dependencies required by the artifact engine, for example PyTorch for engine="pt", ONNX Runtime for engine="onnx", or ONNX Runtime GPU plus TensorRT for engine="trt".

Install the runtime extra that matches the deployment target:

pip install "sciveo[sciveyolo]"
pip install "sciveo[sciveyolo-gpu]"
pip install "sciveo[sciveyolo-trt]"

Use a fresh pyenv for the GPU/TRT extras when possible, because onnxruntime and onnxruntime-gpu provide the same onnxruntime Python module. The TRT extra is intended for CUDA/NVIDIA machines with compatible drivers and TensorRT libraries.

Runtime options:

model = SCIVEYOLO("model.sciveyolo", device="cpu", nms_method="numpy")
model = SCIVEYOLO("model.sciveyolo", fuse=True)
model = SCIVEYOLO("model.sciveyolo", channels_last=True)
model = SCIVEYOLO("model.sciveyolo", device="cuda:0", engine="auto", auto_engine_source="sample-video.mp4")

Native fine-tuning uses UT-style object-detection dataset YAML files:

model = SCIVEYOLO("coco-detector.sciveyolo")
result = model.train(
    data="/datasets/coco128/data.yaml",
    project="/models/object-detection",
    name="coco128-sciveyolo-m",
    epochs=8,
    batch=24,
    augment=False,
    optimizer="SGD",
    lr0=0.005,
    lrf=0.1,
    amp=False,
)

The trainer writes weights/last.pt, weights/best.pt, weights/last.sciveyolo, weights/best.sciveyolo, args.yaml, and results.csv under project/name.

Evaluate an artifact on a dataset split:

model = SCIVEYOLO("coco-detector.sciveyolo")
val_metrics = model.evaluate(data="/datasets/coco128/data.yaml", split="val")
test_metrics = model.test(data="/datasets/coco128/data.yaml")

evaluate() reports both native validation loss for PyTorch artifacts and Sciveo object-detection AP/FP metrics. test() is the same evaluator with split="test" by default.

engine="auto" looks for compatible .sciveyolo artifacts near the requested model, checks what the selected device/runtime can actually load, runs a small probe when multiple candidates are available, and then sets model.engine to the selected engine.

Pass auto_engine_source as an image, video, frame array, or list of sources to make the probe representative of the deployment workload. Without a source, Sciveo logs that it is falling back to a synthetic frame.

For CPU deployment, ONNX Runtime can be faster than the PyTorch engine on some machines. For GPU deployment, engine="pt" is the most portable path, while ONNX/TensorRT performance depends on the installed runtime and GPU generation.

Predictors API Server

Start the predictors API service:

sciveo predictors-server --port 8080

This command starts the configured Sciveo API predictor server and keeps it running as a long-lived process.

PredictorRemoteClient supports explicit image transport formats:

Existing JSON requests without image_format retain their predictor-specific input handling, including base64 image embeddings and text embeddings. The image helper defaults to jpeg-base64; select jpeg-binary to avoid base64 encoding:

from sciveo.tools.remote import PredictorRemoteClient

client = PredictorRemoteClient(url="http://localhost:8080")
embedding = client.predict_image_embedding(
    local_path="frame.jpg", resize_to=None, image_format="jpeg-binary",
)

# Already encoded JPEG files can be sent without decoding or re-encoding them.
from pathlib import Path

result = client.predict({
    "predictor": "ImageEmbedding",
    "image_format": "jpeg-binary",
    "compressed": 1,
    "X": [Path("frame.jpg").read_bytes(), Path("next.jpg").read_bytes()],
})

For direct HTTP clients, POST to /{api_prefix}/predict with the existing Bearer header. Binary requests use a params form field containing JSON metadata (predictor, image_format, and optional compressed), plus one X file part per JPEG with media type image/jpeg. Keep X out of the JSON metadata. File-part order determines prediction order. Responses retain predictor results, optional compression, and stats.count / stats.elapsed.

Explicitly formatted inputs are validated as JPEG and decoded to PIL images before the predictor runs. Invalid images, unsupported formats, and mismatched transports return HTTP 400. NumPy/OpenCV arrays can still be passed to the client image helper for JPEG encoding; direct array and PIL wire formats are not yet supported. Binary transport requires the updated server and client source.

Local S3-Compatible Storage

Start a local S3-compatible service:

sciveo storage --s3 --paths '["/mnt/d1","/mnt/d2"]' --port 9000 --threads 32 --health-check-interval 10800

A JSON config can also be supplied:

{
  "paths": [
    {"path": "/mnt/d1", "require_mount": true},
    {"path": "/mnt/d2", "require_mount": true}
  ],
  "credentials": {
    "access_key": "sciveo",
    "secret_key_file": "/etc/sciveo/storage-s3.secret"
  },
  "region": "us-east-1",
  "storage_name": "storage",
  "db_backend": "sqlite",
  "db_path": "~/.sciveo/storage/storage-s3.sqlite3",
  "lock_path": "~/.sciveo/storage/locks",
  "threads": 32,
  "workers": 4,
  "http": {
    "server": "gunicorn",
    "workers": 1,
    "timeout": 3600,
    "keepalive": 65
  },
  "path_health_check_interval": 10800,
  "capacity_check_interval": 5,
  "min_free_ratio": 0.05,
  "min_free_bytes": 1073741824,
  "resume_free_ratio": 0.08,
  "target_free_ratio": 0.15,
  "limits": {
    "max_keys": 1000,
    "max_request_bytes": 10737418240,
    "buckets": {"*": {"max_object_size": 10737418240}}
  },
  "retention": {
    "buckets": {"archive": {"minimum_age": "24h"}}
  }
}

IAM-Style Access Policies

Sciveo can authenticate several S3 users and authorize each request with Sciveo IAM policies using an AWS-compatible JSON structure. iam is a top-level, storage-independent section; the same policy evaluator can use other action and resource names outside S3. S3 only supplies the s3:* action names and S3 ARN mapping.

{
  "iam": {
    "users": [
      {
        "name": "storage-admin",
        "credentials": {
          "access_key": "storage-admin-key",
          "secret_key_file": "/etc/sciveo/iam/storage-admin.secret"
        },
        "policies": [
          {
            "Version": "2012-10-17",
            "Statement": {
              "Sid": "StorageAdmin",
              "Effect": "Allow",
              "Action": "*",
              "Resource": "*"
            }
          }
        ]
      },
      {
        "name": "ingest-write-only",
        "credentials": {
          "access_key": "ingest-key",
          "secret_key_file": "/etc/sciveo/iam/ingest.secret"
        },
        "policies": [
          {
            "Version": "2012-10-17",
            "Statement": {
              "Sid": "WriteIncomingObjects",
              "Effect": "Allow",
              "Action": "s3:PutObject",
              "Resource": "arn:sciveo:s3:::media/incoming/*"
            }
          }
        ]
      },
      {
        "name": "processor",
        "credentials": {
          "access_key": "processor-key",
          "secret_key_file": "/etc/sciveo/iam/processor.secret"
        },
        "policies": [
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Sid": "ReadWriteWorkObjects",
                "Effect": "Allow",
                "Action": [
                  "s3:GetObject",
                  "s3:PutObject",
                  "s3:DeleteObject"
                ],
                "Resource": "arn:sciveo:s3:::media/work/*"
              },
              {
                "Sid": "ListWorkPrefix",
                "Effect": "Allow",
                "Action": "s3:ListBucket",
                "Resource": "arn:sciveo:s3:::media",
                "Condition": {
                  "StringLike": {
                    "s3:prefix": ["work", "work/*"]
                  }
                }
              }
            ]
          }
        ]
      }
    ]
  },
  "s3": {
    "host": "127.0.0.1",
    "port": 9000,
    "region": "us-east-1",
    "paths": ["/mnt/d1", "/mnt/d2"],
    "db_path": "~/.sciveo/storage/storage-s3.sqlite3"
  }
}

Policy evaluation is deliberately small and predictable:

• Requests are denied by default; an explicit Deny overrides every Allow.
• Action, NotAction, Resource, NotResource, wildcards, and the supported string, boolean, and null conditions follow AWS identity-policy form.
• s3:ListBucket applies to the bucket ARN and can restrict s3:prefix and s3:delimiter. It is separate from object read and write permission.
• AWS-compatible s3:GetObject authorizes both GET and HEAD. Sciveo additionally accepts s3:HeadObject as a policy extension that authorizes HEAD without authorizing GET; boto3 still sends the ordinary S3 HeadObject request and requires no client changes. An explicit deny matching either action denies HEAD. s3:PutObject authorizes regular, presigned, POST, copy-destination, and multipart writes. Copy also requires s3:GetObject on its source. Tagging, multipart listing/abort, bucket lifecycle, and bulk deletion retain their distinct S3 actions.
• Bucket CORS configuration uses the distinct s3:GetBucketCORS, s3:PutBucketCORS, and s3:DeleteBucketCORS actions on the bucket ARN.
• A write-only identity with only s3:PutObject cannot read, HEAD, list, copy from, tag, or delete its objects.
• Unknown and disabled access keys fail authentication. When iam is present, only its users are accepted. Existing single-key configurations retain their full-access behavior when iam is absent.
• Unsupported policy fields, including resource-policy Principal, fail configuration loading instead of being silently ignored. V1 implements identity policies, not the AWS IAM management API or bucket resource policies.

Sciveo-native HTTP operations have separate permissions: storage:SearchObjects, storage:GetStats, storage:GetDiskUsage, storage:GetHealth, storage:GetRoots, storage:GetBuckets, storage:GetMetrics, and cluster storage:GetClusterStatus. Their resource is arn:sciveo:storage:::STORAGE_NAME, optionally followed by /BUCKET. This prevents a narrowly scoped S3 identity from bypassing its policy through the native search or observability API.

Secrets should normally use root-readable files rather than inline JSON. A relative secret_key_file is resolved from the configuration file directory. Policies are loaded at process startup, so restart the storage service after a policy or identity change. StorageLocalClient is trusted in-process administrative access to the same files and metadata database; remote IAM is enforced at the signed HTTP boundary and does not wrap that local API.

threads is the request thread count in each storage worker. workers is the number of supervised storage workers: worker 0 listens on port, worker 1 on port + 1, and so on. parallel remains a backward-compatible alias for threads. When top-level workers is greater than one, keep http.workers at one; http.workers is the older Gunicorn mode where several processes share a single port.

The following starts one master supervising ports 3900 through 3903, with 32 request threads per worker:

sciveo storage --s3 --config /etc/sciveo/storage-s3.json \
  --port 3900 --workers 4 --threads 32 --master

The master role is the default when neither --master nor --slave is given, including when workers is the default value of one. Keeping --master in new service scripts makes the intended process role explicit. The master does not serve S3 requests. It starts the children, restarts an unexpectedly failed child with bounded crash loop protection, and forwards SIGINT/SIGTERM shutdown. Each child is an independent server using the same paths, metadata DB, and root-derived lock directory.

Existing commands without a role option continue to expose the same configured port, now through a master supervising one child by default. Manual direct process management remains available through an explicit slave, which serves only the exact requested port:

sciveo storage --s3 --config /etc/sciveo/storage-s3.json \
  --slave --port 3902 --threads 32

Multi-Node S3 Cluster

Cluster mode is a separate backend under sciveo.storage.cluster; standalone mode remains the default and keeps its existing file layout and behavior. The S3 protocol flag does not change:

sciveo storage --s3 --mode cluster --role controller --config controller.json
sciveo storage --s3 --mode cluster --role node --config node.json
sciveo storage --s3 --mode cluster --role gateway --config gateway.json

--master, --slave, --workers, and --threads still describe local process supervision. --role describes the network responsibility. A gateway exposes the boto3-compatible endpoint, a controller owns cluster metadata and placement, and a node stores immutable replicas on its configured local paths. Object data flows directly between gateways and nodes or between nodes during repair; it never flows through the controller.

Controller configuration uses authoritative PostgreSQL metadata in production:

{
  "s3": {
    "mode": "cluster",
    "cluster": {
      "role": "controller",
      "name": "media-cluster",
      "node_id": "controller-1",
      "token": "CHANGE_TO_A_LONG_RANDOM_INTERNAL_TOKEN",
      "replication": {"factor": 3, "write_quorum": 2, "read_quorum": 1},
      "heartbeat_interval": 10,
      "heartbeat_timeout": 30,
      "repair_interval": 30,
      "tls": {
        "ca_file": "/etc/sciveo/cluster-ca.pem",
        "cert_file": "/etc/sciveo/controller.pem",
        "key_file": "/etc/sciveo/controller.key"
      }
    },
    "host": "10.0.0.10",
    "port": 9443,
    "db": {
      "backend": "postgres",
      "url": "postgresql://sciveo:CHANGE_ME@postgres.service/sciveo_cluster"
    },
    "http": {"server": "gunicorn", "workers": 1}
  }
}

Every gateway and node receives a redundant controller URL list. A node also needs a stable identity, an internally reachable advertised URL, mounted paths, and a failure-domain label:

{
  "s3": {
    "mode": "cluster",
    "cluster": {
      "role": "node",
      "name": "media-cluster",
      "node_id": "storage-1",
      "controllers": [
        "https://controller-1.internal:9443",
        "https://controller-2.internal:9443",
        "https://controller-3.internal:9443"
      ],
      "advertise_url": "https://storage-1.internal:9444",
      "token": "CHANGE_TO_A_LONG_RANDOM_INTERNAL_TOKEN",
      "failure_domain": {"site": "primary", "rack": "rack-1"},
      "replication": {"factor": 3, "write_quorum": 2},
      "tls": {
        "ca_file": "/etc/sciveo/cluster-ca.pem",
        "cert_file": "/etc/sciveo/storage-1.pem",
        "key_file": "/etc/sciveo/storage-1.key"
      }
    },
    "host": "10.0.0.21",
    "port": 9444,
    "paths": [
      {"path": "/srv/storage/d1", "require_mount": true},
      {"path": "/srv/storage/d2", "require_mount": true}
    ],
    "db_path": "~/.sciveo/storage/storage-1-node.sqlite3",
    "workers": 1,
    "http": {"server": "gunicorn", "workers": 1}
  }
}

Cluster PUT stages the incoming stream once, calculates CRC32 during that same pass, writes immutable replicas concurrently, and commits the object generation only after write quorum. ETags are opaque UUID tokens, never MD5 file hashes. For replication factor 3, every object is stored on three distinct active nodes: one primary copy plus two redundant copies. Nodes are selected with capacity-weighted rendezvous hashing over the bucket, key, and stable node ID, while preferring distinct failure domains. Placement is deterministic rather than random, distributes different keys across the full healthy node pool, and does not reshuffle unrelated objects when membership changes.

Reads try live replicas in turn. Each gateway rotates equivalent read candidates so concurrent clients do not always select the same physical node; retries still fall through to another replica. This balancing is local to each gateway process, not a cluster-global least-in-flight scheduler. Deletes commit a tombstone before best-effort physical cleanup. Controllers detect live under-replicated generations and ask a new target node to pull directly from a healthy source. They also remove live replicas above the configured factor after failed nodes recover. Active controllers coordinate repair through an expiring lease in the shared metadata database; each object refreshes replica liveness and retries another source if a peer becomes unavailable during a long repair pass.

Adding a node does not require gateway changes. Give the machine a new unique cluster.node_id, configure its controller URLs, advertised peer URL, TLS/token, and mounted paths, then start the node role. It self-registers, heartbeats, and immediately participates in future placement and repairs. Healthy existing factor-three objects are not automatically rebalanced merely because capacity joined the cluster.

Replacing a dead machine may reuse its stable node_id after the controller has classified the old incarnation offline. Every node derives a persistent incarnation ID from its local metadata database. A replacement with a new incarnation invalidates the dead machine's controller replica rows, then normal repair restores factor three from surviving copies. A second incarnation trying to claim an identity that is still healthy is rejected, preventing accidental split brain.

Peer URLs require HTTPS unless cluster.allow_insecure=true is explicitly set for isolated loopback development. Protect the shared internal token and use mTLS certificates on every peer endpoint. Non-loopback controllers should use PostgreSQL HA; SQLite cluster metadata is intended only for local development. When Gunicorn terminates peer TLS directly, controller and node roles require a client certificate whenever tls.cert_file, tls.key_file, and tls.ca_file are configured. The same requirement can instead be enforced by an internal reverse proxy.

The initial cluster backend supports bucket lifecycle, PUT/GET/HEAD/DELETE, range reads, prefix and delimiter pagination, bulk delete, tags, server-side copy, SigV4 presigned GET/PUT, quorum replication, node failover, and repair. Legacy Signature V2 remains disabled; set boto3 Config(signature_version="s3v4") when generating presigned URLs for a custom endpoint. Distributed multipart upload, presigned POST, ACLs, versioning APIs, erasure coding, multi-site replication, and automatic orphan inventory cleanup remain future cluster work. Standalone support for those already implemented S3 operations is unchanged.

sciveo storage --s3 --config ./storage.json

--paths are mounted directories, not raw block devices. A path entry may be a string or an object with path, require_mount, optional root_id, and optional expected device. Sciveo persists an identity marker on every root so a detached/replaced mount is not silently treated as the original storage. Object metadata is stored in ~/.sciveo/storage/<storage-name>-s3.sqlite3 by default with the SQLite backend. Use db_path/--db-path for another SQLite file, or db_backend=postgres with db_url/--db-url for PostgreSQL. storage_name/--storage-name selects the default DB filename and observability label; the persistent root identities and metadata DB, not the display name, define the actual storage set.

For heavier production metadata concurrency, use PostgreSQL:

{
  "s3": {
    "paths": ["/mnt/d1", "/mnt/d2", "/mnt/d3"],
    "access_key": "sciveo",
    "secret_key": "CHANGE_ME",
    "storage_name": "video-cache",
    "db": {
      "backend": "postgres",
      "url": "postgresql://sciveo:CHANGE_ME@127.0.0.1:5432/sciveo_storage"
    }
  }
}

The same settings can also be supplied as flat keys (db_backend, db_url) or CLI options (--db-backend postgres --db-url ...). Install sciveo[storage-postgres] and verify that psycopg2 can load the host libpq before selecting PostgreSQL.

Each metadata database has a persistent store UUID and owns one logical set of storage roots. Starting a service with a completely disjoint path/identity set against an occupied DB is rejected before metadata can be changed. Temporarily omitted roots are retained with an inactive index and become visible again when the same root identity is re-added.

The service does not run a metadata sync pass on normal startup. Run explicit reconciliation with --sync when you want files that exist on disk but are missing from the SQL metadata DB to be added, metadata rows for files missing from currently healthy paths to be removed, and pending deletion tombstones to be completed. Metadata for an unavailable root is retained and hidden from list/search/GET until that root recovers; sync never converts a detached disk into permanent object deletion.

sciveo storage --s3 --sync --config ./storage.json
sciveo storage --s3 --sync --dry-run --config ./storage.json
sciveo storage --s3 --sync --since 10m --config ./storage.json

--since accepts a Unix timestamp, an ISO timestamp, or a relative duration such as 10m, 2h, or 1d. It is intended for fast catch-up after a full index build, for example when preparing a PostgreSQL metadata DB while the SQLite-backed service is still live. Keep the target PostgreSQL service in sync mode only during this phase; do not run two metadata backends as active writers. Both configurations must use the same physical paths and lock_path. Stop the SQLite writers, run one final incremental/full sync with zero inconsistencies, then cut traffic over to PostgreSQL. Incremental sync only reconciles files with filesystem mtimes newer than the timestamp and only prunes recently updated metadata rows; run a full --sync when you need a complete consistency audit. Foreign in-flight mutation journals are skipped and the affected keys return a retryable unavailable response until their owning metadata DB recovers them.

New objects use capacity-weighted rendezvous hashing by persistent root UUID across writable paths. Reordering paths or changing a mount point therefore does not perturb equal-health placement. Paths above target_free_ratio have equal placement weight; paths below that target receive progressively less new data. A path is excluded from new allocations when writing an object would violate either the configured min_free_ratio or min_free_bytes reserve. It returns to placement only after crossing resume_free_ratio, avoiding rapid full/not-full state changes.

A full path remains healthy and readable: indexed objects on it continue to be listed and GET/HEAD continue to work. Only new allocations are redirected. A path is unhealthy only when it is inaccessible, unreadable, detached/unmounted, has a changed device while active, or reports a real filesystem I/O failure. Metadata for an inaccessible root remains indexed but is filtered from client results; when the same root identity returns, its objects become visible again. Configure health checks with --health-check-interval; configure the capacity policy through the generic --config JSON file or SCIVEO_STORAGE_* environment variables.

Known object sizes are reserved with filesystem preallocation where supported. This makes concurrent processes compete for real disk capacity before consuming the HTTP body and allows a failed candidate to fall through to the next ranked path safely. Unknown-length streams are bounded while they are consumed: Sciveo aborts the temporary write before commit if a configured request, object, bucket, or remaining-capacity limit is crossed.

Object listing uses the bucket/key index with a prefix key range rather than a table scan. Empty bucket deletion removes the whole bucket tree below each healthy storage path in one filesystem operation. Unavailable paths are left untouched, preserving their data for recovery.

Object mutations are coordinated by bucket/key across threads, processes, and metadata backends on one host. Every backend uses the same root-set-derived striped POSIX lock directory; lock_path or SCIVEO_STORAGE_LOCK_PATH can pin that directory for hardened services. SQLite adds WAL/retry handling, while PostgreSQL uses transactional/advisory schema locks and a fork-aware connection pool. Multi-host writers sharing the same files are not supported; use one storage host behind local Gunicorn workers and an nginx endpoint. Writes, copies, and local moves keep a temporary destination backup until the SQL transaction commits. Failed metadata commits restore the previous bytes. Deletes first create a transactional tombstone, hiding the object immediately; sync or maintenance can finish physical cleanup after a transient filesystem failure. Maintenance reconciles only indexed tombstones and does not walk object files. Schema v5 records a persistent metadata-store UUID in mutation journals so a migration target cannot roll back a transaction owned by the source DB. Schema v6 adds the bucket CORS configuration subresource without walking or modifying object files.

Standalone object ETags are lightweight stat-based metadata tokens; cluster ETags are opaque UUID tokens. Neither is a content MD5 hash. This avoids hashing very large video/object streams during upload. Use an explicit application checksum in object metadata when content verification is required.

Authentication uses AWS Signature V4 for boto3 requests and presigned GET/PUT/POST. Header timestamps, credential scope, POST policy conditions, body SHA-256, and supported streaming checksum trailers are validated. Legacy Signature V2 is disabled unless allow_legacy_auth is explicitly enabled. The demo sciveo/secret credentials are rejected by server startup unless allow_demo_credentials is explicitly enabled for a disposable test. Use a protected secret file and terminate public HTTPS at a hardened reverse proxy.

Gunicorn with gthread workers is the Linux production HTTP runtime. Waitress is an explicitly configured compatibility runtime that buffers request bodies and is intended for local tests. Sciveo fails fast instead of attempting Gunicorn prefork on macOS.

Object HEAD, GET, and byte-range responses preserve an explicitly stored Content-Type. When it is missing, standalone and cluster storage infer the MIME type from the object key's extension (for example, .mp4 is video/mp4, .mov is video/quicktime, and .jpg/.jpeg, .png, and .gif use their image types). Unknown extensions use application/octet-stream. This also applies to existing objects without type metadata and presigned GET responses, without a sync, file-content scan, or database rewrite.

Python client example:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://127.0.0.1:9000",
    aws_access_key_id="sciveo",
    aws_secret_access_key="CHANGE_ME",
    region_name="us-east-1",
)

s3.upload_file("clip.mp4", "media", "clips/clip.mp4")
s3.download_file("media", "clips/clip.mp4", "clip-copy.mp4")

Browser CORS And Presigned Requests

Bucket CORS uses the normal boto3 S3 API and is persisted in the storage metadata database. It is not an ACL and does not grant object access: SigV4, presigned policy validation, and Sciveo IAM still authorize the actual request. CORS only tells a browser which cross-origin responses it may expose.

s3.put_bucket_cors(
    Bucket="media",
    CORSConfiguration={
        "CORSRules": [
            {
                "ID": "browser-media",
                "AllowedOrigins": ["https://app.example.com"],
                "AllowedMethods": ["GET", "HEAD", "POST"],
                "AllowedHeaders": ["content-type", "x-amz-*"],
                "ExposeHeaders": ["ETag", "Content-Range", "x-amz-request-id"],
                "MaxAgeSeconds": 300,
            }
        ]
    },
)

get_url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "media", "Key": "clips/clip.mp4"},
    ExpiresIn=300,
)

post = s3.generate_presigned_post(
    Bucket="media",
    Key="incoming/generated-object-id.mp4",
    Fields={"Content-Type": "video/mp4"},
    Conditions=[
        {"Content-Type": "video/mp4"},
        ["content-length-range", 1, 500000000],
    ],
    ExpiresIn=300,
)

generate_presigned_post() returns both url and fields. A browser must submit every returned field plus one file part as multipart/form-data; do not manually set the multipart boundary. Sciveo handles anonymous OPTIONS preflight using the first matching bucket rule and adds AWS-style Access-Control-Allow-*, expose, credentials, max-age, and Vary headers. The reverse proxy must forward Origin, Access-Control-Request-Method, and Access-Control-Request-Headers; it should not replace the bucket policy with global static CORS headers.

Use get_bucket_cors() to inspect the configuration and delete_bucket_cors() to remove it. StorageLocalClient exposes the same three method names for trusted in-process administration.

Object tags use the S3 object-tagging model. Add tags during upload with the normal Tagging/x-amz-tagging mechanism, update them with put_object_tagging, and search them through the Sciveo metadata search API:

s3.put_object(
    Bucket="media",
    Key="clips/clip.mp4",
    Body=open("clip.mp4", "rb"),
    Tagging="kind=raw&project=demo",
)

s3.put_object_tagging(
    Bucket="media",
    Key="clips/clip.mp4",
    Tagging={"TagSet": [{"Key": "stage", "Value": "ready"}]},
)

Sciveo also provides a boto3-like remote client that keeps the normal S3 methods and adds Sciveo-native metadata search endpoints:

import sciveo.storage.s3

s3 = sciveo.storage.s3.client(
    endpoint_url="http://127.0.0.1:9000",
    aws_access_key_id="sciveo",
    aws_secret_access_key="CHANGE_ME",
    region_name="us-east-1",
)

s3.upload_file("clip.mp4", "media", "clips/clip.mp4")
objects = s3.search_objects(Bucket="media", Prefix="clips/", Period=7200)
tagged = s3.search_objects(Bucket="media", Prefix="clips/", Tags={"stage": "ready"})
stats = s3.search_stats(Bucket="media", Prefix="clips/")
metrics = s3.metrics(Bucket="media")

The Sciveo-native HTTP surface is intentionally small and separate from AWS S3 compatibility: /sciveo/storage/search, /sciveo/storage/stats, /sciveo/storage/disk-usage, /sciveo/storage/health, /sciveo/storage/roots, /sciveo/storage/buckets, and /sciveo/storage/metrics. Search and stats are DB-indexed and support exact generic tag filters with tag.NAME=value or Tags={"name": "value"} through the Python client. /sciveo/storage/metrics returns OpenMetrics text for Prometheus-style scraping.

Storage stats also include rolling file-operation counts for the latest 24 UTC hours. The profile is calculated with a SQL aggregation over the existing hourly operation counters, so it adds no counters or writes to the request path. JSON clients read it from operations.hourly_files_utc; terminal users get the same 24-row table with:

sciveo storage --s3 --config /etc/sciveo/storage-s3.json --stats --render text

Each row includes its UTC hour, total file operations, files / 60 average per minute, and write/read/delete counts.

Disk-usage totals count each physical filesystem once even when multiple storage roots share it. Per-root object counts and bytes remain separate so operators can still inspect placement. S3 object tags follow AWS request limits: at most 10 tags, with keys up to 128 characters and values up to 256 characters.

Local workers running on the same storage machine can resolve an object to its committed file path without downloading it through S3:

from sciveo.storage.local import StorageLocalClient

storage = StorageLocalClient.from_config("/etc/sciveo/storage-s3.json")
storage.create_bucket("media")
path = storage.path("media", "clips/clip.mp4")

# Use path directly with local video/image processing code.
storage.copy("media", "clips/clip.mp4", dst_key="processed/clip.mp4")
storage.move("media", "tmp/clip.mp4", dst_key="queue/0/clip.mp4")
storage.delete_objects("media", ["processed/clip.mp4", "queue/0/clip.mp4"])
storage.delete_bucket("media")

StorageLocalClient reads the same storage config and SQL metadata DB as the server. It only returns paths for committed objects whose files still exist, rejects metadata paths outside the configured storage roots, and updates both object files and metadata for local bucket, copy, move, and delete operations. move() defaults to no_copy=True: the file is renamed on its current storage path and only the bucket/key metadata changes, so changing a key does not trigger an inter-disk copy when the destination key hashes to another path. Use move(..., no_copy=False) only when the old copy/delete behavior is desired.

Run bounded metadata, staging, and deferred-delete cleanup explicitly. The deletions section reports tombstones seen, reconciled, and deferred:

sciveo storage --s3 --action maintenance --dry-run --config ./storage.json
sciveo storage --s3 --action maintenance --config ./storage.json

Ordinary boto3 clients do not need an update for these server-side changes. Processes importing StorageLocalClient should run the same current Sciveo release as the servers so they share locking, tombstone, root-identity, and rollback behavior. Update the Sciveo remote client when using newly added native search/health/stats methods; normal delegated S3 method signatures remain boto3-compatible.

When upgrading from a release before metadata schema v5, stop every S3 server and local writer using these roots, update them together, then restart. Do not mix old and new lock/journal implementations during a rolling restart. Schema migration is automatic and does not run --sync; run an explicit dry-run/full sync later as an operational consistency audit.

The repository includes a destructive-only-to-its-marked-directory stress runner. It prints a parameter table before opening sockets and a result table at completion:

PYENV_VERSION=sciveo python test/storage_runtime.py \
  --path /tmp/sciveo-storage-runtime \
  --servers 4 --threads 16 --clients 16 --roots 5 --files 240

The runtime mixes boto3, Sciveo remote, and local clients; exercises presigned requests, transfers, tags, conditions, ranges, copy directives, multipart, pagination, root loss/recovery, process-crash recovery, root omission/re-add, disjoint-DB rejection, live sync, and parallel apply-sync while a local cleanup client deletes the oldest indexed cohort; then verifies reclaimed free space and exact bytes against SQL metadata. Every parallel sync pass, the final apply sync, and the final dry-run must report zero inconsistencies. Every run writes a staged JSON report under /private/tmp unless --report overrides it. Use --db-backend postgres --db-url ... with a fresh disposable database to repeat the same gate with PostgreSQL.

Experiments Client

The experiment helpers expose project runs, parameter sampling, datasets, scoring, plots, metadata, local execution, and optional remote synchronization. They are useful for ML, AI, research, and engineering scripts where each run should have a known configuration and a recorded result.

Minimal pattern:

import sciveo

def run_once():
    with sciveo.open() as experiment:
        experiment.log({"message": "hello experiment"})
        experiment.score(0.0)

sciveo.start(
    project="example-project",
    function=run_once,
    configuration={},
    remote=False,
    count=1,
)

Run a project:

import sciveo

def evaluate_model():
    with sciveo.open() as experiment:
        learning_rate = experiment.config.learning_rate
        experiment.log("learning_rate", learning_rate)
        experiment.score(0.94)

sciveo.start(
    project="example-project",
    function=evaluate_model,
    configuration={"learning_rate": {"values": [0.001, 0.01]}},
    remote=False,
    count=2,
)

Remote synchronization requires a configured Sciveo API account.

Experiment Concepts

• Project: a named ML, research, or engineering workspace containing related runs.
• Experiment: one run with a sampled configuration, measurements, outputs, and score.
• Configuration: parameter values used by the run.
• Dataset records: structured references to input data, split definitions, or generated data artifacts.
• Score: numeric or structured result used for comparison and optimization.
• Local mode: runs experiments on the current machine without API synchronization.
• Remote mode: synchronizes project and experiment data through the Sciveo API when configured.

Example with dataset and score metadata:

import sciveo

def evaluate_dataset():
    with sciveo.open() as experiment:
        experiment.dataset({
            "name": "sensor-window-001.csv",
            "split": {"train": 0.8, "test": 0.2},
        })
        experiment.eval("rmse", 0.034)
        experiment.eval("mae", 0.021)
        experiment.score(0.93)

sciveo.start(
    project="sensor-eval",
    function=evaluate_dataset,
    configuration={},
    remote=False,
    count=1,
)

Example local parameter sweep:

import sciveo

def calibrate():
    with sciveo.open() as experiment:
        window = experiment.config.window
        threshold = experiment.config.threshold
        experiment.log({"window": window, "threshold": threshold})
        experiment.score(0.8)

sciveo.start(
    project="lab-calibration",
    function=calibrate,
    configuration={
        "window": {"values": [32, 64, 128]},
        "threshold": {"values": [0.1, 0.2, 0.3]},
    },
    remote=False,
    sampler="grid",
)

Sciveo does not force a specific analysis or ML stack. Experiments can call NumPy, SciPy, scikit-learn, PyTorch, TensorFlow, OpenCV, custom sensor clients, or any other Python code that fits the workflow.

Python Module Map

Common package areas:

• sciveo.monitoring: metrics, watchdogs, monitoring CLI.
• sciveo.network: scan, SSH, Modbus, SNMP, MQTT, HTTP, emulators.
• sciveo.admin: edge admin web UI and service helpers.
• sciveo.ops: doctor and fleet diagnostics.
• sciveo.ui.term: tables, menus, progress, status, panels, trees, documents, image previews, and stream helpers.
• sciveo.agents: agent console, providers, local runtimes.
• sciveo.chat: encrypted chat server/client transport.
• sciveo.media: capture, media CLI, pipeline workers.
• sciveo.ml.images: image ML helpers, SCIVEYOLO inference, embeddings, descriptions, image transforms.
• sciveo.storage: local S3-compatible object store.
• sciveo.iam: generic Sciveo IAM policy evaluation and access control.
• sciveo.certification: framework profiles, readiness scoring, suggestions, rendering, and certification CLI delegation.
• sciveo.api: API clients and predictor server.
• sciveo.db, sciveo.web, sciveo.content, sciveo.tools: shared support modules.

Development Checks

Useful local checks:

python -m py_compile sciveo/cli.py sciveo/ml/images/sciveyolo/model.py sciveo/ml/images/sciveyolo/torch_net.py
python -m unittest discover -s test -p "test_*.py" -v

For SCIVEYOLO runtime validation, use a UT-capable build environment to create .sciveyolo artifacts, then test production loading in an environment that has PyTorch/OpenCV/NumPy but no Ultralytics installed.

Contact

Pavlin Georgiev
pavlin@softel.bg