edgeserve

Run LLMs on NVIDIA Jetson and NVIDIA GPUs behind an OpenAI-compatible API, using NVIDIA TensorRT Edge-LLM.

• OpenAI API (chat, completions, streaming): works with the OpenAI SDKs and curl.
• Serves Hugging Face models and your own fine-tuned, pruned or quantized checkpoints, in FP16/BF16, INT4 (AWQ, GPTQ), INT8, FP8 or NVFP4, whichever your GPU supports.
• Converts the model to a TensorRT engine for your GPU on first start, and caches it.
• Keeps your request IDs and saves every request, with metrics per request and per token (latency, tokens/s, energy).
• Queues and batches simultaneous requests, with limits per client, and guards small devices against running out of memory.
• edgeserve setup --network adds HTTPS and an API key for other machines; edgeserve keys add gives each client its own key.

Steps: 1. Install · 2. Build the GPU runtime · 3. Export a model · 4. Serve · 5. Send requests · 6. Access from other machines. Also: request format, many requests at once, saved requests, your own models, batch jobs, service, uninstall, commands, options and settings, troubleshooting, documentation.

In the commands below, replace each <placeholder>, angle brackets included, with your own value.

Requirements

Device	Software	Status
Jetson Orin Nano / Orin NX / AGX Orin	JetPack 6.2.x or 7.2.x	tested (Orin Nano 8 GB, JetPack 6.2.1)
Jetson Thor	JetPack 7.x	expected to work
x86-64 PC with an NVIDIA RTX 30/40/50, A100, A10, L4, L40S or B200	Ubuntu 22.04/24.04, CUDA 12 or 13, TensorRT 10	expected to work

Python 3.10 or newer and build tools are needed too. On a Jetson:

sudo apt install nvidia-jetpack git cmake build-essential python3-venv

On a PC, install the NVIDIA driver, CUDA and TensorRT 10 with its development packages, then sudo apt install git cmake build-essential python3-venv. edgeserve setup checks all of this and prints what is missing. edgeserve 0.1 is built on TensorRT Edge-LLM 0.11 (commit 0f52705723ab). Details per platform: docs/PLATFORMS.md.

Disk space. edgeserve's tools take about 7 GB, the same whichever models you use: 1.7 GB for the GPU runtime (step 2, which needs 3 GB free while it builds) and 5.5 GB for the exporter (step 3). Each model needs its own space on top, which grows with its size: its download, its exported folder and its engines. A 0.5B model takes up to 2 GB in INT4 and 3.5 GB in FP16 (details).

1. Install

With pip:

python3 -m venv <venv-dir>
source <venv-dir>/bin/activate
pip install --upgrade pip
pip install edgeserve

Or from the repository:

git clone https://github.com/Surya-Rayala/edgeserve.git
cd edgeserve
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .

<venv-dir> is any new folder for the Python environment, for example ~/edgeserve-venv. Both ways give the same edgeserve command, and both keep their files in the same places: settings in ~/.config/edgeserve/, the GPU runtime and the model exporter in ~/.local/share/edgeserve/, engines and build files in ~/.cache/edgeserve/. In a new terminal, run the source …/bin/activate line again.

2. Build the GPU runtime

edgeserve setup

Run it once per machine, as your normal user (not with sudo). It downloads TensorRT Edge-LLM, builds it for your GPU and tests it. This takes about 30 minutes on a Jetson Orin Nano (less on a PC) and needs internet access and 3 GB of free disk. Run it again after upgrading edgeserve, JetPack, CUDA or TensorRT: it rebuilds only what changed.

3. Export a model

edgeserve serves models in TensorRT Edge-LLM's format: a folder with the model converted to ONNX, its weights and its tokenizer. edgeserve export makes this folder from a Hugging Face checkpoint:

edgeserve export <checkpoint> <model-dir>

Argument	What it is
<checkpoint>	a Hugging Face model ID, such as Qwen/Qwen2.5-0.5B-Instruct-AWQ, or the folder of your own checkpoint (your own models)
<model-dir>	a new folder for the exported model, such as ~/models/qwen2.5-0.5b-instruct-awq; its name becomes the model's name

The first export installs TensorRT Edge-LLM's exporter (a few minutes, about 5.5 GB of disk, once for all models). An export takes a few minutes; it runs on the CPU, so it works on the Jetson itself or on any Linux PC with edgeserve and git installed (no edgeserve setup needed there; copy <model-dir> to the Jetson afterwards). It also copies the model's default sampling settings (generation_config.json). For a gated model such as Llama, accept its license on huggingface.co and pass your access token: HF_TOKEN=<your token> edgeserve export <checkpoint> <model-dir>. A folder that TensorRT Edge-LLM 0.11 already exported needs no export: serve it as it is. GGUF files, ONNX models from other tools and ready-made engines cannot be served (what you can bring).

Which models. Text models of the Llama 3.x, Qwen2/Qwen2.5, Qwen3 and HunYuan (dense) families; Qwen3 MoE, Qwen3.5 and Nemotron-H should work but are untested (families and tested models). Choose a precision your GPU supports:

Weights	Hugging Face name looks like	Jetson Orin	RTX 30/40, A100, A10, L4, L40S	Thor, RTX 50, B200
FP16 / BF16 (the original weights)	Qwen/Qwen2.5-0.5B-Instruct	✅	✅	✅
INT4 (AWQ or GPTQ)	…-Instruct-AWQ, …-GPTQ-Int4	✅	✅	✅
INT8 (SmoothQuant)	quantize it yourself (how)	✅	✅	✅
FP8	nvidia/…-FP8	❌	RTX 40, L4, L40S only	✅
NVFP4	nvidia/…-NVFP4	❌	❌	✅

Memory decides the model size. With edgeserve export, an INT4 model builds its engine in about the memory it serves in; an FP16 model needs two to three times that. On an 8 GB Orin Nano, FP16 models up to about 0.6B parameters and INT4 models of 0.5B to 4B were tested, and a pruned 6B INT4 model with a reduced output vocabulary fits too. Boards and GPUs with more memory run correspondingly larger models, in any precision they support. More: every argument, what the export does, exporter options and errors in docs/EXPORT.md; supported models and memory in docs/MODELS.md.

4. Serve

On a Jetson, the model shares the board's memory with everything else running: close what you do not need first (how).

edgeserve serve --model <model-dir>

The first start converts the model to a TensorRT engine for your GPU (a minute to a few minutes) and caches it; later starts take seconds. The server listens on http://127.0.0.1:8000, which only this machine can reach. Stop it with Ctrl-C. To test it, run edgeserve check in a second terminal.

Its options:

Option	Default	What it does
--model DIR	none	the model folder; after edgeserve setup --model <model-dir>, plain edgeserve serve is enough
--output-dir DIR	none	save every request, answered or not, as JSON (saved requests)
--host ADDRESS, --port N	127.0.0.1, 8000	where the server listens; step 6 opens it to the network
--max-batch-size N	1	how many requests are generated together (many requests at once)
--max-input-len N	3072	the longest prompt, in tokens; a longer one is refused
--max-new-tokens N	the model's default, else 512	the answer length when a request does not set max_tokens
--cache-dir DIR	~/.cache/edgeserve	where the converted engines are kept
--queue-size N	16	how many requests may wait for their turn; more are refused (503). -1: no limit
--queue-timeout SECONDS	600	a request that has waited this long fails (503). 0: no limit
--request-timeout SECONDS	no limit	a request not finished this long after it arrived is stopped (504)
--max-client-requests N	no limit	how many requests one client may have waiting or running; more are refused (429)
--client-rate-limit N	no limit	requests a minute per client: N at once, then one every 60/N seconds; more are refused (429)

To set an option permanently, see options and settings.

Many requests at once

Requests wait in a queue, and the oldest runs first. With --max-batch-size N, up to N waiting requests of the same kind (chat or completion), with the same temperature, top_p, top_k and max_tokens, are generated together. When it starts, the server reserves memory for N requests of up to --max-input-len + --max-new-tokens tokens each, so lower those two to fit a larger N. Changing any of the three builds a new engine (a few minutes; it is cached too). For example, to batch up to 3 requests and keep every request until its turn:

edgeserve serve --model <model-dir> --max-batch-size 3 --max-input-len 1024 --max-new-tokens 256 --queue-size -1

• A client is one API key (a key for each client), or one address on a server without keys.
• --queue-size -1 keeps every request: there is no wait limit unless you also give --queue-timeout, and no connection limit. Otherwise one address can open at most 32 connections, and all clients together 128.
• A refused request (429 or 503) gets a Retry-After header with the seconds to wait. The OpenAI SDK and edgeserve batch wait and retry by themselves; with curl, retry yourself. A request stopped by --request-timeout while running is not retried, as it would run out of time again.
• A waiting client has its own timeout: after 600 s without an answer, the OpenAI SDK drops the request and sends it again at the back of the queue, twice at most; OpenAI(..., timeout=3600) waits longer.

5. Send requests

With curl:

curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" \
  -H "X-Request-Id: demo-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'

With Python (pip install openai):

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")  # no key needed until step 6
model = client.models.list().data[0].id                                  # the served model's name
reply = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "demo-002"},
)
print(reply.choices[0].message.content)
print(reply.model_extra["metrics"]["decode_tokens_per_s"], "tokens/s")

Request format

A chat request (POST /v1/chat/completions) is a JSON object; only messages is required:

{
  "messages": [
    {"role": "system", "content": "Summarize the text in two sentences."},
    {"role": "user", "content": "<the text>"}
  ],
  "max_tokens": 160,
  "temperature": 0,
  "request_id": "meeting-0001",
  "metadata": {"source": "room-12"}
}

Field	Meaning	Default
messages	the conversation, oldest first. role: system (instructions), user, assistant (an earlier answer) or developer (same as system); content: the text. Up to 64 messages of up to 128 KB each	required
max_tokens	the longest answer, in tokens (max_completion_tokens means the same)	--max-new-tokens (the model's generation_config.json, else 512)
temperature	randomness, 0 to 2. 0 always picks the most likely next token: the same answer every time	the model's generation_config.json, else 0
top_p, top_k	when sampling, choose only among the most likely tokens: those that make up top_p (above 0, up to 1) of the probability, or the top_k (up to 1024) most likely	the model's generation_config.json, else no limit
seed	a number that makes sampling repeatable	random, saved in the record
stop	a string, or up to 4, that ends the answer	none
stream	true: send the answer in pieces as it is generated; add "stream_options": {"include_usage": true} for a last piece with the token counts and metrics	false
request_id	your ID for this request (the same as the X-Request-Id header)	req_<random hex>
metadata	up to 16 string pairs of your own (keys up to 64 characters, values up to 512), saved in the record	none
return_record	true: the response also holds the full record, with per-token timings (not with stream)	false
model	the served model's name, as GET /v1/models lists it; optional	the served model
n (only 1), frequency_penalty and presence_penalty (only 0), logprobs (only false), user, safety_identifier, prompt_cache_key, a message's name	accepted so that every OpenAI client works; they change nothing	—

A completion request (POST /v1/completions) takes prompt, one string with the model's chat template already applied, instead of messages. Any other field is refused with HTTP 400, never ignored.

Request IDs. Send your ID in the X-Request-Id header or the request_id field: 1 to 512 printable ASCII characters (with letters, digits, ., _ and -, the record's file name is the ID itself). Without one, the server makes one (req_…). The ID comes back in the response's request_id and id (chatcmpl-<id>) and in its X-Request-Id header. With edgeserve serve --output-dir <output-dir>, every request is saved as <output-dir>/records/<id>.json, which GET /v1/records/<id> also returns. A request with the ID of one that is still running gets HTTP 409.

The response is OpenAI's, plus request_id and metrics:

{"id": "chatcmpl-meeting-0001", "object": "chat.completion", "model": "<model name>",
 "choices": [{"index": 0, "message": {"role": "assistant", "content": "…"}, "finish_reason": "stop"}],
 "usage": {"prompt_tokens": 162, "completion_tokens": 72, "total_tokens": 234},
 "request_id": "meeting-0001", "metrics": {"ttft_ms": 303, "decode_tokens_per_s": 25.9, …}}

For streaming, send "stream": true (stream=True in the SDK). Every field, error code and metric: docs/API.md.

6. Access from other machines

edgeserve setup --network

This creates three things in ~/.config/edgeserve/:

• An API key. It is printed once. To see it again: grep EDGESERVE_API_KEY ~/.config/edgeserve/edgeserve.env.
• An HTTPS certificate, and edgeserve-ca.crt, the file clients need to trust it.
• The setting to listen on the network (all interfaces, port 8000).

Restart edgeserve serve. From then on the server speaks only HTTPS and every /v1/ request needs the key, on this machine too (edgeserve check finds both by itself), so the http:// examples of step 5 stop working.

On each client machine you need the server's edgeserve-ca.crt and the key. Below, <server-ip> is the server's IP address (edgeserve setup --network prints it) and <user> is your user name on the server. The first two lines copy both from the server over SSH; you can also paste the key yourself, e.g. export EDGESERVE_API_KEY=es-….

Linux and macOS (Terminal):

scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env")
curl --cacert edgeserve-ca.crt -H "Authorization: Bearer $EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
curl --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions \
  -H "Authorization: Bearer $EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'

Windows (PowerShell; scp and curl.exe come with Windows 10 and 11):

scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
$env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env"
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt -H "Authorization: Bearer $env:EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
Set-Content -Encoding ascii request.json '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions `
  -H "Authorization: Bearer $env:EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-002" `
  -d "@request.json"

Type curl.exe, not curl (in Windows PowerShell, curl is a different command). --ssl-no-revoke is needed because Windows otherwise asks whether the certificate was revoked, which a private CA cannot answer. The request body goes in a file because PowerShell changes the quotes inside inline JSON.

Python, on any of them: make a virtual environment with the OpenAI SDK (python3 -m venv <client-venv> and <client-venv>/bin/pip install openai httpx; on Windows py -m venv <client-venv> and <client-venv>\Scripts\pip install openai httpx), then run this with its Python:

import os, ssl
import httpx
from openai import OpenAI

client = OpenAI(base_url="https://<server-ip>:8000/v1", api_key=os.environ["EDGESERVE_API_KEY"],
                http_client=httpx.Client(verify=ssl.create_default_context(cafile="edgeserve-ca.crt")))
reply = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "client-003"},
)
print(reply.model_extra["request_id"], reply.choices[0].message.content)

Every other request works as in step 5, with https://<server-ip>:8000, the key and the CA file.

• New key: delete the EDGESERVE_API_KEY= line in ~/.config/edgeserve/edgeserve.env, run edgeserve setup --network (it prints the new key) and restart the server. You can also write your own key on that line.
• New IP address or host name: run edgeserve setup --network again and restart the server. Clients keep their edgeserve-ca.crt.
• Firewall: allow only your own network to reach port 8000. edgeserve setup --network prints the ufw commands for your network.
• This machine only again: set EDGESERVE_HOST=127.0.0.1 in the settings file.
• /metrics and /docs need the key too, except when opened at 127.0.0.1 on the server itself; /health stays open.

A key for each client

Rather than sharing the key from edgeserve setup --network (client default), you can give each person or machine its own. You can then remove one without changing the others, and each saved request shows which client sent it. On the server:

edgeserve keys add <name>       # make a key for client <name>; it is shown only now
edgeserve keys list             # the clients that have a key
edgeserve keys remove <name>    # that key stops working at once

The client uses its own key in place of the default key, and still needs edgeserve-ca.crt. To skip copying the key by hand, make it straight into a file (instead of the plain keys add above), then fetch it on the client over SSH, which also deletes the file:

edgeserve keys add <name> | awk '$1 ~ /^es-/ {print $1}' > ~/<name>.key && chmod 600 ~/<name>.key   # on the server
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "cat <name>.key && rm <name>.key")                # on the client

On Windows (PowerShell), the client line is $env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "cat <name>.key && rm <name>.key".

• Adding or removing a key works at once, also on a running server.
• A client reads only its own saved requests; the default key reads all of them.
• --max-client-requests and --client-rate-limit count each key separately.
• edgeserve stores only a hash of each key, so a lost key cannot be shown again: remove it and add a new one.

Saved requests

With edgeserve serve --output-dir <output-dir>, every request is saved:

File	What it holds
<output-dir>/records/<id>.json	the latest attempt of each request ID: the request, the answer, its client, metrics per request and per token, and its status: completed, failed, cancelled or rejected (refused before it was queued; http_status and error say why)
<output-dir>/results.jsonl	one line per attempt, in OpenAI's batch output format

A refused request gets a results.jsonl line, and a record when it named its ID (X-Request-Id, or request_id in a valid body) and that ID has no record yet: a refusal never replaces an earlier record. At most 60 refusals a minute per client are saved, and requests without a valid key never. GET /v1/records/<id> returns a record, also from another machine (with client keys, only the client's own). The folder is readable only by the user running the server, as records hold prompts and answers. Every field: docs/METRICS.md.

Your own models

Find what you have:

You have	What to do
A Hugging Face checkpoint folder (config.json and *.safetensors): the original model, or your fine-tuned, pruned or quantized version of it	export it: edgeserve export <checkpoint-dir> <model-dir>, then serve <model-dir>
A LoRA adapter	load its base model in FP16/BF16 (not 4-bit), merge the adapter (PEFT's merge_and_unload()), save the model and the tokenizer into one folder with save_pretrained(), then export that folder
A folder already exported (converted to ONNX) by TensorRT Edge-LLM 0.11: its config.json has "edgellm_version": "0.11.…" and sits next to model.onnx	nothing: serve it as it is, edgeserve serve --model <model-dir> (if the export made an llm folder, pass that)
A folder exported by another TensorRT Edge-LLM version	export the checkpoint again with edgeserve export; edgeserve refuses the folder and says so
An ONNX model made by another tool (such as Optimum), a GGUF file (llama.cpp, Ollama), an engine built by TensorRT-LLM or another tool, or an INT4 model in another format (bitsandbytes, compressed-tensors, EXL2, HQQ)	cannot be served: export the Hugging Face checkpoint it was made from, or an AWQ or GPTQ version of it
A model of another family (such as Mistral, Gemma or Phi), or one that needs images or audio	not supported, except Qwen3.5, whose image part the export leaves out (supported families)

A checkpoint folder (<checkpoint-dir>) holds these files, as save_pretrained() writes them:

File	Needed	What it holds
config.json	yes	the architecture, with model_type such as llama, qwen2 or qwen3 (supported families)
model.safetensors (or shards with model.safetensors.index.json; pytorch_model.bin also works)	yes	the weights
tokenizer.json, tokenizer_config.json	yes	the tokenizer, and the chat template (chat_template in tokenizer_config.json, or a chat_template.jinja file)
generation_config.json	no	default sampling settings and end-of-sequence tokens; copied into <model-dir>

• Fine-tuned: no changes needed; save the model and its tokenizer into one folder.
• Pruned: fewer layers, or fewer heads or a smaller hidden size in every layer, need only the matching numbers in config.json. A different MLP width in each layer needs "intermediate_sizes": [<width of layer 0>, <width of layer 1>, …] in the checkpoint's config.json, one entry per layer (multiples of 128 for INT4). An exported <model-dir> needs nothing more: the widths are then part of its model.onnx, and edgeserve info --model <model-dir> lists them (mlp_widths).
• Quantized: INT4 AWQ or GPTQ (group size 128), or FP8, NVFP4, INT8 SmoothQuant or INT4 AWQ made with NVIDIA ModelOpt or Edge-LLM's quantizer, for GPUs that run them (INT4 quality).
• Smaller output vocabulary: add -- --reduced-vocab-dir <vocab-dir> to the export command (<vocab-dir> is the folder with vocab_map.safetensors).

An exported folder is served as a whole: copy all of it (every file). edgeserve info --model <model-dir> checks it without starting a server. A folder made with Edge-LLM's own tensorrt-edgellm-export (without edgeserve) differs in three ways: it has no generation_config.json (copy the checkpoint's in, or requests decode greedily), its INT4 AWQ and asymmetric GPTQ weights are slightly clipped, and unless it was exported with --externalize-weights, its engine build needs more memory. If you have the checkpoint, export it again with edgeserve export (INT4 quality).

To host your model for your network, starting at every boot:

edgeserve export <checkpoint-dir> <model-dir>        # once (skip it for a folder that is already exported)
edgeserve setup --network --model <model-dir>        # make it the default model; API key and HTTPS
edgeserve service install                            # serve it now and at every boot

Details, quantizing a model yourself and checking a pruned export: docs/EXPORT.md.

Batch jobs

edgeserve batch <requests.jsonl> --output-dir <results-dir>

<requests.jsonl> has one request per line, for example:

{"id": "q1", "messages": [{"role": "user", "content": "What is CUDA?"}], "max_tokens": 200}
{"id": "q2", "prompt": "<a prompt with the chat template already applied>"}

The OpenAI Batch format (custom_id, method, url, body) works too. Results go into <results-dir>/records/<id>.json, <results-dir>/results.jsonl and <results-dir>/run.json (counts, throughput, latency, energy). Running the same command again skips the finished IDs, so an interrupted batch resumes. For another machine's server, set EDGESERVE_API_KEY as in step 6 and add --server https://<server-ip>:8000 --ca-cert edgeserve-ca.crt.

Run as a service

edgeserve setup --model <model-dir>    # the model the service serves (add --network for other machines)
edgeserve service install              # starts the server now and at every boot; asks for your sudo password
journalctl -u edgeserve -f             # its log
edgeserve service remove               # stop and remove it

Run these commands as your normal user, not with sudo. The service uses the settings file: after changing it, restart the service with sudo systemctl restart edgeserve. edgeserve service install --print only shows the unit file.

Uninstall

First remove the Hugging Face downloads you no longer need (they are in ~/.cache/huggingface), then:

edgeserve service remove     # if you installed the service
rm -rf ~/.config/edgeserve ~/.local/share/edgeserve ~/.cache/edgeserve

Finally, delete <venv-dir> (or the cloned edgeserve folder) and the model folders you no longer need. After a new install, edgeserve setup --network makes a new key and edgeserve-ca.crt: give both to the clients again.

Command reference

Command	What it does
edgeserve setup [--network] [--model DIR]	build the GPU runtime; --model sets the default model; --network adds an API key, HTTPS and listening on the network
edgeserve export CHECKPOINT MODEL_DIR [-- exporter options]	export a Hugging Face checkpoint (an ID or a folder) into a model folder
edgeserve serve	run the server
edgeserve build	only convert the model to an engine (or check the cached one), then exit
edgeserve info	settings and where each comes from, the GPU runtime, the hardware; with --model, the model, its memory estimate and its cached engine
edgeserve check [--server URL] [--ca-cert FILE]	test a running server end to end
edgeserve keys add NAME, edgeserve keys list, edgeserve keys remove NAME	API keys, one per client
edgeserve batch INPUT --output-dir DIR [--server URL] [--ca-cert FILE] [--concurrency N]	run a JSONL file of requests
edgeserve service install [--print], edgeserve service remove	run the server at boot (systemd)
edgeserve --version	the edgeserve and TensorRT Edge-LLM versions

Options and settings

Each option can also be set as a line NAME=value in ~/.config/edgeserve/edgeserve.env (the file edgeserve setup writes; restart the server after editing it) or as an environment variable. An option on the command line wins over the environment, which wins over the file. edgeserve info shows where each value came from.

The model and its engine (serve, build and info):

Option	Setting	Default	Meaning
--model DIR	EDGESERVE_MODEL	none	the model folder (an absolute path in the file)
--max-input-len N	EDGESERVE_MAX_INPUT_LEN	3072	longest prompt in tokens; longer prompts are refused, never cut
--max-new-tokens N	EDGESERVE_MAX_NEW_TOKENS	the model's generation_config.json, else 512	output length when a request sets no max_tokens
--max-batch-size N	EDGESERVE_MAX_BATCH_SIZE	1	requests generated together; each one needs its own memory
--cache-dir DIR	EDGESERVE_CACHE_DIR	~/.cache/edgeserve	where engines are kept

The server (serve):

Option	Setting	Default	Meaning
--output-dir DIR	EDGESERVE_OUTPUT_DIR	none	save every request here (saved requests)
--host ADDRESS	EDGESERVE_HOST	127.0.0.1	127.0.0.1: this machine only; 0.0.0.0: the network (needs an API key)
--port N	EDGESERVE_PORT	8000	port
--queue-size N	EDGESERVE_QUEUE_SIZE	16	requests that may wait beyond the running batch; -1: no limit, nor a wait or connection limit
--queue-timeout SECONDS	EDGESERVE_QUEUE_TIMEOUT	600	how long a request may wait; 0: no limit
--request-timeout SECONDS	EDGESERVE_REQUEST_TIMEOUT	0	how long a request may take from arrival to its last token; 0: no limit
--max-client-requests N	EDGESERVE_MAX_CLIENT_REQUESTS	0	requests one client may have waiting or running; 0: no limit
--client-rate-limit N	EDGESERVE_CLIENT_RATE_LIMIT	0	requests per minute per client; 0: no limit

Keys, certificates and the runtime (no command-line option, so a key never shows in process lists; set them in the settings file or the environment):

Setting	Meaning
EDGESERVE_API_KEY	the key of client default, sent as Authorization: Bearer <key>; more keys: edgeserve keys
EDGESERVE_TLS_CERT, EDGESERVE_TLS_KEY	HTTPS certificate and private key (PEM files)
EDGESERVE_NATIVE_DIR	use another GPU runtime build (development)

Changing --max-input-len, --max-new-tokens or --max-batch-size builds a new engine (cached too). check and batch use this machine's server unless given --server URL, and trust a server's certificate with --ca-cert FILE; batch --concurrency N sets how many requests are in flight (default: the server's batch size + 1). Environment only: EDGESERVE_CONFIG_DIR (another settings folder), EDGESERVE_API_KEY on a client (the key check and batch send), and for edgeserve setup, TRT_PACKAGE_DIR (a TensorRT tarball folder) and JOBS (parallel compile jobs).

Troubleshooting

Problem	Fix
edgeserve setup lists missing packages	install them with the apt line it prints, then run it again
"the GPU runtime is not built yet" or "… from other sources"	run edgeserve setup
/health/ready says building_engine	normal on the first start of a model or new limits
"not enough memory", or the engine build "ran out of memory"	free memory on a Jetson; use an INT4 or a smaller model, lower --max-input-len, keep --max-batch-size 1
"exported with TensorRT Edge-LLM x.y"	export the model again with edgeserve export
"is a Hugging Face checkpoint, not an exported model"	export it first: edgeserve export <checkpoint-dir> <model-dir>, then serve <model-dir>
"is a GGUF file", "holds GGUF files" or "holds an ONNX model made by another tool"	export the Hugging Face checkpoint the model was made from
"has no config.json"	--model must be the folder edgeserve export made; for export, the checkpoint folder must hold config.json
HTTP 401	send the key: Authorization: Bearer <key>; a key removed with edgeserve keys remove no longer works
HTTP 429 too_many_requests or rate_limit_exceeded	the client is over --max-client-requests or --client-rate-limit: retry after the Retry-After seconds
HTTP 503 server_is_overloaded or queue_timeout	the queue is full or the wait too long: retry after Retry-After; raise --queue-size or --queue-timeout, or batch more with --max-batch-size
HTTP 504 time_limit_exceeded	the request took longer than --request-timeout: raise the limit, or ask for fewer max_tokens (a request stopped while running is not sent again automatically)
certificate errors	give the client edgeserve-ca.crt; use a host name or address the certificate lists, or run edgeserve setup --network again after an address change
<host>.local does not connect	use the IP address; mDNS can announce a Docker bridge address (fix: deny-interfaces=docker0 in /etc/avahi/avahi-daemon.conf)
other machines cannot connect	run edgeserve setup --network and restart; check the firewall (sudo ufw status)
answers differ from Hugging Face	the server log lists generation_config.json options Edge-LLM does not support (e.g. repetition_penalty); for INT4 models see INT4 quality
edgeserve export fails	the error table in docs/EXPORT.md

Documentation

Topic	Document
Exporting: every argument, what the export does, your own fine-tuned, pruned and quantized checkpoints, INT4 quality, exporter options, errors	docs/EXPORT.md
Models: supported families, tested models, precision per GPU, memory, disk space	docs/MODELS.md
The API: endpoints, request fields, sampling, streaming, errors, request IDs, records	docs/API.md
Metrics: every per-request and per-token metric and how it is measured	docs/METRICS.md
Platforms: requirements per device, Jetson settings, measured performance, Edge-LLM patches	docs/PLATFORMS.md
How it works: processes, GPU runtime, engine cache, request lifecycle, worker protocol	docs/ARCHITECTURE.md

Development

pip install -e ".[test]"
python -m pytest            # no GPU needed; needs g++ and nlohmann-json3-dev

License

Apache License 2.0 (LICENSE). edgeserve downloads and builds NVIDIA TensorRT Edge-LLM (Apache 2.0), which is not part of this package. Models have their own licenses.