skylar

A tiny runtime + CLI for the Skylar model family — local, sovereign, from-scratch LLMs you load, run, and serve with one pip install. It covers generative chat with tool calling, an agent in a sandbox (beta), embeddings / retrieval, typed, calibrated decisions ("System One", beta) and COBOL code specialists — 236M–1B class, runnable on a single GPU or CPU, no data leaving your machine.

Models live under Skyl4r-Ai on HuggingFace: Skylar-980M-Cobol · Skylar-980M-Cobol-Base · Skylar-980M-Cobol-Decider · Skylar-390M-Cobol · Skylar-236M-Base · Skylar-236M-Chat · Skylar-236M-Embed. Framework and training code: github.com/skyl4r-ai/skylar.

New in 0.7.10: documentation only (docstrings of the decision API); the code is that of 0.7.9.

New in 0.7.9: a server whose grammar engine is installed but fails to start (another version than the pin, an error at its warm-up) now starts anyway with structured output off and says why in its start line (until 0.7.8 it did not start); a tool marked "strict": true on a server without an engine is served unconstrained, with one line in the log, instead of a 400 (many clients send strict by default); on a constrained request <|im_end|> can no longer end the turn inside the <think> block; a schema the engine rejects, a malformed named tool_choice and concurrent compilations give a 400 instead of a 500. An answer cut by the loop-guard now ends with finish_reason: "length" both streaming and not (the stream said stop), as an answer cut by max_tokens. Importing skylar no longer touches the GPU (0.7.8: the hybrid kernels' library is loaded at the model's first use on CUDA).

New in 0.7.8: structured output in skylar serve (below): JSON mode, JSON Schema, strict tool calls, tool_choice required or a named function, Anthropic output_config.format, with llguidance or xgrammar. Every other request decodes as in 0.7.7.

New in 0.7.7: an opt-in fast engine for skylar serve on a GPU. --cuda-graph replays the decode step as a CUDA graph: on a 24 GB consumer GPU, a ~1B hybrid model in bf16 with a 4K-token context goes from 30 to 132 tokens/s with one request and from 204 to 748 tokens/s with eight. It uses a preallocated cache and other attention kernels, so in bf16 its answers are not token for token those of the default (the quality is the same), and past ~48K tokens of context it is slower than the default (2× at 128K): use it up to ~32K. --prefill-chunk N picks the same engine without the graph and reads long prompts in chunks of N tokens, so a 128K-token context fits a 24 GB GPU (~6 GiB for a ~1B model; with --cuda-graph the chunk is 8,192 unless given). Without either flag nothing changes: the default engine is 0.7.6's, token for token. The same in Python: the model's generate / generate_streaming (so Skylar.generate, Skylar.stream and skylar agent) keep the 0.7.6 decode unless you pass static_cache=True. The hybrid layers' short-convolution cache no longer keeps its whole input alive (less memory after a long prompt, same values). The default engine no longer builds the (prompt × prompt) prefill mask for a hybrid model like Skylar2, which never read it (it took gigabytes on a long prompt; same tokens). --fp32-head now works on CUDA: up to 0.7.6 it ran under the server's bf16 autocast, so on a GPU the head came out in bf16 and the option had no effect (on CPU it worked).

New in 0.7.6: skylar serve on CPU never goes through torch.compile (it needed a C++ compiler at run time: in a slim image every answer was a 500, while /health said ok) — eager was as fast; the server starts listening only after a test generation succeeded (else it exits with code 1), so /health ok means the model answers; --dtype auto on CPU now loads bf16 where the CPU computes it natively (AVX512-BF16/AMX: ~2× faster, same greedy text on our prompts), fp32 elsewhere, and the chosen dtype is always explicit (a bf16 checkpoint loads in fp32 with --dtype fp32); new --fp32-head: final logits in float32 over bf16 weights (off by default). Our model repos now keep bf16 as model.safetensors and the original fp32 as model.fp32.safetensors: --dtype fp32 (or Skylar.load(dtype=torch.float32)) loads that original, --dtype auto the bf16 main file, --variant picks one by hand. Skylar.load() without a dtype now loads the checkpoint as stored on every transformers version (4.x used to cast it to fp32).

New in 0.7.5: streaming decodes each new token on its own (a 32K-token answer reaches the client ~50 s sooner per row, with the same text byte for byte); a tool sent without description or parameters reaches the model in its training format; the embedder is the framework's own, ready for Skylar 2 embedders (the published 236M vectors are unchanged). Answers can be up to 32,768 tokens long (8,192 up to 0.7.3).

Install

pip install skylar                      # CLI + Python API
pip install "skylar[serve]"             # + the OpenAI- and Anthropic-compatible server
pip install "skylar[serve,hybrid]"      # Skylar 2 hybrid models on a CUDA GPU (Triton kernels of their linear-attention layers)

Python ≥ 3.10. Without a GPU the Skylar 2 hybrid layers run in plain PyTorch: correct, but slow on long inputs.

Use it — CLI

# chat with any Skylar generative model (no forced persona — steer it with --system)
skylar chat --model Skyl4r-Ai/Skylar-236M-Chat --system "Sei un assistente che risponde dal contesto."

# embeddings / retrieval (any SkylarEmbedder model)
skylar embed --model Skyl4r-Ai/Skylar-236M-Embed --query "prestito per comprare casa" \
  --docs "Il mutuo ipotecario è un finanziamento per l'acquisto di un immobile." "Domani il meteo prevede pioggia su tutto il nord Italia."

# typed, calibrated decisions on a state: what does this COBOL program print on this input?
#   state.json     {"cobol": "<program>", "stdin": "12\n30\n"}
#   questions.json {"output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print
#                   on the input in `stdin`?", "criteria": {"A": "0042", "B": "0041"}}}
skylar decide --model Skyl4r-Ai/Skylar-980M-Cobol-Decider --state-file state.json --questions questions.json

# one-shot generation (HF repo id or a local checkpoint dir)
skylar generate --model Skyl4r-Ai/Skylar-236M-Chat --prompt "..."

# the COBOL specialist — completes a COBOL stub into a full, compilable program
#   (auto-downloads Skylar-390M-Cobol; it's a stub completer, not a chatbot)
skylar cobol --example
skylar cobol --stub-file my_task.cbl --compile        # your own stub + GnuCOBOL check

# multi-user OpenAI-compatible server (needs the [serve] extra) — full details in "Serve it" below
skylar serve --model <any-skylar-model> --port 8000     # interactive docs at http://localhost:8000/docs

# an agent (beta): bash, read_file and write_file on a folder, inside a Docker sandbox
skylar agent --model <a tool-calling Skylar model> --workdir ./project "Write fizzbuzz.py, run it and check the output."

Decoding is greedy by default (--temperature 0.0) with no repetition penalty (1.0, as vLLM and 🤗 Transformers: a penalty also hits the prompt's tokens, so it works against copying from a document or a code skeleton); a loop-guard trims any runaway repetition. There is no forced system prompt — pass --system "..." to steer a chat model. (The skylar cobol subcommand handles the COBOL prompt format for you.)

skylar agent (beta)

The model gets a task and three tools — bash, read_file, write_file — on the folder you pass with --workdir. At every step it calls tools or answers; each call runs for real and its output goes back to the model, until it answers or --max-steps (default 20) ends. Commands run in a Docker container with no network, a read-only root and only your folder mounted on /work. The image (skylar-agent-sandbox:1: bash, Python, gcc, Java, Node, GnuCOBOL) is built from the Dockerfile shipped in the package the first time you run the agent. --sandbox local runs the commands on your machine instead, asking before each one (--yes to skip asking). --lang it|en sets the language of its instructions; --transcript run.json saves the conversation.

Use it — Python

import skylar

# generative chat — pass your own system prompt (no forced persona)
m = skylar.load("Skyl4r-Ai/Skylar-236M-Chat")          # HF repo id or a local dir
print(m.generate("Domanda: dove ha sede la Banca d'Italia?",
                 system="Rispondi solo dal contesto fornito."))
for delta in m.stream("..."):                          # streaming
    print(delta, end="", flush=True)

# embeddings / retrieval
e = skylar.load_embedder("Skyl4r-Ai/Skylar-236M-Embed")
ranked = e.rank("costo del denaro", ["la BCE alza i tassi", "ricetta pizza"])

# typed, calibrated decisions ("System One", beta): no text generated, probabilities you can trust
d = skylar.load_decider("Skyl4r-Ai/Skylar-980M-Cobol-Decider")
d.decide({"cobol": open("program.cbl").read(), "stdin": "12\n30\n"}, {
    "output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print on the input in `stdin`?",
               "criteria": {"A": "0042", "B": "0041", "C": "0084"}}})
# -> {"output": {"type": "choice", "probabilities": {"A": ..., "B": ..., "C": ...}, "choice": ..., "confidence": ...}}

# the COBOL specialist — a stub completer (not a chatbot)
c = skylar.load("Skyl4r-Ai/Skylar-390M-Cobol")
print(c.complete_cobol(my_stub))                       # -> full, compilable COBOL program

skylar also registers the architecture with 🤗 Transformers, so this works too:

import skylar  # registers skylar2 (and the earlier nano-transformer)
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Skyl4r-Ai/Skylar-236M-Chat")

Serve it — multi-user HTTP API (skylar serve)

skylar serve --model <id> turns any Skylar generative model into an OpenAI-compatible HTTP server built for concurrent users. Requests from many clients are fused into dynamic micro-batches on a single worker that owns the model — so one GPU (or CPU) serves a whole demo without per-request OOM or GPU contention — and each request can stream its tokens.

pip install "skylar[serve]"
skylar serve --model Skyl4r-Ai/Skylar-236M-Chat          # swap the id for ANY Skylar model
#  → http://127.0.0.1:8000   ·   interactive docs: http://127.0.0.1:8000/docs

Open /docs for the auto-generated Swagger UI — every endpoint, schema, and example is described there (or /redoc for ReDoc). The model is whatever you pass to --model (an HF repo id or a local checkpoint dir); an embedder model is auto-detected and served at /v1/embeddings instead.

Method & path	What it does
POST /v1/chat/completions	OpenAI chat format, function calling included (tools → tool_calls, parallel calls, tool turns). "stream": true → Server-Sent Events.
POST /v1/messages	Anthropic-compatible Messages API (same engine): system, content blocks, stop_sequences, thinking → <think> control, CoT returned as thinking blocks, native Anthropic SSE events.
POST /generate	One prompt → one completion.
GET /health	Liveness + which model/device is loaded.
GET /metrics	Throughput, batch sizes, queue depth.

# one-shot completion
curl localhost:8000/generate -H 'content-type: application/json' \
  -d '{"prompt": "Dove ha sede la Banca d'\''Italia?", "max_new_tokens": 64}'

# OpenAI chat format (+ "stream": true for SSE)
curl -N localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role":"system","content":"Sei un esperto COBOL."},
               {"role":"user","content":"Somma due campi PIC 9(4)."}],
  "max_tokens": 256, "stream": true
}'

Drop-in with the official OpenAI client — just point base_url at the server:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="Skyl4r-Ai/Skylar-236M-Chat",
    messages=[{"role": "user", "content": "Spiega cosa fa questo COBOL ..."}],
    stream=True,
)
for chunk in r:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Function calling works as with OpenAI: pass tools, get tool_calls back (finish_reason: "tool_calls"), run them, send the results as tool messages. The model sees the tools in the format it was trained on: every tool written as {"name", "description", "parameters"}, with an empty description and an object without properties when the request leaves them out (until 0.7.4 they were left out of the prompt too).

tools = [{"type": "function", "function": {"name": "get_weather", "description": "Current weather of a city",
          "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="skylar", tools=tools,
                                   messages=[{"role": "user", "content": "Che tempo fa a Torino?"}])
print(r.choices[0].message.tool_calls)  # [ChatCompletionMessageToolCall(function=Function(name='get_weather', arguments='{"city": "Torino"}'), ...)]

…or with the official Anthropic client (/v1/messages speaks the Anthropic wire format, streaming events included; tool use is not supported and returns an explicit 400):

import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="not-needed")
msg = client.messages.create(
    model="Skyl4r-Ai/Skylar-236M-Chat",
    max_tokens=256,
    system="Sei un esperto programmatore COBOL.",
    messages=[{"role": "user", "content": "Somma due campi PIC 9(4)."}],
    stop_sequences=["\n\n\n"],
)
print(msg.content[-1].text)          # a <think> CoT, if any, arrives as a thinking block

Structured output (0.7.8)

With a grammar engine installed, the server can guarantee the shape of an answer: while decoding, each token is drawn only among the tokens the grammar allows, and <|im_end|> is allowed only once the JSON is complete.

pip install "skylar[serve,grammar]"     # llguidance (MIT); or skylar[xgrammar] for xgrammar (Apache-2.0)

You send	You get
response_format: {"type": "json_object"}	a JSON object
response_format: {"type": "json_schema", "json_schema": {"name", "schema"}}	an object valid against the schema, keys in the schema's order
a tool with "strict": true	its calls carry arguments valid against its parameters; the name is always one of the tools
tool_choice: "required"	the turn is tool calls only (parallel_tool_calls: false = exactly one)
tool_choice: {"type": "function", "function": {"name": …}}	exactly one call of that tool
/v1/messages with output_config: {"format": {"type": "json_schema", "schema"}}	the text block is an object valid against the schema

Every example in this section was run against skylar serve 0.7.8 (the outputs in the comments are the real ones: a ~1B chat checkpoint, CPU, greedy). The formats are the ones the model was trained on: a call is <tool_call> + one JSON line + </tool_call>, and with json_schema the schema is also appended to the last user turn after one fixed line, as in the training data (send "schema_in_prompt": false for the grammar alone). An answer cut by max_tokens ends with finish_reason: "length" (stop_reason: "max_tokens") and is not a complete JSON: give it room. Without a grammar engine (or when the one installed fails to start: the start line says why) the server starts as before: a JSON answer and a required or named tool_choice get a 400, a tool marked strict is served unconstrained (strict needs skylar[grammar]); every other request decodes exactly as in 0.7.7.

curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role": "user", "content": "Estrai i dati: Fattura FT-118 del 3 novembre 2026, totale 1.525,00 euro."}],
  "max_tokens": 200,
  "response_format": {"type": "json_schema", "json_schema": {"name": "fattura", "schema": {
    "type": "object", "additionalProperties": false, "required": ["numero", "data", "totale"],
    "properties": {"numero": {"type": "string"},
                   "data": {"type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},
                   "totale": {"type": "number"}}}}}
}'
# → "content": "{\"numero\": \"FT-118\", \"data\": \"2026-11-03\", \"totale\": 1525.0}", "finish_reason": "stop"

from openai import OpenAI
from pydantic import BaseModel

class Invoice(BaseModel):
    number: str
    date: str
    total: float

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.parse(
    model="skylar", max_tokens=200, response_format=Invoice,
    messages=[{"role": "user", "content": "Extract the invoice: INV-7731 dated 2026-10-28, total $3,672.00."}])
print(r.choices[0].message.parsed)      # Invoice(number='INV-7731', date='2026-10-28', total=3672.0)

tools = [{"type": "function", "function": {"name": "get_weather", "strict": True, "parameters": {
    "type": "object", "additionalProperties": False, "required": ["city", "unit"],
    "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}]
r = client.chat.completions.create(model="skylar", tools=tools, tool_choice="required",
                                   messages=[{"role": "user", "content": "Che tempo fa a Torino?"}])
print(r.choices[0].message.tool_calls[0].function.arguments)   # {"city": "Torino", "unit": "celsius"}

import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="not-needed")
msg = client.messages.create(
    model="skylar", max_tokens=200,
    messages=[{"role": "user", "content": "Extract the invoice: INV-7731 dated 2026-10-28, total $3,672.00."}],
    output_config={"format": {"type": "json_schema", "schema": {
        "type": "object", "additionalProperties": False, "required": ["number", "date", "total"],
        "properties": {"number": {"type": "string"}, "date": {"type": "string"}, "total": {"type": "number"}}}}})
print(msg.content[0].text)              # {"number": "INV-7731", "date": "2026-10-28", "total": 3672.0}, indented by 2
# messages.parse(output_format=Invoice) and messages.stream(..., output_config=...) work the same way

Not yet: response_format together with tools (400: send one or the other), tools on /v1/messages, OpenAI allowed_tools / custom tools.

Tuning concurrency

Flag	Default	Meaning
--max-batch	8	Max requests fused into one forward pass. Raise for more throughput until VRAM/latency says stop.
--max-wait-ms	15	How long to wait for stragglers before launching a batch. Higher = bigger batches, slightly more latency.
--max-queue	256	Input-queue depth in requests; beyond it → 503.
--max-queue-tokens	auto	Queue cap in tokens (prompt+max_new) — a few long requests can't saturate the queue. 0 = auto (~2 full batches).
--dtype	auto	Weight dtype: auto (bf16 on GPU; on CPU bf16 if the CPU has native bf16 — AVX512-BF16 or AMX — else fp32) · bf16 · fp16 · fp32. bf16 ≈ half the weight memory and ~2× decode; the rotary frequencies and the decay parameters of the hybrid layers stay fp32. For CPU measurements comparable with earlier fp32 runs pass --dtype fp32.
--variant	—	Weights file next to the main one: fp32 = model.fp32.safetensors, main = always model.safetensors. Unset: the fp32 original when --dtype fp32 is asked and the repo has it, else the main file.
--grammar-backend	auto	Structured output engine: auto (llguidance, then xgrammar, whichever is installed) · llguidance · xgrammar · off. Without one (or if it fails to start), JSON answers and a required / named tool_choice get a 400, strict tools are served unconstrained, nothing else changes.
--fp32-head	off	Final logits in float32 from a float32 copy of the head, over bf16/fp16 weights (+ vocab × d_model × 4 bytes). It can help exact copying at long context; off by default. On CUDA it has an effect only from 0.7.7 (up to 0.7.6 the bf16 autocast turned it back to bf16).

How it works (for implementers / a future maintainer)

The server is skylar/serve.py — the model code (decoder.py / attention.py) is left untouched:

• One worker owns the model. Async routes enqueue requests; a single background thread pulls a micro-batch (up to --max-batch, waiting --max-wait-ms) and runs it. No two CUDA calls race, and there is exactly one set of KV-caches in flight. The input queue is bounded in tokens (--max-queue-tokens), so a flood of requests is shed with 503 rather than accepted blindly — though a single micro-batch of very long prompts can still pressure VRAM until the planned in-flight KV arena lands.
• True batched decoding. generate_batch() left-pads ragged prompts and builds a 4D additive mask (causal + pad) that Skylar2ForCausalLM.forward already accepts (its dense-mask SDPA path), so prompts of different lengths decode together with a shared KV-cache and per-row EOS stop. A Skylar 2 hybrid is prefilled one row at a time (a pad token must never enter a recurrent state), then decodes in the same batch. Per-row sampling mirrors Skylar2ForCausalLM.generate exactly → batched output is token-for-token identical to single-stream (proven by tests/test_batch_equiv.py).
• Batching + streaming coexist. Each request carries its own queue; the worker pushes text deltas into it as tokens are produced, so every request in a batch streams independently.
• Whole characters only. An emoji (or any character split over several byte-level tokens) is sent when its last byte arrives, never as «�»; a character left unfinished at the stop is dropped. The vocabulary's tokens that are «�» (learned from broken web text) are never sampled (0.7.1), nor the byte pieces that would assemble one, as «ï» then «¿½» (0.7.2).
• Streaming in linear time. Each token decodes only the ids after the last complete character (skylar.chatml.StreamDecoder, the prefix offset of vLLM's detokenizer); until 0.7.4 the whole answer was decoded again at every token, ~3 s more for an 8K-token answer and ~50 s for a 32K one, per row. The streamed text is the one of the whole answer decoded at once, byte for byte (tests/test_stream_decoder.py). (0.7.5)
• The loop-guard ignores separators. A run of symbols only (-----, =====, |---|) counts as a loop past 320 characters; words and phrases repeated 4 times past 30 characters are still cut. Until 0.7.2 a single 32-dash token (a COBOL comment line *-----…) stopped the stream. A cut keeps the first copy of the repeated unit. (0.7.3)
• Long answers. max_tokens (chat) and max_new_tokens (generate) accept up to 32,768 new tokens; until 0.7.3 the limit was 8,192. An agent that rewrites a whole COBOL member in one tool call (~7K tokens for a 500-line program) needs more than 8K. Nothing else changes: a request at or under 8,192 behaves as before. (0.7.4)
• Current limits (PoC). Static micro-batching (a batch starts and finishes together). For heavy, time-skewed load the next step is continuous batching (adding requests to an in-flight batch). One model per process; greedy is the default, sampling params are per-request.

python tests/test_batch_equiv.py     # run after touching batching/masking: batched == single-stream

Serve decisions — POST /v1/decide

skylar serve detects a decider and serves it instead of chat:

skylar serve --model Skyl4r-Ai/Skylar-980M-Cobol-Decider --port 8000
curl -s localhost:8000/v1/decide -H 'content-type: application/json' -d '{
  "state": {"cobol": "<program>", "stdin": "12\n30\n"},
  "questions": {"output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print on the input in `stdin`?",
                           "criteria": {"A": "0042", "B": "0041"}}}}'
# -> {"model": ..., "answers": {"output": {"type": "choice", "probabilities": {...}, "choice": ..., "confidence": ...}}}

All the questions about one state run in one batch. A decider answers the kind of questions it was trained on: the published one, on COBOL programs and their input (its card says which questions and how well). The confidence formulas: choice (p_max − 1/n)/(1 − 1/n), score 1 − Σ pᵢ|i − m| / MAD of the uniform rubric, noul |2p − 1|.

What's inside

The Skylar models use a custom decoder (Skylar2ForCausalLM, formerly NanoTransformer, Qwen3-style: RMSNorm + RoPE + GQA + QK-Norm + SwiGLU; Skylar 2 adds the KDA hybrid, Attention Residuals and GatedNorm), trained 100% from scratch (no third-party pretrained weights). This package vendors the framework's model code (kept identical to it) so the published weights load anywhere. Skylar 2 hybrid checkpoints use the Triton kernels on a CUDA GPU (pip install "skylar[hybrid]") and plain PyTorch elsewhere.

License

Apache-2.0 (code). Each model's license is on its model card. IP: A. Ivanovitch, CEO SKYL4R.