Speak Fill

Voice-dictated form filling — build a form, then fill it out by talking. Speech is transcribed locally and in real time by faster-whisper (no cloud STT service), and an LLM maps what you said onto the right form fields.

It's a local, single-user tool: no accounts, no login, no cloud dependency beyond an optional LLM call.

Install

pip install speakfill

Set your API key

Whole-form dictation (see "How it works" below) uses OpenAI (gpt-4o-mini) to map your transcript onto form fields, so it needs OPENAI_API_KEY. Set it one of three ways — checked in this order, first match wins:

1. A real environment variable — simplest for one-off use:

   export OPENAI_API_KEY=sk-...
   speakfill
   

2. A .env file in your project directory (or any parent of it — same lookup git uses):

   echo "OPENAI_API_KEY=sk-..." > .env
   speakfill
   

3. A .env file at ~/.speakfill/.env — set it once, then run speakfill from anywhere without repeating it:

   mkdir -p ~/.speakfill
   echo "OPENAI_API_KEY=sk-..." > ~/.speakfill/.env
   speakfill    # works from any directory now
   

No key at all? Per-field dictation still works fully (it never calls an LLM), and speech-to-text itself never needs an API key either way — only whole-form dictation's field-mapping step does. Or skip OpenAI entirely with LLM_PROVIDER=local (see Extras below).

Run it

speakfill

Speak Fill starting at http://0.0.0.0:8000
INFO:     Uvicorn running on http://0.0.0.0:8000

Open http://localhost:8000. One process serves both the API and the UI — no Docker, no separate frontend server. Data is stored locally in SQLite at ~/.speakfill/speakfill.db, created automatically on first run.

Tutorial: your first form

1. Open http://localhost:8000 — you land on Build, a drag-and-drop form builder.
2. Add a few fields (e.g. a text input for "Name", a number input for "Age", a dropdown for "Department"). Save the form.
3. Switch to My Forms, find the form you just saved, and open it to fill it out.
4. Fill it by voice, in one of two modes:
   • Per-field dictation — click the mic on a single field, speak, the transcript becomes that field's value directly. No API key needed.
   • Whole-form dictation — click the mic on the whole-form recorder, speak freely covering everything the form asks for (e.g. "My name is Alex, I'm 30, and I work in Engineering"), stop recording, and the LLM maps what you said onto the right fields in one shot. Needs OPENAI_API_KEY (or LLM_PROVIDER=local).
5. Submit. Switch to Submissions to see everything you've filled out so far.

Extras

• pip install speakfill[postgres] — use Postgres instead of the default SQLite (DATABASE_URL=postgresql+psycopg://...)
• pip install speakfill[local-llm] — run whole-form field-mapping locally instead of via OpenAI (LLM_PROVIDER=local, LLM_LOCAL_MODEL=Qwen/Qwen2.5-7B-Instruct by default); pulls in torch/transformers, a multi-GB install, so it's opt-in

Using it inside your own project

speakfill isn't only a standalone app — it's a normal importable package, so you can pull pieces of it into an existing FastAPI project instead of running speakfill as its own process.

Mount the whole app under a prefix, API + bundled UI together:

from fastapi import FastAPI
from speakfill import app as speakfill_app

app = FastAPI()
app.mount("/speakfill", speakfill_app)

Everything comes with it at /speakfill/... — forms API, dictation WebSocket, and the UI (Base.metadata.create_all() and the faster-whisper warm-up run automatically, same as standalone). One thing to know: speakfill_app has CORSMiddleware(allow_origins=["*"]) baked in — fine for most embeds, but if you need different CORS behavior, use the next pattern instead.

Or cherry-pick just what you need — e.g. only the forms API, none of our CORS/UI/other routes:

from fastapi import FastAPI
from speakfill.api.forms.routes import app as forms_router
from speakfill.database.session import Base, engine

Base.metadata.create_all(bind=engine)  # creates the template_forms/filled_forms tables

app = FastAPI()  # your own app, your own CORS/middleware/other routes
app.include_router(forms_router)

Speech-to-text and field-mapping work as plain library calls too, independent of any web framework or the forms API above — speakfill.stt.whisper_engine.transcribe_pcm16(audio_bytes) and speakfill.llm.llm_client.LLMClient().fill_form(form_schema, transcript) — if you just want the transcription/mapping logic with no HTTP layer at all.

Note that every piece — the DB engine, faster-whisper model, LLM client — reads its config from the same Settings object (speakfill.config.settings), so it follows whatever OPENAI_API_KEY/DATABASE_URL/etc. your host process already has set, the same way as standalone.

Reading/writing data without any web layer at all

For the common case of "I just want the data as Python objects" — no FastAPI, no HTTP, no Session to manage — speakfill re-exports a handful of plain functions that open/close their own database session per call:

import speakfill

speakfill.list_forms()                # -> [{"id", "title", "json_data" (its fields), "created_at", "updated_at"}, ...]
speakfill.get_form(form_id)            # -> dict, or None if it doesn't exist
speakfill.list_filled_forms()          # same shape, for submissions
speakfill.get_filled_form(filled_id)

speakfill.transcribe(audio_bytes)      # 16kHz mono PCM16 -> str (one-shot; for live/streaming
                                        # dictation, use speakfill.stt.session.WhisperSession instead)
speakfill.map_transcript(form_schema, transcript)  # -> {field_key: value}, same call Mode B uses

form_schema is the json_data["fields"] list from a form returned by get_form/list_forms — the same field definitions (element/field_name/label/options/...) react-form-builder2 produces, unchanged, so it doubles as documentation of what a "form" actually contains if you're building something on top of it.

Customizing the UI

The bundled frontend is a normal static build (index.html + assets) — point SPEAKFILL_STATIC_DIR at a different one to serve that instead:

SPEAKFILL_STATIC_DIR=/path/to/your/build speakfill

Setting it to a path that doesn't exist fails at startup rather than silently falling back, so a typo doesn't quietly serve the wrong UI. This is the mechanism for both "reskin it" (fork the frontend, restyle theme.css's CSS custom properties, rebuild) and "add new form field types" below — build your version, point speakfill at it, done. The API itself needs no changes either way (it's a plain REST + WebSocket API, wide-open CORS by default — see ARCHITECTURE.md).

Adding custom form field types

New field types (like the built-in MicTextInput/MicTextArea) plug into the drag-and-drop builder (react-form-builder2) two ways, both in the frontend source (frontend/src/form-elements/):

1. Register the component so saved forms know how to render it:

   import { Registry } from 'react-form-builder2';
   import { MyField } from './MyField';
   
   Registry.register('MyField', MyField);
   

2. Add it to the builder's toolbar (frontend/src/FormBuilderPage.tsx) so it's actually draggable onto the canvas:

   {
     key: 'MyField',
     element: 'CustomElement',
     component: MyField,
     type: 'custom',
     forwardRef: true,
     field_name: 'my_field_',
     name: 'My Field',       // shown in the builder's palette
     icon: 'fa fa-star',
   }
   

Your component receives the same props any react-form-builder2 CustomElement gets: name, defaultValue, disabled, and the raw field data. See frontend/src/form-elements/MicFields.tsx for a complete working example (it also shows how a field hooks into the dictation system, if that's relevant to what you're building — most custom fields won't need any of that).

Then rebuild (./scripts/build_frontend.sh from the repo, or your own npm run build) and point SPEAKFILL_STATIC_DIR at the output.

Other settings

All optional, set the same way as OPENAI_API_KEY above:

Links

Full documentation, architecture notes, and source: https://github.com/mohpezeshki/speak-fill