🧠 Company Brain MVP

Company Brain is an automated, offline-first knowledge extraction pipeline. It connects to 16 data sources across your company's communication, engineering, and analytics stack, pulls all the scattered information, runs it through a local AI model, and outputs structured "Skills" — machine-readable procedure cards that AI agents can directly execute.

Instead of manually writing instruction manuals for your AI tools, Company Brain generates them automatically by watching how your team communicates and documents things.

---

✨ What Does It Actually Do?

1. Reads Your Data: Securely connects to 16 data sources — Slack, Google Docs, Google Sheets, GitHub, Notion, Discord, Linear, Outlook/Teams, GitLab, Dropbox, Mixpanel, Amplitude, Algolia, Exa, Perplexity, and Facebook — ingesting raw text through a unified connector registry.
2. Understands Context: It breaks the text down and uses a local AI model (gemma4:e4b) to figure out what the text is actually about (e.g., "Is this a refund policy?" or "Is this a server deployment guide?").
3. Synthesizes "Skills": It groups related information together and writes step-by-step procedures, including decision points (if/then rules) and edge cases.
4. Exports for AI Agents: It outputs everything into a clean skills_file.json that you can plug directly into tools like LangChain, AutoGen, or your own custom AI bots so they know exactly how your company operates.

---

🛠️ Technical Architecture & Algorithms

For technical deep-dives, here is exactly how the pipeline operates under the hood:

1. Ingestion & Connectors (Data Layer)

• Abstract Base Connector: All 16 connectors implement BaseConnector, a shared abstract class that enforces a standard ingest(days_back) interface, provides retry_with_backoff() for exponential backoff on API rate limits (429/5xx errors), and checks CONNECTORS_ENABLED for dynamic toggling without code changes.
• Connector Registry: registry.py maintains a central list of all connector classes. get_enabled_connectors() instantiates each one and filters out unconfigured connectors (e.g., if GITHUB_TOKEN is missing, GitHubConnector silently skips). This means SyncScheduler never imports individual connectors — it only loops over whatever is enabled.
• Error Isolation: Each connector runs in a try/except block inside the sync loop. A bad token or an API outage on one connector logs an error and moves on — it will not crash the entire sync run.
• Idempotency: Before processing, the system calls get_processed_source_item_ids() which returns all item IDs already stored in processed_chunks. Any item already in that set is skipped. Running sync 10 times on the same data produces the same result as running it once.

2. Document Chunking Algorithm

• Semantic Sentence Splitting: Instead of naive character-count splitting (which breaks code blocks and sentences in half), the DocumentChunker uses regex-based sentence boundary detection.
• Sliding Window Overlap: Text is chunked into 5-sentence blocks with a 2-sentence overlap. This guarantees that context isn't lost across chunk boundaries, which is critical for accurate LLM extraction.

3. Knowledge Extraction (LLM Pipeline)

• Combined Single LLM Call: The KnowledgeExtractor sends a single optimised prompt that simultaneously classifies the chunk type (procedure, policy, decision, incident, general) AND extracts key concepts as a JSON array. This halves API latency compared to two sequential calls.
• Structured JSON Fallbacks: Because open-source LLMs can hallucinate formatting, the prompt enforces strict JSON output, which is then parsed using Python's built-in json library with regex fallbacks to strip out markdown code fences.

4. Skill Synthesis (Clustering & Generation)

• Context Window Assembly: The SkillsSynthesizer filters the database for high-confidence chunks (confidence_score >= 0.2) classified as actionable knowledge.
• Schema Enforcement: It asks the LLM to act as a technical writer, reading the raw concepts and synthesizing them into a strict domain model (Skill Pydantic class). This generates the final procedure steps, prerequisites, and edge cases.

5. UI Architecture (Event-Driven Dashboard)

• Decoupled State: The rich terminal dashboard runs independently of the backend pipeline.
• Log Interception: Instead of polluting SyncScheduler with UI logic, a custom Python logging.Handler intercepts backend logs (e.g., "Phase 1: Ingesting data"), updates the UI's internal state machine, and computes progress/ETA — the SyncScheduler never knows a UI exists.

---

⚙️ The Pipeline Workflow

graph TD;
    A[16 Data Sources via REST/GraphQL APIs] -->|BaseConnector.ingest| B(ConnectorRegistry)
    B -->|RawDataItem objects| C(SQLite: raw_data_items)
    C -->|Sliding Window Chunker| D{Local LLM: gemma4:e4b}
    D -->|Single combined prompt| E(SQLite: processed_chunks)
    E -->|Confidence filtering + concept clustering| F[LLM Synthesizer]
    F -->|Pydantic Skill schema| G((skills_file.json))

---

🚀 Setup Guide

1. Prerequisites

• Python 3.11+ installed on your machine.
• Ollama installed (Download from ollama.ai).
• API keys/tokens for whichever connectors you want to enable (see .env.example for the full list).

2. Prepare the AI Model

Open a terminal and download the required local AI model:

ollama pull gemma4:e4b
ollama serve  # Leave this running in the background!

3. Configure Credentials

Duplicate the environment template and add your API keys:

cp .env.example .env

Edit .env and fill in only the connectors you want to use. Unused connectors are automatically skipped if their credentials are absent. Key connectors:

• Slack: Create an app at api.slack.com/apps → paste your Bot Token as SLACK_BOT_TOKEN.
• Google Docs/Sheets: Create OAuth credentials at console.cloud.google.com → save as credentials.json.
• GitHub: Generate a Personal Access Token at github.com/settings/tokens → paste as GITHUB_TOKEN.
• Notion: Create an internal integration at notion.so/my-integrations → paste as NOTION_TOKEN (and share pages with the integration).
• Linear: Find your API key in Linear Settings → paste as LINEAR_API_KEY.
• See .env.example for all 16 connectors.

4. Install Dependencies

# Windows
python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt

---

💻 Usage & Commands

Run the CLI tool using the following commands from inside the project folder:

Run an Interactive Sync (Recommended)

Processes sources in small batches of 2 and pauses after each batch.

.\venv\Scripts\python.exe main.py sync --interactive

Run a Full One-Time Sync

Automatically processes everything in one go without stopping.

.\venv\Scripts\python.exe main.py sync --once

Run Continuous Background Sync

Automatically re-syncs every 30 minutes in the background.

.\venv\Scripts\python.exe main.py sync

Check Current Status

View a breakdown of skills generated, categories, and confidence scores.

.\venv\Scripts\python.exe main.py status

Export Your Skills

Export the final structured data so your AI agents can use it.

# Export as JSON (Best for AI Agents)
.\venv\Scripts\python.exe main.py export --output output/skills.json

# Export as Markdown (Best for Humans)
.\venv\Scripts\python.exe main.py export --output output/skills.md --format markdown

---

Company Brain — Complete Technical Reference

This document covers every aspect of the Company Brain project: what it does, how it works under the hood, every API used, every technology used, the full data pipeline, CLI commands, and the engineering decisions made. Written for YC interviews and technical deep-dives.

---

Table of Contents

1. What is Company Brain?
2. The Core Problem It Solves
3. High-Level Architecture
4. Technology Stack
5. Connectors — All 16 Data Sources
6. Full Data Pipeline — Step by Step
7. Database Design
8. The LLM Integration
9. CLI Commands Reference
10. Terminal Dashboard (UI)
11. Key Engineering Decisions
12. Directory Structure

---

1. What is Company Brain?

Company Brain is an offline-first knowledge extraction pipeline. It connects to 16 of your company's existing communication, engineering, and analytics tools, pulls all the scattered information, runs it through a local AI model, and outputs structured "Skills" — machine-readable procedure cards that AI agents can directly execute.

In plain English: Your team's knowledge lives in thousands of Slack messages, GitHub issues, Notion pages, and Linear tickets. Right now, no AI agent can act on that knowledge because it's buried in unstructured text across 16 different platforms. Company Brain reads all of it and converts it into clean, structured instructions.

---

2. The Core Problem It Solves

Without Company Brain	With Company Brain
AI agents have no idea how your company operates	AI agents get a skills_file.json with exact procedures
Building custom SOPs takes weeks of manual work	Company Brain auto-generates them by reading your existing docs
Institutional knowledge lives in people's heads / old Slack threads	It's extracted, structured, and searchable
Onboarding new hires is slow because no one knows where to find information	Skills are tagged by category, have prerequisites, and success criteria
Your knowledge is siloed across 16+ tools	A single unified sync pulls everything into one knowledge base

---

3. High-Level Architecture

┌───────────────────────────────────────────────────────────────────────────┐
│                              Data Sources (16)                             │
│  Slack  GitHub  Notion  Discord  Linear  Google Docs  Google Sheets        │
│  Outlook/Teams  GitLab  Dropbox  Mixpanel  Amplitude                      │
│  Algolia  Exa  Perplexity  Facebook                                        │
└────────────────────────────────┬──────────────────────────────────────────┘
                                 │  REST / GraphQL APIs
                                 ▼
┌───────────────────────────────────────────────────────────────────────────┐
│                     Connector Layer (BaseConnector)                        │
│                                                                            │
│  • Every connector inherits BaseConnector (get_source_name, ingest)        │
│  • registry.py: get_enabled_connectors() loops all 16, skips unconfigured  │
│  • retry_with_backoff() handles 429/5xx with exponential backoff           │
│  • Error isolation: one failing connector never blocks the others          │
│  • CONNECTORS_ENABLED env var toggles connectors without code changes      │
└────────────────────────────────┬──────────────────────────────────────────┘
                                 │  List[RawDataItem] (Pydantic model)
                                 ▼
┌───────────────────────────────────────────────────────────────────────────┐
│                          SQLite Database                                   │
│                      (via SQLAlchemy ORM)                                  │
│   Table: raw_data_items     Table: processed_chunks     Table: skills      │
└────────────────────────────────┬──────────────────────────────────────────┘
                                 │
                                 ▼
┌───────────────────────────────────────────────────────────────────────────┐
│                         Processing Pipeline                                │
│                                                                            │
│  1. DocumentChunker  → sentence-based sliding window (5 sentences,         │
│     2-sentence overlap), URL/email masking, noise filtering                │
│                                                                            │
│  2. KnowledgeExtractor → single optimised prompt to gemma4:e4b via Ollama  │
│     - Classifies chunk type (procedure/policy/decision/incident/general)   │
│     - Extracts key concepts as JSON array                                  │
│     - Computes heuristic confidence score (0.0 – 1.0)                     │
│                                                                            │
│  3. SkillsSynthesizer → filters high-confidence chunks, clusters by        │
│     primary concept, asks Gemma to write a complete Skill card             │
│     (steps, decisions, prerequisites, edge cases)                          │
└────────────────────────────────┬──────────────────────────────────────────┘
                                 │
                                 ▼
┌───────────────────────────────────────────────────────────────────────────┐
│                            Export Layer                                    │
│              skills_file.json  /  skills_file.md                          │
│       (ready to plug into LangChain, AutoGen, or custom bots)             │
└───────────────────────────────────────────────────────────────────────────┘

---

4. Technology Stack

Category	Library / Tool	Why We Used It
Language	Python 3.11	Mature ecosystem for AI/NLP pipelines
Data Validation	pydantic v2	Enforces strict schema for every data model
Database ORM	sqlalchemy v2	Maps Python classes to SQLite tables cleanly
Database	SQLite	Zero-config local persistence, no server needed
HTTP Client	requests	Used by all new connectors for REST/GraphQL API calls
Slack Integration	slack-sdk v3	Official Slack client; handles pagination & auth
Google Integration	google-api-python-client, google-auth-oauthlib	Shared OAuth 2.0 flow for both Google Docs and Google Sheets
Local AI	ollama (Python client)	Runs Gemma 4 locally; zero data sent externally
CLI Framework	click	Clean, composable command-line interface
Terminal UI	rich	Beautiful dashboard with live animations, progress bars
Background Scheduler	apscheduler	Runs sync jobs on a 30-minute interval
Env Management	python-dotenv	Loads .env credentials without hardcoding secrets

---

5. Connectors — All 16 Data Sources

Company Brain uses a pluggable connector architecture. All connectors inherit from BaseConnector and are registered in registry.py. The sync loop dynamically loads only the ones with valid credentials configured in .env.

Tier 1 — Core Business Knowledge

Connector	Auth	What It Ingests	Dedup Key
Slack	Bot Token	Messages, threads, replies from all joined channels	slack:{channel_id}:{ts}
Google Docs	OAuth 2.0	Document paragraph text, headers, table cell contents	gdoc:{doc_id}
Google Sheets	OAuth 2.0 (reused)	Spreadsheet tabs as structured Header: Value row text	gsheet:{spreadsheet_id}:{sheet_name}
GitHub	Personal Access Token	Issues, PR descriptions, labels, review comments	github:{repo}:{issue_number}
Notion	Internal Integration Token	Page block hierarchies: headings, lists, quotes, code	notion:{page_id}
Discord	Bot Token	Server text channels and message history	discord:{channel_id}:{message_id}
Linear	API Key	Issues, descriptions, team context, comment threads (GraphQL)	linear:{issue_id}
Outlook / Teams	Azure OAuth 2.0 (MS Graph)	Outlook email threads + Teams channel messages	outlook:{message_id} / teams:{channel_id}:{message_id}

Tier 2 — Extended Sources

Connector	Auth	What It Ingests	Dedup Key
GitLab	Personal Access Token	Projects, issues, merge requests, issue notes	gitlab:{project}:{type}:{iid}
Dropbox	OAuth Access Token	Text files (.md, .txt, .json, .csv, .doc) from shared folders	dropbox:{file_id}
Mixpanel	API Secret	Tracked event schema catalog	mixpanel:event_schema:{project_id}
Amplitude	API Key + Secret	Event taxonomy definitions and descriptions	amplitude:taxonomy:{key}
Algolia	App ID + API Key	Indexed search records from a named index	algolia:{index}:{objectID}
Exa	API Key	Web research results for configured search queries	exa:{hash(query)}
Perplexity	API Key	AI search completions for configured prompts	perplexity:{hash(prompt)}
Facebook	Page Access Token	Page posts and customer comment threads	facebook:{post_id}

How to Enable/Disable Connectors

Edit the CONNECTORS_ENABLED variable in your .env:

# Enable only what you use — unconfigured connectors are silently skipped
CONNECTORS_ENABLED=slack,google_docs,github,notion,linear

Leave it empty to attempt all connectors (only those with valid credentials will run).

---

6. Full Data Pipeline — Step by Step

Step 1: Ingestion

The SyncScheduler calls get_enabled_connectors() from registry.py. This returns a list of instantiated connectors whose credentials are present. For each connector, ingest(days_back=30) is called and the returned RawDataItem objects are collected.

# Every piece of data becomes this shape regardless of source
class RawDataItem(BaseModel):
    id: str              # e.g., "github:owner/repo:42" or "notion:abc123"
    source: DataSourceType   # enum: "slack", "github", "notion", etc.
    source_id: str       # platform-native ID (channel ID, doc ID, issue ID)
    title: str           # human-readable display name
    content: str         # full normalized text
    author: str          # username or display name
    created_at: datetime
    updated_at: datetime
    raw_metadata: dict   # source-specific extras (labels, state, channel, etc.)

Idempotency check: Before processing, the system calls get_processed_source_item_ids() on the database. This returns the set of all item IDs already processed into chunks. The pipeline filters out any item whose ID is already in this set. This means:

• You can stop the sync halfway through and resume exactly where you left off.
• Running sync again never re-processes already-seen items.

---

Step 2: Chunking — DocumentChunker

Long documents are split into smaller, manageable pieces before being sent to the LLM.

Text Cleaning (always runs first):

- Collapses multiple whitespace characters into single spaces
- Strips control characters (ASCII 0-31, 127-159)
- Replaces URLs with the token [URL]
- Replaces email addresses with the token [EMAIL]

Sliding-Window Sentence Chunking:

• Default chunk size: 5 sentences
• Default overlap: 2 sentences (retains tail context from the previous chunk)
• Uses regex (?<=[.!?])\s+ to split on sentence boundaries (not mid-sentence)
• Chunks shorter than 20 characters are discarded

Why overlap? Without overlap, if a procedure starts at the end of one chunk and continues at the start of the next, the LLM would only see half the context. The overlap ensures key context is preserved across boundaries.

---

Step 3: Knowledge Extraction — KnowledgeExtractor

For every chunk, the extractor makes a single combined LLM call (optimised from two separate calls):

Prompt: "Classify this text as ONE of: procedure, decision, incident, policy, or general.
         Then extract 3-5 key concepts. Return as JSON: { type: ..., concepts: [...] }"

Output: { "type": "procedure", "concepts": ["Handle Refunds", "Verify order status", "Payment processor"] }

Confidence Scoring (no LLM call, pure heuristic):

+0.3  if chunk is longer than 100 words
+0.2  if chunk is 50–100 words
+0.3  if classified as "procedure", "policy", or "decision"
+0.2  if 4+ key concepts were extracted
+0.1  if 2–3 key concepts were extracted
Max score: 1.0

Every processed chunk becomes a ProcessedChunk object and is saved to the database.

---

Step 4: Skill Synthesis — SkillsSynthesizer

This is where the real magic happens.

Phase A — Filtering: Only chunks with confidence_score >= 0.2 move forward.

Phase B — Clustering: Chunks are grouped by their first (primary) key concept. For example, all chunks where the first concept is "Handle Refunds" end up in the same cluster. Clusters with fewer than 2 chunks get merged into a miscellaneous cluster.

Phase C — Skill Generation: For each cluster, the LLM is given all the chunk texts combined and asked to synthesize a complete Skill card:

{
  "name": "clear skill name",
  "description": "what this skill does",
  "category": "refunds / pricing / incidents / ...",
  "procedure_steps": ["Step 1", "Step 2", "..."],
  "decision_points": {"if customer > 30 days": "deny refund"},
  "prerequisites": ["who can perform this", "required access"],
  "success_criteria": ["how to verify completion"],
  "exceptions": ["edge cases to watch for"]
}

The final Skill object is upserted (insert or update) into the SQLite skills table.

---

Step 5: Export

Running python main.py export serializes all skills from the database into a single SkillsFile object and writes it to disk.

{
  "version": "1.0.0",
  "generated_at": "2026-07-25T00:00:00",
  "company_name": "Your Company",
  "skills": [
    {
      "id": "skill-uuid-...",
      "name": "Handle Customer Refunds",
      "description": "...",
      "category": "refunds",
      "procedure_steps": ["Check eligibility", "Verify order", "Issue refund"],
      "decision_points": {"if_disputed": "escalate to manager"},
      "examples": [],
      "prerequisites": ["Customer support access"],
      "success_criteria": ["Refund confirmation sent"],
      "exceptions_and_edge_cases": ["Active subscriptions need billing cancel"],
      "source_items": ["github:myorg/repo:42", "slack-C01-123...", "notion:abc123"],
      "confidence_score": 0.75
    }
  ],
  "metadata": {
    "total_skills": 1,
    "by_category": {"refunds": 1}
  }
}

---

7. Database Design

The system uses SQLite with SQLAlchemy ORM. The database lives at data/company_brain.db (path configurable via DATABASE_URL in .env).

3 Tables:

raw_data_items
├── id (PK)          — unique ID e.g. "github:myorg/repo:42", "notion:abc123"
├── source           — enum: "slack", "github", "notion", "linear", etc.
├── source_id        — platform-native ID (channel ID, doc ID, issue ID)
├── title            — display name
├── content          — full raw text
├── author           — username or display name
├── created_at
├── updated_at
└── raw_metadata     — JSON blob with source-specific fields (labels, state, etc.)

processed_chunks
├── id (PK)          — "chunk-{uuid}"
├── source_item_id   — FK to raw_data_items.id
├── chunk_text       — the actual text block
├── chunk_index      — position within source document
├── key_concepts     — JSON list e.g. ["Refunds", "Policy"]
├── chunk_type       — "procedure", "policy", "decision", "incident", "general"
├── confidence_score — float 0.0 to 1.0
└── created_at

skills
├── id (PK)          — "skill-{uuid}"
├── name             — e.g. "Handle Customer Refunds"
├── description
├── category         — e.g. "refunds"
├── procedure_steps  — JSON list
├── decision_points  — JSON dict
├── examples         — JSON list
├── prerequisites    — JSON list
├── success_criteria — JSON list
├── exceptions_and_edge_cases — JSON list
├── source_items     — JSON list of contributing raw_data_items IDs
├── last_updated
└── confidence_score

---

8. The LLM Integration

Model: gemma4:e4b — Gemma 4 with 4 billion parameters (E4B = Efficient 4B). Always use this model; do not swap to Mistral.

How it runs: Ollama is a lightweight server that runs locally on your machine. You install it once (ollama pull gemma4:e4b), and then the Python code communicates with it via the ollama Python client on localhost:11434.

The ollama client (not raw HTTP):

import ollama
client = ollama.Client()
response = client.generate(model="gemma4:e4b", prompt="...", stream=False)
result = response["response"]

JSON Parsing Robustness: Because LLMs sometimes wrap JSON in markdown code fences (like ```json ... ```), we use a regex fallback:

json_match = re.search(r'\[.*?\]', result_text, re.DOTALL)  # for arrays
json_match = re.search(r'\{.*\}', result_text, re.DOTALL)   # for objects
if json_match:
    data = json.loads(json_match.group())

Why Gemma 4? Outperformed Mistral in adhering to JSON schemas during testing. Smaller footprint than 7B/13B models while producing more structured outputs.

---

9. CLI Commands Reference

All commands are run from inside the project root directory using the venv Python interpreter.

# Start interactive batch sync (RECOMMENDED)
# Processes 2 items at a time, asks whether to continue after each batch
.\venv\Scripts\python.exe main.py sync --interactive

# Run a single full sync (processes everything, no stops)
.\venv\Scripts\python.exe main.py sync --once

# Run continuous auto-sync (syncs every 30 minutes in the background)
.\venv\Scripts\python.exe main.py sync

# Check current database status: skill count, categories, confidence
.\venv\Scripts\python.exe main.py status

# Export skills as JSON (for AI agents)
.\venv\Scripts\python.exe main.py export --output output/skills.json

# Export skills as readable Markdown (for humans)
.\venv\Scripts\python.exe main.py export --output output/skills.md --format markdown

# Disable the Rich dashboard UI (useful for piping output to logs)
.\venv\Scripts\python.exe main.py sync --once --plain

Interactive Batch Menu (appears after each batch of 2 documents):

[Batch 1 complete. 5 total chunks processed so far.]
Do you want to: (1) Process next batch (2) Synthesize skills now and STOP (3) Quit immediately?

• 1 → Continue to next 2 items
• 2 → Stop ingesting new data, run synthesis on what you have right now, and export
• 3 → Exit immediately without synthesizing

---

10. Terminal Dashboard (UI)

The terminal UI is built with the rich library and runs inside a rich.live.Live context. This means the dashboard panel stays fixed at the top of the terminal while log messages scroll underneath it.

Key components:

• SyncState — A plain Python class that holds the current progress percentage, file count, skill count, task statuses, and elapsed time.
• SyncDashboard — A custom __rich_console__ renderable that reads from SyncState and draws the panel using rich.panel.Panel, rich.console.Group, and rich.progress.Progress.
• ProceduralNetwork — The animated neural network animation. It's NOT pre-built frames. It uses math.sin(time.time() * 3 + offset) to procedurally compute the brightness/state of each node and edge in real time at ~10 FPS.
• DashboardLogHandler — A custom Python logging.Handler that intercepts log messages from the backend (like "Phase 1: Ingesting data") and maps them to UI state updates (progress bar %, which task is active). This is how the backend and UI stay decoupled — the SyncScheduler never knows a UI exists.

---

11. Key Engineering Decisions

Why an Abstract Base Connector + Registry pattern?

With 16 connectors, hardcoding each one into SyncScheduler would create a monolithic, hard-to-maintain sync loop. Instead, BaseConnector enforces a uniform ingest() interface, and registry.py acts as a plugin system. Adding a new connector is three steps: write the class, register it in registry.py, add credentials to .env.example. The scheduler code never changes.

Why SQLite instead of a cloud database?

Privacy-first design. Company data stays on-premise. SQLite also requires zero configuration, which makes setup trivial. The path is configurable via DATABASE_URL if you want to swap in PostgreSQL for production.

Why local LLM (Ollama) instead of OpenAI?

Companies have confidential Slack data, GitHub issues, emails. Sending it to a third-party API is a major legal and trust risk. Ollama + Gemma 4 gives equivalent results while keeping everything local. This is a core value proposition: "your data, your machine, your AI."

Why a single combined LLM call instead of two?

The original design made two LLM calls per chunk: one for classification, one for concept extraction. This was optimised to a single JSON-returning prompt that does both simultaneously, cutting per-chunk latency roughly in half without sacrificing output quality.

Why a sliding window chunker instead of splitting by headers?

We also implemented chunk_by_structure() (splits by # headers). But most Slack messages and informal docs don't have formal headers. The sentence-based sliding window handles messy, unstructured text far better across all 16 source types.

Why concept-overlap clustering instead of embeddings/vector search?

For an MVP, cosine similarity over embeddings requires storing large float arrays and running nearest-neighbor search. Instead, we cluster purely by the primary concept extracted by the LLM. It's O(N) instead of O(N²) and produces very interpretable clusters. Embedding-based clustering is a listed future improvement (embedding field already exists on ProcessedChunk, it's just null for now).

Why the interactive batch system?

Running a full sync across 16 connectors through a local 4B model can take a long time. Users need to be able to stop, inspect partial results, and decide whether to continue. The batch system also lets users validate quality early without committing to the full pipeline run.

Idempotency

Every raw item has a globally unique ID (e.g., github:myorg/repo:42, notion:abc123, slack-C05-1234). Before any processing, we query processed IDs from the DB and filter them out. Running sync 10 times on the same data produces the same result as running it once.

---

12. Directory Structure

company_brain_mvp/
│
├── main.py                              ← CLI entry point (self-relative sys.path setup)
├── requirements.txt                     ← All Python dependencies
├── .env                                 ← Your API keys (gitignored)
├── .env.example                         ← Template for .env with all 16 connectors
├── credentials.json                     ← Google OAuth credentials (gitignored)
├── token.pickle                         ← Saved Google OAuth token (gitignored)
├── test_integration.py                  ← Quick smoke test (chunker + LLM + synthesizer)
│
├── src/
│   ├── connectors/
│   │   ├── base_connector.py            ← Abstract BaseConnector: ingest(), retry_with_backoff()
│   │   ├── registry.py                  ← Central connector registry: get_enabled_connectors()
│   │   │
│   │   │   ── Tier 1 Connectors ──
│   │   ├── slack_connector.py           ← Slack: messages, thread replies
│   │   ├── google_docs_connector.py     ← Google Docs: paragraph text, tables
│   │   ├── github_connector.py          ← GitHub: issues, PRs, comments (REST API v3)
│   │   ├── notion_connector.py          ← Notion: page blocks, recursive children
│   │   ├── discord_connector.py         ← Discord: server channels, messages
│   │   ├── google_sheets_connector.py   ← Google Sheets: tabs as key-value row text
│   │   ├── linear_connector.py          ← Linear: issues, comments (GraphQL API)
│   │   ├── outlook_teams_connector.py   ← Outlook emails + Teams messages (MS Graph)
│   │   │
│   │   │   ── Tier 2 Connectors ──
│   │   ├── gitlab_connector.py          ← GitLab: issues, merge requests, notes
│   │   ├── dropbox_connector.py         ← Dropbox: text file downloads
│   │   ├── analytics_connector.py       ← Mixpanel + Amplitude: event schemas
│   │   ├── search_connector.py          ← Algolia + Exa + Perplexity: search results
│   │   └── facebook_connector.py        ← Facebook: page posts, comments
│   │
│   ├── processors/
│   │   ├── chunker.py                   ← Sentence-based sliding window chunker
│   │   ├── knowledge_extractor.py       ← gemma4:e4b: single combined classify+extract call
│   │   └── skills_synthesizer.py        ← Cluster chunks → generate Skill cards
│   │
│   ├── models/
│   │   └── domain.py                    ← Pydantic models: RawDataItem, ProcessedChunk, Skill
│   │                                       DataSourceType enum (all 19 source types)
│   │
│   ├── storage/
│   │   ├── database.py                  ← SQLAlchemy table definitions (3 tables)
│   │   └── storage_manager.py           ← CRUD: save_raw_items, save_chunks, get_processed_ids
│   │
│   ├── sync/
│   │   └── sync_scheduler.py            ← Orchestrates full & interactive sync via registry
│   │
│   └── cli/
│       ├── main.py                      ← Click commands: sync, export, status
│       └── ui/
│           ├── animation.py             ← Procedural neural network animation
│           ├── dashboard.py             ← Rich Live dashboard (SyncDashboard, SyncState)
│           ├── logger.py                ← Custom logging.Handler → UI state bridge
│           └── components.py            ← Rich tables/panels for status & export
│
├── tests/
│   ├── test_github_connector.py         ← GitHub connector unit tests (3 tests)
│   ├── test_notion_connector.py         ← Notion connector unit tests (3 tests)
│   ├── test_discord_connector.py        ← Discord connector unit tests (3 tests)
│   ├── test_google_sheets_connector.py  ← Google Sheets connector unit tests (3 tests)
│   ├── test_linear_connector.py         ← Linear connector unit tests (3 tests)
│   ├── test_outlook_teams_connector.py  ← Outlook/Teams connector unit tests (3 tests)
│   ├── test_gitlab_connector.py         ← GitLab connector unit tests (3 tests)
│   ├── test_dropbox_connector.py        ← Dropbox connector unit tests (3 tests)
│   ├── test_analytics_connector.py      ← Mixpanel + Amplitude unit tests (2 tests)
│   ├── test_search_connector.py         ← Algolia + Exa + Perplexity unit tests (3 tests)
│   └── test_facebook_connector.py       ← Facebook connector unit tests (3 tests)
│
├── data/
│   └── company_brain.db                 ← SQLite database (auto-created on first run)
│
└── output/
    ├── skills_file.json                 ← Final structured output for AI agents
    └── skills_file.md                   ← Human-readable version