extralit-server 0.4.0

Open-source tool for accurate & fast scientific literature data extraction with LLM and human-in-the-loop.

Extralit Server

Extract structured data from scientific literature with human validation

This repository contains developer information about the backend server components. For general usage, please refer to our main repository or our documentation.

Source Code Structure

The server components are split into two main services:

/extralit_server
  /api # Core extraction API endpoints
    /handlers # FastAPI request handlers 
    /schemas # Data models and validation
    /services # Business logic services
    /utils # Helper utilities
  /ml # Machine learning components
    /extractors # Document extraction models
    /ocr # OCR processing
    /pipeline # Extraction pipeline orchestration
  /storage # Data persistence layer
    /models # Database models
    /search # Search engine integration
    /vector # Vector store

/argilla_server 
  /api # Annotation UI API endpoints
    /handlers
    /schemas 
  /models # Database models
  /auth # Authentication
  /tasks # Background jobs

Development Environment

The development environment uses Docker Compose to run all required services. Key commands:

# Start all services
docker-compose up -d

# Run server in dev mode
pdm run dev

# Run tests
pdm test

# Format and lint
pdm format
pdm lint

# Run all checks
pdm all

Key Components

FastAPI Servers

• Extraction Server: Handles document processing, extraction pipeline, and ML model serving
• Annotation Server: Manages UI, data validation workflow, and user collaboration

Databases

• PostgreSQL: Main database for user data, annotations, and metadata
• Elasticsearch: Vector store for semantic search and document indexing
• Weaviate: Vector database for table and section embeddings

Background Processing

Uses Celery for asynchronous tasks like:

• Document OCR and preprocessing
• ML model inference
• Batch extraction jobs
• Data export

CLI Commands

Key management commands:

# Database management
python -m extralit_server db migrate
python -m extralit_server db create-user

# Start servers
python -m extralit_server start
python -m argilla_server start

# Run workers
python -m extralit_server worker

See full CLI documentation in our developer docs.

Contributing

Check our contribution guide and join our Slack community.

Roadmap

See our development roadmap and share your ideas!