Extralit Server

Extract structured data from scientific literature with human validation

This repository contains developer information about the backend server components. For general usage, please refer to our main repository or our documentation.

Source Code Structure

The server components are split into two main services:

/extralit_server
  /api # Core extraction API endpoints
    /handlers # FastAPI request handlers
    /schemas # Data models and validation
    /services # Business logic services
    /utils # Helper utilities
  /ml # Machine learning components
    /extractors # Document extraction models
    /ocr # OCR processing
    /pipeline # Extraction pipeline orchestration
  /storage # Data persistence layer
    /models # Database models
    /search # Search engine integration
    /vector # Vector store

/extralit_server
  /api # Annotation UI API endpoints
    /handlers
    /schemas
  /models # Database models
  /auth # Authentication
  /tasks # Background jobs

Development Environment

The development environment uses Docker Compose to run all required services. Key commands:

# Start all services
docker-compose up -d

# Install dependencies
uv sync --dev

# Run server in dev mode
uv run uvicorn extralit_server:app --host 0.0.0.0 --port 6900 --reload

# Run tests
uv run pytest tests

# Format and lint
uv run ruff format
uv run ruff check

Key Components

FastAPI Servers

• Extraction Server: Handles document processing, extraction pipeline, and ML model serving
• Annotation Server: Manages UI, data validation workflow, and user collaboration

Databases

• PostgreSQL: Main database for user data, annotations, and metadata
• Elasticsearch: Vector store for semantic search and document indexing
• Weaviate: Vector database for table and section embeddings

Background Processing

Uses Celery for asynchronous tasks like:

• Document OCR and preprocessing
• ML model inference
• Batch extraction jobs
• Data export

CLI Commands

Key management commands:

# Database management
python -m extralit_server db migrate
python -m extralit_server db create-user

# Start servers
python -m extralit_server start
python -m extralit_server start

# Run workers
python -m extralit_server worker

See full CLI documentation in our developer docs.

Running Tests

The pytest suite is primarily designed to run in the CI environment using GitHub Actions as defined in .github/workflows/extralit-server.yml. This workflow sets up the necessary dependencies including Elasticsearch, PostgreSQL, Redis, and Minio.

Note that some tests are specifically skipped when running locally due to differences between the CI environment and local development environments. These tests may involve:

• Search engine dynamics (Elasticsearch/OpenSearch compatibility)
• File storage operations with Minio
• Authentication and permission checks

To run tests in CI, create a pull request to trigger the test workflow.

If you need to run a specific test locally for debugging purposes, you can use:

cd extralit-server
python -m pytest [test_path] -v

However, expect some tests to fail or be skipped when running locally.

Contributing

Check our contribution guide and join our Slack community.

Roadmap

See our development roadmap and share your ideas!