fenliu 0.5.2

Monitor and filter Fediverse hashtags, curate quality content, and distribute via external tools like Zhongli

FenLiu (分流)

Created by marvin8 with assistance from DeepSeek AI assistant.

⚠️ DISCLAIMER / PROVISO: This project is a work in progress with major changes still happening. It is in no way anywhere close to finished and is only borderline useful for actual production use. Expect breaking changes, incomplete features, and significant architectural evolution as development continues.

Divide the Fediverse content flow

FenLiu is a web application that monitors Fediverse hashtags, filters spam, allows human review, learns from feedback, and exports quality content for boosting. Inspired by the ancient Chinese Dujiangyan irrigation system (256 BC) that separated silt from water, FenLiu applies 2,300-year engineering wisdom to modern digital content streams.

Current Status — v0.5.0

FenLiu is a fully functional spam filtering and content management system with complete Curated Queue integration. Monitor hashtags, score posts for spam, manually review content, and reliably export quality posts with fine-grained reblog controls.

Latest updates: Curated Queue API complete with ack/nack/error pattern; reblog controls (blocked users/hashtags, attachments-only); API key authentication; Queue Preview UI; automated scheduling with configurable intervals.

Features

Core Functionality

• Hashtag Monitoring: Monitor multiple Fediverse hashtags with customizable instance sources and scheduling
• Spam Scoring: Rule-based detection (0-100 scale) with 7 intelligent detection rules
• Manual Review Interface: Web interface for reviewing and approving/rejecting posts with scoring
• Bulk Operations: Fetch and process posts in bulk with real-time progress tracking
• Curated Queue Export: API-driven queue with ack/nack/error reliability pattern

Reblog Controls (Export Filters)

• "Don't Reblog" User Blocklist: Prevent specific accounts from being boosted
• "Don't Reblog" Hashtag Blocklist: Exclude posts with blocked hashtags
• Attachments-Only Mode: Export only posts with media attachments
• Auto-Reject on Fetch: Automatically reject blocked content before review

Web Interface

• Dashboard: Real-time analytics, top hashtags, review progress
• Streams Management: Create, edit, manage hashtag streams with CRUD operations
• Review Workflow: Approve/reject posts with manual score adjustment and spam breakdown
• Queue Preview: Monitor queue health (pending/reserved/delivered/error counts)
• Statistics: Charts for posts over time and hashtag distribution
• Responsive Design: Fully responsive across desktop, tablet, mobile

REST API

• Hashtag Streams: Full CRUD for stream management and bulk fetching
• Posts: List, filter, update with approval/rejection and scoring
• Curated Queue: /next, /ack, /nack, /error, /requeue endpoints
• Reblog Controls: Manage blocked users and hashtags
• Statistics: Post counts, hashtag distribution, approval rates
• Authentication: API key-based authentication for queue endpoints
• Health: Health check and application info endpoints

Technical Quality

• Type Safety: Comprehensive type hints throughout
• Testing: 300+ tests with 100% pass rate
• Resource Management: Proper cleanup of DB sessions and HTTP connections
• Database Migrations: Alembic with automatic schema migration on startup
• API Key Security: Secure generation and management of API keys
• No JavaScript Bloat: Pure HTML/CSS frontend, no external JS dependencies

Quick Start

Prerequisites

• Python 3.12 or higher
• uv package manager (recommended)

Installation

# Install dependencies
uv sync -U --all-groups

# Optional: Set up pre-commit hooks
uv run pre-commit install

Running the Application

# Development mode with auto-reload
fenliu --reload --debug

# Alternative development mode
uv run python -m fenliu --reload --debug

# Production mode
fenliu --host 0.0.0.0 --port 8000

# See all options
fenliu --help

First Steps

1. Start the server: fenliu --reload
2. Open browser: Navigate to http://localhost:8000
3. Add a hashtag: Go to Streams page and create a hashtag stream (e.g., "python")
4. Fetch posts: Click "Fetch" on the stream to retrieve posts from Fediverse
5. Review posts: Use the Review interface to approve quality content or reject spam
6. Export: Monitor the Queue Preview to see posts flowing to Curated Queue

Debug Logging

Enable detailed debug logging with the --debug flag:

# Enable debug logging to file
fenliu --debug

# View logs in real-time
tail -f logs/fenliu_debug.log

# Custom log directory
fenliu --debug --log-dir=/var/log/fenliu

In your code: from fenliu.logging import get_logger then logger.debug(f"message")

API Usage

Authentication

All queue endpoints require API key authentication. Generate a key in Settings, then include it in requests:

curl -H "X-API-Key: your-api-key-here" \
  http://localhost:8000/api/v1/curated/next

Common Examples

# List all hashtag streams
curl http://localhost:8000/api/v1/streams

# Create a new hashtag stream
curl -X POST http://localhost:8000/api/v1/streams \
  -H "Content-Type: application/json" \
  -d '{"hashtag": "python", "instance": "mastodon.social", "active": true}'

# Fetch posts for a stream
curl -X POST http://localhost:8000/api/v1/streams/1/fetch?limit=20

# Get next post from Curated Queue
curl -H "X-API-Key: your-api-key-here" \
  http://localhost:8000/api/v1/curated/next

# Acknowledge successful reblog
curl -X POST -H "X-API-Key: your-api-key-here" \
  http://localhost:8000/api/v1/curated/123/ack

# Report permanent failure
curl -X POST -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Account suspended"}' \
  http://localhost:8000/api/v1/curated/123/error

# Review a post (approve)
curl -X PATCH http://localhost:8000/api/v1/posts/123 \
  -H "Content-Type: application/json" \
  -d '{"approved": true, "reviewer_notes": "Quality content"}'

# Adjust spam score manually
curl -X PATCH http://localhost:8000/api/v1/posts/123 \
  -H "Content-Type: application/json" \
  -d '{"manual_spam_score": 15}'

API Endpoints

Streams & Posts:

• GET /api/v1/streams - List streams
• POST /api/v1/streams - Create stream
• GET/PUT/DELETE /api/v1/streams/{id} - Stream operations
• POST /api/v1/streams/{id}/fetch - Fetch posts for stream
• POST /api/v1/streams/fetch-all - Fetch all active streams
• GET /api/v1/posts - List posts with filtering
• GET /api/v1/posts/{id} - Get post details
• PATCH /api/v1/posts/{id} - Update post (review, approve, score)
• GET /api/v1/stats - Application statistics

Curated Queue:

• GET /api/v1/curated/next - Get next post (returns 204 if empty)
• POST /api/v1/curated/{post_id}/ack - Confirm successful reblog
• POST /api/v1/curated/{post_id}/nack - Return to queue (transient failure)
• POST /api/v1/curated/{post_id}/error - Mark permanently failed
• POST /api/v1/curated/{post_id}/requeue - Return errored post to queue

Reblog Controls:

• GET /api/v1/reblog-controls/settings - Get reblog filter settings
• PATCH /api/v1/reblog-controls/settings - Update settings
• GET /api/v1/reblog-controls/blocked-users - List blocked users
• POST /api/v1/reblog-controls/blocked-users - Add blocked user
• DELETE /api/v1/reblog-controls/blocked-users/{id} - Remove blocked user
• GET /api/v1/reblog-controls/blocked-hashtags - List blocked hashtags
• POST /api/v1/reblog-controls/blocked-hashtags - Add blocked hashtag
• DELETE /api/v1/reblog-controls/blocked-hashtags/{id} - Remove blocked hashtag

System:

• GET /health - Health check
• GET /info - Application info

Configuration

Environment variables (via .env file):

# Database
DATABASE_URL=sqlite:///./fenliu.db

# Fediverse settings
DEFAULT_INSTANCE=mastodon.social
API_TIMEOUT=30
MAX_POSTS_PER_FETCH=20
RATE_LIMIT_DELAY=1.0

# Application
DEBUG=false
SECRET_KEY=your-secret-key-change-in-production
APP_NAME=FenLiu

# Spam scoring thresholds
VERY_HIGH_THRESHOLD=76
LOW_MAX_THRESHOLD=25

# Queue timeout
RESERVE_TIMEOUT_SECONDS=300

Development

Testing

# Run full test suite
pytest

# Run with coverage
pytest --cov=src/fenliu tests/

# Quick validation
python -m pytest -q

Code Quality

# Linting
ruff check src/fenliu/

# Formatting
ruff format src/fenliu/

# Pre-commit checks
prek run --all-files

# Full CI simulation
nox

Database Migrations

# Apply pending migrations
alembic upgrade head

# Create new migration
alembic revision --autogenerate -m "description"

# Show current revision
alembic current

# View all revisions
alembic history

Development Workflow

# After dependency changes
uv sync -U --all-groups

# Quick validation before commits
prek run --all-files

# Full validation before commits
nox

Project Structure

fenliu/
├── src/fenliu/
│   ├── __init__.py              # Package definition
│   ├── __main__.py              # CLI entry point
│   ├── main.py                  # PyView application
│   ├── config.py                # Configuration
│   ├── database.py              # Database setup
│   ├── models.py                # SQLAlchemy models
│   ├── schemas.py               # Pydantic validation
│   ├── api/                     # REST API endpoints
│   │   ├── curated.py           # Queue API
│   │   ├── reblog_controls.py   # Filter management
│   │   └── api_keys.py          # API key management
│   ├── services/                # Business logic
│   │   ├── spam_scoring.py      # Spam detection
│   │   ├── fediverse.py         # Fediverse client
│   │   ├── export_eligibility.py # Export filtering
│   │   ├── scheduler.py         # Task scheduling
│   │   └── api_key.py           # API key service
│   ├── templates/               # HTML templates
│   └── static/                  # CSS and assets
├── alembic/                     # Database migrations
├── tests/                       # Test suite (300+ tests)
├── docs/                        # MkDocs documentation
├── pyproject.toml               # Project configuration
└── README.md                    # This file

Documentation

Complete documentation available in the docs/ folder built with MkDocs:

# Serve locally with hot reload
mkdocs serve

# Build static site
mkdocs build

📚 Live Documentation: https://marvinsmastodontools.codeberg.page/fenliu/

Includes: Installation, Quick Start, API Reference, Curated Queue Integration, Contributing Guide, Roadmap, and FAQ.

Technical Stack

• Framework: PyView (Starlette-based LiveView) with real-time capabilities
• Database: SQLAlchemy with SQLite, optimized with eager loading
• API Client: minimal-activitypub for Fediverse integration
• Async: Full async/await throughout (sync for SQLite only)
• Type Hints: Comprehensive type annotations with Pydantic validation
• Frontend: Jinja2 templates with Tailwind CSS, responsive design
• Testing: pytest with 300+ tests
• Linting: ruff for formatting and linting
• Migrations: Alembic for schema management
• Package Manager: uv for dependency management

Upcoming Features

See Roadmap for detailed plans. Phase 4 focus:

• Docker containerization and CI/CD
• Multi-user support with roles
• Performance optimization and caching
• Advanced monitoring dashboard
• PostgreSQL/MySQL support

Cultural Context

The name "FenLiu" (分流) means "divide the flow" in Chinese, inspired by the ancient Dujiangyan irrigation system (256 BC). This project applies the same engineering wisdom to digital content streams, separating valuable content from spam and noise while maintaining the natural flow of community conversation.

Key Resources

• Roadmap - Development plans and future features
• LLM System Prompt - Development standards
• Live Docs - Complete documentation

License

AGPL-3.0 License - See LICENSE file for details.

Contributing

1. Follow existing code style (ruff formatted with comprehensive type hints)
2. Write tests for new functionality (maintain 100% test pass rate)
3. Update documentation as needed
4. Run nox before submitting changes
5. Run alembic upgrade head after pulling changes with new migrations

---

Version: 0.5.0 Status: Fully Functional ✅ Tests: 300+ passing ✅ Code Quality: All checks passing ✅ Framework: PyView (Starlette-based LiveView) Architecture: Async Python with comprehensive type hints Repository: https://codeberg.org/marvinsmastodontools/fenliu