fenliu 0.7.8

Monitor and filter Fediverse hashtags, curate quality content, and distribute via external tools like Zhongli

FenLiu (分流)

Created by marvin8 with assistance from Claude and DeepSeek AI assistants.

⚠️ DISCLAIMER / PROVISO: This project is a work in progress with major changes still happening. It is in no way anywhere close to finished and is only borderline useful for actual production use. Expect breaking changes, incomplete features, and significant architectural evolution as development continues.

Divide the Fediverse content flow

FenLiu is a web application that monitors Fediverse hashtags, filters spam, allows human review, learns from feedback, and exports quality content for boosting. Inspired by the ancient Chinese Dujiangyan irrigation system (256 BC) that separated silt from water, FenLiu applies 2,300-year engineering wisdom to modern digital content streams.

Current Status — v0.7.1

Fully functional spam filtering and content management system with complete Curated Queue integration, flexible pattern-based user blocking, automated queue lifecycle management, production-ready containerization, and ML training data collection.

Latest (v0.7.1): Random post selection for the Curated Queue API. 411 tests passing.

Documentation

📚 Live Docs: https://marvinsmastodontools.codeberg.page/dujiangyan/fenliu/

The docs/ folder contains full MkDocs documentation covering installation, API reference, pattern blocking, Curated Queue integration, and more.

mkdocs serve   # serve locally with hot reload
mkdocs build   # build static site

Features

Core Functionality

• Hashtag Monitoring: Monitor multiple Fediverse hashtags with customizable instance sources and scheduling
• Spam Scoring: Rule-based detection (0-100 scale) with 7 intelligent detection rules
• Manual Review Interface: Approve/reject posts with scoring; pagination, bulk actions, and back-to-top link
• Curated Queue Export: Reliable API-driven queue with ack/nack/error pattern

Reblog Controls (Export Filters)

• Pattern-Based User Blocking: exact, suffix, prefix, and contains matching modes
• Hashtag Blocklist: Exclude posts containing blocked hashtags
• Attachments-Only Mode: Export only posts with media attachments
• Auto-Reject on Fetch: Automatically reject blocked content before review
• Blocklist Refresh: Apply Settings changes to the review page instantly

Web Interface

• Dashboard, Streams Management, Review Workflow, Pattern Blocking Settings, Queue Preview, Statistics
• Responsive design — no external JavaScript dependencies

REST API

• Hashtag streams, posts, curated queue, reblog controls, statistics, health
• API key authentication for queue endpoints

Technical Quality

• 411 tests, 100% passing
• Comprehensive type hints; zero type errors under ty check
• All functions pass ruff and complexipy checks
• Alembic migrations run automatically on startup

Quick Start

Prerequisites: Python 3.12+, uv package manager

uv sync -U --all-groups
fenliu --reload --debug

Open http://localhost:8000, create a hashtag stream, fetch posts, and review them.

Container Deployment

podman build -t fenliu -f Containerfile .
cp .env.example .env   # edit with your settings
podman run -d -p 8000:8000 --env-file .env \
  -v fenliu-data:/app/data -v fenliu-logs:/app/logs fenliu

See the Container Deployment guide for full details.

API Endpoints

All curated queue endpoints require X-API-Key header (generate in Settings).

Streams & Posts:

• GET /api/v1/streams — List streams
• POST /api/v1/streams — Create stream
• GET/PUT/DELETE /api/v1/streams/{id} — Stream operations
• POST /api/v1/streams/{id}/fetch — Fetch posts for stream
• POST /api/v1/streams/fetch-all — Fetch all active streams
• GET /api/v1/posts — List posts with filtering
• PATCH /api/v1/posts/{id} — Update post (review, approve, score)
• GET /api/v1/stats — Application statistics

Curated Queue:

• GET /api/v1/curated/next — Next post (204 if empty); ?random=true for random selection
• POST /api/v1/curated/{post_id}/ack — Confirm successful reblog
• POST /api/v1/curated/{post_id}/nack — Return to queue (transient failure)
• POST /api/v1/curated/{post_id}/error — Mark permanently failed
• POST /api/v1/curated/{post_id}/requeue — Return errored post to queue
• POST /api/v1/curated/cleanup — Delete old delivered posts
• POST /api/v1/curated/trim-pending — Trim excess pending posts

Reblog Controls:

• GET/PUT /api/v1/reblog-controls/settings — Reblog filter settings
• GET/POST /api/v1/reblog-controls/blocked-users — Blocked users (pattern-based)
• DELETE /api/v1/reblog-controls/blocked-users/{id} — Remove blocked user
• GET/POST /api/v1/reblog-controls/blocked-hashtags — Blocked hashtags
• DELETE /api/v1/reblog-controls/blocked-hashtags/{id} — Remove blocked hashtag
• POST /api/v1/reblog-controls/reject-blocked — Bulk reject matching posts

System: GET /health, GET /info

See the API docs for full reference.

Configuration

Key environment variables (see .env.example for full list):

Variable	Default	Description
DATABASE_URL	sqlite:///./fenliu.db	Database connection
DEFAULT_INSTANCE	mastodon.social	Default Fediverse instance
RESERVE_TIMEOUT_SECONDS	300	Queue reservation timeout
VERY_HIGH_THRESHOLD	76	Spam score: very high
LOW_MAX_THRESHOLD	25	Spam score: low
DEBUG	false	Enable debug logging

Development

uv run pytest                    # run tests
uv run ruff check .              # lint
uv run complexipy .              # complexity check
uv run ty check                  # type check
nox                              # full CI simulation

alembic upgrade head             # apply migrations
alembic revision --autogenerate -m "description"  # new migration

Project Structure

fenliu/
├── src/fenliu/
│   ├── main.py                  # PyView application and LiveViews
│   ├── models.py                # SQLAlchemy models
│   ├── schemas.py               # Pydantic validation
│   ├── api/                     # REST API (curated, reblog_controls, api_keys)
│   ├── services/                # Business logic (spam scoring, fediverse, scheduler)
│   ├── templates/               # HTML templates
│   └── static/                  # CSS and assets
├── alembic/                     # Database migrations
├── tests/                       # Test suite (402 tests)
└── docs/                        # MkDocs documentation

Technical Stack

• Framework: PyView (Starlette-based LiveView)
• Database: SQLAlchemy + SQLite, Alembic migrations
• Validation: Pydantic v2
• Fediverse: minimal-activitypub
• Frontend: Jinja2 + Tailwind CSS, no external JS
• Testing: pytest (411 tests)
• Tooling: ruff, ty, complexipy, uv

What's New in v0.7.1

• Random queue selection: GET /api/v1/curated/next?random=true returns a randomly chosen eligible post instead of the oldest. If the chosen author has more than one pending post, their oldest is returned to avoid consecutive same-author posts
• 9 new tests for random selection behaviour; 411 total

Previous Release — v0.7.0

• Review pagination: 20 posts/page with prev/next navigation
• Bulk actions: Approve All / Reject All for the current page
• Auto-refresh: Empty page reloads automatically when more posts exist
• ML training snapshots: ReviewFeedback captures 12 feature fields at review time — survives queue cleanup
• Bug fix: Stream deletion cascade fixed (Post → ReviewFeedback)

Previous Release — v0.6.0

• Queue Lifecycle Management: Auto-delete delivered posts (7-day retention), trim excess pending with weighted random deletion, cleanup and trim-pending API endpoints
• Production Containerization: Multi-stage build (~207 MB), non-root user, persistent volumes, automatic migrations on startup

Previous Release — v0.5.3

• Pattern-Based User Blocking: exact, suffix, prefix, and contains matching modes — see PATTERN_BLOCKING_FEATURE.md
• Blocklist Refresh: Apply Settings changes to the review page without losing progress

Cultural Context

"FenLiu" (分流) means "divide the flow" in Chinese, inspired by the Dujiangyan irrigation system (256 BC). This project applies the same engineering wisdom to digital content streams.

Key Resources

• Live Documentation — Full docs
• Roadmap — Development plans
• Pattern Blocking Guide — Pattern matching details
• API Reference — Endpoint documentation

License

AGPL-3.0 — see LICENSE file.

Contributing

1. Follow existing code style (ruff, comprehensive type hints)
2. Write tests for new functionality
3. Run nox before submitting changes
4. Run alembic upgrade head after pulling changes with new migrations

---

v0.7.1 · Phase 4 In Progress · 411 tests ✅ · Codeberg