fenliu 0.4.1

Monitor and filter Fediverse hashtags, curate quality content, and distribute via external tools like Zhongli

FenLiu (分流)

Created by marvin8 with assistance from DeepSeek AI assistant.

⚠️ DISCLAIMER / PROVISO: This project is a work in progress with major changes still happening. It is in no way anywhere close to finished and is only borderline useful for actual production use. Expect breaking changes, incomplete features, and significant architectural evolution as development continues.

Divide the Fediverse content flow

FenLiu is a web application that monitors Fediverse hashtags, filters spam, allows human review, learns from feedback, and exports quality content for boosting. Inspired by the ancient Chinese Dujiangyan irrigation system (256 BC) that separated silt from water, FenLiu applies 2,300-year engineering wisdom to modern digital content streams.

Current Status (1 March 2026) — v0.2.3

✅ Phase 1: Foundation & MVP - COMPLETE

• Web Application: PyView-based LiveView interface with real-time capabilities
• Fediverse Integration: Fetch posts from Mastodon/Fediverse instances
• Database: SQLAlchemy models with SQLite backend, optimized with eager loading
• REST API: Full CRUD operations for all entities with proper HTTP semantics
• Testing: Comprehensive test suite with 267 tests passing

✅ Phase 2: Enhanced Features - COMPLETE

• Spam Scoring: Advanced rule-based system (0-100 scale) with 7 detection rules
• Streams Management: Full CRUD operations with bulk fetch and progress tracking
• UI Consistency: All pages with consistent styling and hyperlinked hashtags
• Manual Review Interface: Complete web interface for reviewing posts with approve/reject workflow
• Learning System Foundation: ReviewFeedback model for historical tracking of review decisions
• Export Foundation: Database schema and API support for Curated Queue integration
• Alembic Migrations: Full Alembic setup with baseline migration; alembic upgrade head runs automatically on startup
• DB Session Management: All database sessions now properly closed via @contextmanager on get_db()
• Refined Review Interface: Enhanced review page with improved styling and user experience
• Dynamic Version Management: Application version automatically loads from package metadata

📋 Phase 3: Advanced Functionality - PLANNED

• Export Implementation: Complete Curated Queue export functionality
• Machine Learning Integration: Train models on collected review feedback
• Real-time Features: Enhanced LiveView for instant updates
• Multi-instance Support: Monitor multiple Fediverse instances simultaneously
• Advanced Filtering: Custom rule system for content classification

Recent Progress (March 2026 - Version 0.2.3)

Bug Fixes & UX Improvements (v0.2.3 — 1 March 2026)

• No JavaScript Popups: All alert(), confirm(), and prompt() calls removed from every template; replaced with inline flash messages and page redirects
  • dashboard.html: addNewStream() (which used prompt()) replaced with direct links to /streams; fetchStream() errors shown via showFlash()
  • hashtag_streams.html: All alert() calls replaced with a showFlash() helper; confirm() before "Fetch All" removed (the progress modal provides sufficient feedback)
  • reblog_controls.html: confirm() guards removed from deleteBlockedUser() and deleteBlockedHashtag() (trash icon already signals intent)
• Auto-Reject on Fetch Fixed: Posts fetched from Fediverse were not being auto-rejected even when the setting was enabled. Root cause: the SQLAlchemy session uses autoflush=False, so newly add()-ed posts were invisible to the SELECT inside reject_blocked_posts(). Fixed by calling db.flush() after all new posts are staged in _save_fetched_posts().
• New Regression Tests: Three tests added covering auto-reject by blocked user, auto-reject by blocked hashtag, and no-reject when the setting is disabled.

Code Quality & Bug Fixes (v0.2.1 — 1 March 2026)

• DB Session Leak Fixed: get_db() decorated with @contextmanager; all 23 call sites updated to with get_db() as db:
• FediverseClient Leak Fixed: httpx.AsyncClient now closed via try/finally after each fetch
• Routing Bug Fixed: /posts/batch-score route moved before /{post_id} to prevent shadowing
• load_stats() Performance: Replaced len(all()) counts with func.count() SQL aggregates
• Alembic Setup: alembic.ini, alembic/env.py, alembic/script.py.mako created; baseline migration generated and stamped; app lifespan now runs alembic upgrade head on startup
• Request Type Hints: All API handler request parameters now typed as Request
• Ruff Clean: All linting issues resolved across production and migration code

Manual Review Interface

• Complete Review Workflow: Approve/reject decisions with scoring adjustment
• Review Feedback Tracking: Historical tracking of review decisions for future ML training
• Refined Review Interface: Enhanced styling and user experience improvements
• Dynamic Version Management: Application version now automatically loads from package metadata
• Test Expansion: Increased test coverage from 159 tests to current suite
• Interactive Scoring: Manual score adjustment with detailed breakdown visualization
• Real-time Stats: Dashboard cards showing review progress and statistics
• Filter System: Filter posts by spam score range and hashtag streams

Streams Management Enhancement

• Complete CRUD Operations: Create, Read, Update, Delete for hashtag streams with modal dialogs
• Bulk Fetch Functionality: Fetch all active streams simultaneously with real-time progress tracking
• API Consistency: Fixed endpoint mismatches and HTTP 204 response handling
• Performance Optimizations: Eager loading of relationships to prevent N+1 queries

User Interface Improvements

• Consistent Styling: All pages now display hyperlinked hashtags matching dashboard behavior
• Dashboard Analytics: Charts for posts over time and top hashtags visualization
• Responsive Design: Fully responsive across desktop, tablet, and mobile devices
• Real-time Feedback: Button state management with spinners during operations
• Enhanced Navigation: Improved mobile menu and accessibility features

Technical Foundation

• Testing: All 267 tests passing with comprehensive coverage
• Type Safety: Enhanced type annotations throughout the codebase
• Code Quality: Pre-commit hooks passing (ruff format, ruff check)
• Performance: Optimized database queries and eager loading
• Version Management: Dynamic version loading from package metadata

Quick Start

Prerequisites

• Python 3.11 or higher
• uv package manager (recommended)

Installation

# Install dependencies
uv sync -U --all-groups

# Optional: Set up pre-commit hooks
uv run pre-commit install

Running the Application

# Development mode with auto-reload (simpler)
fenliu --reload --debug

# Alternative development mode
uv run python -m fenliu --reload --debug

# Production mode (simpler)
fenliu --host 0.0.0.0 --port 8000

# Alternative production mode
uv run python -m fenliu --host 0.0.0.0 --port 8000

# See all options
uv run fenliu --help

Debug Logging

Enable detailed debug logging to file with the --debug flag. This writes all DEBUG level logs to logs/fenliu_debug.log with module name, function name, and line number context. Console output shows INFO and ERROR only. Log files rotate daily at midnight with 3 versions retained.

# Enable debug logging
python -m fenliu --debug

# View logs in real-time
tail -f logs/fenliu_debug.log

# Use custom log directory
python -m fenliu --debug --log-dir=/var/log/fenliu

In your code, use from fenliu.logging import get_logger to add debug entries: logger = get_logger(__name__) then logger.debug(f"context: message with {variable}").

First Steps

1. Start the server: fenliu --reload (or uv run python -m fenliu --reload)
2. Open browser: Navigate to http://localhost:8000
3. Add a hashtag: Go to Streams page and add a hashtag (e.g., "python")
4. Fetch posts: Click "Fetch" on the hashtag stream (or use "Fetch All" for bulk operations)
5. Review posts: Use the Manual Review interface to approve quality content or reject spam

API Usage

Basic Examples

# List all hashtag streams
curl http://localhost:8000/api/v1/hashtags

# Add a new hashtag stream
curl -X POST http://localhost:8000/api/v1/hashtags \
  -H "Content-Type: application/json" \
  -d '{"hashtag": "python", "instance": "mastodon.social", "active": true}'

# Fetch posts for a stream (individual or bulk)
curl -X POST http://localhost:8000/api/v1/hashtags/1/fetch?limit=20

# Update a stream
curl -X PATCH http://localhost:8000/api/v1/hashtags/1 \
  -H "Content-Type: application/json" \
  -d '{"active": false}'

# Get application statistics
curl http://localhost:8000/api/v1/stats

# Review a post (approve)
curl -X PATCH http://localhost:8000/api/v1/posts/123 \
  -H "Content-Type: application/json" \
  -d '{"approved": true, "reviewer_notes": "High-quality content"}'

# Adjust spam score manually
curl -X PATCH http://localhost:8000/api/v1/posts/123 \
  -H "Content-Type: application/json" \
  -d '{"manual_spam_score": 15, "reviewer_notes": "Adjusted score based on context"}'

API Endpoints

• GET /api/v1/hashtags - List all hashtag streams with filtering
• POST /api/v1/hashtags - Create new stream
• GET/PATCH/DELETE /api/v1/hashtags/{id} - Stream CRUD operations
• POST /api/v1/hashtags/{id}/fetch - Fetch posts for specific stream
• GET /api/v1/posts - List posts with advanced filtering (stream, spam score, status, reviewed, approved, exported)
• GET /api/v1/posts/{id} - Get specific post details
• PATCH /api/v1/posts/{id} - Update post (review, approve, adjust scores)
• GET /api/v1/stats - Application statistics and metrics
• GET /health - Health check endpoint
• GET /info - Application information
• GET /api/v1/posts/{id}/score - Get post scoring analysis
• POST /api/v1/posts/{id}/recalculate - Recalculate spam score
• POST /api/v1/posts/batch-score - Batch score multiple posts

Note: The /posts listing page has been removed. Use the /review LiveView interface to browse and manage posts.

Project Structure

fenliu/
├── src/fenliu/
│   ├── __init__.py              # Package definition
│   ├── __main__.py              # CLI entry point
│   ├── main.py                  # PyView application with enhanced queries
│   ├── config.py                # Application settings
│   ├── database.py              # Database configuration
│   ├── models.py                # SQLAlchemy models with relationships (including ReviewFeedback)
│   ├── schemas.py               # Pydantic schemas for validation
│   ├── api/                     # REST API endpoints (full CRUD with review support)
│   ├── services/                # Business logic (spam scoring, Fediverse client)
│   └── templates/               # HTML templates (dashboard, streams, review, stats, post_details)
├── alembic/                     # Database migration scripts
│   ├── env.py                   # Alembic environment (wired to app config and models)
│   ├── script.py.mako           # Migration file template
│   └── versions/                # Migration history
├── alembic.ini                  # Alembic configuration
├── tests/                       # Test suite (267 tests)
├── pyproject.toml               # Project configuration
└── README.md                    # This file

Testing

# Run linting and formatting
ruff check src/fenliu/
ruff format src/fenliu/

# Run pre-commit checks
pre-commit run --all-files

# Run full test suite
pytest --cov=src/fenliu tests/

# Quick validation
python -m pytest -q

# Database migrations
alembic upgrade head                              # apply all pending migrations
alembic revision --autogenerate -m "add column"  # generate a new migration
alembic current                                   # show current revision
alembic history                                   # list all revisions

Configuration

Environment variables (via .env file):

# Database configuration
DATABASE_URL=sqlite:///./fenliu.db

# Fediverse settings
DEFAULT_INSTANCE=mastodon.social
API_TIMEOUT=30
MAX_POSTS_PER_FETCH=20
RATE_LIMIT_DELAY=1.0

# Application settings
DEBUG=false
SECRET_KEY=your-secret-key-change-in-production
APP_NAME=FenLiu
# APP_VERSION is read from package metadata — do not set this env var

# Spam scoring thresholds
VERY_HIGH_THRESHOLD=76
LOW_MAX_THRESHOLD=25

Development Workflow

After dependency changes:

uv sync -U --all-groups

Quick validation:

pre-commit run --all-files

Full validation before commits:

nox

Technical Stack

• Framework: PyView (Starlette-based LiveView) with real-time capabilities
• Database: SQLAlchemy with SQLite, optimized with eager loading
• API Client: minimal-activitypub for Fediverse integration
• Async/await: Full async support throughout the codebase except for interacting with SQLite db.
• Type Hints: Comprehensive type annotations with Pydantic validation
• Frontend: Jinja2 templates with Tailwind CSS and responsive design
• Testing: pytest with 267 tests covering all major functionality
• Linting/Formatting: ruff with pre-commit hooks
• Package Management: uv for fast dependency resolution
• Spam Scoring: Rule-based system with 7 detection rules (0-100 scale)
• Review System: Manual review interface with decision tracking
• Learning Foundation: ReviewFeedback model for future ML training

Cultural Context

The name "FenLiu" (分流) means "divide the flow" in Chinese, inspired by the ancient Dujiangyan irrigation system (256 BC). This project applies the same engineering wisdom to digital content streams, separating valuable content from spam and noise while maintaining the natural flow of community conversation.

Documentation

• Roadmap - 5-phase development plan with current progress
• Project Summary - Overview and detailed status
• LLM System Prompt - Development standards and guidelines

License

AGPL-3.0 License - See LICENSE file for details.

Contributing

1. Follow the existing code style (ruff formatted with comprehensive type hints)
2. Write tests for new functionality (maintain 100% test pass rate)
3. Update documentation to reflect changes
4. Run nox before submitting changes
5. Run alembic upgrade head after pulling changes that include new migrations

---

Status: Phase 1 & 2 Complete ✅, Phase 3 Planned 📋 Version: 0.2.3 Date: 1 March 2026 Test Status: 267/267 tests passing ✅ Code Quality: All ruff and pre-commit checks passing ✅ Next Phase: Curated Queue API queue integration and ML training (Phase 3) Framework: PyView (Starlette-based LiveView) Architecture: Async Python with comprehensive type hints and modern web patterns Recent Focus: JS popup removal, auto-reject on fetch bug fix