ux-journey-scraper 0.3.0

Autonomous web crawler for capturing user journeys

UX Journey Scraper

Autonomous web crawler for capturing user journeys.

A powerful tool for UX designers, developers, and QA teams to autonomously capture complete user flows through websites.

🎯 What It Does

Core Features (v0.3.0)

• 🤖 Autonomous Crawling: Intelligently navigates websites without manual intervention
• 🎯 Smart Element Detection: Finds ALL clickables (buttons, links, onclick handlers, ARIA roles)
• 📸 Journey Capture: Records complete user flows with screenshots at each step
• 🔐 Auth Support: Handles login flows and session management
• 📋 Form Filling: Automatically fills checkout forms (test data only)
• 🕵️ Stealth Mode: Anti-bot detection with human-like behavior simulation
• 🔄 SPA Support: Works with modern single-page applications
• 🛡️ Privacy-Aware: Automatically blurs PII in screenshots

🚀 Installation

PyPI Installation (Recommended)

# Install latest version (v0.3.0 - Autonomous Crawling)
pip install ux-journey-scraper

Installation from Source

# Clone the repository
git clone https://github.com/resabh/ux-journey-scraper.git
cd ux-journey-scraper

# Install in development mode
pip install -e .

# OR install with full UX guidelines support
pip install -e ".[full]"

Note: The [full] option includes ecommerce-ux-guidelines package for detailed UX analysis. The base installation works fully for autonomous crawling and journey capture without it.

Post-Installation

After installation, install Playwright browsers:

playwright install chromium

📖 Quick Start

Autonomous Crawl (v0.3.0)

# Create a configuration file (see scrape-config.example.yaml)
ux-journey crawl --config scrape-config.yaml --output-dir journey_output/

# The tool will:
# 1. Check robots.txt (asks for confirmation if needed)
# 2. Autonomously navigate through the site
# 3. Capture screenshots at each step
# 4. Record complete user journey
# 5. Save journey data as JSON + screenshots

Manual Recording (Deprecated)

# Manually record a journey (interactive mode)
ux-journey record \
  --start-url https://example.com \
  --output my-journey.json

# Note: Use 'crawl' for autonomous navigation

💡 Use Cases

1. Competitor Analysis

Capture competitor websites to understand user flows and patterns.

ux-journey crawl --config competitor-config.yaml --output-dir competitor_journey/

2. User Flow Documentation

Document user flows with screenshots for design/QA teams.

ux-journey crawl --config mysite-config.yaml --output-dir user_flows/

3. Pre-Launch Testing

Capture staging sites before launch.

ux-journey crawl --config staging-config.yaml --output-dir staging_test/

4. QA Automation

Automated journey capture for regression testing.

ux-journey crawl --config qa-config.yaml --output-dir qa_snapshots/

🛠️ Features

Journey Recording

• ✅ Autonomous crawling with priority queue
• ✅ Multi-step journey capture
• ✅ Automatic screenshot on each navigation
• ✅ HTML structure extraction
• ✅ Form field detection and auto-fill
• ✅ CTA and button analysis
• ✅ Navigation element capture
• ✅ Session management and auth support

Accessibility

• ✅ WCAG 2.1 Level A/AA validation
• ✅ Color contrast analysis
• ✅ Keyboard navigation checks
• ✅ Screen reader compatibility
• ✅ ARIA attributes validation

Privacy & Security

• ✅ PII blur in screenshots (emails, credit cards, names)
• ✅ robots.txt respect (with user confirmation)
• ✅ Rate limiting
• ✅ No data collection or tracking

Reports

• ✅ Interactive HTML reports
• ✅ Annotated screenshots with issue markers
• ✅ JSON export for automation
• ✅ Journey flow visualization
• ✅ Issue summaries and scores

📋 CLI Reference

ux-journey record

Record a user journey interactively.

ux-journey record [OPTIONS]

Options:
  --start-url TEXT         Starting URL (required)
  --output TEXT            Output file path (default: journey.json)
  --viewport TEXT          Viewport size (default: 1920x1080)
  --blur-pii               Blur PII in screenshots (default: true)
  --respect-robots         Check robots.txt (default: true)
  --headless              Run in headless mode (default: false)

🔧 Python API

from ux_journey_scraper import JourneyRecorder, Journey
from ux_journey_scraper.config import ScrapeConfig
from ux_journey_scraper.core import AutonomousCrawler

# Option 1: Autonomous crawling with config
config = ScrapeConfig.load("scrape-config.yaml")
crawler = AutonomousCrawler(config=config, output_dir="journey_output/")
journey = await crawler.crawl()

# Option 2: Manual recording (deprecated)
recorder = JourneyRecorder(
    start_url="https://example.com",
    blur_pii=True,
    respect_robots=True
)
journey = await recorder.record()
journey.save("my-journey.json")

# Load existing journey
journey = Journey.load("journey.json")

# Access journey data
for step in journey.steps:
    print(f"Step {step.step_number}: {step.title}")
    print(f"  URL: {step.url}")
    print(f"  Screenshot: {step.screenshot_path}")

📊 Sample Journey Output

{
  "start_url": "https://example.com",
  "start_time": "2026-03-19T12:00:00Z",
  "end_time": "2026-03-19T12:05:30Z",
  "viewport": [1920, 1080],
  "steps": [
    {
      "step_number": 1,
      "url": "https://example.com",
      "title": "Homepage",
      "screenshot_path": "screenshots/step-001.png",
      "html_snapshot": "...",
      "page_data": {
        "forms": [...],
        "buttons": [...],
        "links": [...]
      }
    },
    ...
  ]
}

🔍 How It Works

1. Configuration

   • YAML config defines seed URLs, auth, form fill settings
   • Priority queue manages navigation strategy

2. Autonomous Crawling

   • Launches stealth browser (anti-bot detection)
   • Navigates autonomously using priority queue
   • Finds ALL clickables (buttons, links, ARIA roles, onclick handlers)
   • Handles login flows and session management
   • Fills forms with test data (payment safeguards)

3. Capture

   • Waits for page readiness (DOM stable, no spinners, lazy-load complete)
   • Captures screenshot + HTML at each step
   • Blurs PII automatically
   • Records all page data (forms, buttons, links)

4. Output

   • Saves journey JSON with all steps
   • Screenshots folder with PII blurred
   • Complete DOM snapshots for analysis

🛣️ Roadmap

v0.3.0 (Current) - Pure Journey Capture

• ✅ Autonomous crawling with priority queue
• ✅ Desktop web journey recording
• ✅ Smart element detection (4 strategies)
• ✅ Auth support and session management
• ✅ Form auto-fill with safeguards
• ✅ PII blur in screenshots
• ✅ robots.txt handling
• ✅ Stealth mode (anti-bot detection)

Future Enhancements

• ⬜ Mobile viewports (responsive crawling)
• ⬜ Multi-page site mapping
• ⬜ CAPTCHA handling improvements
• ⬜ Visual regression detection
• ⬜ Performance metrics capture
• ⬜ CI/CD integration

Analysis Features → Moved to BayMAAR

• UX guidelines analysis is now in the separate BayMAAR Analysis Engine (private)
• This package remains a pure journey capture tool

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License - see LICENSE file for details

🔗 Related Projects

• ecommerce-ux-guidelines - 324+ research-backed UX guidelines

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Made with ❤️ for better user experiences