Skip to main content

CDVL Crawler

PyPI version

Python tools for crawling and downloading videos from the CDVL (Consumer Digital Video Library) research video repository.

Contents:

Disclaimer

What This Does

This package provides a unified command-line tool for working with CDVL:

cdvl-crawler, which starts an interactive menu when run without arguments, or runs one of these subcommands directly:

  • crawl - Crawls and extracts metadata from all videos and datasets on CDVL
  • download - Downloads individual videos by their ID
  • generate-site - Generates a searchable, interactive HTML site from crawled metadata
  • export - Exports JSONL data to CSV format for use in spreadsheets

Features:

  • Automatic Login: Handles authentication automatically with username/password
  • Parallel Crawling: Concurrent requests for efficient data collection
  • Smart Enumeration: Automatically discovers videos and datasets by ID
  • Auto-Resume: Continues from the last crawled ID if interrupted; press q to stop a running crawl
  • Structured Output: JSONL format for easy data processing
  • Interactive Mode: Menu-driven usage with search, selection lists and sensible defaults
  • Progress Tracking: Colorful progress bars showing found/empty/failed counts
  • Download Management: Handles large file downloads with progress indicators
  • Static Site Generator: Creates a beautiful, searchable HTML interface for browsing videos

Requirements

  • Python 3.10 or higher
  • An active CDVL account with username and password

Installation

Install with uvx:

uvx cdvl-crawler          # interactive menu
uvx cdvl-crawler --help
uvx cdvl-crawler crawl --help
uvx cdvl-crawler download --help
uvx cdvl-crawler generate-site --help
uvx cdvl-crawler export --help

Or install with pipx:

pipx install cdvl-crawler

Or, with pip:

pip3 install --user cdvl-crawler

We assume you will be using uvx, otherwise just run cdvl-crawler directly without uvx after installing from pipx or pip.

Usage

Interactive Mode

Run cdvl-crawler without a subcommand to open the interactive menu:

uvx cdvl-crawler

The menu shows your configuration and the number of already crawled records, and lets you crawl metadata, download videos, generate the HTML site, or export to CSV. To download videos, you can search the crawled metadata by title, filename or description and select results from a list, or enter IDs directly. The license agreement is shown once per session.

The subcommands below do the same without prompts, which is useful for scripts. Add -v for detailed log output.

Credentials

Before using the tool, you need to provide your CDVL credentials. The tool supports three methods for providing credentials (in order of priority):

  1. Config file: Create a config.json file in your working directory (automatically detected) or specify with --config that contains username and password
  2. Environment variables: Set CDVL_USERNAME and CDVL_PASSWORD
  3. Interactive prompt: The tool will ask for credentials if not found via other methods

Note: If a config.json file exists in your current directory, it will be automatically loaded. You don't need to specify --config unless you want to use a different file.

Choose the method that best suits your workflow. For example:

# Using environment variables (no config file needed)
export CDVL_USERNAME="your.email@example.com"
export CDVL_PASSWORD="your_password"
uvx cdvl-crawler crawl

# Using config.json (automatically detected if in current directory)
# Just create config.json:
# {
#   "username": "your.email@example.com",
#   "password": "your_password_here"
# }

# and run:
uvx cdvl-crawler crawl

Crawling Metadata

To crawl all videos and datasets:

# Basic usage (outputs to current directory)
uvx cdvl-crawler crawl

# Save to specific directory
uvx cdvl-crawler crawl --output-dir ./data

# Accept license automatically (useful for automation/scripts)
uvx cdvl-crawler crawl --accept-license

# Crawl with custom concurrency and delays
uvx cdvl-crawler crawl --max-concurrent 10 --delay 0.2

# Crawl up to specific ID limits
uvx cdvl-crawler crawl --max-video-id 3000 --max-dataset-id 500

# Adjust failure threshold (stop after N consecutive failures)
uvx cdvl-crawler crawl --max-failures 2000

# Advanced: customize ID gap probing
uvx cdvl-crawler crawl --probe-step 50 --max-probe-attempts 40

For more options, run:

uvx cdvl-crawler crawl --help

The crawler will automatically:

  1. Log in with your credentials (from config, env vars, or prompt)
  2. Crawl videos and datasets in parallel
  3. Save metadata to videos.jsonl and datasets.jsonl in the output directory
  4. Resume from the last ID if run again

Example output:

2025-10-09 15:30:03 - INFO - ✓ Login successful!
2025-10-09 15:30:03 - INFO - Starting crawlers in parallel...
Videos:   12543 | Success: 8432 | Empty: 3891 | Failed: 220
Datasets:   142 | Success:   98 | Empty:   34 | Failed:  10

To start fresh, delete the output files before running:

rm ./data/videos.jsonl ./data/datasets.jsonl
uvx cdvl-crawler crawl --output-dir ./data

To resume, just run it again - it will automatically continue from where it left off.

Downloading Videos

Download videos by their ID:

# Download a single video (to current directory)
uvx cdvl-crawler download 42

# Download to specific directory
uvx cdvl-crawler download 42 --output-dir ./downloads

# Accept license automatically (useful for automation/scripts)
uvx cdvl-crawler download 42 --accept-license

# Download multiple videos (comma-separated)
uvx cdvl-crawler download 1,5,10-20 --output-dir ./videos

# Get download URL without downloading
uvx cdvl-crawler download 42 --dry-run

# Download to specific filename (single video only)
uvx cdvl-crawler download 42 --output my_video.avi

# Disable resume capability (always download from beginning)
uvx cdvl-crawler download 42 --no-resume

Downloads include automatic file size verification and support for resuming interrupted downloads (if the server supports HTTP range requests).

For more options:

uvx cdvl-crawler download --help

Generating Static Site

After crawling metadata, you can generate a beautiful, interactive HTML site to browse the video library:

# Generate site from videos.jsonl to index.html
uvx cdvl-crawler generate-site

# Specify custom input and output files
uvx cdvl-crawler generate-site -i ./data/videos.jsonl -o ./website/index.html

For more options:

uvx cdvl-crawler generate-site --help

Exporting to CSV

After crawling metadata, you can export it to CSV format for use in spreadsheets or data analysis tools:

# Export all columns (default)
uvx cdvl-crawler export -i videos.jsonl -o videos.csv

# Export specific columns
uvx cdvl-crawler export -i videos.jsonl -o videos.csv --columns id,title,filename

# Export datasets
uvx cdvl-crawler export -i datasets.jsonl -o datasets.csv --columns id,title,url

Available columns depend on the data, but typically include: id, url, title, content_type, filename, file_size, paragraphs, links, media, tables_count, extracted_at.

For more options:

uvx cdvl-crawler export --help

Output Format

Output files use JSON Lines format (one JSON object per line).

Video Records

{
  "id": 5,
  "url": "https://www.cdvl.org/members-section/view-file/?videoid=5",
  "paragraphs": [
    "Description:Pan over a children's ball pit, seen from above using a crane.",
    "Dataset:NTIA Source Scenes",
    "Audio Specifications:16-bit stereo PCM. Talk in English with balls rustling.",
    "Video Specification:The camera was a professional HDTV camera..."
  ],
  "extracted_at": "2025-10-09T15:30:00+00:00",
  "content_type": "video",
  "title": "NTIA children's ball pit from above, part 1, 525-line",
  "links": [{"text": "NTIA T1.801.01", "href": "/members-section/search?dataset=3"}],
  "media": [{"type": "img", "src": "/uploads/thumbnails/thumb_1.jpg"}],
  "filename": "ntia_bpit1-525_original.avi",
  "file_size": "242.50 MB"
}

Required fields:

  • id: Video ID number
  • url: Source URL on CDVL
  • paragraphs: Structured list of text content from the page
  • extracted_at: Timestamp when data was extracted (ISO 8601 format)
  • content_type: Always "video" for videos

Optional fields:

  • title: Video title
  • links: Related links found on the page
  • media: Images and media elements
  • filename: Download filename (if available)
  • file_size: File size (if available)

Dataset Records

{
  "id": 7,
  "url": "https://www.cdvl.org/members-section/search?dataset=7",
  "paragraphs": ["Description of the dataset...", "Additional information..."],
  "extracted_at": "2025-10-09T15:30:00+00:00",
  "content_type": "dataset",
  "title": "Mobile Quality Dataset",
  "links": [{"text": "Video 123", "href": "/members-section/view-file/?videoid=123"}],
  "tables_count": 2
}

Required fields:

  • id: Dataset ID number
  • url: Source URL on CDVL
  • paragraphs: Structured list of text content from the page
  • extracted_at: Timestamp when data was extracted (ISO 8601 format)
  • content_type: Always "dataset" for datasets

Optional fields:

  • title: Dataset title
  • links: Related links found on the page
  • tables_count: Number of tables in the dataset page

Here are some processing examples using jq.

Count records:

wc -l videos.jsonl datasets.jsonl

View first record:

head -n 1 videos.jsonl | jq .

Extract all titles:

jq -r '.title' videos.jsonl

Filter by keyword in paragraphs:

jq 'select(.paragraphs | join(" ") | contains("codec"))' videos.jsonl

Convert to CSV (or use cdvl-crawler export):

jq -r '[.id, .title, .url] | @csv' videos.jsonl > videos.csv

Configuration File (config.json)

Configuration is optional. The tool has sensible defaults built-in, and you can use:

  • Environment variables for authentication (see Usage above)
  • Command-line options for crawling parameters (see --help)
  • Interactive prompts if credentials are not found

Auto-detection: If a file named config.json exists in your current directory, it will be automatically loaded. You can override this with --config path/to/other.json.

If you want to customize settings permanently or override defaults, create a config.json file:

  1. Download config.example.json from the repository
  2. Rename it to config.json
  3. Edit config.json with your settings:
{
  "username": "your.email@example.com",
  "password": "your_password_here",
  "endpoints": {
    "video_base_url": "https://www.cdvl.org/members-section/view-file/",
    "dataset_base_url": "https://www.cdvl.org/members-section/search"
  },
  "output": {
    "videos_file": "videos.jsonl",
    "datasets_file": "datasets.jsonl"
  },
  "start_video_id": 1,
  "start_dataset_id": 1,
  "max_video_id": null,
  "max_dataset_id": null,
  "max_concurrent_requests": 5,
  "max_consecutive_failures": 1000,
  "request_delay": 0.1,
  "probe_step": 100,
  "max_probe_attempts": 20
}

Configuration Options

All settings are optional with sensible defaults. CLI options override config file values.

Setting Default CLI Option Description
username (from env CDVL_USERNAME or prompt) - Your CDVL account email
password (from env CDVL_PASSWORD or prompt) - Your CDVL account password
start_video_id 1 --start-video-id Starting video ID for crawling
start_dataset_id 1 --start-dataset-id Starting dataset ID for crawling
max_video_id None --max-video-id Maximum video ID to crawl (optional)
max_dataset_id None --max-dataset-id Maximum dataset ID to crawl (optional)
max_concurrent_requests 5 --max-concurrent Number of parallel requests
max_consecutive_failures 1000 --max-failures Stop after N consecutive empty/failed responses
request_delay 0.1 --delay Delay between request batches (seconds)
probe_step 100 --probe-step How far ahead to jump when probing for ID gaps
max_probe_attempts 20 --max-probe-attempts Max probe attempts (20*100=2000 ID range)
videos_file videos.jsonl - Output filename for video metadata
datasets_file datasets.jsonl - Output filename for dataset metadata
endpoints.video_base_url cdvl.org members section - Base URL for video pages
endpoints.dataset_base_url cdvl.org members section - Base URL for dataset pages
headers Browser-like headers - HTTP headers (User-Agent, Accept, etc.)

API

You can also use the package programmatically:

import asyncio
from cdvl_crawler import CDVLCrawler, CDVLDownloader, CDVLExporter, CDVLSiteGenerator

# Crawl videos and datasets
async def crawl():
    # Config file is optional - will use env vars or prompt
    crawler = CDVLCrawler(config_path=None, output_dir="./data")
    await crawler.crawl()

# Download a specific video
async def download():
    # Config file is optional - will use env vars or prompt
    downloader = CDVLDownloader(config_path=None, output_dir="./downloads")
    await downloader._init_session()
    await downloader._login()

    url = await downloader.get_download_link(42)
    if url:
        await downloader.download_file(url, "output.avi")

    await downloader._close_session()

# Generate static site
def generate_site():
    generator = CDVLSiteGenerator(
        input_file="./data/videos.jsonl",
        output_file="./website/index.html"
    )
    success = generator.generate()
    print(f"Site generated: {success}")

# Export to CSV
def export_to_csv():
    exporter = CDVLExporter(
        input_file="./data/videos.jsonl",
        output_file="./data/videos.csv",
        columns=["id", "title", "filename"]  # None for all columns
    )
    success = exporter.export()
    print(f"Exported: {success}")

# Run
asyncio.run(crawl())
asyncio.run(download())
generate_site()
export_to_csv()

Development

To set up a development environment:

# Clone the repository
git clone https://github.com/slhck/cdvl-crawler.git
cd cdvl-crawler

# Install dependencies (including dev dependencies)
uv sync --dev

Running Tests

# Run all tests
uv run pytest tests/ -v

# Run specific test file
uv run pytest tests/test_utils.py -v

# Run with coverage (if pytest-cov is installed)
uv run pytest tests/ --cov=cdvl_crawler

Linting and Formatting

# Check code style
uv run ruff check src/ tests/

# Auto-fix issues
uv run ruff check --fix src/ tests/

# Format code
uv run ruff format src/ tests/

Type Checking

uv run ty check

License

The MIT License (MIT)

Copyright (c) 2025 Werner Robitza

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Metadata

Release files for cdvl-crawler 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cdvl-crawler 0.8.0
File Size Uploaded Provenance Actions
cdvl_crawler-0.8.0.tar.gz 43.1 kB Without provenance Details for cdvl_crawler-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cdvl-crawler 0.8.0
File Interpreter ABI Platform Actions
cdvl_crawler-0.8.0-py3-none-any.whl Python 3 none any Details for cdvl_crawler-0.8.0-py3-none-any.whl

Total release size: 90.6 kB

Release files / cdvl_crawler-0.8.0.tar.gz

Download URL cdvl_crawler-0.8.0.tar.gz
Size 43.1 kB
Tags Source
SHA-256 checksum
How to use checksums
630743d3f9255cf39dcd4b3f461eff42062add14874ea24b969bc42e11bc1690
BLAKE2b-256 checksum
How to use checksums
dee30eb1f1ba73efe9ee24fb6d52576644b20f5870d3a1d197fb1ab00ad33522
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / cdvl_crawler-0.8.0-py3-none-any.whl

Download URL cdvl_crawler-0.8.0-py3-none-any.whl
Size 47.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
59d6ae7c401164c9ac8f62840b5863b16dd82d882e8d501045390da52a0a4d04
BLAKE2b-256 checksum
How to use checksums
61355c700a27945d09f9a3d460bfa206dbbf0fc23224d8f3ae7299675aac3838
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for cdvl-crawler 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page