virtuale-dl 0.1.0

Download materials from Virtuale (Moodle) courses using secure cookie handling

Virtuale Course Materials Downloader

This script downloads downloadable files from a course on Virtuale (Moodle) using your authenticated browser session cookie.

What It Does

• Connects to a Virtuale course page
• Scans course resource/folder pages for downloadable files
• Downloads all discovered files to a target folder
• Overwrites existing files when the filename is the same

Requirements

• Python 3.10+
• requests package
• keyring package (optional, strongly recommended for secure credential storage)

Install dependencies:

python -m pip install requests keyring

Install For Other Users

Option A: install directly from GitHub (fastest)

Recommended for end users that just want the command:

pipx install "git+https://github.com/g-mainardi/scraping-virtuale.git"

Or with plain pip:

python -m pip install "git+https://github.com/g-mainardi/scraping-virtuale.git"

Then verify:

virtuale-dl --help

Option B: install from PyPI (after you publish)

pipx install virtuale-dl
# or
python -m pip install virtuale-dl

Publish The Package (Maintainer)

Option A: GitHub Actions (recommended)

This repository includes an automated workflow at:

• .github/workflows/release.yml

Behavior:

• Push tag v* (example: v0.1.0) -> build + publish to PyPI
• Manual run (workflow_dispatch) -> build + publish to TestPyPI

Required one-time setup:

1. Create package projects on TestPyPI and PyPI (virtuale-dl)
2. In each index, configure Trusted Publisher with owner g-mainardi, repository scraping-virtuale, and workflow release.yml
3. If you add environment restrictions in the publisher settings, create matching GitHub environments first

Release commands:

# Test release (manual from GitHub UI):
# Actions -> Release Python Package -> Run workflow

# Production release:
git tag v0.1.0
git push origin v0.1.0

Option B: manual twine upload

One-time setup:

python -m pip install --upgrade build twine
python -m twine check --help

For each release:

# 1) bump version in pyproject.toml
# 2) build
python -m build

# 3) validate artifacts
python -m twine check dist/*

# 4) publish to TestPyPI first (recommended)
python -m twine upload --repository testpypi dist/*

# 5) publish to PyPI
python -m twine upload dist/*

After publishing, users can install with pipx install virtuale-dl.

Security: Credential Storage

IMPORTANT: Never pass cookies on the command line to production systems or AI agents. This script uses a secure fallback chain:

1. System Keyring (Windows Credential Manager) - most secure, recommended ✅
2. VIRTUALE_MOODLE_SESSION environment variable
3. --moodle-session CLI argument (for special cases only)
4. Interactive prompt with getpass (no echo to terminal)

Recommended Setup: Keyring (One-time)

Store your cookie securely using Python's keyring package:

# Get fresh cookie from browser (see "How to Get MoodleSession" below)
# Then store it securely:
python -m keyring set virtuale-moodle moodle YOUR_COOKIE_VALUE

This stores your cookie in Windows Credential Manager (encrypted, OS-managed).

After one-time setup, you can run:

python scraping_virtuale.py IRS
# No credential needed! Script retrieves it from keyring automatically.

This is ideal for AI Agents - they can invoke the tool without handling any secrets!

Easier: Use the Helper Script

Instead of manually copying the cookie and running the keyring command, use the included helper:

./refresh-virtuale-cookie.cmd

This script will:

1. Guide you to DevTools & the Cookies tab
2. Ask for the cookie (hidden from terminal for security)
3. Store it securely in Windows Credential Manager
4. Clear sensitive data from memory

First-time setup:

./refresh-virtuale-cookie.cmd
# Follow the interactive prompts -> done!

When cookie expires (next day or after logout):

./refresh-virtuale-cookie.cmd
# Get fresh cookie -> done!

If automatic authentication fails (for example: No MoodleSession found in keyring/env or The cookie is invalid or expired), run ./refresh-virtuale-cookie.cmd again and then retry virtuale-dl.

Then use the downloader normally:

virtuale-dl IRS

Alternative setup using environment variable (less secure)

# Set environment variable (consider adding to PowerShell profile for persistence):
$env:VIRTUALE_MOODLE_SESSION = "YOUR_COOKIE_VALUE"
python scraping_virtuale.py IRS

Script Files

• scraping_virtuale.py - main downloader
• course_ids.json - course code -> Moodle course ID mapping
• refresh-virtuale-cookie.ps1 - local helper that stores MoodleSession in keyring
• refresh-virtuale-cookie.cmd - Windows launcher for the local PowerShell helper
• sync-refresh-cookie-global.ps1 - copies local refresh scripts to your global bin folder

Keep Global Script In Sync

If you edit the local repo scripts and also want your globally installed command updated, run:

powershell -NoProfile -ExecutionPolicy Bypass -File .\sync-refresh-cookie-global.ps1

Default source files:

• ./refresh-virtuale-cookie.ps1
• ./refresh-virtuale-cookie.cmd

Default global target folder:

• %USERPROFILE%\\bin

Example with custom destination:

powershell -NoProfile -ExecutionPolicy Bypass -File .\sync-refresh-cookie-global.ps1 -InstallDir "C:\\tools\\bin"

CLI Usage

virtuale-dl SELECTED_COURSE [OUTPUT_DIR] [--moodle-session COOKIE] [--course-ids-file PATH] [--max-pages N] [--debug-scan]

Positional Arguments

• SELECTED_COURSE (required): one of:
  • IRS, ASMD, PM, SAP, SPE, DS, old-ASMD, old-PM
• OUTPUT_DIR (optional): destination folder for downloaded files

If OUTPUT_DIR is omitted, the script uses automatic defaults:

• Standard courses: <BASE_DIR>/<SELECTED_COURSE>/materials
• Historical: old-ASMD -> <BASE_DIR>/ASMD/old_materials, old-PM -> <BASE_DIR>/PM/old_materials

<BASE_DIR> is resolved as:

1. VIRTUALE_OUTPUT_BASE_DIR environment variable (if set)
2. Otherwise, your user home directory

Missing folders are created automatically.

Options

• --moodle-session (optional): MoodleSession cookie value. If omitted, script tries keyring → env var → prompt.
• --course-ids-file (optional): path to JSON file with course IDs (default: built-in course IDs, or course_ids.json if present next to script)
• --max-pages (optional, default: 300): max number of resource/folder pages to crawl
• --debug-scan (optional): print crawler progress details

Course IDs Configuration

The script reads course mappings from course_ids.json.

Example format:

{
  "IRS": "72659",
  "ASMD": "71106",
  "old-ASMD": "59758"
}

Provide a different mapping with --course-ids-file PATH.

How to Get MoodleSession Cookie

1. Log in to Virtuale in your browser
2. Open browser DevTools (F12)
3. Go to Storage/Application → Cookies → https://virtuale.unibo.it
4. Find MoodleSession cookie
5. Copy its value (long alphanumeric string)
6. Store it: python -m keyring set virtuale-moodle moodle <VALUE>

Cookie expires after browser session/inactivity. If you get auth errors, repeat the above steps to get a fresh one.

Usage Examples

After keyring setup (recommended):

# Most basic - retrieves cookie from keyring automatically
virtuale-dl IRS

With optional custom output:

virtuale-dl PM ~/Downloads/PM_Materials

With debug output:

virtuale-dl SAP --max-pages 120 --debug-scan

With environment variable (if keyring not available):

$env:VIRTUALE_MOODLE_SESSION = "cookie_value"
python scraping_virtuale.py IRS

With CLI argument (not recommended, only for special cases):

python scraping_virtuale.py IRS --moodle-session "cookie_value"

Typical Output

On success:

• Authenticated for course ...
• Found N downloadable file link(s).
• Downloaded: ...
• Download completed. Saved N file(s) in: ...

On invalid/expired cookie:

• Error: The cookie is invalid or expired. You need a fresh one.
• Fix: run ./refresh-virtuale-cookie.cmd and then run virtuale-dl IRS again.

Notes

• The script only follows Virtuale links
• It discovers files through resource and folder pages
• Non-download HTML pages are skipped during download
• If a file with the same name already exists, it is replaced.