🎧 dj-soundcloud-digger

A local-first playlist workflow for DJs. Turn SoundCloud playlists, likes, and artist profiles into an auditionable, searchable library, then find the safest available path to download or buy each track.

dj-soundcloud-digger keeps the whole digging loop in one terminal application: collect a playlist, preview and filter it, mark decisions, match music already on disk, and follow verified store or download links. Your library and preferences stay on your machine, and checkout always stays in your hands.

The current implemented behavior, architecture, interfaces, data model, and security/privacy boundaries are maintained in PROJECT-SPECIFICATION.md.

dj-digger https://soundcloud.com/someone/sets/that-playlist

⚡ The workflow

• Collect complete playlists: SoundCloud API v2 reads playlists, likes, profiles, and individual tracks without relying on the page's finite rendered list. A saved-HTML fallback covers private and unlisted sources.
• Audition and narrow down: Preview, seek, search, filter, and sort tracks in the Textual interface without leaving the playlist.
• Remember every decision: Local playlists keep got it / skipped status across playlists, refreshes identify new additions, and a library scan finds tracks already on disk.
• Find the track, not just a link: Store, smart-link, and download-gate URLs are classified and expanded into useful Bandcamp, Beatport, shop, or download destinations.
• Acquire with guardrails: Download artist-provided files, resolve supported gates, verify Bandcamp products, or prepare a Beatport playlist. Ambiguous matches and checkout remain manual.
• Keep control locally: Playlists, decisions, credentials, and browser profiles remain on your machine; there is no project-hosted account or backend.

Additional capabilities

• API-first collection: A 300-track playlist can be read from SoundCloud in about three seconds using batch hydration. Following third-party purchase links takes as long as those providers take; a 484-track playlist is around a minute.
• Store and gate classification: Group links into Bandcamp, Beatport, Traxsource, JunoDownload, record shops, download gates, smart links, and direct SoundCloud downloads.
• In-memory audio preview: Stream, seek, prefetch upcoming tracks, and render block waveforms with a reactive audio meter, powered by miniaudio.
• Multi-playlist local library: Save, switch, refresh, and search playlists stored in ~/.local/share/dj-digger/digger.db.
• Cross-playlist track memory: A decision made for a SoundCloud track applies when that track appears in another playlist.
• 🔓 Download Gate Automation: Resolves follow-to-download gates (Hypeddit, ToneDen, GateRush, Droploud) through their supported download flows. SoundCloud, Instagram, YouTube and similar link steps are click-through markers: the app does not call their follow, like, repost, comment APIs or open external social links to simulate a click. A gate may receive the name or email configured in Settings when its manifest requires them. Spotify steps are reported like the other click-throughs: Hypeddit clears them through its own Spotify app, so no Spotify login is needed or used. CAPTCHA and unknown provider steps remain manual in the private Chromium profile. Gate automation remains subject to the provider's terms.
• 🔗 Link-Hub Expansion: A purchase link that turns out to be a list of shops rather than a download—an ampsuite release page, a gate running in smart-link mode—is opened, and the Bandcamp and Beatport links behind it are added to the track directly instead of a gate badge.
• 🛒 Store Purchase Assistance: An optional, user-triggered flow verifies Bandcamp additions and prepares Beatport tracks as an importable playlist. Login, playlist transfer, and checkout stay manual.
• 🆕 New Since Last Refresh: Refreshing a playlist marks whatever it gained with NEW and sorts it to the top.
• 📄 Saved-HTML Fallback: Fully supports saved HTML pages (Ctrl+S) for private or unlisted SoundCloud playlists.
• ⚙️ CLI & Non-Interactive Mode: Export playlists directly to JSON or CSV for automated pipelines and scripts.

📦 Installation

Recommended (via uv or pipx)

# Install with audio preview; store-cart support is included
uv tool install 'dj-soundcloud-digger[play]'

or with pipx:

pipx install 'dj-soundcloud-digger[play]'

From Source (Development)

git clone https://github.com/fbialogrecki/dj-soundcloud-digger.git
cd dj-soundcloud-digger
uv venv
uv pip install -e '.[play,dev]'

Run the working tree without installing or releasing anything:

uv run --extra play dj-digger

The browser is drawn on standard error, so 2>file would redirect the interface itself and leave you looking at a blank terminal. To keep a log while it is up, use --log-file:

uv run --extra play dj-digger --log-level DEBUG --log-file /tmp/dj-digger.log

To try it against a throwaway library instead of your real playlists, point the XDG directories somewhere temporary for that one run:

XDG_DATA_HOME=/tmp/dj-dev XDG_CONFIG_HOME=/tmp/dj-dev XDG_CACHE_HOME=/tmp/dj-dev uv run --extra play dj-digger

Requires Python 3.12 or newer.

Note on optional extras:

• play: Enables in-memory audio preview via miniaudio. Store-cart support is included. On the first c or C, the app asks before downloading its matching Chromium build automatically.

🔗 Supported Input Links

🖥️ Interactive TUI Workflow

Launching dj-digger opens an interactive Textual browser divided into three key areas:

1. Playlist Sidebar (Ctrl+B): Switch between saved playlists, add new links (a), or refresh existing playlists (r).
2. Track Table: Displays tracks with status badges (· untouched, ○ opened, ✓ got, ✗ skipped), position, artist/title, available store badges, genre, and duration.
3. Player & Waveform Bar: A four-row, 32-level waveform with a real-time VU level meter, over a one-row control strip: previous / play-pause / next buttons, track title, clock, a drag-to-set volume slider, and a close button.

▸ 0 all  1 soundcloud·18  2 bandcamp·12  3 gate·53      83/83 tracks · got 0 · skipped 4

Keybinding Reference

Press ? inside the TUI at any time to view the full grouped keybinding modal.

Track Navigation & Status Marks

Audio Preview Controls

Filtering, Stores & Library

🏷️ Store & Gate Categories

Links are parsed and categorized using strict domain-boundary matching:

Bandcamp carts and Beatport playlists

Pressing c or C does the product checks and the cart clicks in a hidden Chromium; nothing pops up while it works. A window appears only when there is something for you: the completed Bandcamp cart, or items left to finish by hand. That window is a separate browser carrying the same cookies, so the hidden session is never disturbed. A desktop display is needed for those moments only (WSL users need WSLg); without one the additions still stand and the result screen says the window could not be shown. If the matching browser build is absent, the app asks before downloading it in the background and then resumes the cart preflight. The app uses a dedicated persistent browser profile, separate from your everyday browser. A Bandcamp account is not required for its cookie-backed cart; opening a manual Bandcamp session in Settings is optional. Beatport login is never attempted there. The app never reads or fills your password, chooses a payment method, or completes checkout.

• c resolves the highlighted track, then adds it to Bandcamp or prepares it for a Beatport playlist.
• C resolves all currently visible, unhandled tracks, then asks for one batch confirmation. The active Bandcamp or Beatport filter limits the target store.
• When both Bandcamp and Beatport filters are explicitly active, each track is handled for both destinations: Bandcamp is added to its cart and Beatport is retained for the transfer playlist.
• With no store filter, Bandcamp is tried first. Beatport is used only when the exact track is genuinely unavailable for individual purchase on Bandcamp—not when Bandcamp fails technically.

Batch mode resolves every candidate first and shows exact products, prices, currencies, existing cart items, and skips before Bandcamp is changed. Bandcamp rows with seller-approved flexible pricing show their minimum; highlight one and press E to enter a higher value. Fixed-price rows cannot be edited. Bandcamp pages are rechecked immediately before each click. Ambiguous titles, version mismatches, changed prices or product IDs, CAPTCHA, and changed store UI stop the affected Bandcamp operation instead of guessing. Two reusable work tabs bound preflight; Bandcamp mutation is serial, and only its successful final cart tab remains open for format selection and checkout.

If a Bandcamp link moved or points only to an artist/label page, the app uses the site's visible autocomplete as a bounded fallback. It accepts only canonical, exact track matches and inspects at most three returned album pages; it never enters the CAPTCHA-protected full results page. A track sold only as part of a full album is reported as album-only instead of silently adding the whole album. After a click, verification uses the cart count, a visible removable row in the opened side cart, and one reload check, each on its own clock. An uncertain cart remains open for inspection and is never clicked again automatically; the app saves a screenshot and a redacted copy of the page under cart-diagnostics in its data folder (last ten kept) so a broken flow can be reported with the page that broke it. After two unverified clicks in one batch the app stops clicking: it opens the remaining products with Buy expanded and the price filled, asks you to press Add to cart yourself, and then checks the cart once. The result screen offers the same Finish in browser for anything left uncertain.

Beatport login and cart mutation are not automated: c/Shift+C prepare a playlist, and Shift+P opens the exact Beatport track pages in your everyday browser, where you are already logged in, so adding to the cart is one click each. The result screen creates a new Beatport playlist.txt in the playlist's download folder, copies its contents, creates a temporary Soundiiz import with Beatport already selected, and opens the returned review page. Confirm the matches and finish the transfer there; Soundiiz accepts up to 200 tracks per import. The saved file and clipboard remain available as a fallback. Promo prefixes, uploader names, preview markers and trailing label fields are removed before matching, including titles whose artist separator has no surrounding spaces. Exact Beatport track URLs are preferred in that file and replace stored release links so later imports can reuse them. Release links and blocked public pages are looked up for the exact title/remix and fall back to cleaned artist - title, which Soundiiz presents for review before writing the Beatport playlist. Already-exact numeric track URLs bypass Chromium. Old pro.beatport.com links are automatically rewritten and saved under www.beatport.com. In Beatport DJ, that playlist can then be added to the default cart in one action.

Only canonical Bandcamp and Beatport HTTPS domains are inspected. Custom artist domains remain outside this version. The feature depends on the stores' and Soundiiz's public import interface; use it in line with their terms. Showing the completed Bandcamp cart or opening its manual session needs a graphical session; WSL users need WSLg or another working display.

For a timestamped diagnostic log that does not interfere with Textual, run:

uv run --extra play dj-digger --log-level DEBUG --log-file /tmp/dj-digger-cart.log

🔐 Authentication for Artist Downloads

Some artist-provided SoundCloud downloads require a logged-in account even when the track is public. Run dj-digger auth login: an existing valid login or a readable Firefox session is used first, otherwise dj-digger opens a dedicated Playwright Chromium profile and waits up to five minutes for you to log in. Only the verified oauth_token cookie is copied to dj-digger; passwords and other cookies are not written to its credential file. If Chromium cannot start, the command offers a hidden token-paste fallback:

dj-digger auth login
dj-digger auth login --token YOUR_SOUNDCLOUD_OAUTH_TOKEN
dj-digger auth status
dj-digger auth logout

The TUI opens the same choice automatically when a download needs SoundCloud. Cancelling leaves the track untouched. A successful login recreates the API client after active downloads finish and retries the waiting track once.

SoundCloud's private browser profile lives in ~/.local/share/dj-digger/soundcloud-browser; its verified API credential lives in the owner-only ~/.config/dj-digger/auth.json (the standard XDG environment variables override both base directories). If SOUNDCLOUD_OAUTH_TOKEN is set, it deliberately overrides that file; an invalid value must be unset or updated before the CLI wizard can replace a login.

When Hypeddit or GateRush requires an email address, the TUI asks for a real name and email before it submits the gate. It explains who receives those data, rejects placeholders and malformed addresses, and retries only the downloads that were waiting for the profile. Cancelling sends no retry request.

Spotify steps on download gates

Hypeddit gates that show a Spotify step are handled like the other click-through steps: the gate clears that step through Hypeddit's own Spotify app and server session, so nothing dj-digger could do with your Spotify account would reach it. Releases before 1.0 asked for a Spotify developer app and stored a login in ~/.config/dj-digger/spotify.json; that file is no longer read, and you can delete it yourself.

When a gate ends up in the private Chromium profile (a refusal, a CAPTCHA, a provider login), a hidden browser walks the gate's own step slides for you: it ticks the follow and like links (closing the provider pages they open, unread), presses the Connect of a Spotify step - whose login popup comes back by itself once you have signed in to Spotify in that profile - fills the email slide with the address from Settings (and the name, when the gate asks for one), and presses Download. Only a step no program can do alone - a provider asking you to sign in, a CAPTCHA, a missing email or name - opens a window, where the same driver keeps walking the steps before and after the one that needs you. Sign in to Spotify there once and later gates finish out of sight. Nothing outside the Hypeddit page is clicked. Disable gate social actions in Settings to keep the program from reporting or clicking any social step at all. Gates requiring those actions will then remain manual.

🎧 In-Memory Streaming & Waveform Engine

• Zero-Latency Playback: Decodes audio chunks directly off the network socket in memory via miniaudio.
• Pre-Fetching & Gapless Transitions: While listening to a track, the next track's stream URL, waveform data, and initial MBs are buffered 20 seconds prior to track completion.
• SoundCloud Waveform Rendering: Renders SoundCloud's 1800-sample waveform data into a 4-row block ASCII visualizer with logarithmic loudness scaling.
• Reactive Audio VU Metering: Real-time RMS audio signal measurement rendered at 30 FPS.
• Remote SSH / Headless Systems: Automatically degrades to 4 FPS under TEXTUAL_ANIMATIONS=none for smooth operation over SSH connections.

🤖 Non-Interactive CLI & Automation

Add --no-tui to run dj-digger in batch mode for terminal pipelines, cron jobs, or export scripts:

# Export playlist links to CSV
dj-digger https://soundcloud.com/user/sets/playlist --no-tui -f csv -o playlist.csv

# Limit extraction to first 20 tracks and export to JSON
dj-digger https://soundcloud.com/user/likes -n 20 -f json -o likes.json

# Re-open a previously exported JSON playlist in the TUI browser
dj-digger open likes.json

# Open all Bandcamp links directly in browser from a saved playlist
dj-digger open likes.json --category bandcamp

🏗️ Project Architecture

                       ┌─────────────────────────┐
                       │  SoundCloud URL / HTML  │
                       └────────────┬────────────┘
                                    │
                         ┌──────────▼──────────┐
                         │ SoundCloudClient v2 │
                         │ (Resolve & Hydrate) │
                         └──────────┬──────────┘
                                    │
                         ┌──────────▼──────────┐
                         │ Link Categorizer    │
                         │ (Store & Gate Engine│
                         └──────────┬──────────┘
                                    │
           ┌────────────────────────┼────────────────────────┐
           │                        │                        │
┌──────────▼──────────┐  ┌──────────▼──────────┐  ┌──────────▼──────────┐
│   Textual TUI App   │  │  Playlist Library  │  │ In-Memory Player   │
│ (Interactive Grid)  │  │ (~/.../crates/)    │  │ (Miniaudio & VU)   │
└─────────────────────┘  └────────────────────┘  └────────────────────┘

🧪 Testing & Quality Assurance

The codebase includes an extensive offline test suite covering unit tests, API serialization, player buffering, link parsing, cart safety, and TUI reactive widgets.

# Run offline test suite (550+ tests, no network required)
uv run pytest

# Run live integration tests (verifies SoundCloud API v2 contract stability)
uv run pytest -m live

# Open public store pages read-only; never logs in or changes a cart
uv run pytest -m shop_live

📄 License

Distributed under the Apache License 2.0. See LICENSE for details.