Sphinx Extension: Repository Manager

About

This Sphinx extension by Xsolla Backend [XBE] automates the management of multiple documentation repositories as part of building a larger, unified documentation system. It facilitates multithreaded cloning and updating of external repositories specified in a YAML manifest file before Sphinx builds.

📜 See the XBE docgen source code and demo doc production site heavily making use of this extension. Here, you may also find tips for how to utilize this extension to its greatest capabilities.

See how it works or quickstart below >>

Installation

This guide assumes you have a basic understanding of Sphinx and RST

Add to Existing Project

1. Install the extension via pip:

    pip install sphinx-repo-manager
   

2. Add extension to your project's docs/source/conf.py (example template):

   extensions = [ "sphinx_repo_manager" ] ,  # https://pypi.org/project/sphinx-repo-manager
   

3. Ensure a docs/.env file exists next to your Makefile -> set REPO_AUTH_TOKEN=

4. Create a docs/repo_manifest.yml (example template) next to your Makefile

   • 💡 Optionally, set the manifest max_workers_local to a higher number for faster local builds [even 30 is ok for high-end machines!]

Once setup, sphinx-build as normal (typically via make html next to your Makefile)!

Tips

• Windows user? You may want to unlock your max char paths by running tools/admin-enable-long-file-paths.ps1 as admin
• Editing the manifest?
  • Consider purging your docs/source/_repos-available and docs/source/content dirs
• Want speedier build iterations?
  • Test bumping up your max_workers_local counts - even significantly higher - for high-end machines!

Demos

Minimal Demo

1. Clone the source repo for a demo:

• Minimal build architecture begins at at docs/
• repo_manifest.yml contains a minimal sphinx_demo_doc repo to be cloned

Production Demo

Alternately, see sphinx_repo_manager used by Xsolla Backend at a production-grade level:

• Source Code
• Production Site

How it Works

1. repo_manifest.yml lists repositories with their respective clone URLs [and optional rules].
2. docs/source/ creates _repos-available (src repos) and content (symlinked) dirs.
3. Upon running sphinx-build (commonly via make html), the extension either clones or updates each repo defined within the manifest.
4. Source clones will sparse checkout and symlink to the content dir, allowing for flexibility such as custom entry points and custom names (such as for shorter url slugs).
5. All repos in the manifest will be organized in a monolithic doc.

💡 If you want to store local content (eg, static .rst), add it to source/_source-docs/

💡 The only RST file expected for your monolithic repo is the index.rst file (next to your conf.py)

⌛ 5 local workers (default) will take only ~50s to process 30 repos with default manifest settings

Tests

Confirmed compatability with:

• Windows 11 via PowerShell 7, WSL2 (bash)
• Ubuntu 22.04 via ReadTheDocs (RTD) CI, Docker Desktop
• Python 3.10, 3.12
• Sphinx 7.3.7, 8.1.3

Questions?

Join the Xsolla Backend official Discord guild!

License

MIT