ArcGIS Item Dependency Management

Overview

This tool builds and maintains an organization-wide ArcGIS item dependency graph, showing what Web Maps, Dashboards, Feature Services, and other items depend on each other. You can query the graph by item ID or portal search string and receive CSV, Excel, interactive HTML, and GML outputs — making it safe to audit, migrate, and clean up portal content without breaking downstream items.

Quick Start

1. Install

Standard Python / Mac / Linux:

pip install arcgis-item-graph

ArcGIS Pro (Windows) — uses Pro's bundled Python:

"%PROGRAMFILES%\ArcGIS\Pro\bin\Python\Scripts\pip.exe" install arcgis-item-graph

Windows one-click installer (end-user deployment):

Your GIS admin provides a tool folder containing install.bat, install.ps1, launch_query.py, and a pre-configured config/config.yaml. Double-click install.bat to install and launch.

What it does automatically:

1. Detects conda (Miniconda / Anaconda / ArcGIS Pro)
2. Downloads and installs Miniconda silently if conda is not found (~75 MB, one-time)
3. Registers Miniconda's condabin in your user PATH so conda is available in new terminals (skipped if conda was already on PATH)
4. Accepts conda channel Terms of Service for the three default Anaconda channels (required for conda 24.x+ in scripted environments)
5. Creates an arcgis-graph conda environment with Python 3.11
6. Installs the latest ArcGIS API for Python and arcgis-item-graph
7. Launches the interactive query tool

Subsequent runs are fast — existing environments and packages are reused, and pip upgrades to the latest version automatically.

2. Configure

arcgis-graph setup

The wizard prompts for your portal URL, authentication method (named profile or username/password), and output preferences. Your credentials are never stored in config.yaml — they go to a gitignored .env file.

3. Build the graph (run once)

arcgis-graph create

This crawls your portal, saves a dependency graph (outputs/graph.gml), and builds a SQLite metadata cache (outputs/meta.sqlite) used by the health subcommand and governance risk scoring. For large organizations (5,000+ items) it can take 30–90 minutes.

4. Query

arcgis-graph query --item-id abc123
arcgis-graph query --search "owner:jsmith type:Dashboard"

Prerequisites

• Python 3.9 or later
• ArcGIS API for Python 2.4.0 or later (arcgis>=2.4.0)

Setup

See Quick Start above for installation and configuration.

For development setup, see For Contributors below.

Configuration

config/config.yaml controls authentication and all run-time settings. Two auth options are available:

Option 1 — Named ArcGIS profile (recommended for GIS admins)

Set the auth.profile key to the name of a saved ArcGIS credential profile:

auth:
  profile: "my_portal_profile"   # created via arcgis.gis.GIS(profile=...)
  verify_cert: true

Run python -c "from arcgis.gis import GIS; GIS(profile='my_portal_profile')" to verify the profile name is correct.

Option 2 — Environment variables

Leave auth.profile blank and create a .env file in the project root:

ARCGIS_URL=https://your-portal/portal
ARCGIS_USER=your_username
ARCGIS_PASSWORD=your_password

The CLI loads .env automatically when a profile is not set.

Other settings

Usage

All commands are run via the unified CLI entry point:

python -m cli [--config /path/to/config.yaml] {create,update,query,triage,remap,setup,health} [options]

Build the graph (run once)

Crawls the entire portal and saves a GML snapshot. For large organizations (5,000+ items) this can take 30–90 minutes.

python -m cli create

Keep the graph current (run on a schedule)

Finds items modified since the last run and merges changes into the existing GML. Designed for a daily cron job.

python -m cli update

Query the graph

# Query by item ID
python -m cli query --item-id abc123

# Query by portal search string
python -m cli query --search "owner:jsmith type:Dashboard"

# Request specific output formats for a single run
python -m cli query --item-id abc123 --format excel
python -m cli query --item-id abc123 --format csv --format html

# Probe URL-type dependencies for HTTP liveness and cache results in meta.sqlite
arcgis-graph query --item-id abc123 --service-check

# Probe ALL URL-type nodes in the full org graph without running a query
arcgis-graph query --service-check

# Add governance signals from a triage run to the HTML dashboard
arcgis-graph query --item-id abc123 --triage-manifest outputs/reports/triage/<timestamp>/triage_manifest.json

# Use a different config file
python -m cli --config /path/to/other/config.yaml query --item-id abc123

When --config points somewhere other than the standard config/config.yaml layout, the CLI first looks for a .env file next to the config file itself, falling back to the project-root convention if none is found there.

--service-check sends an HTTP GET to each unresolved URL-type dependency found in the query result (e.g., hosted service URLs that are not registered as portal items). Results are stored in outputs/meta.sqlite and surfaced in the Excel Dependency Edges sheet (dep_status column) and in the health report. The probe is opt-in and silently skipped if the metadata cache does not exist.

When run without --item-id or --search, --service-check operates as a standalone org-wide probe: it reads every URL-type node from the full GML graph, probes each in parallel, and prints a summary table. The graph must exist (run arcgis-graph create first); no query outputs are written.

--triage-manifest <path> adds the governance signals a prior arcgis-graph triage run recorded — version-managed status and shared update groups — to each matching service URL node in the HTML dashboard's item summary panel. It's ignored (and logged) if the file is missing or unreadable, so the dashboard always renders with or without it.

Interactive dashboard (live server mode)

Add --serve to any query command to start a local HTTP server and open the dashboard in your browser automatically:

arcgis-graph query --item-id abc123 --serve
arcgis-graph query --search "owner:jsmith" --serve

# Use a different port if 8765 is taken
arcgis-graph query --item-id abc123 --serve --port 9000

The server runs at http://localhost:8765/ by default. It exposes:

• GET / — the interactive HTML dashboard
• GET /query?id=<item_id> — live re-query from inside the dashboard (click any node)
• POST /export/excel — download an Excel report built from the graph currently shown in the dashboard

Press Ctrl+C in the terminal to stop the server.

Note: Opening the saved .html file directly (file://...) will not work for node re-queries or Excel exports because those features require the live server. Always use --serve for the full interactive experience.

Run python -m cli --help or python -m cli <command> --help for the full list of options and overrides.

Triage (migration planning)

Identify the highest-traffic consumer items in your portal and classify the services they depend on — prioritized by view count and dependency breadth. Designed for migration planning and portal housekeeping.

arcgis-graph triage                    # rank top 50 items (config default)
arcgis-graph triage --top-n 20         # rank top 20
arcgis-graph triage --min-dependents 2 # only items with 2+ service dependencies
arcgis-graph triage --deep             # Tier 3 layer introspection + Version Management capability check (slower, more accurate)
arcgis-graph triage --no-usage-stats   # rank by dependency count only (skip portal API)
arcgis-graph triage --force-refresh    # bypass the triage_cache_hours window and re-run

Outputs to outputs/reports/triage/<timestamp>/:

Excel workbook sheets:

Note: data_source_type classification uses URL pattern matching (Tier 1), service JSON inspection (Tier 2), and optionally layer-level inspection (Tier 3 with --deep). Enterprise ArcGIS Server services backed by an Enterprise Geodatabase are classified as egdb; hosted services as hosted_relational; file-based data as fgdb.

version_managed note: whether a branch-versioned (named-version) egdb service has its Version Management capability enabled is a separate, independent check from data_source_type — most egdb services are classified from the URL alone (Tier 1) with no service-JSON fetch, so this check only runs with --deep. Without --deep, version_managed is always null. With --deep: true/false means the capability was confirmed present/absent; null means the check itself failed (e.g. network error) — not the same as "absent." An egdb service silently republished without this capability breaks named-version-aware apps with no signal anywhere else in this tool's output.

shared_update_groups note: only checked (with --deep) for services where version_managed is true and the service resolved to an actual Portal item — sharing/group membership doesn't exist for an unregistered service URL. null means not checked or no resolvable item; an empty result means checked, no update-capable groups found; otherwise it lists the titles of groups whose members are all version administrators for that layer. That grant does not transfer or change when the item's owner changes — treat any non-empty result as a manual governance review item before reassigning ownership, not just informational.

triage_manifest.json per-item dependency fields: each entry in the items array carries three distinct dependency-shaped arrays — dependencies (services this item consumes, classified by data_source_type/tier), consumers (portal items with a graph edge pointing to this item), and item_dependencies (portal items this item points to that are not service endpoints — e.g. a Web AppBuilder app's edge to its underlying Web Map). dependency_count counts all outgoing graph edges regardless of type, so it may exceed len(dependencies) + len(item_dependencies) combined only if a successor could not be resolved as either a service or a portal item at all (rare — e.g. a dangling reference). Each item_dependencies entry is shaped {"id", "title", "type"}, matching consumers' shape.

Health check (broken references and orphan candidates)

After running arcgis-graph create, a SQLite metadata cache (outputs/meta.sqlite) is built alongside the GML graph. Use the health subcommand to query it for org-wide quality issues:

arcgis-graph health

Prints a summary of broken node references (items in the graph that no longer exist in the portal) and orphan candidates (items with no dependents and low recent activity). Writes a health_report_<timestamp>.xlsx workbook to outputs/ with two sheets:

If arcgis-graph query --service-check has been run, the health output also includes two URL service sections:

• Inaccessible Services — URL-type dependencies that returned a non-200 status or SSL error (inaccessible) or a connection/DNS failure (dead), printed in red.
• Live Services / No Portal Item — URL-type dependencies confirmed reachable (HTTP 200) but not registered as portal items (live_no_item), printed in yellow.

When no service URL probes are on record, a hint is printed to run arcgis-graph query --service-check.

Note: health requires the metadata cache. Run arcgis-graph create (or update) first to build outputs/meta.sqlite. If the cache is missing, health exits with status 1 — safe to check in CI/cron via $?.

Remap item references

When an item is replaced (e.g., a service migrated from one URL to another), use remap to update all dependent items that reference the old item:

# Preview what would change (dry run)
arcgis-graph remap --from-id <old-item-id> --to-id <new-item-id> --preview

# Execute the remap
arcgis-graph remap --from-id <old-item-id> --to-id <new-item-id>

# Remap all broken nodes in the health cache (bulk repair workflow)
arcgis-graph remap --from-health-report

# Skip the confirmation prompt (for scripted/CI use)
arcgis-graph remap --from-health-report --yes

The --from-health-report flag reads broken node IDs directly from the metadata cache and walks you through a remap for each one. A JSON manifest is written to outputs/ recording every item updated, the old and new references, and success/failure status.

Confirmation gate: --from-health-report and --force make live writes to production portal items, so remap prompts you to type yes before proceeding (skipped for --preview/dry-run). Pass --yes/-y to bypass the prompt for scripted or CI runs.

Note: remap exits with status 1 if --from-id/--to-id are missing (and neither --from-health-report is set) or if the metadata cache is missing for --from-health-report. If the cache exists but has no broken nodes to remap, that's a no-op and remap exits 0.

Experience Builder note: For Web Experience (ExB) app dependents, remap also patches the source item ID embedded in the app's config/config.json resource file — Item.remap_data() alone does not update it (Esri KB 000033170), which otherwise leaves the cloned app uneditable in the ExB editor. If this patch fails, the item still reports success (the main-JSON remap did succeed) but prints a config/config.json patch FAILED warning — check logs for that item.

Resilience note: A transient failure (e.g. a dropped portal connection) partway through one from_id's remap no longer aborts the whole batch. That from_id is recorded as failed and remap continues to the next one — important for --from-health-report, which can walk many broken nodes in a single run.

Summary (migration dependency closure)

For the ArcGIS 11.1 → 11.5 migration, summary reads the confirmed migration remapping reference CSV, computes the full dependency closure (both directions, unbounded) over the existing item graph for every row not marked External/NA/Verify, and flags EGDB/federated-service risk and cross-domain dependents along the way:

arcgis-graph summary --remapping-csv docs/remapping-reference_latest.csv

Writes dependency_graph_summary.json to outputs/ (or --output <path>) — a machine-readable dependency map (per-item depends_on/consumers refs tagged item_id/url, data_source_type, egdb_risk_flag, depended_on_by_migrating_item) consumed by the Content_Migration repo's ordering_risk.py/group_migration.py to sequence and group the actual migration work. Two top-level fields cover CSV rows the closure could not place: excluded_count (rows marked External/NA/Verify, deliberately skipped) and unresolved (rows whose ID was never found as a node in the existing item graph at all).

Coverage note (standalone server services): the item graph is built by crawling from Web Map/Dashboard/Web Experience/Web Mapping Application/Hub Site Application/StoryMap items only (CONSUMER_TYPES) and following their dependencies — it does not query Feature/Map/Image Services directly. A service referenced by one of those consumer items is discovered transitively and gets real depends_on/consumers data. A standalone service with no consumer item referencing it — common among Tier 3 (SD export + REST republish) services migrated straight at the server level — is never discovered at all, and its CSV row lands in unresolved, not in items with an empty dependency list. Do not read an unresolved row, or a row missing from items altogether, as "confirmed no dependencies." It means the opposite: this tool never got to check. Verify those rows by another means (e.g. server logs, REST service usage reports) before treating them as safe to migrate.

Production runs of .github/workflows/prod-triage.yml now build and upload dependency_graph_summary.json as a GitHub Actions artifact (dependency-summary-<run-id>) on every triage run. Content_Migration downloads it from there rather than running summary itself.

Shared Deployment (Team Use)

For team environments, point paths.gml_file and paths.output_dir at a UNC share so all users read from the same graph without running create individually.

1. Admin: initial setup

# On the admin machine, configure config.yaml to point at the share:
#   paths.gml_file: "\\\\server\\share\\arcgis-graph\\graph.gml"
#   paths.output_dir: "\\\\server\\share\\arcgis-graph\\outputs"

arcgis-graph create   # one-time full crawl (~30-60 min for large orgs)

2. Automation: scheduled updates

Windows Task Scheduler (hourly):

arcgis-graph update --config \\server\share\arcgis-graph\config.yaml --skip-if-fresh

Linux/macOS cron (hourly):

0 * * * *  arcgis-graph update --config /mnt/share/arcgis-graph/config.yaml --skip-if-fresh

--skip-if-fresh prevents double-runs if automation fires while a manual update is in progress.

3. Users: install and run

Option A — Windows installer (no Python required):

Distribute the tool folder (install.bat, install.ps1, launch_query.py, config/) to users. They double-click install.bat. The installer handles everything: conda, packages, and launch.

The tool folder can live on a UNC share — users can run it directly from there:

\\server\share\arcgis-graph\install.bat

Option B — CLI (Python already installed):

Users point their local config.yaml at the share paths and run:

arcgis-graph query --item-id <id>

If the same item was queried within 24 hours, the cached outputs are returned instantly. Use --force-refresh to bypass the cache and re-run the query.

Freshness thresholds (configurable)

cache:
  update_warn_hours: 24    # Warn in query if graph is older than this (24 = daily, the default)
  query_cache_hours: 24    # Reuse cached query outputs within this window

Output files

All output files land in the directory set by paths.output_dir (default: outputs/).

Project structure

arcgis_item_graph/   Core library
  creator.py           Full org crawl → graph.gml
  updater.py           Incremental update since last run
  query.py             Direction-aware graph traversal + live hydration
  graph_io.py          Sole entry point for create_dependency_graph()/load_from_file()/
                        write_to_file() — always normalizes URL-type node ids first
  reporter.py          DataFrame → CSV, Excel (All Items, Dependency Edges, Broken Deps, External Refs)
  visualizer.py        Jinja2 + Cytoscape.js → interactive HTML dashboard
  cache.py             SQLite metadata cache (outputs/meta.sqlite) — broken nodes, orphan detection, risk scoring
  parsers.py           Custom JSON parsers (View Admin, Dashboard, ExB, Web Map) that augment graph edges
  risk.py              Governance risk scoring (RiskScore, score_item) — 0-100 score, green/yellow/red tier
  remapper.py          ItemGraphRemapper — remap item references across all dependents
  auditor.py           ItemDependencyAuditor — audit accuracy via dependent_to() API
  triage.py            ItemTriageRunner — rank high-traffic items and classify service dependencies
  utils.py             Shared helpers (URL classification, batch fetch, retry)
  templates/           graph.html — Jinja2 dashboard template (shipped in the wheel)
  lib/                 Vendored cytoscape.js, dagre, cytoscape-dagre — inlined into generated HTML
cli/                 Unified CLI entry point (python -m cli ...)
config/              config.example.yaml template — copy to config.yaml and fill in credentials
docs/                Documentation and design plans
outputs/             Generated output files (gitignored); outputs/meta.sqlite is the metadata cache
tests/               Unit and integration tests (pytest) — 607 tests

The CLI uses Rich for terminal output. Progress bars, error panels, and completion summaries all go through arcgis_item_graph/console.py — the single file in the project that imports Rich. Library modules (creator, updater, triage, etc.) remain UI-free and communicate with the CLI via on_progress/on_warning/on_step callback kwargs.

For Contributors

1. Clone the repository

git clone https://github.com/Global-Geospatial-IT/expert-octo-spoon.git
cd expert-octo-spoon

2. Install in editable mode with dev dependencies

pip install -e ".[dev]"

3. Activate the git hooks

git config core.hooksPath .githooks

This enables both the commit-msg hook (rejects AI co-authorship trailers) and the pre-commit hook (blocks staging the local-only development-workflow paths listed in CONTRIBUTING.md).

4. Create your configuration file

cp config/config.example.yaml config/config.yaml
# or just run: arcgis-graph setup

Running tests

pytest tests/ -v

Performance & Architecture Notes

Graph Traversal

The query BFS uses collections.deque for O(1) popleft (O(V+E) total). Seed items not found in the cached GML file are fetched live in parallel via ThreadPoolExecutor (default 10 workers, configurable via fetch_workers on ItemGraphQuery).

Traversal direction is controlled by query.traversal_direction in config:

• upstream (default) — follows contained_by() edges: finds items that reference the queried item. Answers "what breaks if X is removed?" — the correct mode for migration impact analysis.
• downstream — follows contains() edges: finds items the queried item depends on. Answers "what does X need to function?"
• both — runs separate upstream and downstream passes. No cross-directional contamination (forward deps of upstream-reached nodes are not included).

Update Hydration

ItemGraphUpdater hydrates all cached graph nodes concurrently using ThreadPoolExecutor (default 10 workers, configurable via hydration_workers). Graph mutations (node removal) happen serially on the main thread after all fetches complete. The modified-items search enforces a max_items cap (defaults to create.max_items from config) and warns when results may be truncated.

Timestamps

All timestamps are stored in milliseconds with sub-second precision (int(t.timestamp() * 1000)).

Excel Reports

ItemGraphReporter.to_excel() builds all four sheets from a single pass through to_dataframe() — node.contains() is called once per node.

The dep_status column in the Dependency Edges sheet reflects the state of each dependency reference:

URL-type dep nodes show the richer statuses (live_no_item, inaccessible, dead, unchecked) only when the metadata cache is available (i.e., after running arcgis-graph create and then arcgis-graph query --service-check). Without the cache they fall back to broken.

The Broken Dependencies sheet includes rows where dep_status is broken, inaccessible, or dead.

The dashboard's POST /export/excel builds its workbook from the graph already loaded in the browser. When the dashboard was rendered with the metadata cache available (the normal --serve case), each URL-type node's health is embedded in that graph data, so the exported dep_status column includes the same live_no_item, inaccessible, and dead statuses as the CLI export. If a node has no recorded health (e.g. the cache was unavailable at render time), dep_status falls back to healthy, broken, or not_in_result based on hydration state alone, matching the previous behavior.