baltimore-camden 0.8.0

Python toolkit for the Baltimore City Mayor's Office of Performance and Innovation (OPI).

Camden

OPI's Python toolkit for civic data, automation, and reporting.

Camden gives analysts and data staff in the Baltimore City Mayor's Office of Performance and Innovation (OPI) one consistent, safe, repeatable way to do the things we do every week: pull SharePoint lists into pandas, query city databases, fetch Open Baltimore layers, geocode addresses, send report emails, and post to Teams.

Part of the Baltimore civic foundations

Camden is one of three independently versioned foundations OPI maintains for Baltimore civic software:

• camden (this repo) — Python data SDK (baltimore-camden): Graph/SharePoint, Open Baltimore, city databases, geocoding, provenance, delivery.
• patapsco — platform / integration point (baltimore-patapsco + @city-of-baltimore/patapsco): request-id & error-envelope wire contract, runtime, auth, typed frontend client, app scaffold.
• bromo / Baltimore Design System — TypeScript presentation layer (@city-of-baltimore/bromo): tokens, theme, shell, accessible components.

How the pieces fit together: patapsco/docs/foundations.md is the canonical architecture overview. The shared naming rule (baltimore-<name> on PyPI, @city-of-baltimore/<name> on npm) is ADR-0005.

Contents

• Part of the Baltimore civic foundations
• Philosophy
• What Camden supports
• Installation
• Configuration
• Quickstart
• Coming from r4OPI
• Development
• Roadmap
• Contributing
• License

Philosophy

Camden complements OPI's internal R package; it does not replace it.

• R remains welcome for statistical analysis, exploratory work, and existing R/Shiny assets.
• Python is the preferred default for new shared, repeatable, production-facing, integrated, or automated workflows.
• Camden exists to lower the ramp: reusable helpers, copy-pasteable recipes, and shared standards, so each analyst doesn't hand-roll their own auth, connection strings, and API calls.

Design principles: plain-English function names, predictable return types (pandas in, pandas out), small composable functions, no magic, no hard-coded credentials, and everything configurable without editing code.

What Camden supports

Area	Module	What you get
Microsoft Graph auth	camden.graph	GraphAuth (app-only or signed-in) and a hardened GraphClient — retries, paging, and $batch built in
SharePoint & OneDrive	camden.sharepoint	Lists ↔ DataFrames with where= filters (non-indexed columns fall back gracefully), display-name labels, choice validation, person lookups, describe() diagnostics; schema admin (create_list/delete_list, typed column builders, add_column/delete_column); document libraries by path (ls, mkdir, versions, recursive download, upload, delete); file transfer by pasted link; extension-aware read_sharepoint() / write_sharepoint()
Email	camden.mail	send_mail() through Graph (attachments, HTML, cc/bcc) — no ad hoc SMTP
Teams	camden.teams	Post to channels by team/channel name, list team members/rosters, or fire a webhook from unattended jobs
Planner	camden.planner	Plans/buckets/tasks as DataFrames; create and update tasks (buckets by name, assignees by email) with Planner's If-Match etag dance handled for you
Notifications	camden.notify	Provider-neutral send() with Graph-mail, SendGrid, and Teams-webhook transports; sent/skipped/failed statuses instead of exceptions; "all" or "failover" strategies
City databases	camden.db	Named registry (SQL Server + Postgres/PostGIS via kind), read_sql()/read_table() → DataFrames, write_sql() bulk writes, read_postgis() → GeoDataFrames, SQLAlchemy URLs, sql_in()
Open Baltimore	camden.opendata	search_datasets() finds datasets; get_layer()/get_table()/download_layer()/iter_layer_rows() fetch them (parameterized, paginated, retried, logged); provenance built in: describe_layer(), assert_fields() schema-drift guards, where_since() incremental clauses, and a manifest_for() sidecar recording exactly what query produced every cached file
Census/ACS	camden.census	acs() Baltimore City estimates at county/tract/block-group level, friendly variable names, numeric columns, join-ready geoid
Geocoding	camden.geocoding	City ESRI locator with Census fallback, address scrubbing; batch geocode_frame() dedupes, rate-limits, and logs progress
Spatial recipes	camden.geo	spatial_join() (which neighborhood is each point in — accepts plain lon/lat DataFrames), buffer_feet() (no accidental degree buffers), points_from_frame(), reproject() with named CRS constants
Excel reports	camden.excel	write_excel(): branded headers, frozen rows, autofilter, currency/percent/date formats — one call from DataFrame to presentable workbook
Report templates	camden.reporting	html_report() (branded email-ready HTML from DataFrames + text) and excel_report() (About sheet with title/timestamp/run id + one sheet per section)
Validation & diffs	camden.validate	check_frame() — required columns/dtypes, nulls, uniqueness, allowed values, all problems in one teaching error; diff_frames() — keyed add/update/delete plans that power list sync
Job runtime	camden.runtime	configure_logging() (UTC, run-id correlation, quiet deps), log_duration() timing on success and failure, safe_summary() config logging with secrets redacted
City utilities	camden.dates, camden.formats, camden.branding	Fiscal years, dollar formatting, CCF→gallons, OPI brand palette

What Camden is not: a stats library, a dashboarding framework, or a place for one-off analysis code. If a helper is only useful to one project, it belongs in that project.

Installation

Camden supports Python 3.13 and 3.14. The repository also pins the family-wide Node.js 24 tooling baseline in .nvmrc; Camden itself ships no JavaScript runtime or npm package.

Releases are published to PyPI as baltimore-camden from a matching vX.Y.Z GitHub release through OIDC trusted publishing. Consumer applications pin a released version rather than a moving Git branch.

# with uv (preferred)
uv add "baltimore-camden[all]==0.8.0"

# with pip
pip install "baltimore-camden[all]==0.8.0"

Extras keep the core light — install only what you need:

Extra	Adds	Needed for
(none)	httpx, msal, pandas, pyarrow, pydantic	Graph, SharePoint, mail, Teams, Open Baltimore tables/Parquet, geocoding, utilities
[db]	pyodbc, SQLAlchemy	camden.db SQL Server profiles
[postgres]	psycopg, SQLAlchemy	camden.db Postgres/PostGIS profiles
[excel]	openpyxl	camden.excel styled workbooks, .xlsx writes in write_sharepoint
[geo]	geopandas	camden.opendata layers, spatial file reading/writing
[all]	everything above	—

This is a private repo: authenticate with gh auth login or a GitHub token.

Configuration

Camden reads CAMDEN_* environment variables (or a local .env file — see .env.example). Every value has an OPI default, and the legacy variables opi_graph_id / opi_graph_secret still work, so most existing machines need zero setup.

Variable	Default	Purpose
CAMDEN_CLIENT_ID	legacy opi_graph_id	Entra app registration ID
CAMDEN_CLIENT_SECRET	legacy opi_graph_secret	Client secret (app-only auth)
CAMDEN_TENANT	bmore.onmicrosoft.com	Entra tenant
CAMDEN_DEFAULT_SITE_URL	OPI SharePoint site	Site used when none is given
CAMDEN_MAIL_SENDER	—	From-mailbox for app-only email
CAMDEN_SENDGRID_API_KEY	—	Enables the SendGrid notification transport
CAMDEN_CENSUS_API_KEY	—	Census API key for camden.census (free signup)
CAMDEN_SQL_DRIVER	auto-detected	ODBC driver name
CAMDEN_DATABASES_FILE	—	TOML file extending the database registry
CAMDEN_CACHE_DIR	~/.cache/camden	Where cached open-data pulls live
CAMDEN_TIMEOUT	30	HTTP timeout (seconds) for outbound requests
CAMDEN_GRAPH_MAX_RETRIES	4	Retry budget for transient Graph failures
CAMDEN_GRAPH_BASE_URL	Graph v1.0	Microsoft Graph endpoint (rarely changed)
CAMDEN_GRAPH_SCOPES	.default	OAuth scopes requested for Graph tokens

Credentials never live in code, and Settings never prints secrets. Any function also accepts an explicit camden.Settings object when you need per-call overrides; inspect the resolved configuration with camden.get_settings().

Quickstart

from baltimore import camden

# SharePoint lists <-> pandas
site = camden.get_site()                        # the OPI site by default
updates = site.get_list("Book of Business - Weekly Updates")
on_track = updates.to_df(where="fields/Status eq 'On track'", labels="display")
updates.choices("Status")                       # valid options before you write
updates.append(new_rows_df)                     # person columns: use site.user_lookup_id(email)

# Sync a tracker safely: dry-run the plan, then apply (deletes are opt-in;
# columns you don't manage — like user-written notes — are never touched)
print(updates.diff(weekly_df, key="Title").summary())   # add 3, update 2, delete 0, unchanged 41
updates.sync(weekly_df, key="Title")

# Paste a link, get data — and write it back the same way
df = camden.read_sharepoint("https://bmore.sharepoint.com/:x:/r/sites/opi/...xlsx")
camden.write_sharepoint(df, "weekly.xlsx", folder_link)   # DataFrame -> xlsx -> SharePoint

# Document libraries, by path
docs = site.drive()                             # the default "Documents" library
docs.ls("General", recursive=True)              # name/path/kind/size/modified
docs.download_dir("General/Reports", "reports/")
docs.upload("weekly.xlsx", "General/Reports")

# City databases -> DataFrames (parameterize with ?, never f-strings)
permits = camden.read_sql(
    "SELECT * FROM dbo.Permits WHERE issued_date >= ?", "citistat", params=["2026-01-01"]
)
neighborhoods = camden.read_table("dbo.Neighborhoods", "citistat")
camden.write_sql(
    camden.check_frame(scores_df, columns={"score": "number"}, no_nulls=["agency"]),
    "dbo.Scores", "opi-sandbox", if_exists="append",
)
conn = camden.connect("citistat")               # raw connection when you need one

# Open Baltimore: search the catalog, then pull any dataset by name, page URL, or id
camden.search_datasets("parking fines")         # title, dataset_id, snippet, modified
hoods = camden.get_shape("neighborhoods")       # GeoDataFrame, EPSG:4326
vacants = camden.get_layer(                     # paste a dataset page straight from the catalog
    "https://data.baltimorecity.gov/datasets/8ed6565f93a04455ae6bd5bcb7d272db_19/explore"
)
camden.describe_layer("real_property")          # provenance: rows, last edit, owner
camden.assert_fields("vbns", {"NoticeDate": "Date"})   # fail fast on schema drift
camden.layer_fields("real_property")            # inspect the schema before filtering
camden.layer_count("real_property")             # size up a big pull first (~238k)
fines = camden.get_table(                       # JSON rows only — no geometry, much faster
    "d2a2330d6a374ad39a24a0d7f7b58f19_0",       # parking fines: ~11M rows
    where="ViolFine > 500",
    fields=["Citation", "Description", "ViolFine", "ViolDate"],
)
camden.download_layer(                          # ginormous pulls: stream to Parquet, flat memory
    "d2a2330d6a374ad39a24a0d7f7b58f19_0", "fines.parquet", where="ViolDate >= DATE '2026-01-01'"
)  # then pd.read_parquet(...) — or pl.scan_parquet(...) if you prefer polars
fines = camden.cached_table(                    # or cache it: downloads at most once per max_age
    "d2a2330d6a374ad39a24a0d7f7b58f19_0",
    where=camden.where_since("ViolDate", "2026-01-01"),  # incremental pulls
    max_age="6h",
)  # cache_layer(...) returns the Parquet path instead; every download writes a
   # manifest sidecar — camden.manifest_for(path) says exactly what query produced
   # the file — and iter_layer_rows() streams row-by-row

# Geocoding + spatial tagging
located = camden.geocode_frame(df["address"], fallback=True, rate_limit_per_second=5)
tagged = camden.spatial_join(located, hoods, keep=["Name"])   # neighborhood per address

# Styled Excel in one call (branded header, filters, number formats)
camden.write_excel(summary_df, "weekly.xlsx", currency=["TotalFines"], percent=["ShareResolved"])

# Email a report; post to Teams
camden.send_mail(to="[e-mail]", subject="Weekly VBN report",
                 body="Attached.", attachments=["vbns.xlsx"])
camden.send_channel_message("OPI", "Book of Business", "Reporting period is <b>open</b>.")

# City utilities
camden.fiscal_year_label("2026-07-01")          # "FY27"
camden.week_bounds("2026-07-08")                # Fri-Thu reporting week: (Jul 3, Jul 9)
camden.dollars(121324520.45)                    # "$121,324,520.45"

Teams note: Graph only allows channel messages from signed-in users (delegated auth). For unattended/scheduled jobs, use camden.teams.send_webhook() with a channel's Workflows URL — no credentials required.

Coming from r4OPI

r4OPI	Camden
sharepointLogin()	GraphAuth() (automatic inside every client)
sharepointConnect()	camden.get_site()
sharedWithMe(url)	camden.sharepoint.download_file(url) — direct API, no browser
sqlConnection("citistat")	camden.connect("citistat")
ordTbl()	camden.db.ordered_select(conn, table) / ordered_columns()
getGeoShapeFromOB("neighborhoods")	camden.get_shape("neighborhoods")
getOBdatasets("vbns")	camden.get_dataset("vbns")
cityGeocoder(addr) / censusGeocoder(addr)	camden.geocode(addr) / camden.geocode(addr, source="census"); batches: camden.geocode_frame(addresses)
scrubAddress(addr)	camden.scrub_address(addr)
whatFY(d) / whatFY(d, abbrv=TRUE)	camden.fiscal_year(d) / camden.fiscal_year_label(d)
dollarWithCents(x)	camden.dollars(x)
ccfToGallons(x)	camden.ccf_to_gallons(x)
vectorToSql(v)	camden.sql_in(v)
opiColors()	camden.opi_colors() / camden.opi_palette()

New in Camden (no r4OPI equivalent): send_mail, send_channel_message, send_webhook, upload_file, read_sharepoint, list update/delete, SharePoint URL parsing, shared_with_me() listing.

Development

One command contract for humans, CI, and agents — Taskfile.yml (install go-task):

git clone https://github.com/city-of-baltimore/camden && cd camden
task setup      # dev environment from the committed uv.lock
task check      # everything a PR must pass: lint, types, tests + coverage gate
task fmt        # auto-format and fix lint findings
task ci         # exactly what CI runs
task docs-serve # live-preview the documentation site (mkdocs-material + mkdocstrings)
uvx pre-commit install   # lint + types on every commit

Unit tests run offline with no credentials. Live-tenant checks are opt-in: put credentials in .env and run task integration (each check skips individually when its target isn't configured — see .env.example).

See AGENTS.md for architecture, conventions, and testing patterns, and CONTRIBUTING.md for the pull-request workflow.

Roadmap

Direction lives in ROADMAP.md; execution is tracked with GitHub milestones and issues. Highlights: Excel read/write helpers, geospatial recipes (joins, buffers, CRS), Postgres/PostGIS support, and standard report outputs.

Contributing

Contributions are welcome — new helpers, recipes, docs, and bug reports. Branch, open a PR to main, and CI must pass. If you have an idea but aren't sure where it fits, open an issue and ask.

License

Distributed under the MIT License. See LICENSE for more information.

Maintainers

Mayor's Office of Performance and Innovation (OPI).