spacescans

Link time-referenced geolocation data to the spatial and contextual exposome.

spacescans links location episodes to environmental and contextual exposures. A location episode is a participant identifier, a location (longitude and latitude) and the first and last day spent there; a residential history is a series of them. For every episode spacescans computes a buffer-weighted value of each exposure source (or the distance to the nearest feature) and weights it by the days the episode overlaps each source period. By default the result has one row per participant: each exposure averaged over all of the participant's episodes, weighted by days. One row per location episode is available with output_level="episode".

Exposures come from a catalog of 11 public data products (each site downloads them itself; see Getting the exposure data) or from datasets you register: tables keyed by census geography, rasters, and point, line or polygon features. The same engine runs behind the Python API, the command line, the R package in r/ and the SPACESCANS web application.

Install

pip install "spacescans[geo]"

The development version installs from GitHub:

pip install "spacescans[geo] @ git+https://github.com/IU-Ultraman/spacescans.git"

You install and import spacescans. (Before 0.3.0 the distribution was called spacescans-pipeline; it was never on PyPI.) It needs Python 3.10 to 3.13 on Linux or macOS. The base install is light; the extras add what each kind of source needs:

pip install "spacescans[geo,rda,nc]" covers every catalog source except TEMIS UV Exposure. An optional reader or pattern whose extra is missing raises MissingExtraError, which names the pip install to run.

Quickstart (no data needed)

spacescans quickstart --output-dir ./demo-out

The quickstart needs the geo extra. It runs one spatial-weighting step (C3) on synthetic sample data that comes with the package: 270 m buffers around 10 synthetic locations, intersected with three sample county polygons. It writes the spatial weights to ./demo-out/quickstart_county_weights.parquet. It does not link exposure values; the next section does.

Link location episodes (Python)

Your cohort

One row per location episode; a participant who moved has one row per location.

Every row needs a pid and a location: the cohort check refuses a file with a blank pid or a missing, non-numeric or out-of-range coordinate, or with an endDate before its startDate, before anything runs. A blank date is allowed, with a warning: that episode contributes no days. Only these five columns are written to disk, into a private (0700) run directory; other columns stay in your session. Cohort values (identifiers, dates, coordinates) never appear in messages, events, warnings or provenance.

Link

import pandas as pd
from spacescans import data
from spacescans.orchestrate import link

# Once: download NPS Soundscape Noise (1.2 GB) from the project mirror.
data.fetch(["noise"], data_dir="~/ss-data")

cohort = pd.DataFrame({
    "pid":       ["P1", "P1", "P2", "P3"],
    "startDate": ["2015-01-01", "2017-06-01", "2016-03-15", "2014-07-01"],
    "endDate":   ["2017-05-31", "2019-12-31", "2019-12-31", "2018-06-30"],
    "longitude": [-86.158, -86.131, -85.680, -87.630],
    "latitude":  [39.768, 39.790, 40.193, 41.880],
})

res = link(cohort, ["noise"], data_dir="~/ss-data")
res.exposures    # one row per participant: pid, l50dba_exi, l50dba_imp, l50dba_nat
res.coverage     # linkage rate of each variable: the share of participants with a value
res.features     # feature dictionary: what each output column is, with its unit
res.provenance   # versions, data receipts, cache keys and settings; no cohort values

by_episode = link(cohort, ["noise"], data_dir="~/ss-data", output_level="episode")

cohort can also be the path of a CSV file. Other arguments: buffer_m (buffer radius in meters, default 270), raster_res_m (default 25), cache (a directory that keeps spatial weights between calls, so further sources on the same geography reuse them), custom (registered datasets, below), on_error ("stop" or "continue" when a source fails), threads and memory_limit (DuckDB limits per step). spacescans.catalog.variables() lists the catalog keys; spacescans.catalog.coverage(cohort, keys) checks before a run which episodes fall inside each source's spatial and temporal coverage.

Participant level and episode level

With output_level="participant" (the default) the result has one row per participant id in the cohort, in order of first appearance: pid, then the requested variables, with no dates, coordinates or episode ids. For participant i and variable v:

x̄(i,v) = Σ_e Σ_t d(e,t) · z(e,t,v)  /  Σ_e Σ_t d(e,t)

where e runs over the participant's episodes, t over the source periods (year, two-week period, day; a single period for time-invariant sources) that overlap the episode, z(e,t,v) is the spatial estimate for the episode's location in that period (the buffer-weighted mean, or the distance to the nearest feature), and d(e,t) is the number of days of the episode in that period, first and last day included. Only periods with a defined estimate count, separately for each variable; a participant with no contributing day keeps their row, with missing values. This is the same as averaging the episode values, each weighted by its contributing days.

• Overlapping location episodes. When two episodes of one participant overlap in time, each contributes its own days: a calendar day that both cover counts once for each of the two locations, so the participant value is a mean over episode-days, not over calendar days. Trim the overlaps in your cohort if a day should count only once.
• Derived variables are derived from the participant's means: FARA Food Access's flags (such as LILATracts_1And10) are recoded from the participant's mean rates and urban share, so they stay 0 or 1, and each ACAG _nbm (non-biomass) column is the participant's mean of the species minus the participant's mean of its biomass part (_bm). At the episode level the same rules apply to each episode's means.

With output_level="episode" the result has one row per location episode, with episode_id (the 0-based row number of the episode in your input) and the variables.

Linkage rates are computed at the level of the output: the share of participants with a value, or the share of episodes in episode mode.

Register your own datasets

spacescans.custom keeps a library of datasets you add. Registration validates the files and prepares them once (normalized Parquet for tables, a checked grid and period index for rasters, GeoParquet in EPSG:5070 for features), stores them with their metadata (dataset.json), and every later linkage reuses the prepared files.

from pathlib import Path
from spacescans import custom

custom.register(["clinics.gpkg"], name="clinics", kind="features", temporal="static")
custom.register(sorted(Path("pm_biweekly").glob("*.tif")), name="pm_biweekly",
                kind="raster", temporal="biweekly")

custom.list_datasets()           # DatasetRecord: id, name, kind, temporal, variables, metadata
res = link(cohort, ["noise"], data_dir="~/ss-data", custom=["clinics", "pm_biweekly"])
custom.remove_dataset("pm_biweekly")

register() also takes key (a table's geography and key column, such as {"boundary": "tract", "column": "GEOID"}; for features optionally a year column or a layer), variables (name, unit and description of each value), metadata (source, version, description, license note, visibility, owner) and overwrite. A record has an id, a name unique in its library, the checksum of the original files and the spacescans version that prepared it. The library lives in root=, else $SPACESCANS_DATASETS_DIR, else the user data directory (platformdirs.user_data_dir("spacescans") followed by datasets). Each record's visibility ("private" or "deployment") is informational in the package; the web application enforces it, sharing a "deployment" dataset with every user of that deployment. get_dataset(name_or_id) returns one record. The reference is docs/custom-datasets.md.

custom also accepts in-memory definitions made with custom.define_polygon() and custom.define_raster(), as a mapping of variable key to definition.

Getting the exposure data

No exposure data comes with spacescans. spacescans data knows where each catalog variable's data comes from and what a complete copy looks like under the data directory (spacescans/data/manifest.json); it needs no extras.

spacescans data status ndi walkability --data-dir ~/ss-data   # present, missing, sizes, next step
spacescans data fetch noise tiger_proximity --data-dir ~/ss-data --dry-run
spacescans data fetch noise tiger_proximity --data-dir ~/ss-data   # resumable; re-run after a drop
spacescans data verify --data-dir ~/ss-data                   # sha256 of everything placed
spacescans data adopt --data-dir /existing/web/pipeline-data  # receipts for data placed earlier

• Files the project mirrors download from the project's OneDrive mirror (or --mirror URL|DIR, $SPACESCANS_DATA_MIRROR), resume after interruptions, are checked against their published sha256, and are recorded in <data-dir>/.spacescans/receipts/.
• EPA FAQSD Air Quality, TEMIS UV Exposure and ACAG PM2.5 Composition download from their providers once you accept the provider's terms in the call (--accept-terms ID; status and fetch --dry-run print each ID with its terms). VIIRS Nighttime Lights needs a manual download; status prints the instructions, license and attribution. Provider terms are never accepted on your behalf.
• The same functions are available from Python: spacescans.data.status(), fetch(), verify(), adopt(), terms(), import_files().

The catalog (every source covers the conterminous United States; the time-varying sources cover 2013–2019, the Neighborhood Deprivation Index 2012–2022):

Command line

export SPACESCANS_DATA_DIR=/path/to/exposome-data
export SPACESCANS_OUTPUT_DIR=/path/to/results

# Copy the editable template configs, then point their paths at your data
spacescans init-config --out ./configs

# Run the pipelines in dependency order
spacescans run ./configs/c3/county.yaml
spacescans run ./configs/c3/zcta5.yaml
spacescans run ./configs/c4/zbp.yaml            # depends on C3 zcta5
spacescans run ./configs/c4/cbp_fallback.yaml   # depends on C3 county + C4 zbp

A C4 config's time.output_grouping is patient (one row per participant, the default) or episode (one row per location episode); the templates state it. Paths resolve in three tiers, CLI flag > $SPACESCANS_DATA_DIR > YAML base_dir:, with ${SPACESCANS_DATA_DIR} expansion inside YAML paths.

How it works

Every linkage runs in two steps, both driven by YAML configs and the same DuckDB engine:

1. Spatial weighting (C3, needs geo): the share of each location's buffer that falls in each polygon or raster cell, or the distance from the location to the nearest line or feature. The weights depend only on the cohort's locations, the source geography, the buffer radius and the rasterization resolution, so they are computed once per link() call (and across calls with a cache directory) and reused by every further source on the same geography.
2. Exposure-value aggregation (C4, base install): the weights applied to each source period's values, weighted in turn by the days each episode overlaps the period, per episode or per participant.

C3 patterns

Distances to your own point, line or polygon features take one of two routes, which give the same exact nearest-feature distance:

• Registered features (custom.register(..., kind="features"), then link(custom=[...]); also R's ss_register_dataset() and the web application's feature uploads): the distances from each location to the nearest feature are computed at the start of each link (spacescans.custom.features, in EPSG:5070 meters) and aggregated by the static_areal or yearly_areal C4 pattern. No C3 step.
• Hand-written YAML run with spacescans run: the proximity pattern does both steps in one. Its config names the features (source.file: one vector or GeoParquet file, or one per year of time.years, or one file with source.year_col), the output column (exposure.value_cols, default distance_m) and optionally source.category_col (one column per category); distances are in meters in buffer.aea_crs (an Albers equal-area projection of the conterminous US) and time.output_grouping applies as in every C4 config.

C4 patterns

Limits

• Catalog sources cover the conterminous United States and, when they vary over time, at least 2013–2019.
• Each pipeline step runs as a single process, using about one processor core. Support for several cores is planned.
• The daily sources need memory roughly in proportion to the number of episodes: EPA FAQSD Air Quality and TEMIS UV Exposure ran out of memory in a 24 GiB container at 100,000 episodes. Measured peaks per source are in docs/memory.md.

R package

The R package in r/ (ss_link(), ss_fetch(), ss_register_dataset() and friends) runs this package in a separate Python process through a JSON-file bridge (docs/bridge-protocol.md). It is released with the same version as the Python package. Install it with remotes::install_github("IU-Ultraman/spacescans", subdir = "r"), then run ss_install() once; see its README.

Project layout

src/spacescans/
├── orchestrate/         link(): a cohort and variables in, exposures out; run dirs, cache, merge
├── catalog/             the catalog variables, feature dictionary, recipes and templates
├── custom/              registered datasets and uploaded values (tables, rasters, features)
├── data/                spacescans data: manifest, status, fetch, verify, adopt
├── cohort.py            cohort checks and date parsing
├── bridge/              python -m spacescans.bridge, the JSON-file protocol the R package uses
├── cli/                 run / quickstart / init-config / data
├── config_resolution.py 3-tier path resolution (CLI > env > YAML) + ${VAR} expansion
├── pipeline/            config loader, pattern registry and runner
├── models/              pydantic config schema
├── engine/              DuckDB aggregation engine
├── io/                  readers (Parquet, CSV, .Rda) and writers
├── geometry/            buffers, overlap, grid weights, proximity (geo)
├── transforms/          filter / recode / derive / date_parse / spatial_fix
├── linkage/             linkage-pattern implementations
├── plugins/             source-specific readers (extras-gated)
└── resources/           synthetic sample data and template configs

import spacescans loads only the base dependencies.

Development

pip install -e '.[all,dev]'
pytest                  # full suite (base + any extras present in the environment)
pytest -m "not extras"  # base-only tests
pytest -m geo           # geo-extra tests

Opt-in suites (real data, golden references, web parity) are described in docs/tier2-notes.md. Changes are listed in CHANGELOG.md; the release procedure is docs/release.md.

License

spacescans is released under the MIT License (LICENSE); the R package in r/ is released under the same license. The license covers the software only: each catalog source keeps its provider's terms, which spacescans data status prints.

Learn more at www.spacescans.com.