openroland-survey-core

The MIT openroland-survey-core distribution contains the canonical survey model, database schema, import/normalization, and validation APIs shared by the Stonex Cube, RW5, and JXL extraction libraries.

The AGPL siscadro-survey-server distribution in server/ contains FastAPI, web orchestration, authentication, and frontend integration and depends on this core package.

Status

This package currently provides the base foundation layer only:

• The immutable SurveyPointRecord canonical record and its supporting types (SourceMetadata, ParseIssue, ExtractionResult, ImportSummary, XlsxSummary).
• A deterministic three-decimal millimetre coordinate identity (openroland_survey.coordinates).
• The SQLAlchemy target schema and database engine/session helpers (openroland_survey.models, openroland_survey.database).
• The single-source-file import orchestration service, including Stereo70/openroland-crs and pyproj-based CRS transformation to WGS84 (openroland_survey.services).
• The common XLSX writer (openroland_survey.xlsx).
• Optional DXF writer (openroland_survey.dxf; install the [cad] extra).
• The SurveyPointExtractor protocol and entry-point discovery (openroland_survey.extractors), including process_file_to_all (import
  • XLSX + DXF for one source file).

The Cube, RW5, and JXL format adapters, the unified CLI ([cli] extra), the FastAPI/Material UI web application ([web] extra), and the survey-points deployment project are implemented in later development phases. Importing openroland_survey.cli, openroland_survey.web, or openroland_survey.jobs today raises an actionable error explaining that the required extra is missing or that the feature is not implemented yet.

Canonical model

openroland_survey.records.SurveyPointRecord is the one in-memory shape every format adapter (Cube, RW5, JXL) produces. north, east, and height are required Decimal values in metres; every other field is optional and None when a source format does not supply it. Unmapped source-specific values are preserved in source_values instead of being dropped.

Coordinate identity

Stored identity is exact: two points share a key when their north, east, and height values are equal after rounding to three decimal places (one millimetre) using half-up rounding. openroland_survey.coordinates.coordinate_key() implements this once, using Decimal arithmetic built from the value's string representation so binary floating-point noise never changes the identity. Non-finite values (missing, NaN, infinity) are rejected with NonFiniteCoordinateError.

Import matching is wider: when linking an observation to an existing row, a neighbor within ±1 mm on every axis (MATCH_TOLERANCE_MM) is reused. Untimed library imports (kind=imported / added) may also link to a timed measured neighbor with the same plan position and height within LIBRARY_HEIGHT_TOLERANCE_MM (20 mm). Measured observations may take over a library row's stored identity when they arrive second. Use openroland-survey-core merge-near-duplicates --database PATH (dry run by default) to collapse library clones already present in a database.

Database schema

openroland_survey.models defines one SQLAlchemy Base with schema version 2. New databases are GeoPackages:

• source_files: one row per imported source file version, unique on (path, sha256) so a byte-identical reimport is idempotent while a modified file at the same path is tracked as a new version.
• source_file_issues: one row per parse/import diagnostic recorded for a source file version (severity, optional record_id, message). Created additively on existing schema version 2 databases by ensure_schema.
• survey_points: one row per unique millimetre coordinate, unique on (north_mm, east_mm, height_mm). The same table is the GIS feature layer: column geom holds a GeoPackageBinary POINT in EPSG:3844 with axis order XY = (east, north). Scalar latitude / longitude remain ordinary columns for the Leaflet web map.
• survey_point_sources: the many-to-many association between points and source files, recorded as a full ORM class (not a bare table) so it can carry occurrence_count and the normalized source observation JSON.
• schema_metadata: a small key/value table recording the schema version.
• gpkg_spatial_ref_sys, gpkg_contents, gpkg_geometry_columns: standard GeoPackage catalogs registering survey_points as an EPSG:3844 features layer.

openroland_survey.database creates engines with SQLite WAL mode and foreign keys enabled, creates missing tables with Base.metadata.create_all() (never dropping existing tables), registers GeoPackage metadata, and rejects an existing target whose tables are missing expected columns or whose schema version does not match. Schema version 1 databases must be copied with:

openroland-survey-core migrate-database --source OLD.sqlite3 --destination NEW.gpkg

Migration builds geom from plan coordinates and fills missing WGS84 columns with the openroland-crs package (openroland-crs.stereo70_to_etrs89). openroland-crs is declared as a normal PyPI dependency. Editable sibling installs remain a local development option but are not required by package metadata or public builds.

Open the resulting .gpkg in QGIS and add the survey_points layer (EPSG:3844). The web UI continues to plot WGS84 latitude/longitude.

Import merge rules

openroland_survey.services.import_source() imports one ExtractionResult in a single database transaction:

• A point's stored coordinate identity is never changed once created.
• An incoming observation links to an existing point when each of north_mm, east_mm, and height_mm differs by at most 1 mm; if several neighbors match, the closest by L1 millimetre distance wins (then lowest id). Otherwise a new row is inserted with the incoming exact key.
• Within one source file, equal-coordinate records pick a primary observation by quality: timed points beat untimed ones; non-imported kinds beat imported (and legacy method="Imported"); kind rank is control/gps/base > auto > unknown > added/imported.
• A None canonical field is filled from an incoming non-None value.
• Collection kind may be upgraded when a higher-ranked observation arrives. Legacy method="Imported" may be cleared when a better observation arrives; new imports leave method empty for imported Type.
• An existing non-None canonical field (other than the kind/Imported method rules above) is never overwritten with a conflicting value.
• Every incoming observation is preserved in the association's observations_json, including values that lost the conflict above.
• Records with equal coordinates within the same source file increment occurrence_count on one association row instead of creating duplicates.

Cube .pd extraction sets kind from Point.PointType, GPS linkage, and GPSCoordinate.RecordMode (PointType 2 added, 3 imported; RecordMode 0/2 gps, 1 control, 3 auto; other GPS-less rows are kind=imported with empty method). Schema version 3 adds the nullable survey_points.kind column; ensure_schema upgrades version 2 GeoPackages additively and backfills from stored observations.

Latitude/longitude are taken directly from the record when present; otherwise they are transformed from (east, north) using pyproj with always_xy=True and the supplied source CRS. A record that has neither geographic coordinates nor a usable source CRS raises UnmappableCoordinatesError, so a source is never imported with points silently missing from the map.

GPS height sanity check (optional)

When enabled (Convert → Sanity-check GPS heights against DEM, or --height-check on CLI; on by default), EPSG:3844 imports compare each primary observation's MN75 height against Copernicus DEM GLO-30 near the point (kind=base records are skipped). Stereo70 plan coordinates and normal heights are converted through openroland-crs (ROMGEO grid) to ellipsoidal height, then to EGM2008 for comparison with the DEM.

Behaviour:

• On by default for each import that writes to the database; turn it off per job on the Convert page or with --no-height-check.
• Before import, required Copernicus tiles and the EGM2008 geoid grid are downloaded into the user cache (platformdirs: openroland-survey-core/height_check). If any required tile or grid cannot be retrieved, the import aborts with no points written.
• After tiles are available, points import normally. Per-point PASS/WARN/FAIL/INCONCLUSIVE results are recorded as source-file issues; height mismatches do not block import.
• Non-EPSG:3844 imports skip the check and record one warning issue.

This is a coarse sanity check against a 30 m surface model, not survey certification.

Frontend (Material UI + Leaflet)

frontend/ contains the React, Material UI, and Leaflet single-page application served by openroland_survey.web (see above). It is a standalone Vite/TypeScript project; building or testing it does not require the Python package to be installed, and installing the base, [cli], or [web] Python extras never runs pnpm.

The application has four pages, reachable from the app bar navigation:

• Convert exposes the CLI conversion workflows (file-to-xlsx, file-to-database, file-to-all, directory-to-xlsx, directory-to-database) behind source-kind/target-kind segmented controls that show only the fields relevant to the current combination (format override, output path/directory, recursive, source CRS, include-KeyedIn, overwrite, fail-fast). Submitting a job polls GET /api/jobs/{id} about once a second, showing counters, the current source path, per-file errors, and parse-issue counts until the job reaches a terminal status; a finished single-XLSX job gets a direct download button, and a finished database or To DB + XLSX + DXF job refreshes the app bar's point/source counts. The all-target mode writes {stem}.xlsx and {stem}.dxf beside the source (requires [cad]).
• Sources is a filtered, keyset-paginated table of imported source file versions (GET /api/sources) with a detail drawer showing full metadata, persisted parse issues, and that source's paged points. All parse issues across the database are also available via GET /api/issues and the CLI list-issues command.
• Points is a filtered, keyset-paginated table of canonical points (GET /api/points) plus an "Around a point" panel that accepts either an existing point ID or explicit north/east coordinates, a radius, and an optional height tolerance (POST /api/points/near), showing ascending-distance results. Selecting any point opens a detail drawer with every contributing source and its observation values.
• Map renders every point with WGS84 coordinates on OpenStreetMap tiles (GET /api/map/points), fed page by page into a chunk-loading Leaflet marker cluster. A bottom-right layers control can switch the base map to QGIS WMS layers discovered via same-origin GET /api/qgis/layers (tiles via GET /api/qgis/wms when the backend proxies MapProxy). Configure SISCADRO_QGIS_INTERNAL_URL and SISCADRO_MAPPROXY_INTERNAL_URL on the server; the browser never needs those hosts. It shows load progress with cancel/reload controls, fits the view to the first loaded batch while preserving later pan/zoom, colors markers by fix/quality status, and opens the same point detail drawer when a marker is selected.

The shell adds accessible labels, keyboard-operable controls, a shared error boundary, and Material UI snackbars for API outcomes.

Frontend development

cd frontend
pnpm install
pnpm run dev

pnpm run dev proxies /api/... requests to http://127.0.0.1:8000, so run python -m openroland_survey.web --database ... (or openroland-survey-core serve --database ... with the [cli] extra) alongside it. Only OpenStreetMap tile requests use the network; every other asset is bundled locally.

Frontend checks and packaging

pnpm run lint
pnpm run test
pnpm run build

pnpm run build writes to frontend/dist/. Copy that output into openroland_survey/static/ before building the Python package so the FastAPI application's GET / and SPA-fallback routes have something to serve and so python -m build packages the compiled assets into the wheel (see MANIFEST.in's recursive-include openroland_survey/static *):

make frontend-static

Or rebuild and start the local UI in one step (defaults: D:/prog/__py_libs__/openroland-survey-core/playground/production.gpkg on 127.0.0.1:8765):

make serve
make serve DATABASE=path/to.gpkg PORT=8765

If HOST:PORT is already in use, make serve asks whether to kill the holding process, switch to another free port, or quit. Non-interactive helpers: make serve SERVE_FLAGS=--kill or make serve SERVE_FLAGS=--alt-port.

Manual equivalent:

pnpm --dir frontend run build
python scripts/sync_frontend_static.py

Vitest/React Testing Library tests cover API error-message formatting, Convert-page field visibility and payload building per source/target mode, job-polling termination at a terminal status, keyset paged-table row accumulation, "Around a point" form validation, and the map feed's page accumulation and stale-request cancellation.

Docker (survey app)

See docker/README.md. Local stack:

.\venv\Scripts\Activate.ps1
cd docker
copy docker-compose.local.yml.example docker-compose.local.yml
cd ..
make d-local

make d-local builds the frontend and private/survey wheels on the host, then starts Compose (app installs from docker/wheels/).

• App: http://localhost:8000

Optional QGIS WMS base maps come from the separate qgis-provider project (CadPlatf/qgis-provider). Start the provider lab first, then the survey app:

cd D:\prog\CadPlatf\qgis-provider
make serve

cd D:\prog\__py_libs__\openroland-survey-core
copy .env.local.example .env.local   # optional
make serve

make serve optionally loads repo-root .env.local (then .env). Missing file or vars is fine (OpenStreetMap only). Shell env still wins. Example host defaults: MapProxy :9081, config API :9500.

make d-local reuses the same .env.local when present (env_file, optional). Compose overrides the URL values with Docker DNS (http://mapproxy/..., http://qgis:5000) so host localhost entries do not leak into the container. Shared flags such as SISCADRO_QGIS_PROXY_ENABLED come from the file when set.

The browser uses /api/qgis/layers and /api/qgis/wms only. See docker/README.md, deploy/production/README.md, and docker-compose.local.yml.example.

Installation

python -m pip install -e .

Base installation depends only on attrs, openpyxl, pyproj, and sqlalchemy. It does not install Click, FastAPI, Pydantic, or Uvicorn.

Install the optional extras once their features are implemented:

python -m pip install -e .[cli]
python -m pip install -e .[web]
python -m pip install -e .[cad]
python -m pip install -e .[cli,web,cad]

The [cad] extra installs ezdxf and enables file-to-all / Convert To DB + XLSX + DXF (and openroland_survey.dxf in general).

Development

Start by creating a virtual environment and installing the development dependencies:

python -m venv venv
venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e .[dev,cli,web,cad]

Or, if you have make available:

make init-d

Run the standard checks before committing:

make delint
make lint
make typecheck
make test