Recip Slice

Documentation index · Current release plan

0.5.0 Beta development release (2026-09-11). Routine pytest is now a compact regression suite; heavyweight scientific and artifact campaigns run explicitly through python tools/validate_release.py. Large implementation files have been split behind stable compatibility facades, and no production Python module exceeds 800 lines. The reconstruction estimator, quadrature, native ABI, fixed-v4 schema, scientific defaults and product definitions are unchanged. This is a Beta development release; publication and final desktop/platform sign-off remain separate release gates.

Scientific release scope: uncalibrated reciprocal-space reconstruction, with CMOS Off and Background None by default. Signal remains background-inclusive; fitted background subtraction is experimental and may remove genuine diffuse signal. This is not a calibrated CMOS or isolated-elastic-intensity release. See the approved scope and remaining acceptance and the release plan and mandatory gates. Passing execution tests alone does not establish scientific accuracy.

Recip Slice reconstructs physically scaled crystallographic Zone Patterns from continuous-rotation electron-diffraction images processed with XDS. A Zone Pattern is a reciprocal-space layer viewed through a user-selected thickness: a practical zone-axis-like representation derived from the measured rotation dataset, not a simulated static diffraction experiment.

Each detector observation is converted to a scattering vector and mapped into a fixed crystal frame. New v4 results store fixed-thickness 2D Zone Patterns. Choose the slice half-thickness before reconstruction; changing it requires a new run. Display resolution, binning and centre hiding remain display-only. The beta GUI, CLI, validation tools, and normal viewer use fixed-thickness v4 results only. Historical v1/v2/v3 outputs and flat legacy arrays are not migrated or reprojected; they require a fresh reconstruction. The signed q_s volume exists only transiently while a requested slab is reconstructed and is never persisted.

The principal use is to inspect hk0, h0l, 0kl, offset-layer, and diagonal reciprocal-space sections from cRED/3D ED data while preserving quantitative provenance and sampling support.

Historical illustration: this screenshot shows the retired Mean display, not a current v4 product or validation of this release.

Scientific products

The normal viewer opens Intensity. Scientific results expose four primary products only:

• Intensity - slab-projected relative physical diffraction intensity, obtained by integrating local mass / measure through the selected reciprocal thickness;
• Signal - additive reciprocal-volume integral used for reflection/ROI integration (Intensity * dq_u * dq_v per displayed pixel);
• Coverage - reciprocal-space measure represented by the experiment;
• Reliability - local geometric/conditioning information for judging the estimate.

The quantitative evidence is accumulated as signed mass, reciprocal-space measure, and conditioning on a local 3D volume. Conceptually, local 3D intensity density is mass / measure; displayed Intensity is its integral through the selected q_s thickness, and Signal additionally integrates the in-plane reciprocal pixel area. Coverage and Reliability are diagnostics and are never intensity divisors.

Signal and Intensity are both floating-point products. Detector counts may start as integers, but background subtraction, Lorentz/reciprocal-volume weighting, fractional deposition and q-area integration produce fractional values and may produce signed residuals. Integer conversion would round weak signal and cannot represent NaN missing support. The viewer's automatic Intensity range is a display-only robust range; it never clips or converts the saved products.

Each SMV pixel is already integrated over detector solid angle, exposure, and one constant Delta phi wedge. For the supported cRED workflow, exposure is required to remain constant across the selected series; recip-slice never performs per-frame exposure normalization. When a frame header provides TIME, EXPOSURE_TIME, EXPOSURE, or COUNT_TIME, every read frame is checked against the first value. If exposure metadata is absent, the manifest records the explicit assumption that the acquisition used constant exposure. A rotation frame therefore measures in (solid angle, phi) coordinates, not directly in reciprocal-volume coordinates. Before deposition, signed Raw counts are multiplied once by the finite-cell normal Ewald crossing speed

v_n = |det(dq/dx,dq/dy,dq/dphi)| / |dq/dx x dq/dy|

which is the rotation-method Lorentz conversion from detector/angular measure to reciprocal-volume mass. Delta phi is not multiplied into the detector count again; it appears once in reciprocal measure through the full local Jacobian J * Delta phi. Consequently mass / measure recovers relative physical diffraction intensity rather than an inverse-crossing-speed-biased count density. At fixed wavelength the remaining k^2 factor is global. For flat detectors, pixels that cross the zero-Jacobian/spindle line use an analytic finite-pixel absolute-value correction rather than a singular centre-point sample. Signed detector input is retained. New reconstructions default to CMOS Off + Background None: ambiguous detector/specimen signal is retained. The products remain uncalibrated and background-inclusive; detector response, offsets, inelastic and dynamical scattering can bias their interpretation. No intensity threshold, morphological cleanup or zero clipping enters the Base. Optional estimated background subtraction is experimental, not a calibrated separation of those components.

Where Mean went: new results do not write Sum or Mean. Observation count or sampling weight is not reciprocal-space measure. The scientifically meaningful successor to Mean is therefore Intensity, normalized by reciprocal-space measure rather than by the number/weight of detector observations. Historical v1/v2 files are outside the beta application contract: update recip-slice and rebuild them instead of relying on a compatibility conversion.

Formal SNR or I/sigma(I) is not a current product. It will only be introduced if a defensible detector/statistical variance model is available.

The current physics gate generates synthetic detector frames from the ideal forward measure I_rel * k^2 dOmega * Delta phi (equivalently physical dI/dOmega * dOmega * Delta phi up to the fixed wavelength scale) rather than from the estimator's reciprocal-volume denominator. On a 260-frame, 1024 x 1024, 12-plane workload, the plane-wise median recovered/imposed intensity is within about 1.4e-6; the bulk 99th-percentile relative error is approximately 2.4e-6 to 4.9e-6. A historical native run before the finite-footprint change completed in 7.33 s at about 0.65 GB peak RSS. The independent finite-cell tensor reciprocal-measure audit remains within the declared 5e-4 tolerance. Synthetic checks establish the estimator and computational envelope; they do not replace the separate real-data and cross-platform beta release gate.

Fixed thickness and view controls

Thickness is fixed when reconstructing; the live Thickness selector is removed. Use Slice half-thickness in advanced Build settings, then reconstruct again to change it. Resolution, Bin, Hide centre, display scale, colour map, and orientation are View operations and do not change the quantitative Base.

Main outputs

Normal reconstructions use isolated run folders; the result name is the filename prefix:

<XDS directory>/recip_slices/<run folder>/<result name>_*

For each selected plane the authoritative scientific payload is one compressed *_pattern.npz containing Intensity, Coverage, Reliability, q_u, q_v, and the exact reciprocal-pixel area for bit-exact Signal recovery. New archives also carry the auxiliary 2D depth_support map, which records the fraction of the selected fixed slab represented at each in-plane cell. It is diagnostic only and does not rescale any scientific product. The v4 manifest records plane definitions, units, sampling/geometry provenance, quantitative-source/background semantics, the fixed thickness, and the absence of depth-adjustment capability. All four scientific products retain their existing precision and definitions.

New saves use lossless ZIP/DEFLATE level 1 without reducing numerical precision. For lower save-time CPU cost on a fast local disk, explicitly select --storage-compression none in reconstruct or tools/run_local_real_data.py. This writes ordinary uncompressed NPZ: identical scientific arrays, larger files. The GUI offers the same choice under Advanced settings → Storage, retaining compression as default and saving the choice in projects/presets/queued runs. Independent plane archives now save with up to four workers; the ready manifest is written only after every save succeeds. See result storage and measured performance. To compact an existing current-format result while keeping the original intact:

python -m recip_slice compact "/path/to/result" --output-dir "/path/to/result-compact"

The destination must be new and outside the source. Every decompressed NPZ member is hash-verified; calibration maps and provenance are copied unchanged. See result storage for measured sizes, validation and GUI latency. Existing result folders are never converted or deleted automatically. Historical v1/v2/v3 and flat-array results are not converted by the beta GUI; update recip-slice and rebuild them.

Optional CMOS and experimental background correction

Normal reconstruction exposes two separately selectable controls. CMOS correction is Auto CMOS or Off; Background is Beam + smooth inelastic or None. All four combinations are valid and are recorded independently in the manifest. The default is Off + None. A single bounded estimation pre-pass is shared when both are enabled; it is not an independent calibration measurement.

Auto CMOS currently abstains from self-estimating a detector-fixed map. Stationary specimen structure cannot be separated reliably from detector texture using one rotation series. Without an explicit PSF fraction, Auto leaves the detector signal unchanged. Independent dark/flat calibration import is planned; it is not an implemented GUI feature. Automatic spill/PSF inference is disabled: intrinsic reflection width cannot be separated from detector spill using one rotation series. A calibrated spill fraction may be supplied explicitly; it is recorded as an explicit opt-in and corrected with the signed bounded inverse for Y=(1-f)X+f H(X) using a native separable five-tap kernel. No zero clipping is used. Random shot/read noise is not blurred away and should ultimately be represented by uncertainty, not by a destructive denoiser.

Estimated subtraction (experimental) uses the existing beam + smooth inelastic model to estimate a beam-centred low-bandwidth background. The lower temporal reference is reduced to a low azimuthal-quantile radial profile, strongly smoothed and constrained to be non-increasing with |q|. The radial fit uses the configured quantile (default 0.2), not the minimum pixel, and is evaluated continuously between radial bands. It can still remove smooth specimen signal, including diffuse rings; preservation must be tested. A perfectly smooth isotropic specimen component can be fundamentally inseparable from unfiltered inelastic background; Recip Slice therefore does not claim to recover an energy-filtered diffraction pattern from unfiltered data. Saturated/overload direct- beam pixels remain missing quantitative evidence rather than being numerically reconstructed. The detector-fixed CMOS map and physical-background map are saved separately for inspection and provenance.

The background is refitted after fixed-pattern removal; broad specimen residuals are not added to its baseline. A falling detector level can relax subtraction per frame, with every additive shift saved in provenance. Brighter frames do not raise the fitted background automatically, so some positive residual may remain. Calibration rejects saturated/invalid observations. PSF inversion is available only for an explicitly supplied independent calibration and is bypassed where its stencil touches missing samples. Native and fallback use the same five-tap PSF and flux-conserving symmetric detector boundary.

Installation

The same source package is used on Linux, Windows, and macOS. Published binary wheels contain the native exact-order sparse accumulator, so normal wheel installation does not require a compiler:

python -m pip install recip-slice

For development from a checkout:

python -m pip install -e .

PyQt5 is installed as a core dependency because the program is GUI-first. The .[gui] extra is an empty convenience alias and is no longer needed.

Source installs attempt to compile the small native sparse bilinear accumulator from the shipped generated C source. The default build policy is RECIP_SLICE_NATIVE=auto: if compilation is unavailable, installation succeeds with the scientifically equivalent serial NumPy fallback. Performance-critical or release builds can set RECIP_SLICE_NATIVE=require to fail rather than silently fall back; RECIP_SLICE_NATIVE=disable explicitly requests the NumPy backend. Run with --verbose to see Sparse accumulator backend: native or numpy-serial; the same value is stored in the manifest.

On an offline machine with NumPy and the build dependencies already installed, python -m pip install -e . --no-build-isolation avoids fetching build dependencies. See docs/installation.md for Linux, Windows, macOS, and wheel release details.

fabio is required to read SMV detector frames. SciPy remains a core dependency for supported detector/noise-processing paths.

Local real-data check

For a local real-data validation, use the dedicated current runner:

python tools/run_local_real_data.py "PATH_TO_XDS_DIRECTORY"

The runner processes the complete frame range declared by DATA_RANGE in XDS.INP; no frame count is assumed. It uses the uncalibrated defaults (CMOS = off, Background = none), reconstructs the fixed 12-plane validation set twice, verifies deterministic evidence digests, records timing/source identity, and automatically builds the shipped native C extension if needed. Use --max-frames N only when you explicitly want a shortened diagnostic run. See docs/local_real_data_run.md.

Historical fixed-dataset milestone certification scripts are no longer part of the maintained tool surface; dated validation outputs remain preserved as provenance.

Subset-stability / banding mechanism campaign

When the scientific question is whether visible banding is stable, angular-sampling-sensitive, or time-dependent, run the fixed-v4 subset campaign from the source checkout:

python tools/run_subset_stability.py "PATH_TO_XDS_DIRECTORY"

The campaign reconstructs five immutable results with identical geometry and scientific settings: the complete frame series, two interleaved sequence-position subsets, and the early/late temporal halves. It writes SUBSET_STABILITY.json and compares only mutually supported fixed-v4 cells, using depth_support >= 0.95 by default when both results provide the map. The default campaign uses the same 12 validation planes, CMOS Off, Background None, the native accumulator, and raw detector coordinates; XDS spatial-distortion tables are never applied.

Interpretation is deliberately conservative: subset_stable means both partition tests are reproducible, not that a feature is proven specimen-derived; angular_sampling_sensitive means the interleaved subsets disagree while the temporal halves are stable; temporal_change means the temporal halves disagree while the interleaved subsets are stable; and mixed_or_low_support is unresolved. Absolute scale ratios are always retained separately from scale-normalized shape metrics. Use --max-frames only for an explicitly partial trial; it is not release evidence.

No Zn/31160 experimental frames are bundled with the package, so the package does not claim a real-data classification for those datasets. The synthetic end-to-end audit verifies the campaign logic with generated SMV cRED series of known cause.

The current native path can reuse conservative detector-block selection across short rotation intervals without changing frames, quadrature or scientific results. An optional temporal plane-candidate cache controls only this selection bookkeeping. See the release plan for current performance status; the complete-reconstruction 10-second target is not yet certified.

Command line

Reconstruct the three principal planes in one frame-reading pass:

recip-slice reconstruct /path/to/xds \
  --xparm GXPARM.XDS \
  --planes hk0 h0l 0kl \
  --q-max 1.0 \
  --dq 0.002 \
  --slice-thickness 0.008 \
  --prefix sample_name \
  --output-dir /path/to/xds/recip_slices \
  --no-plot --verbose

By default the command reads NAME_TEMPLATE_OF_DATA_FRAMES and DATA_RANGE from XDS.INP; frame numbers come from the ? field in the XDS template. Raw-coordinate processing does not apply EXCLUDE_DATA_RANGE or XDS region/pixel masks by default; enable those explicitly with --apply-xds-frame-exclusions and --apply-xds-region-exclusions. X-CORRECTIONS.cbf and Y-CORRECTIONS.cbf are never applied by recip-slice: they can change during an experimental session and are not treated as a stable detector-coordinate calibration. Their presence is recorded for provenance. DATA_RANGE remains the experimental image range in either mode; the preflight and GUI report whether that range is complete for the requested policy. Use --glob only for an explicit manual image selection. Manual paths are first matched through the XDS template; an incompatible filename must contain one unambiguous frame-number group, or the run stops rather than guessing from a run/suffix number. A normal run requires XDS.INP, a refined GXPARM.XDS or XPARM.XDS, and every frame required by the selected mode.

Normal treatment defaults to --cmos-correction off --background none. Auto CMOS is an explicit opt-in that abstains without calibration; experimental --background beam-inelastic may remove genuine diffuse scattering. Save it in a new result directory and retain the background-off baseline for comparison. The CLI refuses an occupied prefix for experimental background output; the GUI always retains the selected old result when starting an experimental background run. Projects, presets and queue snapshots must contain the current independent CMOS and background controls. Unsupported saved settings are rejected with an update-and-recreate message; no combined-selector migration is performed. Morphology-based display operations never modify quantitative evidence.

The scientific path defaults to no pre-reconstruction intensity threshold and no build-time central q mask. In the Qt viewer, a 0.02 A^-1 Centre mask is applied only to the view by default and can be removed instantly; use the Build-time Beam mask only when detector pixels are physically invalid.

SMV reads use a bounded four-worker read-ahead queue by default. On a fast local disk this can be tuned with --prefetch-workers N, or disabled with --no-prefetch when memory is constrained. Temporal plane-candidate reuse is disabled by default. It can be enabled explicitly with --temporal-plane-cache; --no-temporal-plane-cache forces the baseline selection path explicitly.

For a guided custom Miller-index plane, use --custom-plane with an optional offset and in-plane orientation. The offset is the right-hand side of the fractional equation and the slab thickness remains physical reciprocal-space distance:

recip-slice reconstruct /path/to/xds \
  --custom-plane 1 1 0 \
  --plane-name hhl_layer \
  --plane-offset-hkl 2 \
  --in-plane-basis 1 -1 0 0 0 1 \
  --q-max 1.0 --dq 0.002 --slice-thickness 0.008 \
  --no-plot

This reconstructs h + k = 2. The resolved Cartesian normal and orthonormal in-plane basis, equation, offset, units, and sampling settings are recorded in the result manifest and survive result loading. A custom plane is a single plane in this milestone slice; the GUI layer builder and offset-series controls remain planned work.

Open the lightweight viewer:

recip-slice view /path/to/result/folder --prefix sample_name

Open the Qt workspace for a new analysis session (the default when no subcommand is supplied):

recip-slice

Open the Qt workspace:

recip-slice gui /path/to/xds --prefix sample_name

From a checkout, the same GUI can be launched directly without installing the package in editable mode:

python run.py /path/to/xds --prefix sample_name

Interactive GUI launches print a short ASCII welcome banner with the current version and a pointer to recip-slice --help in the command window. The reconstruction, preflight, viewer, and JSON-reporting commands keep their output focused for scripts and pipelines.

GUI workflow

The workspace keeps a simple three-step route:

1. Data: select one parent folder. Every usable nested XDS dataset is indexed and can then be switched from a compact selector without another scan.
2. Planes: select principal, layer, or diagonal sections. All selected planes share one detector-frame pass.
3. Settings: review the resolution limit and workload; optional sampling and filtering controls remain collapsed for beginners.

The run/open actions remain visible from every step. Expert mode removes the guidance and exposes technical controls for a direct folder-to-run workflow. The right-hand workspace prioritises the reciprocal-space image; optional appearance and export controls are collapsed until requested.

Under Settings → Advanced settings, Run all indexed datasets… applies the current reconstruction settings to every usable XDS dataset in the selected folder. A confirmation summary appears first, followed by separate dataset- and detector-frame progress bars.

Recent experiment folders, named plane/settings presets, and display preferences are retained between sessions. Recent folders remain available from the Recent Projects menu, but a new GUI instance never selects the previous folder automatically. Saved reconstructions are selectable as individual runs, with provenance and quality summaries available without opening the process log. Re-running an occupied result name requires an explicit choice to create a timestamped version or replace the existing output.

Compare opens a second viewer for another indexed XDS dataset. Choose dataset… reuses the searchable Find… dataset browser and can browse to another folder. Saved slices are loaded automatically; if none are available, the GUI offers to reconstruct dataset B using the current A settings and shows a cancellable progress bar. Side-by-side and blink views work for any loaded pair; numerical difference and ratio views are available only when the plane and physical q-coordinate grids match exactly, so the GUI never performs hidden scientific resampling. Appearance controls apply to whichever pane was last clicked, allowing A and B to be adjusted independently; the active pane is indicated only by a coloured frame. Hovering either pane reports the nearest hkl coordinate and both intensities. Comparison figures can be exported for reporting.

Scientific scope

The output is geometrically precession-like, but it is not a simulation of a physical precession experiment. Intensity retains the sampling, detector response, dynamical scattering, and other limitations of the original rotation data. The current products are reciprocal-measure-normalized Intensity, additive Signal, measured Coverage, and conditioning-based Reliability. Signal is the quantitative ROI/integration product; it is not an intensity average or a replacement for an absolute calibration.

If XDS spatial correction tables (X-CORRECTIONS.cbf and Y-CORRECTIONS.cbf) are present, they are hashed/recorded for provenance but never parsed as coordinate corrections or applied. BKGPIX.cbf and supported XDS region/pixel exclusions remain an independent opt-in. The manifest records the ignored spatial-table provenance and applied masks, the refined geometry actually used, frame selection, q axes, preflight warnings, and saved filenames. It also records detector samples considered, samples rejected by the central-beam and optional XDS masks, processed frames, and per-plane geometry/signal-covered fractions.

See docs/algorithm.md for the coordinate model, docs/architecture.md for the software design, docs/gui.md for the interaction model, and docs/reproducibility.md for reporting guidance. The prioritized, cross-platform feature plan is in slice-view-roadmap.md.

For an external transform check against indexed XDS observations, run:

recip-slice validate-spot /path/to/xds --xparm GXPARM.XDS

This requires an indexed SPOT.XDS beside the XDS inputs; the package does not fabricate experimental validation data. The diagnostic reports total, indexed, unindexed, and validated-indexed records separately; XDS (0, 0, 0) unindexed records are not included in hkl residuals. SPOT detector positions may be in XDS's spatially corrected coordinate system, so this external oracle is interpreted as a convention diagnostic and never changes recip-slice's raw detector-coordinate reconstruction policy.

Testing

python tools/project_cycle.py check

PyQt5 is a core dependency. GUI integration tests run headlessly in CI with QT_QPA_PLATFORM=offscreen.

Project production cycle

Project housekeeping is separate from dataset housekeeping. From a checkout, run python tools/project_cycle.py release to validate the source, run tests, check every Python file with Ruff and mypy, quarantine known generated artefacts, build the wheel and source distribution, and verify that the canonical packaged style.qss and SVG icons are included in both packages. The standalone clean command is a dry run unless --apply is supplied; it never targets experiment folders or reconstruction results.

Base and View

A saved reconstruction is the Base. It is never changed by display controls. The GUI derives a reversible View from that Base:

• Show to crops the existing reciprocal-space raster to a smaller q range.
• Bin coarsens the displayed raster without rereading detector frames.
• Centre mask hides the pattern centre only in the derived view.

These settings are stored beside a manifest-backed result as <prefix>_view.json. Changing Build to, the native sampling, plane/slab definition, detector mask, or geometry still requires Rebuild pattern. The beta GUI and normal viewer open current fixed-thickness v4 results only. Existing legacy sum/mean, flat arrays, and historical depth-resolved results require a fresh reconstruction; they are not migrated or converted at the display boundary.

Historical performance records

Older development timing notes remain in CHANGELOG.md and dated validation reports. Their retired command-line controls and storage assumptions are not accepted by the current beta. Current measurements and remaining release work are in result storage and the release plan.