OAFuncs

Oceanic and Atmospheric Functions for Python.

OAFuncs provides utilities for NetCDF files, geoscience arrays, plotting, geographic masks, downloads, file operations, model data, and small workflow tools.

Installation

pip install oafuncs

Optional dependency groups:

data, interp, nc, draw, geo, down, office, parallel, sign, science, all, full

Package Structure

oafuncs
|-- oa_nc.py          NetCDF utilities
|-- oa_data.py        data interpolation helpers
|-- oa_geo.py         geographic and land/ocean mask helpers
|-- oa_draw.py        plotting helpers
|-- oa_cmap.py        colormap helpers
|-- oa_file.py        file operation helpers
|-- oa_date.py        date and timing helpers
|-- oa_down/          download helpers
|-- oa_model/         ROMS and WRF helpers
|-- oa_linux.py       Linux, HPC, and Slurm helpers
|-- oa_office.py      Office document helpers
|-- oa_sign/          website sign-in helpers
`-- oa_tool.py        command, progress bar, email, and parallel helpers

Preserved Historical Version

0.0.99.75 is the last version before the Slurm submission redesign on 2026-10-17. Its archive, version/oafuncs_0.0.99.75/, is a permanent reference for the previous implementation. Preserve it during future cleanup: do not delete, overwrite, or modify it, including when pruning other old versions. The redesigned submission behavior below describes the current source tree.

Public API

oa_nc

oa_nc.save(write_mode="w") and oa_nc.merge() write to a temporary file before replacing an existing output. Failed writes preserve the previous file; append mode (write_mode="a") still updates the file in place. For NumPy input, preserve_mask_values=True keeps underlying masked values; False writes them as missing values. Explicit missing_value values remain missing even for integer arrays, and all-NaN arrays remain missing rather than 0. merge() preserves raw encoding and input order: units, calendars, non-merged variables and non-merged coordinates must match. It raises ValueError for incompatible inputs rather than converting time units or aligning different grids. oa_nc.rename() checks variable and dimension name conflicts before changing either name, so a conflicting coordinate rename does not leave a partial change.

oa_nc.draw() accepts a NetCDF path, xarray.Dataset, or xarray.DataArray. File-backed workloads use parallel="auto" by default: small batches remain serial, while sufficiently large collections of independent 2D slices use oafuncs.oa_tool.Pex with automatic worker sizing. Use parallel=False to force serial rendering or parallel=True to force Pex. The default output is 300 dpi; set save_dpi=600 when publication-resolution PNG files are needed. On Windows, Pex calls must run below an if __name__ == "__main__": guard. For strongly skewed fields, robust=True displays colors using the 1st–99th percentiles while the title continues to report the complete data range and the clipped color range. The default robust=False preserves the full range.

oa_data

oa_geo

mask_land_ocean() preserves existing xarray dimension coordinates, including x/y grid indices. Longitude and latitude describe mask positions; two-dimensional longitude/latitude arrays must have the same shape.

oa_draw

oa_draw.gif(frame_duration=0.2) uses seconds (200 milliseconds per frame). Unreadable images are skipped; automatic dimensions come from the first readable image.

oa_cmap

oa_file

Example:

from oafuncs.oa_file import replace

replace(
    input_path="config.txt",
    replace_dict={"old_value": "new_value"},
    output_path=r"D:\run\config.txt",
)

output_path accepts either a file name, such as "result.txt", or an absolute file path. A file name is saved in the input file's directory. Directory-only paths and relative paths containing directories are rejected to avoid ambiguous output. Both direct-text and key-value replacement atomically replace the output path and preserve permission bits. A file link at that path becomes an independent file, leaving its source and other links unchanged. Outputs inside symbolic-link directories or Windows junctions are rejected to avoid modifying source folders. replace() still returns None on success. Read/write, permission and atomic replacement errors now propagate to the caller, including FileNotFoundError, PermissionError and other OSError exceptions; directory-link rejection raises ValueError. Unmatched text or keys retain the existing no-op behavior.

oa_date

oa_down

Literature downloads require a PDF header and the configured min_size_kb before replacing an existing file. This is a basic content check, not a full PDF integrity check. Each mirror receives its own tries_each_url budget; an HTTP failure moves on to the next mirror and failure records are deduplicated.

For hycom_3hourly.download(), interval_hours for a time range must be a positive multiple of 3. When using IDM, idm_timeout controls how many seconds without a completed valid file are allowed before remaining files are reported as failed; it defaults to 24 hours.

oa_model

The little_r helpers retain the existing fixed-width record layout and units. Overlong text fields are truncated to their documented widths; invalid record lengths raise ValueError rather than terminating the caller's Python process.

oa_linux

patch_slurm() replaces selected options in place without accumulating blank lines, preserving other options on the same line and trailing comments. Duplicate target options are removed; empty directive lines are dropped and comments are retained as ordinary comments. Resource counts must be positive integers; booleans, strings, fractions and nonpositive counts return False without writing the script. Updates are written to a sibling temporary file and atomically replace the supplied script path, preserving permission bits. A symbolic or hard link at that path becomes an independent file; its source and other links keep their original contents. This also applies to script updates made by submit(). Paths inside symbolic-link directories or Windows junctions are rejected.

from oafuncs import oa_linux

job_id = oa_linux.submit(
    workdir='/path/to/case',
    queues=['cpu_parallel', 'bigmem', 'dcu'],
    max_jobs=50,
    wait_start=False,
    wait_done=False,
)

submit() filters candidates by partition visibility, node count, time limit, and the per-user debug submission cap. It submits exactly one job: multiple eligible queues use a shared request such as #SBATCH -p cpu_parallel,bigmem,dcu; one queue uses a single partition. No replicas, queue switching, or cancellation/resubmission are performed.

Node limits are read dynamically from partition MinNodes/MaxNodes, including for custom queue names. An explicit node count or range must overlap the partition's allowed range; a queue with MaxNodes=1 is therefore excluded from an eight-node request without any name preset. If no -N/--nodes is specified, Slurm chooses the node count from the remaining resource request. On failed or incomplete queries, known single-node queues retain their fallback restriction; unknown limits produce a warning and remain for Slurm to validate. Account/QOS limits are still checked by Slurm, not inferred from partition node limits.

For multiple eligible queues, an existing active #SBATCH --mem is capped at the smallest queue-specific safe request, normally the smallest configured node memory minus 15 GiB (with existing conservative fallback values when the query fails). Lower requests are preserved. If --mem is absent, memory settings are left unchanged, including any --mem-per-cpu directive. A single eligible queue adjusts memory only when an active --mem directive already exists. For either queue count, missing or commented-out --mem directives are never added or enabled; scheduler defaults apply when unset. Explicit --mem-per-cpu or --mem-per-gpu settings are preserved for either queue count; conflicting memory modes are rejected before writing the script. Including debug sets the shared job time limit to 02:00:00.

max_jobs counts the current user's jobs and waits for one submission slot. Both waiting options default to False: the job ID is returned immediately after submission. wait_start=True waits until the job is observed running; wait_done=True waits for confirmed successful completion and takes precedence over wait_start. If a short job finishes between polls, accounting must confirm success before either waiting mode returns its ID. Failure or an unconfirmed final state returns None. A failed live-state query also checks accounting for a confirmed final state; it does not trigger resubmission. The original working directory is restored on exit.

The obsolete race, race_queues, timeout, and wait arguments have been removed. Remove the first three from existing calls and replace wait=True with wait_done=True. Multi-partition submission is internal to submit() and has no separate public function.

Job matching uses one current-user query and includes configuring, completing, suspended and other active states. Path filters accept ~, relative paths and paths containing spaces; matches are exact, with no recursive directory or prefix matching. get_jobid() returns None for zero or multiple matches. Query/parse failures raise RuntimeError in matching and cancellation APIs. The former dir_queued() has been removed. Use get_jobid(path=workdir) when each directory has at most one active job; remember that None also means multiple matches, so it is not a general test for an empty directory.

job_state(job_id) checks accounting when the live state is unavailable and returns compact codes such as PD, R, CD, F or CA. Set history=False for live queries only. No available record returns None; an unresolved query failure raises RuntimeError, and an invalid numeric ID raises ValueError. CD reports scheduler state; completion waiting also validates exit codes.

Use Slurm's scancel job_id for a single job. The former public cancel_job() has been removed. scancel_dir(path) cancels only current-user active jobs whose WorkDir exactly matches the directory; names, shared script locations and child directories do not match. Returned IDs indicate that scancel accepted the request; cleanup may still be in progress. Query or cancellation failures raise RuntimeError.

oa_office

oa_sign

oa_tool

On Windows, Pex uses its own spawn context without changing the caller's global multiprocessing start method. Calls still require the main guard.

Pex() chooses workers from CPU limits and current memory headroom: the minimum of host available memory, every visible ancestor cgroup's remaining allowance, and the local Slurm allocation minus current job usage. It leaves 2% headroom unassigned. Clean inactive file cache is conservatively treated as reclaimable; dirty/writeback pages are excluded. Slurm's allocation alone is never reported as free memory. If job-cgroup usage is unavailable, a warning and memory_headroom_gb statistics identify the process-tree fallback, which cannot account for sibling job steps or unmapped file cache.

When task memory is unknown, the first real task runs alone; its result is kept and its incremental memory cost, multiplied by 1.05, sizes the remaining pool without a fixed minimum reservation. The Linux probe subtracts the pre-pool parent PSS baseline from parent-plus-worker PSS, and also measures private worker memory (USS). This excludes inherited shared pages while accounting for new private/shared allocations. Small pools and explicit max_workers caps also use this probe. Before creating the actual pool, headroom is refreshed and its process count is capped using the full per-worker reservation, including startup costs; this also covers all workers started together by Linux fork.

Tasks are submitted individually, with at most one outstanding task per worker, and replenishment pauses when current headroom is insufficient. Each worker reserves only the growth still needed to reach its estimated peak; resident idle workers cannot fund other workers' reservations. Unknown worker slots reserve their full cost, and scheduling uses the largest eligible growth costs because the next worker PID is not guaranteed. USS is preferred for reusable private memory, with conservative PSS/RSS fallbacks. Later observed private worker peaks may raise the estimate, including an explicit reservation that proves too small. Already running tasks are allowed to finish. If no task is running and the next estimated worker cannot fit, MemoryError is raised. Sampling cannot guarantee protection from an unseen instantaneous task peak. For known heavy or heterogeneous tasks, memory_per_worker_gb (also available as OAFUNCS_PEX_MEMORY_PER_WORKER_GB) supplies the per-worker reservation in GiB, including worker overhead, and skips probing. Calls and result order are unchanged; worker counts remain automatic by default.

Links

• PyPI: https://pypi.org/project/oafuncs
• GitHub: https://github.com/OAKun/OAFuncs
• Download stats: https://pypistats.org/packages/oafuncs
• Wiki: https://deepwiki.com/OAKun/OAFuncs

License

MIT License. See LICENSE.txt.