# Rust-primary backend SHARPpy Reimagined supports `sharpmod_rs` as its primary numerical and direct pressure-level GRIB point-decoding backend. Official Windows executables bundle the extension, and default `auto` mode selects Rust after validating its package version, backend API, and required operations. Rust works behind the Python facade; the PySide6 GUI, renderer, command-line tools, SHARPpy widget stack, model retrieval, and profile orchestration remain shared Python application layers. The independently optimized Python backend remains fully functional and is the portable fallback. Source and Python-only installations therefore do not need Rust, Cargo, or maturin, and a missing or unloadable extension does not prevent the application from starting in default `auto` mode. ## Backend selection Set `SHARPMOD_BACKEND` before starting Python: | Value | Behavior | | --- | --- | | `auto` | Default. Use Rust when the extension loads successfully; otherwise use Python and retain the fallback reason for diagnostics. | | `python` | Always use the optimized Python implementation. The Rust extension is not required. | | `rust` | Require the Rust implementation. Missing or unloadable native code produces a clear error instead of silently falling back. | PowerShell examples: ```powershell $env:SHARPMOD_BACKEND = "auto" $env:SHARPMOD_BACKEND = "python" $env:SHARPMOD_BACKEND = "rust" ``` POSIX shell examples: ```bash export SHARPMOD_BACKEND=auto export SHARPMOD_BACKEND=python export SHARPMOD_BACKEND=rust ``` An unset variable is equivalent to `auto`. An unsupported value is an error; it is not treated as `auto`. Inspect the resolved state through the Python backend layer: ```python from sharpmod.backends import backend_info print(backend_info()) ``` The diagnostic reports the requested and active backends, whether `sharpmod_rs` is discoverable, its version when it was loaded, and the fallback reason when `auto` selected Python. In forced `python` mode the selector does not import the native module, so `rust_installed` can be true while `rust_version` remains `None`. Application code should import through `sharpmod.backends`; direct `sharpmod_rs` imports are reserved for extension development and benchmark verification. An importable native module is not accepted on import success alone. Before activation, the selector requires its package version to match `sharpmod` exactly, its integer backend API version to equal the Python contract (currently `7`), and all 18 required operations listed below to be callable. In `auto` mode an old or incomplete extension falls back to Python and records the compatibility failure in `fallback_reason`; in forced `rust` mode the same mismatch raises `BackendUnavailableError` instead of reaching an operation with stale native code. Backend selection is process-local. Set the environment variable before the first backend operation rather than changing it while the GUI or workers are already running. ## Accelerated scope Both backends expose the same named operations: - `wind_to_components` converts meteorological wind direction and speed to unit-preserving `u` and `v` components. - `components_to_wind` converts `u` and `v` components to meteorological wind direction and unit-preserving speed. - `interpolate_1d` performs the backend's one-dimensional profile interpolation contract. - `basic_sounding_qc` performs the documented structural and basic-value checks for sounding columns. - `pressure_sort_dedup_indices` returns the ordering/selection needed to sort pressure levels and remove duplicates consistently across all aligned columns. - `parse_sounding_rows` parses the deliberately narrow, simple sounding-row representation supported by the backend contract. - `profile_kinematics` computes Bunkers storm motion plus the standard SFC-to-0.5/1/3/4/6 km shear, mean-wind, storm-relative-wind, and SRH workspace in one coarse-grained call. - `profile_parcels` computes surface-based, lowest-300-hPa most-unstable, and 100-hPa mixed-layer parcel summaries—including CAPE/CIN, LCL/LFC/EL, and CAPE through 3/6 km—in one coarse-grained call. Cached 0–6 km CAPE is used only when reported levels in that layer are no more than 500 m apart; coarser soundings retain the reference integration. - `profile_convective_parcels` computes surface, forecast, most-unstable, mixed-layer, and mean-effective parcel summaries and virtual-temperature plotting traces together with effective-inflow-layer pressure bounds. - `lift_parcel` computes one explicit user-selected parcel summary and plotting trace. - `profile_dcape` computes DCAPE, its source pressure/downrush temperature, and the descending parcel-temperature trace. - `profile_thermodynamics` prepares one owned profile snapshot and shares it across the three-parcel, traced convective, and DCAPE workspaces. Native traces cross the extension boundary as contiguous NumPy buffers plus offsets; the public Python API still exposes the existing immutable tuples. - `profile_batch_analysis` evaluates complete independent soundings in stable input order. Calls below eight profiles are serial; larger calls default to a process-wide Rayon pool bounded by the available CPUs and four workers, remain serial when already inside Rayon, and accept explicit worker/cutoff overrides. Sharing the default pool keeps simultaneous GUI callers under the same native thread ceiling. - `decode_grib_point` and `decode_grib_points` reuse a bounded immutable message inventory, select the nearest point once per distinct grid definition, and return each sounding as one C-contiguous NumPy matrix. The multipoint form opens each selected message once and asks ecCodes for every requested grid index together while retaining request order and duplicate grid cells. When verified surface messages are present, both forms remove below-ground isobars and prepend surface pressure/height, 2-m thermodynamics, and 10-m wind. - The remaining required callables enable, clear, and report the native GRIB inventory cache so tests and benchmarks can distinguish cold scans from inventory reuse. The shared behavior includes missing values, masks, NaNs, stable ordering, boundary results, error conditions, units, and output types. Python/Rust equivalence tests use explicit floating-point tolerances. Unit conversions remain the caller's responsibility unless an operation's contract says otherwise. The wind and interpolation operations default to no explicit sentinel. The QC, pressure ordering, and row-parser operations default to `-9999`; passing `missing=None` disables explicit-sentinel matching. Parser blanks and non-finite values are normalized to `NaN` in that mode. Both backends apply these rules identically. The simple-row parser follows Python `float()` for Unicode decimal digits: the `Nd` blocks in the target interpreter's Unicode table—including Arabic-Indic, Devanagari, fullwidth, and mathematical digits—are normalized to ASCII before the native parse. CPython 3.12's Unicode 15 table additionally accepts Kawi and Nag Mundari digits; the CPython 3.11 wheel follows its Unicode 14 table and rejects those two newer blocks. Numeric characters outside the decimal-digit category, such as Roman numerals and superscripts, remain errors. This deliberately mirrors Python `float()` rather than Rust's broader `char::is_numeric` predicate. No Qt, rendering, or forecast downloading has been ported. A Python `AcceleratedConvectiveProfile` keeps SHARPpy's orchestration and public parcel objects while delegating its standard parcel/effective-layer searches and DCAPE integrations to the backend. The Skew-T user-parcel action delegates its explicit ascent through the same boundary. Backend failure falls back to the pinned SHARPpy routines. Advanced parcel fields that are not part of the traced parcel summary contract (for example cap strength and exact upstream MPL bookkeeping) remain Python-side compatibility concerns. ECAPE also retains a Python MUCAPE check for its exact pinned-oracle upper-bound contract. The Rust decoder receives a cache-owned local file from the Python retrieval layer. Rayon is used only for complete-profile analysis, where the measured 8-profile and 32-profile workloads benefit. GRIB/ecCodes calls remain behind a process-wide native mutex; batch analysis never enters ecCodes. Repository-owned application call sites delegate wind-component creation, generic profile interpolation, ERA5/WRF wind conversion, pressure sorting and deduplication, standard SharpTab kinematics, standard/full parcel workspaces, user parcel ascents, DCAPE, and forecast-model GRIB point extraction through the facade. A real `Profile` caches its immutable kinematics and three-parcel summary workspaces; full convective profiles consume the extended workspace once during construction. Box fast-tier analysis and built-in ensemble summaries submit complete ordered profile groups through `profile_batch_analysis`; load failures retain their original positions and fall back per profile without discarding successful members. The bounded metric cache fingerprints all source columns and rebuilds prepared results after either profile replacement or an in-place array edit. Nonstandard layers and any unavailable workspace continue through the existing Python routines. Basic QC and simple-row parsing remain equivalence-tested APIs without changing the SPC, UWyo, BUFKIT, WRF, or other decoder pipelines. ### Direct GRIB behavior Forecast retrieval still uses Herbie and the existing provider/subregion routes. The selected inventory is reused for transfer planning, and a complete local subset is handed to the active decoder without constructing xarray data. The Python decoder scans message headers once per file identity, performs one ecCodes nearest-point lookup per distinct grid definition, vector-reads all uncached requested indexes from each selected field/level message, and keeps bounded inventory, nearest-point, and decoded-point LRUs. The Rust decoder memory-maps the file, locates message boundaries without copying, uses ecCodes handles that borrow each mapped message, and returns all nine columns in one NumPy-compatible matrix per point. Its adapter batches unique uncached requests and keeps a bounded exact-result cache without conflating different surface grids. Inventory identity includes the canonical path and size plus replacement-safe filesystem metadata: volume/file index and creation/write timestamps on Windows, or device/inode plus modification/change timestamps on Unix. An atomic same-size replacement therefore invalidates both Python and native state. The inventory stores immutable offsets and metadata only--never ecCodes handles--so model-cache leases and mapped-file lifetimes remain unchanged. Python ecCodes access uses its own re-entrant lock and all native ecCodes access uses one native mutex. For HRRR, retrieval includes `PRES` and `HGT` at the surface, `TMP` and `DPT` at 2 m, and `UGRD` and `VGRD` at 10 m. Both direct decoders and the cfgrib/Zarr compatibility routes use surface pressure to remove all subterranean isobars before inserting the real ground row. Model extraction fails closed if this verified merge is unavailable. The JSON sidecar records `surface_pressure_hpa`, `surface_merged`, and `below_ground_levels_removed`. Optimized Python and Rust return matching omega values and missing masks at the published pressure levels. For GEFS, the value published only at 850 hPa stays missing at every other level. The old/new benchmark's `12 -> 1 valid` omega difference records removal of the frozen legacy xarray full-column broadcast; it is not missing Rust functionality or a Python/Rust equivalence exception. `SHARPMOD_GRIB_DECODER=auto` is the default. Unsupported layouts fall back to cfgrib with its persistent on-disk index; split groups are reduced to at most a 3-by-3 neighborhood before xarray merging. This preserves the existing neighbor-wind surface-vorticity calculation without materializing whole model grids. Use `xarray` to force that compatibility path or `direct` to make a direct-decoder error explicit. ## Python-only installation Existing setup continues to work unchanged: ```powershell py -3.11 -m venv .gribenv .\.gribenv\Scripts\Activate.ps1 python scripts\install_sharppy_compat.py --extras dev,render ``` With no extension installed, `auto` uses Python. `python` also works normally, while an explicit `rust` request reports that the extension is unavailable. ## Build the local Rust extension Install Rust 1.88 or newer from the stable channel and make `cargo` available on `PATH`. Then install the optional build frontend into the same `.gribenv` that will import the extension: ```powershell .\.gribenv\Scripts\Activate.ps1 python -m pip install -e ".[dev,rust-build]" sharpmod-rust-sync sharpmod-rust-sync --check ``` `sharpmod-rust-sync` compares the installed native distribution with the checkout version. It performs a locked release rebuild only when the extension is missing or stale, installs it into the active virtual environment, and then requires forced-Rust selection in a fresh Python process. `--check` performs the version and selector checks without modifying the environment. Use `sharpmod-rust-sync --force` after changing native source even when the package version did not change. The lower-level extension-development path remains available: ```powershell Set-Location rust\sharpmod-rs maturin develop --release --locked Set-Location ..\.. ``` `maturin develop` installs `sharpmod_rs` into the active environment; it does not modify the normal setuptools build backend for `sharpmod`. From a POSIX shell, activate `.gribenv/bin/activate` and use the same sync command or crate directory commands. Windows source builds also require the MSVC C/C++ linker toolchain (normally installed through Visual Studio Build Tools). Confirm the extension and selector: ```powershell python -c "import sharpmod_rs; print(sharpmod_rs.__version__)" $env:SHARPMOD_BACKEND = "rust" python -c "from sharpmod.backends import backend_info; print(backend_info())" ``` ## Tests and checks Python-only tests must pass without building Rust: ```powershell Remove-Item Env:SHARPMOD_BACKEND -ErrorAction SilentlyContinue python -m pytest $env:SHARPMOD_BACKEND = "python" python -m pytest sharpmod/tests -k "backend or rust" ``` Run the native checks from the crate and then run the equivalence tests with Rust required: ```powershell Set-Location rust\sharpmod-rs cargo fmt --check cargo clippy --locked --all-targets -- -D warnings cargo test --locked maturin develop --release --locked Set-Location ..\.. $env:SHARPMOD_BACKEND = "rust" python -m pytest sharpmod/tests -k "backend or rust" ``` Rust-specific Python tests skip with an explicit reason when `sharpmod_rs` is not installed during a normal Python-only run. Forced-Rust selector tests must still verify the error path without depending on a globally installed extension. ## Benchmarks The cross-backend harness requires a built extension and compares the Python and Rust backend methods on small profiles, ordinary profiles, a 100,000-value vector, a 2,048-by-128 array batch, repeated calls, scalar 700 hPa lookups on 32/128 levels, six repeated fields, and real cached-selector `Profile` construction. It also measures 32/128-level kinematics, diagnostics-only parcels, traced convective parcels, DCAPE, shared thermodynamic preparation, and complete 8-by-32 and 32-by-128 profile batches. Shared thermodynamics are split into public-adapter, buffered-adapter, and direct-native rows; complete batches split serial and bounded four-worker rows. An equal-work sequential-versus-two-thread diagnostic remains observational: ```powershell $env:SHARPMOD_BACKEND = "rust" python benchmarks\benchmark_backends.py ``` See [the benchmark guide](../benchmarks/README.md) for methodology and reproducibility guidance. Benchmark output is evidence to inspect, not a pass criterion. No operation is described as faster without recorded results on the relevant workload, and common single-profile operations may remain preferable in Python when boundary-crossing or conversion overhead dominates. The two-thread table is observational, not a pass/fail result or a promise of parallel scaling. Small array kernels retain the Python GIL while borrowing NumPy input slices; releasing it could allow another Python thread to mutate storage that Rust is reading. The profile-kinematics and profile-parcel bindings instead create one Rust-owned snapshot before releasing the GIL; their internal prepared structures borrow that snapshot instead of copying the same columns again. Complete-batch workers likewise receive only owned vectors. GRIB decoding releases the GIL and returns capsule-backed NumPy matrices, but native ecCodes calls remain serialized. Python ecCodes/cfgrib calls use a separate lock, so callers should not mix decoder implementations concurrently in one process. ### Recorded local result The following snapshot was recorded on 2026-07-15 on a 12th Gen Intel Core i5-1235U in Windows Balanced mode, with Python 3.11.14, NumPy 2.4.6, `sharpmod_rs` 0.3.1, and Rust 1.97.0. The harness alternated measurement order across 20 repeats after five warmups. The complete header, command, medians, minima, and maxima for every measured row are retained in the [raw result](../benchmarks/results/2026-07-15-windows-amd64.txt). Representative median milliseconds per call were: | Workload | Operation | Python | Rust | Local observation | | --- | --- | ---: | ---: | --- | | 100,000 values | `wind_to_components` | 4.619510 | 3.816955 | Rust median about 17% lower | | 100,000 values | `components_to_wind` | 5.626450 | 4.531640 | Rust median about 19% lower | | 100,000 values | `interpolate_1d` | 3.766315 | 4.706360 | Python median about 20% lower | | 2,048 x 128 values | `wind_to_components` | 11.361770 | 7.825710 | Rust median about 31% lower | | 2,048 x 128 values | `components_to_wind` | 16.628920 | 12.481090 | Rust median about 25% lower | This is a noisy local observation, not a cross-platform guarantee; several small and repeated-operation ranges overlapped substantially and are not used for speed claims. Large wind-vector batches showed the clearest Rust benefit, while NumPy remained preferable for the large interpolation case. `auto` selects one backend for all operations rather than making a per-call performance decision, so benchmark the real workload and use `python` explicitly when its mix favors the Python implementation. ### Recorded GRIB decoder result On 2026-07-16, a three-sample run used a 181,267,625-byte HRRR pressure-level subset at 35.18 N, 97.44 W (40 output levels). The OS file cache was warm, while application decoder caches and cfgrib indexes were disabled for each cold sample. "Old Rust" is the historical hybrid path: cfgrib/xarray still performs the complete decode and Rust only converts the extracted wind column. | Comparison | Old median | Optimized median | Speedup | | --- | ---: | ---: | ---: | | Python | 33.961649 s | 6.601793 s | 5.14x | | Rust / historical hybrid | 31.398744 s | 6.614370 s | 4.75x | The optimized cold decoders were effectively tied in this run (Python was 0.19% lower). With a persistent cfgrib index versus a warm direct-Python inventory at a new point, the medians were 19.569670 s and 7.254023 s (2.70x). For an exact decoded-point cache hit, 100 samples had medians of 0.165650 ms in Python and 0.110250 ms in Rust. See the dated raw result in `benchmarks/results` for minima, maxima, environment details, and the exact command. The current API-7 optimization measurements, including the release baselines, Criterion confidence intervals, complete-adapter JSON, fixed-fixture GRIB cache states, batch worker evaluation, regressions, and validation commands, are in [`2026-09-12-rust-backend-optimization.md`](../benchmarks/results/2026-09-12-rust-backend-optimization.md). ## Platform status The crate is portable Rust stable code. The dedicated `.github/workflows/rust.yml` workflow is configured to build, install, smoke-test, equivalence-test, and retain CPython 3.11 and 3.12 wheel artifacts on: - Windows x86_64 - Linux x86_64 - macOS arm64 - macOS x86_64 Those jobs prepare downloadable native-wheel CI artifacts; the wheels are not published as a separate package-index release. The official Windows release workflow builds and installs the extension before constructing and runtime-checking the one-folder and one-file PyInstaller applications. Rust is therefore the supported primary backend in those binaries. The Python-only package remains the portable fallback for source installs and platforms without a compatible extension. ## Current limitations - Native wheels are CI/build artifacts rather than a separately published package-index release. Official Windows desktop binaries bundle the native extension directly. - The normal `sharpmod` wheel and sdist contain the Python backend layer but not the Rust crate or extension. The `[rust-build]` extra installs maturin only; a repository checkout is required for the documented source build. - The prepared wheels target CPython 3.11 and 3.12 separately and are not `abi3` wheels. Each additional Python minor version requires its own native wheel matrix entry. - NumPy masked arrays, non-contiguous views, non-`float64` data, and sentinel normalization can require Python-side validation or copies. Zero-copy is an optimization for compatible arrays, not an API guarantee. - The declared NumPy range spans NumPy 1.x and 2.x; native compatibility must be tested across that range before publishing wheels. - Small arrays can cost more to cross the Python/Rust boundary than they save in computation. The workspace APIs amortize that cost by returning all standard layers or parcels together and caching each result on a `Profile`. - Borrowed array kernels retain the GIL. Profile kinematics and parcels own copies before releasing it. Direct GRIB decoding also releases it, but Rust-decoder ecCodes access is serialized; concurrent point decodes are not currently expected to scale. Python ecCodes/cfgrib calls are protected separately rather than by a shared cross-language lock. - A `Profile`'s original internal SharpTab caches still follow its normal derived-attribute lifecycle. The cross-profile metric cache separately fingerprints pressure, height, thermodynamic, and wind columns, so its prepared result is discarded if those arrays are edited in place. - Native parcel results intentionally exclude several advanced `Parcel` fields outside the traced parcel summary contract, including cap-strength bookkeeping and the pinned ECAPE upper-bound check. The Python compatibility layer derives or masks those fields and retains the pinned SHARPpy fallback. - Direct GRIB caches are process-local and bounded. A truly cold request still scans message headers; inventory reuse removes that scan but ecCodes must still unpack every selected field/level message. Multipoint calls amortize that unpack across all requested indexes rather than making repeated scalar passes. - When a downloaded inventory has no usable relative/absolute-vorticity field, model extraction keeps the xarray compatibility path so the existing neighbor-wind vorticity estimate is preserved. - Parsing is intentionally limited to the backend's simple row format. Existing SPC, UWyo, BUFKIT, WRF, and other decoders remain Python implementations. - QC is basic and structural. It does not replace the complete SHARPpy profile validation and meteorological analysis paths. - The PyInstaller spec includes `sharpmod_rs` only when the extension is discoverable and collected successfully in the environment performing the build. The official release job builds and installs it before packaging; custom local builds that omit it log a warning and produce a fully functional Python-fallback bundle.