# Foundation Model Inference Operation # # Apply a pretrained foundation model (HighFM, Prithvi, Clay, AlphaEarth, etc.) # to raster input(s). Resolves the model from a model record in registry/models/, # loads weights with a hashable content-addressable identifier, and emits a # RunRecord alongside the prediction for Evidence Bundle composition. # # Differs from `classification_predict`: # - `classification_predict` consumes a local sklearn-style `.model` file # produced by `classification_train`. # - `fm_inference` consumes a registry/models/ record that points at a # publisher's published weights (Hugging Face, Zenodo, R2, NASA Earthdata) # and verifies them by SHA256 at load time. # # Ticket: .tickets/impl/op-fm-inference/ # Spec: docs/spec/ML.md, docs/spec/EVIDENCE_BUNDLE.md # Tie-in: UC-045 advisory recipes, ADR-0046 (real-time catalog), FM_SUBSTRATE_POSITIONING.md id: fm_inference name: Foundation Model Inference description: > Apply a pretrained foundation model from registry/models/ to raster input(s). Resolves the model record, downloads + hash-verifies the weights on first use, runs the inference pass via the model's declared framework (PyTorch/TerraTorch, ONNX, etc.), and emits a per-run RunRecord that the Evidence Bundle composes alongside the prediction. version: 0.1.0 category: ml type: raster-model-to-raster default_implementation: terratorch inputs: - name: image type: raster format: cog description: > Input scene. For temporal models, pass a list under `images:` instead (see params.temporal_input); the op stacks them along the time axis. required: true - name: images type: raster_list format: cog description: > Ordered list of input scenes for temporal models (Prithvi expects T×C×H×W). Mutually exclusive with `image:`. The model record's input.num_frames defines the expected list length. required: false - name: mask type: raster format: cog description: > Optional mask. Pixels with value 0 are skipped (prediction set to nodata). required: false # Parametric on `params.model:` — bands, num_frames, and target concept are # all declared by the model record in registry/models/.yaml. The # resolver must inspect the model record (input.bands, input.num_frames, # output.task) at plan time. The op YAML cannot statically declare bands # because the same op shape runs Prithvi (6 HLS bands), Clay (different # band set), AlphaEarth (multi-modal), etc. requires: {} outputs: - name: prediction type: raster format: cog dtype: uint8 # No fixed output concept — depends on the chosen model's head (burned-area, # land-cover, crop-type, embedding, etc.). Recipes that pin a model should # declare the concept at the recipe layer. description: > Primary prediction output. For segmentation heads: class labels (uint8). For probability heads: scaled to uint8 0-255. Use params.output_format to change. - name: probability type: raster format: cog dtype: float32 description: > Per-class probability map (one band per class). Emitted only when params.emit_probability is true. Useful for uncertainty propagation into Evidence Bundles. - name: embedding type: raster_or_tensor format: zarr dtype: float32 description: > Per-tile or per-pixel embedding tensor. Emitted only when the model record's output.types includes `embedding` and params.emit_embedding is true. Shape determined by the backbone's embedding_dim. - name: run_record type: provenance_record format: json description: > Content-addressable record of the inference call. Carries: - model_id (registry/models/ slug) - model_hash (SHA256 of weights file) - input_hashes (SHA256s of each input raster) - params (canonicalized JSON of all params used) - engine (e.g. terratorch), engine_version (e.g. terratorch==0.99.7) - timestamp (UTC ISO-8601) - device (cpu | cuda:0 | mps) Consumed by `geo:evidence_bundle_emit` (separate ticket) as a signed input to the bundle's PROV-O graph. params: model: type: string required: true description: > Model record slug (filename in registry/models/, without .yaml). Examples: prithvi-eo-2.0-300m, prithvi-eo-2.0-300m-burnscars, clay-v1.5, alphaearth-foundations-v1. The op fails fast with a clear error if the record is missing or its weights URL is unreachable. device: type: string default: auto description: > Inference device: auto | cpu | cuda:0 | cuda:1 | mps. `auto` picks the most powerful available (cuda > mps > cpu). The choice is logged in RunRecord.device so re-runs can match. engine: type: string description: > Inference engine override. Default reads the model record's `framework` field. Valid values: terratorch, pytorch, onnx, jax. The engine version is captured in RunRecord. num_frames: type: integer description: > Number of temporal frames for temporal models (Prithvi). Default reads the model record's input.num_frames. Must match len(images) if `images` passed; ignored otherwise. batch_size: type: integer default: 16 description: > Tiles per forward pass. Lower for memory-constrained runs. tile_size: type: integer description: > Spatial tile edge (pixels). Default reads the model record's input.image_size (224 for Prithvi, 256 for Clay). Inputs larger than one tile are gridded; outputs are stitched. tile_overlap: type: integer default: 0 description: > Pixels of overlap between tiles. Smooths boundaries when stitching segmentation outputs. Set to ~16 for thin-feature segmentation. threshold: type: float default: 0.5 description: > Confidence threshold for binary segmentation heads. Pixels with probability >= threshold map to class 1. output_format: type: string default: mask description: > mask | probability | embedding. `mask` returns a uint8 class label raster; `probability` returns the float32 probability raster; `embedding` returns the per-tile/per-pixel embedding tensor (only valid when the model supports it). emit_probability: type: boolean default: false description: > Emit the float32 probability map alongside the primary prediction. emit_embedding: type: boolean default: false description: > Emit the embedding tensor. Requires the model record's output.types to include `embedding`. Useful for downstream similarity / retrieval. weights_cache_dir: type: string description: > Directory for cached weight downloads. Default $FOLIA_CACHE_DIR/models/ or ~/.folia/cache/models/. Weights are content-addressed by SHA256, so multiple recipes can share a cache safely. reject_on_hash_mismatch: type: boolean default: true description: > If the downloaded weights file's SHA256 does not match the model record's weights_hash field, fail the run instead of using the weights. Set to false only for development against pre-release model records where the hash is not yet known. license_acceptance: type: boolean default: false description: > Required to be true when the model record's license tier is `free_nc` (non-commercial) or `commercial`. Forces the recipe author to acknowledge license terms. Free / Apache / MIT / CC-BY models do not require this flag. granularity: min_resolution_m: 1 max_resolution_m: 30000 recommended_resolution_m: 30 output_resolution: inherit requires_continuous: false suitable_aoi_km2: min: 0.5 max: 100000 realtime_max_km2: 1000 uncertainty: error_model: unknown factors: - name: weights_provenance description: > Foundation model weights are published by third parties. Provenance is verifiable only via SHA256 hash match against the model record. A drift between hash claims and actual weights file is a hard failure. severity: high mitigations: - "params.reject_on_hash_mismatch defaults to true; do not disable in production" - "Pin the model record version; bump explicitly on backbone updates" - name: engine_version_drift description: > PyTorch 2.5 vs 2.6 can produce numerically-different outputs for the same weights. The RunRecord captures engine + version; downstream verification re-runs in the same environment. severity: medium mitigations: - "Pin engine version in the deployment environment" - "Re-verify Evidence Bundles in the same engine version they were emitted in" - name: input_domain_shift description: > Foundation models trained on HLS may not transfer cleanly to non-HLS sensors. The op does not check for this; the model record describes the trained input distribution. severity: high mitigations: - "Use the model only with sensors listed in the model record's input.sensor" - "For cross-sensor inference, fine-tune on the target sensor first" - name: license_tier description: > Non-commercial license blocks commercial deployment. Setting license_acceptance: true does not change the license — it only acknowledges the recipe author has read it. severity: high mitigations: - "Inspect the Evidence Bundle's model.license field before downstream use" limitations: - "Inference results depend entirely on the foundation model's training distribution" - "Determinism modulo hardware float-point noise; document the noise floor in tests" - "Out-of-distribution inputs may produce overconfident or systematically biased outputs" execution: realtime_max_km2: 1000 cost_per_km2: 0.002 # placeholder; tune after first profiling time_per_km2_sec: 0.3 memory_profile: high # FM weights are GB-scale; budget accordingly profile: cpu: medium memory: high io: medium gpu: high scaling: model: linear parallelizable: true min_chunk_km2: 0.5 # backends: audited 2026-08-14 (defect 50). A key means a runtime that DISPATCHES this op — # folia-engine `dispatch_op` (products/sdk/folia-engine/src/lib.rs), a `registerOp`/OP_TABLE # entry in packages/compute, `_BUILTIN_OP_MAP` in folia/compute.py, or a backend manifest # (folia/backends/*/backend.yaml). backends: python: function: ml.eofm.predict dispatch: folia/compute.py _BUILTIN_OP_MAP display_hints: map: renderer: maplibre palette: categorical # default; recipes can override per head opacity: 0.7 cache_policy: regional_precompute: [] ttl_days: 7 # FMs evolve; short TTL forces re-run on update invalidate_on: [model_update, source_update, engine_version_update] examples: - name: Prithvi burn-scar mapping over pre/post HLS pair description: > Apply the Prithvi-EO-2.0-300M burn-scars fine-tune to a pre/post HLS-L30 pair over an AOI. Emits a uint8 mask + a RunRecord for Evidence Bundle. inputs: images: ["pre_event.tif", "post_event.tif"] params: model: prithvi-eo-2.0-300m-burnscars num_frames: 2 threshold: 0.5 emit_probability: true - name: Clay embeddings over a single Sentinel-2 scene description: > Compute Clay-v1.5 per-tile embeddings on a Sentinel-2 L2A scene; useful for downstream similarity / retrieval recipes. inputs: image: "sentinel2_tile.tif" params: model: clay-v1.5 output_format: embedding emit_embedding: true - name: Future — HighFM fire detection on SEVIRI (placeholder; weights pending) description: > Once HighFM weights drop, this exact same op invocation runs the active-fire head on a SEVIRI scene. Demonstrates L1 substitutability. inputs: image: "seviri_radiance.nc" params: model: highfm-mt-fire # not yet authored; placeholder record threshold: 0.5