Loading Data

Loaders, DEA/STAC acquisition, and local cache surfaces. See Usage Guide: The four ways to run it and Advanced: DEA acquisition internals for narrative context before diving into individual signatures below.

hydroseason.io

Source-agnostic extent and raster loaders (re-export facade).

Implementation lives in _io_extent (pandas-only), _io_geo (AOI/raster loading and georeferencing), and _io_resolution (resolution planning and the amplitude probe). This module exists so from hydroseason.io import X keeps working for every name that was importable here before the split, including the private helpers already used directly by scripts and tests.

HistoricalWaterMask dataclass

The exact, immutable (count_wet > 0) AND AOI raster and its provenance.

mask is a 2D boolean array (row-major, shape shape) at the Statistics' native grid -- never a polygon. pixel_count is the fixed number of True cells, which becomes the constant n_aoi denominator for every month in the requested analysis period.

source_item_ids and source_lineage are recorded as sorted tuples so two builds over the same underlying STAC items always compare equal regardless of search-result ordering. aoi_sha256/mask_sha256 are stable SHA-256 digests -- see :func:build_historical_water_mask for exactly what each is computed over.

WetPlanningFootprint dataclass

A conservative, coarse pruning aid for planning remote WOfS reads.

This is a SEPARATE, performance-only artifact from anything zoning-related (build_zones()). It must never be fed into zoning as frequency or support, and it must never change what counts as inside/outside the catchment for any metric denominator -- pixels it skips for I/O still represent dry/outside-the-planning-footprint, not a smaller catchment.

native_mask is count_wet > 0 at the statistics' native grid. coarse_mask is native_mask aggregated by factor via aligned max-pooling (coarsen(..., boundary="pad").max()) plus an optional safety_cells-wide coarse-grid dilation halo -- never nearest/mode/mean, so an isolated native wet pixel can never disappear when coarsened. Every accepted footprint satisfies the round-trip proof: expanding coarse_mask back to the native grid is always a superset of native_mask (see :func:build_wet_planning_footprint's tests).

active_windows are storage/source-aligned native-grid GridWindows covering the wet coarse cells -- a chunk/window predicate, not a polygon. geometry stays None on the default path (no Shapely close/buffer); it exists only for a consumer that explicitly needs a vectorised polygon.

WOfSCacheHandle dataclass

A resolved pointer to a (possibly complete) on-disk WOfS cache store.

HistoricalMaskCoverageWarning

Bases: UserWarning

The requested analysis window is not fully inside the mask's coverage.

HistoricalMaskRefreshedWarning

Bases: UserWarning

A cache hit's coverage was extended by rebuilding against a newer, wider DEA Water Observation Statistics vintage.

Deliberately a SEPARATE class from :class:HistoricalMaskCoverageWarning: "your window overhangs the cached coverage" (an observation about the request) and "your numbers just moved" (an observation about the denominator itself changing under a caller that may be tracking extent_pct over time) are different events, and a caller filtering warnings may reasonably want to treat them differently -- e.g. always surface a refresh, but silence routine overhang notices.

load_aoi

load_aoi(aoi, *, to_crs: str | int | None = None)

Load a non-empty GeoDataFrame from vector path or GeoDataFrame.

load_extent_csv

load_extent_csv(path: str | PathLike[str], *, date_col: str = 'date', value_col: str = 'extent_pct') -> pd.DataFrame

Read a monthly extent CSV into date-indexed form for detection.

This loader only parses dates and coerces the value column; it does not gapfill missing months or quality-screen invalid coverage. The CSV is valid input for detect_hydrological_years only if the upstream extent series already went through mask completion and quality screening (see the migration plan's gapfilling recommendation).

load_monthly_masks

load_monthly_masks(input_dir: str | PathLike[str], start_date: str, end_date: str, *, aoi=None, encoding: MaskEncoding | None = None, classifier: Callable | None = None, chunk_x: int = 512, chunk_y: int = 512, time_chunk: int = 24, majority: bool = True, duplicate_month_policy: Literal['raise', 'warn'] = 'raise')

Load AOI-clipped TIFF masks as lazy canonical time/y/x data.

Explicit encoding prevents ambiguous uint8 masks from being mistaken for raw WOfS flags. Canonical values: dry 0, water 1, invalid -1, outside -2.

load_monthly_masks_zarr

load_monthly_masks_zarr(zarr_path: str | PathLike[str], start_date: str, end_date: str, *, chunk_x: int = 512, chunk_y: int = 512, time_chunk: int = 24, duplicate_month_policy: Literal['raise', 'warn'] = 'raise')

Open an already-canonical, already-AOI-clipped Zarr mask cube lazily.

load_wofs_from_stac

load_wofs_from_stac(stac_url: str, collection: str, aoi, start_date: str, end_date: str, *, crs: int | str | None = 3577, chunk_x: int = 512, chunk_y: int = 512, time_chunk: int = 24, majority: bool = True, duplicate_month_policy: Literal['raise', 'warn'] = 'raise', resolution: float | None = None, groupby: str = 'solar_day', historical_water_mask=None)

Load WOfS observations in annual batches, compose monthly, and clip to the AOI.

A calendar year is sent to odc.stac.stac_load at once. The returned lazy cube is then split into monthly composites, avoiding the substantial graph/setup overhead of one loader call per month while retaining the same monthly result. groupby (default "solar_day") controls same-day scene mosaicking before compositing -- see :func:_load_wofs_items. historical_water_mask, when supplied, is applied during that loader's single AOI clip so callers do not need a second full-cube masking pass.

load_wofs_monthly_extent

load_wofs_monthly_extent(stac_url: str, collection: str, aoi, start_date: str, end_date: str, *, cache_dir: str | PathLike[str] | None = None, mask_cache_dir: str | PathLike[str] | None = None, offline: bool = False, crs: int | str | None = 3577, resolution: float | None = None, chunk_x: int = 512, chunk_y: int = 512, time_block: int = 12, majority: bool = True, force: bool = False, tile_pixels: int | None = None, wet_aoi=None, precompute_wet_aoi: bool = False, persistence_min: float = 0.0, close_m: float = 150.0, buffer_m: float = 300.0, progress: bool = False, progress_desc: str | None = None, progress_position: int | None = None, on_warning: Callable[[str], None] | None = None, auto_tiling: bool = True, read_workers: int | None = None, diagnostics_callback: Callable[[dict[str, int]], None] | None = None, resampling_policy: Literal['categorical_safe', 'native_aligned'] = 'categorical_safe', year_workers: int | None = None, wet_mask: Literal['off', 'dea_stats'] = 'off', use_historical_water_mask: bool = True, historical_water_mask: HistoricalWaterMask | None = None, historical_mask_cache_dir: str | PathLike[str] | None = None, statistics_stac_url: str = DEFAULT_WO_STATISTICS_STAC_URL) -> pd.DataFrame

Compute monthly WOfS extent in resumable calendar-year pieces.

Each year is loaded and reduced independently, bounding graph size and allowing a stopped run to resume from its last completed year. When cache_dir is supplied, CSV cache identity includes all data-affecting inputs, the AOI content hash, and (when a wet AOI is in play) its content hash too -- a different wet AOI never silently reads a stale cache.

progress=True shows a tqdm bar ticking once per calendar year processed (cache hits included), so long tiled STAC pulls give visible feedback instead of blocking silently.

on_warning, if given, is called once per non-fatal provenance notice; the notice is also emitted as a UserWarning regardless.

read_workers, if given (and > 0), overrides dask's threaded-scheduler worker count while the lazy STAC/COG graph is materialised -- both the precompute whole-cube pass and every per-year reduction.

PROFILING RESULT (see git history for the investigation): forcing a worker count here does NOT help and usually HURTS. Measured on a 133-scene AOI-year, dask's own default (unset) decoded in ~18-23s; explicit worker counts of 4, 8, 16, 32, and 64 were all slower than unset, and got monotonically worse above ~8 workers (64 workers: ~30s, nearly 2x unset). The workload is decode/warp-CPU-bound, not I/O-latency-bound as originally assumed -- a raw concurrent-HTTP probe fetching the same 133 scenes' headers took ~6s, so the wall is GDAL's per-scene GeoTIFF decode and reprojection, which dask's own worker-count heuristic already sizes close to this machine's real parallel-decode capacity. Forcing a higher number adds thread-scheduling/lock contention on top of an already-saturated decode step. The default is therefore None (leave dask's scheduler untouched); only pass an explicit value if you have profiled your own machine and confirmed it helps there -- it very likely will not.

year_workers controls independent calendar-year acquisitions. The default None uses two workers for an uncached, untiled first pass; this overlaps independent DEA/STAC reads without multiplying dask's decode worker count. Cached acquisition remains serial by default. Pass 1 to force serial execution or a profiled positive integer to tune it.

auto_tiling=True (the default) degrades a requested tiled load to the plain untiled path when the AOI's bounding box provably fits inside one load tile (tile_pixels * resolution metres per side). In that case tiling can only ever yield a single tile, so its output is bit-identical to the untiled reduction, but precompute_wet_aoi would still pay for a full extra whole-cube read to prune a one-cell grid. Degrading skips that dead cost. It is suppressed when a wet_aoi is caller-supplied (that genuinely changes n_wet_aoi/wet_fill_pct), and can be turned off with auto_tiling=False to force the tiled path regardless of AOI size.

When tile_pixels is set, each annual window is loaded tile-by-tile via :func:hydroseason.io.iter_wofs_tiles_from_stac instead of as one whole-AOI load. Already-cached tile CSVs (under a per-year tile-cache directory derived from the annual cache path) are read and their ids passed as skip_tile_ids, so an interrupted year resumes at tile granularity on the next call. This has no effect on the annual cache's identity: a complete annual result is tile-shape-independent, so it is written to and read from the same cache file as the untiled path.

wet_aoi, if given, is an already-computed wet-AOI GeoDataFrame (see :func:hydroseason.io.compute_wet_aoi) threaded into every tiled per-year load's n_wet_aoi/wet_fill_pct computation, and -- only when a full_ts cube is also available to reconcile against (see below) -- as a second, independent tile-skip gate too. If precompute_wet_aoi is True and wet_aoi is not supplied, one full-time-series load_wofs_from_stac pass over the whole requested window is used to derive it via :func:hydroseason.io.compute_wet_aoi (using persistence_min, close_m, buffer_m), before any per-year tiled loads happen. precompute_wet_aoi requires tile_pixels -- pruning only exists on the tiled path, so precomputing a wet AOI without tiling would be a no-op the caller almost certainly didn't intend.

KNOWN LIMITATION: this full-time-series precompute pass runs unconditionally, even when every year in the requested range is already cached from a prior run -- it is not skipped on a fully-cached resume. This is because the per-year cache key includes a hash of the derived wet_aoi itself (see wet_aoi_hash below), which cannot be known before wet_aoi is actually derived; there is no cheaper way to check "is this already cached" without first paying the cost being checked for. A fully correct fix would persist the derived wet-AOI geometry as its own cache artifact (keyed on the non-wet-AOI-hash-dependent inputs) so it can be reloaded cheaply on a later call instead of re-derived -- tracked as follow-up work, not implemented here. This is a performance regression on repeat calls, not a correctness issue: results are still correct, just not resumed cheaply.

Pruning tiles that the wet AOI excludes guarantees those tiles contribute no water -- but iter_wofs_tiles_from_stac never loads them, so their n_aoi/n_valid/n_invalid pixel counts (which genuinely can differ per month, e.g. cloud-affected pixels, and cannot be inferred from geometry alone) would otherwise silently vanish from the tiled aggregate's denominator instead of being counted as unseen-but-real AOI pixels -- corrupting extent_pct even though pruning never touches which pixels contribute water. When wet_aoi was derived internally here (not supplied by the caller), the already-loaded full_ts cube covers the exact same AOI/CRS/resolution as the tiled path and was fetched regardless of pruning, so it is reduced once per year with :func:hydroseason.hydro_year.monthly_water_extent and its n_water/n_aoi/n_valid/n_invalid/extent_pct/ invalid_pct replace the (potentially pruning-truncated) tiled aggregate's for that year -- an exact, no-extra-STAC-cost source of truth, computed from data already resident rather than reconstructed. Only n_wet_aoi/wet_fill_pct are left to the tiled aggregate, since those two are legitimately allowed to differ under pruning.

This reconciliation only applies when precompute_wet_aoi derived wet_aoi here, because only then is a full_ts cube available to reconcile against. A caller-supplied wet_aoi has no accompanying full-time-series cube, so there is no ground truth to correct the tiled aggregate's denominator against if pruning were allowed to run -- and running it anyway would silently corrupt extent_pct/invalid_pct. To guarantee correctness, pruning is therefore automatically disabled (falling back to loading every tile, unpruned) whenever there is no full_ts to reconcile against -- i.e., only a precompute_wet_aoi- derived wet_aoi currently benefits from tile-skip pruning; an externally-supplied wet_aoi does not, today. This is a real, documented capability boundary, not a bug: an externally-supplied wet_aoi still gets its n_wet_aoi/wet_fill_pct computed correctly against the real wet-AOI geometry (that calculation only reads pixels from tiles actually loaded, and every tile is loaded when pruning is disabled, so it has no missing-tile denominator problem of its own) -- it just does not skip loading any tiles.

complete_monthly_axis

complete_monthly_axis(masks, start_date: str, end_date: str, *, invalid_value: int = -1, duplicate_month_policy: Literal['raise', 'warn'] = 'raise')

Reindex a lazy mask cube to complete monthly starts; gaps become invalid.

open_wo_statistics

open_wo_statistics(aoi: Any, *, product: str = DEFAULT_WO_STATISTICS_PRODUCT, stac_url: str = DEFAULT_WO_STATISTICS_STAC_URL, resolution: float = 30.0, crs: str = 'EPSG:3577', chunks: Mapping[str, int] | None = None) -> 'xr.Dataset'

Load native DEA Water Observation Statistics for aoi.

Public, general-purpose statistics loader: a single STAC search against product (default the all-time ga_ls_wo_fq_myear_3 summary), requesting exactly the two raw count bands (count_wet, count_clear) and deriving frequency (0-100) lazily as 100 * count_wet / count_clear. This is the SAME ratio DEA's own precomputed frequency band encodes, requested this way so the derivation is explicit and auditable via provenance rather than trusting an opaque upstream band.

Unlike :func:fetch_dea_stats_wet_aoi, this function does no planning reduction: no union-of-years, no vectorisation, no buffering. It returns the raw, dask-backed statistics cube at whatever native grid crs and resolution describe -- by default DEA's native 30 m WOfS/Albers grid. Turning statistics into a planning mask is a separate, later concern (WetPlanningFootprint / build_wet_planning_footprint, not part of this function).

resolution/crs are passed to odc.stac.load explicitly so the output grid is never implicitly resampled or coarsened; there is no scientific-resolution knob here; callers that need a coarser working grid resample the returned dataset themselves, deliberately, downstream.

Returns a lazy (Dask-backed) xarray.Dataset with data variables count_wet, count_clear, frequency and .attrs["provenance"] recording product, stac_url, the resolved STAC item IDs, and how frequency was derived. Never calls .load()/.compute().

Raises :class:WoStatisticsUnavailable (a :class:DEAStatsUnavailable) if the endpoint is unreachable, or the STAC search fails, times out, or returns no items. Does not raise on a geographic CRS -- CRS (guard_area_metric_crs); this loader is source-agnostic and hands whatever grid was asked for, unsigned COG reads and all, straight back.

build_historical_water_mask

build_historical_water_mask(stats: 'xr.Dataset', aoi: Any) -> HistoricalWaterMask

Build the exact (count_wet > 0) AND AOI historical water mask.

stats is the xr.Dataset returned by :func:hydroseason._io_dea_stats.open_wo_statistics (must be the all-time Multi-Year product, ga_ls_wo_fq_myear_3). aoi is a user AOI geometry/GeoDataFrame, rasterized onto stats's native grid via :func:hydroseason._io_geo._inside_aoi_mask_like (the same AOI-onto-grid rasterizer _clip_to_aoi already uses elsewhere in the package). The result is never closed, buffered, dilated, or converted through a polygon: it is exactly (count_wet > 0) & rasterized_aoi, materialized as a plain boolean numpy array.

Raises :class:hydroseason._io_dea_stats.DEAStatsUnavailable (fail-closed, matching the identical three-category validation :func:hydroseason._io_dea_stats.build_wet_planning_footprint already performs against the same stats.attrs["provenance"] contract) for:

• an incompatible source product/lineage (anything other than ga_ls_wo_fq_myear_3, or a version token that does not match the monthly WOfS collection ga_ls_wo_3) -- message contains "incompatible WOfS lineage";
• an exact mask with no True cells after the AND-with-AOI step -- message contains "no historically observed water".

load_or_build_historical_water_mask

load_or_build_historical_water_mask(aoi: Any, *, cache_root, offline: bool = False, stac_url: str | None = None, product: str | None = None, crs: str = 'EPSG:3577', resolution: float = 30, end_date: str | None = None, refresh_historical_mask: bool = True) -> HistoricalWaterMask

Resolve a verified :class:HistoricalWaterMask for aoi, cache-first.

Resolution order: (1) a verified cache hit via :func:read_historical_water_mask -- zero network access; (2) exactly one :func:hydroseason._io_dea_stats.open_wo_statistics load plus :func:build_historical_water_mask, persisted via :func:write_historical_water_mask before being returned.

In offline=True mode, or after a Statistics load/build failure, returns ONLY a verified cache -- never falls through to constructing a full-AOI mask by any other means. If no verified cache exists in either case, raises :class:hydroseason._io_dea_stats.DEAStatsUnavailable (offline with no cache: "no cached historical water mask"; online with a Statistics failure: the underlying failure, re-raised).

stac_url/product default to :data:hydroseason._io_dea_stats.DEFAULT_WO_STATISTICS_STAC_URL / :data:hydroseason._io_dea_stats.DEFAULT_WO_STATISTICS_PRODUCT when not given. cache_root is required and has no default -- this function caches to disk, and picking a directory on the caller's behalf (e.g. relative to the current working directory) is not a decision this module makes silently. This mirrors every other cache-bearing entry point in the package (:mod:hydroseason._io_wofs_zarr, :mod:hydroseason._io_wofs_acquire, :mod:hydroseason._io_stac_cache), none of which default cache_root either. A shared default cache root across the high-level API is expected to be wired through by a later task, not invented here.

end_date is the caller's requested analysis-window end, used ONLY to decide whether a cache hit is worth a refresh check -- see :func:_maybe_refresh_cached_mask. It is never baked into the cache request/artifact digests (an artifact still serves any requested window; see :class:HistoricalWaterMaskRequest). Defaults to None for callers with no analysis window of their own (e.g. scripts that only need a mask, not a monthly run) -- None skips the refresh check entirely and behaves exactly like the pre-refresh strict-pinning behavior, since there is no requested window to compare against coverage_end.

On a cache hit whose requested window extends past the cached artifact's recorded coverage_end, and refresh_historical_mask is True (the default) and offline is False, a cheap metadata-only probe checks whether DEA now has wider Statistics coverage; if so, the mask is rebuilt and a NEW artifact is written and repointed to (the old artifact is never deleted). Set refresh_historical_mask=False to restore strict pinning -- a cache hit is always returned as-is, useful for deterministic regeneration. The probe is never entered for offline=True or for a window already inside the cached coverage, and a probe/rebuild failure of any kind falls back to the cached artifact rather than becoming fatal.

build_wet_planning_footprint

build_wet_planning_footprint(stats: 'xr.Dataset', *, factor: int = 4, safety_cells: int = 1, requested_years: Collection[int]) -> WetPlanningFootprint

Build a conservative coarse wet-pixel planning footprint from stats.

stats is the xr.Dataset returned by :func:open_wo_statistics (count_wet/count_clear/frequency at native resolution, with .attrs["provenance"]). This is a PERFORMANCE-ONLY artifact: it gates which spatial windows a later remote read touches, and must never be confused with or fed into zoning (build_zones()).

Steps, matching the task's correctness contract exactly:

1. native_mask = count_wet > 0 at native resolution.
2. coarse_mask = native_mask.coarsen(y=factor, x=factor, boundary="pad").max() -- aligned max-pooling only (never nearest/mode/mean), with boundary="pad" so a trailing partial block is padded (False) rather than dropped, preserving edge cells.
3. A safety_cells-wide coarse-grid dilation halo is applied on top, covering grids that are not exactly aligned.

Every accepted footprint satisfies native_mask <= expand(coarse_mask) (the round-trip proof) -- see the test suite in test_io_dea_stats.py.

active_windows are derived directly from coarse_mask as a raster/chunk predicate (:func:hydroseason._spatial_plan.active_windows_from_mask); no polygon is vectorised on this path. geometry stays None.

Fails open (raises :class:DEAStatsUnavailable) rather than returning a partial or empty mask when:

• count_wet has no wet pixels at all -- an empty footprint could be mistaken for "nothing here to prune" and must never be returned.
• stats.attrs["provenance"] is absent or missing a resolvable collection/version -- the statistics/daily-observation lineage/version contract is itself unverifiable.
• stats's recorded time_span does not cover every year in requested_years.

acquire_wofs_cache

acquire_wofs_cache(*args, **kwargs)

Acquire or reuse the canonical local WOfS mask cache.

open_completed_mask_cache

open_completed_mask_cache(*args, **kwargs)

Lazily open the canonical WOfS water-mask cube for a completed cache store.

Public reader counterpart to :func:acquire_wofs_cache: given a :class:WOfSCacheHandle (as returned by acquire_wofs_cache) and a [start_date, end_date] range, opens every completed annual Zarr group that overlaps the range, concatenates them in year order, and fills any still-missing months with the package's standard missing-month convention (-1 invalid, never fabricated). See :func:hydroseason._io_wofs_zarr.open_completed_mask_cache for the full contract, including its FileNotFoundError/ValueError cases.

open_completed_extent_counts

open_completed_extent_counts(*args, **kwargs)

Internal facade for the extent counts reader.

open_completed_dual_extent_counts

open_completed_dual_extent_counts(*args, **kwargs)

Read back the second (any-day-wet max_water) composite's per-month pixel counts.

Public reader counterpart to :func:acquire_wofs_cache when it was called with composite_bundle="dual_composite_v1": given a :class:WOfSCacheHandle and a [start_date, end_date] range, returns a pandas.DataFrame combining every completed year's years/<year>/dual_extent_counts.json sidecar -- the SECONDARY composite's per-month wet/valid pixel counts alongside the fixed full-AOI/analysis-mask pixel-count denominators -- or None if any requested year is incomplete, the sidecar is missing/malformed for any requested year (including a store acquired with the default composite_bundle="single_mask", which never writes this file), or the resulting range has no rows. See :func:hydroseason._io_wofs_zarr.open_completed_dual_extent_counts for the full contract.

verify_cache_footprints

verify_cache_footprints(*args, **kwargs)

Read, independently re-rasterize, and verify a cache's persisted AOI/analysis footprints.

Public reader/verifier counterpart to :func:acquire_wofs_cache: given a :class:WOfSCacheHandle, reads the full-AOI and analysis-footprint geometry/counts/digests persisted in the store's root manifest, re-rasterizes each geometry from its persisted canonical WKB, and cross-checks both the digest and the pixel count against what was persisted -- never trusting either alone. Raises ValueError on any tamper/corruption mismatch, or FileNotFoundError/ValueError if no manifest or no footprints metadata exists yet. See :func:hydroseason._io_wofs_zarr.verify_cache_footprints for the full contract.