# Polling and triggers This reference describes the schedules configured in the repository. It does not establish that a deployed schedule is healthy. Source observations, completed assets, published revisions, and delivered alerts are separate checkpoints. ## Trigger ownership | Work | Owner and trigger | Persistent checkpoint | | -------------------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | Live MRMS discovery | data, cron \*/2 \* \* \* \* (every two minutes, UTC) | D1 source cursors and source assets | | Configured historical benchmarks | data, the same cron; incomplete registry entries are reconsidered | D1 backfill state and event-scoped assets | | MRMS processing | Ingestion queue consumer | D1 asset outcome, R2 raw object and rendered frame, revision products | | Revision notification | Revision queue consumer; outbox dispatch follows ingestion and scheduled discovery | D1 outbox, then the event room's last notice | | Operational artifact cleanup | data, the scheduled invocation at 04:00 UTC | R2 object deletion; no archive-completeness checkpoint | | NWS warning evaluation | hail-alerts, cron \*/2 \* \* \* \* | Application-local Durable Object state and delivery ledger | | Place search | A client request, with response caching | No scheduled source ingestion | Wrangler owns these triggers and queue consumer settings. Terraform owns the backing resources. The application owners are [Data](/stormtrust/apps/stormtrust-data/) and [Hail Alerts](/stormtrust/apps/stormtrust-hail-alerts/). Their source contracts are listed in [Datasets and source contracts](/stormtrust/product/platform-and-data/datasets-and-source-contracts/). Storm Forecast ingestion, county/work-area ingestion, Storm Report generation, and customer archive maintenance do not yet have configured schedules. Their product requirements must not be interpreted as running jobs. ## Live MRMS discovery The current poller reads `MESH_00.50` and ` MergedReflectivityAtLowestAltitude_00.50` from ` https://noaa-mrms-pds.s3.amazonaws.com`. These are the configured products; adding a source requires an explicit contract and implementation change. | Bound | Implemented behavior | | ---------- | ---------------------------------------------------------------------------------------------------------- | | Date scope | Current UTC date; also the previous date during the first ten minutes after midnight | | Listing | One S3 listing per product/date, at most 1,000 keys and 768 KiB of listing data; no continuation-page loop | | Selection | At most the newest 12 eligible objects in each product/date listing | | Cursor | Per-product last\_source\_key, supplied as start-after when it belongs to that date prefix | | Input | .grib2.gz objects, with compressed input bounded to 16 MiB | | Identity | Source, object key, and source version derived from last-modified time, size, and ETag | The cursor advances after enqueueing the selected objects. A large backlog can therefore leave older objects unprocessed. An overwrite or late object behind the cursor is not automatically rediscovered. Source-version identity prevents some duplicate processing; it does not provide a complete correction scan. A discovery failure prevents that invocation's subsequent outbox dispatch and cleanup. Benchmark discovery runs alongside live discovery; a benchmark failure is logged separately and does not by itself stop live discovery. The MRMS fetch paths bound response sizes but do not configure an explicit request-abort timer. ## Queue and publication contract | Queue | Concurrency | Maximum batch / wait | Retries / delay | Dead-letter queue | | ------------------------- | ----------- | ----------------------- | --------------- | ----------------------------- | | stormtrust-data-ingestion | 1 | 4 messages / 5 seconds | 5 / 30 seconds | stormtrust-data-ingestion-dlq | | stormtrust-data-revisions | 4 | 20 messages / 2 seconds | 5 / 10 seconds | stormtrust-data-revisions-dlq | Ingestion messages use schema version 2\. Processing failures record the asset failure and request queue retry. Already completed assets are skipped. Invalid message shapes are logged and acknowledged, so they do not reach a dead-letter queue through repeated failure. A completed asset is not yet a complete revision. Publication requires both configured products for the two-minute observation window. The revision and manifest identify the evidence; the notification only tells a consumer that a revision is available. See [Live revisions and historical evaluation](/stormtrust/product/platform-and-data/live-revisions-and-historical-evaluation/). Revision messages use schema version 1\. The outbox dispatches up to 50 pending entries per invocation, ordered by creation time. An ingestion-side dispatch failure leaves pending work for a later dispatch. The event room retains its last notice and suppresses older sequences. Consumers reconcile through the [read API](/stormtrust/product/build-and-operate/api/) after reconnecting; the WebSocket is not a complete history log. Neither queue has a repository-configured automatic dead-letter drain. Invalid messages and exhausted retries require an operator to inspect the failed stage, identity, and persisted output before authorizing recovery. ## Historical acquisition and recovery bounds The benchmark registry is checked on each data cron. A registry entry marked complete is skipped. The internal backfill interface supports windows of at most 12 hours whose end is at least five minutes in the past; it is an internal acquisition interface, not a customer replay operation. Use [Inspect a local benchmark](/stormtrust/engineering/operations/local-benchmark/) for the supported local procedure. Historical discovery inspects one listing page per product/date and filters by observation time. It does not paginate the complete bucket history. A backfill marked complete means its registered assets completed; it does not prove the expected upstream object count was discovered. Completed backfills do not continuously check for upstream corrections. Rediscovered assets in `discovered` or ` failed` state can be enqueued. Assets left in ` queued` or ` processing` do not have a separate automatic recovery scan. A benchmark backfill uses its own event identity; it does not repair the live feed's cursor or guarantee that missing live frames become available. ## Persistence and cleanup D1 stores discovery, processing, revision, and outbox metadata. R2 stores source objects, images, and manifests. Event rooms retain notification state. These stores have different lifecycles. At 04:00 UTC, cleanup removes operational raw objects older than 72 hours and operational frames older than 30 days, measured from R2 upload time. Frames referenced by the current manifest are retained. Cleanup requires the current manifest; without it, the routine deletes nothing. Benchmark event prefixes are outside this operational cleanup scope. A missed or failed cleanup has no separate catch-up timer. The next matching scheduled invocation is the next configured cleanup opportunity. Cleanup deletes R2 objects without pruning all D1 revision records or historical manifests, so an old revision listing can reference an expired image. Report retention pins and a complete customer archive are future requirements in the [History and Replay contract](/stormtrust/product/platform-and-data/intelligence/history-replay-contract/). ## NWS evaluation and delivery Hail Alerts evaluates the configured Missouri severe-thunderstorm-warning feed for its Kansas City market. It persists application-local state in the `kc-mo-hail-alerts-v1` coordinator. This is not a shared warning archive for all Storm Intelligence capabilities. | Condition | Implemented behavior | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- | | Active feed | Conditional request with ETag; HTTP 304 updates fetch metadata without replacing warnings | | Request bounds | 12-second timeout, 8 MiB response limit, at most 1,000 features | | Missing warning geometry | At most 20 allowed NWS zone URLs may supply fallback geometry | | HTTP 429 or 5xx | Error retains Retry-After text when supplied; no dedicated retry timer honors that value | | Poll failure | Error is persisted; a later scheduled or manual poll retries. The scheduled promise can resolve after an error was recorded | | UI state read | Reads stored state without polling NWS | | Manual poll | Evaluates the source without automatically sending email | | Acceptance replay | Fetches a rolling recent window with a 500-result limit and no pagination; uses a labeled fixture when needed | Scheduled live evaluation may deliver pending alerts only when delivery mode, allow-list, and cap checks permit it. The default shadow mode does not establish real delivery. Replay never automatically sends email. The application's acceptance replay can reset its local replay state; it is separate from the proposed customer [Storm Replay](/stormtrust/product/platform-and-data/intelligence/storm-replay/) capability, which reads retained evidence without rerunning acquisition or delivery. ## Operational evidence and next action `GET /v1/data/status` returns the latest discovered asset per product, including benchmark assets. Its statuses describe processing, not freshness or complete live coverage. An empty product list means no matching asset was found. A successful HTTP response or cron invocation does not establish healthy data. | Observation | Evidence needed before recovery | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | No recent source discovery | Scheduled invocation outcome, queried prefix, cursor, source response, and last discovery time | | Queued, processing, or failed asset | Exact source/version and event identity, queue attempts or dead-letter record, D1 state, and any existing R2 output | | Missing revision | Required product outcomes for the observation window and revision publication state | | Missed live notice | Published revision, outbox state, event-room sequence, and a fresh API reconciliation | | Missing historical image | R2 object availability and applicable retention scope; do not assume a listed revision guarantees retained bytes | | NWS or delivery failure | Coordinator error, last successful fetch, qualifying warning revision, delivery mode, and delivery ledger | An operator records the affected identity, timestamps, diagnostic evidence, and intended recovery scope before a retry or backfill. Recovery is complete only when new evidence establishes the expected output. Repeating a poll must not be presented as proof that skipped history or expired artifacts were restored. Source-specific freshness thresholds, escalation recipients, and automated stale-data alerts are not configured by this reference. Before relying on a capability operationally, its owner must define the accepted data age, expected scope, observation/check frequency, failure action, and retained recovery evidence. ## Implementation references Repository paths are the authority for the configured behavior: - `org/apps/data/wrangler.json`: cron and queue configuration. - `org/apps/data/worker/index.ts`: scheduled work, status response, and cleanup gate. - `org/apps/data/worker/discovery.ts` and ` backfill.ts`: listing, cursors, acquisition bounds, and benchmark selection. - `org/apps/data/worker/ingestion.ts`, ` revisions.ts`, and ` event-room.ts`: retries, publication, outbox, and notifications. - `org/apps/data/worker/mrms.ts`: artifact persistence and cleanup. - `org/apps/hail-alerts/wrangler.jsonc` and ` worker/index.ts`: NWS schedule and entry points. - `org/apps/hail-alerts/worker/nws.ts` and ` coordinator.ts`: fetch bounds, stored state, replay, and delivery decisions.