# Observability

Rask is instrumented for production operations with standard .NET primitives — no extra packages,
OpenTelemetry-exportable out of the box. Five surfaces:

1. **Structured logging** — framework faults flow into your `ILogger` pipeline.
2. **Metrics** — a `Meter` named `Rask.Server` with session, handler, and frame counters/histograms, plus
   one meter per DB-backed pillar (`Rask.Jobs`, `Rask.Outbox`, `Rask.Mail`) carrying throughput, failures
   and **dead letters**.
3. **Tracing** — an `ActivitySource` named `Rask.Server` spanning handler dispatch.
4. **Health checks** — a live-session capacity check.
5. **Durable logs** — opt-in: [`Rask.Logging`](https://rask.sh/docs/guides/logging/) keeps the `ILogger` pipeline in a database of its
   own, so what happened survives the restart that hid it.

Everything is on by default with zero configuration; you opt in to *exporting* it.

## Structured logging

`Rask.Core` and the WASM host take no dependency on `Microsoft.Extensions.Logging` — instead they
report diagnostics through an internal seam (`RaskDiagnostics`). On the server, `MapRask<TApp>()`
bridges that seam to your application's `ILogger` automatically. From then on every framework
diagnostic — a lifecycle hook that threw in a component rendered outside a host, a component `Dispose`
that faulted, a duplicate sibling `Key`, a malformed WebSocket frame, an event handler that threw —
is logged through an `ILogger` named for the diagnostic's category, at the matching level, with the
original exception attached:

| Category | Emitted for |
| --- | --- |
| `Rask.Errors` | Every exception an `ErrorBoundary` took — a render, a handler, a lifecycle hook — once, at Error, with the exception and the reference the visitor's [error page](https://rask.sh/docs/guides/lifecycle/#what-a-visitor-sees-when-something-throws) shows |
| `Rask.Lifecycle` | Faulting lifecycle hooks / disposes with no ancestor `ErrorBoundary` |
| `Rask.Diff` | Duplicate sibling `Key` (keyed reconciliation silently disabled) |
| `Rask.Live` | Handler / navigate / JS-invoke faults, malformed frames, a session asked for renders without end (which components asked) |
| `Rask.JsInvoke` | Failures surfacing a JS-invoke fault back to the caller |
| `Rask.HotReload` | Dev-time hot-reload failures: a registry refresh that threw, a repaint that faulted, or an undeliverable applied-notification |

No wiring is needed — configure log levels for these categories like any other:

```jsonc
// appsettings.json
{
  "Logging": {
    "LogLevel": {
      "Rask.Live": "Warning",
      "Rask.Diff": "Warning"
    }
  }
}
```

When no `ILoggerFactory` is registered (e.g. a bare test host), the seam keeps its default behaviour
and writes the same diagnostics to `stderr`.

A WebAssembly app gets the same bridge from `WasmHostBuilder.Run`, into the app's own logging. If
the app registers no `ILoggerProvider` of its own, the host adds one that writes to the browser console:
errors through `console.error`, everything from `Information` up through `console.log`. That way a
framework warning is never silently dropped. Register any provider yourself and the host adds nothing,
so your providers are the only ones that see the entries:

```csharp
var host = WasmHostBuilder.CreateDefault();
host.Services.AddSingleton<ILoggerProvider, MyTelemetryLoggerProvider>(); // replaces the console default
await host.Run<App>();
```

### Keeping the log

Everything above is a *transport*: the diagnostics reach whatever sinks you configured, which on a
single-box deployment usually means the container's stdout — gone with the next restart. Add
[`Rask.Logging`](https://rask.sh/docs/guides/logging/) to keep them:

```csharp
builder.Services.AddRaskLogging();   // a SQLite file of its own, at Rask:ConnectionStrings:Logs
```

It registers an `ILoggerProvider`, so it captures exactly what every other sink sees — the categories above
included — into a SQLite file of its own (or, on PostgreSQL or SQL Server, into the app's own database with
`AddRaskLogging<AppDbContext>()`), with retention by age and row count and a searchable view in the
[dashboard](https://rask.sh/docs/guides/dashboard/). Its `rask.logs.dropped` counter is the one metric that tells you the stored log is
incomplete.

## Metrics

All metrics publish on the meter named **`Rask.Server`** (`RaskTelemetry.MeterName`).

| Instrument | Kind | Tags | Meaning |
| --- | --- | --- | --- |
| `rask.sessions.created` | Counter | | Live sessions created (component tree + DI scope). |
| `rask.sessions.rejected` | Counter | | Creations refused because `MaxSessions` was reached. |
| `rask.sessions.evicted` | Counter | | Sessions removed (disconnect grace elapsed / shutdown). |
| `rask.sessions.active` | Gauge | | Live sessions currently held. |
| `rask.handlers.dispatched` | Counter | | Client event handlers dispatched to user code. |
| `rask.handlers.faulted` | Counter | | Handler dispatches that threw (isolated; session survives). |
| `rask.handlers.timedout` | Counter | | Handler dispatches cancelled by `HandlerTimeout`. |
| `rask.handlers.stale` | Counter | | Events that ran nothing because the page had changed under them: the handler they were sent to was gone, or could not be shown to be the same one. A few are normal (a double click on a row being removed); a steady stream is clicks being dropped — see [An event that outlived its render](https://rask.sh/docs/guides/js-interop-runtime/#an-event-that-outlived-its-render). |
| `rask.handler.duration` | Histogram (ms) | | Wall-clock duration of an event-handler dispatch. |
| `rask.ws.frames.rejected` | Counter | `reason` = `size` \| `rate` \| `backlog` \| `idle` \| `hello` | Inbound frames refused by a safety limit. |
| `rask.sessions.resumed` | Counter | | Pages rebuilt on a host that had never heard of the session, from the client's [resume record](https://rask.sh/docs/guides/configuration/#surviving-a-restart-or-a-redeploy). |
| `rask.sessions.resume_rejected` | Counter | `reason` = `malformed` \| `unprotect` \| `principal` \| `toolarge` \| `atcapacity` | Resume records refused. |
| `rask.shutdown.sessions.abandoned` | Counter | | Sessions still connected when the shutdown drain budget ran out; their sockets were aborted. |
| `rask.sessions.connected` | Gauge | | Sessions with a socket attached — people actually looking at the app, as opposed to `active`, which also counts GET-minted sessions whose socket never arrived and sessions riding out their reconnect grace. |
| `rask.handlers.pending` | Gauge | | Handler dispatches queued across all sessions. The backpressure breaker's *input* — `frames.rejected{reason=backlog}` is its output. |
| `rask.render.duration` | Histogram (ms) | | Time to render a session and write its frame. The framework's half of a slow interaction; `handler.duration` is your half. |
| `rask.payload.bytes` | Histogram (By) | | Size of a frame sent to a client. Watch the distribution: a page that quietly stops diffing jumps to its full size here long before anyone notices the bandwidth. |

A nonzero `rask.shutdown.sessions.abandoned` is the signal that `ShutdownDrainTimeout` — or the
`HostOptions.ShutdownTimeout` containing it — is shorter than your app's real shutdown, so some users saw
an abnormal disconnect instead of a clean "Updating…" reload. Because it is emitted *during* shutdown, its
delivery depends on your exporter's final flush; the same fact is logged, which is the more reliable read.

The `rask.ws.frames.rejected` counter is the headline DoS-visibility signal: a spike on
`reason=rate` or `reason=backlog` means a client is being throttled by the per-connection frame-rate
cap or the pending-handler backpressure breaker. `reason=idle` is the `IdleSocketTimeout` reclaiming a
silently-idle connection, not an attack. `reason=hello` is a socket that sent a second session-establishing
`hello`; the browser runtime sends one per connection, so any count here is a client other than Rask's own.

`rask.sessions.resume_rejected` is worth an alert of its own. A steady trickle is normal — expired records
from laptops that slept. **A spike on `reason=unprotect` immediately after a deploy means your
data-protection key ring is not surviving the deploy**, so every user is getting a reload instead of their
page back (and, if you use cookie auth, being signed out). See
[surviving a restart](https://rask.sh/docs/guides/configuration/#surviving-a-restart-or-a-redeploy).

## Pillar metrics

Each DB-backed pillar publishes on its own meter, registered by its `AddRaskX` call. The shape is the same
for all three — only the noun changes.

| Instrument | Kind | Tags | Meaning |
| --- | --- | --- | --- |
| `rask.jobs.processed` | Counter | `job.type` | Jobs that ran to completion. |
| `rask.jobs.failed` | Counter | `job.type` | Attempts that threw — **every attempt**, not every job. |
| `rask.jobs.deadlettered` | Counter | `job.type` | Jobs that exhausted `MaxAttempts`, counted once. |
| `rask.jobs.interrupted` | Counter | `job.type` | Cancelled by shutdown past `ShutdownGracePeriod`. Re-runs on restart; counts **no** attempt. |
| `rask.jobs.duration` | Histogram (ms) | `job.type` | Wall-clock duration of one execution. |
| `rask.jobs.pending` | Gauge | | Not yet processed and not yet exhausted. |
| `rask.jobs.deadletters` | Gauge | | **The number worth alerting on.** |

`Rask.Outbox` publishes the same seven as `rask.outbox.*` tagged by `message.type`; `Rask.Mail` publishes
`rask.mail.sent` / `failed` / `deadlettered` / `interrupted` / `duration` / `pending` / `deadletters`.

**`rask.*.interrupted` says your shutdown budget is too short.** It counts work that a `SIGTERM` cancelled
after its `ShutdownGracePeriod` — which re-runs from the top on the next boot, so any non-idempotent handler
is repeating its side effects. For `rask.mail.interrupted` it is sharper than that: because a cancelled SMTP
conversation can be accepted by the server before the row is marked, it is the direct answer to *"did that
deploy duplicate any mail?"* See [Shutdown and redeploy](https://rask.sh/docs/guides/configuration/#shutdown-and-redeploy).

**`rask.*.deadletters` is the alert.** Delivery is at-least-once with backoff, so a pillar retrying itself
to death still shows a healthy `processed` rate — the dead-letter gauge is the one that says work has been
abandoned. Pair it with the [dashboard](https://rask.sh/docs/guides/dashboard/), which shows *which* rows and why.

> **Why mail has no type tag.** Jobs and the outbox tag by their registered type, a closed set fixed at
> build time by a source generator. Mail's only per-message dimensions are subject and recipient, both
> unbounded — tagging by either would mint a time series per email sent.

> **The queue-depth gauges cost nothing until you subscribe.** They are sampled by the processor's existing
> poll and only while a listener is attached, rather than running `COUNT(*)` inside the observable-gauge
> callback — which would otherwise put read load on the app's database on the collector's schedule.

### Reading metrics locally

```bash
dotnet-counters monitor --counters Rask.Server,Rask.Jobs,Rask.Outbox,Rask.Mail --process-id <pid>
```

### Exporting via OpenTelemetry

```csharp
builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m
        .AddMeter(RaskTelemetry.MeterName)
        .AddMeter(JobMetrics.MeterName)
        .AddMeter(OutboxMetrics.MeterName)
        .AddMeter(MailMetrics.MeterName))
    .WithTracing(t => t.AddSource(RaskTelemetry.ActivitySourceName));
```

## Tracing

Each event-handler dispatch starts an `Activity` named `rask.handler.dispatch` on the
`Rask.Server` `ActivitySource`, tagged with `rask.handler.id` and marked `Error` (with the exception
message) when the handler throws. With no listener registered the span is never materialised, so the
instrumentation costs nothing. Subscribe with `.AddSource(RaskTelemetry.ActivitySourceName)` as
above (or any `ActivityListener`).

## Health checks

`AddRaskLiveSessions()` registers two checks, because they answer different questions: capacity
(`RaskLiveHealthCheck`, tagged `live`) and readiness (`RaskReadinessHealthCheck`, tagged `ready`).
Register them on your health-checks pipeline:

```csharp
builder.Services.AddRask();   // "Rask": { "Live": { "MaxSessions": 1000 } } in appsettings.json
builder.Services.AddHealthChecks().AddRaskLiveSessions();
// ...
app.MapHealthChecks("/health");
app.MapRask<App>();
```

| Status | Condition |
| --- | --- |
| `Healthy` | Below 80% of `MaxSessions` (or uncapped), and memory below 80% of the limit. |
| `Degraded` | At or above 80% of `MaxSessions`, **or** memory at 80% of the limit. |
| `Unhealthy` | At `MaxSessions` (new sessions are being refused with `503`), **or** memory at 92%. |

**Memory is checked as well as the session count, and outranks it.** A cap alone can't keep a host
healthy, because what a session costs is a property of the page rather than of the user: the same host
holds ~45,000 sessions of a trivial page or ~1,250 of a 200-row grid. A cap sized for the small page is no
protection on the big one. The reading comes from `GCMemoryInfo`, which honours a container memory limit,
so it reflects the ceiling a deployed app actually runs under — and an uncapped host, which is what most
apps run, now reports something before an OOM does. A memory position the runtime won't disclose is
treated as healthy, not as full: a host must not shed load because it can't measure itself.

`activeSessions`, `connectedSessions`, `maxSessions` and `memoryLoad` are attached to the health-check
result `data` for dashboards. Pair the capacity cap with a reverse-proxy rate limit for precise admission
control.

### Readiness

`RaskReadinessHealthCheck` answers one question: *is this instance still accepting live sessions?* It is
`Healthy` normally and `Unhealthy` from the moment a graceful shutdown begins, so an aggregate `/health`
returns `503` while draining and a load balancer with active probes stops routing here.

It is kept separate from the capacity check on purpose — that one's `Unhealthy` already means "at
`MaxSessions`, refusing with `503`", and folding a second cause into it would make both readings
ambiguous exactly when someone is diagnosing an incident. Split them with the tags when you want distinct
probes:

```csharp
app.MapHealthChecks("/health/ready",
    new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") });
app.MapHealthChecks("/health/live",
    new HealthCheckOptions { Predicate = c => c.Tags.Contains("live") });
```