Skip to main content

Problems and alerts

Datagrok watches itself. When a service fails, one error hits many users, requests slow down, an account keeps failing to sign in, a data source stops answering, or a user reports a problem, the server records a problem and raises an alert about it. The alert goes to your monitoring system as a log record, once, and stays open until the problem ends or a person resolves it.

ProblemAlert
What it iswhat is wrong, for example "service Jupyter fails"the notification that it went wrong
How long it livesone per condition; forgotten 30 days after it ended with no open alert, unless it is muted or not a problemuntil the problem ends or a person resolves it
Who changes ityou set its status; Datagrok tracks whether it is happeningDatagrok opens it; you acknowledge and resolve it

What Datagrok detects​

Services that fail their health checks, errors that hit many users or keep recurring, errors users keep making, slow or failing requests, data connections and scheduled jobs that fail, containers that don't start, storage, identity providers and Log sync destinations that keep failing, problem reports, the database, storage or disks growing too much (metric alerts), tables that no vacuum reached for long (vacuum: 10K rows or more, 20% of them dead, not vacuumed for 7 days), and the conditions of your own problem rules, such as the default login-failures. When each one opens and ends, who acts on it, and which thresholds an administrator can change in Health > Alerts > Rules: What Datagrok detects in the operator guide.

Statuses​

Each problem has a status that decides whether it alerts:

StatusMeaningWhen it happens again
Activeneeds attentionalerts
Mutedknown, quiet for now: for a while, until a version, or until you lift the muteno alert; active again when the mute ends
Not a problemexpected behaviornever alerts. An error marked so stops counting as an error
Fixedsomeone fixed italerts again only if it comes back after the fix, marked as a regression

What to do with an alert:

  • Resolve it when you have seen it. The problem stays active: if the condition still holds, a new alert opens on the next check.
  • Mute the problem to stop its alerts. Muting resolves the open alert, and is the way to stop alerts about a problem.
  • Mark the problem fixed to hear about it only if it comes back.
  • Each action has its undo: Unacknowledge, Reopen a resolved alert, Unmute, Undo not a problem, Undo fixed.

An alert that is open does not page again while its condition persists. When the condition ends, the alert is cleared and resolved.

The Health overview​

Browse > Platform > Health shows the state of the instance on one page:

  • Status: "All clear", or how many problems need attention, by severity. A problem needs attention while its condition holds and nobody muted it, marked it fixed or not a problem, or resolved its alert. A resolved alert's problem needs attention again when its condition raises a new alert.
  • Vitals of the last 24 hours against the day before: requests and the share answered with an error status (400 or above), p95 latency, errors and the users who met them, the function queue, the database size and cache hit rate, and blocked database connections; and the HTTP connections and open sockets of all server instances, with the busiest one's share of its open files limit. A tile opens Metrics; Errors opens the error events in Logs; Requests, DB connections, HTTP connections and Open sockets open each instance and isolate in Metrics. Growing sockets raise the metric alerts socket-growth and too-many-sockets.
  • Needs attention: the most severe and latest problems. Click one to see it in the context panel; hover it to acknowledge or resolve its alert, mute it, or open its evidence.
  • Top errors of the last 24 hours. Click one to see its events in Logs.
  • Services: every health check, the failing ones first. Click a service to open its logs.
  • Monitoring setup: the alert rules, the log settings, and how long records are kept.

The page refreshes every minute.

Every Health page works the same way:

  • It appears at once as a skeleton; each card and block then loads on its own and refreshes in place.
  • The ribbon shows when the page was last updated; the refresh icon spins while it loads.
  • Hover a card, a column header or a field to read what it is, which values are good and which are a concern.
  • Click a card to drill down to what is behind the number.
  • Export as dashboard (the archive icon) saves every table of the page as a zip. Drop the file into Datagrok to open it as a dashboard, for example to send the state of an instance to support.

Alert rules​

Health > Alerts > Rules holds everything that raises a problem:

  • Built-in detectors: error incidents, user errors, degraded requests, connections, service health, scheduled jobs, dashboards and data growth. Each card has the detector's thresholds and says how many of its problems are failing now.
  • Custom rules: your own conditions on logged events.

One Save saves both.

Metrics, logs and log settings​

  • Metrics: the database, connections, the function queue, storage and disks, errors, latency and sessions for the chosen window; data growth; the slowest queries; the largest and the least healthy tables; and the HTTP routes. Click a failed share of a route to see the failed requests and their log.
  • Logs: one explorer for the platform's events and the logs of its services. Service logs are read from what the services pushed into Datagrok; switch to Live to follow a container as it writes. On AWS deployments the explorer also reads CloudWatch and the log archive.
  • Log settings: the log groups, where records are sent, and the in-memory buffer.

The Alerts view​

Browse > Platform > Health > Alerts lists the active problems: those whose condition holds or whose alert is not resolved yet, the failing and most severe first. Set Show to All to see the problems that cleared as well. A problem's row reads:

ColumnWhat it showsValues
Severityhow bad it iscritical, warning, info
Conditionwhether it is happening now, as Datagrok sees itfailing (red), ok (green)
Alertits alertopen (red), acknowledged (orange), resolved (green)
Handlingthe status you setwatched (blue): an active problem, guarded and alerting when it fails; muted, not a problem (grey); fixed (green)

The icons in Actions acknowledge or resolve the alert, mute the problem, and open its evidence. The Actions section of a problem's context panel also marks it fixed or not a problem, or watches it again. The Diagnose section links to where to look next: the rule or the detector that raised the problem, the logs of its service, the errors of the last day, or the request and database metrics. Hover a column header to read what the column means.

Where alerts go​

Alerts are log records, all of one alert with the same alert id, plus a heartbeat every 5 minutes while the instance is alive (the records and when they arrive). Send them to your monitoring system with an OpenTelemetry destination in Log sync (its Alerts option is on by default) and page from them:

Your stackRule
Prometheus and Alertmanager behind an OpenTelemetry Collectorcount the records by alert id; fire from opened or firing until resolved
Grafana Cloud, Datadog, New Relic, Elastic, Dynatraceopen on datagrok.alert.opened, close on datagrok.alert.resolved with the same datagrok.alert.id
A log store or SIEMalert on the alert-opened record

Page on a missing heartbeat too: no heartbeat for 15 minutes means the instance is down, hung, or cut off. A reference collector, alert rules and routing are in the operator guide.

Datagrok also notifies people in the product when an alert opens or gets worse, once per alert:

  • An owner's alert goes to the owner: the author of the connection or of the scheduled function, or the owners of a problem rule. It links to the connection or function.
  • A platform alert, and an owner's alert with no owner, goes to everyone with the Manage Alerts permission.

Turn these notifications off in your notification settings. A query on a connection that was down on its last check says so in its Details, and warns before it runs. A scheduled function shows its last successful and last failed run in its Details. Each record carries the problem's audience: platform for the platform's operators, owner for problems that belong to the owner of a data connection or a rule, so the people who run the platform are not paged for them.

Working with problems​

Problems and alerts need the Manage Alerts global permission, which administrators have. Use the grok CLI or the REST API (/api/problems). A problem is named by its id or by kind:key, for example health:Jupyter@<server>, report:<report id> or connection:<connection id>:

grok s observe problems list --alert-status open # the open alerts
grok s observe problems resolve report:<report id> --reason "duplicate"
grok s observe problems list --status active --since 7d
grok s observe problems status connection:<connection id> --data '{"status": "muted", "reason": "monthly maintenance", "until": "2026-10-04T06:00:00Z"}'
grok s observe problems status health:Jupyter@<server> --data '{"status": "fixed", "reason": "kernel image rebuilt"}'
grok s observe problems history health:Jupyter@<server> # everything that happened to it

An error incident's Diagnose links open the errors of the last day and the platform events in Health > Logs; Top errors on the overview opens each error's events there.