Skip to main content

Metric alerts

Datagrok keeps track of how its own data grows: the size and rows of each table of its database (and the counts of categories such as events by source or entities by type), its file storage by kind of data and location, each server's disk and open sockets, and what the garbage collector deleted and still has to delete. A metric alert watches these numbers and raises a problem when one is too high or grows too fast, with the same statuses, alerts and notifications as every other problem.

You can see the numbers in Browse > Platform > Health > Metrics: the data growth cards and the Data growth panel, with each value's change over 1, 7 and 30 days and its daily trend. Seeing them needs the View Telemetry global permission.

What is measured​

MetricKeyMeasured
db.bytesa table (events), or (total) for the whole databasehourly
db.rowsa tablehourly, an estimate
db.counta category of a table: events/source=usage, entities/type=Project, func_calls/status=Failed, …daily; estimated on tables over 5 million rows
storage.bytes, storage.objectsshared, shared/<kind> (packages, app-data, tables, user-files, …), shared/packages/<package>; local@<server> for a server's own diskevery 6 hours
disk.used_bytes, disk.free_bytes, disk.used_pctdisk@<device>:<size>G, such as disk@/dev/sda1:484G: the disk of the servers' data folder. The pods of a node share its disk, so it is sampled once, by the first server to see it duehourly
gc.deletedwhat the garbage collector shrank: events/source=debug, test_runs, shared/packages, …after each collection
gc.pendingthe same keys, for the steps that can be counteddaily, and after each collection
server.sockets, server.http_connections, server.db_connectionslocal@<server>: the sockets the server process holds, the HTTP connections it accepted, its connections to the platform databaseevery 5 minutes

The schedule and the categories counted are settings in Settings > Alerts > Data growth: metricsEveryMinutes (60), metricsBreakdownHours (24), metricsStorageHours (6) and metricsBreakdowns. The samples are kept for 400 days, the hourly ones for 14 (metricSamplesRetention in the garbage collector settings).

Where alerts live​

The alerts are one setting, Metric alerts (metricAlerts), in Settings > Alerts > Data growth: a JSON array, at most 30 alerts. Saving checks every alert and refuses the save with the first error. Changing it needs the Edit Plugins Settings global permission.

Out of the box it holds:

AlertMetric, keyRaises whenSeverity
database-growthdb.bytes, (total)the database grew 25% in 7 dayswarning
table-growthdb.bytes, any tablea table is over 1 GiB and doubled in a daywarning
storage-growthstorage.bytes, sharedthe storage grew 25% in 7 dayswarning
disk-lowdisk.free_bytes, any diska disk has under 20 GB freewarning
disk-fulldisk.free_bytes, any diska disk has under 5 GB freecritical
gc-backloggc.pending, any keythe garbage collector has over 10 million rows to deletewarning
socket-growthserver.sockets, any servera server holds over 1,000 sockets and twice as many as a day before: a leakwarning
too-many-socketsserver.sockets, any servera server holds over 10,000 socketscritical

Growth needs history: no growth alert holds until there is a sample old enough to compare with, so a new installation stays quiet for the first week. Saving your own alerts replaces the defaults, so keep the ones you want in the array.

An alert​

{
"name": "table-growth",
"metric": "db.bytes",
"key": "*",
"above": 1073741824,
"growth": {"percent": 100, "days": 1},
"severity": "warning",
"summary": "{key} {metric} {value} ({growth} in {days} d)"
}
FieldDefaultDescription
namerequired2–27 lowercase letters, digits or dashes, starting with a letter; unique. Its problems are of kind growth, keyed <name>:<key>, and its alert name is Datagrok and the name in Pascal case (table-growth: DatagrokTableGrowth)
enabledtrue
metricrequiredone of the metrics above
keyrequireda key, or a pattern: * stands for any characters, ? for one (events/source=*, local@*)
abovenoneholds when the latest value is greater
belownoneholds when the latest value is smaller, such as free space under a limit
growthnone{"percent": 25, "days": 7} or {"by": 1000000, "days": 1}: holds when the latest value grew by that much since the sample days (1–90) earlier
severitywarninginfo records the problem and never alerts; warning; critical
summarygeneratedthe alert text, with placeholders {alert}, {metric}, {key}, {value}, {previous}, {growth}, {percent}, {days}, {above}, {below}. Sizes read like 12.3 GB

Give above, below, growth or several; all that are given must hold, so a size floor keeps small tables from alerting on a doubling.

A problem ends when its alert no longer holds for two collections of its metric (two hours for the hourly ones, ten minutes for the sockets), so a value hovering at the threshold is one problem, not many. Removing or disabling an alert ends its problems. Alerts go to everyone with the Manage Alerts permission.

An alert that cannot be evaluated, or that holds for more than 100 keys at once, raises a problem-rule problem keyed metric-alert:<name>; the other alerts still run.

Examples​

Error events grow by a million a day:

{"name": "error-flood", "metric": "db.count", "key": "events/source=error", "growth": {"by": 1000000, "days": 1}}

Any package's files in the storage pass 5 GB:

{"name": "big-package", "metric": "storage.bytes", "key": "shared/packages/*", "above": 5368709120}

Debug events waiting for the garbage collector grow by half in a week (it is off, or falls behind):

{"name": "gc-behind", "metric": "gc.pending", "key": "events/source=debug", "growth": {"percent": 50, "days": 7}}

HTTP connections to one server pass 2,000:

{"name": "http-flood", "metric": "server.http_connections", "key": "*", "above": 2000}

From the API​

grok.dapi.admin.getGrowthMetrics({area, days}) returns the same numbers as the Metrics page: one row per key with its value now, its change over 1, 7 and 30 days, its daily trend, what the garbage collector did to it, and the metric alerts that cover it. It needs View Telemetry; the ids of the alerts' problems also need Manage Alerts.

The sockets of every server instance and of each of its isolates, the threads that serve requests, are gathered when you ask for them:

  • GET /api/admin/connections (View Telemetry): one entry per instance with its process's openSockets, httpConnections, dbConnections and openFilesLimit, and isolates: for each, the HTTP connections it accepted (httpActive, httpIdle) and its database pool connections (dbInUse, dbAvailable, dbWaiting). Each isolate also counts the requests it served in the last 5 minutes and the last hour (requests5m, requests60m) and those in flight (requestsInFlight), and the most its pools may open (dbMax); the instance sums its isolates (dbPoolConnections, dbInUse, dbWaiting, dbMax) and has the df of its data folder (disk).
  • GET /api/admin/metrics has them under instances, and their sum under process, with usedPct, the busiest instance's share of its open files limit.
  • GET /api/admin/health?instances=true (View Telemetry) describes them all in the record sockets, such as 3 instances, 9 isolates: 210 open sockets, 7 HTTP connections, 165 to the database; ..., then each instance. Without instances, as the Kubernetes probes call it, the record covers only the server that answered.

The overview shows them in Instances and isolates: a row per instance, with its isolates a click away, and a bar for each one's share of the requests of the last hour, so an instance or isolate that takes most of the load stands out.

Instances and isolates​

Browse > Platform > Health > Metrics shows each server instance in Instances and isolates: a card an instance, with a row for each of its isolates. Each measure is a bar against its limit, green while far from it, orange close to it, red at it:

  • Requests: served in the last hour and the last 5 minutes, and in flight; the bar is the share of all.
  • HTTP: connections open now, and for an isolate, active and idle.
  • DB pools: connections in use (dark) and open (light), of the most the pools may open (dbMax). Orange at 80% in use, red when a request waits for a connection. The Database line compares every connection to the database with its max_connections, and shows when the pools together may open more than that.
  • Sockets: open sockets, with a mark at the threshold of each enabled server.sockets alert, such as socket-growth and too-many-sockets; red at 80% of the open files limit.
  • Disk: the disk of the data folder of the instance; orange at 75% used or under 5 GB free, red at 90% or under 2 GB.

The alerts that hold for an instance show on its card; a click opens one in the context panel. All, Requests, HTTP, Database, Sockets and Disk pick what the cards show and sort them by it. Instances and isolates, table holds the same numbers, a row per instance and per isolate, each value colored good or bad, and the worst of the row in state. The overview's Requests, DB connections, HTTP connections and Open sockets, and the Metrics Servers and Disk free cards, open them with that measure picked (/health/metrics?focus=db).