Monitoring

PuppyGraph makes it easy to keep an eye on your system's health and performance. With built-in health check endpoints and detailed metrics collection, you can quickly monitor how your PuppyGraph instances are running and catch any issues early.

Health Check

PuppyGraph has two kinds of health checks. A liveness check tells you that a container is running. A readiness check tells you that PuppyGraph can serve Gremlin and openCypher queries. Use the readiness check before sending queries, and to decide when a deployment or upgrade is done.

Version requirement

/readyz and the image HEALTHCHECK are available in PuppyGraph 1.15.0 and later.

Endpoints

None of these endpoints requires a login, even with SSO, RBAC or authentication for Gremlin and openCypher clients turned on.

Endpoint	Served by	What it reports
`:8081/readyz`	The PuppyGraph container, or the controlplane in a cluster	Whether the whole deployment can serve queries now. Returns `200` when ready, `503` when not.
`:8081/healthz`	The PuppyGraph container, or the controlplane in a cluster	Liveness only. Returns `200` as soon as the Web UI and REST API are up, long before queries work.
`:8083/healthz`	Each leader and compute container, and the single PuppyGraph container	Whether this container is ready to take part in serving queries. Returns `200` with `YES`, or `500` with `NO`.

If you set BASE_PATH, add it before the path, for example :8081/puppygraph/readyz.

Readiness response

/readyz returns a JSON body:

{"ready": true, "degraded": false, "reasons": []}

ready is true when at least one leader and at least one compute container are ready, or, in a single-container deployment, when the container can serve queries. The HTTP status is 200. Otherwise ready is false and the status is 503.
degraded is true when PuppyGraph is ready but some leader or compute containers are not, so the cluster runs with less capacity or redundancy than configured.
reasons explains a 503, or a degraded answer. It holds stable codes only, never container names. Signed-in users can see which container each reason belongs to in GET /status, which has the same ready, degraded and ready_reasons fields plus not_ready_reasons for each container.

A fresh instance with no graph schema is ready, so it can accept its first schema. While a schema loads, for example for a few seconds after the first upload or after a restart, /readyz returns 503 with SCHEMA_SYNC_PENDING or SCHEMA_NOT_LOADED. Probes with a few retries ride this out.

Wait for the current schema

Add ?schema=current to wait until a graph schema is uploaded and every leader that serves queries has applied its latest version. Use it after you upload a schema, and after an upgrade or restart. A single container, or a cluster whose only leader cannot load the stored schema, stays at 503.

until curl -sf -o /dev/null "http://localhost:8081/readyz?schema=current"; do
  sleep 3
done

In a cluster with several leaders, /readyz?schema=current returns 200 once every leader that serves queries has applied the current schema. A ready leader on an older schema version keeps it at 503 with SCHEMA_NOT_CURRENT. A leader that is not ready, for example because it is still restarting or cannot load the schema, does not block the answer but makes it degraded. To confirm that every leader and compute container has recovered, for example at the end of an upgrade, also wait for "degraded":false:

until curl -sf "http://localhost:8081/readyz?schema=current" | grep -q '"degraded":false'; do
  sleep 3
done

On an instance with no schema, /readyz?schema=current stays at 503 with NO_SCHEMA, so use plain /readyz before the first schema upload. Any other value for schema returns 400.

Which endpoint to use

Deployment	Readiness	Liveness
Single container (Docker, Docker Compose)	`:8081/readyz`	`:8081/healthz`
Cluster controlplane	`:8081/healthz`	`:8081/healthz`
Cluster leader or compute container	`:8083/healthz`
Helm chart, `single-node` mode	`:8083/healthz`	`:8081/healthz`
Scripts and clients waiting for a cluster	`:8081/readyz` on the controlplane

Don't use /readyz as the controlplane's own readiness or health check in a cluster. Leader and compute containers register through the controlplane, so a controlplane that waits for the cluster to be ready never becomes ready. The Helm chart already sets these probes.

Upgrading from 1.14.0 or earlier

Existing health checks and wait scripts keep working. :8081/healthz still reports liveness only, and GET /status keeps all its fields and only gains new ones. Two things change without any action on your part:

:8083/healthz on leader and compute containers is stricter. A compute container answers YES only once it has finished starting and can execute its share of query work. A leader answers YES only once it can serve queries and, if a graph schema has been uploaded, has a graph schema loaded. Pods probed on :8083, as in the Helm chart, become Ready later at startup. They also leave their Service while the container cannot serve queries: briefly while a schema loads, and for as long as a leader cannot load its stored schema. In the chart's single-node mode the Web UI is behind the same Service, so to investigate such a pod, forward the port of the pod itself: kubectl -n $NAMESPACE port-forward pod/$RELEASE_NAME-single-node-0 8081:8081.
The image has a built-in HEALTHCHECK, described below. A healthcheck in your Compose file replaces it. Tools that restart unhealthy containers, such as Docker Swarm, now restart a single container that stays not ready, for example one that cannot load its stored schema.

No action is needed unless one of these applies:

Your own scripts or monitors treat NO from :8083/healthz as down. Expect NO for longer at startup and briefly while a schema loads, and allow for it in their retries.
You use :8081/healthz to decide that PuppyGraph can serve queries. Switch to /readyz, because :8081/healthz reports liveness only.
You don't want the built-in HEALTHCHECK, for example because your orchestrator runs its own check. Set healthcheck: disable: true in Compose or pass --no-healthcheck to docker run.

Usage

Wait until PuppyGraph is ready

Poll /readyz before you send queries or upload a schema from a script:

until curl -sf -o /dev/null http://localhost:8081/readyz; do
  sleep 3
done
echo "PuppyGraph is ready."

Docker container health check

The PuppyGraph image has a built-in HEALTHCHECK. On a single container it checks :8081/readyz. On cluster containers it checks :8081/healthz for the controlplane and :8083/healthz for leader and compute containers. docker ps shows the container as healthy once the check passes.

In Docker Compose, you can make another service wait for PuppyGraph without adding a healthcheck of your own:

services:
  puppygraph:
    image: puppygraph/puppygraph:latest
    # ...
  my-app:
    image: my-app:latest
    depends_on:
      puppygraph:
        condition: service_healthy

docker compose up -d --wait also returns only after PuppyGraph is healthy.

Kubernetes readiness probe (cluster deployments)

In a cluster, probe each leader and compute pod on :8083/healthz, and the controlplane pod on :8081/healthz. A pod's readiness should describe that pod, so that Kubernetes stops routing queries to a leader that cannot serve them. /readyz describes the whole cluster: it stays 200 while any leader is ready, so a broken leader would stay in the Service. The Helm chart configures these probes for you. For your own manifests, use this on leader and compute pods:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8083
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 5

For more information about cluster deployment, see Cluster Deployment.

Metrics

PuppyGraph exposes metrics in Prometheus / OpenMetrics format on two endpoints:

Endpoint	Served by	Metrics	Credentials
`:8083/metrics`	The node agent. In a single-container deployment (Docker, or the Helm chart's `single-node` mode), the PuppyGraph container. In a cluster, each leader and compute container.	Node metrics: memory, JVM, component health, and optionally the Gremlin server	The admin credentials, `PUPPYGRAPH_USERNAME` and `PUPPYGRAPH_PASSWORD`, unless `METRICS_AUTH_ENABLED=false`
`:8081/metrics`	The controlplane. In a single-container deployment (Docker, or the Helm chart's `single-node` mode), the PuppyGraph container. In a cluster, the controlplane container.	Controlplane metrics: local-table loads and Periodic Load	A user or service account with `CLUSTER:read`

Both endpoints use HTTP basic authentication. If you set BASE_PATH, add it before the path, for example :8083/puppygraph/metrics.

Configuration

Set these on every PuppyGraph container: the single container, or the controlplane, leaders, and compute nodes of a cluster.

Environment Variable	Default Value	Description
`METRICS_ENABLED`	`false`	Enables both metrics endpoints. While it is `false`, `:8083/metrics` returns `403`, and `:8081/metrics` returns `404` to a signed-in caller.
`METRICS_AUTH_ENABLED`	`true`	Requires the admin credentials on `:8083/metrics`. Set it to `false` to serve node metrics without credentials. It has no effect on `:8081/metrics`, which always requires credentials.
`GREMLINSERVER_METRICS_ENABLED`	`false`	Adds the `puppy_gremlin_*` metrics of the Gremlin server to `:8083/metrics`.
`BOLTSERVER_METRICS_ENABLED`	`false`	Adds the `puppy_bolt_*` metrics of the Bolt (openCypher) server to `:8083/metrics`.

Node metrics

Scrape :8083/metrics on each leader and compute container. In a single-container deployment, one endpoint covers both roles. The Gremlin and Bolt server metrics (puppy_gremlin_*, puppy_bolt_*) come only from leader containers, or the single container, so sum them across leaders for cluster totals.

With METRICS_AUTH_ENABLED=true, the endpoint accepts only the admin username and password set with PUPPYGRAPH_USERNAME and PUPPYGRAPH_PASSWORD (default puppygraph and puppygraph123). Other users and service accounts get 401.

curl -u "puppygraph:puppygraph123" http://localhost:8083/metrics

The table lists common node metrics. Others appear depending on the features you enable and the data sources you use, for example puppy_compute_jdbc_pool_* for each JDBC data source, and puppy_leader_iceberg_metadata_cache_max_age_seconds and puppy_leader_iceberg_metadata_last_success_age_seconds for each Iceberg catalog.

Metric Name	Description
`puppygraph_node_alive`	Whether each node of the cluster is alive.
`puppygraph_component_running`	Whether each process of a node (`dataaccess`, `bolt`, `gremlinserver`, `nodeagent`) is running.
`puppygraph_component_health`	Whether each process of a node is healthy.
`puppygraph_client_status`	Whether the `gotty`, `bolt`, and `notebook` clients are alive.
`puppygraph_gremlin_server_status`	Whether the Gremlin server is alive.
`puppy_compute_memory_usage_bytes`	Current memory consumption of the compute node.
`puppy_compute_memory_limit_bytes`	Configured memory limit of the compute node.
`puppy_compute_query_memory_usage_bytes`	Memory used by queries on the compute node.
`puppy_compute_data_cache_memory_usage_bytes`	Memory used by the compute node's data cache.
`puppy_compute_query_cache_memory_usage_bytes`	Memory used by the compute node's query cache.
`puppy_compute_local_load_memory_usage_bytes`	Memory used by local-table loads on the compute node.
`puppy_compute_jvm_heap_size_bytes`, `puppy_compute_jvm_non_heap_size_bytes`	Compute JVM heap and non-heap size, by type.
`puppy_leader_jvm_heap_size_bytes`, `puppy_leader_jvm_non_heap_size_bytes`	Leader JVM heap and non-heap size, by type.
`puppy_leader_jvm_young_gc`, `puppy_leader_jvm_old_gc`	Leader JVM garbage collection statistics, by type.
`puppy_leader_iceberg_metadata_refresh_total`	Completed Iceberg table metadata refreshes, by outcome.
`puppy_gremlin_*`	Gremlin server metrics, such as `puppy_gremlin_op_eval_count` (Gremlin queries; Cypher queries are not counted), `puppy_gremlin_op_eval_mean` and `puppy_gremlin_errors_count`. Values reset when the Gremlin server restarts. Only with `GREMLINSERVER_METRICS_ENABLED=true`.
`puppy_bolt_*`	Bolt server metrics, such as `puppy_bolt_queries_total` (queries received over the Bolt protocol, by `result` and `language`) and `puppy_bolt_connections_active`. Queries from the Web UI or the REST API are not counted. Only with `BOLTSERVER_METRICS_ENABLED=true`.

Controlplane metrics

Version requirement

Controlplane local-table load metrics are available in PuppyGraph 1.16.0 and later.

The controlplane reports on local-table loads and the Periodic Load scheduler at :8081/metrics. The endpoint uses the same sign-in as the REST API, not METRICS_AUTH_ENABLED:

Request	Response
No credentials, or wrong credentials	`401`
A user or service account without `CLUSTER:read`	`403`, `permission denied: requires CLUSTER:read`
A user or service account with `CLUSTER:read`, and `METRICS_ENABLED=false`	`404`, `metrics are not enabled; set METRICS_ENABLED=true`
A user or service account with `CLUSTER:read`, and `METRICS_ENABLED=true`	`200` with the metrics

Of the built-in roles, Admin, UserAdmin, and GraphAdmin have CLUSTER:read. None of them is limited to reading metrics, so treat a scraper's credentials like those of an administrator. For a scraper, create a service account with one of these roles and use its name and secret as the basic authentication username and password:

curl -u "<service-account-name>:<secret>" http://localhost:8081/metrics

Every series carries a hostName label with the controlplane's host name. No series carries a per-table label.

Metric Name	Type	Labels	Description
`puppy_controlplane_local_table_load_tasks_total`	counter	`hostName`, `trigger`, `result`	Load task attempts, by what started the load and how the attempt ended.
`puppy_controlplane_local_table_load_task_duration_seconds`	histogram	`hostName`, `trigger`	Duration of successful load tasks.
`puppy_controlplane_local_table_load_active_task_max_age_seconds`	gauge	`hostName`	Seconds since the oldest unfinished (pending, running, or retrying) load task was submitted. `0` when there is none.
`puppy_controlplane_local_table_load_periodic_submissions_total`	counter	`hostName`, `result`	Periodic Load submissions, by result.
`puppy_controlplane_local_table_load_periodic_poll_last_timestamp_seconds`	gauge	`hostName`	Unix time of the Periodic Load scheduler's last successful check. `0` before the first.

trigger is what started the load:

Value	Load started by
`on_demand`	An explicit request to load now: `POST /ui-api/loadLocalTable` or `loadLocalTables`, the Web UI's Import from Source, PuppyGraph AI, or a Gremlin load action.
`periodic`	The Periodic Load scheduler, which reloads a local table on a fixed interval.
`schema_upload`	A schema upload with `postUploadBehavior=load` or `switch`.
`table_create`	Creating a local table for a node or edge, for example with Enable Local Replication.
`unknown`	A load recorded without a trigger, such as one submitted before an upgrade to a version with these metrics.

result of puppy_controlplane_local_table_load_tasks_total:

Value	Meaning
`success`	The task succeeded.
`failure`	The task failed, and no retry follows.
`retried`	The task failed, and a retry was submitted. The retry is counted again when it finishes.
`cancelled`	The task was cancelled.

result of puppy_controlplane_local_table_load_periodic_submissions_total:

Value	Meaning
`accepted`	The load was submitted. Its tasks are counted in `puppy_controlplane_local_table_load_tasks_total` with `trigger="periodic"`.
`rejected`	The load could not be submitted, for example because no leader was available. The scheduler tries again later.
`superseded`	The schema changed before the load was submitted, for example because Periodic Load was turned off, so it was not submitted.

How to read them:

puppy_controlplane_local_table_load_tasks_total counts load tasks, not requests. A load of a partitioned table or of a time range runs one task per partition, and each retry is another task. A retry keeps the trigger of the load it retries, so a load that succeeds on its second attempt counts retried once and success once under the same trigger.
failure counts only failures that are final. The load status FAILED shown in the Web UI also covers attempts that were retried.
puppy_controlplane_local_table_load_task_duration_seconds includes successful tasks only, including any time a task waited in a queue after it was submitted.
puppy_controlplane_local_table_load_active_task_max_age_seconds keeps growing while a task is stuck. Such a task also stops Periodic Load from loading its table again. Find the table with getLocalTableLoadStatus.
puppy_controlplane_local_table_load_periodic_poll_last_timestamp_seconds shows that the scheduler is running, not that any table was due.
The counters belong to the controlplane process. They start at zero when a controlplane starts, or takes over from another one, and rate() and increase() treat that as a counter reset.

Example queries:

# Final load failures per second, by trigger
sum by (trigger) (rate(puppy_controlplane_local_table_load_tasks_total{result="failure"}[5m]))

# Successful periodic load tasks per second
rate(puppy_controlplane_local_table_load_tasks_total{trigger="periodic", result="success"}[5m])

# Rejected periodic submissions in the last hour
increase(puppy_controlplane_local_table_load_periodic_submissions_total{result="rejected"}[1h])

# A load task that has not finished for more than 13 hours (the default task timeout is 12 hours)
puppy_controlplane_local_table_load_active_task_max_age_seconds > 13 * 3600

# Seconds since the scheduler last checked; alert when it stays above a few minutes
time() - puppy_controlplane_local_table_load_periodic_poll_last_timestamp_seconds

Prometheus Integration

Use the following Prometheus scrape configuration as a template. It has one job for node metrics and one for controlplane metrics. Replace the placeholder values to match your environment.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: puppygraph-node
    metrics_path: /metrics
    static_configs:
      - targets: ["<PUPPYGRAPH_HOST>:8083"]
    basic_auth:
      username: <PUPPYGRAPH_USERNAME>
      password: <PUPPYGRAPH_PASSWORD>

  - job_name: puppygraph-controlplane
    metrics_path: /metrics
    static_configs:
      - targets: ["<PUPPYGRAPH_HOST>:8081"]
    basic_auth:
      username: <SERVICE_ACCOUNT_NAME>
      password: <SERVICE_ACCOUNT_SECRET>

For puppygraph-node, list :8083 of every leader and compute container, or of the single PuppyGraph container. Use the admin credentials set with PUPPYGRAPH_USERNAME and PUPPYGRAPH_PASSWORD, or remove basic_auth if METRICS_AUTH_ENABLED=false.
For puppygraph-controlplane, use the controlplane, or the single PuppyGraph container, and a service account with CLUSTER:read. Leave this job out before PuppyGraph 1.16.0.
Adjust the scrape intervals to match your monitoring requirements.

For a complete walkthrough, see Monitoring PuppyGraph with Prometheus.

Datadog Integration

Here's an example of how to integrate PuppyGraph's metrics endpoints with Datadog for monitoring.

Run Datadog agent

Follow Start the Datadog Agent with Docker to start the Datadog agent.

Pass the credentials to the Datadog agent container as environment variables: the admin credentials for node metrics, and a service account with CLUSTER:read for controlplane metrics:

docker run -d --name dd-agent \
  -e PUPPYGRAPH_USERNAME="<YOUR_PUPPYGRAPH_USERNAME>" \
  -e PUPPYGRAPH_PASSWORD="<YOUR_PUPPYGRAPH_PASSWORD>" \
  -e PUPPYGRAPH_SA_NAME="<SERVICE_ACCOUNT_NAME>" \
  -e PUPPYGRAPH_SA_SECRET="<SERVICE_ACCOUNT_SECRET>" \
  -e DD_API_KEY="<DATADOG_API_KEY>" \
  -e DD_SITE="<YOUR_DATADOG_SITE>" \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  gcr.io/datadoghq/agent:latest

Run PuppyGraph with Datadog labels

Add labels to a PuppyGraph container so that Datadog discovers and scrapes its metrics. For example, if you are running PuppyGraph with Docker, add the following arguments to the docker run command. They configure the openmetrics check twice: the first instance collects node metrics from :8083, and the second collects controlplane metrics from :8081:

-l com.datadoghq.ad.check_names='["openmetrics","openmetrics"]' \
-l com.datadoghq.ad.init_configs='[{},{}]' \
-l com.datadoghq.ad.instances='[{"openmetrics_endpoint":"http://%%host%%:8083/metrics","username":"%%env_PUPPYGRAPH_USERNAME%%","password":"%%env_PUPPYGRAPH_PASSWORD%%","namespace":"puppy","metrics":[".*"]},{"openmetrics_endpoint":"http://%%host%%:8081/metrics","username":"%%env_PUPPYGRAPH_SA_NAME%%","password":"%%env_PUPPYGRAPH_SA_SECRET%%","namespace":"puppy","metrics":[".*"]}]' \

If METRICS_AUTH_ENABLED=false, remove username and password from the :8083 instance. Keep them on the :8081 instance, which always requires credentials.
Before PuppyGraph 1.16.0, configure the check once: check_names '["openmetrics"]', init_configs '[{}]', and only the :8083 instance.