Cluster Deployment

Running PuppyGraph 0.x?

This page documents the PuppyGraph 1.0 cluster architecture (controlplane / leader / compute, Helm chart, env vars, healthz ports). For the 0.x version, see the archived page.

This article covers deploying PuppyGraph in cluster mode for two outcomes: horizontal scalability (each query is parallelized across compute nodes, so adding nodes speeds up individual queries and raises total throughput) and high availability (the system stays operational when nodes fail).

The recommended deployment path is the PuppyGraph Helm chart. This page covers the cluster architecture, the configuration knobs the chart exposes (env vars, services, resources), day-2 operations like scaling and upgrades, and a manual Kubernetes manifest for environments without Helm.

Cluster architecture

A PuppyGraph cluster has three node types: a controlplane node, leader nodes, and compute nodes.

For production deployments, plan for one (1) controlplane node, at least three (3) leader nodes, and at least three (3) compute nodes.

Controlplane node

The controlplane manages the cluster registry, serves the Web UI, and exposes the gRPC endpoint that leader and compute nodes use for discovery and registration. It is the central coordination point for the cluster.

Leader nodes

Leader nodes manage query execution and coordinate the compute nodes. They run leader election among themselves, monitor cluster health, and manage cluster state. Each leader registers with the controlplane on startup.

Compute nodes

Compute nodes execute queries and process data. Each compute node registers with the controlplane on startup and communicates with the leader nodes during query execution.

Deploying with the Helm chart

The PuppyGraph Helm chart provisions all three node types, their services, secrets, persistent volumes, and readiness probes. For the install procedure (adding the repo, generating values, deploying, accessing the Web UI), see the Helm Chart Installation Guide.

The rest of this page documents what the chart deploys and the configuration available, so you can tune values.yaml and run scale or upgrade operations confidently.

Controlplane configuration

Environment variables

Variable	Helm values key	Description
`CONTROLPLANE`	(set by chart)	Identifies the node as a controlplane. Set to `true`.
`CONTROLPLANE_REGISTRY_STORAGEPATH`	(set by chart)	Path for the controlplane registry database. Default: `/data/registry.db`.
`CLUSTER_ID`	`env.leader.CLUSTER_ID`	Cluster ID. Must match all leader nodes in the cluster.
`AUTHENTICATION_JWT_SECRETKEY`	`secrets.AUTHENTICATION_JWT_SECRETKEY`	Secret key for JWT authentication between the controlplane and other nodes. Must be 16, 24, or 32 bytes for AES-128, AES-192, or AES-256.
`PUPPYGRAPH_USERNAME`	`secrets.PUPPYGRAPH_USERNAME`	Username for PuppyGraph login.
`PUPPYGRAPH_PASSWORD`	`secrets.PUPPYGRAPH_PASSWORD`	Password for PuppyGraph login.

JWT secret key in production

AUTHENTICATION_JWT_SECRETKEY must be a strong, unique value in production. A 32-byte (AES-256) key is recommended. The Helm chart generates one automatically if secrets.AUTHENTICATION_JWT_SECRETKEY is left blank.

Service

The controlplane service exposes the Web UI (port 8081) and the gRPC endpoint (port 8082) used by leader and compute nodes for registration and discovery. The Helm chart deploys this as <release>-controlplane-service. To expose the Web UI externally, set service.controlplane.type=LoadBalancer (default ClusterIP).

Leader configuration

Basic environment variables

Variable	Helm values key	Description
`LEADER`	(set by chart)	Identifies the node as a leader.
`CONTROLPLANE_ADDR`	(set by chart)	Address of the controlplane gRPC endpoint, for example `<release>-controlplane-service:8082`.
`PRIORITY_IP_CIDR`	`env.common.PRIORITY_IP_CIDR`	IP range for priority IPs for leader nodes. Should match the Kubernetes cluster CIDR. Verify with `kubectl get nodes -o wide`.
`STORAGE_PATH_ROOT`	(set by chart)	Root path for persistent storage.

Cluster environment variables

Variable	Helm values key	Description
`POD_NAME`	(set by chart)	Name of the current leader node. Used to detect when to start a new cluster or wait for others.
`CLUSTER_ID`	`env.leader.CLUSTER_ID`	Cluster ID. All leader nodes in the same cluster must use the same ID.
`LEADER_SERVICE_ADDRESS`	(set by chart)	Address or name of the leader service.
`AUTHENTICATION_JWT_SECRETKEY`	`secrets.AUTHENTICATION_JWT_SECRETKEY`	Secret key for JWT authentication. Must match the value set on the controlplane node.
`DATAACCESS_DATA_REPLICATIONNUM`	`env.leader.DATAACCESS_DATA_REPLICATIONNUM`	Replication factor for graph metadata and the local cache. Default `3`. Set to `1` for single-compute (non-HA) deployments.

Startup timeouts

These environment variables tune how long a leader node waits during startup:

Variable	Description
`CLUSTER_STARTUPTIMEOUT`	Max time the startup process can take before being considered failed. Default 24h.
`CLUSTER_POTENTIALLEADERTIMEOUT`	Max time a leader can take to discover other potential leaders before starting a new cluster. Default 30s.
`CLUSTER_OTHERLEADERTIMEOUT`	Max time a leader waits for the leader election to complete before failing. Should be larger than `CLUSTER_POTENTIALLEADERTIMEOUT`.

Services

The chart deploys two services that select leader pods:

A headless leader service (<release>-leader-service) that exposes the REST port (8081) and the Gremlin port (8182). Used for inter-leader communication and for direct Gremlin access.
A query service (<release>-query-service) that exposes the openCypher Bolt port (7687). Set service.query.type=LoadBalancer to expose externally.

Leader startup process

When a leader node starts, it connects to the controlplane via CONTROLPLANE_ADDR to register and discover the cluster state. If a cluster is already formed, the leader joins it.

If the leader cannot find any other leaders within CLUSTER_POTENTIALLEADERTIMEOUT (30 seconds by default), it will only start a new cluster if it is the first leader to start. Eligibility is determined by POD_NAME: only the leader whose POD_NAME ends with -0 will start a new cluster. This avoids multiple leaders forming separate clusters at the same time (split-brain).

The whole leader startup has a timeout of CLUSTER_STARTUPTIMEOUT (24 hours by default). If the startup doesn't finish in that window, the leader is considered failed and must be restarted manually.

Readiness probe

The chart configures a readiness probe that checks /healthz on port 8083. Override under leader.readinessProbe in values.yaml if you need different timing.

Compute configuration

Environment variables

Variable	Helm values key	Description
`COMPUTE_NODE`	(set by chart)	Identifies the node as a compute node.
`CONTROLPLANE_ADDR`	(set by chart)	Address of the controlplane gRPC endpoint.
`PRIORITY_IP_CIDR`	`env.common.PRIORITY_IP_CIDR`	IP range for priority IPs. Should match the Kubernetes cluster CIDR.
`STORAGE_PATH_ROOT`	(set by chart)	Root path for persistent storage.
`SCRATCH_PATH_ROOT`	(set by chart)	Root path for temporary storage.

Service

The compute service (<release>-compute-service) is headless and exposes the REST port (8081) for inter-cluster communication.

Compute startup process

When a compute node starts, it connects to the controlplane via CONTROLPLANE_ADDR to register and discover the cluster state. Once registered, leader nodes coordinate query execution against it.

Cluster management

High availability

PuppyGraph high availability has two parts: leader HA and compute HA.

Leader HA

Leader nodes consist of one leader and multiple followers. Followers continuously synchronize metadata with the leader and monitor each other's health. When the leader becomes unavailable, the followers elect a new one, keeping cluster management available.

Leader HA uses a 2n+1 deployment formula, where n is the number of failures the cluster can tolerate. Three leaders tolerate one failure, five tolerate two, and so on.

Enabling HA

Set leader.replicas: 3 (or more) in values.yaml to enable HA. Three is typically sufficient. For production, plan for at least three leader nodes.

The leader nodes store cluster metadata in STORAGE_PATH_ROOT. A restarted leader will re-register itself with the cluster from that metadata.

Compute HA

Compute HA is implemented through multi-replica data storage, configured by DATAACCESS_DATA_REPLICATIONNUM (chart default 3). Each data block is stored as that many replicas on different compute nodes.

Leader nodes continuously monitor compute nodes. When a compute node becomes unavailable, leaders mark it unhealthy and serve queries from replicas on other nodes. The cluster automatically restores the configured replica count by replicating data to remaining nodes.

When a compute node restarts, it re-registers with the leaders and reports its existing data state, and the leaders update accordingly.

Replication vs. node count

The replication factor (DATAACCESS_DATA_REPLICATIONNUM) is independent of the number of HA nodes. You can have, for example, five compute nodes with a replication factor of three.

Persistent volumes and local cache

Leader nodes use persistent volumes to store cluster metadata. If a leader pod is deleted and recreated, it retains its metadata as long as the volume isn't deleted.

Compute nodes use persistent volumes for the local cache. When you delete a compute node from the Web UI, its cache data is automatically migrated to other compute nodes before the node is removed, preserving the configured replica count. If a compute node fails unexpectedly (not through Web UI deletion), the cluster detects the replica mismatch and redistributes data automatically.

Stop and restart

To stop the cluster, uninstall the Helm release:

helm uninstall <release-name> -n <namespace>

Persistent volumes

Uninstalling does not delete the persistent volumes. Metadata and local cache remain available for the next restart. Delete them manually if you no longer need them:

kubectl get pvc -n <namespace> -o name | grep -E "data-<release-name>" | xargs -r kubectl delete -n <namespace>

To restart, reinstall the chart with the same values:

helm install <release-name> puppygraph/puppygraph -n <namespace> -f values.yaml

To restart a single leader or compute node (for example, if it's stuck), delete the pod and let Kubernetes recreate it:

kubectl delete pod <pod-name> -n <namespace>

To start the recreated pod from a clean volume, delete the PVC first:

kubectl delete pvc <pvc-name> -n <namespace>
kubectl delete pod <pod-name> -n <namespace>

The PV is removed once the pod releases the volume.

Scaling

Scaling has two directions: scaling up (adding nodes) and scaling down (removing nodes).

Scaling up

Update the corresponding replicas in values.yaml (leader.replicas or compute.replicas) and apply with helm upgrade. The chart re-renders the StatefulSets with the new replica counts.

For compute scaling, the persistent local cache on existing nodes is unaffected. New compute nodes automatically rebalance data on startup.

Single leader to HA

To scale from a single leader to HA, set leader.replicas: 3 (or more) and apply. New leader nodes join the existing cluster with no downtime.

HA scaling

In HA deployments, scaling up or down has no downtime; services remain available throughout.

After applying changes, verify pod status:

kubectl get pods -n <namespace>

and confirm in the Web UI.

Scaling down compute nodes

Delete the compute nodes with the highest indices from the Web UI cluster management page (for example, delete <release>-compute-3 and <release>-compute-4 when scaling from 5 to 3 replicas, considering 0-based indexing).
Wait for deletion and local cache migration to finish. Refresh the Web UI to verify. This step takes a while because cache data migrates to remaining nodes.
Update compute.replicas in values.yaml to the target value and run helm upgrade.
Manually delete the persistent volumes associated with the removed compute nodes, so future scale-up starts from a clean PV.

Pod status after UI deletion

After deleting a node in the Web UI, the corresponding pod stays in running status. This is expected. The pod is removed when you update the StatefulSet replicas and apply.

Scaling down leader nodes

Same as compute nodes, with one exception around the elected leader:

Delete follower leader nodes with the highest indices from the Web UI.
If the elected leader is among the nodes to be deleted, delete the elected leader pod with kubectl delete pod <release>-leader-<n> -n <namespace>. This triggers a new leader election among the remaining followers. Kubernetes recreates the deleted pod, which rejoins as a follower. You can then delete that follower from the Web UI.
Wait for the Web UI to show all targeted nodes deleted.
Update leader.replicas in values.yaml to the target value and run helm upgrade.
Manually delete the persistent volumes associated with the removed leader nodes.

Version upgrade

Pre-upgrade support contact

Before any upgrade, contact PuppyGraph support to verify version compatibility and any specific migration requirements.

Image upgrade

Upgrading PuppyGraph involves updating container images for the controlplane, leader, and compute pods. Upgrade them in this order: controlplane first, then compute, then leaders.

Set the image tag in values.yaml and run helm upgrade:

Set controlplane.image.tag to the new version, run helm upgrade, and wait for the controlplane pod to become ready.
Set compute.image.tag to the new version and run helm upgrade.
Set leader.image.tag to the new version and run helm upgrade.

compute.image.tag and leader.image.tag override the global image.tag. If neither is set, both default to image.tag.

After each step, verify the pod status and confirm in the Web UI.

Upgrade behavior

During upgrade or downgrade, affected pods are recreated through a rolling update by default. The local cache is not affected. PuppyGraph itself isn't a database, so no data migration is required for PuppyGraph data or the local cache.

Whether the cluster can serve queries during upgrade depends on the specific version pair, but services typically remain available.

Helm chart upgrade

If you're upgrading the Helm chart itself (not just the PuppyGraph image) on already-running clusters, review the release notes, check the chart changes, and contact PuppyGraph support to verify compatibility.

Chart vs. image upgrade

Upgrading the chart is different from upgrading the PuppyGraph image. Image upgrades happen through values.yaml. Chart upgrades may not change the PuppyGraph version or trigger pod rolling updates. The chart and image evolve together, so as long as both are aligned and up to date, they remain compatible.

Troubleshooting

Pods stuck in CrashLoopBackOff

Download logs from the Web UI and check for startup errors or missing configuration. Verify the environment variables are set correctly. Restart the cluster after fixing the issues.

Persistent volumes not attaching

Verify the storage class and PVC binding. If you're on a managed cloud service, check that the storage provisioner is configured correctly.

Cluster not forming properly

Wait for all pods to enter the Running status. If leader or compute pods remain not ready for an extended period, download logs from the Web UI for details. You can also follow Stop and restart to restart the affected pods.

High memory or CPU usage

Tune resource limits in values.yaml (leader.resources and compute.resources) or investigate query patterns. Adding more compute nodes will distribute the load.

Service not reachable

Verify ingress and service configuration. If the issue appears during a cluster update, refresh the page. During rolling updates, connections to the previous leader pod may be interrupted as it restarts; refreshing reconnects to another leader.

Manual Kubernetes deployment

For environments where Helm isn't available, the manifest below produces an equivalent deployment to the Helm chart with default values. Most users should prefer the Helm chart since it manages templating, secrets, and upgrades for you.

The manifest depends on a secret-puppygraph Secret holding PUPPYGRAPH_USERNAME, PUPPYGRAPH_PASSWORD, and AUTHENTICATION_JWT_SECRETKEY, plus an optional configmap-puppygraph ConfigMap for additional configuration files. The example below mounts the ConfigMap at /etc/config/puppygraph/ on the leader StatefulSet only; mirror the volume / volumeMount onto the compute StatefulSet if you also need files there.

Example PuppyGraph cluster manifest

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-puppygraph
data: {}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: puppygraph-controlplane
  labels:
    app: puppygraph-controlplane
spec:
  serviceName: puppygraph-controlplane-service
  replicas: 1
  selector:
    matchLabels:
      app: puppygraph-controlplane
  template:
    metadata:
      labels:
        app: puppygraph-controlplane
    spec:
      containers:
        - name: puppygraph-controlplane
          image: puppygraph/puppygraph:latest
          imagePullPolicy: Always
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CONTROLPLANE
              value: "true"
            - name: CONTROLPLANE_REGISTRY_STORAGEPATH
              value: "/data/registry.db"
            - name: CLUSTER_ID
              value: "1000"
            - name: PUPPYGRAPH_USERNAME
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: PUPPYGRAPH_USERNAME
            - name: PUPPYGRAPH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: PUPPYGRAPH_PASSWORD
            - name: AUTHENTICATION_JWT_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: AUTHENTICATION_JWT_SECRETKEY
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
            limits:
              memory: "4Gi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: ebs-sc
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: puppygraph-controlplane-service
spec:
  type: LoadBalancer
  selector:
    app: puppygraph-controlplane
  ports:
    - name: web
      protocol: TCP
      port: 8081
      targetPort: 8081
    - name: grpc
      protocol: TCP
      port: 8082
      targetPort: 8082
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: puppygraph-leader
  labels:
    app: puppygraph-leader
spec:
  serviceName: puppygraph-leader-service
  podManagementPolicy: "Parallel"
  replicas: 3
  selector:
    matchLabels:
      app: puppygraph-leader
  template:
    metadata:
      labels:
        app: puppygraph-leader
    spec:
      containers:
        - name: puppygraph-leader
          image: puppygraph/puppygraph:latest
          imagePullPolicy: Always
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CLUSTER_ID
              value: "1000"
            - name: PUPPYGRAPH_USERNAME
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: PUPPYGRAPH_USERNAME
            - name: PUPPYGRAPH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: PUPPYGRAPH_PASSWORD
            - name: AUTHENTICATION_JWT_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: secret-puppygraph
                  key: AUTHENTICATION_JWT_SECRETKEY
            - name: LEADER
              value: "true"
            - name: CONTROLPLANE_ADDR
              value: "puppygraph-controlplane-service:8082"
            - name: LEADER_SERVICE_ADDRESS
              value: "puppygraph-leader-service"
            - name: DATAACCESS_DATA_REPLICATIONNUM
              value: "3"
            - name: PRIORITY_IP_CIDR
              value: "172.31.0.0/16"
            - name: STORAGE_PATH_ROOT
              value: "/data/storage"
            - name: SCRATCH_PATH_ROOT
              value: "/data/scratch"
            - name: CLUSTER_STARTUPTIMEOUT
              value: "10m"
          volumeMounts:
            - name: volume-config-puppygraph
              mountPath: /etc/config/puppygraph/
              readOnly: true
            - name: data
              mountPath: /data
          resources:
            requests:
              cpu: "8"
              memory: "32Gi"
            limits:
              memory: "32Gi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8083
            initialDelaySeconds: 60
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/home/ubuntu/prestop.sh"]
      terminationGracePeriodSeconds: 60
      volumes:
        - name: volume-config-puppygraph
          configMap:
            name: configmap-puppygraph
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: ebs-sc
        resources:
          requests:
            storage: 200Gi
---
apiVersion: v1
kind: Service
metadata:
  name: puppygraph-leader-service
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: puppygraph-leader
  ports:
    - name: rest
      protocol: TCP
      port: 8081
      targetPort: 8081
    - name: gremlin
      protocol: TCP
      port: 8182
      targetPort: 8182
---
apiVersion: v1
kind: Service
metadata:
  name: puppygraph-query-service
spec:
  type: LoadBalancer
  selector:
    app: puppygraph-leader
  ports:
    - name: bolt
      protocol: TCP
      port: 7687
      targetPort: 7687
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: puppygraph-compute
  labels:
    app: puppygraph-compute
spec:
  serviceName: puppygraph-compute-service
  podManagementPolicy: "Parallel"
  replicas: 3
  selector:
    matchLabels:
      app: puppygraph-compute
  template:
    metadata:
      labels:
        app: puppygraph-compute
    spec:
      containers:
        - name: puppygraph-compute
          image: puppygraph/puppygraph:latest
          imagePullPolicy: Always
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: COMPUTE_NODE
              value: "true"
            - name: CONTROLPLANE_ADDR
              value: "puppygraph-controlplane-service:8082"
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: COMPUTE_NODE_DOMAIN_NAME
              value: "$(POD_NAME).puppygraph-compute-service.$(NAMESPACE).svc.cluster.local"
            - name: PRIORITY_IP_CIDR
              value: "172.31.0.0/16"
            - name: STORAGE_PATH_ROOT
              value: "/data/storage"
            - name: SCRATCH_PATH_ROOT
              value: "/data/scratch"
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            requests:
              cpu: "16"
              memory: "64Gi"
            limits:
              memory: "64Gi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8083
            initialDelaySeconds: 5
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/home/ubuntu/prestop.sh"]
      terminationGracePeriodSeconds: 60
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: ebs-sc
        resources:
          requests:
            storage: 200Gi
---
apiVersion: v1
kind: Service
metadata:
  name: puppygraph-compute-service
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: puppygraph-compute
  ports:
    - name: rest
      protocol: TCP
      port: 8081
      targetPort: 8081