Querying Polaris Data as a Graph

Archived PuppyGraph 0.x documentation

This is the archived 0.x documentation. For new deployments, use puppygraph/puppygraph:latest (PuppyGraph 1.x) and the current docs. The 0.x schemas and UI described here don't apply to 1.x; see Migrating from v0.

Summary

In this tutorial, you will:

1. Start Apache Polaris, RustFS, Spark, and PuppyGraph locally with Docker Compose.
2. Create an Iceberg catalog named modern_catalog and load sample data into it.
3. Upload a graph schema JSON to PuppyGraph and query the data as a graph.

Prerequisites

• Docker
• Docker Compose

Verify that both commands are available:

docker version
docker compose version

Accessing the PuppyGraph Web UI requires a browser. If you prefer a CLI-only workflow, you can upload the graph schema with curl and connect to Gremlin on port 18182.

Note

This tutorial uses demo credentials and local container hostnames for a self-contained setup:

• Polaris bootstrap credential: root:s3cr3t
• RustFS access key: rustfs_root
• RustFS secret key: rustfspwd
• PuppyGraph login: puppygraph / puppygraph123

Deployment and Data Preparation

This tutorial uses four containers:

• polaris-rustfs for S3-compatible object storage
• polaris for the Iceberg REST catalog
• polaris-spark-iceberg to create the sample Iceberg tables
• polaris-puppygraph for the PuppyGraph server

Create docker-compose.yaml

Create a file named docker-compose.yaml with the following content:

docker-compose.yaml

services:
  polaris-rustfs:
    container_name: polaris-rustfs
    image: rustfs/rustfs:latest
    restart: always
    environment:
      RUSTFS_ACCESS_KEY: rustfs_root
      RUSTFS_SECRET_KEY: rustfspwd
      RUSTFS_CONSOLE_ENABLE: "true"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://127.0.0.1:9000/health"]
      interval: 2s
      timeout: 10s
      retries: 30
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - polaris-rustfs-data:/data

  polaris:
    container_name: polaris
    image: apache/polaris:latest
    restart: always
    depends_on:
      polaris-rustfs:
        condition: service_healthy
    environment:
      AWS_REGION: us-west-2
      AWS_ACCESS_KEY_ID: rustfs_root
      AWS_SECRET_ACCESS_KEY: rustfspwd
      POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
      polaris.realm-context.realms: POLARIS
      quarkus.otel.sdk.disabled: "true"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://127.0.0.1:8182/q/health"]
      interval: 2s
      timeout: 10s
      retries: 30
      start_period: 10s
    ports:
      - "8181:8181"
      - "8182:8182"

  polaris-spark-iceberg:
    container_name: polaris-spark-iceberg
    image: tabulario/spark-iceberg
    restart: always
    depends_on:
      polaris:
        condition: service_healthy
    entrypoint: ["/bin/bash", "-lc", "tail -f /dev/null"]

  polaris-puppygraph:
    image: puppygraph/puppygraph:stable
    pull_policy: always
    container_name: puppy
    restart: always
    networks:
      default:
    environment:
      - PUPPYGRAPH_USERNAME=puppygraph
      - PUPPYGRAPH_PASSWORD=puppygraph123
    ports:
      - "8081:8081"
      - "18182:8182"
      - "7687:7687"
    depends_on:
      - polaris-spark-iceberg

volumes:
  polaris-rustfs-data:

networks:
  default:
    name: polaris_net

Create setup-polaris.sh

Create a file named setup-polaris.sh with the following content. This script starts the containers, creates the RustFS bucket, provisions the Polaris catalog, grants permissions, and loads the sample Iceberg tables.

setup-polaris.sh

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
cd "$SCRIPT_DIR"

docker compose up -d

ROOT_TOKEN=$(
  docker run --rm --network polaris_net alpine/curl:8.17.0 sh -lc '
    apk add --no-cache jq >/dev/null
    curl -s --user root:s3cr3t \
      -H "Polaris-Realm: POLARIS" \
      -d grant_type=client_credentials \
      -d scope=PRINCIPAL_ROLE:ALL \
      http://polaris:8181/api/catalog/v1/oauth/tokens | jq -r .access_token
  '
)

docker compose exec -T polaris-rustfs sh -ec '
  curl -fsS --aws-sigv4 aws:amz:us-west-2:s3 --user rustfs_root:rustfspwd \
    --head http://localhost:9000/bucket457 >/dev/null 2>&1 ||
  curl -fsS --aws-sigv4 aws:amz:us-west-2:s3 --user rustfs_root:rustfspwd \
    -X PUT http://localhost:9000/bucket457
'

docker run --rm --network polaris_net -e ROOT_TOKEN="$ROOT_TOKEN" alpine/curl:8.17.0 sh -lc '
  apk add --no-cache jq >/dev/null

  if ! curl -s http://polaris:8181/api/management/v1/catalogs \
    -H "Authorization: Bearer ${ROOT_TOKEN}" \
    -H "Polaris-Realm: POLARIS" | jq -e ".catalogs[]? | select(.name == \"modern_catalog\")" >/dev/null; then
    curl -s -X POST http://polaris:8181/api/management/v1/catalogs \
      -H "Authorization: Bearer ${ROOT_TOKEN}" \
      -H "Polaris-Realm: POLARIS" \
      -H "Content-Type: application/json" \
      -d "{\"catalog\":{\"name\":\"modern_catalog\",\"type\":\"INTERNAL\",\"readOnly\":false,\"properties\":{\"default-base-location\":\"s3://bucket457/modern_catalog\"},\"storageConfigInfo\":{\"storageType\":\"S3\",\"allowedLocations\":[\"s3://bucket457/modern_catalog\",\"s3://bucket457\"],\"endpoint\":\"http://polaris-rustfs:9000\",\"endpointInternal\":\"http://polaris-rustfs:9000\",\"pathStyleAccess\":true}}}"
  fi

  curl -s -X PUT http://polaris:8181/api/management/v1/catalogs/modern_catalog/catalog-roles/catalog_admin/grants \
    -H "Authorization: Bearer ${ROOT_TOKEN}" \
    -H "Polaris-Realm: POLARIS" \
    -H "Content-Type: application/json" \
    -d "{\"type\":\"catalog\",\"privilege\":\"TABLE_WRITE_DATA\"}"

  curl -s -X PUT http://polaris:8181/api/management/v1/principal-roles/service_admin/catalog-roles/modern_catalog \
    -H "Authorization: Bearer ${ROOT_TOKEN}" \
    -H "Polaris-Realm: POLARIS" \
    -H "Content-Type: application/json" \
    -d "{\"name\":\"catalog_admin\"}"
'

docker compose exec -T -e ROOT_TOKEN="$ROOT_TOKEN" polaris-spark-iceberg bash -lc '
  cat >/tmp/prepare.sql
  /opt/spark/bin/spark-sql \
    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-aws-bundle:1.10.1 \
    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
    --conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.polaris.type=rest \
    --conf spark.sql.catalog.polaris.uri=http://polaris:8181/api/catalog \
    --conf spark.sql.catalog.polaris.oauth2-server-uri=http://polaris:8181/api/catalog/v1/oauth/tokens \
    --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \
    --conf spark.sql.catalog.polaris.client.region=us-west-2 \
    --conf spark.sql.catalog.polaris.token="${ROOT_TOKEN}" \
    --conf spark.sql.catalog.polaris.warehouse=modern_catalog \
    --conf spark.sql.defaultCatalog=polaris \
    -f /tmp/prepare.sql
' <<'SQL'
CREATE DATABASE IF NOT EXISTS modern;

CREATE TABLE modern.person (id string, name string, age int) USING iceberg;
INSERT INTO modern.person VALUES
    ('v1', 'marko', 29),
    ('v2', 'vadas', 27),
    ('v4', 'josh', 32),
    ('v6', 'peter', 35);

CREATE TABLE modern.software (id string, name string, lang string) USING iceberg;
INSERT INTO modern.software VALUES
    ('v3', 'lop', 'java'),
    ('v5', 'ripple', 'java');

CREATE TABLE modern.created (id string, from_id string, to_id string, weight double) USING iceberg;
INSERT INTO modern.created VALUES
    ('e9', 'v1', 'v3', 0.4),
    ('e10', 'v4', 'v5', 1.0),
    ('e11', 'v4', 'v3', 0.4),
    ('e12', 'v6', 'v3', 0.2);

CREATE TABLE modern.knows (id string, from_id string, to_id string, weight double) USING iceberg;
INSERT INTO modern.knows VALUES
    ('e7', 'v1', 'v2', 0.5),
    ('e8', 'v1', 'v4', 1.0);

SELECT COUNT(*) AS person_cnt FROM modern.person;
SELECT COUNT(*) AS software_cnt FROM modern.software;
SELECT COUNT(*) AS created_cnt FROM modern.created;
SELECT COUNT(*) AS knows_cnt FROM modern.knows;
SQL

Run the setup

Run the following commands from the directory that contains both files:

chmod +x setup-polaris.sh
./setup-polaris.sh

When the script completes:

• Polaris is available at http://localhost:8181/api/catalog
• PuppyGraph Web UI is available at http://localhost:8081
• PuppyGraph Gremlin is available at localhost:18182
• The sample Iceberg catalog is modern_catalog
• The sample namespace is modern

The script creates the following tables:

personsoftwarecreatedknows

id	name	age
v1	marko	29
v2	vadas	27
v4	josh	32
v6	peter	35

id	name	lang
v3	lop	java
v5	ripple	java

id	from_id	to_id	weight
e9	v1	v3	0.4
e10	v4	v5	1.0
e11	v4	v3	0.4
e12	v6	v3	0.2

id	from_id	to_id	weight
e7	v1	v2	0.5
e8	v1	v4	1.0

Modeling the Graph

PuppyGraph models external data sources by using a graph schema JSON. In this tutorial, the schema connects PuppyGraph to Polaris through the Iceberg REST catalog and maps the four sample tables to the Modern graph.

Create schema.json

Create a file named schema.json with the following content:

schema.json

{
  "catalogs": [
    {
      "name": "polaris_data",
      "type": "iceberg",
      "metastore": {
        "type": "rest",
        "uri": "http://polaris:8181/api/catalog",
        "warehouse": "modern_catalog",
        "security": "oauth2",
        "credential": "root:s3cr3t",
        "scope": "PRINCIPAL_ROLE:ALL",
        "oauthServerUri": "http://polaris:8181/api/catalog/v1/oauth/tokens"
      },
      "storage": {
        "type": "S3",
        "useInstanceProfile": "false",
        "accessKey": "rustfs_root",
        "secretKey": "rustfspwd",
        "enableSsl": "false",
        "endpoint": "http://polaris-rustfs:9000",
        "enablePathStyleAccess": "true"
      }
    }
  ],
  "graph": {
    "vertices": [
      {
        "label": "person",
        "oneToOne": {
          "tableSource": {
            "catalog": "polaris_data",
            "schema": "modern",
            "table": "person"
          },
          "id": {
            "fields": [
              {
                "type": "String",
                "field": "id",
                "alias": "id"
              }
            ]
          },
          "attributes": [
            {
              "type": "String",
              "field": "name",
              "alias": "name"
            },
            {
              "type": "Int",
              "field": "age",
              "alias": "age"
            }
          ]
        }
      },
      {
        "label": "software",
        "oneToOne": {
          "tableSource": {
            "catalog": "polaris_data",
            "schema": "modern",
            "table": "software"
          },
          "id": {
            "fields": [
              {
                "type": "String",
                "field": "id",
                "alias": "id"
              }
            ]
          },
          "attributes": [
            {
              "type": "String",
              "field": "name",
              "alias": "name"
            },
            {
              "type": "String",
              "field": "lang",
              "alias": "lang"
            }
          ]
        }
      }
    ],
    "edges": [
      {
        "label": "created",
        "fromVertex": "person",
        "toVertex": "software",
        "tableSource": {
          "catalog": "polaris_data",
          "schema": "modern",
          "table": "created"
        },
        "id": {
          "fields": [
            {
              "type": "String",
              "field": "id",
              "alias": "id"
            }
          ]
        },
        "fromId": {
          "fields": [
            {
              "type": "String",
              "field": "from_id",
              "alias": "from_id"
            }
          ]
        },
        "toId": {
          "fields": [
            {
              "type": "String",
              "field": "to_id",
              "alias": "to_id"
            }
          ]
        },
        "attributes": [
          {
            "type": "Double",
            "field": "weight",
            "alias": "weight"
          }
        ]
      },
      {
        "label": "knows",
        "fromVertex": "person",
        "toVertex": "person",
        "tableSource": {
          "catalog": "polaris_data",
          "schema": "modern",
          "table": "knows"
        },
        "id": {
          "fields": [
            {
              "type": "String",
              "field": "id",
              "alias": "id"
            }
          ]
        },
        "fromId": {
          "fields": [
            {
              "type": "String",
              "field": "from_id",
              "alias": "from_id"
            }
          ]
        },
        "toId": {
          "fields": [
            {
              "type": "String",
              "field": "to_id",
              "alias": "to_id"
            }
          ]
        },
        "attributes": [
          {
            "type": "Double",
            "field": "weight",
            "alias": "weight"
          }
        ]
      }
    ]
  }
}

Note

The schema uses the container hostnames polaris and polaris-rustfs. These names resolve correctly because PuppyGraph is running in the same Docker Compose network.

Upload the graph schema

You can upload the schema in either of the following ways.

Option 1: Web UI

Log into PuppyGraph Web UI at http://localhost:8081 with username puppygraph and password puppygraph123.

1. Open http://localhost:8081.
2. Log in with username puppygraph and password puppygraph123.
3. Go to the schema page.
4. Use the Upload Graph Schema JSON section to upload schema.json.

Option 2: REST API

Alternatively, run the following command to upload the schema file:

curl -X POST \
  -H "content-type: application/json" \
  --data-binary @./schema.json \
  --user "puppygraph:puppygraph123" \
  http://localhost:8081/schema

After the upload succeeds, PuppyGraph will visualize the schema and make the graph queryable.

Querying the Graph

After the schema is loaded, open the PuppyGraph Dashboard to inspect the graph summary, or use the Query UI to explore the data interactively.

Run the following example Gremlin queries:

g.V().hasLabel('person').valueMap('name', 'age')

g.V().hasLabel('person').has('name', 'marko').out('knows').values('name')

g.V().hasLabel('person').has('name', 'josh').
  outE('created').
  project('software', 'weight').
    by(inV().values('name')).
    by(values('weight'))

With the sample data in this tutorial, the graph contains:

• 4 person vertices
• 2 software vertices
• 2 knows edges
• 4 created edges

Cleaning up

Run the following command to stop and remove the containers, network, and volume created by this tutorial:

docker compose down -v