Querying Snowflake Horizon Iceberg Data as a Graph

Summary

Snowflake Horizon Catalog exposes the Snowflake-managed Iceberg tables in your Snowflake account through an Iceberg REST catalog endpoint. PuppyGraph connects to that endpoint like any other Iceberg REST catalog: it reads table metadata from Horizon, receives short-lived storage credentials vended by Snowflake for each table, and reads the Parquet data files directly from your cloud storage. Graph queries do not run on a Snowflake warehouse.

In this tutorial, you will:

• Create Snowflake-managed Iceberg tables and a read-only service user with a programmatic access token (PAT).
• Start a PuppyGraph container and connect it to the Horizon Catalog endpoint of your Snowflake account.
• Run Cypher and Gremlin queries against the Iceberg data as a graph.

Requires a Snowflake account

This tutorial uses a real Snowflake account and a cloud storage location (Amazon S3, Google Cloud Storage, or Azure) registered in Snowflake as an external volume. PuppyGraph runs self-hosted in Docker. See Connecting to Iceberg for the catalog reference.

Prerequisites

• docker is available on the host where you'll run PuppyGraph.
• A Snowflake account and a role that can run every SQL statement in this tutorial: create a database, Iceberg tables, a role, a user, and an authentication policy, and apply the policy to the user. The statements are written to run under a single role, such as ACCOUNTADMIN, without switching roles.
• A Snowflake virtual warehouse that the role can use, to run the INSERT statements that load the sample data. This tutorial calls it <warehouse>. It is not the same as the REST Warehouse field in PuppyGraph, which is set to the Snowflake database name.
• An external volume for the storage location that will hold the Iceberg data. This tutorial calls it <external_volume>.
• The PuppyGraph host can reach https://<org>-<account>.snowflakecomputing.com and the storage location over HTTPS.

Setup

Create Snowflake-managed Iceberg tables

The tutorial creates the database PUPPYGRAPH_HORIZON_DEMO, the role PUPPYGRAPH_HORIZON_READER, and the user PUPPYGRAPH_HORIZON_SVC, and Cleanup drops all three. The statements that create them fail if an object with the same name already exists, so the tutorial never reuses and later drops an existing object. If a name is already taken, use a different name throughout.

In Snowsight, open a SQL worksheet and run the following, replacing <warehouse> with the name of your virtual warehouse and <external_volume> with the name of your external volume:

modern.sql

USE WAREHOUSE <warehouse>;
CREATE DATABASE PUPPYGRAPH_HORIZON_DEMO;
USE DATABASE PUPPYGRAPH_HORIZON_DEMO;
CREATE SCHEMA IF NOT EXISTS MODERN;
USE SCHEMA MODERN;

CREATE ICEBERG TABLE person (
  id   STRING,
  name STRING,
  age  INT
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = '<external_volume>'
  BASE_LOCATION = 'modern/person';
INSERT INTO person VALUES
  ('v1', 'marko', 29),
  ('v2', 'vadas', 27),
  ('v4', 'josh',  32),
  ('v6', 'peter', 35);

CREATE ICEBERG TABLE software (
  id   STRING,
  name STRING,
  lang STRING
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = '<external_volume>'
  BASE_LOCATION = 'modern/software';
INSERT INTO software VALUES
  ('v3', 'lop',    'java'),
  ('v5', 'ripple', 'java');

CREATE ICEBERG TABLE created (
  id      STRING,
  from_id STRING,
  to_id   STRING,
  weight  DOUBLE
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = '<external_volume>'
  BASE_LOCATION = 'modern/created';
INSERT INTO created VALUES
  ('e9',  'v1', 'v3', 0.4),
  ('e10', 'v4', 'v5', 1.0),
  ('e11', 'v4', 'v3', 0.4),
  ('e12', 'v6', 'v3', 0.2);

CREATE ICEBERG TABLE knows (
  id      STRING,
  from_id STRING,
  to_id   STRING,
  weight  DOUBLE
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = '<external_volume>'
  BASE_LOCATION = 'modern/knows';
INSERT INTO knows VALUES
  ('e7', 'v1', 'v2', 0.5),
  ('e8', 'v1', 'v4', 1.0);

CATALOG = 'SNOWFLAKE' makes these Snowflake-managed Iceberg tables, the table type this tutorial uses with Horizon Catalog. In a Snowflake-managed Iceberg table, INT is written as the Iceberg int type and BIGINT as long, while an explicit NUMBER(p,0) becomes an Iceberg decimal(p,0).

Because the identifiers above are not quoted, Snowflake stores them in upper case: the database is PUPPYGRAPH_HORIZON_DEMO, the schema is MODERN, the tables are PERSON, SOFTWARE, CREATED, and KNOWS, and the columns are ID, NAME, and so on. Use the upper-case names everywhere in PuppyGraph.

personsoftwarecreatedknows

id	name	age
v1	marko	29
v2	vadas	27
v4	josh	32
v6	peter	35

id	name	lang
v3	lop	java
v5	ripple	java

id	from_id	to_id	weight
e9	v1	v3	0.4
e10	v4	v5	1.0
e11	v4	v3	0.4
e12	v6	v3	0.2

id	from_id	to_id	weight
e7	v1	v2	0.5
e8	v1	v4	1.0

Create a read-only role and service user

PuppyGraph authenticates to Horizon Catalog as a Snowflake user, and every request runs with a single Snowflake role. Create a role that can only read the demo tables, and a service user that holds it:

CREATE ROLE PUPPYGRAPH_HORIZON_READER;
GRANT USAGE ON DATABASE PUPPYGRAPH_HORIZON_DEMO TO ROLE PUPPYGRAPH_HORIZON_READER;
GRANT USAGE ON SCHEMA PUPPYGRAPH_HORIZON_DEMO.MODERN TO ROLE PUPPYGRAPH_HORIZON_READER;
GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA PUPPYGRAPH_HORIZON_DEMO.MODERN TO ROLE PUPPYGRAPH_HORIZON_READER;
GRANT SELECT ON FUTURE ICEBERG TABLES IN SCHEMA PUPPYGRAPH_HORIZON_DEMO.MODERN TO ROLE PUPPYGRAPH_HORIZON_READER;

CREATE USER PUPPYGRAPH_HORIZON_SVC
  TYPE = SERVICE
  DEFAULT_ROLE = PUPPYGRAPH_HORIZON_READER;
GRANT ROLE PUPPYGRAPH_HORIZON_READER TO USER PUPPYGRAPH_HORIZON_SVC;

Iceberg tables need their own grants

GRANT SELECT ON ALL TABLES does not cover Iceberg tables. Grant SELECT ON ALL ICEBERG TABLES (and FUTURE ICEBERG TABLES), or grant the tables individually.

For PuppyGraph to receive vended storage credentials, the role that owns the Iceberg tables (the role you used to create them above) must also have USAGE on the external volume. If it does not, grant it:

GRANT USAGE ON EXTERNAL VOLUME <external_volume> TO ROLE <table_owner_role>;

Create a programmatic access token

PuppyGraph authenticates with a programmatic access token (PAT).

By default, Snowflake does not let a service user create or use a PAT unless the user is subject to a network policy. Without one, creating the token fails with Network Policy is required when creating a programmatic access token. Horizon Catalog does not support network policies set at the user level, so remove or meet the requirement in one of these ways:

• Attach an authentication policy to the service user that does not require a network policy for PATs. This is the setup used in this tutorial.
• Have your Snowflake administrator activate a network policy for the whole account that allows the egress IP addresses of the PuppyGraph host. An account-level network policy applies to every user in the account, so do not set one up only for this tutorial.

Create the authentication policy and attach it to the service user:

CREATE AUTHENTICATION POLICY PUPPYGRAPH_HORIZON_DEMO.PUBLIC.PUPPYGRAPH_HORIZON_PAT_POLICY
  PAT_POLICY = (NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);
ALTER USER PUPPYGRAPH_HORIZON_SVC
  SET AUTHENTICATION POLICY PUPPYGRAPH_HORIZON_DEMO.PUBLIC.PUPPYGRAPH_HORIZON_PAT_POLICY;

ENFORCED_NOT_REQUIRED lets the user authenticate with a PAT without a network policy, and still enforces a network policy if one applies to the user. The PAT is then the only credential that protects access, so keep it secret and give it a short expiry.

Create a PAT restricted to the read-only role:

ALTER USER PUPPYGRAPH_HORIZON_SVC ADD PROGRAMMATIC ACCESS TOKEN PUPPYGRAPH_HORIZON_PAT
  ROLE_RESTRICTION = 'PUPPYGRAPH_HORIZON_READER'
  DAYS_TO_EXPIRY = 30;

Copy the token_secret value from the result. Snowflake shows it only once. This tutorial calls it <pat>.

Find your account identifier

The Horizon Catalog endpoint is https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog, where <org>-<account> is your account identifier in the <organization_name>-<account_name> form. If your organization or account name contains underscores, replace them with hyphens in the host name (for example my_org-my_account becomes my-org-my-account).

Check the token (optional)

Before you configure PuppyGraph, you can confirm that the PAT, role, and database work together. Exchange the PAT for an access token. read -rs prompts for the PAT without echoing it, and curl reads it from standard input, so the PAT does not end up in your shell history or in the curl command line:

read -rs PAT   # paste <pat>, then press Enter
printf '%s' "$PAT" | curl -s -X POST "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'scope=session:role:PUPPYGRAPH_HORIZON_READER' \
  --data-urlencode 'client_secret@-'

Then use the returned access_token to read the catalog configuration for the database. As before, read -rs keeps the token out of your shell history and the curl command line:

read -rs TOKEN   # paste <access_token>, then press Enter
printf 'Authorization: Bearer %s' "$TOKEN" | curl -s "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/config?warehouse=PUPPYGRAPH_HORIZON_DEMO" -H @-

A JSON response whose overrides.prefix is PUPPYGRAPH_HORIZON_DEMO means the endpoint, token, and role are correct. PuppyGraph performs the same token exchange itself. It needs the PAT, not the access token.

Start PuppyGraph

Start the PuppyGraph container:

docker run -d --name puppygraph \
  -p 8081:8081 -p 8182:8182 -p 7687:7687 \
  -e PUPPYGRAPH_USERNAME=puppygraph \
  -e PUPPYGRAPH_PASSWORD=puppygraph123 \
  --pull=always puppygraph/puppygraph:latest

Default password

Change PUPPYGRAPH_PASSWORD before running on a publicly accessible machine.

Modeling a Graph

We model the data as the TinkerPop modern graph: two node types (person, software) and two edge types (knows, created).

Modern Graph

Log into the PuppyGraph Web UI at http://localhost:8081 with puppygraph / puppygraph123.

Build the graph in the Schema Builder

Open the Catalogs page, click Add Catalog, then pick Apache Iceberg under Data Lakes.

Fill in the connection form:

Field	Value
Catalog name	horizon_data
Metastore type	Snowflake Horizon (Iceberg REST)
REST Endpoint URL	https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog
REST Warehouse	PUPPYGRAPH_HORIZON_DEMO (the Snowflake database name, not a Snowflake warehouse)
REST Authentication Type	OAuth 2.0 Client Credentials
OAuth 2.0 Credential	<pat>
OAuth 2.0 Scope	session:role:PUPPYGRAPH_HORIZON_READER
OAuth 2.0 Server URI	https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens
Iceberg view support	Disabled — skip Iceberg views on this catalog (selected automatically for Snowflake Horizon; leave it)
Storage type	Get from metastore

Snowflake Horizon (Iceberg REST) is a preset of the Iceberg REST catalog. PuppyGraph saves the catalog as an ordinary Iceberg REST catalog (metastore type rest), with Iceberg view support disabled.

Keep Iceberg view support disabled

Horizon Catalog advertises the Iceberg view endpoints but rejects view requests with HTTP 403. If PuppyGraph asks for views, the whole table listing fails with "Failed to list tables from the REST catalog because its view endpoints rejected the request." With view support disabled, PuppyGraph never requests views, and tables list normally. Iceberg views in Horizon Catalog are therefore not available in PuppyGraph.

With Get from metastore, PuppyGraph uses the storage credentials that Horizon Catalog vends for each table, so you do not configure any cloud keys.

Click Create Catalog. PuppyGraph validates the connection by authenticating and listing the catalog. The catalog horizon_data then shows the schema MODERN with the tables PERSON, SOFTWARE, CREATED, and KNOWS.

Open the Graph page and use Add Node and Add Edge to model the graph:

• person node from MODERN.PERSON, with ID as the ID and NAME and AGE as attributes.
• software node from MODERN.SOFTWARE, with ID as the ID and NAME and LANG as attributes.
• knows edge from MODERN.KNOWS, from person (FROM_ID) to person (TO_ID), with ID as the ID and WEIGHT as an attribute.
• created edge from MODERN.CREATED, from person (FROM_ID) to software (TO_ID), with ID as the ID and WEIGHT as an attribute.

See Modeling a Graph through the Schema Builder for a walkthrough of the node and edge dialogs.

Upload a schema file

Instead of using the Schema Builder, you can upload the whole graph schema, including the catalog, as one file.

The upload replaces the whole graph

The schema.json below is the complete graph definition, not an addition to the current graph. When you upload it, PuppyGraph removes every node, edge, and local table that is not in the file. Use a fresh PuppyGraph instance for this tutorial. To upload to an instance that already has a graph, first download its current schema and merge its nodes, edges, and local tables into the file. See How an upload is applied.

If you already created the catalog in the Web UI

The upload is rejected when the horizon_data catalog already exists with a different definition, for example after you created it with the form above. Upload to a PuppyGraph instance without that catalog, or allow the change to the existing catalog as described in How an upload is applied.

Create a file schema.json with the following content. Replace <org>-<account> and <pat>. The file then contains the PAT, so do not commit or share it, and delete it after the upload:

schema.json

{
  "catalog": [
    {
      "name": "horizon_data",
      "type": "iceberg",
      "metastore": {
        "type": "rest",
        "uri": "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog",
        "warehouse": "PUPPYGRAPH_HORIZON_DEMO",
        "authType": "oauth2",
        "security": "oauth2",
        "credential": "<pat>",
        "scope": "session:role:PUPPYGRAPH_HORIZON_READER",
        "oauthServerUri": "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens",
        "restCatalogViewEndpointsMode": "off"
      }
    }
  ],
  "node": [
    {
      "label": "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "horizon_data",
          "schema": "MODERN",
          "table": "SOFTWARE",
          "mappedField": [
            { "sourceFieldName": "ID",   "targetFieldName": "id"   },
            { "sourceFieldName": "NAME", "targetFieldName": "name" },
            { "sourceFieldName": "LANG", "targetFieldName": "lang" }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "lang", "type": "STRING" }
      ]
    },
    {
      "label": "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "horizon_data",
          "schema": "MODERN",
          "table": "PERSON",
          "mappedField": [
            { "sourceFieldName": "ID",   "targetFieldName": "id"   },
            { "sourceFieldName": "NAME", "targetFieldName": "name" },
            { "sourceFieldName": "AGE",  "targetFieldName": "age"  }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "age",  "type": "INT"    }
      ]
    }
  ],
  "edge": [
    {
      "label":         "created",
      "fromNodeLabel": "person",
      "toNodeLabel":   "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "horizon_data",
          "schema": "MODERN",
          "table": "CREATED",
          "mappedField": [
            { "sourceFieldName": "ID",      "targetFieldName": "id"      },
            { "sourceFieldName": "FROM_ID", "targetFieldName": "from_id" },
            { "sourceFieldName": "TO_ID",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "WEIGHT",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    },
    {
      "label":         "knows",
      "fromNodeLabel": "person",
      "toNodeLabel":   "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "horizon_data",
          "schema": "MODERN",
          "table": "KNOWS",
          "mappedField": [
            { "sourceFieldName": "ID",      "targetFieldName": "id"      },
            { "sourceFieldName": "FROM_ID", "targetFieldName": "from_id" },
            { "sourceFieldName": "TO_ID",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "WEIGHT",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    }
  ]
}

The catalog entry is what the Snowflake Horizon preset produces: an Iceberg REST metastore with restCatalogViewEndpointsMode set to "off". Leaving that setting out in a schema file means auto, which fails table listing against Horizon Catalog. There is no storage section, which is the schema-file equivalent of Get from metastore.

In the Web UI, click Graph in the sidebar, then Upload Schema, and select schema.json.

Upload via CLI

curl -X POST -H "content-type: application/json" \
  --data-binary @./schema.json \
  --user "puppygraph:puppygraph123" \
  http://localhost:8081/schema

Querying the Graph

In the PuppyGraph Web UI, click Query in the sidebar. You can run graph queries in either Cypher or Gremlin.

First, confirm that PuppyGraph reads the Iceberg data. The query returns 4, the number of rows in PERSON:

CypherGremlin

MATCH (p:person) RETURN count(p);

g.V().hasLabel('person').count()

The following query answers "What software was created by people that marko knows?"

CypherGremlin

MATCH path = (p:person)-[:knows]->()-[:created]->()
WHERE p.name = 'marko'
RETURN path;

g.V().hasLabel('person').has('name', 'marko')
  .out('knows').out('created').path()

The query returns two paths, from marko through josh to ripple and to lop.

Limitations

• This tutorial covers Snowflake-managed Iceberg tables (CATALOG = 'SNOWFLAKE'). Snowflake native tables do not appear in Horizon Catalog. Snowflake offers a preview of Horizon Catalog access to externally managed Iceberg tables in catalog-linked databases, which this tutorial does not cover. To graph Snowflake native tables, use Connecting to Snowflake instead.
• PuppyGraph cannot read Iceberg views from Horizon Catalog. Horizon Catalog rejects Iceberg view requests with HTTP 403, so PuppyGraph reads only the Iceberg tables.
• The PAT expires after DAYS_TO_EXPIRY days. Before it expires, create a new token and update the catalog's OAuth 2.0 Credential.
• Horizon Catalog also accepts key-pair JWTs as the credential, but a key-pair JWT expires one hour after it is created by default and PuppyGraph does not create new ones. Use a PAT for a catalog that should keep working.
• See Snowflake's considerations for Horizon Catalog for limitations on the Snowflake side, such as supported clouds and cloned or converted tables.

Troubleshooting

Symptom	Likely cause and fix
Catalog creation fails with an authentication or OAuth error, or the token request returns unauthorized_client	The PAT is wrong, expired, or revoked, or a network policy that applies to the user does not allow the PuppyGraph host's IP address. Check the token with the curl commands above from the PuppyGraph host.
OAuth error mentioning the role or scope	The scope must be session:role:<role>, and <role> must be the PAT's ROLE_RESTRICTION role (a role granted to the user).
Table listing fails with "Failed to list tables from the REST catalog because its view endpoints rejected the request"	Iceberg view support is enabled. Edit the catalog and set Iceberg view support to Disabled — skip Iceberg views on this catalog (restCatalogViewEndpointsMode: "off" in a schema file).
The catalog or schema shows no tables	The role is missing SELECT ON ALL ICEBERG TABLES, the tables are not Snowflake-managed Iceberg tables, or REST Warehouse is not the database name in upper case.
Tables list, but queries fail with a storage access error	The table owner role lacks USAGE on the external volume, or the PuppyGraph host cannot reach the storage location. For S3 encrypted with SSE-KMS, see Snowflake's notes on KMS key access for vended credentials.
Connection times out	The PuppyGraph host cannot reach <org>-<account>.snowflakecomputing.com or the storage endpoint. If your bucket is reachable only through a private endpoint, see Get from metastore for how to set the endpoint while keeping vended credentials.

Cleanup

Stop the PuppyGraph container and delete schema.json if you created it:

docker stop puppygraph && docker rm puppygraph
rm -f schema.json

Remove the Snowflake objects:

ALTER USER PUPPYGRAPH_HORIZON_SVC REMOVE PROGRAMMATIC ACCESS TOKEN PUPPYGRAPH_HORIZON_PAT;
DROP USER IF EXISTS PUPPYGRAPH_HORIZON_SVC;
DROP ROLE IF EXISTS PUPPYGRAPH_HORIZON_READER;
DROP DATABASE IF EXISTS PUPPYGRAPH_HORIZON_DEMO;