Connecting to Iceberg

Apache Iceberg is a high-performance format for huge analytic tables. This document describes how to connect to Iceberg tables in PuppyGraph.

See also Tutorials for end-to-end tutorials for connecting to different Iceberg implementations.

Prerequisites

• Both the Iceberg Catalog and Storage are accessible over the network from the PuppyGraph instance.
• PuppyGraph supports various Iceberg implementations. Please see Metastore Configuration and Data Storage Configuration for supported implementations.

Configuration

The configuration consists of two parts: Metastore (Catalog) and Data Storage. Please configure them according to you Iceberg implementation.

Metastore Configuration

Iceberg REST

Iceberg REST Catalog defines a REST protocol allowing different Iceberg catalog implementations. To use it, select Iceberg REST as the metastore type in the Web UI.

Configuration	Explanation
RestUri	The server endpoint URI of the REST Catalog.
Warehouse	The name of the Iceberg warehouse.
Security	Security authentication mode of the REST catalog.
Credential	The Oauth2 client authentication credential.
Scope	Authorization scope for client credentials or token exchange

The following sections provide specific configurations for different REST Catalog implementations.

Nessie

Project Nessie is a Transactional Catalog for Data Lakes with Git-like semantics. PuppyGraph supports Nessie through Iceberg REST Protocol.

PuppyGraph works with Nessie version 0.95.0 or higher. Please first configure Nessie Catalog with Iceberg REST before connecting.

After selecting Iceberg REST as the metastore type:

• Set RestUri to the Nessie REST endpoint. Typically, this ends with /iceberg like http://127.0.0.1:19120/iceberg.
• Set Warehouse to the Nessie warehouse. This is typically the (base) storage location, for example s3://my-bucket/.

See also the following tutorials:

• Querying Nessie Data as a Graph

Polaris

Apache Polaris is an open-source, fully-featured catalog for Iceberg that implements Iceberg's REST API. PuppyGraph supports both Apache Polaris and Snowflake Open Catalog, a managed service for Polaris.

After selecting Iceberg REST as the metastore type:

• Set RestUri to the Polaris REST endpoint.
• Set Warehouse to the Polaris warehouse accordingly.
• Set Scope to the proper PRINCIPAL_ROLE in Polaris. The format is PRINCIPAL_ROLE:<ROLE_NAME>.
• Set Credential to the client auth credential if needed. The format is <CLIENT-ID>:<CLIENT-SECRET>.

See also the following tutorials for Polaris and Snowflake Open Catalog:

• Querying Snowflake Open Catalog Data as a Graph
• Querying Polaris Data as a Graph

Snowflake Horizon

Snowflake Horizon Catalog exposes the Snowflake-managed Iceberg tables of a Snowflake account through an Iceberg REST endpoint. PuppyGraph reads the table metadata from Horizon Catalog and reads the data files directly from storage, using credentials that Snowflake vends for each table.

In the Web UI, select Snowflake Horizon (Iceberg REST) as the metastore type. This is a preset of Iceberg REST: PuppyGraph saves the catalog as an Iceberg REST catalog ("type": "rest") with Iceberg view support disabled.

Field	Value
REST Endpoint URL	https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog. Replace underscores in the organization or account name with hyphens.
REST Warehouse	The name of the Snowflake database that contains the Iceberg tables. Use upper case if the database was created without quotes.
REST Authentication Type	OAuth 2.0 Client Credentials.
OAuth 2.0 Credential	A Snowflake programmatic access token (PAT).
OAuth 2.0 Scope	session:role:<role>, where <role> is the PAT's role restriction. The role needs USAGE on the database and schema and SELECT on the Iceberg tables.
OAuth 2.0 Server URI	https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens.
Iceberg view support	Disabled — skip Iceberg views on this catalog. The preset selects it. Keep it selected.
Storage type	Get from metastore, so PuppyGraph uses the storage credentials vended by Horizon Catalog.

Iceberg view support must be disabled ("restCatalogViewEndpointsMode": "off" in a schema file). Horizon Catalog advertises the Iceberg view endpoints but rejects view requests with HTTP 403, which fails the whole table listing rather than only the views. With view support disabled, PuppyGraph does not request views, so Iceberg views in Horizon Catalog are not available.

Catalogs created from a schema file or the API

A catalog in a schema file has no Horizon preset. Set "type": "rest" and "restCatalogViewEndpointsMode": "off" explicitly, and omit the storage section to get vended credentials. See Snowflake Horizon + vended credentials for a complete example.

This setup is for Snowflake-managed Iceberg tables. It does not cover Snowflake's preview of Horizon Catalog access to externally managed Iceberg tables in catalog-linked databases. Snowflake native tables are not in Horizon Catalog. Use Connecting to Snowflake for them.

Use a PAT as the OAuth 2.0 Credential. By default, Snowflake lets a service user create or use a PAT only when a network policy applies to the user, and Horizon Catalog does not support network policies set at the user level. Before you create the PAT, attach an authentication policy to the service user that removes this requirement:

CREATE AUTHENTICATION POLICY <policy_name>
  PAT_POLICY = (NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);
ALTER USER <service_user> SET AUTHENTICATION POLICY <policy_name>;

Alternatively, your Snowflake administrator can activate an account-level network policy that allows the PuppyGraph host's egress IP addresses. See Create a programmatic access token in the tutorial for the full setup.

Do not use a key-pair JWT as the credential for a lasting connection. Horizon Catalog accepts one, but it expires one hour after it is created by default, and PuppyGraph does not create a new one.

See also the following tutorial:

• Querying Snowflake Horizon Iceberg Data as a Graph

Tabular

Tabular is a cloud-native managed storage engine that provides a range of services on top of Apache Iceberg tables.

After selecting Iceberg REST as the metastore type:

• Set RestUri to the Tabular REST endpoint. It is https://api.tabular.io/ws by default.
• Set Warehouse to the name of the Tabular warehouse
• Set Credential to the Tabular member or service credential.

Google Cloud Lakehouse

Google Cloud Lakehouse (formerly BigLake) is a storage engine that unifies data warehouses and lakes. PuppyGraph supports Google Cloud Lakehouse through its Iceberg REST Catalog API.

PuppyGraph uses Application Default Credentials (ADC) for authentication. See Authentication for PuppyGraph to access Google Cloud resources for how to configure credentials.

In the Web UI, after selecting Google BigLake REST as the metastore type:

• Set Endpoint URL to the REST catalog URI, which is generally https://biglake.googleapis.com/iceberg/v1/restcatalog.
• Set Warehouse to the GCS bucket path. The format is gs://BUCKET_NAME.
• Set GCP User Project to your Google Cloud project ID.
• Set Storage type as Get from metastore.

See also the following tutorial:

• Querying Google Cloud Lakehouse Iceberg Tables Data as a Graph

AWS Glue

Configuration	Explanation
Region	The region of the AWS Glue Data Catalog. Example: us-east-1. See AWS Glue endpoints and quotas for more details.
Use instance profile	Whether to use role-based authentication (Explicit IAM roles or instance-profile attached)
IAM Role ARN	The ARN of the IAM role for accessing the AWS Glue Data Catalog. Required by authentication with IAM roles.
Access key	The access key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.
Secret key	The secret key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.

Amazon S3 Tables

Amazon S3 Tables provide S3 storage that’s optimized for analytics workloads, with features designed to continuously improve query performance and reduce storage costs for tables.

To connect to the data from PuppyGraph, you need a user account with proper permissions (at least AmazonS3TablesReadOnlyAccess).

Configuration	Explanation
useDefaultCredentialsProviderChain	Whether to use Default Authentication. Default value false. See Using Default Authentication in PuppyGraph for more details.
Region	The region of the AWS S3 Table Bucket. Example: us-east-1. See S3 Tables AWS Regions, endpoints, and service quotas for more details.
Access key	The access key of the IAM user for accessing the Amazon S3 Tables. Required by authentication with IAM User Access keys.
Secret key	The secret key of the IAM user for accessing the Amazon S3 Tables. Required by authentication with IAM User Access keys.
Warehouse	The table bucket ARN of the Amazon S3 table bucket.

Hive Metastore

Configuration	Explanation
Hive metastore URI	The URI of your Hive metastore. Format: thrift://<metastore_IP_address>:<metastore_port>.

OneLake Table APIs for Iceberg

OneLake offers a REST API endpoint for interacting with tables in Microsoft Fabric. This endpoint can be used with clients and libraries that are compatible with the Iceberg REST Catalog (IRC) API open standard or the Unity Catalog API open standard.

You need to gather the following information before connecting PuppyGraph to OneLake Table APIs for Iceberg:

Placeholder	Description
<fabric_workspace_id>	The ID of your Microsoft Fabric workspace
<fabric_data_item_id>	The ID of the lakehouse data item
<client_id>	The client ID of the service principal
<client_secret>	The client secret of the service principal
<tenant_id>	Your Microsoft Entra ID tenant ID - see endpoint

After selecting Iceberg REST as the metastore type:

• Set Endpoint URL as https://onelake.table.fabric.microsoft.com/iceberg.

• Set Warehouse as <fabric_workspace_id>/<fabric_data_item_id>.

• Set Credential to the OAuth2 client authentication credential. The format is <client_id>:<client_secret>. Optionally, set Client ID and Client Secret directly.

• Set Scope to https://storage.azure.com/.default.

• Set Rest oauth2 server uri to https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token.

Data Storage Configuration

Get from metastore

There is no need to specify Storage configuration with the following implementation of Iceberg:

• HDFS with Hive Metastore
• Iceberg REST Catalog with credential vending.

Select Get from metastore in the Web UI for these implementations.

Vended credentials do not include a storage endpoint. If the catalog vends credentials for Amazon S3 but the bucket is reachable only through a private endpoint (for example AWS PrivateLink), select Amazon S3 instead of Get from metastore, set S3 Authentication Type to Credentials vended by the catalog, and set Region and Endpoint for the private endpoint. PuppyGraph then uses the vended credentials with that endpoint.

Amazon S3 (Simple Storage Service)

PuppyGraph supports Amazon S3 (Simple Storage Service) for Iceberg.

Configuration	Explanation
Region	The region of the Amazon S3. Example: us-east-1. See Amazon Simple Storage Service endpoints and quotas for more details.
Use instance profile	Whether to use role-based authentication (Explicit IAM roles or instance-profile attached).
IAM Role ARN	The ARN of the IAM role for accessing the Amazon S3. Required by authentication with IAM roles.
Access key	The access key of the IAM user for accessing the Amazon S3. Required by authentication with IAM User Access keys.
Secret key	The secret key of the IAM user for accessing the Amazon S3. Required by authentication with IAM User Access keys.

Amazon S3 Compatible Storage

PuppyGraph supports S3 Compatible Storage (e.g. RustFS or MinIO AIStor) for Iceberg.

Configuration	Explanation
Endpoint	The S3 compatible storage endpoint.
Access key	The access key of an IAM user for accessing the S3 compatible storage.
Secret key	The secret key of an IAM user for accessing the S3 compatible storage.
Enable SSL	Whether to enable SSL connection for accessing the S3 compatible storage.
Enable path style access	Whether to use path-style access method when accessing the S3 compatible storage.

GCS (Google Cloud Storage)

PuppyGraph supports GCS (Google Cloud Storage) for Iceberg.

Using VM instance

Configuration	Explanation
useComputeEngineServiceAccount	Use Compute Engine service account, set to true here

Using VM instance impersonate a service account

Configuration	Explanation
useComputeEngineServiceAccount	Use Compute Engine service account, set to true here
impersonationServiceAccount	Impersonation service account

Using service account

Configuration	Explanation
serviceAccountEmail	Service account email address associated with your Google Cloud project
serviceAccountPrivateKeyId	Unique identifier for the service account's private key
serviceAccountPrivateKey	Service account's private key for authentication

Using service account impersonate another service account

Configuration	Explanation
serviceAccountEmail	Service account email address associated with your Google Cloud project
serviceAccountPrivateKeyId	Unique identifier for the service account's private key
serviceAccountPrivateKey	Service account's private key for authentication
impersonationServiceAccount	Impersonation service account

Azure Onelake Storage

PuppyGraph supports Azure OneLake Storage for Iceberg.

Configuration	Explanation
Client ID	The Application (client) ID of the Azure AD app registered for authentication.
Client Secret	The client secret (password) generated in the Azure AD app registration. Used together with the client ID for authentication.
Client Endpoint	The OAuth2 token endpoint of your Azure Active Directory tenant.

Tutorials

• Querying Google Cloud Lakehouse Iceberg Tables Data as a Graph
• Querying Iceberg Data as a Graph
• Querying Nessie Data as a Graph
• Querying Azure OneLake Data as a Graph
• Querying Snowflake Open Catalog Data as a Graph
• Querying Snowflake Horizon Iceberg Data as a Graph
• Querying Polaris Data as a Graph
• Querying Amazon S3 Tables Data as a Graph
• Run Graph Queries on Apache Iceberg Tables with Dremio & Puppygraph

Data Type Mapping

Iceberg Type	PuppyGraph Type	Description
boolean	Boolean	True or false
int	Int	32-bit signed integers
long	Long	64-bit signed integers
float	Float	32-bit IEEE 754 floating point
double	Double	64-bit IEEE 754 floating point
decimal(P, S)	Decimal(P, S)	Fixed-point decimal; precision P, scale S
string	String	Arbitrary-length character sequences
date	Date	Calendar date without timezone or time
time	Time	Time of day, microsecond precision, without date, timezone
timestamp	DateTime	Timestamp, microsecond precision, without timezone
list	Array<E>	A collection of values with some element type
struct	Struct<field1 E1[,field2 E2...]>	A tuple of typed values

If you need a data type that is not supported, or one that is not listed here, contact us.

Example Configurations

Catalog Type	Storage Type	Example Configuration
REST Catalog	Amazon S3	#rest-catalog-amazon-s3
REST Catalog	RustFS	#rest-catalog-rustfs
REST Catalog	OneLake	#rest-catalog-azure-onelake
Snowflake Horizon	Vended credentials	#snowflake-horizon-vended-credentials
AWS Glue	Amazon S3	#aws-glue-amazon-s3
Hive Metastore	HDFS	#hive-metastore-hdfs
Hive Metastore	Amazon S3	#hive-metastore-amazon-s3
Hive Metastore	RustFS	#hive-metastore-rustfs
Hive Metastore	Google GCS	#hive-metastore-google-gcs
Hive Metastore	Azure Blob	#hive-metastore-azure-blob
Hive Metastore	Azure Data Lake Gen2	#hive-metastore-azure-data-lake-gen2

REST Catalog + Amazon S3

{
  "catalog": [
    {
      "name": "iceberg_rest_s3",
      "type": "iceberg",
      "metastore": {
        "type": "rest",
        "uri": "http://127.0.0.1:8181"
      },
      "storage": {
        "useInstanceProfile": "false",
        "region": "us-west-2",
        "accessKey": "AKIAIOSFODNN7EXAMPLE",
        "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "enableSsl": "false"
      }
    }
  ]
}

REST Catalog + RustFS

{
  "catalog": [
    {
      "name": "iceberg_rest_rustfs",
      "type": "iceberg",
      "metastore": {
        "type": "rest",
        "uri": "http://127.0.0.1:8181"
      },
      "storage": {
        "useInstanceProfile": "false",
        "accessKey": "admin",
        "secretKey": "password",
        "enableSsl": "false",
        "endpoint": "http://127.0.0.1:9000",
        "enablePathStyleAccess": "true"
      }
    }
  ]
}

REST Catalog + Azure OneLake

{
  "catalog": [
      {
        "name": "az_iceberg",
        "type": "iceberg",
        "metastore": {
          "type": "rest",
          "uri": "http://127.0.0.1:8181"
        },
        "storage": {
          "type": "AzureDLS2",
          "clientId": "<oauth-2.0-client-id>",
          "clientSecret": "<oauth-2.0-client-secret>",
          "clientEndpoint": "<oauth-2.0-token-endpoint>"
        }
      }
    ]
}

Snowflake Horizon + vended credentials

{
  "catalog": [
    {
      "name": "horizon_data",
      "type": "iceberg",
      "metastore": {
        "type": "rest",
        "uri": "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog",
        "warehouse": "<SNOWFLAKE_DATABASE>",
        "authType": "oauth2",
        "security": "oauth2",
        "credential": "<programmatic_access_token>",
        "scope": "session:role:<ROLE>",
        "oauthServerUri": "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens",
        "restCatalogViewEndpointsMode": "off"
      }
    }
  ]
}

AWS Glue + Amazon S3

{
  "catalog": [
    {
      "name": "iceberg_glue_s3",
      "type": "iceberg",
      "metastore": {
        "type": "glue",
        "useInstanceProfile": "false",
        "region": "us-west-2",
        "accessKey": "AKIAIOSFODNN7EXAMPLE",
        "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      },
      "storage": {
        "useInstanceProfile": "false",
        "region": "us-west-2",
        "accessKey": "AKIAIOSFODNN7EXAMPLE",
        "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "enableSsl": "false"
      }
    }
  ]
}

Amazon S3 Tables + Amazon S3

{
  "catalog": [
        {
          "name": "iceberg_s3tables_s3",
          "type": "iceberg",
          "metastore": {
            "type": "s3tables",
            "region": "us-west-2",
            "accessKey": "AKIAIOSFODNN7EXAMPLE",
            "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "warehouse": "arn:aws:s3tables:us-west-2:111122223333:bucket/bucket-name123",
            "useDefaultCredentialsProviderChain": "false"
          },
          "storage": {
            "useDefaultCredentialsProviderChain": "false",
            "useInstanceProfile": "false",
            "region": "us-west-2",
            "accessKey": "AKIAIOSFODNN7EXAMPLE",
            "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "enableSsl": "true",
            "type": "S3" 
          }
        }
      ]
}

Amazon S3 Tables + Amazon S3 using Default Authentication

{
  "catalog": [
        {
          "name": "iceberg_s3tables_s3",
          "type": "iceberg",
          "metastore": {
            "type": "s3tables",
            "region": "us-west-2",
            "useDefaultCredentialsProviderChain": "true",
            "useInstanceProfile": "false",
            "warehouse": "arn:aws:s3tables:us-west-2:111122223333:bucket/bucket-name123"
          },
          "storage": {
            "useDefaultCredentialsProviderChain": "true",
            "useInstanceProfile": "false",
            "region": "us-west-2",
            "type": "S3"
          }
        }
      ]
}

Hive Metastore + HDFS

{
  "catalog": [
    {
      "name": "iceberg_hms_hdfs",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      }
    }
  ]
}

Hive Metastore + RustFS

{
  "catalog": [
    {
      "name": "iceberg_hms_rustfs",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      },
      "storage": {
        "useInstanceProfile": "false",
        "accessKey": "admin",
        "secretKey": "password",
        "enableSsl": "false",
        "endpoint": "http://127.0.0.1:9000",
        "enablePathStyleAccess": "true"
      }
    }
  ]
}

Hive Metastore + Amazon S3

{
  "catalog": [
    {
      "name": "iceberg_hms_s3",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      },
      "storage": {
        "useInstanceProfile": "false",
        "region": "us-west-2",
        "accessKey": "AKIAIOSFODNN7EXAMPLE",
        "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "enableSsl": "false"
      }
    }
  ]
}

Hive Metastore + Google GCS

{
  "catalog": [
    {
      "name": "iceberg_hms_gcs",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      },
      "storage": {
        "type": "GCS",
        "serviceAccountEmail": "acc_name@project.iam.gserviceaccount.com",
        "serviceAccountPrivateKeyId": "AKIAIOSFODNN7EXAMPLE",
        "serviceAccountPrivateKey": "-----BEGIN PRIVATE KEY-----\nabcded\n-----END PRIVATE KEY-----\n"
      }
    }
  ]
}

Hive Metastore + Azure Blob

{
  "catalog": [
    {
      "name": "iceberg_hms_azblob",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      },
      "storage": {
        "type": "AzureBlob",
        "storageAccount": "account_name",
        "storageContainer": "container_name",
        "sasToken": "sp=rl&st=2020-12-15T03:19:48Z&se=2024-12-12T11:19:48Z&sv=2022-11-02&sr=c&sig=1"
      }
    }
  ]
}

Hive Metastore + Azure Data Lake Gen2

{
  "catalog": [
    {
      "name": "iceberg_hms_azgen2",
      "type": "iceberg",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://127.0.0.1:9083"
      },
      "storage": {
        "type": "AzureDLS2",
        "clientId": "000000-avaf-aaaa-bbbb-aba988azfa",
        "clientSecret": "EXAMPLEvonefPJabcde",
        "clientEndpoint": "https://login.microsoftonline.com/000000-avaf-aaaa-bbbb-aba988azfa/oauth2/token"
      }
    }
  ]
}