Skip to content

Connecting to Delta Lake

Delta Lake is an open-source storage framework that enables building a format agnostic Lakehouse architecture. This document describes how to connect to Delta Lake tables in PuppyGraph.

See also Tutorials for end-to-end tutorials for connecting to different Delta Lake implementations, including Databricks.

Prerequisites

  • Both the Delta Lake Metastore and Data Storage are accessible over the network from the PuppyGraph instance.

Configuration

The configuration consists of two parts: Metastore and Data Storage. Please configure them according to you Delta Lake setup.

Metastore Configuration

Unity Catalog

Configuration Explanation
Databricks host The hostname of the Databricks URL. The format is $databricks-customer-prefix.cloud.databricks.com
Databricks token The access token of the Databricks user. See this page for more details.
Databricks catalog name The catalog name under the Unity Catalog instance. See this page for more details.

AWS Glue

Configuration Explanation
Region The region of the AWS Glue Data Catalog. Example: us-east-1. See AWS Glue endpoints and quotas for more details.
Use instance profile Whether to use role-based authentication (Explicit IAM roles or instance-profile attached)
IAM Role ARN The ARN of the IAM role for accessing the AWS Glue Data Catalog. Required by authentication with IAM roles.
Access key The access key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.
Secret key The secret key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.

Hive Metastore

Configuration Explanation
Hive metastore URI The URI of your Hive metastore. Format: thrift://<metastore_IP_address>:<metastore_port>.

Data Storage Configuration

Get from metastore

There is no need to specify Storage configuration when Delta Lake uses one of the following metastore and storage combinations:

Select Get from metastore in the Web UI for these implementation combinations.

Amazon S3 (Simple Storage Service)

PuppyGraph supports Amazon S3 (Simple Storage Service) for Delta Lake.

Configuration Explanation
Region The region of the Amazon S3. Example: us-east-1. See Amazon Simple Storage Service endpoints and quotas for more details.
Use instance profile Whether to use role-based authentication (Explicit IAM roles or instance-profile attached).
IAM Role ARN The ARN of the IAM role for accessing the Amazon S3. Required by authentication with IAM roles.
Access key The access key of the IAM user for accessing the Amazon S3. Required by authentication with IAM User Access keys.
Secret key The ARN of the IAM role for accessing the Amazon S3. Required by authentication with IAM User Access keys.

Amazon S3 Compatible Storage

PuppyGraph supports S3 Compatible Storage (e.g. RustFS or MinIO AIStor) for Delta Lake.

Configuration Explanation
Endpoint The S3 compatible storage endpoint.
Access key The access key of an IAM user for accessing the S3 compatible storage.
Secret key The secret key of an IAM user for accessing the S3 compatible storage.
Enable SSL Whether to enable SSL connection for accessing the S3 compatible storage.
Enable path style access Whether to use path-style access method when accessing the S3 compatible storage.

Google Cloud Storage

PuppyGraph supports Google Cloud Storage for Delta Lake.

Configuration Explanation
useComputeEngineServiceAccount Use Compute Engine service account
Service Account Email Service account email address associated with your Google Cloud project
Service Account Private Key Id Unique identifier for the service account's private key
Service Account Private Key Service account's private key for authentication
Impersonation Service Account Impersonation service account

Azure Data Lake Gen2

PuppyGraph supports Azure Data Lake Gen2 for Delta Lake.

Configuration Explanation
Storage Account The name of your Azure v2 Storage account
Shared Key The access key used to authenticate with the storage account
Tenant Id The tenant ID for your organization in Entra ID
Client Id The OAuth 2.0 Client ID
Client Secret The OAuth 2.0 Client secret
Client Endpoint The OAuth 2.0 token endpoint

Azure Blog Storage

PuppyGraph supports Azure Blog Storage for Delta Lake.

Configuration Explanation
Storage Account The name of the Azure Storage account
Shared Key The Shared Key of the Azure Storage account
Storage Container The name of the Storage Container
SAS Token The account or container SAS (Shared Access Signatures) Token.

Azure Onelake Storage

PuppyGraph supports Azure OneLake Storage for Delta Lake.

Configuration Explanation
Client Id The OAuth 2.0 Client ID
Client Secret The OAuth 2.0 Client secret
Client Endpoint The OAuth 2.0 token endpoint

Metadata Cache

Version requirement

Turning off the metadata cache is available in PuppyGraph 1.18.0 and later.

With AWS Glue or Hive Metastore, PuppyGraph caches each Delta table's metadata: its current version, schema and transaction log. Repeated queries then skip re-reading the log from storage, which makes query planning much faster. A cached table version is reloaded at most once a minute, so new commits to a table can take up to a minute to appear in query results.

The same delay applies when a table is dropped and created again under the same name: for up to a minute, queries can still return the old table's rows. Before PuppyGraph 1.16.0, the transaction log was not cached and a recreated table usually showed up at once. To get that behavior back, turn the cache off for the catalog. In the Web UI, clear Cache Delta Lake Metadata under Advanced configuration in the catalog form. You can do this when you create the catalog, or later with Edit on the Catalogs page. In a schema file, add a metadataCache object at the top level of the catalog entry, next to metastore and storage:

{
  "metadataCache": {
    "enabled": "false"
  }
}
Web UI Schema field Default Description
Cache Delta Lake Metadata metadataCache.enabled true Whether to cache Delta table metadata. Set to false to read the table's current metadata and transaction log on every query. A recreated table, or a new commit, then shows up on the next query, at the cost of slower query planning and more requests to the metastore and storage.

The setting applies to Delta Lake catalogs on AWS Glue and Hive Metastore. Unity Catalog ignores it, and the other metadataCache fields described for Iceberg do not apply to Delta Lake. Changing it on an existing catalog takes effect without recreating the catalog or restarting PuppyGraph.

To change it through a schema file on a catalog that already exists, edit its metadataCache object and upload the schema with catalog changes allowed: tick Allow changes to existing catalogs in the upload dialog, or add allowCatalogUpdate=true to the REST request. See How an upload is applied.

Tutorials

Data Type Mapping

Delta Lake Type PuppyGraph Type Description
BOOLEAN Boolean True or false
TINYINT Byte 8-bit signed integers
SMALLINT Short 16-bit signed integers
INT Int 32-bit signed integers
BIGINT Long 64-bit signed integers
FLOAT Float 32-bit IEEE 754 floating point
DOUBLE Double 64-bit IEEE 754 floating point
DECIMAL(P, S) Decimal(P, S) Fixed-point decimal; precision P, scale S
STRING String Arbitrary-length character sequences
BINARY Binary Byte sequence values
DATE Date Calendar date without timezone or time
TIMESTAMP_NTZ DateTime Timestamp, microsecond precision, without timezone
TIMESTAMP DateTime Timestamp, microsecond precision, with timezone1
ARRAY Array<E> A collection of values with some element type
STRUCT Struct<field1 E1[,field2 E2...]> A tuple of typed values
  1. Datetime with timezone will be converted to UTC in PuppyGraph. e.g. 2024-12-01T12:00:00-08:00 is equivalent to 2024-12-01T20:00:00Z and will be stored as 2024-12-01T20:00:00 without timezone.

If you need a data type that is not supported, or one that is not listed here, contact us.

Example Configurations

Catalog Type Storage Type Example Configuration
AWS Glue Amazon S3 #aws-glue-s3
Hive Metastore HDFS #hive-metastore-hdfs
Hive Metastore Azure OneLake #hive-metastore-azure-onelake
Unity Catalog (without credential vending) AWS S3 #unity-catalog-s3-without-credential
Unity Catalog (credential vending) AWS S3/Azure #unity-catalog-credential-vending

Hive Metastore + HDFS

{
  "catalog": [
    {
      "name": "delta_hms_hdfs",
      "type": "deltalake",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "<hive_metastore_uri>"
      }
    }
  ]
}

Hive Metastore + Azure OneLake

{
  "catalog": [
    {
      "name": "delta_hms_onelake",
      "type": "deltalake",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "<hive_metastore_uri>"
      },
      "storage": {
        "type": "AzureDLS2",
        "clientId": "<oauth-2.0-client-id>",
        "clientSecret": "<oauth-2.0-client-secret>",
        "clientEndpoint": "<oauth-2.0-token-endpoint>"
      }
    }
  ]
}

AWS Glue + S3

{
  "catalog": [
    {
      "name": "delta_glue_s3",
      "type": "deltalake",
      "metastore": {
        "type": "glue",
        "useInstanceProfile": "false",
        "region": "<aws_glue_region>",
        "accessKey": "<iam_user_access_key>",
        "secretKey": "<iam_user_secret_key>"
      },
      "storage": {
        "useInstanceProfile": "false",
        "region": "<aws_s3_region>",
        "accessKey": "<iam_user_access_key>",
        "secretKey": "<iam_user_secret_key>",
        "enableSsl": "false"
      }
    }
  ]
}

Unity Catalog + AWS S3 (without credential vending)

{
  "catalog": [
    {
      "name": "unity_s3",
      "type": "deltalake",
      "metastore": {
        "type": "unity",
        "host": "<unity_server_host>",
        "token": "<unity_access_token>",
        "databricksCatalogName": "<catalog_name>"
      },
      "storage": {
        "useInstanceProfile": "false",
        "region": "<aws_s3_region>",
        "accessKey": "<iam_user_access_key>",
        "secretKey": "<iam_user_secret_key>",
        "enableSsl": "false"
      }
    }
  ]
}

Unity Catalog + AWS S3/Azure (credential vending)

{
  "catalog": [
    {
      "name": "unity_credential_vending",
      "type": "deltalake",
      "metastore": {
        "type": "unity",
        "host": "<unity_server_host>",
        "token": "<unity_access_token>",
        "databricksCatalogName": "<catalog_name>"
      }
    }
  ]
}

To change, delete, or troubleshoot this catalog after you save it, see Managing catalogs.