Connecting to Delta Lake
Last updated
Last updated
Both the Delta Lake Data Catalog and Storage are accessible over the network from the PuppyGraph instance
The configuration consists of two parts: Metastore (for Data Catalog) and Data Storage. Please configure them according to you Delta Lake setup.
Configuration | Explanation |
---|---|
Configuration | Explanation |
---|---|
Configuration | Explanation |
---|---|
PuppyGraph supports Amazon S3 (Simple Storage Service) for Delta Lake.
PuppyGraph supports S3 Compatible Storage (e.g. MinIO) for Delta Lake.
In the demo, the delta lake data source stores people and referral information. To query the data as a graph, we model people as vertices and the referral relationship between people as edges.
The demo assumes that PuppyGraph has been deployed at localhost
according to the instruction in Launching PuppyGraph from AWS Marketplace or Launching PuppyGraph in Docker.
In this demo, we use the username puppygraph
and password puppygraph123
.
The demo uses people and referral information as shown above.
Here is the shell command to start a SparkSQL instance for data preparation assuming that the delta lake data are stored on HDFS at 172.31.19.123:9000
and the Hive metastore is at 172.31.31.125:9083
.
Now we can use the following SparkSQL query to create data in the database onhdfs
. The catalog name is puppy_delta
as specified in the command above.
Now the data are ready in Delta Lake. We need a PuppyGraph schema before querying it. Let's create a schema file deltalake.json
:
Here are some notes on this schema:
A catalog catalog_test
is added to specify the remote data source in Delta Lake. Note the hiveMetastoreUrl
field has the same value as the one we used to create data.
The label of the vertices and edges do not have to be the same as the names of corresponding tables in Delta Lake. There is a mappedTableSource
field in each of the vertex and edge types specifying the actual schema (onhdfs
) and table (referral
).
Additionally, the mappedTableSource
marks meta columns in the tables. For example, the fieldsfrom
and to
describe which columns in the table form the endpoints of edges.
PuppyGraph supports query Iceberg / Hudi / Delta Lake with metastore: Hive metastore/ AWS Glue and with storage: HDFS/ AWS S3/ MinIO.
For more catalog parameters details, please refer to Data Lake Catalog.
Now we can upload the schema file deltalake.json
to PuppyGraph with the following shell command, assuming that the PuppyGraph is running on localhost
:
Connecting to PuppyGraph at http://localhost:8081 and start gremlin console from the "Query" section:
Now we have connected to the Gremlin Console. We can query the graph:
Please refer to Data Lake Catalog for detailed parameters for each type of catalog and storage.
Configuration | Explanation |
---|---|
Configuration | Explanation |
---|---|
ID | Age | Name |
---|---|---|
RefID | Source | Referred | Weight |
---|---|---|---|
You can refer to catalog configuration examples we provide: .
Catalog Type | Storage Type | Example Configuration |
---|---|---|
Region
The region of the AWS Glue Data Catalog. Example: us-east-1
. See AWS Glue endpoints and quotas for more details.
Use instance profile
Whether to use role-based authentication (Explicit IAM roles or instance-profile attached)
IAM Role ARN
The ARN of the IAM role for accessing the AWS Glue Data Catalog. Required by authentication with IAM roles.
Access key
The access key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.
Secret key
The secret key of the IAM user for accessing the AWS Glue Data Catalog. Required by authentication with IAM User Access keys.
Hive metastore URI
The URI of your Hive metastore. Format: thrift://<metastore_IP_address>:<metastore_port>
.
Databricks host
The hostname of the Databricks URL. The format is $databricks-customer-prefix.cloud.databricks.com
Databricks token
The access token of the Databricks user. See this page for more details.
Databricks catalog name
The catalog name under the Unity Catalog instance. See this page for more details.
Region
The region of the Amazon S3. Example: us-east-1
. See Amazon Simple Storage Service endpoints and quotas for more details.
Use instance profile
Whether to use role-based authentication (Explicit IAM roles or instance-profile attached).
IAM Role ARN
The ARN of the IAM role for accessing the Amazon S3. Required by authentication with IAM roles.
Access key
The access key of the IAM user for accessing the Amazon S3. Required by authentication with IAM User Access keys.
Secret key
The ARN of the IAM role for accessing the Amazon S3. Required by authentication with IAM User Access keys.
Endpoint
The S3 compatible storage endpoint.
Access key
The access key of an IAM user for accessing the S3 compatible storage.
Secret key
The secret key of an IAM user for accessing the S3 compatible storage.
Enable SSL
Whether to enable SSL connection for accessing the S3 compatible storage.
Enable path style access
Whether to use path-style access method when accessing the S3 compatible storage.
v1
29
marko
v2
27
vadas
e1
v1
v2
0.5
AWS Glue
Amazon S3
Hive Metastore
HDFS