Skip to content

Querying Kerberized Hive Data as a Graph

Summary

In this tutorial, you will:

  • Start a self-contained Hive container that bundles HDFS, the Hive Metastore, HiveServer2, and a Kerberos KDC, then load example data through Beeline.
  • Run PuppyGraph against the Kerberized Hive cluster: copy the Hadoop/Kerberos config files into PuppyGraph, generate a keytab for it, and kinit against the bundled KDC.
  • Connect Hive to PuppyGraph and define a graph schema.
  • Run Cypher and Gremlin queries against the Hive data as a graph.

Self-contained Kerberized Hive

This tutorial uses a prebuilt container that bundles HDFS, Hive Metastore, HiveServer2, and a Kerberos KDC, so the whole secured stack starts with one docker run. PuppyGraph runs in a separate container and shares a Docker volume with the Hive bundle to exchange config files and the keytab.

In real deployments, point PuppyGraph at your existing kerberized Hive Metastore. See Connecting to Hive for the connection reference.

Prerequisites

docker is available on the host:

docker version

See https://www.docker.com/get-started/ if you need to install it.

Accessing the PuppyGraph Web UI requires a browser. The schema upload and query steps also have CLI alternatives via curl and the bundled Gremlin console.

Setup

Create a shared network and volume

▶ Create a Docker network and a named volume. The volume is mounted into both containers so they can exchange Hadoop / Kerberos config files and the keytab:

docker network create puppy-hive
docker volume create puppy-hive

Start the Kerberized Hive bundle

▶ Start the bundled Hive container (the hostname datametaserver.com matters because it appears in Kerberos principals and the metastore URL):

docker run -d --name datametaserver \
  --hostname datametaserver.com \
  -p 88:88 -p 9000:9000 -p 9083:9083 -p 10000:10000 -p 10002:10002 \
  --network puppy-hive \
  -v puppy-hive:/home/share \
  puppygraph/hive:kerberos-hdfs-1.0

Allow a minute or so for HDFS, the Hive Metastore, HiveServer2, and the KDC to finish starting.

Stage Hadoop config and the PuppyGraph keytab

PuppyGraph needs the cluster's core-site.xml, hdfs-site.xml, hive-site.xml, and krb5.conf, plus a keytab for its own Kerberos principal. The Hive bundle already has those XML files baked in; the steps below copy them to the shared volume (so PuppyGraph can mount them) and create a fresh keytab.

▶ Open a shell in the Hive container. All of the remaining steps in this section run inside this shell:

docker exec -it datametaserver bash

▶ Copy the Hadoop config files and krb5.conf into the shared volume:

cp /usr/local/hive/conf/hive-site.xml      /home/share/
cp /usr/local/hadoop/etc/hadoop/core-site.xml /home/share/
cp /usr/local/hadoop/etc/hadoop/hdfs-site.xml /home/share/
cp /etc/krb5.conf /home/share/

▶ Rewrite the shared krb5.conf so:

  1. The KDC hostname matches the container hostname (datametaserver.com) that PuppyGraph will resolve over the Docker network.
  2. The ticket cache prefix matches what PuppyGraph reads (/tmp/krb5cc_<uid> instead of /tmp/krb5cc_cli_<uid>).
sed -i 's/localhost/datametaserver.com/g' /home/share/krb5.conf
sed -i 's/krb5cc_cli_/krb5cc_/' /home/share/krb5.conf

▶ Generate a Kerberos principal for PuppyGraph and dump its keytab onto the shared volume:

kadmin.local -q "add_principal -randkey puppygraph/puppygraph.com"
kadmin.local -q "ktadd -k /home/share/puppygraph.keytab puppygraph/puppygraph.com@HADOOPKRB"

Why -randkey?

PuppyGraph authenticates with the keytab, not a password. -randkey makes kadmin.local create the principal with a random key (no interactive password prompt); the next ktadd line writes that same key into the keytab.

After this, /home/share contains:

core-site.xml  hdfs-site.xml  hive-site.xml  krb5.conf  puppygraph.keytab

Data Preparation

Still inside the datametaserver container:

▶ Start a Beeline shell:

beeline -u jdbc:hive2://

▶ Paste the following SQL into the Beeline prompt to create the database and insert data:

modern.sql 
CREATE DATABASE modern_demo
  LOCATION "hdfs://datametaserver.com:9000/user/hive/warehouse/modern_demo.db";

CREATE TABLE modern_demo.software (
  id   string,
  name string,
  lang string
) LOCATION "hdfs://datametaserver.com:9000/user/hive/warehouse/modern_demo.db/software";
INSERT INTO modern_demo.software VALUES
  ('v3', 'lop',    'java'),
  ('v5', 'ripple', 'java');

CREATE TABLE modern_demo.person (
  id   string,
  name string,
  age  int
) LOCATION "hdfs://datametaserver.com:9000/user/hive/warehouse/modern_demo.db/person";
INSERT INTO modern_demo.person VALUES
  ('v1', 'marko', 29),
  ('v2', 'vadas', 27),
  ('v4', 'josh',  32),
  ('v6', 'peter', 35);

CREATE TABLE modern_demo.created (
  id      string,
  from_id string,
  to_id   string,
  weight  double
) LOCATION "hdfs://datametaserver.com:9000/user/hive/warehouse/modern_demo.db/created";
INSERT INTO modern_demo.created VALUES
  ('e9',  'v1', 'v3', 0.4),
  ('e10', 'v4', 'v5', 1.0),
  ('e11', 'v4', 'v3', 0.4),
  ('e12', 'v6', 'v3', 0.2);

CREATE TABLE modern_demo.knows (
  id      string,
  from_id string,
  to_id   string,
  weight  double
) LOCATION "hdfs://datametaserver.com:9000/user/hive/warehouse/modern_demo.db/knows";
INSERT INTO modern_demo.knows VALUES
  ('e7', 'v1', 'v2', 0.5),
  ('e8', 'v1', 'v4', 1.0);

The above creates four Hive tables under the modern_demo database backed by HDFS.

id name age
v1 marko 29
v2 vadas 27
v4 josh 32
v6 peter 35
id name lang
v3 lop java
v5 ripple java
id from_id to_id weight
e9 v1 v3 0.4
e10 v4 v5 1.0
e11 v4 v3 0.4
e12 v6 v3 0.2
id from_id to_id weight
e7 v1 v2 0.5
e8 v1 v4 1.0

▶ Type !exit to leave Beeline, then exit to leave the container shell.

Start PuppyGraph and acquire a Kerberos ticket

▶ Start PuppyGraph with the shared volume mounted and Hadoop/Kerberos config paths set:

docker run -d --name puppygraph \
  --hostname puppygraph.com \
  -p 8081:8081 -p 8182:8182 -p 7687:7687 \
  -e PUPPYGRAPH_USERNAME=puppygraph \
  -e PUPPYGRAPH_PASSWORD=puppygraph123 \
  -e AUTHENTICATION_HADOOP_CORESITEXMLPATH=/home/share/core-site.xml \
  -e AUTHENTICATION_HADOOP_HIVESITEXMLPATH=/home/share/hive-site.xml \
  -e AUTHENTICATION_HADOOP_HDFSSITEXMLPATH=/home/share/hdfs-site.xml \
  -e AUTHENTICATION_KERBEROS_CONFIGPATH=/home/share/krb5.conf \
  --network puppy-hive \
  -v puppy-hive:/home/share \
  --pull=always puppygraph/puppygraph:latest

▶ Inside the PuppyGraph container, kinit against the bundled KDC using the keytab so PuppyGraph has a valid Kerberos ticket cached at /tmp/krb5cc_0:

docker exec -uroot -it puppygraph bash
export KRB5_CONFIG=/home/share/krb5.conf
kinit -kt /home/share/puppygraph.keytab puppygraph/puppygraph.com@HADOOPKRB
klist
exit

klist should report Default principal: puppygraph/puppygraph.com@HADOOPKRB with a krbtgt/HADOOPKRB@HADOOPKRB entry. Refresh the ticket cache before it expires (10 hours by default in this image).

Modeling a Graph

We model the data as the TinkerPop modern graph: two node types (person, software) and two edge types (knows, created).

Modern Graph
Modern Graph

▶ First, log into the PuppyGraph Web UI at http://localhost:8081 with the credentials configured above:

Field Value
Username puppygraph
Password puppygraph123

There are two ways to define the schema in PuppyGraph: build it interactively in the Schema Builder, or upload a JSON file directly. Pick whichever you prefer; both produce the same graph.

Build the graph in the Schema Builder

The Schema Builder is the visual editor in the PuppyGraph Web UI for adding catalogs, nodes, and edges step by step. It's the recommended path when you're modeling a graph for the first time or want to inspect what each click produces. For a deeper visual walkthrough of every dialog and field, see Modeling a Graph through the Schema Builder. The summary below covers what's needed to build the modern graph against the kerberized Hive metastore.

Connecting to Hive

▶ Click Create Catalog, then expand Data Lakes and pick Apache Hive.

▶ Fill in the connection form:

Field Value
Catalog name hive_data
Metastore type Hive metastore
Hive metastore URI thrift://datametaserver.com:9083
Storage type Get from metastore

Kerberos auth uses the keytab and ticket cache that PuppyGraph already loaded via kinit. No extra credentials in the form.

Hive catalog form
Hive catalog form

▶ Click Create Catalog.

Adding nodes and edges

▶ Add the software and person nodes, then the created and knows edges, the same way as in the Iceberg tutorial.

Select the software table for a node
Select the software table for a node
Configure the software node
Configure the software node
Configure the knows edge
Configure the knows edge
Completed modern graph schema
Completed modern graph schema

The catalog tree shows the source under hive_data > modern_demo. For each edge, set the From / To Node, map from_id and to_id as the FROM / TO Select Column, and assign id as the edge identifier.

Upload a schema file

If you've already built the graph in the Schema Builder above, you can skip this section. The resulting schema is the same.

This method writes the full schema to a JSON file and uploads it directly. It's useful when you already have a schema for an environment and want to recreate it elsewhere (e.g. for CI, scripted setup, or copy-pasting between PuppyGraph instances).

▶ Create a file schema.json with the following content:

schema.json 
{
  "catalog": [
    {
      "name": "hive_data",
      "type": "hive",
      "metastore": {
        "type": "HMS",
        "hiveMetastoreUrl": "thrift://datametaserver.com:9083"
      }
    }
  ],
  "node": [
    {
      "label": "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "hive_data",
          "schema": "modern_demo",
          "table": "software",
          "mappedField": [
            { "sourceFieldName": "id",   "targetFieldName": "id"   },
            { "sourceFieldName": "name", "targetFieldName": "name" },
            { "sourceFieldName": "lang", "targetFieldName": "lang" }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "lang", "type": "STRING" }
      ]
    },
    {
      "label": "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "hive_data",
          "schema": "modern_demo",
          "table": "person",
          "mappedField": [
            { "sourceFieldName": "id",   "targetFieldName": "id"   },
            { "sourceFieldName": "name", "targetFieldName": "name" },
            { "sourceFieldName": "age",  "targetFieldName": "age"  }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "age",  "type": "INT"    }
      ]
    }
  ],
  "edge": [
    {
      "label":         "created",
      "fromNodeLabel": "person",
      "toNodeLabel":   "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "hive_data",
          "schema": "modern_demo",
          "table": "created",
          "mappedField": [
            { "sourceFieldName": "id",      "targetFieldName": "id"      },
            { "sourceFieldName": "from_id", "targetFieldName": "from_id" },
            { "sourceFieldName": "to_id",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "weight",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    },
    {
      "label":         "knows",
      "fromNodeLabel": "person",
      "toNodeLabel":   "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "hive_data",
          "schema": "modern_demo",
          "table": "knows",
          "mappedField": [
            { "sourceFieldName": "id",      "targetFieldName": "id"      },
            { "sourceFieldName": "from_id", "targetFieldName": "from_id" },
            { "sourceFieldName": "to_id",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "weight",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    }
  ]
}

▶ In the Web UI, click Graph in the sidebar, then Upload Schema, and select schema.json.

Upload via CLI

You can also POST the schema directly:

curl -X POST -H "content-type: application/json" \
  --data-binary @./schema.json \
  --user "puppygraph:puppygraph123" \
  http://localhost:8081/schema

Querying the Graph

In the PuppyGraph Web UI, click Query in the sidebar. You can run graph queries in either Cypher or Gremlin.

The following query answers "What software was created by people that marko knows?"

MATCH path = (p:person)-[:knows]->()-[:created]->()
WHERE p.name = 'marko'
RETURN path;

g.V().hasLabel('person').has('name', 'marko')
  .out('knows').out('created').path()

There are two paths in the result: marko knows josh, who created lop and ripple.

Cleanup

▶ Stop and remove the containers, network, and volume:

docker stop puppygraph datametaserver
docker rm puppygraph datametaserver
docker network rm puppy-hive
docker volume rm puppy-hive