Querying Nessie and MinIO AIStor Data as a Graph with TLS

Archived PuppyGraph 0.x documentation

This is the archived 0.x documentation. For new deployments, use puppygraph/puppygraph:latest (PuppyGraph 1.x) and the current docs. The 0.x schemas and UI described here don't apply to 1.x; see Migrating from v0.

Summary

In this tutorial, you will:

Create a Nessie-backed Apache Iceberg data lake and load it with example data;
Deploy TLS-enabled MinIO AIStor, Nessie, Spark-Iceberg, and PuppyGraph services using Docker Compose.
Configure each service to communicate securely via HTTPS.
Use PuppyGraph to query and visualize Nessie data as a graph.

Note: For the non-TLS version, see Querying Nessie Data as a Graph.

Prerequisites

MinIO AIStor license

Before continuing, obtain a valid MinIO AIStor license, such as the AIStor Free Tier license for a single-node deployment. Save the license as minio.license in your working directory, where you will create docker-compose.yaml below.

This tutorial requires a valid AIStor license to create the bucket and write data. Docker Compose mounts the license file read-only into the AIStor container.

Docker

Docker is required to run the PuppyGraph server. You can download Docker from here.

Please ensure that docker compose is available. The installation can be verified by running:

docker compose version

OpenSSL and keytool

OpenSSL and the Java keytool utility are needed to generate TLS certificates and truststores.

For Ubuntu, you can install it using the following commands:

sudo apt update
sudo apt install openssl default-jdk

Accessing the PuppyGraph Web UI requires a browser. However, the tutorial offers alternative instructions for those who wish to exclusively use the CLI.

TLS Certificate Setup

To enable secure communications, we generate certificates for MinIO AIStor and Nessie.

1. Create a Shell Script

Create a file named generate_crt.sh with the following content.

generate_crt.sh

#!/usr/bin/env bash
set -euo pipefail

# ----------------------------
# Configuration parameters
# ----------------------------
# Base directories
CERT_ROOT="${CERT_ROOT:-./certs}"
CA_DIR="${CA_DIR:-${CERT_ROOT}/ca}"
MINIO_DIR="${MINIO_DIR:-${CERT_ROOT}/minio}"
NESSIE_DIR="${NESSIE_DIR:-${CERT_ROOT}/nessie}"

# Validity periods (in days)
CA_DAYS="${CA_DAYS:-3650}"
SERVER_DAYS="${SERVER_DAYS:-365}"

# Password settings
CA_PASSWORD="${CA_PASSWORD:-ca_password}"
NESSIE_PASSWORD="${NESSIE_PASSWORD:-nessie_password}"

# Common Names
CA_CN="${CA_CN:-MyRootCA}"
MINIO_CN="${MINIO_CN:-minio}"
NESSIE_CN="${NESSIE_CN:-nessie}"

# Subject Alternative Names
MINIO_SANS=( ${MINIO_SANS:-minio localhost 127.0.0.1} )
NESSIE_SANS=( ${NESSIE_SANS:-nessie localhost 127.0.0.1} )

# Helper: build SAN entries for OpenSSL config
build_san() {
  local sans=("$@")
  local idx=1
  for san in "${sans[@]}"; do
    if [[ $san =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
      echo "IP.$idx = $san"
    else
      echo "DNS.$idx = $san"
    fi
    ((idx++))
  done
}

# Clean and prepare directories
rm -rf "$CERT_ROOT"
mkdir -p "$CA_DIR" "$MINIO_DIR" "$NESSIE_DIR"

# 1. Generate Root CA
echo "[1/4] Generating Root CA..."
openssl genrsa -out "$CA_DIR/ca.key" 2048
cat > "$CA_DIR/ca.cnf" <<EOL
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
prompt = no

[req_distinguished_name]
C = US
ST = State
L = City
O = Organization
OU = Unit
CN = $CA_CN

[v3_ca]
basicConstraints = critical, CA:TRUE
keyUsage = critical, keyCertSign, cRLSign
EOL
openssl req -new -x509 -days "$CA_DAYS" \
  -key "$CA_DIR/ca.key" \
  -out "$CA_DIR/ca.crt" \
  -config "$CA_DIR/ca.cnf"

# 2. Generate MinIO AIStor Server Certificate
echo "[2/4] Generating MinIO AIStor certificate..."
cat > "$MINIO_DIR/minio.cnf" <<EOL
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = State
L = City
O = Organization
OU = Unit
CN = $MINIO_CN

[v3_req]
subjectAltName = @alt_names

[alt_names]
$(build_san "${MINIO_SANS[@]}")
EOL
openssl genrsa -out "$MINIO_DIR/private.key" 2048
openssl req -new -key "$MINIO_DIR/private.key" \
  -out "$MINIO_DIR/minio.csr" \
  -config "$MINIO_DIR/minio.cnf"
openssl x509 -req -days "$SERVER_DAYS" \
  -in "$MINIO_DIR/minio.csr" \
  -CA "$CA_DIR/ca.crt" \
  -CAkey "$CA_DIR/ca.key" \
  -CAcreateserial \
  -out "$MINIO_DIR/public.crt" \
  -extfile "$MINIO_DIR/minio.cnf" \
  -extensions v3_req

# 3. Create CA truststore (PKCS12)
echo "[3/4] Creating CA truststore (PKCS12)..."
keytool -import -trustcacerts -noprompt \
  -alias root-ca \
  -file "$CA_DIR/ca.crt" \
  -keystore "$CA_DIR/ca-truststore.p12" \
  -storetype PKCS12 \
  -storepass "$CA_PASSWORD"

# 4. Generate Nessie Server Certificate
echo "[4/4] Generating Nessie certificate..."
cat > "$NESSIE_DIR/nessie.cnf" <<EOL
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = State
L = City
O = Organization
OU = Unit
CN = $NESSIE_CN

[v3_req]
subjectAltName = @alt_names
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
$(build_san "${NESSIE_SANS[@]}")
EOL
openssl genrsa -out "$NESSIE_DIR/private.key" 2048
openssl req -new -key "$NESSIE_DIR/private.key" \
  -out "$NESSIE_DIR/nessie.csr" \
  -config "$NESSIE_DIR/nessie.cnf"
openssl x509 -req -days "$SERVER_DAYS" \
  -in "$NESSIE_DIR/nessie.csr" \
  -CA "$CA_DIR/ca.crt" \
  -CAkey "$CA_DIR/ca.key" \
  -CAcreateserial \
  -out "$NESSIE_DIR/nessie.crt" \
  -extfile "$NESSIE_DIR/nessie.cnf" \
  -extensions v3_req

# Create Nessie keystore (PKCS12)
echo "Creating Nessie keystore (PKCS12)..."
openssl pkcs12 -export \
  -in "$NESSIE_DIR/nessie.crt" \
  -inkey "$NESSIE_DIR/private.key" \
  -name "nessie-cert" \
  -out "$NESSIE_DIR/nessie.p12" \
  -passout "pass:$NESSIE_PASSWORD"

# Tighten permissions
chmod 600 "$CA_DIR/ca.key" "$MINIO_DIR/private.key" "$NESSIE_DIR/private.key"
chmod 644 "$NESSIE_DIR/nessie.p12"

echo -e "\nCertificate generation complete!"
echo "Generated files:"
echo "  CA:                 $CA_DIR/ca.crt"
echo "  Truststore (PKCS12): $CA_DIR/ca-truststore.p12"
echo "  MinIO AIStor cert:         $MINIO_DIR/public.crt"
echo "  MinIO AIStor key:          $MINIO_DIR/private.key"
echo "  Nessie cert:        $NESSIE_DIR/nessie.crt"
echo "  Nessie key:         $NESSIE_DIR/private.key"
echo "  Nessie keystore:    $NESSIE_DIR/nessie.p12"

2. Generate the Certificate Files

Run the following commands to generate the server certificates and keys signed by your root CA

chmod +x generate_crt.sh

./generate_crt.sh

3. Prepare TLS Certificates for Docker Volume Mounts

mkdir -p ./minio_certs
cp ./certs/minio/public.crt ./minio_certs/
cp ./certs/minio/private.key ./minio_certs/

cp ./certs/ca/ca.crt ./minio_certs/
# The AIStor image runs as root and can read this private directory.
chmod 700 ./minio_certs
chmod 600 ./minio_certs/private.key

mkdir -p ./nessie_certs
cp ./certs/nessie/nessie.p12 ./nessie_certs/
cp ./certs/ca/ca-truststore.p12 ./nessie_certs/

mkdir -p ./spark_certs
cp ./certs/ca/ca-truststore.p12 ./spark_certs/

mkdir -p ./puppygraph
cp ./certs/ca/ca.crt ./puppygraph/MyRootCA.crt
cp ./certs/minio/public.crt ./puppygraph/minio.crt
cp ./certs/nessie/nessie.crt ./puppygraph/nessie.crt

Deployment

1. Create the Docker Compose File

Create a file named docker-compose.yaml with the content below. Notice that TLS parameters are integrated into the MinIO AIStor and Nessie service configurations.

docker-compose.yaml

services:
  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    networks:
      iceberg_net:
    depends_on:
      nessie:
        condition: service_started
      minio-init:
        condition: service_completed_successfully
    volumes:
      - ./spark_certs:/spark_certs
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - JAVA_TOOL_OPTIONS=-Djavax.net.ssl.trustStore=/spark_certs/ca-truststore.p12 -Djavax.net.ssl.trustStorePassword=ca_password
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001

  minio:
    image: quay.io/minio/aistor/minio:latest
    container_name: minio
    command: ["server", "/data", "--console-address", ":9001", "--certs-dir", "/minio_certs", "--license", "/minio.license"]
    environment: &minio-credentials
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    networks:
      - iceberg_net
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./minio_certs:/minio_certs:ro
      - type: bind
        source: ./minio.license
        target: /minio.license
        read_only: true
        bind:
          create_host_path: false
    healthcheck:
      test: ["CMD", "curl", "-fsS", "--cacert", "/minio_certs/ca.crt", "https://localhost:9000/minio/health/ready"]
      interval: 2s
      timeout: 5s
      retries: 30
      start_period: 10s
  minio-init:
    image: quay.io/minio/aistor/minio:latest
    depends_on:
      minio:
        condition: service_healthy
    networks:
      - iceberg_net
    environment: *minio-credentials
    volumes:
      - ./certs/ca/ca.crt:/ca.crt:ro
    entrypoint:
      - /bin/sh
      - -ec
      - |
        curl -fsS --cacert /ca.crt --aws-sigv4 aws:amz:us-east-1:s3 --user "$${MINIO_ROOT_USER}:$${MINIO_ROOT_PASSWORD}" \
          --head https://minio:9000/my-bucket >/dev/null 2>&1 ||
        curl -fsS --cacert /ca.crt --aws-sigv4 aws:amz:us-east-1:s3 --user "$${MINIO_ROOT_USER}:$${MINIO_ROOT_PASSWORD}" \
          -X PUT https://minio:9000/my-bucket
  nessie:
    depends_on:
      minio-init:
        condition: service_completed_successfully
    image: ghcr.io/projectnessie/nessie:0.104.1
    container_name: nessie
    networks:
      iceberg_net:
    ports:
      - 19121:19121
    volumes:
      - ./nessie_certs:/nessie_certs
    environment:
      - nessie.version.store.type=IN_MEMORY
      - nessie.catalog.default-warehouse=warehouse
      - nessie.catalog.warehouses.warehouse.location=s3a://my-bucket/
      - nessie.catalog.service.s3.default-options.region=us-east-1
      - nessie.catalog.service.s3.default-options.endpoint=https://minio:9000
      - nessie.catalog.service.s3.default-options.path-style-access=true
      - nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key
      - nessie.catalog.secrets.access-key.name=admin
      - nessie.catalog.secrets.access-key.secret=password
      - nessie.server.authentication.enabled=false
      - quarkus.http.ssl-port=19121
      - quarkus.http.ssl.certificate.key-store-file=/nessie_certs/nessie.p12
      - quarkus.http.ssl.certificate.key-store-password=nessie_password
      - quarkus.http.ssl.certificate.key-store-type=PKCS12
      - JAVA_TOOL_OPTIONS=-Djavax.net.ssl.trustStore=/nessie_certs/ca-truststore.p12 -Djavax.net.ssl.trustStorePassword=ca_password

  puppygraph:
    image: puppygraph/puppygraph:stable
    container_name: puppygraph
    networks:
      iceberg_net:
    environment:
      - PUPPYGRAPH_USERNAME=puppygraph
      - PUPPYGRAPH_PASSWORD=puppygraph123
      - CERTIFICATE_BASEPATH=/home/ubuntu/tls-certs
    ports:
      - "8081:8081"
      - "8182:8182"
      - "7687:7687"
    volumes:
      - ./puppygraph:/home/ubuntu/tls-certs
    depends_on:
      - spark-iceberg

networks:
  iceberg_net:

2. Start the Services

Then run the following command to start Nessie-backed Iceberg services and PuppyGraph:

docker compose up -d

[+] Running 6/6
 ✔ Network   iceberg_net       Created
 ✔ Container nessie         Started
 ✔ Container minio         Started
 ✔ Container minio-init    Exited
 ✔ Container spark-iceberg  Started
 ✔ Container puppygraph     Started

The minio-init service uses the same root credentials as AIStor and creates the bucket with a signed S3 request before the catalog starts. The bucket stays private; the clients use the credentials configured above.

Data Preparation

This tutorial is designed to be comprehensive and standalone, so it includes steps to populate data in Nessie. In practical scenarios, PuppyGraph can query data directly from your existing Nessie tables.

Run the following command to start a Spark-SQL shell connected to Nessie.

The command sets the MinIO AIStor endpoint and enables path-style access, overriding the endpoint bundled in the Spark image.

docker exec -it spark-iceberg spark-sql \
  --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.demo.uri=https://nessie:19121/iceberg/ \
  --conf spark.sql.catalog.demo.warehouse=s3a://my-bucket/ \
  --conf spark.sql.catalog.demo.type=rest \
  --conf spark.sql.catalog.demo.s3.endpoint=https://minio:9000 \
  --conf spark.sql.catalog.demo.s3.path-style-access=true

The shell will be like this:

spark-sql ()>

Then execute the following SQL statements in the shell to create tables and insert data:

CREATE DATABASE demo.modern;

CREATE EXTERNAL TABLE demo.modern.person (
  id string,
  name string,
  age int
) USING iceberg;

INSERT INTO demo.modern.person VALUES
  ('v1', 'marko', 29),
  ('v2', 'vadas', 27),
  ('v4', 'josh', 32),
  ('v6', 'peter', 35);

CREATE EXTERNAL TABLE demo.modern.software (
  id string,
  name string,
  lang string
) USING iceberg;

INSERT INTO demo.modern.software VALUES
  ('v3', 'lop', 'java'),
  ('v5', 'ripple', 'java');

CREATE EXTERNAL TABLE demo.modern.created (
  id string,
  from_id string,
  to_id string,
  weight double
) USING iceberg;

INSERT INTO demo.modern.created VALUES
  ('e9', 'v1', 'v3', 0.4),
  ('e10', 'v4', 'v5', 1.0),
  ('e11', 'v4', 'v3', 0.4),
  ('e12', 'v6', 'v3', 0.2);

CREATE EXTERNAL TABLE demo.modern.knows (
  id string,
  from_id string,
  to_id string,
  weight double
) USING iceberg;

INSERT INTO demo.modern.knows VALUES
  ('e7', 'v1', 'v2', 0.5),
  ('e8', 'v1', 'v4', 1.0);

The above SQL creates the following tables:

personsoftwareknowscreated

id	name	age
v1	marko	29
v2	vadas	27
v4	josh	32
v6	peter	35

id	name	lang
v3	lop	java
v5	ripple	java

id	from_id	to_id	weight
e7	v1	v2	0.5
e8	v1	v4	1.0

id	from_id	to_id	weight
e9	v1	v3	0.4
e10	v4	v5	1.0
e11	v4	v3	0.4
e12	v6	v3	0.2

When finished, exit the Spark-SQL shell by entering:

quit;

Modeling the Graph

Step 1: Connecting to Nessie

Log into PuppyGraph Web UI at http://localhost:8081 with username puppygraph and password puppygraph123..

Click on Create graph schema to create a new graph schema.

Fill in the fields as follows.

Parameter	Value
Catalog type	`Apache Iceberg`
Catalog name	Some name for the catalog as you like.
Metastore Type	`Iceberg REST`
RestUri	`https://nessie:19121/iceberg` (note the HTTPS).
Warehouse	Same as nessie.catalog.warehouses.warehouse.location in `docker-compose.yaml`.
Storage type	`S3 Compatible`
Endpoint	`https://minio:9000` (note the HTTPS). Same as nessie.catalog.service.s3.default-options.endpoint in `docker-compose.yaml`.
Access key	Same as AWS_ACCESS_KEY_ID in `docker-compose.yaml`
Secret key	Same as AWS_SECRET_ACCESS_KEY in `docker-compose.yaml`
Enable SSL	`true`
Enable path style access	`true`

Click on Save, then Click on Submit to connect to Nessie.

Step 2: Building the Graph Schema

In the Schema Builder, add the first node (vertex) to the graph from the table person.

After that use the Auto Suggestion to create other nodes and edges. Select person as the start node (vertex) and add the auto suggested nodes and edges.

The graph schema should look like this: Submit the schema to create the graph.

Step 3: Querying the Graph

PuppyGraph provides a Dashboard that gives the summary of the graph.

Use the Interactive Query UI to further explore the graph by sending queries.

Cleaning up

Run the following command to shut down and remove the services:

docker compose down --volumes --remove-orphans