Skip to content

Querying Elasticsearch Data as a Graph

Summary

In this tutorial, you will:

  • Start a PuppyGraph container alongside an Elasticsearch container and load example data.
  • Connect Elasticsearch to PuppyGraph and define a graph schema.
  • Run Cypher and Gremlin queries against the Elasticsearch data as a graph.

Self-contained Elasticsearch Data

This tutorial bundles an Elasticsearch container and seeds it with the TinkerPop modern graph sample data.

In real deployments, PuppyGraph queries your existing Elasticsearch clusters directly. See Connecting to Elasticsearch for the connection reference.

Prerequisites

Please ensure that docker compose is available. The installation can be verified by running:

docker compose version

See https://docs.docker.com/compose/install/ for Docker Compose installation instructions and https://www.docker.com/get-started/ for more details on Docker.

Accessing the PuppyGraph Web UI requires a browser. The schema upload and query steps also have CLI alternatives via curl and the bundled Gremlin console.

Setup

Deployment

▶ Create a file docker-compose.yaml with the following content:

docker-compose.yaml 
version: "3"
services:
  puppygraph:
    image: puppygraph/puppygraph:latest
    pull_policy: always
    container_name: puppygraph
    environment:
      - PUPPYGRAPH_USERNAME=puppygraph
      - PUPPYGRAPH_PASSWORD=puppygraph123
    networks:
      - es_net
    ports:
      - "8081:8081"
      - "8182:8182"
      - "7687:7687"
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.2.4
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - ELASTIC_PASSWORD=es_password
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    networks:
      - es_net
    ports:
      - "9200:9200"
networks:
  es_net:
    name: puppy-es

The Elasticsearch container runs a single node with security enabled, so all requests authenticate with the built-in elastic user and the password set by ELASTIC_PASSWORD. This matches how PuppyGraph connects to Elasticsearch in production: basic authentication with a username and password. TLS on the HTTP layer is disabled to keep this tutorial self-contained; see Connecting to Elasticsearch for notes on TLS.

Default passwords

The compose file ships with default credentials for convenience. Change ELASTIC_PASSWORD and PUPPYGRAPH_PASSWORD before running on a publicly accessible machine.

▶ Start the stack:

docker compose up -d

[+] Running 3/3
 ✔ Network puppy-es           Created                                      0.1s
 ✔ Container elasticsearch    Started                                      0.6s
 ✔ Container puppygraph       Started                                      0.7s

Elasticsearch takes a short while to become ready. Verify it responds before continuing:

curl -u elastic:es_password http://localhost:9200

The response is a JSON document with the cluster name and version 9.2.4.

Data Preparation

The tutorial data consists of four Elasticsearch indices modeling people, software, and the relationships between them. Each relationship index stores one document per edge with from_id and to_id fields referencing the connected documents.

▶ Create the four indices with explicit field mappings:

Create indices 
curl -u elastic:es_password -H 'Content-Type: application/json' \
  -X PUT 'http://localhost:9200/person' -d '
{
  "mappings": {
    "properties": {
      "id":   { "type": "keyword" },
      "name": { "type": "text" },
      "age":  { "type": "integer" }
    }
  }
}'

curl -u elastic:es_password -H 'Content-Type: application/json' \
  -X PUT 'http://localhost:9200/software' -d '
{
  "mappings": {
    "properties": {
      "id":   { "type": "keyword" },
      "name": { "type": "text" },
      "lang": { "type": "text" }
    }
  }
}'

curl -u elastic:es_password -H 'Content-Type: application/json' \
  -X PUT 'http://localhost:9200/knows' -d '
{
  "mappings": {
    "properties": {
      "id":      { "type": "keyword" },
      "from_id": { "type": "keyword" },
      "to_id":   { "type": "keyword" },
      "weight":  { "type": "double" }
    }
  }
}'

curl -u elastic:es_password -H 'Content-Type: application/json' \
  -X PUT 'http://localhost:9200/created' -d '
{
  "mappings": {
    "properties": {
      "id":      { "type": "keyword" },
      "from_id": { "type": "keyword" },
      "to_id":   { "type": "keyword" },
      "weight":  { "type": "double" }
    }
  }
}'

▶ Load the documents through the bulk API:

Load data 
curl -u elastic:es_password -H 'Content-Type: application/x-ndjson' \
  -X POST 'http://localhost:9200/person/_bulk?refresh=wait_for' --data-binary @- <<'EOF'
{ "index": { "_id": "1" }}
{ "id": "v1", "name": "marko", "age": 29 }
{ "index": { "_id": "2" }}
{ "id": "v2", "name": "vadas", "age": 27 }
{ "index": { "_id": "3" }}
{ "id": "v4", "name": "josh", "age": 32 }
{ "index": { "_id": "4" }}
{ "id": "v6", "name": "peter", "age": 35 }
EOF

curl -u elastic:es_password -H 'Content-Type: application/x-ndjson' \
  -X POST 'http://localhost:9200/software/_bulk?refresh=wait_for' --data-binary @- <<'EOF'
{ "index": { "_id": "1" }}
{ "id": "v3", "name": "lop", "lang": "java" }
{ "index": { "_id": "2" }}
{ "id": "v5", "name": "ripple", "lang": "java" }
EOF

curl -u elastic:es_password -H 'Content-Type: application/x-ndjson' \
  -X POST 'http://localhost:9200/knows/_bulk?refresh=wait_for' --data-binary @- <<'EOF'
{ "index": { "_id": "1" }}
{ "id": "e7", "from_id": "v1", "to_id": "v2", "weight": 0.5 }
{ "index": { "_id": "2" }}
{ "id": "e8", "from_id": "v1", "to_id": "v4", "weight": 1.0 }
EOF

curl -u elastic:es_password -H 'Content-Type: application/x-ndjson' \
  -X POST 'http://localhost:9200/created/_bulk?refresh=wait_for' --data-binary @- <<'EOF'
{ "index": { "_id": "1" }}
{ "id": "e9", "from_id": "v1", "to_id": "v3", "weight": 0.4 }
{ "index": { "_id": "2" }}
{ "id": "e10", "from_id": "v4", "to_id": "v5", "weight": 1.0 }
{ "index": { "_id": "3" }}
{ "id": "e11", "from_id": "v4", "to_id": "v3", "weight": 0.4 }
{ "index": { "_id": "4" }}
{ "id": "e12", "from_id": "v6", "to_id": "v3", "weight": 0.2 }
EOF

The loaded documents look like this:

id name age
v1 marko 29
v2 vadas 27
v4 josh 32
v6 peter 35
id name lang
v3 lop java
v5 ripple java
id from_id to_id weight
e7 v1 v2 0.5
e8 v1 v4 1.0
id from_id to_id weight
e9 v1 v3 0.4
e10 v4 v5 1.0
e11 v4 v3 0.4
e12 v6 v3 0.2

In PuppyGraph, every Elasticsearch index appears as a table under a single database named default_db, with one row per document and one column per mapped field.

Modeling a Graph

We model the data as the TinkerPop modern graph: two node types (person, software) and two edge types (knows, created).

Modern Graph
Modern Graph

▶ First, log into the PuppyGraph Web UI at http://localhost:8081 with the credentials configured above:

Field Value
Username puppygraph
Password puppygraph123

There are two ways to define the schema in PuppyGraph: build it interactively in the Schema Builder, or upload a JSON file directly. Pick whichever you prefer; both produce the same graph.

Build the graph in the Schema Builder

The Schema Builder is the visual editor in the PuppyGraph Web UI for adding catalogs, nodes, and edges step by step. It's the recommended path when you're modeling a graph for the first time or want to inspect what each click produces. For a deeper visual walkthrough of every dialog and field, see Modeling a Graph through the Schema Builder. The summary below covers what's needed to build the modern graph against this tutorial's Elasticsearch data.

Connecting to Elasticsearch

▶ Click Create Catalog, then expand NoSQL & Search and pick Elasticsearch.

▶ Fill in the connection form:

Field Value
Catalog name es_data
Username elastic
Password es_password
Server Hosts http://elasticsearch:9200
Elasticsearch catalog form
Elasticsearch catalog form

▶ Click Create Catalog.

Adding nodes

▶ Click Add Node in the toolbar. The Select Table for Node dialog opens. Expand es_data then default_db, pick software, then click Next.

Select the software table for a node
Select the software table for a node

▶ In the Add Node wizard, click Add to ID and select id from the dropdown. The wizard moves id into ID Columns, leaving name and lang as attributes. Click Next, leave Enable Local Replication off, then click Add Node.

Configure the software node
Configure the software node

▶ Repeat for person. The flow is the same: click Add Node, pick the table, click Next, assign id to ID Columns, leave replication off, click Add Node.

Adding edges

▶ Click Add Edge in the toolbar, pick created from the catalog tree, then click Next.

▶ In the Add Edge wizard, set:

Field Value
From Node person
To Node software
FROM Select Column from_id
TO Select Column to_id
Configure the created edge
Configure the created edge

▶ Click Add to ID and select id to set the edge identifier. Click Next, leave Enable Local Replication off, then click Add Edge.

▶ Repeat for knows with both From Node and To Node set to person. The other settings are identical to created.

Completed modern graph schema
Completed modern graph schema

Upload a schema file

If you've already built the graph in the Schema Builder above, you can skip this section. The resulting schema is the same.

This method writes the full schema to a JSON file and uploads it directly. It's useful when you already have a schema for an environment and want to recreate it elsewhere (e.g. for CI, scripted setup, or copy-pasting between PuppyGraph instances).

▶ Create a file schema.json with the following content:

schema.json 
{
  "catalog": [
    {
      "name": "es_data",
      "type": "elasticsearch",
      "elasticSearch": {
        "hosts": ["http://elasticsearch:9200"],
        "username": "elastic",
        "password": "es_password"
      }
    }
  ],
  "node": [
    {
      "label": "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "es_data",
          "schema": "default_db",
          "table": "person",
          "mappedField": [
            { "sourceFieldName": "id",   "targetFieldName": "id"   },
            { "sourceFieldName": "name", "targetFieldName": "name" },
            { "sourceFieldName": "age",  "targetFieldName": "age"  }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "age",  "type": "INT"    }
      ]
    },
    {
      "label": "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "es_data",
          "schema": "default_db",
          "table": "software",
          "mappedField": [
            { "sourceFieldName": "id",   "targetFieldName": "id"   },
            { "sourceFieldName": "name", "targetFieldName": "name" },
            { "sourceFieldName": "lang", "targetFieldName": "lang" }
          ]
        }
      },
      "id":        [{ "name": "id",   "type": "STRING" }],
      "attribute": [
        { "name": "name", "type": "STRING" },
        { "name": "lang", "type": "STRING" }
      ]
    }
  ],
  "edge": [
    {
      "label":         "knows",
      "fromNodeLabel": "person",
      "toNodeLabel":   "person",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "es_data",
          "schema": "default_db",
          "table": "knows",
          "mappedField": [
            { "sourceFieldName": "id",      "targetFieldName": "id"      },
            { "sourceFieldName": "from_id", "targetFieldName": "from_id" },
            { "sourceFieldName": "to_id",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "weight",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    },
    {
      "label":         "created",
      "fromNodeLabel": "person",
      "toNodeLabel":   "software",
      "dataSourceGroup": {
        "externalDataSource": {
          "enabled": true,
          "catalog": "es_data",
          "schema": "default_db",
          "table": "created",
          "mappedField": [
            { "sourceFieldName": "id",      "targetFieldName": "id"      },
            { "sourceFieldName": "from_id", "targetFieldName": "from_id" },
            { "sourceFieldName": "to_id",   "targetFieldName": "to_id"   },
            { "sourceFieldName": "weight",  "targetFieldName": "weight"  }
          ]
        }
      },
      "id":        [{ "name": "id",      "type": "STRING" }],
      "fromKey":   [{ "name": "from_id", "type": "STRING" }],
      "toKey":     [{ "name": "to_id",   "type": "STRING" }],
      "attribute": [
        { "name": "from_id", "type": "STRING" },
        { "name": "to_id",   "type": "STRING" },
        { "name": "weight",  "type": "DOUBLE" }
      ]
    }
  ]
}

▶ In the Web UI, click Graph in the sidebar, then Upload Schema, and select schema.json.

Upload via CLI

You can also POST the schema directly:

curl -X POST -H "content-type: application/json" \
  --data-binary @./schema.json \
  --user "puppygraph:puppygraph123" \
  http://localhost:8081/schema

Querying the Graph

In the PuppyGraph Web UI, click Query in the sidebar. You can run graph queries in either Cypher or Gremlin.

The following query answers "What software was created by people that marko knows?"

MATCH path = (p:person)-[:knows]->()-[:created]->()
WHERE p.name = 'marko'
RETURN path;

g.V().hasLabel('person').has('name', 'marko')
  .out('knows').out('created').path()

There are two paths in the result: marko knows josh, who created lop and ripple.

Cleanup

▶ Shut down and remove the containers:

docker compose down