---
title: "Connect Elasticsearch (ES|QL)"
description: "Connect Datarelix to Elasticsearch 8.11+ over ES|QL: API-key or basic auth, index scoping, TLS certificates, and what the query language cannot do."
canonical: https://docs.datarelix.ai/guides/connections/elasticsearch/
---

# Connect Elasticsearch (ES|QL)

Datarelix connects to Elasticsearch 8.11+ through a read-only query service. Queries go through [ES|QL](https://www.elastic.co/docs/reference/query-languages/esql) — Elastic's piped query language, available in the free Basic tier on ES 8.11+.

## Prerequisites

- Elasticsearch **8.11+** — ES|QL went GA in 8.11. Older versions will reject the `/_query` endpoint.
- Security features enabled (default on Elastic Cloud; for self-managed clusters, ensure `xpack.security.enabled: true` and TLS configured).
- An API key with [`read` and `view_index_metadata` privileges](https://www.elastic.co/docs/reference/elasticsearch/security-privileges) on the target indices.

> **This connector is for Elasticsearch, not OpenSearch.** AWS OpenSearch Service and the OpenSearch project are forks that don't implement ES|QL (they use PPL instead) and aren't supported here.

**Deployment types** — the same connector works across all three, but where you get the endpoint and API key differs:

- **Elastic Cloud Hosted** — Cloud console → your deployment → copy the Elasticsearch endpoint; create API keys in Kibana.
- **Elastic Cloud Serverless** — uses a different auth model: create a **project API key** from the serverless project's management UI, and use the project's Elasticsearch endpoint. The same `read` + `view_index_metadata` privileges apply.
- **Self-managed** — your cluster's HTTPS endpoint (e.g. `https://es.example.com:9200`); API keys via Kibana or the API.

**Licensing:** ES|QL on a single cluster is in the free **Basic** tier on ES 8.11+. Only ES|QL cross-cluster search requires an Enterprise subscription — Datarelix uses single-cluster ES|QL.

## Auth modes

### API key (recommended)

A scoped API key authenticates every request. The key is stored encrypted and is revocable without restarting any service.

**Setup — create a scoped API key**

**Option A: Kibana UI**

1. In Kibana, [create an API key](https://www.elastic.co/docs/deploy-manage/api-keys/elasticsearch-api-keys) (**Stack Management → Security → API Keys → Create API key**).
2. Give it a name (e.g. `datarelix-prod`).
3. Under **Restrict privileges**, add index privileges: select your target indices, add the `read` and `view_index_metadata` privileges.
4. Click **Create**. Copy the **API key** (base64 encoded `id:secret`) — it's only shown once.

**Option B: Elasticsearch API**

```bash
POST /_security/api_key
{
  "name": "datarelix-prod",
  "role_descriptors": {
    "logs_reader": {
      "indices": [
        { "names": ["logs-*", "metrics-*"], "privileges": ["read", "view_index_metadata"] }
      ]
    }
  }
}
```

The response contains `id` and `api_key` — the form expects these as separate **API key ID** and **API key secret** fields.

**Where to find your credentials**

- **Elastic Cloud**: Elastic Cloud console → your deployment → **Copy endpoint** for the cluster URL; API keys via Kibana as above.
- **Self-managed**: Cluster URL is your Elasticsearch HTTP endpoint (e.g. `https://es.example.com:9200`); API keys via Kibana or the API.

**What to enter**

```ini
Cluster URL:     https://your-deployment.es.us-east-1.aws.elastic-cloud.com:9243
API key ID:      your-key-id
API key secret:  ••••••••
Verify TLS:      on
Allowed indices: logs-*,metrics-*
```

---

### Basic auth (dev only)

Username and password via the Elasticsearch SDK's `basic_auth`. Surfaces a warning in the UI.

> **Not recommended for production.** Basic auth attributes every request to the same user (no per-connection audit trail), and rotating credentials requires editing the connection.

**Setup**

Use an existing Elasticsearch user with `read` and `view_index_metadata` on the target indices, or create a dedicated role and user:

```bash
# Create a role
POST /_security/role/datarelix_reader
{
  "indices": [{ "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] }]
}

# Create a user with that role
POST /_security/user/datarelix_reader
{
  "password": "choose-a-strong-password",
  "roles": ["datarelix_reader"]
}
```

**What to enter**

```ini
Cluster URL:     https://your-deployment.es.example.com:9200
Username:        datarelix_reader
Password:        ••••••••
Verify TLS:      on
Allowed indices: logs-*
```

---

## Connection form

```ini
Cluster URL:        https://es.example.com:9243
API key ID:         your-key-id           (API key mode)
API key secret:     ••••••••              (API key mode)
Verify TLS:         on
CA cert PEM:        (optional, for clusters with a self-signed or internal CA cert)
Allowed indices:    logs-*,metrics-*
```

## TLS / CA cert

If your cluster uses a self-signed cert or an internal CA, paste the PEM into **CA cert PEM**. Where to get it:

- **Elastic Cloud**: deployment page → **Security** → download the CA certificate.
- **Self-managed (default install)**: `$ES_HOME/config/certs/http_ca.crt` on any cluster node.
- **Docker default install**: `docker cp <es-container>:/usr/share/elasticsearch/config/certs/http_ca.crt ./http_ca.crt`

Paste the **entire** PEM block including `-----BEGIN CERTIFICATE-----` / `-----END CERTIFICATE-----`.

Disabling **Verify TLS** is supported but only for dev — it makes the connection vulnerable to active MITM attacks.

## Scope semantics

The connection persists two scope fields:

- **Allowed indices** — a list of patterns (comma-separated in the UI). Evaluated first.
- **Index pattern** — a single fallback pattern used only when Allowed indices is empty.

At least one must be set. One connection can cover multiple index groups by listing them in Allowed indices (e.g. `logs-*,metrics-*`).

## Discovery

Fully supported. The introspector calls the Elasticsearch cat indices API and retrieves field mappings for each in-scope index, flattening nested objects to dot-notation columns (`user.name`, `request.headers.user_agent`). Elasticsearch has no PK/FK concept — the LLM enrichment pass infers cross-index links from naming conventions if you opt in.

## ES|QL primer

Generated queries use ES|QL. You don't write ES|QL yourself — Datarelix generates it. For context:

```text
FROM logs-*
| WHERE response.status == 500
| STATS count() BY url.path
| SORT count desc
| LIMIT 10
```

- `FROM <index-or-pattern>` is the source.
- `|` pipes one step's output into the next operator.
- `KEEP` selects columns; `WHERE` filters; `STATS` aggregates; `SORT` orders; `LIMIT` caps rows.
- Nested fields flatten with dot notation (e.g. `user.name`).

See the [ES|QL reference](https://www.elastic.co/docs/reference/query-languages/esql) for the full language.

## Limitations

- **No mutations** — the validator rejects any mutation keywords as defense-in-depth.
- **ES|QL `LOOKUP JOIN` requires ES 8.13+** — older ES versions have no JOIN; keep cross-index analysis to a single `FROM` pattern or pre-enrich upstream.
- **Large field counts** (>1k fields per index) may hit prompt-size limits during planning. Discovery still works, but you may need to narrow the allowed indices scope.
- **Row limit** — server-side cap of 5 000 rows by default; the validator appends a `LIMIT` when missing.
- **Partial results on shard failure** — if a shard is unavailable, ES returns what it can plus a `partial=true` flag. The run UI shows a warning badge; re-run when the cluster recovers.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `401 Unauthorized` | API key rotated, expired, or missing privileges | Recreate the key with `read` + `view_index_metadata`. |
| `TLS verify failed` | Self-signed cluster or custom CA | Paste the CA cert PEM into the connection form, or disable TLS verify in dev. |
| `parsing_exception` | ES|QL syntax error | [Contact us](https://datarelix.ai/contact/) with the question and the generated query. |
| `Index 'foo' is not in allowlist` | Generated query targets an index outside Allowed indices | Add the index pattern to the connection's allowlist. |
| Empty schema after discovery | ES|QL not enabled or missing `view_index_metadata` privilege | Confirm ES 8.11+ and re-check the API key privileges. |
| `unknown URI [/_query]` | Cluster is older than 8.11 | Upgrade — the legacy `_sql/query` endpoint is not used by this integration. |
| `partial=true` warning | One or more shards unavailable | Re-run when cluster health returns to yellow/green. |