Elasticsearch (ES|QL)

datarelix.ai connects to Elasticsearch 8.11+ through a read-only query service. Queries go through ES|QL — Elastic’s piped query language, available in the free Basic tier on ES 8.11+.

Prerequisites

Elasticsearch 8.11+ — ES|QL went GA in 8.11. Older versions will reject the /_query endpoint.
Security features enabled (default on Elastic Cloud; for self-managed clusters, ensure xpack.security.enabled: true and TLS configured).
An API key with read and view_index_metadata privileges on the target indices.

This connector is for Elasticsearch, not OpenSearch. AWS OpenSearch Service and the OpenSearch project are forks that don’t implement ES|QL (they use PPL instead) and aren’t supported here.

Deployment types — the same connector works across all three, but where you get the endpoint and API key differs:

Elastic Cloud Hosted — Cloud console → your deployment → copy the Elasticsearch endpoint; create API keys in Kibana.
Elastic Cloud Serverless — uses a different auth model: create a project API key from the serverless project’s management UI, and use the project’s Elasticsearch endpoint. The same read + view_index_metadata privileges apply.
Self-managed — your cluster’s HTTPS endpoint (e.g. https://es.example.com:9200); API keys via Kibana or the API.

Licensing: ES|QL on a single cluster is in the free Basic tier on ES 8.11+. Only ES|QL cross-cluster search requires an Enterprise subscription — datarelix.ai uses single-cluster ES|QL.

Auth modes

API key (recommended)

A scoped API key authenticates every request. The key is stored encrypted and is revocable without restarting any service.

Setup — create a scoped API key

Option A: Kibana UI

In Kibana, create an API key (Stack Management → Security → API Keys → Create API key).
Give it a name (e.g. datarelix-prod).
Under Restrict privileges, add index privileges: select your target indices, add the read and view_index_metadata privileges.
Click Create. Copy the API key (base64 encoded id:secret) — it’s only shown once.

Option B: Elasticsearch API

POST /_security/api_key
{
  "name": "datarelix-prod",
  "role_descriptors": {
    "logs_reader": {
      "indices": [
        { "names": ["logs-*", "metrics-*"], "privileges": ["read", "view_index_metadata"] }
      ]
    }
  }
}

The response contains id and api_key — the form expects these as separate API key ID and API key secret fields.

Where to find your credentials

Elastic Cloud: Elastic Cloud console → your deployment → Copy endpoint for the cluster URL; API keys via Kibana as above.
Self-managed: Cluster URL is your Elasticsearch HTTP endpoint (e.g. https://es.example.com:9200); API keys via Kibana or the API.

What to enter

Cluster URL:     https://your-deployment.es.us-east-1.aws.elastic-cloud.com:9243
API key ID:      your-key-id
API key secret:  ••••••••
Verify TLS:      on
Allowed indices: logs-*,metrics-*

Basic auth (dev only)

Username and password via the Elasticsearch SDK’s basic_auth. Surfaces a warning in the UI.

Not recommended for production. Basic auth attributes every request to the same user (no per-connection audit trail), and rotating credentials requires editing the connection.

Setup

Use an existing Elasticsearch user with read and view_index_metadata on the target indices, or create a dedicated role and user:

# Create a role
POST /_security/role/datarelix_reader
{
  "indices": [{ "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] }]
}

# Create a user with that role
POST /_security/user/datarelix_reader
{
  "password": "choose-a-strong-password",
  "roles": ["datarelix_reader"]
}

What to enter

Cluster URL:     https://your-deployment.es.example.com:9200
Username:        datarelix_reader
Password:        ••••••••
Verify TLS:      on
Allowed indices: logs-*

Connection form

Cluster URL:        https://es.example.com:9243
API key ID:         your-key-id           (API key mode)
API key secret:     ••••••••              (API key mode)
Verify TLS:         on
CA cert PEM:        (optional, for clusters with a self-signed or internal CA cert)
Allowed indices:    logs-*,metrics-*

TLS / CA cert

If your cluster uses a self-signed cert or an internal CA, paste the PEM into CA cert PEM. Where to get it:

Elastic Cloud: deployment page → Security → download the CA certificate.
Self-managed (default install): $ES_HOME/config/certs/http_ca.crt on any cluster node.
Docker default install: docker cp <es-container>:/usr/share/elasticsearch/config/certs/http_ca.crt ./http_ca.crt

Paste the entire PEM block including -----BEGIN CERTIFICATE----- / -----END CERTIFICATE-----.

Disabling Verify TLS is supported but only for dev — it makes the connection vulnerable to active MITM attacks.

Scope semantics

The connection persists two scope fields:

Allowed indices — a list of patterns (comma-separated in the UI). Evaluated first.
Index pattern — a single fallback pattern used only when Allowed indices is empty.

At least one must be set. One connection can cover multiple index groups by listing them in Allowed indices (e.g. logs-*,metrics-*).

Discovery

Fully supported. The introspector calls the Elasticsearch cat indices API and retrieves field mappings for each in-scope index, flattening nested objects to dot-notation columns (user.name, request.headers.user_agent). Elasticsearch has no PK/FK concept — the LLM enrichment pass infers cross-index links from naming conventions if you opt in.

ES|QL primer

Generated queries use ES|QL. You don’t write ES|QL yourself — the planner does. For context:

FROM logs-*
| WHERE response.status == 500
| STATS count() BY url.path
| SORT count desc
| LIMIT 10

FROM <index-or-pattern> is the source.
| pipes one step’s output into the next operator.
KEEP selects columns; WHERE filters; STATS aggregates; SORT orders; LIMIT caps rows.
Nested fields flatten with dot notation (e.g. user.name).

See the ES|QL reference for the full language.

Limitations

No mutations — the validator rejects any mutation keywords as defense-in-depth.
ES|QL LOOKUP JOIN requires ES 8.13+ — older ES versions have no JOIN; keep cross-index analysis to a single FROM pattern or pre-enrich upstream.
Large field counts (>1k fields per index) may hit prompt-size limits during planning. Discovery still works, but you may need to narrow the allowed indices scope.
Row limit — server-side cap of 5 000 rows by default; the validator appends a LIMIT when missing.
Partial results on shard failure — if a shard is unavailable, ES returns what it can plus a partial=true flag. The run UI shows a warning badge; re-run when the cluster recovers.

Troubleshooting

Symptom	Likely cause	Fix
`401 Unauthorized`	API key rotated, expired, or missing privileges	Recreate the key with `read` + `view_index_metadata`.
`TLS verify failed`	Self-signed cluster or custom CA	Paste the CA cert PEM into the connection form, or disable TLS verify in dev.
`parsing_exception`	ES	QL syntax error
`Index 'foo' is not in allowlist`	Generated query targets an index outside Allowed indices	Add the index pattern to the connection’s allowlist.
Empty schema after discovery	ES	QL not enabled or missing `view_index_metadata` privilege
`unknown URI [/_query]`	Cluster is older than 8.11	Upgrade — the legacy `_sql/query` endpoint is not used by this integration.
`partial=true` warning	One or more shards unavailable	Re-run when cluster health returns to yellow/green.