Skip to content

Query Files & Object Storage

Datarelix can query data files — Delta Lake, Parquet, CSV, JSON, and Excel — directly from AWS S3, Azure Blob/ADLS, Google Cloud Storage, or a direct HTTPS URL, powered by an embedded, read-only DuckDB engine. No database server required.

Instead of schemas and tables, you define a list of datasets. Each dataset is a logical table backed by a file, a folder of files, or a Delta table. Datarelix exposes each dataset as a queryable table named however you choose — the AI plans SELECT … FROM your_table and never sees or constructs the underlying file paths.

Prerequisites

  • A storage location Datarelix can reach over the network.
  • Credentials for that storage (or a public HTTPS URL).
  • One or more files in a supported format: Parquet, CSV, JSON, Excel (.xlsx), or Delta Lake.

Connection form

FieldMeaning
Storage providerSelects the storage backend: AWS S3, Azure, GCS, or Direct HTTPS.
DatasetsOne or more named tables. For each: a name, format, which files, and a path/URI.

Datasets

For each dataset you provide:

  • Table name — the identifier the AI queries (e.g. sales). Letters, digits, and underscores.
  • Format — parquet, csv, json, excel, or delta.
  • Which files (non-Delta): All files in directory (union), Latest file only, or A single file.
  • Path / URI — an exact file for A single file; a directory or glob for the other modes (e.g. s3://bucket/sales/ or s3://bucket/sales/*.parquet).
  • Version (Delta only): blank reads the latest version; set a number to pin a specific version.
  • CSV options: header row toggle and delimiter. Excel: optional sheet name.

“Latest file” picks the newest file by name (descending sort), which matches date/sequence-stamped filenames such as events-2026-07-01.parquet.

When All files or Latest file points at a bare directory (no * wildcard), a format-specific pattern is applied automatically — s3://bucket/sales/ is read as s3://bucket/sales/*.parquet for a Parquet dataset. Use an explicit glob when you need finer control.

Two combinations are not supported: Direct HTTPS URLs are always a single file (HTTP servers can’t list directories), and Excel datasets read one file — use Latest file or A single file.

Auth modes

AWS S3

FieldNotes
AWS Regione.g. us-east-1.
Access Key ID / SecretOptional — leave blank only for publicly readable objects.

Use s3://bucket/path/... URIs.

Azure Blob / ADLS

FieldNotes
Storage AccountYour Azure storage account name.
Account Key or SAS TokenOne is required.

Use az://container/path/... URIs.

Google Cloud Storage

FieldNotes
HMAC Key ID / SecretCreate interoperability HMAC keys for your service account.

Use gs://bucket/path/... URIs.

Direct HTTPS URL

FieldNotes
Authorization HeaderOptional — for token-protected URLs (e.g. Bearer abc123). Leave blank for public files.

Use https://host/path/file.parquet URIs.

Query behavior

  • Read-only. Only SELECT queries run. Mutations and IO/admin statements are rejected.
  • The AI references your dataset names only. File-reading functions (read_parquet, delta_scan, glob, …) and arbitrary paths in generated SQL are rejected — paths come solely from your saved connection.
  • Results are capped at 5,000 rows per query.

Discovery

After connecting, discovery reads each dataset’s columns and types. Optional AI schema analysis adds table and column descriptions. Files carry no declared keys or foreign keys, so relationships are suggested from naming conventions.

Limitations & troubleshooting

  • Delta version pinning — if the pinned version cannot be read, the dataset fails to load rather than falling back to another version. Clear Version to read the latest.
  • “Latest file” is filename-based, not modification-time based.
  • Access denied / cannot read file — check the credentials and that the path/glob matches actual objects. For S3, check that the access key you entered can read the bucket.
  • Excel requires the DuckDB excel extension; Delta requires the delta extension. Both are already installed — nothing to set up on your side.