Data lake

Huntbase can query telemetry that lives in a data lake as well as data it reaches through a connection. There are two ways in, and you can use both at once:

• Bring your own lake — point Huntbase at an Apache Iceberg store you already own. Your data stays where it is.
• Ship logs to Huntbase — send events from a log shipper (Fluent Bit, Vector, rsyslog, syslog-ng) straight into Huntbase's own store. This needs no store of your own, and is the shortest way to get data in.

Either way you query the result with KQL (Huntbase Lake) in the Explorer, and a single query searches every store available to you at once. See Query languages.

Beta

The data lake is currently rolling out and may not be enabled for your organization.

Bring your own lake

A lake store is set up like any other connection — see Create a connection — with one difference that matters, covered below.

Choose how your tables are catalogued

The first question on the Connect step is How your tables are catalogued, and it decides what else you are asked for. There are three answers:

Catalog type	Use it when	What Huntbase then asks for
Give us a bucket — Huntbase runs the catalog	You have object storage and nothing else. This is the default and the shortest path.	Bucket / warehouse location
AWS Glue Data Catalog	You are on AWS and already use Glue. Nothing to stand up.	Bucket / warehouse location, and a Glue Catalog ID only if the catalog is in a different AWS account from your key
I already run an Iceberg REST catalog	You operate an Iceberg REST catalog service of your own.	The catalog's URI

The form only shows the fields your answer needs, so switching the catalog type changes which settings appear.

What to have ready

You'll need	Notes
Object store endpoint	Pick your AWS regional endpoint from the list, or type your own for MinIO, Ceph, Cloudflare R2 or any other S3-compatible store.
Region	Pick from the AWS regions, or type the region string your store expects. us-east-1 is the usual answer for MinIO.
Access key and secret key	S3-compatible credentials. Read access to the warehouse prefix is enough for queries; write is used only for Iceberg table metadata.
Catalog settings	Whichever the table above says your catalog type needs.

Apache Iceberg only

Bring your own lake reads Apache Iceberg tables over Parquet in an S3-compatible store. Delta Lake tables are not read directly.

The rest catalog type currently connects to an Iceberg REST catalog that does not require authentication. Managed catalogs that speak the same protocol but authenticate — Databricks Unity Catalog, Snowflake Open Catalog, Apache Polaris, AWS S3 Tables — are not connectable yet. Support for them is planned; talk to us if you need one.

Creating and provisioning the store

Search for the lake product on the Product step, fill in the settings and keys on Connect, then click Create & check.

Creating the connection is only half of it. Unlike every other product, a lake store has to be provisioned before it will answer anything, and that starts automatically when the connection is created. The Verify step shows the progress:

Step	What is happening
Reach the lake	Connecting to the Iceberg catalog and object store.
Register catalog	Creating the store's query catalog.
Attach tables	Binding the telemetry tables Huntbase reads.
Join federation	Including the store in federated lake queries.

Provisioning continues if you close the page. You can follow it, and re-run it, from the Lake store card on the connection's own page.

warning

Until provisioning completes the store answers nothing, and lake queries skip it. A store that is merely healthy is not necessarily queryable — check the federation badge below.

In federation, or excluded

The Lake store card carries a badge that answers the only question that matters for results:

Badge	Meaning
In federation	The store is active and provisioned, so lake queries read it.
Excluded	Lake queries skip it. Hover the badge for the reason — never provisioned, provisioning failed, provisioning still running, or the connection is inactive.

This matters because an excluded store makes your results narrower without making them look wrong. When a query skips a store, the results panel says so — see Which stores answered.

When provisioning fails

The trail names the step that failed and gives you a short remediation. The technical diagnostic is deliberately terse (something like probe: ConnectionRefusedError) because Huntbase does not echo back what your catalog endpoint returned. Treat it as a hint, and check in this order:

1. Are the object store endpoint — and the catalog URI, if you supplied one — reachable from outside your network?
2. Do the access keys still work, and does their policy cover the warehouse prefix?
3. Are the region and warehouse location right?

Then use Retry provisioning.

Ship logs to Huntbase

If you would rather send events than expose a store, mint a telemetry ingest key and point a log shipper at Huntbase.

Mint a key

Two places, depending on where the data should land:

• Into Huntbase's managed store — the Send telemetry to Huntbase card on the Connections page, or the same card during onboarding. No store of your own required.
• Into a lake you own — that store's connection page, under Telemetry ingest keys, so the key is bound to it.

Either way, click New key.

If Telemetry ingest keys shows Request access instead of a key list, ingest is not enabled for the organization you are working in. Use the button to ask — it fills in the details support needs to turn it on.

Give it a Source label that names the sender — something like edge-fluentbit-eu. One key per shipper or source means a single noisy or compromised sender can be revoked on its own.

warning

The token is shown once, when you create it. It is stored only as a hash, so it cannot be recovered or shown again. Copy it before you close the dialog — the dialog will ask you to confirm if you try to dismiss it without doing so.

Afterwards the key is identified only by its prefix (for example hbik_3f9a2c1).

Where to send it

The Send telemetry to Huntbase card shows the endpoint for your deployment, so what you read there is what you paste. It is the same host this app talks to:

Method and path	POST /api/v1/telemetry/ingest
Host	Your Huntbase host, shown on the card.
Port	443
Auth	Authorization: Bearer <your ingest key>

Shippers that want host, port and path separately (Fluent Bit, for example) will find them listed individually on the card.

Before wiring anything up, you can prove the network path from the machine that will do the shipping:

curl -fsS https://<your-huntbase-host>/api/v1/telemetry/ingest/health

That probe is unauthenticated by design, so it works before any key exists — and it distinguishes "the firewall blocks this" from "the key is wrong", which is otherwise a long afternoon.

Configure your shipper

The mint dialog generates ready-to-paste configuration for Fluent Bit, Vector, rsyslog and syslog-ng, plus a command that checks Huntbase is reachable before you involve any secret. You can bring the configuration back up later from the key's Config action — with a placeholder in place of the token, since we cannot show it again.

What the endpoint accepts:

• A single JSON object, a JSON array, or newline-delimited JSON.
• Optionally gzipped.
• Up to 10 MB per request, before and after decompression.

It answers with a count of accepted and rejected records. A malformed record is counted, not retried, and never fails the rest of the batch — so a shipper should treat the response as success and not resend.

Revoking a key

Click Revoke, then confirm. Shippers using it start failing immediately. The row stays in the list, visibly marked as revoked, so the history of what existed is preserved.

The Last used column is the quickest way to tell a key that is wired up and working from one that was minted and never configured.

Querying the lake

Pick KQL (Huntbase Lake) in the query bar's language picker. Unlike the other languages it needs no connection — the bar shows Data lake instead of a connection picker, because a lake query searches Huntbase's own store plus every one of your provisioned stores together.

Which stores answered

Because one query can read several stores, the results panel tells you what it actually searched:

Notice	What it means
Searched N stores	Expand it to see which. Each store can be used to narrow the results to just its rows.
N stores were excluded	A warning, not a footnote: the store was unavailable, so your results may be incomplete. The notice links to the connection.
Showing the first N rows	A row cap was applied. Add `
A column is empty for some events	No mapping exists for that field in those events, so those rows are empty rather than guessed.

Result rows also carry a store column showing which store each row came from, and you can narrow to one store from there.

Next steps

• Query languages — writing lake KQL, and what the dialect supports
• Connections — creating and managing connections
• Query workspace — running queries and reading results
• Watchers — standing rules over the data you have connected