Skip to main content

Apache Iceberg

Keep your telemetry in your own object storage and let Huntbase query it where it lies. In most cases all we need is a bucket and a scoped access key — we create and operate the Apache Iceberg catalog for you, in your bucket. If you already run an Iceberg REST catalog or use AWS Glue, point us at that instead.

info

Apache Iceberg has its own guide: Data lake. This page summarises the connection settings.

CategoryData warehouse
DirectionData destination
Sign-inStatic Access Keys
Query languagesSQLite
Websitehttps://huntbase.io

Before you start​

Huntbase signs in to Apache Iceberg with Static Access Keys. Create the credential in Apache Iceberg first, then keep it to hand for the Connect step.

  1. Pick a bucket and prefix for your telemetry, such as s3://your-bucket/huntbase/.
  2. On AWS, attach an IAM policy to a dedicated IAM user that allows s3:ListBucket on the bucket and read and write object actions such as s3:GetObject and s3:PutObject on arn:aws:s3:::your-bucket/huntbase/* only.
  3. In the IAM console, choose Security credentials for that user, then choose Create access key in Access keys.
  4. On Retrieve access keys, choose Show or Download .csv file. This is your only chance to save the secret access key.
  5. Note the object-store endpoint and Region.
Watch out for
  • Read access is used for queries, and write access only for the Iceberg table metadata that the catalog maintains. Huntbase never writes your event data and never reads outside the prefix you scope the key to.
  • Creating the connection doesn't provision the store. It's provisioned after the connection is created, and only then does it join federated queries.

For the vendor's own instructions, see AWS example policy for S3 bucket object access.

tip

Use a dedicated, read-only credential for Huntbase where the product allows it. Huntbase only needs to read.

Connect Apache Iceberg​

  1. Go to Connections and click New connection, or click New connection on the Apache Iceberg product page.
  2. On Product, pick Apache Iceberg and choose the Owner.
  3. On Details, give the connection a Name and, optionally, a Description.
  4. On Connect, fill in the settings and credentials described below.
  5. On Verify, review the summary and click Create & check.

For everything else on the setup page — saving a draft, I'll do this later, and what each check result means — see Connections.

Settings​

Where your Apache Iceberg lives:

FieldRequiredNotes
How your tables are cataloguedYesmanaged needs nothing you have to install. glue uses the Data Catalog you already have on AWS. rest points at a catalog service you operate. One of managed, glue, rest. Defaults to managed.
Object Store EndpointYesHTTP(S) endpoint of your object store. Pick the AWS regional endpoint, or type your own for MinIO, Ceph, Cloudflare R2 or any other S3-compatible store. One of https://s3.us-east-1.amazonaws.com, https://s3.us-east-2.amazonaws.com, https://s3.us-west-2.amazonaws.com, https://s3.eu-west-1.amazonaws.com, https://s3.eu-west-2.amazonaws.com, https://s3.eu-central-1.amazonaws.com, https://s3.ap-southeast-2.amazonaws.com.
RegionYesFor MinIO and similar stores, use the region string your store expects. This is usually us-east-1.
Bucket / warehouse locationYesThe bucket and prefix, such as s3://your-bucket/huntbase/. For the managed catalog, this is the only location you need to provide. Shown only when How your tables are catalogued is managed or glue.
Iceberg REST Catalog URIYesHTTP(S) endpoint of the Iceberg REST catalog you run, e.g. https://catalog.example.eu:8181. Databricks Unity Catalog and Snowflake Open Catalog both expose one — point us at theirs. Shown only when How your tables are catalogued is rest.

Advanced settings​

These settings are under Advanced. You can usually leave them alone.

FieldRequiredNotes
Glue Catalog IDNoOnly when the Glue catalog lives in a different AWS account from the access key. Shown only when How your tables are catalogued is glue.

Credentials​

The only Method is Static Access Keys. Enter a Credential label (for example Production), then fill in:

FieldRequiredNotes
Access Key IDYesS3-compatible access key id. Secret — not shown again after you save it.
Secret Access KeyYesS3-compatible secret access key. Secret — not shown again after you save it.

Query it​

Once connected, Apache Iceberg can serve these languages in a query tab, and Scout can use it when you ask in Auto:

LanguageUse it for
SQLiteSQL across this source and every other connected source.

Next steps​