Databricks

Databricks is a unified data analytics and lakehouse platform built on Apache Spark that combines data warehousing, data engineering, machine learning, and analytics capabilities. This integration provides comprehensive access to Databricks Unity Catalog and workspace metadata, enabling organizations to query, inventory, monitor, and govern their data platform resources, infrastructure, and configurations.

Category	Data warehouse
Direction	Query source
Sign-in	Personal Access Token, OAuth Service Principal, Basic Authentication, Databricks CLI Config File
Query languages	SQLite
Tables	51
Query templates	362
Website	databricks.com

Before you start

Huntbase can sign in to Databricks with any of these methods: Personal Access Token, OAuth Service Principal, Basic Authentication, Databricks CLI Config File. Pick one, create the credential in Databricks, then keep it to hand for the Connect step.

Personal Access Token

Databricks workspace Personal Access Token or account SCIM token

1. In your Databricks workspace, click your user name in the top bar and select Settings.
2. Click Developer, then click Manage next to Access tokens.
3. Click Generate new token, enter a name, and set the token's lifetime in days.
4. Click Generate, copy the token, then click Done.

OAuth Service Principal

OAuth machine-to-machine (M2M) authentication using a service principal client ID and client secret.

1. As a workspace admin, click your user name in the top bar and select Settings › Identity and access. Next to Service principals, click Manage.
2. Click Add service principal › Add new, enter a name, and click Add.
3. Select the service principal, open the Secrets tab, and click Generate secret.
4. Set the secret's lifetime in days (up to 730). Under Scopes, select only the API scopes you need rather than all APIs.
5. Click Generate, copy the secret and the client ID, then click Done.

Watch out for

• An OAuth secret is shown only once. If you lose it, generate a new one. A service principal can have up to five secrets.
• If you can't create a personal access token, your workspace admin may have disabled tokens or restricted who can use them.
• Basic authentication with a user name and password only works for workspaces on *.cloud.databricks.com (AWS).

For the vendor's own instructions, see Databricks OAuth machine-to-machine authentication.

tip

Use a dedicated, read-only credential for Huntbase where the product allows it. Huntbase only needs to read.

Connect Databricks

1. Go to Connections and click New connection, or click New connection on the Databricks product page.
2. On Product, pick Databricks and choose the Owner.
3. On Details, give the connection a Name and, optionally, a Description.
4. On Connect, fill in the settings and credentials described below.
5. On Verify, review the summary and click Create & check.

For everything else on the setup page — saving a draft, I'll do this later, and what each check result means — see Connections.

Settings

Where your Databricks lives:

Field	Required	Notes
Config Profile	No
Account ID	No	Your Databricks account ID. Only needed for account-level data, together with the Account Console URL.
Account Console URL	No	The account console URL, typically https://accounts.cloud.databricks.com. Only needed for account-level data.
Workspace URL	No	Your workspace URL, such as https://dbc-a1b2c3d4-e6f7.cloud.databricks.com.
Config File Path	No
OAuth Client ID	No	The service principal's application ID, shown with the secret when you generate it.

Credentials

Choose a Method, enter a Credential label (for example Production), then fill in the fields for that method.

Personal Access Token

Field	Required	Notes
Workspace Personal Access Token	Yes	Secret — not shown again after you save it.
Account SCIM Token	No	Secret — not shown again after you save it.

OAuth Service Principal

Field	Required	Notes
OAuth Client Secret	Yes	Secret — not shown again after you save it.

Basic Authentication

Field	Required	Notes
Username	Yes	Secret — not shown again after you save it.
Password	Yes	Secret — not shown again after you save it.

Databricks CLI Config File

No fields — choose this method to connect without credentials.

Query it

Once connected, Databricks can serve these languages in a query tab, and Scout can use it when you ask in Auto:

Language	Use it for
SQLite	SQL over the 51 tables listed below, alongside every other connected source.

Example: SQLite

List Databricks IAM groups basic info — Explore which user groups exist within your Databricks account, including their unique identifiers and display names, to manage and control access effectively.

select
  id,
  display_name,
  account_id
from
  databricks_iam_group;

Databricks ships with 362 query templates. Find them in Library › Queries — see Query templates.

Tables

Databricks adds 51 tables. Browse their columns from Schema in the query bar's ⋯ menu.

All 51 tables

Table	Contains
databricks_catalog	Gets an array of catalogs in the metastore.
databricks_catalog_connection	Gets an array of connections for the workspace.
databricks_catalog_external_location	Gets an array of external locations from the metastore.
databricks_catalog_function	List functions within the specified parent catalog and schema.
databricks_catalog_metastore	Gets an array of the available metastores.
databricks_catalog_schema	List schemas for a catalog in the metastore.
databricks_catalog_storage_credential	Gets an array of storage credentials.
databricks_catalog_system_schema	Gets an array of system schemas for a metastore.
databricks_catalog_table	Gets an array of the available tables.
databricks_catalog_volume	Gets an array of the available volumes.
databricks_compute_cluster	Gets a list of clusters.
databricks_compute_cluster_node_type	Returns a list of supported Spark node types.
databricks_compute_cluster_policy	Gets an array of cluster policies.
databricks_compute_global_init_script	Gets a list of all global init scripts for this workspace.
databricks_compute_instance_pool	Gets a list of instance pools with their statistics.
databricks_compute_instance_profile	List the instance profiles that the calling user can use to launch a cluster.
databricks_compute_policy_family	Retrieve a list of policy families.
databricks_files_dbfs	List the contents of a directory, or details of the file.
databricks_iam_account_group	List group details associated with a Databricks account.
databricks_iam_account_user	List details for all the users associated with a Databricks account.
databricks_iam_current_user	Gets details for the current user of the workspace.
databricks_iam_group	List group details associated with a Databricks workspace.
databricks_iam_service_principal	List the set of service principals associated with a Databricks workspace.
databricks_iam_user	List details for all the users associated with a Databricks workspace.
databricks_job	Get details for all the jobs associated with a Databricks workspace.
databricks_job_run	List details for all the job runs.
databricks_ml_experiment	Get details for all the experiments associated with a Databricks workspace.
databricks_ml_model	List all available registered models.
databricks_ml_webhook	List all registry webhooks.
databricks_pipeline	List pipelines defined in the Delta Live Tables system.
databricks_pipeline_event	Retrieves events for a pipeline.
databricks_pipeline_update	List updates for an active pipeline.
databricks_serving_serving_endpoint	List all serving endpoints.
databricks_settings_ip_access_list	Gets all IP access lists for the specified workspace.
databricks_settings_token	List all the valid tokens for a user-workspace pair.
databricks_settings_token_management	List all tokens associated with the specified workspace or user.
databricks_sharing_provider	Gets an array of available authentication providers.
databricks_sharing_recipient	Gets an array of all share recipients within the current metastore.
databricks_sharing_share	List all data object shares from the metastore.
databricks_sql_alert	Gets a list of alerts.
databricks_sql_dashboard	Gets details for all the dashboards associated with a Databricks workspace.
databricks_sql_data_source	Retrieves a full list of SQL warehouses available in this workspace.
databricks_sql_query	Gets a list of queries.
databricks_sql_query_history	List the history of queries through SQL warehouses.
databricks_sql_warehouse	Gets a list of warehouses.
databricks_sql_warehouse_config	Gets the workspace level configuration that is shared by all SQL warehouses in a workspace.
databricks_workspace	List all secret workspaces available in the workspace.
databricks_workspace_git_credential	Lists the calling user's Git credentials.
databricks_workspace_repo	Returns repos that the calling user has Manage permissions on.
databricks_workspace_scope	List all secret scopes available in the workspace.
databricks_workspace_secret	List the secret keys that are stored.

Next steps

• Connections — health, credentials and settings after you connect
• Query languages — syntax, parameters and time ranges
• Chatting with Scout — ask questions without writing a query
• All integrations — the rest of the catalog