Configuration Reference
Ready-to-use configurations, cluster settings, YAML snippets, and environment setups.
Cluster Configuration — ETL Workloads
Optimised cluster config for large-scale PySpark ETL: node types, autoscaling, Spark settings.
Databricks Secrets & Key Vault Integration
Linking Azure Key Vault secret scopes to Databricks for secure credential management.
Unity Catalog Setup — Metastore & Permissions
Step-by-step Unity Catalog metastore configuration, workspace binding, and RBAC setup.
Databricks Asset Bundles (DABs) — Project Structure
Reference project layout and databricks.yml for deploying jobs and notebooks via DABs.
Snowflake Connection — Python & Spark Connector
Connection string templates and authentication patterns for Python, PySpark, and ADF.
Linked Service Templates — ADLS, SQL, Databricks
ARM-ready linked service JSON for ADLS Gen2, Azure SQL, Databricks, and Snowflake.
Databricks CI/CD Pipeline — azure-pipelines.yml
Full YAML pipeline: lint, test, bundle, and deploy Databricks jobs via DABs.
settings.json — Data Engineering Setup
Recommended VS Code settings for Python, PySpark, YAML, SQL, and Databricks extension.
Orchestrator Agent — Design Template
State schema, routing logic, and retry strategy for the Orchestrator agent.