Azure Databricks
Cluster Configuration — ETL Workloads
Optimised cluster config for large-scale PySpark ETL: node types, autoscaling, Spark settings.
Snowflake
Snowflake Connection — Python & Spark Connector
Connection string templates and authentication patterns for Python, PySpark, and ADF.
Azure Data Factory
Linked Service Templates — ADLS, SQL, Databricks
ARM-ready linked service JSON for ADLS Gen2, Azure SQL, Databricks, and Snowflake.
Azure DevOps / CI-CD
Databricks CI/CD Pipeline — azure-pipelines.yml
Full YAML pipeline: lint, test, bundle, and deploy Databricks jobs via DABs.
VS Code Settings
settings.json — Data Engineering Setup
Recommended VS Code settings for Python, PySpark, YAML, SQL, and Databricks extension.
Multi-Agent Templates
Orchestrator Agent — Design Template
State schema, routing logic, and retry strategy for the Orchestrator agent.
Git Configuration
.gitconfig & Branching Strategy
Global git config, aliases, and branching conventions for data engineering projects.
Apache Airflow
DAG Templates — Databricks Job Trigger
Airflow DAG to trigger and monitor Databricks jobs with retry and alerting config.