Articles
Architecture decisions, technical deep-dives, patterns, and lessons learned from real-world projects.
Building a Multi-Agent AI System on Databricks with LangGraph
Orchestrator, Planner, Executor, Deployment and Test agents — architecture, prompts, and lessons learned.
Mainframe COBOL to PySpark: Lessons from a Real Migration
Architecture decisions, GitHub Copilot usage, and achieving 60–70% effort reduction in a Banking mainframe migration.
Designing a Metadata-Driven ETL Framework on Azure Databricks
How metadata-driven design eliminates hardcoded logic and simplifies onboarding new data sources.
Data Lakehouse Architecture — Medallion Layers on Databricks
Bronze → Silver → Gold design patterns, data quality enforcement, and Unity Catalog governance.
Semantic Layer Architecture — Metrics as Code
Designing a metrics layer on top of Databricks and Snowflake — metric views, dimensions, and governance.
GitHub Copilot as a Migration Accelerator: Real Numbers
Honest look at 40–60% effort reduction in Informatica-to-PySpark migration — what worked, what didn't.
Prompt Engineering for Data Engineers
Effective prompt patterns for SQL generation, schema analysis, ETL code conversion, and data quality rules.
CI/CD for Databricks: Azure DevOps Pipelines End-to-End
Production-grade CI/CD for Databricks notebooks and jobs — branching, testing, and automated deployments.
Data Quality Framework Design
Designing a configurable data quality framework with rules engine, alerting, and quarantine pattern.
My Journey: Transitioning from VMware ESXi 7 to Proxmox 9
Migrating 30+ VMs from VMware ESXi to Proxmox — planning, phased migration, and post-migration optimisation.