Om Jay Mishra

Migrated 40+ production scripts from a legacy Ubuntu server to Databricks notebooks and Workflows, decommissioning the legacy environment
Reduced a thrice-daily pipeline’s runtime from ~1 hr 50 min to ~30 min per run by resolving data skew and migrating to job compute clusters, cutting daily compute hours by ~70% and lowering DBU costs
Containerized a VPN-dependent pipeline using Docker on Azure containers; accelerated delivery through AI-assisted development (Claude + VS Code + Databricks)

Led end-to-end migration from legacy databases to Databricks Lakehouse, transforming SQL-based workflows into scalable PySpark and Delta Lake architectures
Resolved partition skew on 500M+ record pipelines using Liquid Clustering, reducing runtime from 6 hours to 30 minutes
Developed a Gold-layer Star Schema in Databricks, improving query performance by 65% across 50M+ records
Reduced Databricks DBU consumption by 25% through cluster optimization, autoscaling, and Photon Engine utilization
Orchestrated Azure Data Factory and Databricks pipelines with custom Python automation, eliminating 270+ hours of manual effort

Automated data transformation workflows for 10K+ student records using Google Apps Script,
improving data preparation efficiency
Consolidated multi-source data into a centralized Power BI reporting layer, enabling stakeholders to track engagement and enrolment metrics