SAURABH ZAMBARE

Designed and implemented a Lakehouse Architecture (Medallion layers) on Azure Databricks using Delta
•
•
•
•
•
•
•
Lake with data stored in ADLS Gen2.
Built and orchestrated end-to-end data pipelines ADF and Azure Databricks for scalable ingestion and
transformation.
Implemented incremental data ingestion using Auto Loader from ADLS Gen2, enabling efficient and low
latency processing.
Managed data governance and security using Unity Catalog and Azure Key Vault, enabling fine-grained
access control, data lineage tracking, and centralized metadata management.
Optimized Spark workloads, reducing execution time from 2 hours to 20 minutes, and ensured data reliability
using Delta Lake features with monitoring via Azure Monitor.
Performed Spark performance tuning and optimization, significantly reducing execution time and improving
resource utilization.
Utilized Claude Code for code assistance, query optimization, and debugging, improving development
efficiency and reducing troubleshooting time.

Led and facilitated stand-up meetings, allocated tasks, and conducted code reviews to ensure quality and
•
•
•
•
efficiency in project delivery.
Coordinated with the product team to discuss new feature development, assess impact, ensure compliance
with Data Governance policies, and communicate updates to clients.
Documented new features and technical details to provide clear and comprehensive resources for both the
sales and engineering teams.
Created an efficient and budget-friendly testing environment using Terraform, Docker, and GitHub Actions,
streamlining infrastructure management and deployment processes.

Designed and implemented real-time and batch data pipelines using Kafka, PySpark, AWS Glue, and EMR,
•
•
•
•
•
•
•
processing over 10k messages per second with Airflow-based orchestration.
Built scalable streaming and storage solutions by integrating Kafka, DynamoDB, and Elasticsearch, enabling
efficient querying across structured and unstructured data.
Designed and implemented a Data Warehouse using AWS Redshift and AWS S3, optimizing schema design and
query performance.
Implemented monitoring and observability using AWS CloudWatch and Azure Monitor, ensuring system
health and proactive issue resolution.
Developed and orchestrated ETL pipelines using ADF, integrating multi-source data into ADLS.
Utilized Azure Event Hub for real-time data ingestion, complementing Kafka-based streaming in a multi-cloud
architecture.
Contributed to Data Governance initiatives, improving data quality, visibility, and pipeline monitoring.

•
• Used Beautiful Soup with Python to scrape multiple websites, gathering and organizing relevant information.
Automated data loading process, increasing efficiency by 15%, by implementing a Cron job for database
integration.
Collaborated with the team to handle data validation and ensure the accuracy of the scraped data before
•
database loading.