رافي Teja

Currently leading end-to-end data pipeline engineering and forecasting solutions, owning 10-12 pipelines across ADF and
Databricks with zero major productionfailures.
• Cloud Data Pipeline Engineering: Designed and maintained 10-12 end-to-end ADF pipelines covering ingestion,
transformation, and delivery across Dev, QA, and Prod environments; integrated Control-M scheduling to ensure
SLA-compliant weekly deployments.
• Distributed Computing & Performance Optimisation: Optimised PySpark transformations and introduced multithreading
and Spark partitioning strategies, cutting pipeline execution from 20-25 hours to 5-8 hours and model cycle time from 4 hours
to 50-60 minutes.
• Cloud Data Warehousing & Lakehouse Architecture: Architected a Delta-based precomputed result storage integrated
with Snowflake and ADLS Gen2, implementing Medallion Architecture layers across bronze, silver, and gold zones to
support 2 downstream teams.
• ML Pipeline Integration & Generative AI Workflows: Developed time series forecasting models (XGBoost, ARIMA,
Holt-Winters) improving prediction accuracy by 15-20%; built PySpark pipelines integrating MLflow and LLM APIs to
automate business summarisation workflows.
• Data Quality, Governance & CI/CD: Implemented data validation rules (Azure Purview and Databricks Unity Catalog) for
lineage tracking and compliance; managed CI/CD pipelines via Azure DevOps and Git across all deployment stages.

Built 7-9 ELT pipelines processing 300M-500M records per month across Snowflake and ADLSfor enterprise analytics in P&G
and telecom domains.
• Cloud Data Pipeline Engineering: Developed a 7-9 ELT pipelines ingesting data from 5-6 source systems into Snowflake
and ADLS, utilising UC4 scheduler for orchestration and ensuring timely data loads across reporting workloads.
• Distributed Computing & Performance Optimisation: Refined PySpark jobs in Azure Databricks using window functions,
advanced aggregations, and execution tuning, reducing scheduled pipeline execution time and resolving bottlenecks across
5-6 member engineering team deliverables.
• Cloud Data Warehousing & Lakehouse Architecture: Consolidated Snowflake data loading operations using UC4,
automating ingestion tasks and designing data models to support BI analytics and reporting for enterprise clients across P&G
and telecom verticals.
• Data Quality, Governance & CI/CD: Established data validation rules and error logging within PySpark jobs, cutting
downstream reporting errors (by 35-40%) and raising data accuracy to 99%; version-controlled ETL scripts via SVN for
team-wide consistency.

Monitored and maintained ADF and
Hadoop-based data pipelines, resolving 25-30 incidents per month at 95%+ SLA compliance and reducing recurring failures
by 30-35% through root cause fixes.