Pawan Kumar Singh

Lead Data Engineer with 7+ years of experience building and scaling modern data platforms across Azure and Databricks environments. Strong track record of leading data engineering teams through the full project lifecycle - from architecture design to production deployment - using Python, PySpark, and SQL for high-performance data processing. Experienced in implementing Medallion architecture, optimizing Spark workloads, and enabling self-service BI through Power BI. Known for stakeholder engagement, mentoring junior engineers, and delivering scalable data solutions that drive measurable business value across finance, retail, and logistics sectors.

Architected a Medallion Lakehouse (Bronze/Silver/Gold) on Azure Databricks and Azure Synapse Analytics, processing 2TB+ of data daily with 99.9% pipeline reliability using Auto Loader for automated ingestion.

Designed and built end-to-end ELT pipelines using Azure Data Factory, PySpark, and Delta Live Tables, reducing data latency from 6 hours to 2.5 hours (~58% improvement).

Optimized Delta Lake performance through Z-Ordering and Auto Compaction, achieving a 40% improvement in query performance and a 30% reduction in pipeline failures.

Implemented enterprise data governance using Unity Catalog across 50+ Delta tables, supporting 7 cross-functional teams and 200+ data assets with fine-grained access control and lineage tracking.

Delivered 15+ production-grade ELT pipelines on Azure Databricks and Azure Data Factory, supporting 100, 000+ customer records and improving data processing speed by 30%.

Built a robust data quality framework using Python and SQL, reducing downstream data errors by 95% and improving overall data reliability for business reporting.

Automated manual ingestion workflows, eliminating 50% of manual effort and saving 15+ engineering hours weekly, while enabling same-day credit risk decisioning for business teams.

Analyzed 500, 000+ daily transactions using rule-based fraud detection models, reducing false-positive cases by 50% and improving analyst efficiency.

Designed and deployed a near real-time fraud detection pipeline to flag suspicious accounts, accelerating investigation turnaround time by 40%.

Collaborated with risk and compliance teams to define fraud detection rules and thresholds, aligning technical outputs with regulatory reporting requirements.

Built monitoring dashboards in Power BI to track fraud trends, false-positive rates, and investigation SLAs, enabling proactive risk management for stakeholders.

Partnered with data science teams to integrate machine learning-based anomaly detection alongside rule-based models, improving overall fraud detection accuracy.

Analyzed 5 million+ daily digital wallet and payment transactions using SQL and Python, identifying 15+ distinct fraud patterns and proactively preventing fraudulent activity.

Automated 20+ monitoring and reporting workflows using SQL and Power BI, reducing manual processing effort by 80% and enabling faster, data-driven decision-making for risk teams.

Documented fraud pattern findings and shared insights with cross-functional risk and product teams, contributing to continuous improvement of fraud detection rules.

Engineered advanced SQL analytics using CTEs and window functions to process 1 million+ monthly transactions across 50, 000+ customer accounts for anti-money laundering monitoring.

Revamped 15+ compliance and AML dashboards, reducing report generation time by 60% and enabling faster, data-driven decisions for risk leadership.

Supported regulatory compliance reporting by identifying suspicious transaction patterns and ensuring adherence to AML/KYC monitoring standards.