Prabhat Yadav

1. Project : SAAS and Data Platform
Duration : July 2024 - Present Role : Senior Data Platform Engineer
Aimed at developing a centralized SaaS platform for RTA's ECCD department, enabling real-time monitoring, control, and management of road and transportation infrastructure. The platform enhances fleet maintenance operations through predictive analytics, failure detection, and operational insights, integrating external data sources such as weather and traffic conditions.
• Developed high-performance real-time data pipelines leveraging Kafka and Spark Streaming, efficiently distributing data to sockets, Memcached, and MongoDB.
• Integrated diverse external data sources using custom data connectors, enabling scalable and real-time processing for transportation analytics.
• Built an AI-powered automated reporting system using LangChain, enabling stakeholders to generate operational summaries, compliance reports, and incident insights by analysing structured and unstructured data.
• Executed the migration of PostgreSQL from on-premise VM to Oracle Cloud Managed PostgreSQL, ensuring minimal downtime and complete data validation across 200+ tables.
Technologies: Spark, Python, Kafka, Scala, MongoDB, Sockets, REST APIs, OpenShift, Shell Scripting, LangChain, Postgres

2. Project : United Airlines (Migration + Development)
Duration : April 2023 - July 2024 Role : Senior Data Platform Engineer
The project aimed to enhance inventory and revenue management by improving inventory allocation efficiency and maximizing revenue through advanced business strategies. It involved migrating from a legacy data warehouse to a cloud-based solution, covering discovery, design assessment, and full implementation using AWS and Snowflake.
• Developed scalable ELT pipelines using PySpark and Snowflake, efficiently processing multi-year historical datasets for inventory forecasting.
• Designed and deployed a Streamlit-based self-service analytics application, integrating ChatGPT, Snowflake, and DBT, enabling business users to query and visualize data without SQL expertise.
• Built a Python-based data validation framework, ensuring 99% data accuracy and consistency during cloud migration.
• Engineered a scalable data processing platform, supporting both real-time and batch workflows using AWS services (EMR, Lambda, S3, DMS, Step Functions).
• Built interactive Power BI dashboards for operational reporting, combining real-time and historical data models to support business decision-making.
Technologies: Snowflake, AWS, CloudFormation, Snowpark, Streamlit, PySpark, Python, Informatica, DBT, Git and GitHub.

Working as a Data Platform Engineer in Nagarro, Bangalore.

Working as Data Engineer in TCS.

The project was to implement application for User data & Complaint Analysis.
• Developed and maintained real-time data ingestion pipelines using Spark Streaming
and Kafka.
• Introduced Schema Registry Integration to improve the performance of pipelines
and reduce the manual intervention.
Technologies: Hive, Kafka, Spark-streaming, Scala.