Suvigya Jain

Client: Largest Government Entity in Dubai, Government Agency for Insurance in KSA, Smart City in KSA, Wealth fun d organization in KSA, Largest Bank in Dubai, Properties Management in Australia

• Architected and delivered a comprehensive end-to-end data platform, integrating 30+ data sources, optimizing ETL and data pipeline processes, and implementing real-time analytics. This resulted in a 40% reduction in data processing time, enhancing operational efficiency and enabling data-driven decision-making across the organization.
• Designed and implementing AI-driven frameworks leveraging Generative AI and Agentic AI, enabling intelligent automation and adaptive learning for enterprise-scale applications.
• Engaged with key client stakeholders to deeply understand business processes and data workflows,
translating insights into strategic, high-impact data solutions.
• Designed and led data strategy initiatives, guiding clients on transforming their data ecosystems to
drive innovation, scalability, and improved performance.
• Conducted multiple executive sessions with clients, providing expert guidance on data architecture, best practices, and implementation strategies, ensuring alignment with industry leading standards.
• Implemented CI/CD and DataOps best practices using tools like GitLab CI/CD, Terraform, Apache Airflow, dbt to data pipeline deployment, ensure scalability, and enhance data reliability.
Optimized workflows, reducing deployment time and improving data quality.
• Defined and implemented a scalable data architecture, enabling the seamless delivery of complex use cases on a Lakehouse infrastructure with Data Mesh, achieving a 45% reduction in data processing time and ensuring long-term scalability.
• Identified critical gaps in existing architectures and established best-in-class data engineering practices, optimizing system performance and reliability.
• Led and managed a high-performing team of 40+ professionals, spanning multiple streams, fostering collaboration, mentorship, and execution excellence.
• Directed end-to-end project delivery from onsite, leveraging strong communication, stakeholder engagement, and negotiation skills to resolve conflicts, ensure seamless execution, and achieve project milestones on time and within budget.
• Spearheaded client pursuits and business development initiatives, leading proposal writing, client presentations, and strategic engagements to secure new business opportunities and strengthen partnerships.

Technologies: Iceberg, Airbyte, Dremio, Minio, dbt, Airflow, AWS, Azure, GCP, Big Query, Snowflake, Synpase, Azure Data Factory, Glue, Apache Spark, Hive, Apache Hundi, IICS, Informatica IDQ, Cloudera, Tableau, OCI, Terraform, Github.

Client: R & D Data lake for a leading American multinational Bio-Pharmaceutical company

• Designed, implemented, and deployed a centralized sequencing repository with predefined analytics, enhancing Research & Development capabilities.
• Engineered an AWS Cloud Data Lake architecture for NGS Sequencing, addressing critical aspects such as data transfer, analytics, and validation, optimizing research workflows.
• Identified and addressed technical gaps in R&D data lake tools, working closely with vendors to validate and implement improvements.
• Developed and optimized complex Snowflake queries for sales reporting, improving performance and data-driven insights.
• Led and managed cross-functional onshore and offshore data teams, ensuring successful project execution while strengthening client relationships and stakeholder engagement.

Technologies: AWS Glue, boto3, AWS Batch, Serverless EMR, AWS s3, React.js, Snowflake and Labkey.

Client: Functional Data Lake for a leading Indian multinational Conglomerate company

• Designed and built a data lake from the ground up for energy, petrochemicals, natural gas, and retail sectors, integrating cross-functional data from Finance, Manufacturing, HR, and Administration.
• Developed innovative, scalable solutions with functionalities such as restartability and robust validation framework, supporting diverse data types including Batch, Real-time CDC, IoT.
• Enabled real-time IoT data streaming via Kafka Spark Streaming, enhancing defect detection in manufacturing. Implemented MySQL-based offset management for advanced restart capabilities.
• Architected and automated data ingestion pipelines using Spark-Scala and Shell scripts, optimizing data loading into HDFS and Hive from RDBMS (using Sqoop). Established the ABC framework for error logging, failure recovery, and automated job restarts, ensuring resilience and efficiency.
• Pioneered real-time Change Data Capture (CDC) solutions using Kafka Spark Streaming, enabling seamless incremental data loads and efficient storage in Phoenix, HBase, and Hive.
• Led and managed a diverse team of Data Engineers, Data Analysts, BI Developers, and Data Modelers, driving successful project delivery and innovation.

Technologies: Hortonworks 3.0, Hive, Spark, Shell, Jenkins, MySQL, Spark Streaming, Atlas, Ranger, Kafka, Phoenix, Hbase, Jenkins and Centos.

Project: Campaign Funnel and Finnone - Data Lake

• Designed and built an on-premise Hortonworks Data Lake with a robust data model and analytics framework, enabling batch and streaming data ingestion from diverse banking sources, including Core Banking Systems, Net Banking, Card Systems, legacy CRM, and Digital Marketing channels (SMS, Email, Clickstream).
• Empowered banking functions and internal teams by leveraging the data lake for advanced marketing campaign design, personalized product recommendations, and sales lead generation, optimizing advertising strategies and enhancing customer engagement for Relationship Managers.

Technologies: Ambari, Hortonworks (Hive, Pig, HDFS, Sqoop, Spark Sql), RHEL6.5, SQL server 2016,
Pentaho PDI, Informaica, Oracle, AIX and Autosys.

Project: Opinio - Social Media Sentiment Analysis Tool & Digital Solution (Saint-Gobain)
• Designed and implemented a real-time data pipeline for ingesting Twitter data into HDFS, performing data cleaning and analysis using Hive and Pig, and conducting sentiment analysis with the SO-CAL algorithm. Integrated sentiment insights into Elasticsearch and developed interactive visualizations in Kibana, enabling real-time social media analytics.
• Developed and automated enterprise-scale data processing workflows using SSIS packages, leveraging advanced SSIS tasks such as For Each Loop Container, Execute SQL Task, FTP Task, Data Flow Task, Send Mail Task, XML Task, Execute Package Task, Execute Process Task, and Sequence Container, ensuring seamless deployment and process efficiency.

Technologies: Hadoop, Hive, Sqoop Flume, Pig, ElasticSearch, Kibana, SSIS.