Ayushi Malviya

The NEMO platform, hosted in Azure, enables the advanced analysis of
employee-related data through predictive analytics and machine learning.
This analysis facilitates the identification of key performance indicators (KPIs)
and other relevant metrics. These metrics are subsequently utilized to predict
attrition outcomes, including forecasting attrition rates and providing
valuable insights for market analysis.
Data is ingested from multiple sources like SFTP, APIs, AWS S3 bucket to ADLS.
Synapse runs pyspark jobs to load data to raw layer in ADLS implementing
Pseudonymization and data quality checks. Orchestration is done through
Synapse Pipelines which loads the data to staging and target layers (SCD2
and CDC) following Medallion architecture. Semantic Layer with BU Security
is implemented in Azure SQL on top of target layer. Previously worked on
pipelines and notebooks in ADF, Databricks which later migrated to Synapse.
Key Roles:
• Developed comprehensive end-to-end pipelines for data refining.
• Solely responsible for all data-related concerns, transformation, and
scenario handling.
• Developed the Reversal script (De-Pseudonymization) from scratch,
providing a generic solution for decrypting ids.
• Responsible for translating business requirements from the LBU to the
DS team.
• Worked in Agile and Waterfall Methodology.
• Continuous collaboration with several team members, including
LBUs, DS, DevOps, Data Architect, and other DE in the project.

A professional with over 8.5 years of experience in designing and developing on-premises and cloud-based data products. Proficient in working on distributed system environments and leveraging Big Data Cloud stacks to efficiently process substantial volumes of data from diverse sources.
Currently working as Consultant with T-systems Private Ltd. on NEMO project to ingest and transform data utilizing Synapse. ADF, Databricks, Spark, Azure SQL, Python, and ADLS for processing of data. Spark jobs have been optimized for improved performance. Migration has been done for existing jobs to Azure Synapse.
Former Senior Data Engineer with Saama Technologies to develop the Patient Data Lake specifically designed for the medical research in pharma domain. Developed Pyspark jobs in Dataproc cluster to process blinded and unblinded studies various source files from GCS to load data to Bigquery. All jobs were scheduled through Cloud Composer (Airflow).
Previously worked as Senior System Engineer in Infosys on Big Data Solutions to organize and manage Data Lake. Streamlined the data ingestion process using Apache Spark and Python for optimized data loading.
Strong technical Knowledge of advanced python libraries like Pandas, Multiprocessing, Multithreading.
In-depth understanding of On-Premises Big Data solutions like Hadoop, MapReduce, Apache Spark, Apache Hive, HDFS, Hbase, Apache NiFi, Sqoop, Apache Kafka.
Open to relocate |Immediate joiner

Client Enterprise Data Lake (CEDL) (Infosys) (Aug ’17- Jan ’19)
This project was to streamline the Ingestion process for a Data Lake Server. Data was ingested from various sources into a raw
directory in local System in UNIX server using Pandas Data frames and then moved to HDFS on Data Lake. Reading that data from
HDFS directories into Spark Data frames and then applying various transformations to generate final copy of data which is loaded
into partitioned Hive tables stores as ORC files.
Now merge layer is created from loading data from input hive tables to output hive tables by joining various input tables and then
applying transformations to load data in integration layer. Performance tuning is done on queries to optimize joins, and partitioning
was done in this final layer. Integration layers serve as a single source of data for whole organization and make it available to various
data analytics platforms on premise or on cloud for further cleansing.
Scheduling was done in control-m to trigger all jobs in various layers.
Key Roles:
• Ingesting Data into Data Lake and doing data cleansing in ingestion layer.
• Applying transformations in Spark Data frames to load data in Hive tables.
• Performance tuning of hive queries for merge layer.
• Pushing, Pulling and syncing data to bitbucket, and coordinating with other team members.