Senior data engineer
Etihad Airways
Total years of experience :8 years, 11 Months
Project Description : To migrate Huge amount of data from Aims legacy system to new system
using Hadoop datalake. To replace existing aircraft movement feeds with a
new real time feeds to serve different downstream application.
Overview
It include end to end requirement gathering, data ingestion, data analysis
, processing of Etihad flight operation data.
Responsibilities • Worked on requirement gathering, business analysis and making high
level design for this project.
• Ingesting new data feed (adix format) in the form of XML into Etihad
data hub using apache kafka.
• Consuming real time data from kafka and processing using spark
streaming to parse and store in structured format.
• Creating near real time pipeline to create data marts, transform into
summarize data using hive for data vizualization.
• Create end to end workflow using zaloni(bedrock) to schedule end to
end ingestion, transformation and exporting data to sql server.
• Working on cloudera which is deployed on Microsoft azure .
• Supporting end to end projects for all the issues related to projects
• Analysis and Migrating huge amount of flight movement data from one
platform to another using hive and sqoop through bedrock workflow.
• Working in agile methodology having responsibilities as team lead to
design, develop and deliver project.
• responsible for developments, fixes, testing and maintaining
architectures, Hadoop/Spark processing systems.
• Providing technical support and monitor Big Data/Hadoop applications,
handle and identify possible production failure scenarios.
Solution
Environment :
Hadoop, hive, kafka, sqoop, bedrock, Sql Sever, spark streaming, Azure, python.
Operating System UNIX, Windows XP and Windows 7
Project Description This project provides a Solution to Chhattisgarh Police for Analysis of data
through various social media platform like facebook, twitter, mails etc which
will help police department to tract and analyse each and every activity on
social media.
Overview
It includes ingestion, analysis and storing of streaming data from various
sources like facebook, twitter, mails etc.
Responsibilities • Fetching real time data from various social media sources such as
facebook, twitter using Rest Api as well as streaming api.
• Ingesting and processing streaming data using apache storm and solr.
• Data Cleaning, Processing and Summarisation of data according to the
various ues cases.
• Build end to end solution to tract and store every post coming from
facebook, twitter etc to help police department analysis .
• Transformation and analysis of streaming data to find locations, name,
persons, organisation and other meaningful entities using open sources
Nlp machine learning models
• Storing processed data in apache solr to for index bases fast searching
which is used in various dashboards reports .
Solution
Environment :
Hadoop, Storm, Solr, Java, Linuxs
Operating System UNIX, Windows XP and Windows 7
Hadoop Cluster HDP(Hortonworks Data Platform) 2.3, 2.4
3: Project Title : Unum Data Migration
Organization Tata Consultancy Services
Client RBC
Type
Project Description The project is to develop a central authoritative data source, which will house
the data from owned and operated properties, and provide capabilities to
perform business reporting, data merging and query analysis on the data.
This consolidated data repository is expected to form the back-end for AOL
LLC’s decision-support, business reporting, as well as the Ad-targeting and
Ad-serving platforms interfacing with other applications forming an
integrated data unit.
Overview
It includes loading of raw data into Hdfs and use Hive and pig to extract,
transform and load data. Also migrated new process to AWS cloud using
AWS EMR, AWS S3, AWS EC2 and scripts using AWS CLI commands.
Responsibilities • Gathering the requirement from client and provide business as well as
technical solution.
• Creation and testing of Hive sqls unix scripts, Autosys jobs, scripts using
aws commands.
NA