Lead Data Engineer
ABSA
مجموع سنوات الخبرة :9 years, 5 أشهر
Lead data engineer in Physical Channels team responsible for building data pipelines and maintenance.
● Involved in a big project as lead engineer where the bank migrated all their data from ESP system to Customer1View for RBB till July 2020 which will was eventually decommissioned.
● Was Involved in CIB project moving data from front arena and Pegasus systems data to Hadoop and Postgres using Kafka, Spark, and Scala on CA Work Automation Load Client
● Develop Kafka producer and consumers as per data ingestion request/specification. ● Utilize various Spark transformations and actions to cleanse data. Write Map Reduce jobs using Java API
● Design, develop and execute automation scripts in python and scala that automate migration of data from SQL servers’ sources, mainframe, various internal systems such salesforce, branch systems and Customer1View.
● Use ETL tools such as Streamsets, Menas, MS SSIS, Informetica, Jitterbit and CA Wade, AWS Glue
● Building of AWS data-pipelines, storing data in S3, creating python lambda functions to move data to Redshift.
● Develop Hive scripts for business help with scripts that show them ad hoc progress on the day-to-day business analysis required from the data lake.
● Cluster coordination via Zookeeper ● Use Spark & Spark SQL to read parquet files and create tables in Hive using Scala API. ● Extract real time feed using Kafka and Spark streaming, convert to RDD, process as data frame and store as parquet in HDFS.
● Use HDFS & Linux commands, Pyspark, Scala, Hivesql, java and python to develop most of the solutions.
Design, develop and execute automation scripts that automate branch audits done manual using big data from Hadoop data lake.
● Investigate and research solutions that could be used in the group internal audit space to create new innovative solutions.
● Mine and analyze data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.
● Create Hive scripts to automate audit processes to our robotic engine. ● Data Analysis and warehousing designs on new features brought forth by Audit Executives. ● Use SQL Management Studio & Visual Studio Data tools to create SSIS packages
● Statistical computer languages (R, Python, SLQ, etc.) to manipulate data and draw insights from large data sets using libraries such as pandas.
● Coordinate with different functional teams to implement models and monitor outcomes. ● SQL Scripting and ERD designing (snowflake and star schema) for Power BI dashboards.
● Building of AWS data-pipelines, storing data in S3, creating python lambda functions to move data to Redshift and more.
● Collecting data using Hive scripts on central data lake to create termly reports in the form of website generated audit reports, scorecards and dashboard visuals on Power BI.
● Managing business accounts, making sure invoices and statements correspond always for our weekly reconciliation report.
● Distributed data/computing tools: MapReduce, YARN, HDFS, Hive, Spark, MySQL etc. ● Line manager to Data Analysts and developers under my portfolio.
● Design, develop and execute automation scripts that automate branch audits done manual using big data from Hadoop data lake.
● Investigate and research solutions that could be used in the group internal audit space to create new innovative solutions.● Create Hive scripts to automate audit processes to our robotic engine.
● Web Development (Python, Django framework, HTML5, JavaScript, and CSS) for an internal job scheduler website.
● Data Analysis and warehousing designs on new features brought forth by Audit Executives. ● Use SQL Management Studio & Visual Studio Data tools to create SSIS packages ● Code logic using SQL to create SSAS packages used for SSRS reporting. ● Maintenance of Customer 1st system used for branch and corporate data capturing throughout the bank. (Oracle SQL, Python)
● SQL Scripting and ERD designing (snowflake and star schema) for Power BI dashboards. ● Java and Selenium Automation Testing on our automation engine for full regression testing. ● A wealth of expertise in several bank systems such as Thomson Routers, Teammate, ECM products, IBM FILENET, NDS.
● Documentation of Systems with User Manuals used throughout the Group Internal Audit and flow diagrams for devOps teams.
● Collecting data using Hive scripts on central data lake to create termly reports in the form of website generated audit reports, scorecards and dashboard visuals on Power BI.
● Managing business accounts, making sure invoices and statements correspond always for our weekly reconciliation report.
● Meeting with the team to interpret concepts and note requirements. Responsible for 3 Graduates upskilling in Hadoop scripts (hive and phoenix), data warehousing and basic TSQL mentoring.
I create documentation such as User Stories, Business process
maps and Issue Reports used by iStore Sales team.
Star schema, Snowflake designs and methodologies for
dimensions and fact table to develop data warehouses.
Use SQL Management Studio & Visual Studio data tools to
create SSIS packages
Code logic using SQL to create SSAS packages used for SSRS
reporting.
Gathering requirements and implementation of solution.
Run SQL queries on Excel to clean up/update database tables
with new data or new categorizing conditions for inventory,
sales or statistics tables on our Syspro server.
Use VBA code to automate data analysis requirements on
given worksheets and convert out to pivot tables.
Lias with external developers from EOH to
Test cases and supporting documentation for test done on
system.
Use of Bi tools such as Power BI and Qlikview
Data science with Excel and python libraries such as Panda,
numpy
Assist with Salesforce querying for Marketing team
I create documentation such as User Stories, Business process
maps and Issue Reports used by PwC software developers.
Second Tier support for software developers and the service
desk consultants.
Defining a solution approach to determine the nature and
extent of technology and business process changes to be made
as part of implementing the solution to the primary business
objectives.
Gathering requirements and implementation of solution.
Provide new system training to the Service Desk consultants.
Implement advanced strategies for gathering, reviewing and
analyzing data requirements for product owners and
stakeholder.
Conduct grooming sessions with the software development
technical leads.
Test cases and supporting documentation for test done on
system.
Liase with Product Owners
Log user queries with developers
Excel (Pivot tables, data connections)
Gather Requirements
Databases Architecture
PC remote access assistance
Liaise with Developers and PMs for software deliverables Assist Test Analyst with system tests and upgrade tests (UATs)
C# (ASP.Net MVC5 web development)
Web Development Fundamentals (XML, HTML, JavaScript, OOP, RDBMS)
Software Engineering
Systems Analysis and Design (Adobe Ps)
Java Web Programming Fundamentals(Servlets and JSP) Advanced JavaScript
SQL Scripting
Grade 73% overall average
A levels with Cambridge Internation Examination