Big Data Consultant
Al Rajhi Bank
مجموع سنوات الخبرة :14 years, 5 أشهر
Responsible for developing and managing data pipelines in the Cloudera Hadoop cluster, exploring and performing POC using the new HPE Ezmeral Data Fabric Cluster. Some of the other key job functions include but not limited to
Understanding the business requirements, needs and drawing the roadmap for big data platform.
Design, Implement and Maintain ETL/ELT processes and integrating data from different sources.
Exploring HPE Ezmeral Data Fabric Database and Event Streams for different use cases.
Creating and batch and streaming jobs using Kafka, Spark and storing the data into HPE Ezmeral Data Fabric data lake.
Developing web services, REST APIs to connect and interact with different systems.
Understanding the business requirements, needs and drawing the roadmap for big data platform.
Design and Develop streaming processes to enable real-time data ingestion, processing and transformation.
Create and operate streaming jobs and applications with Spark Streaming; integrate Spark Streaming with other Spark APIs.
Explore system interface design using APIs, REST, and pub/sub systems.
Configure OOZIE workflow engine to automate different types of Hadoop jobs such as Map Reduce, Hive, Pig, and SQOOP.
Development of Hive Scripts for data analysis and processing.
Building Batch and Streaming data pipelines using Spark APIs.
Creating and operating streaming jobs and applications with Spark Streaming
Debugging, monitoring, and tuning Hadoop/Spark cluster and applications.
• Implemented and maintained application modules for Web-based database applications using MVC Architecture.
• Used JSP, Servlets, JDBC and Java Beans.
• Designed and developed SQL queries, DB scripts, and store procedures.
• Test Case preparation and execution.
• Worked with QA and Business team for defect fixes and release activities.
Worked with Technical team members in interpreting and elaborating requirements into design specifications.
Requirement analysis and analyzed workflow for application.
Database and User Interface design.
Involved in business logic development.
Involved in interaction with client to set up environment ready for any new component, it includes making web server ready and raising all connections required.
Worked with development and testing team for any deployment.
To make sure that all web servers are up and running properly.
Analysed and developed deployment management system (in house project).
Big Data, Business Analytics, Distributed Computing, Parallel Processing