Big Data Architect
Dana Gas
مجموع سنوات الخبرة :9 years, 10 أشهر
Dana Gas is the Middle East’s first and largest regional private sector natural gas company, with offices across UAE, Egypt, IRAQ and other middle-east countries. As a Big Data Hadoop architect my primary focus was to evaluate, design, develop, test and maintain big data solutions.
Provided Leadership role to solve organization’s big data challenges (storage, processing, high velocity and real time streaming).
Applied skills in modeling, design and development of big data solutions.
Bridged the gap between functional and technical teams especially in an agile development environment
Managed and trained highly talented big data engineers for real time big data solutions (Assign tasks, monitor the team performance and create test cases for improving performance of team members if needed).
Requirement gathering, designing and developing scalable big data solutions with Hadoop.
Proposed architecture with Proof of Concept (POC) documentation for big data.
Responsible for implementation and ongoing administration of Hadoop cloudera infrastructure.
Big Data Hadoop capacity and architecture planning: Achieving Right Sized Hadoop Clusters and Optimized Operations.
Aligning with the systems engineering team to propose and deploy new hardware and software environments required for Hadoop and to expand existing environments.
Installation of Apache Hadoop Framework (HDFS, MapReduce, HBase, Hive, Pig, Impala, Flume, Sqoop, Oozie, and Kafka) Cloudera now having 45 nodes with data replication of factor 3.
Responsible for upgrading Cloudera Manager and CDH up to date.
Translating (SQL, Oracle, SAP and transition logs) use cases into Hadoop Cloudera integrated with (Sqoop, Flume and Quest Sqoop plugin).
Design, implement and deploy ETL to load data into Hadoop/NoSQL.
Integrated Dana gas Data warehouse with big data platform.
Working with data delivery teams to setup new Hadoop users. This job includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.
Responsible for Hadoop day to day operations, support and maintenance as agreed organization SLA.
Cluster maintenance as well as creation and removal of nodes using tools like Cloudera Manager Enterprise.
Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required.
Designed and developed MapReduce programs in Java by using agile software development principles.
Performance tuning of Hadoop clusters and Hadoop MapReduce jobs.
Screen Hadoop cluster job performances, capacity planning and Solr search.
Implemented and monitor Hadoop cluster connectivity and security.
Manage and review Hadoop log files for minimizing downtime.
Prepared insights from unstructured data for reporting production data into Qlikview.
File system management and monitoring in Hadoop.
Diligently teaming with the infrastructure, network, application and business intelligence teams to guarantee high data quality and availability.
Captured and analyzed data with AWS services ( Machine Learning, Lambda, QuickSight, EC2, and ElesticSearch)
DevOps Tools (Chef, Docker, Puppet, Bamboo, Jenkins)
Strong hands-on knowledge of Big Data technologies, such as:
HDFS and related technologies (Avro, Parquet, Snappy)
Map Reduce (Used Java for development)and Yarn
Spark and Spark Streaming (Used Scala programing language)
Kafka, Flume, Sqoop
Hive, Pig, Impala
Oozie
Independently manage and develop Data Analyst projects and work streams.
Design and deployed Splunk for Operational Intelligence at Abu Dhabi pension fund project
Create services on SOA architecture by using java which update, modify, insert, and delete records in their talk servers which uses progress DB in Abu Dhabi Retirement Pensions & Benefits Fund (ADRPBF) and proposed to handle Big Data challenge.
Develop BI and employs performance system by using Microsoft SharePoint.
Prepare and present deliverables and proposals to client's top management and project teams
Build lasting client relationships in cross-cultural environments
Structure, provide and lead analytical solutions and statistical expertise on client engagements and internal projects, develop frameworks together with clients
Apply statistical, optimization, and other types of analytical models to real world business problems
Lay out an analytical framework and a plan to execute solutions using appropriate tools
Interpret and put into context analytical findings for impacted business stakeholders; develop recommendations and quantify their impact
Actively lead and manage internal activities related to knowledge management and people management
Developed, design, test and apply Java applications
Created right code structures to resolve specific tasks
Unit Testing Java components
Developed system features based on system requirements
Delivered proper guidance and support the overall features of the system
Provide Development and Production support for databases. This includes creating primary database storage structures; designing and creating primary objects; modifying database structure as necessary for enhancements or performance; monitoring and optimizing performance of the database; planning for backup and recovery of the database; allocating system storage and planning future storage needs; and creating database-related scripts and programs to support development and production environments.
Execute deployment scripts
Monitor DBs
Perform capacity management planning
Review logs for compliance issues with people attempting to access the DB
DB errors, thresholds, indexing, performance monitoring and management
Review and apply weekly application releases
Release management
Testing in Dev / QA staging environments
SQL Server Upgrade and Patching
Monitor SQL Agent Job Failures
Change Management for object Modifications
Environment Availability Monitoring
Escalations and Alerts
Problem and Incident Management
Backup and Recovery
Numerical Computing Random Variables and schotastic process Information Theory Research Design and Inferential Analysis Advance Mobile Communication Advance Broadband Communication Advance Satellite Communication Network Management Network Security Network Technology
Four year BS.Telecommunication with CGPA 3.66, persentage 74% CERTIFICATE OF INTERNSHIP Working as a Internee At PIA Where I worked, Hardware maintenance, Software Development, PC Troubleshooting, Networking, Data & Massage Switching. From 13th June to 27th June 2006 CERTIFICATE OF INTERNSHIP Working as a Internee PTCL in Fiber Optics Dept: Where They Tell Me about Splicing, Sonnet, Multiplexer, BN-Switches and WLL (Wireless Local Loop). From 9th July 2004 to 10th August 2004
61.8%
68%