Senior Data Engineer
MTN Irancell
مجموع سنوات الخبرة :7 years, 7 أشهر
- Led development efforts of a big data platform from scratch
- Successfully handled 2+ billion/day records at peak rates of 50k/s using multiple PubSub pipelines
- Slashed analytical query response times from 25s to 10s on multi-TB data with innovative solutions
- Successfully led a huge transition of 30+ data pipelines to Airflow
- Led development of a core Django Backend and multiple FastAPI microservices
- Played key role in inception & supervision of the operation team, responsible for maintaining multiple Kubernetes, Docker Swarm and Clickhouse clusters spanning over 60 Physical Servers
- Received MTN Irancell “Gold Performer” award for outstanding performance in 2023
Our stack included:
CLICKHOUSE, POSTGRESQL, AIRFLOW, RABBITMQ, KAFKA, KUBERNETES, ORACLE, DJANGO, FASTAPI, ANGULAR, POSTGIS
- Deployed ELT/ETL pipelines on Azure for 20+ public and proprietary sources
- Optimized and tweaked pipelines using Apache Spark and Kafka
- Quickly adapted to Azure Cloud environment in under one month
Project stack included:
AZURE DEVOPS, AZURE DATA FACTORY, DATABRICKS, AZURE SQL, SPARK, KAFKA
- Architectured a Big Data Platform for social media analysis
- Pivot person in the design and development of a comprehensive crawling and data ingestion engine using pure Python
- Made several contributions to the AI/ML repository of project incl. NLP, text classification, sentiment analysis, social networks graph analysis, image classification, etc.
- Developed enhanced visualization solutions using PowerBI and React
- Designed and implemented the API gateway of the platform using Django
Technologies I've had hands-on experience:
- ELK stack for text storage and indexing
- HDFS / HBase / MySQL & PostgreSQL databases
- MinIO object storage
- KAFKA
- Apache AirFlow & Celery
- Prometheus & Grafana
- Kubernetes
- PowerBI
- Developed, deployed and optimized data ingestion and API gateway solutions using Python and Go
- Done thorough R&D about Elasticsearch optimization and enhanced ES query response times by 20%
- Made several contributions to the frontend React repo
Our tech stack included:
DJANGO, FASTAPI, ELK, KAFKA, KUBERNETES, AIRFLOW, PROMETHEUS, REACT, MINIO
Responsible for designing and implementation of a Big Data Analytics platform using Oracle Technologies and ELK stack
Responsible for Ingestion, Cleaning and Analyzing 300+ million records daily from different national banking & payment systems(SHETAB/ACH/RTGS/etc.) for detecting variousscenarios of fraud, money laundering and abuse
Responsible for recruitment, training and supervision of data analysts
Technical Head of CBI’s “Social Media Monitoring” Project -Analyzing 4+ million records daily from Telegram, Instagram, Twitter, and News Agencies using NLP and Social Network Analysis (SNA) techniques
Received Exceptional Talent Award