Muhammad Waleed Khan

Software Engineer at Northbay Solutions

• Pakistan - Islamabad
• My current job since June 2021

Software Engineer - Northbay Solutions, US, https://www.northbaysolutions.com/ 03-June-2021 - Current
Projects:
- UPIT | Role: Big Data Engineer
- I created and led the ETL pipeline for data migration from on-premises Oracle to cloud AWS S3 for Data Lake.
- Implemented Elasticity in the AWS Architecture which resulted in annual cost saving of $30000.
- Achieved Performance Enhancement of ETL pipeline for data migration of decreasing ingestion time by 7 hours.
- Developed the Email Notification System in Python using Boto3, AWS Lambda, Cloud Watch Logs, SNS and Step Functions.
- Developed rules engine to improve reliability of the data where we had to apply custom rules in Regex and SQL functions
on the migrated data before merging it into the data mart.
- Performed major refactoring of the existing code of AWS Glue jobs and AWS Lambda functions written in Python.
- Data Lake Automation Framework | Role: Big Data Engineer
- Helped the team in developing Big Data Architecture of the framework using Lake House Architecture on AWS.
- I created Source Data Mapping by creating ERD diagrams to present the flow of the data from the Source followed by
Bronze-Ingestion Layer, Silver-Curation Layer and Gold-Aggregation Layer.
- Created and managed Technical Documents for the project on Confluence.
- Created the Configuration files in JSON for the data migration glue jobs.