Data Engineer / Data Developer
Trans Mountain Pipeline
Total des années d'expérience :10 years, 4 Mois
Project: Land Communications Stakeholder system Responsibilities:
●Overseeing the migration from a legacy application into third party application, saving manual efforts in data load with a performance increase of 14%
●Working with the stakeholders to understand business needs and translate those business needs and translating those needs into actionable reports in Borealis (third party application) which has an in build reporting system, saving 18 hours of manual work each week.
●Created ADF monitor service to observe ADF Resource, Integration runtime, Trigger runs, Activity, and Pipeline runs.
●Carried out Data Orchestration by automating all the built pipelines and Data flows in ADF to process all data without manual intervention.
●Managed a team of 1 full-time employee and 3 contractors, and frequently collaborated with project managers and analysts
●Building, Designing, Construction, Maintenance of SSIS packages and scheduling jobs to migrate data from disparate sources into SQL server and vice versa. ISMAIL PAGE 2
●Performing root cause analysis on failures. Resolve performance tuning related issues and queries.
●Data Cleansing, Extraction and modification of data from and to different data sources like flat files, databases, csv, excels. Leverage on Power Query for Data Transformation.
●Working with Business Analyst to understand the requirement and building logic to load packages.
●Performing analysis to assess quality and meaning of data, finding inefficiencies, design and implement improved processes. Key achievement: Developed SQL scripts for query automation to remove the need for human interaction when executing SQL queries, this increased the volume and accuracy of queries being ran by 64%.
Project: Creation of an Enterprise Warehouse and Analytic Environment Responsibilities:
●Drove initiatives to enhance Automation of ETL processes across billions of rows of data, which reduced manual workload by 29% monthly
●Communicated with project managers and analysts about data pipelines that drove efficiency KPIs up by 26%
●Designed Power BI data Visualization utilizing Maps, scatter plots, pie, bar and density charts.
●Designed and created de-normalized dimension tables, fact tables, and entity-relationship using SQL Server Database Diagram and Star and Snowflake Schema Dimensional Modelling approach.
●Produce Source to Target Mapping, Transformation, and Data Quality documentation for client projects.
●Designed SSIS packages to extract, transform, and load (ETL) data from CSV files to Staging and Production SQL Server Databases using OLE DB and Flat File connection managers, Foreach container, Lookup, Merge Join, Merge, Conversion and Derived Column.
●Developed, test, and deploy Multidimensional and Tabular Analytic Databases, calculate measures and dimensions using SSAS.
●Developed Power BI reports and dashboards from multiple data sources
●Created SQL Objects such as Tables, Views, Store Procedures, functions, and indexes. Key achievement: Communicated with project managers and analysts about data pipelines that drove efficiency KPIs up by 26% leveraging the efficiency of process enhancement.
Responsibilities:
●Developed reports and dashboards from multiple data sources using data blending technology utilising Power BI.
●Management of the Vice President's Operation Command Centre for the economic and power sector dashboard.
●Worked with Business Intelligence manager to plan BI strategies, create BI road map and budget plans to deliver on organization Business intelligence needs.
●Performed Ad hoc Queries for Sales and Operations team.
●Worked closely with the Sales, Finance and Operations team to build investors' dashboard and drove initiatives to enhance automation of data reporting across the organization.
●Improved reporting and simplified overall communication by using Slack, Teams and Trello.
●Improved planning and overall efficiency by introducing story mapping and backlog refinement.
●Participated in requirements gathering and brainstorming sessions with project team members.
●Transformed and cleansed the source data using Power Query Editor and Excel Power Pivot.
●Wrote DAX expressions to calculate required measures for data modelling. PAGE 3 Key achievement: Refactored ssis packages by implementing the use of variables and parameters and redesigned a new dashboard resulting in a 15% time decrease in medical record analysis.
Responsibilities:
●Worked with cross-functional teams to structure problems, identify appropriate data sources, extract data and develop integrated information delivery solutions.
●Played key roles in medium to large projects (e.g. data modeller).
●Participated and provided technical leadership in all phases of BI projects: from discovery and planning through implementation and delivery.
●Performed cleansing, deduplication, and harmonization of data across various systems.
●Supported the planning and coordination of data governance activities.
●Developed and maintained an enterprise data architecture roadmap to improve the maturity and effectiveness of enterprise data management
●Coordinated daily updates and help team members to resolve impediments (scrum master)
●Documented software defects in confluence and report findings to the team. Key achievement: Trained Junior Analysts in data transformation and cleaning using excel and power pivot, which boosted and improved data quality by 20%.