آرون جيجو

Big Data Engineer في Exl Clairvoyant

• الهند
• أشغل هذه الوظيفة منذ فبراير 2022

Autonomus Insights Detection
• Developed and implemented an end-to-end, configuration-driven framework for autonomous insight detection.
• Utilized Apache Spark to generate trackers based on predefined Key Performance Indicators (KPIs).
• The framework automatically analyzes the defined KPIs using Spark SQL transformations and employs time series jobs for advanced forecasting.
• Deployed and managed Spark jobs within Azure Databricks, and orchestrated workflows using Azure Data Factory (ADF).
• Executed two distinct proofs of concept: one employing Apache Kafka for real-time data streaming and another for orchestrating workflows in Databricks, enhancing data processing efficiency and operational agility.
• Enhanced the framework's cloud agnosticism by transitioning from Azure Data Factory to Apache Airflow, facilitating broader compatibility across different cloud environments.
• Achieved cloud cost optimization by refining job execution and cloud infrastructure management.
• Established a Continuous Integration/Continuous Deployment (CI/CD) pipeline using Azure DevOps to streamline development and deployment processes.