Andrey Kozlov

New data marts for bank in Hadoop (parquet) or optimization of old data marts. Development environment IntelliJ Idea, PyCharm, DBeaver. Applications for dags are written on a Scala. Dags has different sensors, branches. Some key info gets into Postgres database through sql queries or api.

The result of my work is report with end-to-end analytics. The main data is taken from Hadoop with orc structure (count of records in one table about one billion) and is processed by scala+spark to new data mart. Then this data mart is loaded to MS SQL database with aid of scala+spark. Next, the data from MS SQL database is loaded to SSAS (tabular model) by ETL (SSIS packages). And in the end I made report over SSAS in Power BI. Jobs for starting scala programs in Azcaban - scheduler. Spark SQL in Zippelin. Development environment in IntelliJ Idea. CI/CD - git+teamcity.

Tasks in Oracle: process data for Sheremetevo airlines - statistics about passangers, aircrafts, pilots. Put processed data as messages to IBM MQ and output new data. Optimization of queries PL-SQL from execution time from some days to 30 seconds.

Tasks in T-SQL: loading data from different systems to DWH, automation of bank processes, automatic calculation of bond portfolios. Service Broker to exchange data with outside organizations. Some java development to organize small services. Get data from outside services. CI/CD - TeamCity/Git.
Data exchange by pipes(C#)/ServiceBroker: preparing buffer tables, queues, data marts, stored procs, configuring services.

Fast load by T-SQL from operational DB to DWH: simple recovery model, partitioning, parallel loads by four temporary jobs.