Experience in Azure Data Factory (ADF) creating multiple pipelines and activities using Azure for full and incremental data loads into Azure Data Lake Store and Azure SQL DWWorking experience with DatabricksSpark framework experience as well as experience optimizing Spark queriesExperience for Azure Data Lake Storage and working with Parquet files and partitions.
Azure experience in designing and implementing redundant systems, policies, and procedures for disaster recovery and data archiving to ensure effective protection and integrity of data assets.Manage and resolve database access and performance issues
Hands-on experience in building ETL/ELT solutions using Apache Spark
Experience working with python specifically pySpark framework Good written Spark SQL experience is required
Experience working with Spark RDD’s and DataFrames
Strong experience in python 3 development Design, develop test, deploy, maintain and improve data integration pipeline.Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes etc.Experience in data wrangling and strong SQL skills.Monitor, optimize and allocate physical data storage for database systems.Working technical experience with designing, building, installing, configuring and supporting Microsoft SQL ServerAzure Key Vault and Azure Policies is a plusAzure certification would be desiredExperience in architecture and implementation of large and highly complex projects.E-mail id :