Job description (Azure Data Engineer/ Senior Azure Data Engineer)
Designation: Consultant (B1/B2) or Manager (C1) or Senior Manager (C2)
Job Responsibilities
To collaborate with various teams/regions in driving facilitating data design, identifying architectural risks and key areas of improvement in data landscape, and developing and refining data models and architecture frameworks
Technical experience and knowledge in Cloud Data Warehousing, data migration and data transformation
Develop and test ETL components to high standards of data quality and performance as a hands-on development lead
Familiarity with Data Lakes, Data Warehouses, MDM, BI, Dashboards, AI, ML
Design data architecture patterns and ecosystems including data stores (operational systems, data lakes, data warehouses, data marts), ingress patterns (API, streaming, ETL/ELT), and egress patterns (analytics/decision tools, BI tools). Lead, consult or oversee multiple architectural engagements
Oversee and contribute to the creation and maintenance of relevant data artifacts (data lineages, source to target mappings, high level designs, interface agreements, etc.) in compliance with enterprise level architecture standards
Experience in leading and delivering data centric projects with concentration on Data Quality and adherence to data standards and best practices
Experience in data modeling, metadata support, development and testing for enterprise wide data solutions
Azure cloud experience is a must have with familiarity of the services: Azure Databricks, Azure Datafactory, Azure Datalake, Spark SQL, PySpark, Airflow, SQL server and Informatica MDM.
Additional exposure to GCP and AWS is good to have
Must have:
Bachelor’s degree in a technical field (Comp. Science degree preferred not mandatory); Master’s degree is a plus.
Expertise in ETL, Data architecture and Data modeling
Expertise working in Relational and Big Data platforms, including MySQL, and large-scale distributed systems such as Hadoop and Spark
Expertise in Data Engineering, Data Governance, Data Quality, Data Lake, Data Warehousing and Reporting/Analytics concepts
Expertise building Data pipelines, Data Lakes and Data Warehouses in Azure using Cloud services such as ADF, Synapse, Databricks Notebook etc.
Expertise in data analysis and writing and developing complex queries using SQL
Experience in designing Big Data/Cloud Solution Designs, data models
Knowledge of Pyspark, Shell scripting, SQL, Python & some of the standard data science packages (Pandas, Numpy, etc.).
We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy.
Feedback
We believe in making Analytics Vidhya the best experience possible for Data Science enthusiasts.
Help us by providing valuable Feedback.