Job Description:
Job Description:
We are seeking an experienced Azure Data Engineer to join our team. The ideal candidate will have a strong background in designing, developing, and managing ETL processes and data pipelines on the Microsoft Azure platform. Proficiency in Azure Data Factory (ADF), PySpark, and CI/CD pipelines is essential. The role requires a hands-on approach to design, modeling, development, and delivery, with a focus on scalable and reliable enterprise solutions.
Key Responsibilities:
Development:
- Hands-on design, modeling, development, and delivery of data solutions.
- Strong working knowledge of Azure services and components, with the ability to integrate various components into a scalable, reliable, and sustainable enterprise solution.
- Develop and maintain ETL processes and data pipelines using Azure Data Factory (ADF).
- Optimize data transformation processes using PySpark.
- Production experience delivering CI/CD pipelines across Azure and vendor products.
- Contribute to the design and development of enterprise standards.
- Key knowledge of architectural patterns across code and infrastructure development.
Requirements:
Technical Skills and Experience:
- Bachelor’s or master’s degree in computer science, Engineering, Data Science, or equivalent experience, with a preference for experience and a proven track record in advanced, innovative environments.
- 7-8 years of professional experience in data engineering.
- Strong expertise in Microsoft Azure data services, particularly Azure Data Factory (ADF) and PySpark.
- Experience with data pipeline design, development, and maintenance.
- Proficiency in SQL and experience with NoSQL databases.
- Knowledge of ETL/ELT processes and tools.
- Familiarity with big data technologies such as Hadoop and Spark.
- Experience with Azure Databricks, Data Lake, SQL Database, and other Azure data services.
- Production experience delivering CI/CD pipelines across Azure and vendor products.
- Knowledge of architectural patterns across code and infrastructure development.
- Fully conversant with big-data processing approaches and “schema-on-read” methodologies, with a preference for a deep understanding of Spark, Databricks, and Delta Lake.
- Experience with Agile delivery frameworks and tools (SAFe, Jira, Confluence, Azure DevOps).
- Experience with visualization tools (Power BI, developing reports, dashboards, KPI scorecards).
Preferred Qualifications:
- Azure Data Engineer certification.
- Experience with data warehousing concepts and technologies.
- Understanding of data security and compliance requirements.