Azure Data Engineers:
The role
As Azure data engineer, you will build, configure, deploy and support a production-grade, scalable and secure data platform. On top of this platform, you will build and support long-term, production-grade data pipelines that ingest, store, transform, process, and expose data from producer to consumer, based on business requirements. Data pipelines are developed using a code-first approach in line with the platform standards and practices.
You translate functional requirements into a technical design.
You provide and maintain technical documentation.
You train and handover common data engineering related maintenance and monitoring activities to the Data Operations & Quality Team.
You provide 2nd and 3rd level support in a DevOps operational mode or when required (occasionally outside of office hours).
You share technology and platform expertise with colleague data engineers, product owners and other people in the organization.
Important areas of expertise are:
Storage: Best of breed (Cloud) storage solutions, both for unstructured and structured data.
Ingestion: Streaming and batch ingestion from source systems both in the cloud and on-premise, private and public.
Transformation: Code-first approach using Spark (Databricks).
Analytic engines: Deploy, configure and operationalize analytics engines and its clustered infrastructure.
Exposure/Integration: Expose data based on consumption use-cases like data science, self-service analytics, business intelligence and operational API’s for process integration.
Data Platform: Health, logging, monitoring, debugging, automation.
Experience
A bachelor’s or master’s degree in computer science, engineering or equal through experience.
The ideal candidate has strong and demonstrable hands-on experience with data engineering on cloud-based data platforms using Databricks (Spark), preferably based on MS Azure technology: Azure Storage, Azure SQL database, Azure data factory, Azure Databricks.
Good coding skills (Python, SQL or Scala).
Experience with following concepts or technologies is considered a plus:
Data Lake Storage, SQL, Hive, Presto, …
Databricks, Spark, Delta Lake, Hadoop, HDInsights, Cloudera, Hortonworks, MapR, …
Azure Functions, Azure Synapse, Azure Analysis Services, Power BI
Python, SQL, Scala, Java, C#, …
Azure AD, Ranger, …
Docker, Kubernetes, …
Azure, AWS, …
Kafka, StreamSets, REST, …