Intermediate Data Engineer

Intermediate Data Engineer

Minimum Requirements:

• Matric (Grade 12)
• Masters in Software Engineering, Data Engineering, Computer Science or related field
• 2-3 years of relevant work experience
• Strong Scala or Java background
• Knowledge of AWS, GCP, Azure, or other cloud platform
• Understanding of data modeling principles
• Ability to work with complex data models
• Experience with relational and NoSQL databases (e.g. Postgres, ElasticSearch/OpenSearch, graph databases such as Neptune or neo4j)
• Experience with technologies that power analytics (Spark, Hadoop, Kafka, Docker, Kubernetes) or other distributed computing systems
• Knowledge of API development and machine learning deployment

Responsibilities:

• Build and scale data infrastructure that powers real-time data processing of billions of records in a streaming architecture
• Build scalable data ingestion and machine learning inference pipelines
• Build general-purpose APIs to deliver data science outputs to multiple business units
• Scale up production systems to handle increased demand from new products, features, and users
• Provide visibility into the health of our data platform (comprehensive view of data flow, resources usage, data lineage, etc) and optimize cloud costs
• Automate and handle the life-cycle of the systems and platforms that process our data