Specialist - Data Engineering

Senior Specialist - Data Engineering

Job Summary: We are seeking a highly skilled Senior Specialist - Data Engineering with strong Spark Scala knowledge, mandatory for a significant and complex engagement. An AWS background is welcome. As a Big Data Full Stack Developer, you will work with various underlying tools such as Spark/Scala, Oozie, Postgre, and Couchbase. The role requires collaborating with architects to implement the application solution successfully.

Job Responsibilities:

1. Develop Scala/Spark jobs for data transformation and aggregation.
   
2. Modify and enhance the underlying Scala framework, as required.
   
3. Work with Spark Streaming jobs, handle in-memory Spark state, and process messages for near-real-time systems.
   
4. Optimize Spark queries for improved performance.
   
5. Integrate with SQL databases, especially Postgre and Couchbase.
   
6. Collaboratively solve technical problems within specified timelines with minimal guidance.
   
7. Implement various engineering tasks, including gap analysis, data analysis, and impact analysis.
   
8. Independently perform low-level design configuration, code customization, refactoring, and code reviews.
   
9. Demonstrate expertise in design, development, testing, build, and deployment stages of the software lifecycle.
   
10. Ensure thorough documentation and provide reliable engineering support to all stakeholders.
    

Required Skills:

• Mandatory expertise in Spark Scala.
  
• AWS background and familiarity with Big Data technologies.
  
• Proficiency in modifying and enhancing Scala frameworks.
  
• Experience with Spark Streaming and in-memory processing.
  
• Strong SQL database integration skills, particularly with Postgre and Couchbase.
  
• Problem-solving abilities and the ability to meet project deadlines.
  
• Effective collaboration with team members and architects.
  
• Documentation skills for maintaining comprehensive project records.