Senior Data Engineer

Job Description

As a Lead Data Engineer, you will design and implement robust data solutions using Big Data, Hadoop (Cloudera), and Google Cloud Platform (GCP), working closely with enterprise data teams, architects, and analysts.

Key Responsibilities:

• Lead development efforts for ingesting and transforming data from diverse sources.
• Work hands-on in a Big Data/Hadoop environment, writing code, building scripts, and ensuring end-to-end delivery of data to the Enterprise Data Lake.
• Design, build, and operationalize distributed, reliable, and scalable data pipelines using GCP services (DataProc, Dataflow, CloudSQL, BigQuery, CloudSpanner) and third-party tools (Spark, Apache Beam/Composer, DBT, Confluent Kafka, Cloud Functions).
• Design and implement data ingestion patterns for batch, streaming, and API interfaces (Ingress and Egress).
• Guide and mentor a team of data engineers, developing custom code and frameworks using best practices (Java, Python, Scala, SQL).
• Provide end-to-end technical guidance on effectively leveraging cloud infrastructure and platform services to solve business problems.

Skills and Experience:

• 2+ years of experience in data engineering, including performance optimization for large OLTP applications.
• Strong knowledge of Big Data technologies: Hadoop HDFS, Hive, Pig, Flume, Sqoop, and core Hadoop concepts.
• Proficiency with GCP managed data services: DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub.
• Experience with Cloud Dataflow (Java/Python) for deploying streaming and batch jobs.
• Working experience in HQL and with relational/NoSQL databases.
• Knowledge of data streaming technologies such as Kafka and Spark-streaming.
• Familiarity with Infrastructure as Code (IaC) practices (e.g., Terraform).
• Knowledge of Java microservices, Spring Boot, and developing/scaling JAVA REST services.
• Strong architectural knowledge, communication, and problem-solving skills.

Nice-To-Have Skills:

• Understanding of the Wealth business line and required data domains for end-to-end solution building.

Benefits & Perks:

• Upskilling opportunities through online courses, cross-functional development, and tuition assistance.
• Competitive rewards program, including bonus, flexible vacation, personal/sick days, and day-one benefits.
• Dynamic work environment with space for team collaboration.