Experience: 8-14 years
Notice period: 30 days
Key Responsibilities:
● Technical Leadership:
○ Lead and mentor a team of data engineers, providing technical guidance and support.
○ Drive the design, development, and deployment of scalable data pipelines and solutions using Databricks, Airflow, PySpark/Spark, Kafka, and cloud technologies.
○ Ensure best practices and standards are followed in coding, testing, and documentation.
● Architecture and Design:
○ Architect and design data processing systems that are robust, scalable, and maintainable.
○ Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical solutions.
○ Evaluate and recommend tools and technologies to enhance our data infrastructure.
● Data Pipeline Development:
○ Develop and maintain ETL/ELT pipelines using Databricks, Airflow, PySpark/Spark, and Kafka.
○ Implement data integration and data processing solutions to handle large-scale, complex datasets.
○ Optimize data pipelines for performance, scalability, and reliability.
● Cloud Integration:
○ Design and implement cloud-based data solutions, leveraging platforms such as AWS, Azure, or Google Cloud.
○ Ensure data security, compliance, and governance in cloud environments.
● Collaboration and Communication:
○ Work closely with cross-functional teams including data science, analytics, and product development to deliver data-driven solutions.
○ Communicate technical concepts and solutions effectively to both technical and non-technical stakeholders.
● Continuous Improvement:
○ Stay current with industry trends and emerging technologies in data engineering.
○ Propose and implement improvements to existing data infrastructure and processes.
Qualifications:
● Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
● Experience:
○ 8+ years of experience in data engineering or related field.
○ Proven experience as a technical lead or senior data engineer.
○ Extensive experience with Databricks, Apache Airflow, PySpark/Spark
○ Knowledge on Apache Kafka is preferred.
○ Must have system designing & documentation/architecture experience
○ Strong experience with cloud platforms such as AWS, Azure, or Google Cloud.
● Soft Skills:
○ Excellent problem-solving and analytical skills.
○ Strong leadership and team management abilities.
○ Effective communication and interpersonal skills.
○ Ability to work in a fast-paced, collaborative environment.