Description
- Design and build scalable data pipelines using PySpark, SQL, Hadoop.
- Develop and implement data quality rules, validation checks, and monitoring dashboards.
- Collaborate with data architects, analysts, and QE engineers to ensure end-to-end data integrity.
- Establish coding standards, reusable components, and version control practices for data engineering workflows.
- Optimize performance of ETL/ELT processes and troubleshoot data issues in production environments.
- Support regulatory compliance and data governance by integrating data lineage, metadata, and audit capabilities.
ResponsibilitiesSenior Data Engineer
- Design and build scalable data pipelines using PySpark, SQL, Hadoop.
- Develop and implement data quality rules, validation checks, and monitoring dashboards.
- Collaborate with data architects, analysts, and QE engineers to ensure end-to-end data integrity.
- Establish coding standards, reusable components, and version control practices for data engineering workflows.
- Optimize performance of ETL/ELT processes and troubleshoot data issues in production environments.
- Support regulatory compliance and data governance by integrating data lineage, metadata, and audit capabilities.
QualificationsBachelor Degree 7 - 10 years