We are seeking a detail-oriented Data Engineer to join our data team. In this role, you will be responsible for ensuring the ingestion, quality, integrity, and reliability of our data engineering processes and outputs. You will work closely with data engineers, analysts, and other stakeholders to develop and implement data imports and processes for our data pipelines and systems.
What You'll be Responsible for:- Design, develop, and execute test plans for ETL processes, data pipelines, and data warehousing solutions
- Perform thorough testing of data transformations, integrations, and migrations
- Perform data lineage audits and ensuring applied updates reflect incoming data sources
- Develop and maintain automated test suites for continuous integration and deployment
- Conduct performance testing and optimization of data workflows
- Collaborate with other data engineers to troubleshoot and resolve data quality issues
- Implement data validation checks and monitoring systems
- Participate in code reviews and provide constructive feedback
- Document testing procedures, results, and best practices
- Stay current with industry trends and emerging technologies in data engineering and quality assurance
What You'll Bring to the Position:- 1+ years of experience as a data engineer or in a related role
- Experience with SQL and Python programming
- Experience with big data technologies such as Hadoop and Spark
- Familiarity with cloud platforms (AWS, Azure, or GCP)
- Experience with Docker and Kubernetes
- Knowledge of data modeling and ETL processes
- Experience with version control systems (e.g., Git)
- Proficiency in Linux/Unix command-line operations.
Preferred Skills and Experience:- Familiarity with workflow management tools like Apache Airflow, Argo Workflows, and AWS Step Functions
- Experience with data transformation tools like PySpark, Jupyter, and SQL
- Experience with AWS tooling like Athena, Lambda functions, Glue, RDS Postgres, Redis, etc
- Familiarity with CI/CD pipelines and practices
- Experience with test automation frameworks (e.g., pytest)
- Knowledge of data privacy and security best practices