Director Data Engineering - QA Automation
Core Specialty Insurance Services-
Core Specialty’s Data Environment is the organizational data ecosystem that supports enterprise requirements for data management, consolidation, insights, and reporting. The Data & Analytics organization ingests and manages data from over 200 distinct sources and growing, which serves business intelligence, AI/ML models, and enterprise decision-making. Maintaining the integrity, accuracy, and completeness of this extensive data environment requires a scalable and automated quality assurance strategy. The Director of Engineering – QA Automation will be responsible for designing and implementing this strategy, integrating quality processes throughout the data platform—including ingestion pipelines, semantic models, and machine learning workflows. This role will require 40% hands on involvement to start in order to build out the architecture, blueprints, and coding standards required for QA Engineers to adhere to. It may also include development efforts as needed.
Key Accountabilities/Deliverables:
Define and lead the QA automation strategy for data engineering, analytics, and machine learning initiatives.
Architect and implement scalable, reusable automation frameworks for data pipelines, ingestion models, semantic layers, and ML workflows.
Collaborate with Data Engineering, Data Science, and DevOps teams to integrate automated testing into CI/CD pipelines.
Establish coding standards, architectural patterns, and best practices for test automation across the organization.
Develop automated validation for data quality, schema enforcement, lineage tracking, and model performance.
Mentor and grow a high-performing QA team with deep domain knowledge in data and analytics.
Drive continuous improvement in testing tools, processes, and observability.
In addition to the above key responsibilities, you may be required to undertake other duties from time to time as the Company may reasonably require.
Technical Knowledge and Understanding:
Programming Languages: Python (preferred), SQL, Java/Scala (for Spark-based pipelines)
Automation Frameworks: PyTest, Great Expectations, dbt tests, Selenium, Airflow test suites
Data Technologies: Spark, Kafka, Airflow, dbt, Delta Lake, Parquet, Avro
Cloud Platforms: AWS (Glue, S3, Redshift), Azure, GCP (BigQuery, Dataflow), Databricks
Data Warehouses: Snowflake, BigQuery, Redshift
CI/CD & DevOps: GitHub Actions, Jenkins, Docker, Kubernetes
ML Testing: MLflow, TensorFlow Model Analysis, custom validation pipelines
Observability & Monitoring: Monte Carlo, Databand, OpenLineage
SDLC: Agile/SCRUM Frameworks, Kanban, JIRA
Data Quality Management: MS Purview
Preferred Skills
Experience with synthetic data generation and data fuzzing tools
Experience with Data Contracts and the Producer vs. Consumer model
Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA)
Exposure to AI-assisted testing platforms
Experience:
Bachelor’s degree in computer science, data engineering, or related field required
Master’s degree in computer science, data engineering, or related field preferred
Minimum of 15 years of experience in Data Engineering (hands-on roles)
Minimum of 5 years of experience in a leadership position
Prior QA and automation experience within data engineering space required
Proven experience in testing large-scale data systems and ML workflows
Strong architectural thinking and ability to define scalable testing patterns
Excellent leadership, communication, and cross-functional collaboration skills
Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa for this position.
#LI-Hybrid
-
At Core Specialty, you will receive a competitive salary and opportunities for professional development and advancement. We offer medical, dental, vision, and life insurances; short and long-term disability; a Company-match of 100% of a 6% contribution 401(k) plan; an Employee Assistance Plan; Health Savings Account, Flexible Spending Account, Health Reimbursement Account, and a wellness program