Principal Data Engineer

About us

Xiatech is the pioneer of Xfuze, the world’s first AI-powered composable Integration, Data Management, Analytics and Orchestration Platform that innovatively connects systems, creates a single view of data and delivers actionable insights in one cloud-native, SaaS solution.  Business, technology and data teams use Xfuze to shorten time-to-insights, accelerate digital transformation, and extend the value of their legacy technology investments.

We’re a culture-first organisation and put our people at the forefront of everything we do. We believe that a great working environment leads to a happy and productive team which is why we offer our staff the flexibility to work remotely or from our beautiful office in Fitzrovia, Central London. 

The Role

The Principal Databricks Data Engineer will take a lead role in optimising and evolving the Xfuze data platform for one of our key retail clients, Flying Tiger. This is a critical, hands-on role focused on resolving performance bottlenecks, implementing robust cost controls, and ensuring our Databricks environment on Microsoft Azure is built on a foundation of best practices.

You will be responsible for creating and executing low-level physical designs with a focus on the implementation of data pipelines, optimisation, performance, and cost to form robust data solutions. Mentoring team members, working collaboratively to eliminate single-person dependencies, ensuring the platform is scalable, efficient, and delivers maximum value to the client.  You will work closely with the client business teams, the Xfuze project team, the stakeholders and their peers in the data engineering team and third-party partners such as Microsoft and Databricks.

This role is an opportunity to join an established UK-based Software as a Service organisation who believes in using the latest leading-edge technology to drive business growth.  

Key Responsibilities

• Design, build, and maintain efficient, reliable, and scalable data pipelines using PySpark, SQL, and Databricks on Azure.

• Lead the performance tuning and cost optimisation of the client's Databricks platform, identifying and rectifying inefficient pipelines and configurations.

• Take ownership of creating and documenting low-level physical designs for new features and architectural adjustments, ensuring transparency and team alignment.

• Provide technical leadership and mentorship to the existing data engineering team on Databricks best practices to uplift team capability and reduce reliance on key individuals.

• Participate in a joint client / Xiatech Technical Design Authority, taking full accountability for the solution and representing designs in governance meetings.

• Work as part of a cross-functional team to deliver new features using agile methodologies.

• Implement rigorous data quality patterns that clean and enrich data reliably.

• Ensure data security and compliance within an ISO27001 framework by implementing best practices for data encryption, access control, and secure data transmission within the Azure and Databricks ecosystems.

Your Knowledge & Experience

• 5+ years of experience in a Data Engineering or Data Architecture role.

• Expert-level proficiency in Databricks, including deep experience with Delta Lake, Photon, Unity Catalog, and Databricks SQL.

• Proven experience with performance tuning and cost optimisation of Spark workloads and deep knowledge of Databricks

• You must be able to demonstrate how you've previously tackled and solved high-cost data platforms, ideally handling event streams.

• Extensive hands-on experience with the Microsoft Azure cloud platform, especially services like Azure Data Lake Storage (ADLS), Azure Data Factory, and Azure DevOps.

• Strong proficiency in SQL & Python (PySpark).

• Experience designing and building data models using medallion and star schema architectures within a data lakehouse.

• Familiarity with containerisation and IaC tools (e.g., Docker, Terraform) is desirable.

• Previous experience in the retail domain is desirable.

• Excellent communication skills, with an ability to articulate complex technical concepts to diverse stakeholders both internally and externally to our client and our partners.

Your Personality

• Innovative

• Driven

• Resilient

• Insightful

• Empathetic

• Analytical

• Self-Starter

• Creative

• Problem solver

If you would like the opportunity to join an established UK-based Software as a Service organisation who believe in using the latest leading-edge technology to drive business growth, please enquire now by emailing maddie@xiatech.co.uk.

If you receive an offer of employment, this will be conditional upon satisfactory completion of a right to work & identity check, a reference check and a basic criminal record check for any unspent convictions.

Xiatech is proud to be an equal opportunity employer and prohibits discrimination and harassment of any kind.