Senior Data Engineer

Senior Data Engineers lead the development and optimization of complex data systems and pipelines. Responsibilities include managing data projects, ensuring data quality, and mentoring junior engineers. You will work with stakeholders to align data solutions with business goals and drive improvements. Extensive experience in data engineering and leadership skills are required.

Lead design and optimization of complex data solutions, Oversee data engineering projects, Mentor junior engineers, Ensure solutions align with organizational goals, Provide strategic guidance.

Required Qualifications:
 

10+ years in data engineering/ETL roles, with at least 6+  years in Azure cloud ETL leadership. Azure certifications (e.g., Azure Analytics Specialty, Solutions

• Azure Data Sources: Azure Data Lake Storage (ADLS), Blob Storage, Azure SQL Database, Synapse Analytics. External Sources: APIs, on-prem databases, flat files (CSV, Parquet, JSON). 
• Tools: Azure Data Factory (ADF) for orchestration, Databricks connectors. 
• Apache Spark: Strong knowledge of Spark (PySpark, Spark SQL) for distributed processing. 
• Data Cleaning & Normalization: Handling nulls, duplicates, schema evolution. 
• Performance Optimization: Partitioning, caching, broadcast joins.
• Delta Lake: Implementing ACID transactions, time travel, and schema enforcement. 
• Azure Data Factory (ADF): Building pipelines to orchestrate Databricks notebooks. 
• Azure Key Vault: Secure credential management. 
• Azure Monitor & Logging: For ETL job monitoring and alerting. 
• Networking & Security: VNET integration, private endpoints.