Data Engineer
ThoucentricData Engineer<\/span><\/span><\/span> Location -New Delhi<\/span><\/span> <\/span><\/span><\/span><\/span> Exp -3 -5 years<\/span><\/span><\/span> <\/span><\/span><\/span> - Design, develop, and maintain robust data pipelines
using <\/span><\/span>Azure Data Factory (ADF)<\/span><\/span><\/b>. - Build ETL/ELT processes that ingest,
transform, and load data from various sources into data lakes, warehouses, or databases.
- Implement both batch and near real -time data flows.<\/span><\/span><\/span> - Create, schedule, and monitor ADF pipelines. -
Implement complex data workflows, including <\/span><\/span>dependency management,
parameterization, and dynamic pipeline design<\/span><\/span><\/b>. - Integrate ADF with other
Azure services (Blob Storage, Synapse, Databricks, Key Vault, etc.). -
Implement logging, alerting, and retry logic for production stability.<\/span><\/span><\/span> - Analyze large datasets to identify trends, issues, and
transformation needs. - Write performant SQL queries and transformations. -
Collaborate with data analysts to understand data requirements and structure
transformations accordingly. - Ensure data is accurate, consistent, and aligned
with business logic.<\/span><\/span><\/span> - Quickly diagnose and resolve data quality issues,
pipeline failures, or performance bottlenecks. - Perform root cause analysis
for pipeline errors and implement long -term fixes. - Debug and optimize Spark
or SQL transformations in environments like Databricks.<\/span><\/span><\/span> - Implement data validation and quality checks within pipelines.
- Perform reconciliation and sanity checks across data sources. - Maintain data
dictionaries and metadata repositories.<\/span><\/span><\/span> - Work closely with data architects, analysts, BI
developers, and business users. - Translate business data needs into technical implementations.
- Document pipeline logic, data flows, and troubleshooting procedures.<\/span><\/span><\/span> - Work closely with data architects, analysts, BI
developers, and business users. - Translate business data needs into technical implementations.
- Document pipeline logic, data flows, and troubleshooting procedures.<\/span><\/span><\/span> - Optimize ADF pipeline execution time and cost. -
Improve query performance using indexing, partitioning, and efficient
transformation logic. - Recommend improvements in data infrastructure and
tooling.<\/span><\/span><\/span>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/span><\/span><\/p>
<\/p>
<\/div><\/span>Benefits<\/h3>
<\/span><\/span><\/div>
<\/span><\/span><\/div>
<\/span><\/span><\/div>
<\/span><\/span><\/div>
<\/span><\/span><\/div>
<\/span><\/span><\/div>
<\/div><\/span>