Data Engineer (5+ years)

🔹 1. Data Architecture Design

·      Design end-to-end data architectures using Azure services (e.g., Azure Synapse Analytics, Azure Data Lake, Azure SQL Database, Cosmos DB).

·      Define data models, data flows, and integration patterns aligned with business and technical requirements.

·      Ensure solutions support scalability, reliability, and performance goals.

🔹 2. Data Integration and ETL/ELT

·      Design and oversee data ingestion pipelines using tools like Azure Data Factory, Synapse Pipelines, or Databricks.

·      Establish processes for data transformation, cleansing, and enrichment.

·      Integrate structured, semi-structured, and unstructured data from multiple sources (on-premises and cloud).

🔹 3. Data Storage and Management

·      Choose appropriate storage solutions (e.g., Data Lake Gen2, Blob Storage, SQL, Cosmos DB) based on use cases.

·      Define data retention policies, partitioning strategies, and lifecycle management.

·      Optimize storage performance and cost.

🔹 4. Data Governance and Security

·      Implement data governance frameworks, including data lineage, cataloging, and metadata management (e.g., Azure Purview / Microsoft Purview).

·      Ensure compliance with GDPR, HIPAA, and other regulations.

·      Define and implement data security, encryption, and role-based access control (RBAC) in Azure.

🔹 5. Analytics and BI Enablement

·      Architect analytical environments for Power BI, Azure Synapse, and Databricks.

·      Enable real-time and batch analytics.

·      Ensure that data is accessible and trustworthy for reporting and machine learning.

🔹 6. Collaboration and Stakeholder Management

·      Work with data engineers, analysts, scientists, and business stakeholders to understand data requirements.

·      Translate business goals into technical data strategies.

·      Provide architectural guidance to cross-functional teams.

🔹 7. Performance Optimization

·      Monitor and optimize query performance, data pipelines, and storage costs.

·      Implement data partitioning, caching, and indexing strategies.

·      Use Azure Monitor, Log Analytics, and other tools for system health and performance tracking.

🔹 8. Cloud Strategy and Best Practices

·      Develop and maintain a data strategy aligned with overall Azure cloud architecture.

·      Stay current on Azure updates and industry trends to recommend improvements.

·      Promote DevOps and CI/CD practices for data pipeline deployment and management.

🔹 9. Documentation and Standards

·      Create and maintain architecture diagrams, design documents, and data dictionaries.

·      Define naming conventions, coding standards, and documentation best practices for data teams.

🔹 10. Mentorship and Leadership

·      Guide and mentor data engineers and developers in best practices for Azure data solutions.

·      Provide architectural oversight during project execution.

·      Conduct code and design reviews to ensure consistency and quality.