Senior Pyspark Data Engineer

Job Title:<\/span><\/span><\/span><\/span><\/span><\/b> <\/span>PySpark Data Engineer
<\/span><\/span>Experience:<\/span><\/span><\/b> <\/span>8+ Years
<\/span><\/span>Location:<\/span><\/span><\/b> <\/span>Hyderabad
<\/span><\/span>Employment Type:<\/span><\/span><\/b> <\/span>Full -Time<\/span><\/span><\/span><\/span><\/span><\/p>

 <\/span><\/span><\/span><\/span><\/span>
<\/p>

Job Summary:<\/span><\/span><\/span><\/span><\/span><\/b>
<\/p>

We are looking for a skilled and experienced <\/span><\/span><\/span>PySpark Data Engineer<\/span><\/span><\/b> <\/span>to join our growing data engineering team. The ideal candidate will have 8+ years of experience in designing and implementing data pipelines using <\/span><\/span><\/span>PySpark<\/span><\/span><\/b>, <\/span><\/span><\/span>AWS Glue<\/span><\/span><\/b>, and <\/span><\/span><\/span>Apache Airflow<\/span><\/span><\/b>, with strong proficiency in <\/span><\/span><\/span>SQL<\/span><\/span><\/b>. You will be responsible for building scalable data processing solutions, optimizing data workflows, and collaborating with cross -functional teams to deliver high -quality data assets.
<\/span><\/span><\/span><\/span><\/span>
Key Responsibilities:<\/span><\/span><\/span><\/span><\/span><\/b>
<\/p>

• Design, develop, and maintain large -scale <\/span><\/span><\/span>ETL pipelines<\/span><\/span><\/b> <\/span>using <\/span><\/span><\/span>PySpark<\/span><\/span><\/b> <\/span>and <\/span><\/span><\/span>AWS Glue<\/span><\/span><\/b>.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Orchestrate and schedule data workflows using <\/span><\/span><\/span>Apache Airflow<\/span><\/span><\/b>.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Optimize data processing jobs for performance and cost -efficiency.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Work with large datasets from various sources, ensuring data quality and consistency.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Collaborate with Data Scientists, Analysts, and other Engineers to understand data requirements and deliver solutions.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Write efficient, reusable, and well -documented code following best practices.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Monitor data pipeline health and performance; resolve data -related issues proactively.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li>
• Participate in code reviews, architecture discussions, and performance tuning.<\/span><\/span><\/span><\/span><\/span>
  <\/span><\/li><\/ul>
  
  <\/div><\/span><\/span><\/span><\/span>

  
  <\/div><\/span>
  

  Requirements<\/h3>

  
  <\/h3>

  • 8+ years of experience in <\/span>data engineering<\/b> <\/span>roles.<\/span><\/span><\/span>
    <\/span><\/li>
  • Strong expertise in <\/span>PySpark<\/b> <\/span>for distributed data processing.<\/span><\/span><\/span>
    <\/span><\/li>
  • Hands -on experience with <\/span>AWS Glue<\/b> <\/span>and other AWS data services (S3, Athena, Lambda, etc.).<\/span><\/span><\/span>
    <\/span><\/li>
  • Experience with <\/span>Apache Airflow<\/b> <\/span>for workflow orchestration.<\/span><\/span><\/span>
    <\/span><\/li>
  • Strong proficiency in <\/span>SQL<\/b> <\/span>for data extraction, transformation, and analysis.<\/span><\/span><\/span>
    <\/span><\/li>
  • Familiarity with data modeling concepts and data lake/data warehouse architectures.<\/span><\/span><\/span>
    <\/span><\/li>
  • Experience with version control systems (e.g., Git) and CI/CD processes.<\/span><\/span><\/span>
    <\/span><\/li>
  • Ability to write clean, scalable, and production -grade code.<\/span><\/span><\/span>
    <\/span><\/li><\/ul><\/div><\/span><\/span><\/span><\/span>
    
    <\/div><\/span>
    

    Benefits<\/h3>

    Company Standard Benefits.<\/div><\/span>