Data Engineer ADB+Pyspark Freelance Remote

Qualifications and Skills

• Proficiency in Azure Databricks (ADB) and Pyspark for efficient data processing and management.
• Extensive experience in designing and implementing scalable data solutions in a cloud environment.
• Strong knowledge of data warehousing concepts and ETL processes, ensuring data quality and integrity.
• Familiarity with cloud platforms such as AWS, Google Cloud, or Azure to deploy and manage applications.
• Ability to work with various data storage systems including SQL, NoSQL, and data lakes.
• Understanding of distributed data processing frameworks like Apache Spark for big data computation.
• Strong problem-solving skills to diagnose and resolve data-related issues and optimizations.
• Effective communication skills to collaborate with cross-functional teams and stakeholders.

Roles and Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Pyspark.
• Collaborate with various teams to understand data requirements and optimize data solutions.
• Implement ETL processes to ensure data quality, accuracy, and availability for analytical purposes.
• Maintain and improve data architecture to support ongoing and future data initiatives.
• Conduct performance tuning and optimization for real-time and batch data processing.
• Provide technical expertise in identifying innovative data solutions and strategies.
• Ensure compliance with data governance and security protocols within the organization's framework.
• Document and maintain up-to-date records of data architecture, processes, and protocols.