Intermediate Data Engineer

The Intermediae Data Engineer is accountable for creating and maintaining the technological infrastructure of Spot Money and implementing methods to improve data reliability and quality. Ensure the consistent delivery of data architecture throughout ongoing projects by creating data models, building data pipelines and coordinating ETL (extract, transform, load).

Responsibilities :

Data Ingestion & Processing

• Assist in building ETL/ELT pipelines using AWS services like AWS Glue, Lambda, and Step Functions.
• Work with Amazon S3 for data storage and ingestion.
• Support real-time and batch data processing jobs.

Data Storage & Management

• Load and manage data in AWS
• Work with Parquet, Avro, JSON, and CSV file formats in data lakes.

Data Transformation & Integration

• Support data transformations using PySpark, SQL, or AWS Glue ETL.
• Assist in writing SQL queries for data aggregation, cleansing, and structuring.

Automation & Orchestration

• Monitor scheduled workflows
• Write basic Python or Shell scripts for automating tasks.

Monitoring & Debugging

• Assist in monitoring and troubleshooting failed data pipelines.
• Perform unit testing and basic debugging of data workflows.

Requirements :

Knowledge, skills and attributes:

• Programming Python, SQL.
• AWS Services S3, Glue, Lambda, Redshift, DynamoDB, RDS, CloudWatch, IAM.
• Data Processing Spark (PySpark), SQL-based transformations.
• Orchestration Step Functions, Airflow, or Lambda.
• Version Control Git.
• Flexibility to perform ad-hoc tasks as per business needs.

Education and training:

• Diploma in Computer Science, Information Technology, Engineering, or equivalent.
• Bachelor’s degree in computer science, Information Technology, Engineering, or equivalent advantageous.
• Relevant certification (Certified Data Engineer, Certificate Professional, etc).

Experience:

• 2 years in a similar role.
• Experience with architecture and design of secure, highly available and scalable systems.
• Experience with writing scripts using scripting language.
• Experience working with data sets and ordering data through various functions.
• Experience working with internet technologies.
• Experience dealing with internal and external stakeholders.
• Experience working with and/or on Power BI, Snowflake and AWS system advantageous.