About the role:
This role seeks a Data Engineer with expertise in building and maintaining scalable data pipelines for the financial industry. You will contribute to the design, implementation, and optimization of data pipelines for ingesting and enriching financial data, ultimately improving platform capabilities and data accuracy.
Key Responsibilities:
- Design and implement scalable data lake solutions for handling customer and market data.
- Build and improve pipelines for ingesting and enriching financial data.
- Identify and implement optimizations for existing pipelines to enhance scalability, throughput, and data accuracy.
- Analyze and map source data fields to expand platform functionalities.
- Gather and document system enhancement requirements collaborating with business and core systems teams.
- Work closely with the production team to troubleshoot and resolve data pipeline issues.
Technical Skills (Must Have):
- Expert: SQL, PySpark, Python, Shell Scripting, Airflow, AWS services (EC2, RDS, S3, Spark, EMR, Athena, Presto, Hudi, etc.)
- Significant experience: RDBMS (relational database management systems) and either JIRA or Jenkins (project management/continuous integration tools).
Technical Skills (Good to Have):
- MySQL Database Management System
- Dremio (data virtualization tool)
- Starburst (data virtualization tool)
Location: Bangalore, India ( Work from Office / Hybrid )
Qualifications:
- Bachelor's degree in Computer Science or related field (BE/BTech)
- 5-10 years of work experience (8+ years for senior and lead engineer positions) with Python, Spark, and SQL, preferably on high-volume data applications in a Linux environment using MySQL or S3 storage.
- Minimum of 4 years of experience with RDBMS, working on high-volume daily processing solutions in the financial industry.
- Knowledge of AWS cloud services for data processing, storage, and computation.
- Basic understanding of financial concepts (instruments, trades, positions, corporate actions).
Additional Preferences:
- Knowledge of data virtualization/analytics tools like Dremio and Starburst.
- Experience working with Airflow.
- Advanced knowledge of financial concepts.