Data Engineer - Lead (100% Remote, Work From Home)
Bitcoin Devs Company
Full-time
On-site
India
Overview:
The Data Engineer - Lead plays a crucial role in our organization, working remotely to design, develop, and maintain scalable data pipelines and infrastructures for our data platforms. This position is vital in ensuring the efficiency, reliability, and quality of our data solutions, enabling our teams to make informed decisions and drive business growth.
The Data Engineering team is part of the broader DevOps department, and sits at the core of the business, bridging the fast-paced and never-ending data feeds from the trading platform to the entirety of the firm. Whether it is about business development, financial products, risk management, compliance & market surveillance, marketing – consistent and reliable data is key, making Data Engineering the cornerstone to the firm’s success.
Key Responsibilities:
- Design and maintain enhancements to our data warehouse, data lake, and data pipelines to increase their reliability and consistency
- Improve queriability of large historical datasets through industry-standard tools, careful data representation and aggregation for both technical and business units
- Ensure data governance and security/retention policies can be implemented and enforced
- Ensure that operational system integrations driven from the data stack are running, monitored and available
- Continually review the data platform to ensure that it is fit for purpose and meeting the needs of the business
- Support and maintain downstream integrations from our data lake, for example business intelligence and visualization tools and third party systems
Required Qualifications:
- Experience in the data engineering field with demonstrated design and technical implementation of data warehouses
- Experience with OLAP databases, how they differ from OLTP databases, and data structuring/modeling with understanding of key data points in a business sense for trade-offs between storage/performance and usability
- Experience with building, deploying, and troubleshooting reliable, idempotent and consistent data pipelines working with disparate in-house and external data-sources, e.g. using Airflow DAGs
- Experience with AWS Redshift, Glue Data Catalog, S3, PostgreSQL, Parquet, Iceberg, Trino, and how they are managed using Terraform & Kubernetes
- Experience with data loading, extraction, manipulation and preparing data for ingestion and integration with visualization platforms such as Tableau.