Senior Data Engineer

Description

Summary: Seeking a Senior Data Engineer to lead the design and implementation of robust, scalable data pipelines within a large-scale capital markets data migration program. Highlights: 1. Lead data pipeline development for critical financial data migration 2. Design and implement robust, scalable ETL/ELT pipelines with AWS Glue & dbt 3. Guide and mentor engineers, shaping best practices in data processing Project overview: The program is a multi phase data migration initiative aimed at replacing legacy capital markets systems with a modern platform ecosystem. It includes the migration of critical datasets across custody, clearing and settlement, derivatives processing, and CCP operations. The project spans 18 months and follows an incremental approach with strong emphasis on data integrity, reconciliation, and traceability. * Position overview: We are looking for a Senior Data Engineer to lead the development and optimization of data pipelines within a large scale data migration program in a capital markets environment. * This role is responsible for designing and implementing robust, scalable, and production grade data pipelines, ensuring data quality and consistency across multiple systems. You will play a key role in shaping the data processing layer, guiding best practices, and supporting the successful migration of critical financial data. * The position requires strong technical expertise, architectural thinking, and the ability to mentor other engineers while collaborating closely with data architects and business stakeholders. Technology stack: SQL, AWS Glue, dbt, Oracle, AWS, Git * Responsibilities: Design and implement scalable, reliable, and maintainable ETL/ELT pipelines using AWS Glue and dbt * Define and enforce best practices for data engineering, including coding standards, testing, and deployment * Lead the integration of data from multiple legacy systems into the target platform * Ensure data quality, consistency, and integrity across all stages of the data pipeline * Collaborate with data architects to implement target data models and transformation strategies * Optimize data pipelines for performance, scalability, and cost efficiency * Implement robust error handling, logging, and monitoring mechanisms * Guide and mentor mid level engineers, providing technical direction and code reviews * Work closely with data analysts and business stakeholders to ensure data meets functional requirements * Troubleshoot complex data issues and drive root cause analysis * Contribute to the design of the overall data platform and processing architecture * Document data pipelines, standards, and technical decisions * Requirements: Strong expertise in SQL and data transformation at scale * Extensive experience with AWS Glue and cloud based data processing * Proven experience building production grade ETL/ELT pipelines * Strong experience with dbt or similar transformation frameworks * Experience working with large volumes of data in complex environments * Strong understanding of data modeling and data warehouse architectures * Experience implementing CI/CD practices in data environments * Experience designing scalable and maintainable data solutions * Strong problem solving skills and attention to detail * Nice to have: Experience in capital markets (custody, clearing, settlement, derivatives, CCP) * Experience in large scale data migration or transformation programs * Experience with Oracle based legacy systems * Familiarity with data governance, data lineage, and metadata management * Experience with observability tools and monitoring frameworks