Senior Data Engineer

Description

Job Summary: We are seeking a Data Engineer with experience in architecture design, data management, technical leadership, optimization, and data quality to work on Data Lineage. Key Highlights: 1. Design and maintenance of robust and efficient data pipelines. 2. Implementation of ETL processes to integrate data from diverse sources. 3. Technical leadership and mentoring of junior engineers. IDS Comercial is a 100% Mexican IT company with 40 years of experience and presence in Mexico, Latin America, and the USA. We are accredited at CMMI Level 5 (Capability Maturity Model Integration), a quality model established by the Software Engineering Institute (SEI). Certified by ISO under standard ISO/IEC 27001:2013\. We hold the Socially Responsible Enterprise (ESR) distinction. WE ARE LOOKING FOR YOU!! **Data Lineage** **Work Mode: Hybrid (3 days onsite with client, 2 days remote)** **Location: Av. Paseo de la Reforma 347, Cuauhtémoc, 06500 Ciudad de México, CDMX** **Project Duration: 6 Months** **Requirements:** **3+ years of experience as a Data Engineer.** Experience in: * Architecture Design: Creating and maintaining robust and efficient data pipelines. * Data Management: Implementing ETL (Extract, Transform, Load) processes to integrate data from diverse sources. * Technical Leadership: Guiding and mentoring junior engineers, as well as defining coding and development best practices. * Optimization and Quality: Improving query performance and ensuring data integrity, security, and availability. **Mandatory Knowledge:** * Experience: Proven extensive background in data engineering or similar roles. * Technical Skills (Hard Skills): Proficiency in programming languages (Python, Scala, Java, SQL), Big Data tools (Spark, Hadoop), and cloud platforms (AWS, Google Cloud, Azure). * Data Modeling: Skills in designing relational and non-relational (NoSQL) databases. * Soft Skills: Leadership, excellent communication, critical thinking, and ability to solve complex problems. **Responsibilities:** Deliver specialized services through a development cell that provides the following: **1\. Data Flow Diagramming:** a. Documentation of services, applications, and interfaces. b. Identification of data structures. c. Identification of elements or fields within these data structures. d. Validation and approval by process owners. **2\. Information Mapping and Lineage:** a. Identification of original data sources. b. Identification of lineage from source table field to destination table field, including transformation logic. c. Validation and approval by process owners. **3\. Data Requirements Document:** a. Explanation of the transformation process from source to destination. b. Validation and approval by process owners