Data Lineage

Description

Job Summary: We are seeking a Data Engineer with experience in data architecture, management, and optimization to lead a development cell for data lineage projects. Key Highlights: 1. Lead data lineage and information mapping projects. 2. Design robust and efficient data architectures. 3. Mentor junior engineers and define best practices. IDS Comercial is a 100% Mexican IT company with 40 years of experience and presence in Mexico, Latin America, and the USA. We are accredited at CMMI Level 5 (Capability Maturity Model Integration), a quality model established by the Software Engineering Institute (SEI). We are ISO certified under ISO/IEC 27001:2013\. We hold the Socially Responsible Company (ESR) distinction. WE ARE LOOKING FOR YOU!! **Data Lineage** **Work Mode: Hybrid (3 days at client site, 2 days remote)** **Location: Av. Paseo de la Reforma 347, Cuauhtémoc, 06500 Mexico City, CDMX** **Project Duration: 6 Months** **Requirements:** **3+ years of experience as a Data Engineer.** Experience in: * Architecture Design: Creating and maintaining robust and efficient data pipelines. * Data Management: Implementing ETL (Extract, Transform, Load) processes to integrate data from diverse sources. * Technical Leadership: Guiding and mentoring junior engineers, as well as defining coding and development best practices. * Optimization and Quality: Improving query performance and ensuring data integrity, security, and availability. **Essential Knowledge:** * Experience: Proven extensive track record in data engineering or similar roles. * Technical Skills (Hard Skills): Proficiency in programming languages (Python, Scala, Java, SQL), Big Data tools (Spark, Hadoop), and cloud platforms (AWS, Google Cloud, Azure). * Data Modeling: Skills in designing relational and non-relational (NoSQL) databases. * Soft Skills: Leadership, excellent communication, critical thinking, and ability to solve complex problems. **Responsibilities:** Deliver specialized services through a development cell that provides the following: **1\. Data Flow Diagramming:** a. Document services, applications, and interfaces. b. Identify data structures. c. Identify elements or fields within these data structures. d. Validation and approval by process owners. **2\. Information Mapping and Lineage:** a. Identify original data sources. b. Trace lineage from source table field to destination table field, including transformation steps. c. Validation and approval by process owners. **3\. Data Requirements Document:** a. Explain the transformation process from source to destination. b. Validation and approval by process owners