Senior Data Engineer

Description

Job Summary: Design, develop, and maintain data integration, preparation, and transformation processes for Business Intelligence and Advanced Analytics solutions in Big Data and Cloud environments. Key Highlights: 1. ETL/ELT pipeline design and automation 2. Building scalable data infrastructures with Spark and Cloud 3. Collaboration with data scientists and information architects DESCRIPTION The role aims to design, develop, and maintain data integration, preparation, and transformation processes for Business Intelligence and Advanced Analytics solutions, ensuring consistency, traceability, and performance in Big Data and Cloud environments. The Data Engineer is responsible for building end-to-end data pipelines (ETL/ELT), implementing automated processes, and optimizing data movement between sources and analytical destinations. Their mission is to create scalable data infrastructures using Spark, Hadoop, Hive, Databricks, and cloud platforms (Azure/GCP), ensuring data readiness for predictive modeling, BI, and machine learning. REQUIREMENTS * Academic Background: Degree in Systems Engineering, Computer Science, Data Science, Information Technology, or related fields * Professional Experience: Minimum 6 years implementing ETL processes and 2 years in Big Data environments * Languages: Technical English (intermediate) **Responsibilities** * Design and maintain ETL and ELT processes to integrate, clean, and transform data in Big Data environments. * Develop data transformations in Spark (SQL, Scala, or Python) for batch and streaming processing. * Perform columnar data modeling oriented toward consumption, optimization, and performance. * Select and optimize Big Data technologies (Hadoop, Hive, Databricks) based on data volume and source type. * Design dimensional models (star schema and snowflake schema) for data warehouses and BI analytics. * Understand business requirements and workflows to propose data integration solutions. * Implement orchestration and automation workflows (Airflow, Oozie, etc.). * Develop processes for extracting data from OLTP systems and loading into OLAP/DWH repositories. * Ensure performance and scalability by optimizing SQL queries and distributed disk operations. * Integrate services via REST APIs and other mechanisms. * Guarantee data quality, security, and consistency across all stages. * Collaborate with Data Scientists, BI Analysts, and Information Architects. * Document processes, data dictionaries, and transformation methodologies. **Essential Technical Skills** * Required Knowledge: Spark, Hadoop, Hive, Databricks, Azure Data Factory, **GCP, Airflow,** DataFlow, ETL, SQL, OLTP/OLAP, Python, Scala, Java, dimensional modeling (star/snowflake), APIs * Key Competencies: ETL / ELT Processes: Pipeline design and automation * Big Data Ecosystems: Spark (SQL, Scala, Python), Hadoop, Hive, Databricks * Cloud Data Platforms: Azure Data Factory, Azure Databricks, Google Cloud Dataproc/Dataflow, BigQuery * Databases and SQL: Advanced SQL and query optimization * Data Modeling: Dimensional models (star/snowflake) * Integration and APIs: Connecting OLTP/OLAP systems and exposing data via REST APIs * Automation and Orchestration: Apache Airflow, Oozie, ADF, CI/CD 034 Integration...gineer.pdf. * Programming Languages: Python, Scala, Java, R * Data Quality and Monitoring: Dataset validation and error control * Agile Methodologies: Scrum/SAFe **Desirable Technical Skills** * Preferred Certifications: Azure Data Engineer Associate, Google Cloud Data Engineer, Databricks Certified Data Engineer, Scrum Fundamentals