Lead Data Engineer

Description

Summary: Seeking an experienced Lead Data Engineer to design and rapidly deliver an Oil and Gas data processing prototype leveraging Azure Databricks, ADLS Gen2, and Delta Lakehouse architecture. Highlights: 1. Design and build a scalable data processing prototype for Oil & Gas 2. Leverage Azure Databricks, ADLS Gen2, and Delta Lakehouse architecture 3. Focus on data mastering with exact matching and geospatial calculations We are seeking an experienced **Lead Data Engineer** to build an Oil and Gas data processing prototype leveraging Azure Databricks, ADLS Gen2, and Delta Lakehouse medallion architecture. This role focuses on data mastering using exact matching and geospatial calculations. The prototype must be delivered rapidly to enable extension into a fully\-fledged solution. **Responsibilities** * Design and build a data processing prototype for the Oil and Gas domain * Develop scalable data pipelines using Azure Databricks and PySpark * Implement Delta Lakehouse medallion architecture for structured data flow * Manage and organize data storage in ADLS Gen2, working with Parquet and JSON formats * Perform data mastering using exact matching and geospatial calculations * Write and optimize queries using Spark SQL for efficient data transformation * Collaborate with stakeholders to ensure the prototype meets business requirements * Deliver the prototype rapidly to support extension into a full\-scale solution **Requirements** * 5\+ years of experience in data engineering * At least 1 year of relevant leadership experience * Expertise in Azure Databricks, Azure Data Lake Storage, and ADLS Gen2 (Parquet and JSON) * Proficiency in PySpark and Spark SQL * Background in building data pipelines using Delta Lakehouse medallion architecture * Capability to deliver prototypes rapidly under tight timelines * Excellent written and verbal communication skills in English (B2\+ level) **Nice to have** * Skills in data matching and deduplication using geospatial calculations * Familiarity with Databricks DLT, Databricks Unity Catalog, and Databricks Asset Bundles * Oil and Gas domain experience