Data Engineer – ETL & Streaming

Data Engineer – ETL & Streaming (Kafka, Airflow, Spark)

Location: UAE
Experience: 4–7 years
Department: Data Engineering

About the Role

We are looking for a Data Engineer to develop robust and scalable data pipelines for both batch and real-time processing using Apache Kafka, Airflow, and Spark. This role focuses on implementing business logic, transformations, and data delivery into data lakes and warehouses.

Key Responsibilities

• Build ETL and ELT pipelines integrating multiple source systems (RDBMS, APIs, events).
• Develop data ingestion and transformation jobs in Spark (PySpark or Scala).
• Design and deploy Airflow DAGs to orchestrate end-to-end data workflows.
• Implement real-time data ingestion and processing using Kafka (streams, connectors, topics).
• Ensure data quality, consistency, and lineage across datasets.
• Collaborate with data modelers and BI teams to deliver curated datasets.
• Troubleshoot pipeline failures, optimize jobs for performance and cost efficiency.
• Work closely with the platform team to integrate pipelines into the enterprise data platform.
• Follow DevOps and CI/CD best practices for deployment and monitoring.

Skills & Qualifications

• Strong programming skills in Python (preferred) or Scala/Java.
• Hands-on experience in Kafka (producers, consumers, stream processing, schema management).
• Proficient in Airflow DAG development, scheduling, and task dependency handling.
• Solid experience with Spark (batch, streaming, dataframes, and optimization).
• Working knowledge of SQL and experience with data modeling (Star/Snowflake).
• Experience with data lake and warehouse environments (Azure Synapse, Databricks, Snowflake, Redshift, etc.).
• Understanding of data versioning, partitioning, and incremental load strategies.
• Familiar with Git-based development, CI/CD, and observability practices.

Job Type: Contract
Contract length: 12 months

Pay: AED15,000.00 - AED30,000.00 per month