Senior Data Engineer

Role Overview

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.
Responsibilities

- Design and maintain batch and real-time data pipelines.

- Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services.

- Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.

- Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR.

- Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift.

- Design analytical data models, including Star and Snowflake schemas.

- Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik.

- Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.

- Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control.

- Monitor production data pipelines and proactively address operational and performance issues.

Mandatory Requirements

- Proven experience as a Senior Data Engineer or in a similar role.

- Advanced proficiency in Python and SQL.

- Hands-on experience with:
- Apache Spark / PySpark

- Apache Kafka

- Apache Airflow

- Strong AWS experience, particularly with:
- Amazon S3

- Amazon Redshift

- AWS EMR

- Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.

- Strong understanding of batch and streaming architectures.

- Experience with performance optimization, Git, testing practices, and production support.

Technical Stack

- Python

- SQL

- Apache Spark / PySpark

- Apache Kafka

- Apache Airflow

- AWS EMR

- Amazon S3

- Amazon Redshift

- Git

- Qlik

Originally posted on Himalayas