Intermediate Data Engineer

Intermediate Data Engineer
Minimum Requirements:

- Matric (Grade 12)

- Masters in Software Engineering, Data Engineering, Computer Science or related field

- 2-3 years of relevant work experience

- Strong Scala or Java background

- Knowledge of AWS, GCP, Azure, or other cloud platform

- Understanding of data modeling principles

- Ability to work with complex data models

- Experience with relational and NoSQL databases (e.g. Postgres, ElasticSearch/OpenSearch, graph databases such as Neptune or neo4j)

- Experience with technologies that power analytics (Spark, Hadoop, Kafka, Docker, Kubernetes) or other distributed computing systems

- Knowledge of API development and machine learning deployment

Responsibilities:

- Build and scale data infrastructure that powers real-time data processing of billions of records in a streaming architecture

- Build scalable data ingestion and machine learning inference pipelines

- Build general-purpose APIs to deliver data science outputs to multiple business units

- Scale up production systems to handle increased demand from new products, features, and users

- Provide visibility into the health of our data platform (comprehensive view of data flow, resources usage, data lineage, etc) and optimize cloud costs

- Automate and handle the life-cycle of the systems and platforms that process our data

Originally posted on Himalayas