Lead Data Engineer
We are looking for a Lead Data Engineer who can own more than pipelines. Beyond building and maintaining ETL and integrations, you'll own and implement data engineering best practices that make a platform trustworthy at scale: orchestration, observability, lineage, data modeling, and data contracts.
This is a design-forward, lead level role. You'll set the patterns the rest of the team builds on, and you'll be the person who can trace any record end to end, where it came from, every transformation it passed through, and every downstream consumer that depends on it so that when something upstream changes, you know exactly what it touches and can be confident the change propagates cleanly.
What you'll own:
-
Ingestion. Own how data enters the platform across all channels: automated feeds, batch loads, and source-system extracts from CRM and case-management systems. Design ingestion that validates structure and mandatory elements at the door, handles varying formats and volumes across markets and products. You'll build ingestion to be reusable and configurable, so onboarding a new source or market is a known pattern rather than a one-off, and you'll make sure every inbound record carries the lineage and contract metadata the downstream layers depend on.
-
Orchestration. Design the data flow layer that routes datasets through the correct processing sequence and propagates change to everything that depends on it. You'll build dependency-aware refresh, ordered execution, retry and error handling, and clean rollback, with controls on changes to core routing and strong backup/disaster recovery practices. You'll favor versioned, auditable orchestration-as-code in the existing stack over bolt-on tooling.
-
Observability. Stand up the monitoring that proves the platform is healthy and ensure data integrity. This means data-quality checks at key transformation points, deviation monitoring for volume and expectations, automated flagging of anomalies, and exception reporting on breaches. You'll put monitoring on the seams where systems hand off to each other.
-
Lineage. Build native lineage from ingestion through to reporting, so every element is traceable across transformations. You'll make lineage a first-class capability that supports debugging, impact analysis, audit, and compliance.
-
Data modeling. Own the models that underpin the platform: the medallion layers (raw → validated → interim → canonical, with history/SCD handling), canonical entity and master-ID design, and the linkage between datasets. You'll design master-data and naming conventions and build models flexible enough to absorb differing data requirements by market and product rather than locking to a rigid schema. Sound entity/master-ID modeling is central to this role.
-
Data contracts. Establish declared contracts between producers and consumers so interfaces are explicit and changes are safe.
-
Pipelines and integrations. Design, build, and harden ETL/ELT across ingestion channels into the lake house and out to the operational serving layer. Own reliability, performance, and reusability.
Requirements
- 6+ years in data engineering, with senior-level ownership of production data platforms.
- Deep hands-on experience with Databricks (Spark, notebooks, Delta Lake, orchestration/workflows, cluster management) and PostgreSQL (complex SQL, stored procedures, materialized views, performance tuning).
- Proven experience building orchestration for complex, dependency-heavy flows, including dependency-aware refresh, error handling, and rollback.
- Demonstrated data observability / data quality work: automated checks, anomaly and deviation detection, alerting — with a clear understanding that job success is not the same as correct data.
- Experience implementing or operating data lineage and a grasp of why traceability matters for debugging, impact analysis, and audit.
- Strong data modeling across relational and lake-house paradigms: layered/medal