Senior Data Engineer

Every day, somewhere in the world, important decisions are made. Whether it is a private equity company deciding to invest millions into a business or a large corporation implementing a new strategic direction, these decisions impact employees, customers and other stakeholders.

Consulting and private equity firms come to proSapient when they need to discover knowledge to help them make great decisions and succeed in their goals. It is our mission to support them in their discovery of knowledge.

We help our clients find industry experts who can provide their knowledge via interview or survey; we curate this knowledge in a market-leading software platform; and we help clients surface knowledge they already have through expansive knowledge management.

About the role:

We are looking for a Senior Data Engineer to build and own those pipelines. This is a hands-on, delivery-focused role on a team that ships frequently, working directly with the Lead Search Engineer, an AI/ML Engineer and a PM measured on match quality and CSAT.

Key Duties & Responsibilities:

Pipelines & Data Products
- Build and operate batch and real-time pipelines feeding the matching and search systems.

- Work closely with real-time data streams and search infrastructure using tools like Kafka and Elasticsearch.

- Model the datasets the product layer consumes — candidate features, interaction events, ranking signals — and keep the contracts stable.

- Own the pipelines you build in production: monitoring, alerting, backfills, and the on-call reality of a broken 3am job.

Feedback Loops & Instrumentation
- Instrument recruiter and client behaviour so matching quality can be measured and improved.

- Build the event pipelines behind our CSAT and workflow metrics.

- Conduct and analyse experiments to rapidly iterate on the quality of data processing pipelines and make experiment data queryable without a bespoke analysis every time.

Quality & Reliability
- Add data quality checks and tests at the boundaries that matter and make failures loud instead of silent.

- Reduce cost and latency where it counts, leaving the rest alone.

- Partner with QA to make data correctness a testable property, rather than a hope.

- Identify and provide useful data sources, internal or external, for our structured search and GenAI tools.

Required Skills and Experience:
- 5+ years building production data pipelines.

- Strong Python programming skills, with practical experience writing clean, production-ready code.

- Strong SQL and proficiency with PostgreSQL or other relational databases.

- Strong hands-on experience with Elasticsearch / OpenSearch, including performance optimisation and query tuning.

- Deep understanding of both batch and event/streaming patterns, and a clear sense of when each is appropriate.

- Practical data modelling skills for analytics and for serving application features.

- Experience operating your own pipelines in production (and learned from it), alongside comfort working in a codebase with tests, code review and CI.

Nice to have skills:
- Message brokers like Kafka or RabbitMQ.

- Cloud data warehouse — BigQuery preferred, equivalents fine.

- Docker, Kubernetes, and cloud environments (AWS and/or GCP).

- Experience feeding search, ranking or ML systems specifically.

- Building LLM-based extraction or enrichment pipelines.

- Experience with dbt or a comparable transformation framework.

- Observability tools (e.g. Prometheus, Datadog) and background job processing frameworks.

- Exposure to Elixir services (part of our platform).

What we can offer you:

- Tenure gifts, including vouchers, extra holiday and sabbaticals for each year of employment.

- Health insurance through Vitality.

- Remote working for up to 20 days each year, giving you flexibility and a change of scenery.

- Employee Assistance Programme with personalised health and wellbeing advice from specialist teams.

- Enhanced maternity and paternity pay.

- 25 days’ annual leave plus bank holidays, including a week’s closure over Christmas.

- MyMindPal app for online mental fitness support.

- Corporate events, from quarterly gatherings to annual winter and summer parties.

We are committed to building an inclusive workplace. Marginalised groups are often less likely to apply unless they meet every requirement listed, so if you are interested in this role but do not tick every box, we encourage you to apply anyway — it could still be a great match.