QA Engineer (SDET) — AI, Data & Platform Quality
Programmatic QA • Testing for LLMs & Agents • Data Quality • Platform Reliability
About Finalytics.ai
Finalytics.ai is the leading provider of personalization for the financial industry. Our platform combines data integrations, machine learning, and real-time technology to make digital experiences more relevant and higher-converting for credit unions and banks. We're a growing startup led by industry veterans, building the next generation of AI-driven personalization.
Why This Role Is Different
QA at Finalytics goes well beyond clicking through a UI. Our platform makes model-driven decisions, runs LLMs and agents that generate content and answer questions, and depends on data pipelines that feed those models every day — and all of it has to be tested programmatically.
We're looking for an engineering-minded QA team contributor to help build quality across three areas: our core personalization features, our LLM and agentic capabilities, and the data that powers them. This is a coding role, embedded in the same repo and release flow as our engineers that will report directly to the CTO. You won't just find bugs — you'll build the automated tests, evals, and data checks that let a small team ship trustworthy AI every sprint.
Our stack is Python/Django with a JavaScript personalization tag, backed by MySQL, Celery, BigQuery, and AWS.
What You'll Do
1. Programmatic QA of Core Features
-
Extend our scenario test runner — a proprietary harness that captures real production personalization requests and replays them across environments, asserting on expected algorithms and content selection. Grow it into automated regression across every client.
-
Write automated tests in Python with pytest across our tiers — unit, integration, HTTP, and end-to-end.
-
Build headless Playwright end-to-end tests to verify how personalized content and tracking render on real client pages.
-
Harden the pre-deploy quality gate and pre-commit checks that block bad changes automatically.
2. Testing & Standardizing LLMs and Agents
-
Design evals for non-deterministic AI features — our conversational analytics assistant, AI content builders, and generative SEO — measuring correctness, grounding, and regression across prompt and model versions.
-
Test the tool-calling and agentic layers — that function-calling loops pick the right tools and guardrails hold on adversarial input.
-
Validate our agent/MCP interface — contract conformance, rate limiting, authorization, and safe failure.
-
Help set our standards for shipping AI — catching hallucinations and drift, and benchmarking prompt/model changes before clients see them.
3. Data Quality Engineering
-
Build automated data-health checks that flag stale rollups, incomplete coverage, and broken aggregations before they hit a client dashboard.
-
Validate data pipelines end-to-end — rollups, funnel/rate/financial ingestion, and BigQuery — with drift detection across environments.
-
Guard model inputs so the signals our ML depends on stay accurate and complete.
4. Reliability & Performance
-
Track platform performance — response times, JS load, and page speed — and help keep it fast.
-
Stand up quality dashboards — uptime, coverage, data-health, and eval scores.
5. Collaboration & Bug Lifecycle
-
Work in the codebase alongside engineers to diagnose issues across development, release, and deployment.
-
Drive the bug lifecycle — reproduce, capture with a failing test, and verify the fix.
What We're Looking For
-
3+ years in QA/SDET or test automation with a code-first approach.
-
Strong Python — you write clean test code and can read the app you're testing.
-
pytest (preferred) and browser automation (Playwright or Selenium).
-
API and contract testing experience.
-
A genuine interest in testing AI — comfortable with non-determinism, evals, and prompts.
-
Data-savvy — strong SQL, and the instinct to validate pipelines and reconcile data.
-