Lead AI Engineer

Lead AI Engineer - Agentic Test Automation Tysons, VA

*All candidates selected for an interview are required to complete our mandatory identity verification process.

JOB DESCRIPTION

1) Agentic test automation foundation (reusable patterns + reference implementations) • Design and implement

agentic testing patterns

that can be adopted by multiple Underwriting teams (and later other domains). • Create

reference implementations

(sample repos / templates) demonstrating: o Test generation assistance (from requirements, APIs, contracts, schemas) o Test maintenance assistance (auto-updating selectors/contracts, flaky test triage) o Failure analysis assistance (root cause suggestions, log correlation, defect drafting) • Establish a

standard architecture

for test code organization, tagging, data management, and execution across UI + API + service layers.

2) Coverage standards, templates, and governance

Define and publish

coverage standards

(what "good" looks like) including: o Minimum coverage expectations by service/component o Test type mix (unit vs API vs UI vs contract vs integration) o Risk-based prioritization and traceability to requirements

Provide

templates

usable across teams: o Test plan templates o Test case/spec templates (Gherkin-style or equivalent) o Definition of Ready / Definition of Done quality checklists

Create a

scalable tagging/metadata strategy

(e.g., feature, service, risk, priority, data sensitivity) to support reporting and quality gates.

3) GenAI-assisted reporting and quality insights across microservices

Build automated reporting that

aggregates test + service data

across multiple microservices, such as: o Test execution results (Karate/Playwright + CI runs) o Service health signals (logs/metrics/traces if available) o Defect signals (issue tracker metadata if available)

Generate

GenAI-driven summaries : o Release readiness narratives o Failure clustering and trend analysis o "What changed?" insights (commit/PR correlation)

Produce outputs consumable by engineering leadership and teams (dashboards, markdown summaries in PRs, artifacts in CI).

4) "Quality gates" via agents

Build automated review agents that evaluate user stories/requirements for

minimum required clarity and data

before development/testing starts: o Required fields present (acceptance criteria, testable outcomes, data needs, dependencies) o mbiguity detection and missing edge cases o Data/privacy considerations and environment needs

Integrate gates into workflow (PR checks, issue templates, GitHub Actions) to reduce churn and rework.

Required Technical Skills (must-have) GenAI / LLM + agentic development

Hands-on experience building

LLM-powered agents

(tool-using, multi-step reasoning, guardrails). Experience with

prompting patterns , structured outputs (JSON schemas), evaluation, and reducing hallucinations. Ability to design

agent workflows

for: o Test generation/augmentation o Requirements review and completeness validation o Report generation and summarization

GitHub platform + GHCP (Copilot) for engineering workflows

Strong proficiency with

GitHub Copilot

in day-to-day development. Deep experience with GitHub platform capabilities: o

GitHub Actions

(CI/CD pipelines, reusable workflows, composite actions) o PR checks, branch protections, CODEOWNERS, templates

Automation via GitHub APIs/webhooks (as needed)

Test automation engineering (framework expertise)

Advanced experience designing and implementing automation with: o

Karate

(API testing, contract-like checks, data-driven testing, mocks) o

Playwright

(UI automation, selectors strategy, parallelization, trace/video artifacts)

Strong understanding of test design and coverage: o Happy path scenarios o Negative/validation scenarios o Edge/boundary scenarios o Data setup/teardown strategies and test isolation

Cross-service reporting and data aggregation

Proven ability to aggregate and normalize results from

multiple microservices

and multiple pipelines. Experience producing actionable automated reports (trend analysis, failure clustering, service correlation).

Automated requirements review agents

Experience implementing automated checks that validate: o cceptance criteria completeness o Required test data and environment dependencies o Non-functional requirements (performance, security, observability) when applicable

Deliverables / What success looks like (for the posting)

A reusable

agentic testing automation kit

adopted by multiple teams. Published

coverage standards + templates

and onboarding documentation. A working

GenAI-assisted reporting pipeline

aggregating results across microservices. Automated

quality gates

integrated into GitHub workflows that measurably reduce story churn.