AI Engineer / Generative AI Engineer

TMS LLC is seeking an

AI Engineer / Generative AI Engineer

to help build production-grade AI agents and bring them through deployment, monitoring, and continuous improvement. This

contract

role is based in

Los Angeles, CA

with a

hybrid

work model, and focuses on agent development, evaluation, and observability in real environments. The work centers on developing and operating AI agents using Python and the Claude Agent SDK, deploying via AWS Bedrock AgentCore, and strengthening quality through golden datasets, LLM-as-a-Judge evaluations, and CI/CD quality gates. What you will do

Develop AI agents using

Python

and

Claude Agent SDK . Deploy agents using

AWS Bedrock AgentCore Runtime . Design and maintain

golden datasets

for reliable evaluation coverage. Implement

LLM-as-a-Judge

evaluations, including calibration against human labels. Integrate automated evaluation workflows into

CI/CD pipelines

as

quality gates . Set up

Langfuse

and/or

OpenTelemetry

for AI observability, tracing, and monitoring. Evaluate agent outcomes including

accuracy ,

task completion ,

tool execution , and

performance . Support

RAG pipelines

and

AWS data processing workflows . Required skills

Experience in

Python

and

agentic AI

development. Hands‑ on work with

AWS Bedrock AgentCore Evaluations

(including

Bedrock evaluation ). Ability to create, curate, and use

golden datasets . Experience designing

LLM-as-a-Judge

systems and calibrating them against

human labels . Automated evaluation implementation and integration into

CI gates . Experience with

Langfuse

and/or

OpenTelemetry . Technology stack

Python, Claude Agent SDK AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime LLM-as-a-Judge, CI/CD Langfuse, OpenTelemetry RAG, AWS data processing workflows Bedrock evaluation Nice to have

RAGAS

or equivalent evaluation frameworks. Trajectory evaluation

and agentic

red‑teaming . Bedrock Guardrails ,

Strands Agents SDK , and

GitHub Actions . Qualifications

15+ years

of experience (minimum). 6–8+ years of software engineering, cloud engineering, or AI/ML experience. Hands‑on experience with production AI agents and

continuous evaluation . Strong problem‑solving and troubleshooting skills.

#J-18808-Ljbffr