Gen AI Engineer
Position: Contract
Location:
Mountain View (hybrid) - Remote also fine.
Duration: 12+ months
Job description:
Core Responsibilities
1. Evaluation & Metrics (Primary Focus)
• Define and measure:
o Accuracy (primary metric)
o Precision, recall, confusion metrics
o Hallucination rate
o Faithfulness / grounding
• Build ground truth datasets
• Evaluate retrieval:
o Precision@k
o Recall@k
• Perform:
o Prompt A/B testing
o Bias reduction
2. Evaluation Tooling
• RAG / RAGAS
• Databricks / MLflow
• Arize, TruLens (production monitoring)
• DeepEval (CI/CD integration)
• LangSmith (enterprise)
• LLM-as-judge frameworks
3. GenAI Workflow Orchestration
• Control flow and logging
• Memory management
• Observability pipelines
• Document comparison workflows
4. Agentic AI Evaluation
• Measure:
o Tool correctness
o Agent success rate
• Use:
o LLM-as-judge for agent evaluation
5. Observability
• Monitor:
o Latency
o Token/cost usage
o Response quality
o Hallucination trends
o Retrieval accuracy
6. System Thinking
• Structured problem-solving
• Metric-driven decision making
• Clear articulation of tradeoffs:
o Precision vs recall
o Cost vs performance
Engineering Requirements
• Strong Python (data + evaluation focus)
• Strong analytical thinking (more important than complex coding)
Nice to Have
• Cost optimization awareness
• Platform tradeoff understanding
Key Hiring Insight
Most candidates fail because they:
• Lack depth in evaluation frameworks
• Cannot clearly explain metrics or evaluation methodology