Senior AI Engineer, ARIA Team

Job Summary :

Klaviyo is a company that empowers creators to own their destiny through first-party data. They are seeking a Senior AI Engineer to design, build, and scale production-grade AI systems, focusing on LLM integrations and automation tools, while collaborating closely with product and data science teams.

Responsibilities : • Design and build LLM-powered applications: Architect and develop production-grade systems leveraging large language models (GPT-4, Claude, Mistral, Llama) for internal automation, knowledge retrieval, and intelligent workflow assistance. • Build and maintain AI integration pipelines: Design robust, scalable API integrations between AI models and internal systems such as HRIS, ticketing, CRM, and knowledge bases, ensuring reliability, low latency, and high availability. • Develop full-stack AI features end-to-end: Own the complete development of AI features across the stack from backend model orchestration and API layers to frontend interfaces that make AI accessible to non-technical employees. • Architect and manage CI/CD pipelines for AI systems: Build and maintain automated deployment pipelines for AI models and services, including model evaluation frameworks, A/B testing infrastructure, and safe rollout strategies. • Implement RAG and retrieval systems: Design and build retrieval-augmented generation systems grounded in Klaviyo internal knowledge, ensuring accuracy, relevance, and up-to-date responses at scale. • Evaluate and fine-tune AI models: Lead benchmarking, evaluation, and fine-tuning of foundation models for Klaviyo use cases, including prompt engineering and parameter-efficient fine-tuning techniques. • Establish AI engineering best practices: Define coding standards, testing frameworks, and architectural patterns for AI development at Klaviyo, enabling the team to ship reliable AI features consistently. • Build AI observability and monitoring tooling: Instrument AI systems with latency monitoring, output quality evaluation, hallucination detection, cost dashboards, and production alerting. • Collaborate with Product and Data Science: Work closely with PMs, data scientists, and ML engineers to translate requirements into technical implementations, maintaining a tight feedback loop between experimentation and production. • Drive security and compliance in AI systems: Ensure AI integrations follow data privacy best practices, implement guardrails for sensitive data handling, and maintain compliance with Klaviyo security and governance standards.

Qualifications : Required : • 3+ years of software engineering experience, with at least 2 years focused on building AI, ML, or LLM-powered systems in production. • Deep expertise with large language models such as OpenAI, Anthropic, Mistral, or Llama, and experience integrating them into production applications via API. • Strong full-stack engineering skills across Python (FastAPI, Flask, or Django) and TypeScript or JavaScript, with experience in modern frontend frameworks. • Proven experience designing and building CI/CD pipelines for software or ML systems using tools such as GitHub Actions, Jenkins, or CircleCI. • Experience with RAG systems, vector databases (Pinecone, Weaviate, pgvector), and embedding models. • Proficiency with cloud infrastructure (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code tools such as Terraform or Pulumi. • Experience with prompt engineering, model evaluation frameworks, and LLM observability tools such as LangSmith or Weights and Biases.

Company :

Klaviyo is an automation and email platform designed to help grow businesses. Founded in 2012, the company is headquartered in Boston, USA, with a team of 1001-5000 employees. The company is currently Late Stage.