Expertise in LangChain, LlamaIndex, Semantic Kernel, AutoGen, or equivalent
Key Responsibilities
• Agent Development: Build and orchestrate autonomous AI agents with multi-step reasoning, tool usage, and workflow chaining using frameworks like LangChain, CrewAI, AutoGen, Semantic Kernel, or LlamaIndex.
• LLM Integration & Optimization: Deploy, fine-tune, and serve open-source LLMs (e.g., Llama 3) using Databricks Model Serving; optimize latency, throughput, and cost.
• RAG & Knowledge Systems: Design advanced RAG pipelines leveraging vector search, embeddings, semantic ranking, and enterprise data sources (structured + unstructured).
• Context Engineering: Develop prompt strategies, memory frameworks, and metadata tagging to improve contextual accuracy and response quality.
• UI & Experience Design: Build intuitive AI-driven applications using Databricks Apps (Streamlit/Dash) or modern web frameworks to enable business consumption.
• Data Engineering for AI: Build reliable data pipelines (batch & streaming) supporting training, inference, and feature generation using Delta Lake.
• Security & Governance: Implement enterprise-grade controls using Unity Catalog (row/column-level security, lineage, auditability) aligned with compliance standards.
• LLM Guardrails & Responsible AI: Implement guardrails (e.g., NeMo Guardrails) for prompt injection prevention, hallucination mitigation, and safe output handling.
• MLOps & AIOps: Establish CI/CD pipelines for AI models and agents, including versioning, monitoring, drift detection, observability, and incident response.
• Performance & Cost Optimization: Optimize model performance, GPU/compute usage, and inference cost efficiency across environments.
• Testing & Evaluation
• Collaboration & Stakeholder Engagement
• Documentation & Knowledge Transfer
Required Skills and Qualifications
Databricks & Lakehouse
• Strong experience with Unity Catalog, Delta Lake, Vector Search, Databricks Workflows, and Model Serving • Hands-on with Lakehouse architecture patterns
LLMs & Generative AI
• Experience with open-source LLMs (Llama, Mistral, etc.), prompting techniques, and fine-tuning approaches • Strong knowledge of RAG architectures and embedding strategies
AI Engineering & Frameworks
• Expertise in LangChain, LlamaIndex, Semantic Kernel, AutoGen, or equivalent • Experience building agentic workflows and multi-agent systems
Programming
• Advanced Python proficiency (APIs, web apps, orchestration, data processing) • Familiarity with REST APIs and microservices architecture
MLOps & Monitoring
• Experience with MLflow, CI/CD pipelines, model lifecycle management, and observability tools • Knowledge of drift detection and model performance monitoring
Data Engineering Foundations
• Experience with Spark, SQL, and large-scale data processing • Familiarity with streaming frameworks (Kafka, Structured Streaming)
Security & Governance
• Expertise in AI security risks (prompt injection, jailbreaks, data leakage) • Experience implementing governance frameworks and compliance controls
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.