We are seeking a highly skilled Senior Databricks AI Engineer to design, develop, and deploy scalable data and AI solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in Data Engineering, Machine Learning, Generative AI, and cloud technologies, with hands-on experience building enterprise-grade AI applications and data platforms.
Key Responsibilities
Data Engineering
Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
Build ELT/ETL solutions for structured and unstructured data.
Implement Medallion Architecture (Bronze, Silver, Gold layers).
Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency.
Manage data ingestion from multiple cloud and on-premise sources.
AI & Machine Learning
Develop and deploy Machine Learning models using Databricks ML and Python.
Implement MLOps practices for model training, deployment, monitoring, and governance.
Build predictive analytics solutions and recommendation engines.
Create feature stores and reusable ML pipelines.
Generative AI
Design and develop GenAI applications using:
Large Language Models (LLMs)
Retrieval Augmented Generation (RAG)
Vector Databases
AI Agents and Multi-Agent Frameworks
Integrate Databricks Mosaic AI, LangChain, LangGraph, and OpenAI/Azure OpenAI services.
Fine-tune and evaluate foundation models.
Develop enterprise chatbots, knowledge assistants, and intelligent automation solutions.
Cloud & Architecture
Implement solutions on Azure, AWS, or GCP.
Design secure and scalable Lakehouse architectures.
Work with Unity Catalog, Delta Sharing, and data governance frameworks.
Collaborate with business stakeholders, architects, and data scientists to deliver AI-driven business solutions.
Required Skills
Databricks Lakehouse Platform
PySpark, Spark SQL
Python
Delta Lake
Databricks Workflows
Databricks Unity Catalog
Azure Data Factory / AWS Glue
SQL and Data Modeling
Git, CI/CD, DevOps
AI & GenAI Skills
Machine Learning
Deep Learning
LLMs
RAG Architecture
Prompt Engineering
LangChain / LangGraph
Vector Databases (FAISS, Pinecone, ChromaDB)
Databricks Mosaic AI
MLflow
Cloud Platforms
Microsoft Azure (Preferred)
AWS
Google Cloud Platform
Qualifications
Bachelor's or Master's degree in Computer Science, Data Science, Engineering, Information Technology, or related field.
6+ years of experience in Data Engineering.
3+ years of experience with Databricks.
Experience delivering AI/ML or Generative AI solutions in production environments.
Databricks Certified Data Engineer or Machine Learning certification preferred.
Preferred Experience
Healthcare, Life Sciences, Banking, Retail, or Manufacturing domain experience.
Experience with Azure OpenAI Service.
Knowledge of Agentic AI and AI Governance.
Experience in enterprise-scale data modernization programs.
Key Deliverables
Scalable Databricks data pipelines.
Enterprise AI and GenAI solutions.
Real-time analytics and AI-powered insights.
AI-enabled automation and decision-support systems.
Secure and governed Lakehouse architecture.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.