D
AI Engineer
DataJobs.com
2 hours ago
Full-time
On-site
Own the AI safety layer
for Realtor.com’s RealAssist, an LLM-powered real-estate assistant used across web and mobile. On the AI Integrations Team at News Corp, you will build and tune
LLM safety guardrails
that act as a launch gate for Fair Housing compliance and content moderation, with runtime prompt-injection and jailbreak screening designed for production. What you’ll do
Build, tune, and take end-to-end ownership of
LLM-as-a-judge classifiers
for Fair-Housing compliance and content moderation, targeting
high recall
on disallowed content without overblocking legitimate users Design and run
guardrail evaluation pipelines , including labeled dataset curation, train/test/validate splits, and offline
prod-replay
to evaluate overblocking Use defensible metrics by reporting
confusion matrices
and
precision/recall/F1
per category to guide safety decisions Integrate and operate cloud guardrail services as a
runtime prompt-injection/jailbreak screening
layer, including
fail-open behavior
and alerting Red-team the assistant and wire
safety regression checks
into CI to prevent silent guardrail degradation as the product grows Manage guardrail infrastructure as code using
Terraform
(templates, IAM, and project shape), and participate in technical design reviews and architecture discussions Partner with ML, backend, product, and legal/compliance stakeholders to define
risk tiering
and get new AI capabilities through safety review before launch Independently design, implement, and tune guardrail and evaluation systems with a focus on balancing
safety, latency, and user experience ; write clean, maintainable, well-tested code Keep eval datasets calibrated to real traffic and emerging attack patterns, staying current with AI-safety research and new guardrail tooling What you bring
AI Safety & Evaluation Expertise
(required) 4+ years
of professional software/ML engineering experience with hands‑on LLM application work Bachelor’s degree or equivalent experience Strong
Python
proficiency Real classification-metrics literacy:
precision/recall/F1
(macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distribution Experience evaluating LLM prompts/classifiers with an eval framework such as
DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals , or a homegrown harness Ability to turn small seed sets into robust labeled eval datasets using dataset-synthesis and prompt-optimization loops Familiarity with prompt-injection/jailbreak defense concepts, including
OWASP LLM Top 10 , input/output filtering, least-privilege tool access, and adversarial testing Hands‑on experience with cloud guardrail services:
Google Cloud Model Armor ,
AWS Bedrock Guardrails , or
Azure AI Content Safety Terraform/IaC
for cloud infrastructure Experience with
Google Cloud Vertex AI / Gemini Understanding of monitoring and observability tools (for example,
New Relic
or similar) Exposure to regulated, compliance-sensitive domains (fair housing, fair lending, healthcare, finance, trust & safety) Red‑teaming or AI‑security background Tools you’ll work with
Python ,
DeepEval ,
RAGAS ,
Arize Phoenix ,
LangSmith ,
OpenAI Evals ,
OWASP LLM Top 10 ,
Google Cloud Model Armor ,
AWS Bedrock Guardrails ,
Azure AI Content Safety ,
Terraform ,
Google Cloud Vertex AI / Gemini ,
New Relic Onsite role with strong support and collaboration
The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. The work blends creativity and innovation with in-person collaboration. For most roles, employees work
four or more days in office . Benefits
Inclusive and competitive
medical, Rx, dental, and vision
coverage Family forming benefits 13 Paid Holidays Flexible Time Off 8 hours
of paid volunteer time off Immediate eligibility into the
Company 401(k)
plan with
3.5% company match Tuition Reimbursement
for degreed and non-degreed programs 1:1 personalized Financial Planning Sessions Free snacks and refreshments in each office location Student Debt Retirement Savings Match
program
#J-18808-Ljbffr
for Realtor.com’s RealAssist, an LLM-powered real-estate assistant used across web and mobile. On the AI Integrations Team at News Corp, you will build and tune
LLM safety guardrails
that act as a launch gate for Fair Housing compliance and content moderation, with runtime prompt-injection and jailbreak screening designed for production. What you’ll do
Build, tune, and take end-to-end ownership of
LLM-as-a-judge classifiers
for Fair-Housing compliance and content moderation, targeting
high recall
on disallowed content without overblocking legitimate users Design and run
guardrail evaluation pipelines , including labeled dataset curation, train/test/validate splits, and offline
prod-replay
to evaluate overblocking Use defensible metrics by reporting
confusion matrices
and
precision/recall/F1
per category to guide safety decisions Integrate and operate cloud guardrail services as a
runtime prompt-injection/jailbreak screening
layer, including
fail-open behavior
and alerting Red-team the assistant and wire
safety regression checks
into CI to prevent silent guardrail degradation as the product grows Manage guardrail infrastructure as code using
Terraform
(templates, IAM, and project shape), and participate in technical design reviews and architecture discussions Partner with ML, backend, product, and legal/compliance stakeholders to define
risk tiering
and get new AI capabilities through safety review before launch Independently design, implement, and tune guardrail and evaluation systems with a focus on balancing
safety, latency, and user experience ; write clean, maintainable, well-tested code Keep eval datasets calibrated to real traffic and emerging attack patterns, staying current with AI-safety research and new guardrail tooling What you bring
AI Safety & Evaluation Expertise
(required) 4+ years
of professional software/ML engineering experience with hands‑on LLM application work Bachelor’s degree or equivalent experience Strong
Python
proficiency Real classification-metrics literacy:
precision/recall/F1
(macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distribution Experience evaluating LLM prompts/classifiers with an eval framework such as
DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals , or a homegrown harness Ability to turn small seed sets into robust labeled eval datasets using dataset-synthesis and prompt-optimization loops Familiarity with prompt-injection/jailbreak defense concepts, including
OWASP LLM Top 10 , input/output filtering, least-privilege tool access, and adversarial testing Hands‑on experience with cloud guardrail services:
Google Cloud Model Armor ,
AWS Bedrock Guardrails , or
Azure AI Content Safety Terraform/IaC
for cloud infrastructure Experience with
Google Cloud Vertex AI / Gemini Understanding of monitoring and observability tools (for example,
New Relic
or similar) Exposure to regulated, compliance-sensitive domains (fair housing, fair lending, healthcare, finance, trust & safety) Red‑teaming or AI‑security background Tools you’ll work with
Python ,
DeepEval ,
RAGAS ,
Arize Phoenix ,
LangSmith ,
OpenAI Evals ,
OWASP LLM Top 10 ,
Google Cloud Model Armor ,
AWS Bedrock Guardrails ,
Azure AI Content Safety ,
Terraform ,
Google Cloud Vertex AI / Gemini ,
New Relic Onsite role with strong support and collaboration
The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. The work blends creativity and innovation with in-person collaboration. For most roles, employees work
four or more days in office . Benefits
Inclusive and competitive
medical, Rx, dental, and vision
coverage Family forming benefits 13 Paid Holidays Flexible Time Off 8 hours
of paid volunteer time off Immediate eligibility into the
Company 401(k)
plan with
3.5% company match Tuition Reimbursement
for degreed and non-degreed programs 1:1 personalized Financial Planning Sessions Free snacks and refreshments in each office location Student Debt Retirement Savings Match
program
#J-18808-Ljbffr