Responsibilities
- Build foundational evaluation infrastructure
- Develop comprehensive evaluation systems including golden dataset creation, automated benchmarking, and model comparison tools
- Automate knowledge generation by creating synthetic guides, documentation, and metadata
- Optimize AI agent accuracy via retrieval systems, prompt optimization, and automated evaluation
- Develop intelligence infrastructure for model upgrades, A/B testing, and scalable evaluation
- Drive volume automation for automated resolutions across voice, chat, and email
About You
- 5+ years of software engineering experience as an individual contributor
- Experience with AI evaluation systems, data pipelines, or AI model optimization
- Passion for building systems to measure and improve AI performance
- Experience with retrieval systems, knowledge bases, or information extraction
- Enjoy building tools and infrastructure to enhance engineer and AI system performance
- Highly ambitious with a focus on customer impact and problem-solving
- Adaptable to fast-paced environments and new insights
- Creative and innovative mindset
- Proven impact in small teams or startups
- Humble, open to feedback, and values teamwork
Technologies You'll Work With
- Python/Golang for evaluation systems and data processing
- LLMs and prompt optimization frameworks
- Vector databases and retrieval systems
- AI evaluation and benchmarking tools
- Data pipeline and automation infrastructure
Our U.S. benefits
- Generous medical, dental, and vision benefits
- Paid company holidays, sick time, and unlimited time off
- Monthly credits for professional development, wellness, customers, and commuting
- Paid parental leave
- Hybrid work model with catered lunches, snacks, and beverages in SF & NY offices
- 401(k) plan enrollment
