The Role
We are looking for a detail-oriented Product Quality & AI Evaluation Specialist to help us build the future of civic tech. In this role, you will be the bridge between our cutting-edge AI features and the real-world impact they have on organizers. You won't just be checking boxes; youβll be developing a deep intuition for how LLMs behave, ensuring our models provide accurate, helpful, and reliable support to our users.This is a hybrid role that blends AI evaluation, systematic QA, and user research support. Youβll spend your time rigorously testing our product, building automated test suites to catch regressions, and working closely with our engineering team to prioritize and squash bugs. Beyond the technical side, youβll help us stay grounded in user needs by coordinating research sessions and synthesizing feedback from the field.
Key Responsibilities
- LLM Evaluation: Review and score outputs from our AI features, helping us understand where the model gets it right and where it falls short. You'll develop a real intuition for what "good" looks like in AI-assisted civic tech.
- Automated Testing: Contribute to building out our automated LLM test suite, including writing and maintaining test cases that help us catch regressions as the product evolves.
- QA: Systematically test product features, document bugs with clear reproduction steps, and help us maintain a high bar for what ships.
- Bug Triage: Help review incoming bug reports, identify patterns, and prioritize issues alongside the engineering team.
- User Research Support: Help coordinate and set up user interviews with organizers, including scheduling, note-taking, and synthesizing what we hear.
- Other duties and projects as assigned.
Knowledge, Skills, and Abilities
- Ability to proactively lead, design, and create research projects, workflows, and initiatives.
- Demonstrated excellence in advanced research and analytical skills, including primary and secondary source research, writing, and editing.
- Project management skills, including capabilities to drive efforts independently and ensure effective implementation.
- Ability to coordinate efforts across multiple stakeholders, partners, and teams.
- Ability to work under pressure and across multiple work-streams to meet timelines.
- Exceptional analytical mind with the ability to solve complex, multifaceted problems, synthesize arguments, and generate new ideas.
- Strong teamwork skills and a record of impact in high-intensity, team-based environments.
- Commitment to the highest standards of ethics and confidentiality.
- Excellent written and verbal communication skills, with reasonable ability to use common software tools for word processing, presentation, and spreadsheets.
- Curiosity and optimism in the face of uncertainty.
Requirements
- Maintain the highest level of ethics, integrity, and discretion for confidentiality.
- Be based at the Schmidt offices in New York City and be able to travel within the U.S. as needed.
