Responsibilities
- Participate in the entire software development lifecycle, encompassing all stages from requirements analysis to test planning, execution, defect tracking, through to product release and maintenance.
- Go to person in relation to A.I Agents evaluation and continuously monitoring.
- Create comprehensive and effective test strategies and hands-on testing to ensure the accuracy, reliability, and performance of AI and data applications.
- Root cause analysis of test failures and product issues in an effective manner, and drive optimization for future enhancements.
- Design and develop internal tools leveraging AI technology to improve engineering and testing work efficiency.
Requirements
- Bachelors or Masters degree in Computer Science, Artificial Intelligence, Data Science, or a related field.
- Strong understanding of Large Language Models (LLMs), autonomous AI agents, and their system architectures.
- Experience with AI evaluation methodologies, including offline benchmarking, online monitoring, and hybrid human-AI evaluation approaches.
- Familiarity with software engineering best practices such as Test-Driven Development (TDD), Behavior-Driven Development (BDD), and their limitations in AI contexts.
- Proficiency in designing adaptive, lifecycle-spanning evaluation frameworks that incorporate both quantitative and qualitative metrics.
- Experience with evaluation tools and frameworks (e.g., Opik,LangSmith) is a plus.
- Ability to analyze complex system-level behaviors, including reasoning pipelines, tool integrations, and emergent agent actions.
- Strong analytical skills with experience in data-driven diagnostics and root cause analysis.
- Excellent communication skills to document evaluation plans, results, and recommendations clearly.
- Experience working in cross-functional teams and managing feedback loops between evaluation and development.
- Experience collaborating with infrastructure or platform teams to improve AI tooling and automation platforms.
