AI Terminal-Bench Evaluation Specialist

# AI Specialist – Terminal-Bench & Outlier Projects

**Position:** AI Specialist / AI Evaluation Engineer
**Work Type:** Remote, Contract
**Project Focus:** Terminal-Bench, AI agent evaluation, and coding task benchmarks

## About the Role

We are looking for an experienced AI specialist with practical experience working on Outlier AI projects, particularly Terminal-Bench and terminal-based coding agent evaluations.

The ideal candidate understands how to evaluate AI agents on complex software engineering tasks in command-line environments, analyze model behavior, troubleshoot execution failures, and produce accurate, reproducible evaluation results.

This role is best suited to someone who has already worked on similar AI training, coding evaluation, or agent benchmarking projects and can demonstrate strong technical judgment.

## Responsibilities

- Perform technical evaluations of AI-generated solutions in terminal-based environments.
- Work with software development tools, command-line interfaces, Git, Linux, and automated test suites.
- Analyze AI agent performance on coding, debugging, system administration, and software engineering tasks.
- Investigate failed executions, incorrect outputs, environment configuration issues, and test failures.
- Review task instructions, expected outcomes, evaluation criteria, and test reliability.
- Produce clear, evidence-based assessments of AI-generated work.
- Maintain reproducible testing procedures and document technical findings.
- Collaborate remotely with project leads and other AI evaluation specialists.

## Required Qualifications

- Demonstrable experience with Outlier AI projects, especially Terminal-Bench or closely related coding-agent evaluation work.
- Strong command-line and terminal troubleshooting skills.
- Proficiency in Linux, Git, scripting, and software debugging.
- Experience with at least one programming language, such as Python, JavaScript, Java, or C++.
- Ability to read code, execute tests, diagnose failures, and distinguish genuine solution errors from environment issues.
- Strong analytical thinking and attention to detail.
- Excellent written English and the ability to explain technical findings clearly.
- Ability to work independently in a remote, project-based environment.

## Preferred Qualifications

- Experience evaluating AI coding agents and autonomous terminal agents.
- Familiarity with benchmark-driven evaluation, automated grading, test harnesses, and containerized environments such as Docker.
- Experience with software engineering benchmarks, reproducibility, and task verification.
- Familiarity with Python-based evaluation infrastructure and CI/CD workflows.
- Experience with other AI training or evaluation platforms.

## Application Requirements

Please include the following in your application:

1. A brief summary of your experience with Outlier AI projects.
2. Specific experience with Terminal-Bench or similar terminal-based AI evaluation benchmarks.
3. Your technical background, including programming languages and development environments.
4. Examples of relevant evaluation, debugging, or benchmark work, without disclosing confidential project information.
5. Your availability and preferred contract arrangement.

**Important:** We are prioritizing candidates with verifiable, hands-on experience rather than general familiarity with AI tools.

## How to Apply

Submit your technical background and a short description of your most relevant AI evaluation experience. Qualified candidates may be invited to complete a practical technical assessment.