· AI Talent Report Editorial · Emerging Roles · 5 min read
Evaluation Engineer: Career Transition Plan
A 90-day plan for QA and ML engineers transitioning into Evaluation Engineer roles in 2026, with concrete portfolio projects at each stage.
Who This Plan Is For
This transition plan is built for two starting populations: traditional QA/software test engineers who want to move into AI-specific evaluation work, and ML engineers who have been doing evaluation as a side responsibility and want to formalize it into a dedicated career track. Both groups have real advantages and real gaps, and the 90-day plan below is sequenced to close the gaps fastest for each.
Starting Point Assessment
| Background | Existing Strength | Primary Gap |
|---|---|---|
| QA/Software Test Engineer | Test case design discipline, regression thinking, pipeline/CI experience | Statistical testing, LLM-specific failure modes, adversarial prompting |
| ML Engineer | Statistical fluency, model internals knowledge, training/inference pipeline experience | Rubric design discipline, red-teaming technique, structured failure taxonomy |
Know which column you start from before beginning the plan. QA-background candidates should front-load statistics and adversarial prompting. ML-background candidates should front-load rubric design and human-in-the-loop grading discipline.
Days 1-30: Foundation
Spend the first month building fluency with at least one eval framework and running your first real evaluation project on a public or personal dataset, not a tutorial toy example.
- Week 1: Set up and run an existing open-source eval harness end to end (OpenAI Evals or promptfoo are the most accessible starting points). Reproduce a published benchmark result to confirm your pipeline is correctly wired.
- Week 2: Study statistical testing fundamentals as applied to model comparison: bootstrap confidence intervals, paired significance tests, and sample size / power calculations for eval sets. Apply these to the benchmark you reproduced in Week 1.
- Week 3: Build your first rubric from scratch for a non-trivial capability (e.g., “did this model correctly refuse an unsafe request without being unhelpfully over-cautious”). Test inter-rater agreement between yourself and an LLM-judge.
- Week 4: Read published red-teaming methodology from at least two frontier labs (public model cards and safety reports are the best source material). Identify the jailbreak pattern families they document and replicate two or three against an open model in a controlled, ethical test environment.
Portfolio artifact by Day 30: a written eval report on a public model, including your rubric, your statistical comparison methodology, and at least one adversarial test case with documented results.
Days 31-60: Depth and Breadth
The second month is about building the specific artifacts that hiring managers ask candidates to walk through in interviews.
- Weeks 5-6: Build a small regression-gating pipeline: take two versions of a prompt or fine-tune, run your eval set against both, and produce an automated pass/fail decision with statistical backing. Wire this into a CI-style script (GitHub Actions or a local script that mimics one) so you can speak concretely to “operationalizing” evaluation, not just running it manually.
- Weeks 7-8: Write a benchmark critique. Pick a well-known public benchmark and write a structured analysis of its construction flaws: contamination risk, narrow task coverage, grader reliability issues, or distributional mismatch with real-world use. This single artifact demonstrates the critical thinking that separates senior candidates from junior ones.
Portfolio artifact by Day 60: a GitHub repo containing your regression pipeline and your benchmark critique write-up, structured so an interviewer can review both in under fifteen minutes.
Days 61-90: Positioning and Interview Readiness
- Weeks 9-10: Reframe your resume and LinkedIn around evaluation-specific language: rubric design, statistical significance testing, red-team methodology, regression gating, failure taxonomy. Map every bullet point back to the Tier 1 and Tier 2 skills described in the companion skill-map piece in this series.
- Weeks 11-12: Run mock interviews focused on the two exercise types most commonly used in real loops: live eval design (“design a test suite for X capability in 30 minutes”) and benchmark critique discussion. Practice narrating your Day 30 and Day 60 artifacts fluently, since interviewers will ask you to walk through real work, not hypotheticals.
Portfolio artifact by Day 90: a condensed one-page case study combining your eval report, regression pipeline, and benchmark critique into a single narrative you can present in a 10-minute interview walkthrough.
Milestones Checklist
- Day 30: Working eval harness + first rubric + documented red-team test
- Day 60: Automated regression pipeline + benchmark critique write-up
- Day 90: Resume repositioned + polished case study + mock interview reps completed
Common Mistakes During Transition
The most common failure mode is spending all 90 days on tooling and none on the statistical and rubric-design thinking that interviewers actually probe. A candidate who can run promptfoo but cannot explain why a 50-example eval set is statistically underpowered will stall in technical screens. The second most common mistake is treating red-teaming as a party trick rather than building it into a repeatable, documented process, which is exactly the distinction hiring managers are testing for.
Interview Preparation Resource
Once your 90-day plan is complete, dedicated interview preparation closes the remaining gap between a strong portfolio and an offer. The 0-to-1 AI Engineer Interview Playbook (Amazon: https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20) is built for exactly this transition point, covering exercise formats, structured answer frameworks, and the kind of walkthrough narration this plan’s Day 90 artifact is designed to support.
Bottom Line
Ninety days is enough time to build a credible, interview-ready evaluation portfolio if you sequence foundation before breadth and breadth before positioning. The plan above is deliberately artifact-driven: by Day 90 you should have three concrete pieces of work an interviewer can inspect, not just a list of concepts you have read about.