About the Member of Technical Staff Role at Reflection

Reflection is a research lab on a mission to make intelligence open and accessible for everyone to use, customize, and build on. This Member of Technical Staff role sits on the Engineering team in San Francisco and centers on evaluation: conducting critical comparative analysis to advance understanding of model capabilities, and building the evaluation systems and processes that create tight feedback loops between data, evals, and model behavior. You will develop generalizable evaluation frameworks that capture what matters for reasoning, alignment, and usefulness, and push the boundaries of what is measurable, from synthetic evals to human feedback and real-world interaction data.

This is a highly collaborative role that works closely with pre-training, post-training, and applied teams to translate insights into concrete model improvements. Reflection is looking for someone with strong statistical analysis and experimental design skills, familiarity with LLM evaluation methodologies (static benchmarks, human preference evals, and/or agentic tasks), and high agency in a fast-paced startup environment. You should be excited to help define how a new frontier lab measures and accelerates progress toward more capable open models.

Quick Stats

* Typical process: 4 to 5 rounds, roughly 3 to 5 weeks end to end

* Format: Recruiter phone screen, then video technical and collaboration rounds, likely a final onsite in San Francisco

* Core focus: LLM evaluation methodologies, statistical analysis and experimental design, eval system building, cross-team collaboration, high agency

* Difficulty: Hard, because you must combine rigorous statistics with deep understanding of frontier model evaluation and the ability to build systems that actually change model behavior

What Reflection Looks For

* Strong statistical analysis and experimental design skills to rigorously measure model improvements

* Familiarity with LLM evaluation methodologies: static benchmarks, human preference evals, and agentic tasks

* High agency with a bias for impact over process in a fast-paced startup

* Collaborative and detail-oriented, motivated by building feedback loops that make models truly improve

Round 1: Recruiter Screen (~30 minutes)

What to Expect

A recruiter or hiring manager kicks things off with a phone or video call to understand your background, motivation, and fit for a frontier research lab. Expect a quick walkthrough of your experience with LLM evaluation, data analysis, or ML tooling, and questions about why Reflection's mission of open, accessible intelligence resonates with you. They will also cover logistics such as San Francisco location, timing, compensation expectations, and any export control or work authorization considerations.

Example Questions

* "Walk me through your background and what draws you to model evaluation work."

* "Why Reflection, and why a mission focused on open foundation models?"

* "What is your experience with LLM evals, whether static benchmarks, preference evals, or agentic tasks?"

* "Are you based in or open to relocating to San Francisco, and what are your compensation expectations?"

Tips

* Have a crisp two-minute story that connects your stats or ML background to evaluation and measurement.

* Show genuine excitement for building feedback loops in a talent-dense, early-stage lab.

* Rehearse this opener with Nora's Standard Mode so your pitch and motivation answers sound natural and concise.

Round 2: Statistics and Experimental Design (~60 minutes)

What to Expect

This technical round probes the core skill the posting names first: rigorous statistical analysis and experimental design to measure model improvements. Expect to reason about how you would design an experiment to detect whether a new model is genuinely better, how to handle noise and confounds, sample sizes, significance, and how to avoid fooling yourself with a good-looking number. You may be asked to critique a flawed eval setup or design one from scratch.

Example Questions

* "How would you design an experiment to determine whether a post-training change actually improved reasoning?"

* "A benchmark score went up two points. How do you decide if that is signal or noise?"

* "How would you control for confounds when comparing two model checkpoints on real-world interaction data?"

* "Walk me through choosing sample size and significance thresholds for a human preference eval."

Tips

* Talk out loud about assumptions, variance, and what could make a result misleading.

* Connect statistical rigor back to the practical goal: creating tight feedback loops that improve models.

* Drill this in Nora's Technical Mode, practicing experimental design and stats reasoning until you can defend your choices under follow-up.

Round 3: LLM Evaluation Deep Dive (~60 minutes)

What to Expect

Here the focus is directly on evaluation methodology and eval system building. You will discuss static benchmarks, human preference evals, and agentic tasks, their trade-offs, and how you would build a generalizable framework that captures reasoning, alignment, and usefulness. Expect questions on synthetic evals versus human feedback versus real-world interaction data, benchmark contamination, and how to make an eval that resists gaming and actually predicts model quality.

Example Questions

* "How would you build a generalizable eval framework that captures reasoning, alignment, and usefulness?"

* "When would you trust a static benchmark over human preference data, and vice versa?"

* "How do you design an agentic task eval, and how do you keep it from being gamed or contaminated?"

* "How would you combine synthetic evals, human feedback, and real interaction data into one feedback loop?"

Tips

* Show you understand the limits of every eval type and how they complement each other.

* Emphasize how your eval work would translate into concrete model improvements, not just dashboards.

* Use Nora's Technical Mode to rehearse eval design questions and to sharpen how you explain trade-offs clearly.

Round 4: Collaboration and High Agency (~45 minutes)

What to Expect

The posting stresses close collaboration with pre-training, post-training, and applied teams, plus high agency and a bias for impact over process. This behavioral round explores how you work across teams, drive projects with ambiguity, translate insights into action, and stay detail-oriented under startup speed. Expect STAR-style questions about times you owned something end to end, influenced others without authority, or shipped something imperfect but impactful.

Example Questions

* "Tell me about a time you turned an analysis or eval result into a concrete change others acted on."

* "Describe a situation where you had high ambiguity and had to define the approach yourself."

* "How have you influenced a team that owned the model or data you were evaluating?"

* "Give an example of choosing impact over process when moving fast mattered."

Tips

* Use concrete STAR stories that highlight agency, cross-team influence, and measurable outcomes.

* Show you can hold a high detail bar while still moving fast and prioritizing impact.

* Practice these with Nora's Behavioral Mode so your ownership and collaboration stories stay structured and specific.

Round 5: Final Onsite and Offer (~half day, then negotiation)

What to Expect

A final onsite in San Francisco typically combines a deeper technical or research discussion, a chat with leadership about mission and vision, and cultural fit for a talent-dense early-stage team. You may be asked to present past work or whiteboard how you would stand up evaluation from scratch. If it goes well, a compensation conversation follows covering salary, equity and stock options, benefits, and any visa or export control logistics.

Example Questions

* "If you joined next month, how would you set up evaluation for our models from the ground up?"

* "What would you measure first to accelerate progress toward more capable models?"

* "What does making intelligence open and accessible mean to you, and how does your work advance it?"

* "What are your salary and equity expectations, and what would make this the most impactful role of your career?"

Tips

* Come with a concrete 30-60-90 style plan for building evals and feedback loops early.

* Remember Reflection offers top-tier comp plus equity, so negotiate on total package, not just base.

* Run the money conversation through Nora's Salary Negotiation Mode to practice anchoring on salary and stock options without underselling yourself.

Frequently Asked Questions (FAQ)

1) How many rounds are there?

Expect roughly 4 to 5 rounds: a recruiter screen, a statistics and experimental design round, an LLM evaluation deep dive, a collaboration and high-agency behavioral round, and a final onsite that includes an offer and compensation discussion. Reflection has not published its exact process, so this reflects the typical structure for an evaluation-focused Member of Technical Staff role at a frontier lab of this stage.

2) What topics are most common?

* Statistical analysis, experimental design, and how to measure model improvements rigorously

* LLM evaluation methodologies: static benchmarks, human preference evals, agentic tasks, synthetic evals, and real-world interaction data

3) How long does the process take?

Generally about 3 to 5 weeks from recruiter screen to offer, though a fast-moving startup like Reflection can compress this considerably for a strong candidate.

4) How should I prepare?

* Sharpen your statistics and experimental design, and be ready to reason about signal versus noise in model comparisons.

* Study the trade-offs across benchmarks, human preference evals, and agentic tasks, and be able to design a generalizable eval framework.

* Prepare STAR stories that show high agency, cross-team collaboration, and turning insights into model improvements.

* Practice with Nora AI: use Technical Mode for stats and eval-design questions, Behavioral Mode for collaboration and ownership stories, Standard Mode for the recruiter screen, and Salary Negotiation Mode to lock in salary and equity.