How Listen Labs ran 1,000+ autonomous eval runs on Niteshift

About Listen Labs

Listen Labs helps companies like Anthropic, Microsoft, and Sweetgreen understand what their customers want and why. Its AI market researchers recruit participants, conduct in-depth interviews, and turn conversations into actionable insights in hours instead of weeks.

Experimenting 24/7 in the cloud

Experiments

50

Eval runs

1,000+

Improvement in quote extraction

8.6%

Reduction in average survey response prediction error

8.5%

Listen Labs has run 50 experiments and more than 1,000 eval runs on Niteshift, lifting quote extraction accuracy from 89.2% to 96.9% along the way. Here’s how.

Why local evals broke down

Inspired by autoresearch, Listen Labs implemented autonomous eval loops to optimize its research platform’s performance and quality. Locally, these experiments broke down, as they demanded:

1. Heavy, long-running evals: Large eval sets required significant memory and compute, with experiments running for hours or even days.
2. Complex orchestration: Agents needed to track code versions, coordinate experiments, and share results with one another.
3. Isolated, realistic environments: Parallel experiments each needed their own development environment and dedicated datasets to avoid contamination.

Across hundreds of eval cases, with rate limits on top, one run takes about two hours and 32GB of RAM to parallelize properly. Not something you run locally, and most cloud environments choke on it too.

Niteshift is built for this workload. Every eval task gets its own cloud sandbox with the memory and compute to run for hours, and if something fails mid-run, Niteshift recovers it automatically.

Autoresearch on Niteshift

With autoresearch (a term coined by Andrej Karpathy), users define a high-level goal for agents to loop until they achieve it. For quantitative objectives like reducing query latency, setting a goal is straightforward. For qualitative objectives, there’s more of an art to defining success, and Listen Labs developed thoughtful techniques to evaluate large features like quote extraction and report quality.

Here’s how autoresearch runs on Niteshift. The agent drives the loop, and Niteshift gives every eval task an isolated cloud sandbox with the necessary service dependencies, benchmark datasets, test cases, and constraints to establish a baseline. The agent proposes improvements, runs the changes against the development environment for realistic results, and compares them to earlier runs, iterating until it finds a credible improvement. Agents typically run 3–6 eval variants concurrently within a task’s sandbox, and can fan out across more sandboxes as needed, up to 8 concurrent sessions.

As Listen Labs’ engineering blog puts it, sometimes a local dev server and a coding agent are enough, and sometimes the job needs a dedicated cloud environment with a full copy of the data and elaborate eval scripts. Autoresearch was the second kind. Niteshift gave Listen Labs an isolated cloud environment for each eval task, and with it, the freedom to experiment 24/7 across product workflows.

Florian Juengermann, co-founder of Listen Labs, says that “engineers point their agent at an eval script and let it run for hours optimizing the whole codebase on Niteshift.”

A cloud environment designed for reliability

Heavy eval suites put significant pressure on cloud infrastructure, requiring high memory, sustained compute, isolation from competing workloads, and reliable sandbox lifecycle management. Achieving reliability becomes increasingly challenging for long-running evals, as even specialized sandbox providers struggle to suspend and resume environments consistently. “We trust Niteshift to run our evals autonomously for hours at a time,” says Florian Juengermann, co-founder of Listen Labs, “because if something goes wrong, we know Niteshift will automatically recover the sandbox.”

Set up automatically

Niteshift’s onboarding agent automatically set up Listen Labs’ full stack, including its Next.js frontend, backend services, Temporal workflows, and local Supabase instance. Coding agents sped up previously tedious work like porting eval scripts and establishing secure access to large, realistic datasets needed for testing.

Evals run on Niteshift

Optimizing emotion detection

Listen Labs’ Emotional Intelligence uses AI to detect emotions in video calls through a combination of micro-expressions, tone of voice, and word choice. When Gemini 3.6 Flash was released, the team found it initially scored 12.6% lower than Gemini 3 Flash on its standard benchmark. With an agent on Niteshift optimizing the eval setup and prompts for Gemini 3.6 Flash across several rounds of experiments, the best version scored 0.455, beating the production baseline of 0.448.

Increasing quote extraction accuracy

Listen Labs customers can search transcripts for participant quotes using keywords or broader topic matches. In some cases, the extracted quotes were too short to be meaningful, returning fragments like “I mean, I’m excited” without enough surrounding context. Running on Niteshift, the agent built an eval suite with 65 test cases from reviewed research data. The winning approach improved the pass rate from 89.2% to 96.9% while increasing median quote length from 151 to 219 characters.

Improving synthetic study prediction accuracy

Listen Twins, Listen Labs’ AI-powered human simulation product, uses digital twins of real respondents to predict how people would answer study questions. Niteshift used a study completed by 50 people as a benchmark, generating synthetic responses based on respondent profiles and comparing against real study results. Across approximately 60 experiments and 13 versions, the agent iteratively refined the response generator, ultimately reducing average prediction error by 8.5% and raising the accuracy score from 94.57% to 95.03%.

What’s next

Listen Labs continues to push the boundaries of what coding agents can optimize, developing sophisticated approaches to evaluating research report quality using LLM-based judgment.

Listen Labs now uses Niteshift beyond evals, including general feature work. PR previews make changes easy to validate and help them ship faster.

Niteshift gives any coding agent a full-stack cloud environment with real data, parallel sandboxes and automatic recovery, so eval loops can run unattended for hours and end in a reviewable PR.

Ready to run evals on Niteshift?