Case study
About Elicit
Elicit helps researchers be 10x more evidence-based, using AI to find, analyze, and synthesize scientific research. Backed by Spark Capital and Footwork and used by more than 400,000 researchers every month, Elicit helps organizations make rigorous, evidence-based decisions.
Expanding the scope of autonomous development
Elicit set out to build their own software factory, with the goal of expanding the scope of what agents could autonomously handle. That meant maximizing one-shot issue resolution, so agents could go deeper into their backlog of features, bugs, maintenance, security audits, and UI improvements with less human intervention.
What followed was a rigorous, multi-week evaluation process. Elicit was already using Claude Code locally with git worktrees for parallel work and had tried most major coding agents. They needed a solution that maintained their control and flexibility, letting them plug in existing skills, triage rules, implementation standards, and dev env definition while freely switching between Claude Code and Codex. “We wanted to own the software factory, not be locked into one agent,” says Andreas Stuhlmüller, Co-founder & CEO of Elicit. “We tested more established platforms where the agent struggled to follow directions, but we couldn’t switch it. That was a nonstarter. Niteshift is clearly the better product.”
After evaluating several platforms, they chose Niteshift.
“Niteshift produced the best results on our evaluations, both because of the agents available and the tooling around them.”
How Elicit’s software factory works
Niteshift became an essential component of Elicit’s multi-step development pipeline. Elicit owned the software factory logic, while Niteshift provided the infrastructure for the implementation and validation steps.
FIG · Elicit’s software factory pipeline from trigger to mergeable PR
Elicit’s software factory, The Line, uses an in-house orchestration agent to manage the handoffs between each step. When a Linear issue or Aikido security finding is created, a triage agent determines if it’s agent-workable based on scope, security risk, and reproducibility. Next, a planning agent creates an implementation brief, including root cause analysis for bugs. An agent in Niteshift picks up the implementation brief, makes the smallest change that addresses the issue, and validates its changes against their standardized development environment in the cloud. It then creates a mergeable PR with screenshots and video evidence that the feature works or the bug is fixed. Human review is still required to merge, but the evidence makes review much quicker.
After every deploy, The Line orchestrates Niteshift to run a Playwright suite of E2E smoke tests against the deployed build and investigates failures against changes since the last verified deploy. Rollbacks can be manually triggered in Slack, after which The Line handles PR creation and merging autonomously.
Standardizing the development environment
Niteshift enabled Elicit to run agents against a standardized, multi-service development environment, including their Next.js frontend, TypeScript API, Redis and MongoDB, Python services running in Docker, and AWS SQS emulated through LocalStack.
FIG · What runs inside Elicit’s Niteshift environment
Elicit relies heavily on GROBID, an open source Java app that converts documents like academic papers into structured data for their research agent, but its memory and disk requirements made it impractical to run in every sandbox. They worked directly with the Niteshift team to customize process management per sandbox, spinning up a container with a GROBID image on demand for relevant tasks, then shutting it down once documents had been parsed and the structured data was extracted.
FIG · Getting GROBID running on demand in Elicit’s sandboxes
Scaling to hundreds of autonomous PRs in 2 months
Elicit integrated Niteshift into their software factory in June, and by August, The Line had shipped hundreds of PRs with a 79% merge rate, frequently running multiple sessions in parallel and peaking at 14 concurrent sessions across Claude Code and Codex. 32% of merged PRs are “one-shot” (not requiring human re-review).
FIG · Two months of The Line: 205 PRs merged, 79% merge rate, 32% one-shot
Work autonomously shipped with Niteshift
Over the last 2 months, The Line’s merged PRs have spanned bug fixes, new features, code cleanup, UI improvements, security work, and more.
Fixing a PDF export timeout by finding its breaking point
A user exporting a literature review report as a PDF timed out after 16 retries over 30 hours, burning 8 minutes of compute per attempt. An Elicit engineer flagged the failure in Slack, citing errors from Datadog. Elicit’s investigation agent then traced the issue to a massive report title (>55k characters) being inserted directly into LaTeX, disproving the original hypothesis that the table contents were responsible by removing only the title and confirming the report rendered in about 4 seconds. Niteshift reran the rendering pipeline at various title lengths to determine where timeouts started, then capped titles at 2k characters, tracked failure counts, and added a failsafe to stop retrying after 3 attempts. It then added unit and integration tests, completed a smoke test to verify the blast radius was contained, and opened a mergeable PR with video evidence.
Launching a usage insights dashboard for customers
The Line shipped a reporting dashboard for Elicit org admins showing key usage stats like seats, workflows, papers processed, active users, and feature usage. This required querying multiple sources of usage data, enforcing admin-only access and org-level data isolation, and adapting reporting windows to each org’s billing period. Since org data was sparse locally, the agent created a test org and seeded sessions, notebooks, workflows, and paper activity to validate the API and dashboard end-to-end. During human review, it caught and fixed inconsistencies in how active members were scoped across metrics, adding regression tests with each fix, then smoke-tested adjacent product areas to verify the blast radius was contained before shipping through a few rounds of human review.
What’s next
Today, approximately 1 in 5 PRs ship autonomously through The Line with Niteshift. As agents improve, Elicit plans to increase this percentage by taking on work across multiple repositories and expanding their triage criteria to cover larger, more complex tasks, without having to rebuild the rest of the pipeline.
Ready to build your own software factory on Niteshift?