In October 2026 I set up the open-source research agents from the Reading list on our university cluster and gave each one task, mostly from my own work, then checked every output by hand. This page is the picture version. It is a small feasibility diary, not a ranking: each tool got one task that suited it, once, on one cheap model. A failure can belong to the tool, the model, the gateway or my task.
One key, many tools
Most of these tools do not need their own account. They can all talk to one gateway (OpenRouter), which passes each request to whichever model you name. One key with a spending cap covered five tools; the whole day cost $0.49.
What each one did
Tool
Task
Setup
The work
The interpretation
Aider
Write tests from a plan
ready
tests caught 4 of 4 planted bugs
none made
Codex CLI
Review a figure helper
ready
found untested edge cases
one stated output was wrong
Paper2Agent
Query my two manuscripts
an afternoon per paper
61 of 61 checks pass
answers only from tables
PaperQA2
Question over 20 papers
agent mode crashed
every sentence cited
"at most a third" became "the majority"
Biomni
Compare 5 tandem genes
3 packages missing
table matched UniProt
"more GO terms" read as "diversified"
Virtual Lab
Plan how to test young genes
clean
sensible ranked plan
parent species called outgroups
zotero-skills
Read a public Zotero library
one minute
122 of its tests pass; read worked, write refused without a key
its status check says "connected" without connecting
CellVoyager
Reanalyse a public dataset
5 failed starts
never ran
never ran
Read the last column. In these eight small trials, the errors I found were mostly in the sentence after a correct-looking output. That is a reason to check both the table and the claim. It is not a measurement of what agents are good at: one task is not a rate, and I chose tasks I could check.
What each row did not test. PaperQA2: only its manual local-papers path, not its literature-search agent. Biomni: without its data lake, so as a code-writing database assistant, not its curated tools. Virtual Lab: the meeting, not the design-and-validation workflow in its paper. Paper2Agent: its checks confirm it reproduces my tables, not that a new question is answered correctly. CellVoyager: a setup failure through one gateway key, not a verdict on its analyses.
Who drives, and can you check it
Left to right: how much the tool decides on its own. Bottom to top: how easy its output is to check (rerun the code, look up the passage, or take its word). This is a sketch, not a measurement, and the placement is my reading. "Easy to check" means there is an affordable check for that kind of output; a notebook you can rerun can still be hard to judge. The corner to be careful in is the bottom right.
A router that picks the model for you
Jev Router (on OpenRouter) chooses a model per request. Each cost is shown against my fixed model (DeepSeek V4 Pro) on the same prompt. Two examples, not an evaluation: they show routing can save money on an easy request and cost more on a harder one. I have not measured its average cost, answer quality or stability.
Why one tool never ran
CellVoyager sends the model name down two code paths. Through a gateway, each path needs the name written differently, so one of them always fails. It expects its own OpenAI and Anthropic keys.
Notes for each tool
Codex CLI read-only review of a figure helper I use
Found four edge cases nobody had tested, such as values below 1e-10 printed as "0". One of its concrete claims was wrong when I ran the input: it said a value printed as "1e+09"; it printed "1000M". Notes
Aider write tests from a plan, $0.0016
Wrote working tests in one call. I then broke the code four ways on purpose and the tests caught all four. On the way, Python reused a stale compiled copy of the file and made a correct file look broken, a quiet failure of its own. Notes
Paper2Agent my two manuscripts as tools Claude Code can query
Its own checks still pass (61 of 61). It answers "how many genes, from which file, at which commit", and warns when an old number is still in circulation. Setup takes an afternoon per paper.
PaperQA2 a question over 20 open-access papers on young genes
Answered in three paragraphs with a citation on every sentence. Checking the citations found one overreach: "at most a third come from divergence" became "the majority are de novo", which the paper does not say. Its agent mode crashed on a version mismatch; the plain mode works. Notes
Biomni look up and compare five tandem-duplicated Arabidopsis genes
Wrote and ran its own database code; the table matched UniProt. Then it concluded one gene had "functionally diversified" because it had more GO terms, which mostly means it was studied more. Its install left out three packages. Notes
Virtual Lab four role-played experts plan how to show young genes are real, $0.01
A sensible ranked plan in six minutes, close to what my chapter actually did. It also listed the two parent species of my hybrid as outgroups, which they are not, and used stock thresholds with no reason given. This tested the meeting only. Notes
zotero-skills let an agent search and edit a Zotero library
I have no Zotero library on the cluster, so this was a read-only trial on a public group. Search worked and a write was correctly refused. Its status check reports "connected" whenever a key is set, without making a request; with a made-up key it said connected and the next call failed. The write paths, which are most of what it is for, are untested. Notes
CellVoyager reanalyse a public single-cell dataset
Did not run. Five attempts failed before any analysis: a stale package picked up from my home directory, then two parts of its code that need the same model named two different ways when everything goes through one gateway key. It expects OpenAI and Anthropic keys of its own. Notes
Not run
Gemini CLI (needs a Google login or key), Robin and Kosmos (need the paid Edison platform), AI co-scientist (not generally available), AI Scientist (machine-learning experiments on GPUs, not genomics). Why