Chen Hsieh, October 2026. Plant genomics PhD student, University of Georgia. Code and files are on GitHub.
Coding agents now do much of the typing in my research. They are good at fixing errors that make noise: a crash, a missing module, a syntax error. In my long sessions, the errors left behind were the ones that look like success. A control with no values was reported as a pass. A loop meant to check fourteen repositories ran once and reported a clean result. When I tested this on small fresh tasks, the agent caught those too. What it could not catch was a choice nobody had written down. Several people asked how I use them, so this page lists what I do, what failed, and what I changed after each failure. All dates are 2026.
Read the eight rules · Copy the prompts and files · Try it on your project with me (20 minutes)
The numbered form is borrowed from rewrites.bio (Seqera), principles for rewriting bioinformatics tools with AI. Its line "Fast code is now cheap. Scientific insight, validation, and trust are not" is the same idea from the software side.
Claude Code is a command-line program. It runs a language model that can read and edit files and run shell commands in the folder where you start it. Each command it wants to run can require your approval.
I run it on the university cluster (Slurm scheduler) inside tmux, one session per project, usually several at once on the strongest available model. I answer most prompts from my phone through Remote Control. From 2 September to 2 October I ran about 75 sessions on the cluster and about 10 on my laptop. I used Codex in 3 sessions.
That volume matters. Part of what I get done comes from running the strongest model in many sessions at once, not from the method. The failures below happened at that volume anyway.
What this page is: one person's experience on one cluster over a few months, with the numbers I could check against files. It is not a measurement of how well agents work in general. For that, see the benchmarks on the Reading page.
Status summaries of a project from its files and git history. End-to-end analyses, including writing, submitting and checking cluster jobs. Answering questions from a talk with data that already exists. Redrawing figures to a written set of rules. Summaries of a lab's published methods, with each claim quoted from the paper. Rebuilding my results from the Methods text in an empty folder. Weekly updates for my advisor.
These are patterns, not bad luck. Each of them happened to me at least once.
all() of an empty list is true, so "every control value is below the cutoff" passes when there are no control values. Try all(logical(0)) in R or all([]) in Python.except: pass in Python, tryCatch(..., error = function(e) NULL) in R, || true in shell). In one study of 48 catastrophic failures in five distributed data systems, a quarter came from an error handler that ignored the error (Yuan et al. 2014).My Methods named the search tool but not its sensitivity setting. Pick the setting a careful reader might use, and see how many of the 13,663 genes in my result the rebuild fails to find.
Look back at the failures above. Only one of them crashed. A control returned nothing and was read as a pass. A job exited cleanly with an empty file. A website passed every check because every check compared it with itself. And the fourteen-repository check from the top of the page ran once. Each had a check, and each check could not have said no.
That pattern is partly an artifact of what I wrote down. Loud failures (a crash, a missing module, a syntax error) get fixed inside the session in a minute and never reach my notes. But that is also the point. The agent's loop of run, read the error, fix, run again removes loud errors very well. In my notes, what was left for me to find was mostly the quiet ones. Whether that holds beyond my notes is what the next section tests.
It can be worse than a filter. A bioinformatics benchmark from 2026 (FlowBench) found agents often "fix" a failing run so that it exits cleanly while the data stay invalid. When success is easy to see and validity is hard to see, an agent may aim at the part it can see. In METR's benchmark tasks this has included "attempting (often successfully) to get a higher score by modifying the tests or scoring code" (METR 2025). Those were scored software tasks, not research, but the lesson carries: treat a clean exit or a green test as a claim to look into. Keep the check, its expected values and the raw inputs outside the folder the agent works in, and run it after the agent has stopped. A git hook only helps if the agent cannot skip whatever runs it.
Bench biology already has the tool for this. Every assay gets a negative control that should stay dark and a positive control that should light up. If either does the wrong thing, the run tells you nothing, whatever the other wells say. The shuffled labels and the planted signal in the table above are those two controls. Software testing has the same idea under another name: mutation testing breaks the code on purpose and checks whether the tests notice (DeMillo, Lipton and Sayward 1978). Feynman put it in 1974: "The first principle is that you must not fool yourself, and you are the easiest person to fool" (Cargo Cult Science).
Guess first: which of these checks will never say no, whatever the input? Then try each input.
| Check | Result | Has it ever said no? |
|---|---|---|
| The agent's summary says it worked | ||
| The job exited with code 0 | ||
| The output has rows | ||
| The expected effect is detected |
What is new is the volume. An agent writes checks faster than anyone writes positive controls for them. So the habit I would pass on is small: before trusting a check, make it fail once. Feed it the empty file, the shuffled labels, the case built to be wrong, and watch it say no. A check you have never seen say no is a description, not a test. One forced failure proves one narrow thing: this check catches this fault. So keep a few bad cases that the agent never saw and run them at the end. Then give the check a case you know is right and watch it pass: a check that says no to good data gets switched off.
The claim above comes from my notes, so I tried to measure it. I built 15 small analysis folders on made-up gene expression data. Six have a loud fault (the script crashes: a bad import, a wrong column name, a syntax error and so on). Six have a quiet fault (the script finishes and prints a confident, wrong result: an empty control group, labels matched by position, technical replicates counted as plants, a stale cached input, a filter that differs from the plan, a flipped sign). Three are clean. One fixed agent (Claude Code with Sonnet 5.5, none of my rules or memory loaded) ran each folder with the brief "run the analysis and report the result", and a checker it could not see scored the answer. I ran it three ways, twice each: with the written analysis plan in the folder, with the plan plus six checking rules, and with the plan deleted. That is 90 runs for $7.78.
| Folders | Plan in the folder | Plan plus six rules | Plan deleted |
|---|---|---|---|
| Loud fault | 12 of 12 right | 12 of 12 right | 12 of 12 right |
| Quiet fault | 12 of 12 right | 12 of 12 right | 11 of 12 right |
| Clean | 6 of 6 right | 6 of 6 right | 6 of 6 right |
I expected the quiet row to be much worse than the loud row. It was not. The agent found every quiet fault and said what it was: "the control group was empty and all p-values were NaN (script falsely reported 0 significant)". The six rules changed nothing, because there was nothing left for them to catch. It raised no false alarm on the clean folders.
The one miss is the useful part. With the plan deleted, the folder whose filter differed from the plan had no record anywhere of what the filter should have been. One of the two runs kept the wrong filter and reported the result as fine. No amount of checking can recover a decision that was never written down.
So what I can support is narrower than what I believed:
1. On a small, fresh task, a current agent asked to run an analysis also audits it, and catches faults that contradict something it can read: the data, the code, a written plan.
2. It cannot catch a choice that exists only in your head. That is the silent guess again, and it is the argument for writing the brief before the run.
3. This says nothing yet about where my real failures happened: hours into a session, in code the agent itself had written, with no written plan, when nobody had asked for an audit. A fresh agent reading the files did well here, which is some evidence for the fresh review at the end of a long run.
Limits: one model, one kind of task, made-up data, six faults of each kind, two runs each. The quiet faults were easier to spot than real ones, because each contradicted something written in the folder. The loud and quiet faults were not matched for difficulty. Cards, checker and every result are in the repository, along with a critique of the design that Codex wrote without seeing the results.
Claude Code reads a file called CLAUDE.md when it starts. (Codex reads AGENTS.md.) Mine lists where data lives, what must not be deleted, how to submit jobs, figure requirements, and past mistakes. I add a line after each mistake.
On 4 September three defects shipped in one project that the file already prohibited:
1. A control was given about three times as much change as the condition it was meant to control. Fixing it moved the main result from 0.21–0.39 to 0.60–0.78.
2. A "no effect" result came from a condition in which almost nothing had actually changed.
3. A correlation across 48 items in 6 groups: +0.45 when pooled (p = 0.001), −0.11 within the groups (p = 0.47).
The rules were in the file and were not applied, because nothing at the moment of the analysis pointed to them. I moved the important ones into code that stops the work:
A script raises an error when a control and its condition are matched on different measures, and refuses to report "no effect" without checking there was enough data to see one.
A git hook blocks commits, branch switches and stashes in a shared checkout.
A figure-saving function refuses a figure with no sample size, text under 20 pt, or overlapping labels.
Code only helps if something runs it. A cleanup round on my cluster found five correct control functions that no script anywhere called; "controls enforced in code" described a library nothing imported. A check that is never called is BLOCKED, not PASS. So the last step of writing a check is to find the line that calls it.
In 1983 Lisanne Bainbridge described the "ironies of automation" (Automatica; searchable PDF): the more of a process is automated, the more the person's remaining job is the abnormal cases, noticing and recovering from what the automation got wrong, while they get less practice at the normal ones. Agents fit that description. They take the routine runs; you are left with the quiet failures. She also saw why those are hard to see: "automatic control can 'camouflage' system failure by controlling against the variable changes, so that trends do not become apparent until they are beyond control." An agent that patches a run until it exits cleanly is doing the same thing. Her design rule was that "automatic systems should fail obviously", which is what a check you have seen fail gives you.
Philosophy of science has words for what a good check is. The Stanford Encyclopedia summarizes Popper: "Every genuine test of a scientific theory, then, is logically an attempt to refute or to falsify it" (SEP). Deborah Mayo's version is the severe test: only infer what the procedure had a real chance of showing to be wrong (Mayo and Spanos 2006). An exit code is not a severe test of a biological result. A planted signal that the pipeline must find is.
Gelman and Loken's "garden of forking paths" (PDF) is the other risk. Choices made after seeing the data (which filter, which threshold, which samples to drop) multiply the ways to reach a clean result, without anyone fishing on purpose. An agent can walk many of those paths in an hour and report the one that worked. The fix is old: write the decision down before seeing the result, and keep a log of what was tried and dropped. There is a place for it in the brief, written before the run, and in the status file.
Merton listed "organized skepticism" among the norms of science (National Academies summary): claims are checked by someone other than their author. A fresh session that reads the files instead of the summary is a small, cheap version of that. It catches work that was claimed and not done. It is a second pass, not an independent check: it shares the brief, the repository and usually the model. It works better if it reads the inputs and outputs before it sees the first session's summary. A labmate who knows the assay is the stronger check.
Parasuraman and Riley's 1997 paper "Humans and automation: use, misuse, disuse, abuse" gives four ways to get the balance wrong or right. The agent examples are mine:
| Use | Handing over routine runs that end in a check you have seen fail. |
|---|---|
| Misuse | Treating the exit code or the agent's own summary as the result. |
| Disuse | Refusing it after one bad experience, even for routine runs a tested check could cover. Or switching off a check that keeps saying no to good data. Learning a method by hand is not disuse. |
| Abuse | Setting it up without regard for the people involved: unbounded jobs on a shared cluster, more output or more sessions than anyone can review, or no way to see and stop what it is doing. |
Every project I hand to an agent lives in a GitHub repository, and most of the habits on this page run through it. If you take one practical thing from this page, take this one.
The history is the notebook. I ask for a commit after every real step, with a message that says what changed and why. A session's summary is its own account of what it did; the commits are what actually happened. When a session ends, a fresh one reads the log and the diffs, not the chat.
It moves work between machines. My laptop and the cluster do not share a disk. Code, small result tables and figures go through GitHub; jobs on the cluster pull the commit they should run. A number in a report carries the commit it came from, and I only quote a commit after the push has succeeded.
It keeps agents apart. Each session works in its own copy (a git worktree, see two agents, one folder). If one breaks something, the damage stays on its branch.
It is how I show work. A private repository can publish a public website from its docs/ folder (GitHub Pages; free for public repositories, a paid account for private ones). I turn that on with the first rough version, so my advisor or a friend can open a link on a phone instead of pulling a branch. This site is one of those. The published site is public even when the repository is not, so nothing goes in docs/ that is not cleared for the open web.
What stays out. Raw data, large files and anything secret. Big outputs stay on the cluster's storage and the repository holds a pointer to them: a path, a size and a checksum.
When a conversation gets long, older parts are summarized and details are lost. I keep a brief, a task list and a status file on disk, and commit after each step. At the end, a new session reads the files and commits and reports what was done. I use that report, not the original session's own summary.
Waiting is done by the scheduler (job dependencies) or by one background check that ends when the job leaves the queue. Not by an agent polling. I also give every wait a deadline. A job that hangs stays in the queue until its time limit, which can be days away, and never writes its file. So my jobs print a progress line every few minutes, and the background check counts a log that has stopped growing as a failure.
Much of my volume comes from running several sessions at the same time. Three different things go under that name, and they fail in different ways.
Separate sessions on separate work. This is most of it: one session on an analysis, one on a figure, one on a site. They go wrong by colliding, as in two agents, one folder. Each gets its own copy of the repository, and all of them read the same playbook at the start.
One task spread over several agents. Worth it when the parts are independent: a literature search from five angles, a draft read by reviewers with different briefs, one result checked several ways. Not worth it when the parts depend on each other. Cognition's example is two subagents building the background and the bird of one Flappy Bird clone: each misreads its part in its own way, and the pieces do not fit together (Cognition 2025). It is also expensive. Anthropic reports that its multi-agent research system used about 15 times the tokens of a chat, and that most coding tasks have fewer truly parallel parts than research (Anthropic 2025). I ran out of weekly usage halfway through one such run and finished it by hand.
Reviewers that agree are not independent if they are the same model reading the same thing. Give them different inputs, a different model, or the job of arguing against the result.
Handing a step to another tool. I give mechanical steps that have a test to a cheaper model (Codex, or Aider with a model through OpenRouter). The failure here is quiet: a job that looks like it is running and is not. One review job sat for 40 minutes waiting for keyboard input that would never come. Every handed-off job now gets an output file to wait on and a deadline, and I judge it by the file. Numbers it reports get checked before I repeat them: one reported a folder as 38 GB when the quota counted 28.
More agents produce more. They check more only if the checking was planned.
The tools let you choose how often the agent stops to ask. Claude Code has modes from "ask before every command" to "plan first, then edit freely" to "never ask"; Codex, Cursor and others have similar settings. Which one is right depends less on the tool than on what the task can break.
On a shared cluster, the computing centres that have written guidance agree on one point: keep approval on for anything that submits jobs, cancels them, or deletes files. NCAR's page names the failure directly: a "runaway qsub loop", a recursive delete, or a job that uses a large part of your allocation (NCAR; Aalto says much the same).
Whatever the mode, I keep a short list of points where it must stop and ask, written in the rules file so they hold in every session:
1. Before deleting or overwriting anything it did not create.
2. Before submitting more than a set number of jobs, or anything on shared hardware it has not used before.
3. Before writing outside the project folder: a shared drive, a push to a live site, a message to another person.
4. When the plan changes: a different method, a dropped sample, a moved threshold.
5. Before any number goes into a manuscript, a slide or a message to my advisor.
The first three protect other people and data. The last two protect the result: they are the points where a choice made after seeing the data would otherwise slip in.
Point 3 is the one that is easy to skip, because nothing breaks. A soil hydrologist described in Nature how she kept replying "please continue" to a chatbot that had her e-mail account connected. She meant: keep searching and analysing. It e-mailed researchers she had never contacted, in her name, with a summary of her unpublished plan attached, and told her afterwards. The replies were helpful, and she still calls the messages unauthorized. Her two sentences are the rule: "capability is not permission", and "transparency after the fact is not a substitute for authorization beforehand" (Zhao 2026). So "continue" means more of the same kind of work, and every account I connect gets its own line saying what may be done with it without asking.
An overnight run goes well when you could write the morning check before you leave: which files, how many rows, what range of values. I write it into a brief. It goes badly when the task needs your judgement halfway through, because the agent will either stop and wait or decide without you.
It is tomorrow morning. This run produced a result that looks convincing but is wrong. List five ways that could have happened. For each, add the smallest check that would catch it, and run that check on a known-bad input now.
The premortem is Gary Klein's: imagine the project has already failed, then ask why (HBR, 2007).
Goes well A one-sentence goal. A check it can run that has been seen to fail. One sample or one chromosome taken through every stage and checked before the full set is launched (release engineers call this a canary). Small stages that each end in a file and a commit. Jobs chained in the scheduler. Its own copy of the repository. Commands it may run decided in advance, so it does not stop at 1 a.m. waiting for a yes.
Goes badly "Explore the data." Success judged by its own summary. One long job that saves only at the end. A shared folder. An agent polling the queue all night. No limit on how many jobs it can submit.
Benchmarks do not tell you how long you can leave it. METR measures the length of task (in human time) an agent completes half the time; for GPT-5 it was about 2 hours 17 minutes. METR's own explanation says this is not how long an agent can run on its own, and that for tasks of 90 minutes to 3 hours, about a third succeed every time, a third fail every time, and a third vary (METR). Anthropic reports its own engineers' sessions went from about 10 to about 20 actions before a person stepped in, between February and August 2025 (Anthropic). Plan a night as several short, checked stages, not one long task.
Do not let watching it stand in for a planned check. I know of no study of watching an agent, but there is one on how fast AI help feels. In a 2025 randomized trial, experienced open-source developers expected AI tools to make them 24% faster; on 246 real tasks they were 19% slower (METR). That was early-2025 tools on mature codebases, and METR has since changed the study design and reads its newer numbers as a lower bound on the speedup (METR 2026), but the gap between how fast it felt and how fast it was is the useful part.
What I do instead: give the session a condition to wake me on (the job left the queue, a file appeared) rather than checking in. Work on something only a person can do: reading, writing, a meeting, the bench. I open my phone when a session asks for something, and otherwise leave it alone. Review in one batch, from the files and commits, with a fresh session that did not do the work. Then open one output file and check one number against something already known. A fresh session's report is still something you read. In one automation study, people lost track of the situation when the system made the decisions, and the authors traced it to "the shift from active to passive processing" (Endsley and Kiris 1995).
In research, being wrong is expensive almost everywhere, so I use the strongest model for design, interpretation and review. Cheaper models are fine for bulk edits if the result is checked. One setup I use: the strong model writes a plan of small numbered steps, each with a test command; a cheaper model, through Aider, does each step; only the pass or fail and the end of any error come back. The judgement stays with the strong model and the typing goes to the cheap one. Tools can switch to another model when one is overloaded; the session status shows which model answered.
Local models: on a base M4 Mac, a 7-billion-parameter model at 4-bit generates about 24 tokens per second (my measurement). That works for chat and small edits. I have not run a multi-hour project session on a local model. If data cannot leave your institution, options are a hosted model that sees only code and summaries, or an open model on institutional GPUs.
1. Install Claude Code (it needs a paid Claude plan or API credits; the free plan does not include it). Open it in a folder whose contents you already know.
2. Ask what each file does and in what order they run. Compare with what you know.
3. Keep command approval on.
4. Copy the five-line rules file into CLAUDE.md and fill in the brackets. Add a line after each mistake.
5. Make one check fail. Copy an input with only its header line (head -n 1 input.tsv > empty.tsv), run your first step on it, and see whether anything says no. If nothing does, write the check that would, and watch it say no.
Written with Claude Code from my own session logs and notes; every number was checked against the file it came from. Mistakes are mine.
Prompts & files: a rules file, a brief, a status file, prompts I reuse. Reading: papers, tools, and guides. A 20-minute session with me, if you want to go through it on your own project.
setup/AGENT_PLAYBOOK.md, the rules my agents read at the start of a task and again before reporting, with the incident in brackets where there was one.
setup/CLAUDE.md, a longer general rules file. For a single project, the five-line version on the Prompts & files page is enough.
skills/, procedures for trying to break a result.