We Tried the Prompting Techniques. Most Did Nothing.
A survey that June counted well over fifty distinct prompting techniques in the literature. We took the eleven that plausibly applied to our work and ran them against the same labelled set. The results were duller than the catalogue suggests, and the dullness is the useful part.
The Catalogue and the Question
Schulhoff and colleagues published a systematic survey of prompt engineering that June, assembling terminology and a taxonomy across a large body of papers and cataloguing dozens of distinct named techniques for text-based prompting.
Reading it raised an obvious and slightly uncomfortable question. We use perhaps four of these deliberately. Are we leaving quality on the table, or is most of this catalogue about tasks that do not resemble ours? The only way to find out was to run them.
How We Ran It
Two of our labelled sets: the eighty-case extraction set and a sixty-case question answering set. Eleven techniques from the taxonomy that plausibly applied, each implemented once, each run against both sets with everything else held fixed.
Three runs per configuration, because a single run on sixty cases has enough variance to invent a result. That detail matters more than the choice of techniques: without repeats we would have reported two of the eight null results as improvements.
What Helped
Two things. Giving the model a small number of worked examples chosen to resemble the input, rather than fixed examples, improved extraction accuracy consistently and by a margin large enough to survive the repeats.
The second was asking for reasoning before the answer on the question answering set only. On extraction it did nothing, which fits: extraction is a lookup with formatting, and there is no intermediate reasoning to expose. A technique helping on one task and not the other was the most common pattern we saw.
| Technique | Effect on our sets |
|---|---|
| Similar worked examples | Clear gain on extraction |
| Reasoning before answering | Gain on question answering, none on extraction |
| Assigning the model a persona | Measurably worse on extraction |
| Self-critique and revision | No measurable change, double the cost |
| Emotional or urgency framing | No measurable change |
| Restating the question first | No measurable change |
The One That Hurt
Telling the model it is an experienced logistics clerk made extraction worse. Not dramatically, but consistently across repeats and on both document types. Our reading is that a persona pushes towards plausible domain-shaped output, and plausible is precisely the wrong pressure when the correct behaviour is to leave a missing field empty.
We include this because persona framing is among the most widely repeated pieces of prompting advice, and it was the only technique in our run that moved a number in the wrong direction. That is one measurement on our tasks, not a refutation, but it is enough that we stopped doing it.
What the Null Results Are Worth
Eight of eleven produced no measurable change. That is a real result and we treat it as one: it means those techniques are not where our remaining quality is, and time spent tuning them is time not spent on retrieval or on the schema, which is where our measurable gains have come from.
It also means we can decline them in conversations without a shrug. When someone asks why we are not using a technique, the answer is that we ran it on our set, it did not move, and here is the number, which is a much better answer than a preference.
Why a Catalogue Is Still Useful
A survey like this is a list of things to test, not a list of things to adopt. Its value is that it names techniques precisely enough for two engineers to mean the same thing, and that it makes coverage visible: we could see which parts of the space we had never tried.
The two we adopted came from exactly that. Neither was new to us as an idea, and neither would have been prioritised without a taxonomy showing that we had been sampling one corner of the space and calling it prompt engineering.
What We Would Do Differently
Run it earlier and on smaller sets. The whole exercise took about three days including implementation, and we had been arguing about several of these techniques informally for far longer than that. The measurement was cheaper than the discussion it replaced.
We would also fix the prompt before optimising anything else next time. Two of our null results are suspect because the baseline prompt had accumulated its own instructions, and a technique cannot demonstrate an effect that the baseline already covers by other means.
What We Do Not Claim
We do not claim these techniques do not work. We claim they did not move our two tasks, which are narrow, schema-bound and retrieval-backed. A survey covering a wide range of tasks reports gains that our sets are not shaped to show, and that is a limitation of our measurement rather than of the techniques.
We also do not claim eleven is thorough. We selected the ones that plausibly applied, which is a judgement that could easily have skipped the one that would have helped most.