The One Sentence That Changes the Output
Add the phrase “think step by step” or “explain your reasoning before giving the final answer” to a prompt, and on many tasks — math, logic puzzles, multi-step planning, debugging — the accuracy of the response jumps noticeably. This is chain-of-thought prompting, and it’s one of the few prompting techniques with real research behind it rather than just forum folklore. The surprising part isn’t that it helps; it’s why it helps.
Why Skipping Straight to the Answer Fails
Large language models generate text one piece at a time, left to right, and each new word is predicted using everything written so far — including the model’s own prior words. When you ask directly for a final answer with no reasoning, the model has to arrive at something hard — a multi-step calculation, a chain of logical deductions — essentially in one leap, with no intermediate scratch space. When you instead ask it to reason first, each reasoning step becomes additional context the model can reference while generating the next one, effectively giving it room to “work through” the problem instead of guessing at the end state in one shot.
Where It Helps Most — and Where It Doesn’t
Chain-of-thought gives the biggest lift on tasks with multiple dependent steps: arithmetic word problems, code that needs to track changing state, multi-part comparisons (“which of these three plans is cheapest after fees”), and reasoning puzzles. It does much less for simple factual lookups or short creative requests, where there’s no chain of dependent steps to walk through — asking a model to “think step by step” before naming the capital of a country just adds filler.
Three Ways to Trigger It
The lazy version is appending “think step by step” to any prompt, and it genuinely works more often than it has any right to. A stronger version gives the model a structure to fill in: “First list the relevant facts, then identify what’s being asked, then solve.” The strongest version, called few-shot chain-of-thought, shows the model one or two worked examples in the prompt itself, each with its reasoning spelled out before the answer — this anchors the style and depth of reasoning far more reliably than an instruction alone.
The Trade-Off Nobody Mentions
Reasoning tokens cost time and, on metered APIs, money — a chain-of-thought response can run two to five times longer than a direct one. For high-volume, low-stakes tasks (categorizing a support ticket, extracting a date from text) that overhead isn’t worth it. The technique earns its keep specifically on tasks where getting it right matters more than getting it fast — which is also a useful way to decide, prompt by prompt, whether to ask the model to show its work at all.
