Can You Trust a Fishing Chatbot?
For technique and explanation, a fishing chatbot is genuinely useful. For facts that are local, recent or regulatory, the measured evidence says no — language models fail most on exactly those, and they fail fluently. NIST calls it confabulation: confidently stated, erroneous content. We ship a chatbot in our own app, and this article is the case for using it warily.
Key takeaways
- On legal reference tasks, models hallucinated 58% to 88% of the time in one peer-reviewed study.
- In a medical citation study, 47% of generated references were fabricated and only 7% were fully accurate.
- Longer explanations increased user confidence even when accuracy did not improve.
- Retrieval-grounded legal tools marketed as hallucination-free still erred 17-33% of the time.
- A tutoring trial where the model was boxed in by expert-written content doubled learning gains — constraint is what worked.
The numbers, before the reassurance
Most writing about AI assistants starts with what they can do. Start instead with the measured failure rates, because they calibrate everything else.
On reference tasks about US federal court cases, large language models hallucinated between 58% and 88% of the time in a peer-reviewed study, with the hardest tasks — holdings and reasoning — reaching higher still. In a study of generated medical papers, 47% of the references were fabricated outright, 46% were authentic but inaccurate, and 7% were both authentic and accurate. A citation study across 42 topics found 55% of one model’s citations fabricated, improving to 18% for its successor — better, and nowhere near zero.
Two patterns inside those numbers matter more than the headlines. Errors rose with task complexity. And they concentrated on low-prominence material — obscure courts, less-cited cases. Models are weakest exactly where the fact is small, local and rarely written about.
Hold that thought until the section on regulations.
Why you cannot hear the error
The uncomfortable part is not that chatbots err — every source errs — but that their errors carry no audible signature.
NIST’s generative AI profile gives the failure its proper name: confabulation, defined as the production of confidently stated but erroneous or false content. The risk, as the framework puts it, arises because users believe false content due to the confident nature of the response. And it compounds: models produce confabulated logic and citations that purport to justify the answer, so the supporting argument can be as invented as the claim.
Models are also poor witnesses about themselves. The legal study found they struggle to gauge their own certainty without recalibration, and that they accepted false premises embedded in questions at rates from 27% to 99% — ask about a case that does not exist, and many models will discuss it.
Human psychology completes the trap. A study in Nature Machine Intelligence found users overestimate the accuracy of model responses given default explanations — and that longer explanations increased confidence even when the extra length did not improve accuracy. Length buys trust. It does not buy truth. Your instinct for hedging and vagueness, tuned on people, has nothing to grip here.
Regulations are the worst case, by construction
Put the pieces together and fishing regulations emerge as almost a designed stress test for language models.
A bag limit is fast-changing: seasons open and close, emergency orders amend rules mid-season with the force of law. On a benchmark built around fast-changing knowledge, all models struggled — the best stayed under 15% accuracy on such questions and declined to answer most of them. A bag limit is also local and low-prominence: one county’s rule for one water, the direct analogue of the obscure court cases where legal hallucinations clustered.
So the failure modes stack precisely where the legal stakes sit. This is why our assistant’s own feature page says regulations are not something it can be trusted with, and why every regulatory question on this site is answered the same way: your state or country’s fish and wildlife agency publishes the current rules, and that is the only source worth acting on.
The same logic covers safety. Weather on the water, tides and bar conditions belong to official forecasts and warnings — a conversation is the wrong interface for a life-safety fact.
What grounding fixes, and what it does not
The standard rebuttal is retrieval: connect the model to real documents and the hallucinations stop. That claim has been tested directly.
Researchers evaluated retrieval-grounded legal research tools — commercial systems, some marketed as eliminating hallucinations, one guaranteeing hallucination-free citations — and measured hallucination between 17% and 33% of the time. Grounding genuinely reduced errors relative to bare chatbots. The providers’ stronger claims were found overstated.
That result is the right frame for every grounded assistant, ours included. Our assistant sees your live conditions, which makes its answers more relevant; relevance is not the same property as reliability, and nothing published shows retrieval driving error to zero.
Where the evidence turns positive
After all that, the honest case for fishing chatbots is real — it just lives in a specific place.
A randomised controlled trial at Harvard compared an AI tutor against the university’s best in-class active learning. The AI-tutored students learned more than twice as much in less time, with effect sizes rarely seen in education research. The design detail is the lesson: instructors wrote comprehensive, step-by-step content and constrained the model to it, deliberately avoiding reliance on the model’s own recall.
Explanation under constraint — that is the winning configuration. Asking how to rig a soft plastic, why fish hold behind a boulder, or what to change when fish follow but refuse: these are established-knowledge questions where a conversational tutor shines, and they map exactly to what our assistant’s feature page claims and no more.
The same division runs through this whole site. Nothing the assistant says gets published here; our articles are written from verified sources under an editorial policy a chatbot could not follow. Use the chatbot as a knowledgeable friend on the bank. Use the agency for the law. And when any assistant — ours included — answers with beautiful fluency, remember that fluency was never the part that made an answer true.
What it cannot do
- Never a source for regulations. Licences, seasons and limits are fast-changing, local, low-prominence facts — the measured worst case for language models.
- Never a safety authority. Weather, tides and bar conditions come from official forecasts, not from a conversation.
- It cannot reliably signal its own uncertainty. Models accepted false premises in questions at rates from 27% to 99%, and struggle to gauge their own certainty.
- Fluency is not evidence. There is no tonal difference between a right answer and a confabulated one — that is the definition of the failure.
- Grounding reduces errors but does not eliminate them, whatever the marketing says. This applies to our assistant too.
Frequently asked questions
How often are chatbots actually wrong?
It depends heavily on the task, and the measured numbers deserve respect. On reference tasks about US federal court cases, models hallucinated between 58% and 88% of the time. In a study of generated medical papers, 47% of references were fabricated and 46% were real but inaccurate — 7% were fully correct. Technique questions with stable answers fare far better. The pattern, not a single number, is the finding.
Why do wrong answers sound so convincing?
Because generation and correctness are separate processes. NIST's generative AI profile defines confabulation as the production of confidently stated but erroneous or false content, and warns that users believe it precisely because of the confident nature of the response. Worse, models produce confabulated logic and citations that appear to justify the answer — the explanation itself can be invented.
Can I ask a chatbot about fishing regulations?
Ask if you like, but never act on the answer. Regulations are the measured worst case: models struggle most on fast-changing knowledge — the best model in one benchmark stayed under 15% on such questions — and errors concentrate on low-prominence facts, which is what a county bag limit is. Your fish and wildlife agency publishes the current rules. That applies fully to the assistant in our own app.
Doesn't giving the model live data fix this?
It helps and does not cure. A study of retrieval-grounded legal research tools — some marketed as hallucination-free, one with a guarantee — measured hallucination between 17% and 33% of the time. Grounding shrinks the problem. Nothing published shows it eliminating the problem, and claims that it does have specifically been tested and found overstated.
So what is a fishing chatbot actually good for?
Explanation and technique, where the evidence is genuinely positive. A randomised tutoring trial found students learning with an AI tutor gained more than double a comparison lecture group — but the tutor was heavily constrained, with experts writing the underlying content rather than trusting the model's recall. That is the honest recipe: a chatbot explaining established knowledge is strong; a chatbot as a live fact database is where the failures cluster.
Related reading
Sources
- Large Legal Fictions: hallucinations in large language models — Dahl et al., Journal of Legal Analysis 16(1). Accessed August 8, 2026.
- Fabrication and errors in bibliographic citations generated by ChatGPT — Walters & Wilder, Scientific Reports 13:14045. Accessed August 8, 2026.
- Accuracy of references generated for medical papers — Bhattacharyya et al., Cureus (PMC10277170). Accessed August 8, 2026.
- What large language models know and what people think they know — Steyvers et al., Nature Machine Intelligence. Accessed August 8, 2026.
- Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1) — National Institute of Standards and Technology. Accessed August 8, 2026.
- FreshLLMs: refreshing large language models (preprint) — Vu et al., arXiv:2310.03214. Accessed August 8, 2026.
- Hallucination-Free? Assessing the reliability of leading AI legal research tools (preprint) — Magesh et al., arXiv:2405.20362. Accessed August 8, 2026.
- AI tutoring outperforms in-class active learning: a randomised controlled trial — Kestin et al., Scientific Reports 15:17458. Accessed August 8, 2026.
How we choose sources: sources policy.
More in this series
See the conditions for your own spot.
Fishing Club AI turns live weather, pressure and moon data into an hourly forecast — and shows you which factors moved the number.
Get the App
