Forum Posts
as Validation
Evidence

There is a scene in Arthur Conan Doyle that practitioners of any kind of structured inquiry should read carefully. Sherlock Holmes, examining a case with no clear positive evidence, turns to a Scotland Yard detective and raises "the curious incident of the dog in the night-time." The detective points out that the dog did nothing in the night-time. Holmes replies: "That was the curious incident."

The absence of evidence, read correctly, is itself evidence. The inverse is also true and more dangerous: the presence of something that looks like evidence, read carelessly, is not evidence at all. This article is about the difference — specifically in the context of forum posts as a substitute for primary research when primary research fails.

How We Got to This Question

The venture was a payment protection product for US owner-operators and independent freight carriers. The problem, briefly: carriers complete loads correctly, deliver on time, and then get stiffed. The invoice sits unpaid. Bond claims require process knowledge most carriers don't have at the moment they need it. Legal pursuit costs more than the disputed amount in manual effort alone. So carriers absorb the loss and move on. The enforcement friction trap.

Getting to these carriers through conventional outreach turned out to be harder than expected. Cold outreach to US carriers ran into the structural reality that owner-operators are a hard-to-reach audience through any channel outside their own communities — and those communities are tight-knit and skeptical of outsiders. The outreach infrastructure existed. The access was slow.

This is a situation most practitioners in early-stage validation will recognise. You have a clearly defined audience. You have a real hypothesis about their problem. And the channels you expected to use are either closing, slow, or producing compromised signal because the audience knows they're being studied. What do you do?

The temptation — and it is a genuine temptation — is to find a workaround that looks like primary research but isn't. A survey sent to a convenience sample. Conversations with people who are adjacent to the target audience but not in it. These produce data. They do not produce signal. And data that isn't signal is worse than no data, because it gives you the confidence of evidence without the substance of it.

The alternative we chose was passive evidence. Specifically: treating first-person incident posts from public owner-operator forums as a legitimate substitute for direct interviews, under strict conditions.

The Ethnography Argument

Anthropologists have a concept called naturalistic observation — studying behaviour in its natural setting, without the distortion that comes from telling subjects they're being studied. The Hawthorne Effect, documented through studies conducted between 1924 and 1932 at the Hawthorne Works, showed that workers changed their behaviour simply because they knew they were being observed. The observation itself corrupted the data.

Forum posts written by owner-operators about broker non-payment are naturalistic data. Nobody wrote that post for a researcher. Nobody was performing their pain for an audience of validators. They wrote it because they were angry, or looking for advice, or trying to warn others. That authenticity is, under the right conditions, an asset rather than a liability.

The discipline question is what "right conditions" actually means. Because there is an obvious and significant objection to treating forum posts as primary evidence, and it needs to be addressed before the method has any credibility.

The Self-Selection Problem

People who post in owner-operator forums about broker non-payment are not a random sample of carriers. They are a self-selected sample of carriers who experienced non-payment and were frustrated or motivated enough to post about it publicly. That is a heavily skewed population, and availability bias makes it worse: the posts that get the most replies, the most visible outrage, the most engagement are disproportionately the ones describing the worst outcomes. A carrier who got paid with minor friction after one follow-up email does not write a forum post about it.

This is not a reason to abandon the method. It is a reason to build the method around the bias rather than pretending the bias doesn't exist.

Nassim Taleb, in The Black Swan, describes the problem of silent evidence through the story of Diagoras, a nonbeliever shown painted tablets of shipwreck survivors who had prayed. Diagoras asked: "Where were the pictures of those who prayed, then drowned?" The forum equivalent is the thousands of broker-carrier payment interactions that resolved without drama and left no trace on the internet. If you only read the posts that exist, you are reading the visible sample and mistaking it for the full distribution.

What We Required Before a Post Counted

Two conditions had to be met before a post entered the evidence base.

The first was that the post had to be first-person. The author describing their own direct experience with a specific incident — not "I heard this happens a lot," not "a guy in my crew had this problem," not a general observation about the industry. When someone writes "I delivered the load, got a signed POD, invoiced, the broker stopped responding, and I lost $5,000 with no practical way to fight back" — that is a different category of evidence from "broker non-payment is a real problem."

The second was incident-level detail. A specific load. A dollar amount. What they tried. What happened. The specificity is what separates recalled experience from ambient opinion. Anyone can tell you non-payment is a problem. Only people who experienced it can tell you what the documentation looked like and why the bond claim wasn't filed in time.

Posts that expressed frustration without incident detail counted for less. Not for nothing — a pattern of frustration at scale is still a signal — but for less. The distinction matters because it keeps the method honest about what it's actually measuring.

The Popper Requirement

Karl Popper argued that the distinguishing feature of a genuine scientific claim is that it can be falsified. A hypothesis that cannot in principle be proven wrong is not a hypothesis at all — it is an article of faith. The practical implication for any evidence-gathering exercise is that you have to actively look for the evidence that would prove you wrong, not just accumulate evidence that confirms what you already suspect.

When you go into an owner-operator forum looking for evidence that carriers get stiffed on completed loads, you will find it. The forum exists partly for that purpose. The question is whether you looked for the counter-evidence with the same effort.

We ran counter-searches. Positive terms. "Broker paid same day." "No issues with broker." "Good experience, paid on time." The same platforms, the same period, the same structure.

What came back was sparse — genuinely sparse, not just underrepresented. Positive payment experiences appeared as brief context in threads primarily about something else. They didn't generate the same volume or the same intensity of engagement. That result is itself meaningful. It doesn't prove the problem is universal. It suggests the problem is common enough that the people who experience it are motivated to document it, while the people who don't have no particular reason to document that either.

The Platform Diversity Rule

A single forum is a community, not a market. Communities develop shared narratives, shared language, shared grievances. A pattern that appears strongly across one forum might reflect the culture of that forum rather than the reality of the market. The carriers on r/OwnerOperators might be unusually vocal about non-payment relative to carriers on TruckersReport or Facebook trucking groups, for reasons that have nothing to do with the underlying frequency of the problem.

The requirement we set was that passive evidence could only support a proceed decision if it drew from three or more independent platforms, with no single source accounting for more than 40% of the qualifying posts. The diversity requirement forces the evidence base to cross community lines. If the same pattern appears among carriers posting in Reddit communities, TruckersReport forums, and Facebook trucking groups — separately, with no membership overlap — the pattern is less likely to be a community artefact and more likely to reflect something structural about the market.

The Devil's Advocate Requirement

The final condition was the most uncomfortable to apply: every findings document produced from passive evidence had to include a devil's advocate section. A section written in bad faith toward the evidence. An argument, made as strongly as possible, that everything we found could be explained by something other than the problem being real and prevalent.

Possible alternative explanations for a high volume of non-payment forum posts:

• The audience skews toward less commercially sophisticated carriers who don't vet brokers properly before accepting loads
• The posting behaviour reflects personality type, not problem frequency
• The forums are dominated by a vocal minority
• Incidents involve smaller carriers who lack the factoring infrastructure that more established operators use to mitigate broker risk
• The amounts involved are manageable enough that the posts reflect frustration rather than a structural business problem

These arguments were written seriously, not dismissed. Then they were weighed against the evidence. Where they held — where the alternative explanation was plausible — that reduced confidence. Where they didn't hold — where the evidence pattern was inconsistent with the alternative explanation — that increased it.

Charlie Munger is well known for his principle of inversion — approaching problems backwards to identify what you are trying to avoid, not just what you are trying to achieve. The devil's advocate requirement is the operational version of that discipline applied to validation research: identify the strongest case against your own findings before committing to a proceed decision.

What This Produced

Twelve Tier 1 signals emerged from passive evidence collection across three platforms — specific incidents, named dollar amounts ranging from $1,300 to $7,900 per load, documented outcomes including bond claims filed, bond companies pushed back against, and losses absorbed. The pattern held across Reddit communities, TruckersReport forums, and Facebook trucking groups. Non-payment on completed loads, documentation gaps as the mechanism of disputes, and bond claim execution barriers all appeared independently across every platform searched without being prompted. No contradicting signal was found in counter-searches.

The Argument

When a primary outreach channel fails, there are three options. The first is to stop and wait for access — which may never come, or may come too slowly to be useful. The second is to manufacture signal from a compromised sample — people who aren't quite the target audience, surveys that substitute stated preference for behaviour. The third is to use passive evidence, gathered rigorously, with the bias built into the design rather than ignored.

The case for the third option is not that it's clean. It isn't. The case is that imperfect evidence gathered with explicit bias controls, counter-searches, platform diversity requirements, and a genuine devil's advocate argument is better than no evidence — and substantially better than evidence that doesn't know it's compromised.

The condition is rigour. Rules that are specific enough to be broken, conditions that are explicit enough to be tested, and a willingness to let the counter-evidence land rather than explain it away.

Applied to early-stage validation, that means designing your evidence-gathering so that a negative result is actually possible, not just theoretically acknowledged. Forum posts can meet that standard. The question is whether you hold them to it.