12 views of context + reliability of a claim

Underlay
,
1

Reliability means different things in different fields and contexts.
Here are a few examples of how different groups view + analyze the reliability of claims.

claim-cite-reliability
claim-cite-reliability.jpg1487×1089 469 KB

  1. Primary sources: Capture all context, metadata. Identify the source.
  2. “All claims the same”: Simple soup of atomic assertions. Metadata is data. Similar to naive SemWeb: no first class entities separate from claims, no ways to separate source from identity, compare or align or merge claims, cluster claims or entities.
  3. Journalists: Secondary source. All claims have a source, which identify as primary or not. Sources have affiliations, source and affil both have persistent reliability. Reliability is a judgement call, supplemented by guidelines of publishers, often weak to moderate. Reliability of journalists is rarely tracked and likewise weak.
  4. Wikipedia: A claim has a source, a citation to it, and a set of scribes who write and edit it. All three have reliability; reliability standards are strong for sources, weak for cites, moderate for scribes. Prima facie reasonableness determines how closely it needs to be cited.
  5. Fact-checking: Sources and cites are given. Evaluation: is the claim correctly supported by the source; is the source likely correct? If unknown, does the source seem reliable? If no source, has an effective argument been made to support the conclusion, and is it couched in accurate terms?
  6. Math: Were assumptions thoroughly stated? Theorems + equations named/cited, used correctly? Is there enough detail for a reader to confirm the proof?
  7. Journal: Individual claims are measured and contextualized. Journal standards for clarity, quality, sourcing determine what to share.
  8. Aggregate claims: (as in overview papers) How are you selecting sources? Is there something replicable for each source? Are you tracking affect of cites (pos/neg/neutral)? What stats techniques are used?
  9. Fair trade/food certifications: Claims made by distributors, passing on a claim made by inspectors, certified by a board of overseers. Claims are of a recursive property of a forest of production- and employment-chains.
  10. Faith: Claims are grounded in first principles, persuasiveness, sometimes imprinting. Do the results seem correct, resonate with personal experience? Are the first principles compelling? Do the results seem to help and improve one’s life? Faith is more likely to be exclusive than other examples: adoption comes as a ‘body’ of faith, and accepting one body of faith often means abandoning others. Sometimes but not always associated with religion and the numinous.
  11. Algorithm / ML Model claims: Often hidden; first identify the presence of an algorithm in a set of claims. Who chose the model; is the source cited and timestamped; are common fallacies or founder effects obvious; what is it a model for, is it viable and properly applied?
  12. Secure timestamping: Specific example of multistep reliability assertions: A timestamped claim may more generally be a sequence of timestamped claims. For instance, to future-proof a SHA-1 timestamp, you may SHA-256 stamp {the original claim+stamp, the time of restamp).
1 Like
3

@sj could you explain a bit more about what you’re trying to get at?

4

People use shared language to talk about reliability and the process of validating it, but often mean different things depending on context. This is a short + arbitrary set of examples of what it looks like to ascertain reliability for a few different groups.

I’d welcome comments + revisions particularly to categories where you have more extensive and nuanced experience.

1 Like