Trust & Methodology
G

Trust & Methodology

Trust & Methodology

SHIOS puts evidence before AI: 43,564 real sources underpin 649 forecasts, with 386 resolved and a 46% confirmation rate. SHIOS publishes every forecast before outcomes are known, then scores it against what actually happened — including misses.

How this is measured

Don't trust the AI. Inspect the evidence. This page exists because a skeptical visitor deserves a real answer, not a reassurance.

Documents read

43,564

real sources

Forecasts published

649

all-time

Resolved

386

have a verdict

Confirmation rate

46%

of resolved forecasts

How SHIOS actually works

Signal → Insight → Forecast → Evidence → Result

  1. 1.We collect signals — real job postings, real GitHub repositories, real documents.
  2. 2.We identify the facts in them — skills, technologies, roles mentioned.
  3. 3.We cross-check what we can before publishing anything.
  4. 4.AI helps analyze patterns in that evidence — it does not invent the evidence itself.
  5. 5.Every claim we publish is tied to the documents behind it, visible on request.
  6. 6.Forecasts are published before the outcome is known — never adjusted afterward to look right.
  7. 7.Forecasts are later compared against what actually happened.
  8. 8.The results stay visible, including the ones we got wrong.

AI-assisted research

What AI does here, and what it doesn't

AI helps SHIOS collect, organize, compare, and summarize information across job postings, repositories, and other documents. It does not grade its own forecasts — every forecast is scored by comparing a stored baseline against what was actually observed later, using fixed arithmetic, not a second AI opinion. AI may help explain a result in plain English after the fact; it never decides the result.

Source hierarchy

Not every source is treated the same

Primary evidence — the actual job postings, repositories, and documents SHIOS collects directly. This is what nearly everything on this site is currently built from.

Independent research — third-party firms like Lightcast, Gartner, or the Stanford AI Index have no live data feed here and are referenced only as a point of comparison, never as a claim that they validated anything on this site. Two exceptions have a real, free public feed: Revelio Labs' RPLS dataset (monthly labor-market statistics) and CB Insights' free newsletter. Both are collected automatically and shown under "External Benchmarks" on the Evidence page — always as corroboration alongside our own numbers, never blended into any trend or forecast calculation.

Secondary sources — journalism, industry commentary. Used sparingly, always labeled.

Synthetic/demo data — clearly labeled wherever it appears, and never mixed silently with real evidence.

Frameworks that inform our approach

We don't claim their endorsement

  • NIST AI Risk Management Framework — our approach to AI risk is informed by its structure.
  • OECD AI Principles — informs how we think about transparency and accountability.
  • ISO/IEC 42001 — informs our approach to AI management practices.
  • BLS (Bureau of Labor Statistics) — the kind of primary labor data we aim to eventually incorporate directly.
  • O*NET — a reference point for how occupations and skills are classified.
  • Stanford AI Index — a reference for how AI-adoption trends are typically measured.

None of these organizations certify, validate, or endorse SHIOS. Where we say a framework "informs" our approach, that's the only claim being made.

Signal classification

How strong is the evidence behind a signal?

Reports and practical guides label how much real evidence stands behind a pattern, so a single honest observation is never dressed up as a market trend.

Early observation
A single or very small number of examples — too little evidence yet to call it a pattern.
Directional signal
Real evidence pointing a direction, but under the recurring-theme threshold.
Recurring pattern in this sample
Meets the recurring-theme threshold across enough distinct employers in this sample.

Corrections & updates

What happens when something changes

Reports and guides carry a simple lifecycle status, always visible next to the content itself:

  • Published — live and current, the first time a report or guide goes public.
  • Updated — refreshed with new evidence since it was first published.
  • Superseded — replaced by a newer report on the same topic.
  • Archived — kept for reference, no longer actively maintained.
  • Withdrawn — removed from public display because of a correction or error.

If a published report turns out to need a factual correction, we say so directly on that report rather than silently editing it — never a quiet rewrite.