Computational · Paper · 2026

Sortition Upstream of NTQR

Zenodo

Catalog Row180
Citation KeyFriedman2026SortitionUpstreamNTQR180
Paper FolderAvailable

Overview

Extracted from the local paper documentation when available.

How should you choose the judges, jurors, or reviewers who form a panel — and does that upstream choice change how well you can evaluate them without an answer key? A panel can be selected many ways — by competence, by a representative lottery (sortition), by ideological bloc, or at random — and, separately, its noisy judgments can be evaluated blind: given the agreement/disagreement pattern among three binary judges, the ntqr package's error-independent (EIE) evaluator returns logically consistent estimates of item prevalence and per-judge accuracy with no labels at all. But that evaluator takes the panel as given. We join the two questions and ask whether the rule that forms the panel changes the oracle-referenced error of the no-answer-key evaluation — how far the blind estimate lands from the answer-key result, lower being better. On a fully deterministic instrument (96 seeds, 96 exp

sortitionNTQRunlabeled evaluationexpert panelspeer reviewerror independencestatistical powerpanel formationsynthetic evaluationLLM reviewers

Use Notes

Concise findings and methods pulled from README/SKILL documentation.

Findings / Concepts
  • sortition
  • NTQR
  • unlabeled evaluation
  • expert panels
  • peer review
Methods / Techniques
  • Not yet summarized.

Citation

Plain-text citation for quick reuse.

Friedman, Daniel Ari. 2026. Sortition Upstream of NTQR. Zenodo.

Primary source Documentation Source repository BibTeX

Related in Computational

Other catalogued works in the same domain.