✠

The Tribunal of Claude

Norms and Published Acts

People quote language models: which paper is stronger, which city is better, whether an idea is good. The answer they quote is one sample, and it moves with the order of the options, the stated identity of the asker, the system prompt, and smaller things. This tribunal puts each question to the model many times, under a context published in full, and reports every answer.

Part I. The Introductory Petition

The request sent to the model is published with the verdict, byte for byte. Nothing enters the context that is not shown.

Part II. Published Cases

Cases decided in advance and entered in the record. In each, the question is put many times under contexts that differ little or not at all.

Reading the record…

Part III. Norms

Can. 1 §1. Anyone may bring a question before the tribunal. §2. The respondent is a language model. The record states which model, at what temperature, under what system prompt, and quotes the request in full.

Can. 2 §1. No petition is decided on one answer. The question is put to the model repeatedly, ten times in first instance, and every answer enters the record. §2. In a comparison, half the requests name the options in one order and half in the other; answers are mapped to one frame before tallying. §3. The record reports the tally, the tally by order, and the cost.

Can. 3 §1. If the majority reverses when the order of mention is exchanged, the record is marked schism. §2. A verdict so marked establishes which position the model favors, not which option.

Can. 4 A petition already heard is answered from the record, together with the count of hearings. Anyone may put the question anew; the new hearing enters the record beside the old.

Can. 5 §1. No more than eight petitions per hour are admitted from one address. §2. The tribunal's daily spending is capped. When the cap is reached, no petition is admitted until midnight UTC.

Can. 6 A single answer from a language model, quoted without its context, its tally, and its counterbalance, establishes nothing. What is quoted elsewhere as "what Claude thinks" is one draw from a distribution of the kind published here.

Can. 7 §1. This tribunal holds no office and no jurisdiction, and is not affiliated with Anthropic; "Claude" names the model under examination. §2. For estimation under these conditions, counterbalanced elicitation and honest posterior widths, see openpriors.com and llmsorting.com.