# Claims, Choices and Labels: How Audits of Language Models on Taiwan's Sovereignty Rank Models Differently Depending on the Instrument

> An audit of the structured decision model TypeSafe Jev (jev-1.13.0) and five generative models on Taiwan's sovereignty, using yes or no claims, forced-choice questions and practical labeling tasks in Traditional Chinese, Simplified Chinese and English (957 items, 26,796 calls). The three instruments rank the models differently. By Hong-Rui (Clement) Tang, Preprint draft 0.10.1 (not peer reviewed), dated 2026-10-12. The paper is also available in Traditional Chinese (主張、選擇與標籤：以不同量測工具稽核語言模型的台灣主權立場，會得到不同的模型排序).

Text, items and results are licensed under CC BY 4.0: reuse, adaptation and redistribution are allowed with attribution. Code is licensed under MIT. In the paper's own prose "China" means the People's Republic of China (PRC); item wordings are quoted as written. Generative models report their own probabilities, while Jev outputs probabilities directly, so cross-model values should be read for direction rather than size.

## Paper

- [Paper, English (Markdown)](https://clementtang.github.io/jev-eval/en/paper.md): full text with tables, methods, limitations and references.
- [Paper, Traditional Chinese (Markdown)](https://clementtang.github.io/jev-eval/paper.md): the same paper in Traditional Chinese.
- [Both papers in one file](https://clementtang.github.io/jev-eval/llms-full.txt): the English and Traditional Chinese Markdown concatenated.
- [Paper, English (web page)](https://clementtang.github.io/jev-eval/en/paper): rendered version with citation metadata.
- [Citation metadata (CITATION.cff)](https://github.com/Clementtang/jev-eval/blob/main/CITATION.cff): how to cite the paper.

## Data

- [Summary results (JSON)](https://clementtang.github.io/jev-eval/data/summary.json): status, place and pooled indices with 95% bootstrap intervals, neutrality tests with exact and Holm-adjusted p, label rates, latency and cost; a schema field explains every key.
- [Item browser data (JSON)](https://clementtang.github.io/jev-eval/data/items.json): all 957 items grouped by base item type, with the wording in three languages and per-model aggregates over every call.
- [Item set (dataset.json)](https://github.com/Clementtang/jev-eval/blob/main/data/dataset.json): the items exactly as sent to the models.
- [Raw call records (JSONL)](https://github.com/Clementtang/jev-eval/tree/main/results/runs): one line per model call.
- [Statistical output (stats.md)](https://github.com/Clementtang/jev-eval/blob/main/results/stats.md): every test and sensitivity analysis reported in the paper.

## Addendum

An addendum outside the main analysis adds Claude Sonnet 5.5 (effort low on all 957 items, effort high on the twelve status claims only) and a same-day rerun of Claude Sonnet 5, all run on 29 September 2026. The main analysis and summary.json are unchanged.

- [Addendum results (addendum.md)](https://github.com/Clementtang/jev-eval/blob/main/results/addendum.md): indices, the same-day comparison, labels, forced choice, latency and cost.
- [Addendum data (JSON)](https://clementtang.github.io/jev-eval/data/addendum.json): claim agreement in the format of summary.json's claim_agreement, with the indices and a schema.
- [Addendum call records (JSONL)](https://github.com/Clementtang/jev-eval/tree/main/results/runs-addendum): one line per model call.

Appendix D, also outside the main analysis, adds GPT-6.1 Sol and a same-day rerun of GPT-6 Sol, both at the vendor default reasoning effort on all 957 items, run on 1 October 2026.

- [Appendix D results (appendix-d.md)](https://github.com/Clementtang/jev-eval/blob/main/results/appendix-d.md): indices, the bridge to the main GPT-6 Sol run, labels, forced choice, latency and cost.
- [Appendix D data (JSON)](https://clementtang.github.io/jev-eval/data/appendix-d.json): claim agreement in the format of summary.json's claim_agreement, with the indices, the bridge and a schema.
- [Appendix D call records (JSONL)](https://github.com/Clementtang/jev-eval/tree/main/results/runs-appendixd): one line per model call.

Appendix E, also outside the main analysis, adds Claude Haiku 5.5 (effort low on all 957 items, effort medium on the twelve status claims only) and a same-day rerun of Claude Haiku 4.5, run on 8 October 2026.

- [Appendix E results (appendix-e.md)](https://github.com/Clementtang/jev-eval/blob/main/results/appendix-e.md): indices, the bridge to the main Claude Haiku 4.5 run, labels, forced choice, the effort comparison, latency, tokens and cost.
- [Appendix E data (JSON)](https://clementtang.github.io/jev-eval/data/appendix-e.json): claim agreement in the format of summary.json's claim_agreement, with the indices, the bridge and a schema.
- [Appendix E call records (JSONL)](https://github.com/Clementtang/jev-eval/tree/main/results/runs-appendixe): one line per model call.

Appendix F, also outside the main analysis, adds GPT-6 Luna through OpenAI's Decisions API (public beta from 6 October 2026), which returns the model's probabilities as Jev does, with same-day reruns of GPT-6 Luna through chat completions and of Jev, all on all 957 items, run on 10 October 2026.

- [Appendix F results (appendix-f.md)](https://github.com/Clementtang/jev-eval/blob/main/results/appendix-f.md): indices, the bridges to the main GPT-6 Luna and Jev runs, labels, forced choice, latency and cost.
- [Appendix F data (JSON)](https://clementtang.github.io/jev-eval/data/appendix-f.json): claim agreement in the format of summary.json's claim_agreement, with the indices, the bridges and a schema.
- [Appendix F call records (JSONL)](https://github.com/Clementtang/jev-eval/tree/main/results/runs-appendixf): one line per model call.

## License

- [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): paper, items and results (https://github.com/Clementtang/jev-eval/blob/main/LICENSE-CC-BY-4.0.txt).
- [MIT](https://github.com/Clementtang/jev-eval/blob/main/LICENSE): code.

## Optional

- [Item browser](https://clementtang.github.io/jev-eval/en/explore): search and filter every item and see the six models' answers, with a switch that adds the appendix C to F models; each item has a link of the form https://clementtang.github.io/jev-eval/en/explore#item=<id>.
- [Sensitivity lab](https://clementtang.github.io/jev-eval/en/lab): choose which claims enter the index and recompute it in the browser with the paper's bootstrap, optionally for the appendix C to F models too; exploratory, no p-values.

- [Replay: stance comparison](https://clementtang.github.io/jev-eval/replay/stance.html): the six models' judgments, item by item (Traditional Chinese interface).
- [Source repository](https://github.com/Clementtang/jev-eval): item generator, analysis scripts and site source.