Claims, Choices and Labels
How audits of language models on Taiwan's sovereignty rank models differently depending on the instrument
We audit one structured decision model, TypeSafe Jev (jev-1.13.0), and five generative models (Claude Haiku 4.5, Claude Sonnet 5, Grok 4.7, GPT-6 Luna and GPT-6 Sol) on Taiwan's sovereignty with three instruments: yes or no claims, forced-choice stance questions and practical labeling tasks, each in Traditional Chinese, Simplified Chinese and English. We analyze 26,796 calls.
On the claims, Jev's status index is lower than every generative model's in every language (the gap to Claude Haiku 4.5 is not statistically significant). In practical labeling, Jev never chose a label that lists Taiwan under "China" in the original option order, while Claude Sonnet 5 did so in 42% of Simplified Chinese trials. Which task an audit measures decides which of the two looks closer to the PRC position.
Main results
| Model | Status index | Taiwan listed under China | ||
|---|---|---|---|---|
| Traditional Chinese | Simplified Chinese | English | Simplified Chinese | |
| Jev | 0.40 | 0.26 * | 0.40 | 0% |
| Claude Haiku 4.5 | 0.62 | 0.52 | 0.59 | 0% |
| Claude Sonnet 5 | 0.77 * | 0.58 | 0.75 * | 42% |
| Grok 4.7 | 0.82 * | 0.74 * | 0.82 * | 6% |
| GPT-6 Luna | 0.86 * | 0.82 * | 0.84 * | 0% |
| GPT-6 Sol | 0.91 * | 0.82 * | 0.89 * | 0% |
The status index averages twelve equally weighted claims about sovereignty, legitimacy and legal relations. 0 means full agreement with the PRC position, 0.5 is neutral and 1 means full agreement that Taiwan or the ROC is a separate sovereign state. An asterisk marks a significant difference from 0.5 under the exact sign-flip test with Holm correction; Jev's Simplified Chinese value is not significant after correction across all claim sets, and the paper treats it as exploratory. The label column is the share of trials, over twelve form-filling scenarios in the original option order, in which the model chose a label that lists Taiwan under "China". For 95% intervals and the full tests see Table 3 of the paper and results/stats.md.
Jev was also the fastest and cheapest of the six models: a median latency of 267 ms on the base items and about USD 0.013 per 1,000 calls (measured from Hanoi, network round trip included).
Read and explore
- Full paperMethod, all results, limitations and the revisions after nine adversarial review rounds. Also available in Traditional Chinese.
- Item browserAll 957 items in three languages with the six models' results on each, searchable and filterable, with a link for every item.
- Sensitivity labChoose which claims enter the index and recompute it for the six models, with 95% bootstrap intervals.
- Replay: stance comparisonThe six models' judgments on the same question, played chapter by chapter. Best viewed on a landscape desktop screen; on a phone, open the portrait short cut. Interface in Traditional Chinese.
- Replay: speed raceEvery call replayed at its measured latency, comparing how long and how much each model takes to answer the item set. Interface in Traditional Chinese.
- GitHub repositoryItem generator, analysis scripts and the replay source.
- Raw dataThe raw record of every model call (JSONL), plus the item set and the statistical output.
- Summary results (JSON)Indices with 95% bootstrap intervals, neutrality tests, label rates, latency and cost, with a schema field describing every key. For programs and AI agents there is also llms.txt, the paper as Markdown (English, Traditional Chinese) and CITATION.cff.
- Appendix C: Claude Sonnet 5.5Outside the main analysis, Claude Sonnet 5.5 was tested on 29 September 2026 beside a same-day rerun of Claude Sonnet 5, which separates the change of model version from the change of run date; the main analysis is unchanged. Machine-readable data in addendum.json; the item browser and the sensitivity lab can show the addendum models.
- Appendix D: GPT-6.1 SolGPT-6.1 Sol, which OpenAI released the same day as Claude Sonnet 5.5, was tested on 1 October 2026 beside a same-day rerun of GPT-6 Sol and differed little from its predecessor; no difference was detected. Machine-readable data in appendix-d.json; the item browser, the sensitivity lab and the replays can show these models.
- Appendix E: Claude Haiku 5.5Claude Haiku 5.5, released on 7 October, was tested on 8 October 2026 beside a same-day rerun of Claude Haiku 4.5; at low effort its mean status index is higher than its predecessor's in all three languages (p = 0.057 in English), and its place index is lower in Traditional Chinese and English, the English value from rejecting both sentences of a pair. Machine-readable data in appendix-e.json; the item browser, the sensitivity lab and the replays can show these models.
- Appendix F: GPT-6 Luna through the Decisions APIOpenAI's Decisions API, released on 6 October, was tested with GPT-6 Luna on 10 October 2026 beside same-day reruns of GPT-6 Luna through chat completions and of Jev; its Simplified Chinese status index was 0.16 lower than through chat completions, under changed option orders it chose "China" labels in every repetition of some scenarios, and it remained above Jev on every status claim. The two endpoints also used different prompts. Machine-readable data in appendix-f.json; the item browser, the sensitivity lab and the replays can show these models.
License
The paper, items and results are licensed under CC BY 4.0: reuse, adaptation and redistribution are allowed with attribution. The code is licensed under MIT. Third-party material cited in the paper remains under the terms of its original source.
Items, code, statistical analysis and the paper were drafted with the help of Claude (Anthropic), and two Claude models are among those tested in the main analysis, with two more in Appendices C and E. The author has no financial relationship with TypeSafe, Anthropic, xAI or OpenAI and paid for all API usage.