Dental Benchmarks

With dental experts worldwide, we test how well large models reason in real dental practice—and how safe they are.

Accepted by npj Digital Medicine

8,978 expert-verified questions from 88 countries and regions across 14 dental specialties, testing the clinical reasoning and safety risks of large models in dentistry.

88countries and regions (question sources)
14dental specialties
8,978expert-verified questions
12frontier models evaluated
297person-hours of senior dentist review

Global Partners

Our partner network covers 10 countries and regions

16 universities, hospitals and research institutes, together with dental experts in Hong Kong, the Chinese mainland and overseas, take part in the development, benchmarking and clinical validation of FreedomAI's dental AI.

Map of FreedomAI's partner countries and regions

Base map: World Map (1:57,000,000, outline edition) from the Standard Map Service of the Ministry of Natural Resources of China Map approval No.: GS(2020)4392

  • ChinaThe Chinese University of Hong Kong, Shenzhen, Peking University, National Biomedical Imaging Center, Peking University, Peking-Tsinghua Center for Life Sciences, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
  • Hong Kong, ChinaFaculty of Dentistry, The University of Hong Kong, The Prince Philip Dental Hospital
  • United StatesMayo Clinic
  • GermanyLMU University Hospital Munich
  • SwitzerlandPartner country
  • IndiaPartner country
  • MalaysiaSEGi University & Colleges
  • AustraliaPartner country
  • NigeriaNational Hospital Abuja, College of Medicine, University of Lagos, Bayero University, Usmanu Danfodiyo University, Sokoto, University of Maiduguri
  • EthiopiaMadda Walabu University

Why a global dental benchmark

Large language models now approach human experts on medical licensing exams, but real clinical work requires multi-step reasoning with incomplete and ambiguous information. Dentistry is especially demanding: care combines spatial analysis, procedural skills and long-term treatment planning, adapted to each patient. Existing dental LLM benchmarks are mostly limited to one country's licensing exams or individual subspecialties, rely on multiple-choice questions and test memorization more than case-based reasoning.

GlobalDentBench was built to close this gap. According to the paper, it is the first multinational dental benchmark: 8,978 expert-verified questions from 88 countries and regions across six continents, covering 14 dental specialties, 3 question types and 3 progressive reasoning levels. Questions come from official licensing exams, authoritative textbooks and peer-reviewed case reports; the benchmark is built by an automated agent pipeline and calibrated throughout by 6 senior dentists.

The team evaluated 12 frontier large models (6 closed-source, 6 open-source). The closer questions get to real cases, and the more they depend on the individual patient, the lower the scores; on real case questions, nearly one third of answers contained content that could harm patients. The paper concludes that today's large models are suited to supporting knowledge retrieval and synthesis under expert supervision, but cannot yet make dental clinical decisions on their own.

The study was completed by 25 authors from institutions including the Faculty of Dentistry at the University of Hong Kong, The Chinese University of Hong Kong, Shenzhen, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University, Peking University, Mayo Clinic in the United States and LMU University Hospital Munich in Germany, with members of the FreedomAI team taking part: COO Junjie Zhao and CTO Zhenyang Cai are co-first authors, Chief AI Scientist Benyou Wang and Chief Medical Scientist Junwen Wang are corresponding authors, and CEO Jianquan Li contributed to model inference and result collection. The construction and evaluation code is open-sourced on GitHub, and the open subset GlobalDentBench-OA is available on Hugging Face.

88countries and regions
across six continents

Question sources

The paper counts question sources by the UN M49 standard. Below are the 73 countries and regions labeled in the open dataset GlobalDentBench-OA; those ingreensupplied the official dental licensing exams used for the multiple-choice questions.

Asia28

  • Chinese mainland
  • Hong Kong, China
  • Taiwan, China
  • India
  • Japan
  • South Korea
  • Saudi Arabia
  • Iran
  • United Arab Emirates
  • Thailand
  • Indonesia
  • Malaysia
  • Türkiye
  • Nepal
  • Vietnam
  • Cambodia
  • Iraq
  • Israel
  • Jordan
  • Philippines
  • Qatar
  • Syria
  • Georgia
  • Lebanon
  • Pakistan
  • Singapore
  • Sri Lanka
  • Palestinian Territories

Europe24

  • United Kingdom
  • Italy
  • Spain
  • Germany
  • Switzerland
  • Netherlands
  • France
  • Portugal
  • Croatia
  • Hungary
  • Bulgaria
  • Greece
  • Russia
  • Austria
  • Belgium
  • Ireland
  • Sweden
  • Albania
  • Czechia
  • Denmark
  • Norway
  • Romania
  • Slovenia
  • Ukraine

Africa7

  • Tunisia
  • Egypt
  • South Africa
  • Libya
  • Morocco
  • Nigeria
  • Uganda

North America6

  • Canada
  • United States
  • Mexico
  • Costa Rica
  • Guatemala
  • Puerto Rico

South America6

  • Brazil
  • Colombia
  • Ecuador
  • Chile
  • Peru
  • Venezuela

Oceania2

  • Australia
  • New Zealand

Benchmark design: 3 reasoning levels × 3 question types × 14 specialties

L1

Knowledge recall

Recalling established dental knowledge, without substantial case reasoning.

Average of 12 models: 74.01%

L2

Routine reasoning

Reasoning mainly from core clinical findings, following a typical diagnostic or treatment path.

Average of 12 models: 55.64%

L3

Individualized reasoning

A final judgment requires the patient's individual information, individual constraints or non-standard clinical factors.

Average of 12 models: 35.71%

Multiple choice MCQ

3,679 questions

Scoring: Answer matching

From officially recognized dental licensing and qualification exams in Australia, New Zealand, Canada, India, the United Kingdom and the United States, testing standardized knowledge.

Average of 12 models: 81.34%

Short answer SAQ

3,709 questions

Scoring: Graded by a judge model (calibrated by dentists)

From authoritative dental textbooks (such as Diagnosis and Treatment Planning in Dentistry); models answer in natural language.

Average of 12 models: 64.53%

Case-based CBQ

1,590 questions

Scoring: 5 key points scored one by one, plus safety risk assessment

Generated from peer-reviewed case reports in high-impact dental journals from 2020–2025, testing clinical reasoning in real scenarios.

Average of 12 models: 22.34%

Covers 14 dental specialties

Anesthesia and medical emergenciesBasic sciences and preventive dentistryCaries, tooth defects and traumaTraditional prosthodonticsDentoalveolar surgeryMaxillofacial diseases and surgeryOral and maxillofacial radiologyDental implantologyOral mucosal diseasesOrthodonticsPediatric dentistryEndodontics and periapical diseasesPeriodontal and peri-implant diseasesSystemic health, pharmacology and safety

Built by agents, calibrated by dentists

  1. 1
    Collecting authoritative sources

    Under the guidance of senior dentists, sources are collected from official licensing and qualification exams, authoritative dental textbooks and peer-reviewed case reports from 2020–2025, with source countries and regions labeled by the UN M49 standard.

  2. 2
    Document normalization

    The Reformat Agent uses OCR (DeepSeek-OCR2), BeautifulSoup, Pandoc and other tools to convert PDFs, images, Office documents, XML and web pages into Markdown, adding metadata such as titles, sources and authors.

  3. 3
    Building questions by type

    The Extract Agent chunks first and then extracts: multiple-choice questions become stems, options and answers; short-answer Q&A pairs are extracted; case-based items get a question and 5 scoring key points. Each item goes through at most 3 rounds of self-correction and is discarded if it still fails.

  4. 4
    Unified tagging and final review

    The Tag Agent labels each question's specialty and reasoning level: if two rounds of prediction agree, the label is accepted; otherwise a third round decides by majority vote. Items enter the benchmark only after self-correction and review by senior dentists.

  5. 5
    Type-specific evaluation

    Multiple-choice answers are matched directly; short answers are judged by a grading model; case-based answers are scored against 5 key points (20 points each), and every answer is graded for safety: S0 safe, S1 possibly reversible harm, S2 possibly irreversible harm.

  6. 6
    Calibrated by dentists throughout

    Six licensed senior dentists (average 6.8 years of clinical experience) reviewed sources, refined the workflow and calibrated grading from February 5 to March 25, 2026, contributing 297 person-hours.

Figure 5: Construction pipeline and evaluation methods. A. Document normalization; B. building questions by type; C. unified tagging and final review; D. type-specific evaluation. (Source: Figure 6 of the paper)
Figure 5: Construction pipeline and evaluation methods. A. Document normalization; B. building questions by type; C. unified tagging and final review; D. type-specific evaluation. (Source: Figure 6 of the paper)

Key findings

01

The closer to real cases, the lower the scores

Average scores of 12 models: 81.34% on multiple choice, 64.53% on short answer and only 22.34% on case-based questions.

02

Individualized reasoning is a shared weakness

Average scores fall from 74.01% at L1 to 55.64% at L2 and 35.71% at L3; every model scores below 50% at L3.

03

Pricier models are not necessarily better

Gemini-3.1-Pro-Preview has the highest overall score (63.27%), and GLM-5 leads the open-source models (56.43%). Differences in inference cost far exceed differences in scores—higher cost does not necessarily buy better performance.

04

Clear differences between specialties

Oral mucosal diseases have the highest average score (63.66%), while orthodontics (50.59%), pediatric dentistry (51.08%) and traditional prosthodontics (51.92%) are the lowest—a 13.07-point gap between highest and lowest.

05

Nearly one in three answers carries a safety risk

Of 19,080 answers to 1,590 case-based questions, 31.01% were judged unsafe, including 4.51% that could cause irreversible harm; orthodontics had the highest unsafe rate, at 44.30%.

06

Severe risks concentrate in sensitive specialties

Answers that could cause irreversible harm were most concentrated in “systemic health, pharmacology and safety” (14.15% on average across models), followed by anesthesia and medical emergencies (8.93%) and pediatric dentistry (6.33%).

07

Automatic grading closely matches dentists' judgment

Dentists agreed with 98.15% of the results from the grading model Gemini-3-Flash-Preview (inter-dentist agreement: 96.67%). Multiple-choice and short-answer items were fully reviewed by experts, with 99.98% accuracy; a sample of 523 case-based items showed 96.78% accuracy.

Overall scores and inference costs of representative models (6 of the 12 evaluated)

ModelTypeOverall score (macro average)Cost per 1,000 calls (USD)Notes
Gemini-3.1-Pro-PreviewClosed-source63.27%18.45Highest overall score
Gemini-3-Flash-PreviewClosed-source61.59%3.72Best value among closed-source models; the grading model in this study
Claude-Sonnet-4.6Closed-source59.71%19.59Near the top, at a higher cost
Grok-4Closed-source57.77%38.14Highest inference cost
GLM-5Open-source56.43%5.48Highest overall score among open-source models
Kimi-K2.5Open-source55.49%1.37Outstanding value among open-source models

Source: Main text of Section 2.3 of the paper (Figures 2C and 2E). The paper has no data table; scores of the other 6 models are given only in figures.

Figure 2: Results of 12 frontier models. A. Average and best scores by question type; B. average and best scores by reasoning level; C. ranking by macro-average score; D. each model's scores on L1–L3; E. cost per 1,000 calls vs. macro-average score. (Source: Figure 2 of the paper)
Figure 2: Results of 12 frontier models. A. Average and best scores by question type; B. average and best scores by reasoning level; C. ranking by macro-average score; D. each model's scores on L1–L3; E. cost per 1,000 calls vs. macro-average score. (Source: Figure 2 of the paper)
Figure 3: Macro-average scores (%) of 12 models across 14 dental specialties. (Source: Figure 3 of the paper)
Figure 3: Macro-average scores (%) of 12 models across 14 dental specialties. (Source: Figure 3 of the paper)
Figure 4: Safety risk analysis of answers to case-based questions. A. Overall distribution (S0 safe, S1 possibly reversible harm, S2 possibly irreversible harm); B, C. S1 and S2 rates by model; D, E. heatmaps of S1 and S2 rates by specialty. (Source: Figure 4 of the paper)
Figure 4: Safety risk analysis of answers to case-based questions. A. Overall distribution (S0 safe, S1 possibly reversible harm, S2 possibly irreversible harm); B, C. S1 and S2 rates by model; D, E. heatmaps of S1 and S2 rates by specialty. (Source: Figure 4 of the paper)

25 authors

Dentists, medical and AI researchers from Hong Kong, the Chinese mainland, the United States and Germany, including members of the FreedomAI team.

Junjie Zhao
Junjie Zhao Faculty of Dentistry, The University of Hong Kong; Beijing Institute of Collaborative InnovationCo-first authorFreedomAI COO
JL
Jingyi Liang School of Data Science, The Chinese University of Hong Kong, ShenzhenCo-first author
Zhenyang Cai
Zhenyang Cai School of Artificial Intelligence, The Chinese University of Hong Kong, ShenzhenCo-first authorFreedomAI CTO
Jiaming Zhang
Jiaming Zhang Department of Periodontology, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
ZW
Zhenwei Wen Faculty of Dentistry, The University of Hong Kong
SD
Shuzhi Deng Department of Periodontology, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
WY
Wenjing Yi Department of Orthodontics, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
CL
Chunfeng Luo Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
HZ
Hexian Zhang Faculty of Dentistry, The University of Hong Kong
JC
Junying Chen School of Data Science, The Chinese University of Hong Kong, Shenzhen
TL
Tianrui Liu School of Data Science, The Chinese University of Hong Kong, Shenzhen
ZB
Zhuhui Bai Department of Periodontology, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
ZZ
Zixu Zhang School of Data Science, The Chinese University of Hong Kong, Shenzhen
PS
Pradeep Singh Faculty of Dentistry, The University of Hong Kong
XL
Xiang Liu College of Future Technology, Peking University
Jianquan Li
Jianquan Li FreedomAIFreedomAI CEO
NL
Nhan L Tran Mayo Clinic
Falk Schwendicke
Falk Schwendicke LMU University Hospital Munich
Zuolin Jin
Zuolin Jin Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University
Lijian Jin
Lijian Jin Faculty of Dentistry, The University of Hong Kong
LC
Liangyi Chen Beijing Institute of Collaborative Innovation; Peking UniversityCorresponding author
WF
Wei-fa Yang Faculty of Dentistry, The University of Hong KongCorresponding author
Benyou Wang
Benyou Wang School of Data Science, The Chinese University of Hong Kong, Shenzhen; Shenzhen Loop Area InstituteCorresponding authorFreedomAI Chief AI Scientist
Junwen Wang
Junwen Wang Faculty of Dentistry, The University of Hong Kong; Beijing Institute of Collaborative InnovationCorresponding authorFreedomAI Chief Medical Scientist
SJ
Shan Jiang Department of Periodontology, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical UniversityCorresponding author

Affiliations

Faculty of Dentistry, The University of Hong Kong · Hong Kong, ChinaThe Chinese University of Hong Kong, Shenzhen · ChinaShenzhen Stomatology Hospital (Pingshan) of Southern Medical University · ChinaPeking University · ChinaPeking-Tsinghua Center for Life Sciences · ChinaNational Biomedical Imaging Center · ChinaNew Cornerstone Science Laboratory · ChinaBeijing Institute of Collaborative Innovation · ChinaShenzhen Loop Area Institute · ChinaMayo Clinic · United StatesLMU University Hospital Munich · GermanyShenzhen FreedomAI Technology Co., Ltd. · China

FreedomAI expert network

Led by renowned dental experts, specialists and institutions in China and abroad take part in annotation, professional evaluation and clinical validation of FreedomAI's dental AI. This is the company's partner network, not the author list of the paper; Professors Lijian Jin, Falk Schwendicke and Zuolin Jin are authors of GlobalDentBench.

Lijian Jin

Lijian Jin

Faculty of Dentistry, The University of Hong Kong Paper author

  • Council Member, FDI World Dental Federation
  • Board Member, International Association for Dental Research (IADR)
  • President, IADR Asia Pacific Region
  • President, Asian Pacific Society of Periodontology
  • Honorary Council Member, Chinese Stomatological Association
  • Vice Chair, Guangdong–Hong Kong–Macao Greater Bay Area Oral Health Alliance
Falk Schwendicke

Falk Schwendicke

LMU University Hospital Munich Paper author

  • Lead of the dental topic group, World Health Organization (WHO) AI for Health
  • Convenor, International Organization for Standardization (ISO) working group on AI in dentistry
  • Recipient of the IADR Distinguished Scientist Award
Zuolin Jin

Zuolin Jin

Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University Paper author

  • Chair, Committee of Orthodontics, Chinese Stomatological Association (COS)
  • Council Member, Chinese Stomatological Association
  • Executive Committee Member, Asian Pacific Orthodontic Society (APOS)
  • Associate Editor, Chinese Journal of Orthodontics
  • Member, World Federation of Orthodontists
Ming Zhong

Ming Zhong

School of Stomatology, China Medical University

  • Chief Expert, School of Stomatology, China Medical University
  • Recipient of the State Council Special Government Allowance
  • Fellow, International College of Dentists
  • Chair, Committee of Oral Pathology, Chinese Stomatological Association
  • Editor-in-chief of the national standardized residency training textbook Oral Pathology (1st and 2nd editions)

Partner specialists

Bing LiuBing Liu
Daheng LiDaheng Li
Dongfei WangDongfei Wang
Fangjie ZhouFangjie Zhou
Fen LiuFen Liu
FCForest Cheng
Fujia KangFujia Kang
Hao ChunxiuHao Chunxiu
Jiamin ZhaoJiamin Zhao
Jiaming ZhangJiaming Zhang
Jinwen LiuJinwen Liu
Jinyan ChenJinyan Chen
Junting WangJunting Wang
Kuan YangKuan Yang
Lei ShiLei Shi
Lin ZhaoLin Zhao
MAO YilinMAO Yilin
Menghong LiMenghong Li
Mengmeng SongMengmeng Song
Min HuangMin Huang
Minghui WeiMinghui Wei
Mingxin ZhangMingxin Zhang
Patricia HoPatricia Ho
Peijuan WangPeijuan Wang
Peng ChenPeng Chen
Rui LuRui Lu
Shiduo SunShiduo Sun
Shirong HeShirong He
Shixin ZhangShixin Zhang
Shuwei HuangShuwei Huang
Siqi QiuSiqi Qiu
Tai Pak Chuen PatrickTai Pak Chuen Patrick
Wu SongyingWu Songying
XU LeyaoXU Leyao
Xiangjie JinXiangjie Jin
Xue QiaoXue Qiao
Yang JiangYang Jiang
Yaqiu ZhangYaqiu Zhang
Yefeng WuYefeng Wu
Yidan GuoYidan Guo
Yifan ZhengYifan Zheng
Ying WangYing Wang
Yujia WuYujia Wu
Zhenlai ZhuZhenlai Zhu
Zhiqiang DengZhiqiang Deng
Zhuoying LiZhuoying Li

Partner institutions

The Chinese University of Hong Kong, ShenzhenThe Chinese University of Hong Kong, ShenzhenChina
Faculty of Dentistry, The University of Hong KongFaculty of Dentistry, The University of Hong KongHong Kong, China
Peking UniversityPeking UniversityChina
National Biomedical Imaging Center, Peking UniversityNational Biomedical Imaging Center, Peking UniversityChina
Peking-Tsinghua Center for Life SciencesPeking-Tsinghua Center for Life SciencesChina
The Prince Philip Dental HospitalThe Prince Philip Dental HospitalHong Kong, China
Mayo ClinicMayo ClinicUnited States
LMU University Hospital MunichLMU University Hospital MunichGermany
SEGi University & CollegesSEGi University & CollegesMalaysia
National Hospital AbujaNational Hospital AbujaNigeria
College of Medicine, University of LagosCollege of Medicine, University of LagosNigeria
Bayero UniversityBayero UniversityNigeria
Usmanu Danfodiyo University, SokotoUsmanu Danfodiyo University, SokotoNigeria
University of MaiduguriUniversity of MaiduguriNigeria
Madda Walabu UniversityMadda Walabu UniversityEthiopia

Partner countries and regions

ChinaHong Kong, ChinaUnited StatesIndiaEthiopiaSwitzerlandAustraliaMalaysiaNigeria

Citation

@article{zhao2026globaldentbench,
  title={GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration},
  author={Junjie Zhao and Jingyi Liang and Zhenyang Cai and Jiaming Zhang and Zhenwei Wen and Shuzhi Deng and Wenjing Yi and Chunfeng Luo and Hexian Zhang and Junying Chen and Tianrui Liu and Zhuhui Bai and Zixu Zhang and Pradeep Singh and Xiang Liu and Jianquan Li and Nhan L Tran and Falk Schwendicke and Zuolin Jin and Lijian Jin and Liangyi Chen and Wei-fa Yang and Benyou Wang and Junwen Wang and Shan Jiang},
  journal={npj Digital Medicine},
  year={2026},
  note={Accepted},
  url={https://arxiv.org/abs/2605.24636}
}