Research journey | Five generations of CUHK-Shenzhen's HuatuoGPT series: from medical dialogue to RL-only adaptation, with a 27B model scoring 70.1 on HealthBench

FreedomAI2026.11.16Technology
Research journey | Five generations of CUHK-Shenzhen's HuatuoGPT series: from medical dialogue to RL-only adaptation, with a 27B model scoring 70.1 on HealthBench

Turning a general-purpose large model into a reliable medical assistant means clearing several hurdles: it has to talk like a doctor, master solid medical knowledge, understand medical images, and reason step by step through complex problems.

Since 2023, researchers at The Chinese University of Hong Kong, Shenzhen and their collaborators have pursued medical large-model research through the HuatuoGPT series: from professional medical dialogue, medical knowledge adaptation and medical image–text understanding, all the way to complex medical reasoning and domain adaptation with reinforcement learning alone, building datasets such as Huatuo-26M and PubMedVision and the CMB benchmark along the way. Several FreedomAI core team members took part in many of these projects. This line of research also provides the methodological foundation in data construction, model training and professional evaluation for DentalGPT, FreedomAI's multimodal oral-health large model.

First released Work Focus Published in
May 2023 HuatuoGPT Professional medical dialogue Findings of EMNLP 2023
November 2023 HuatuoGPT-II Medical knowledge adaptation COLM 2024
June 2024 HuatuoGPT-Vision Medical image–text understanding EMNLP 2024
December 2024 HuatuoGPT-o1 Complex medical reasoning Findings of ACL 2025
October 2026 HuatuoGPT-3 Domain adaptation with reinforcement learning arXiv preprint (2026)

Table: Timeline of the HuatuoGPT series. Dates are when each paper was first made public; the last column gives the conference or preprint platform where it was published.

2023: from answering questions to communicating like a doctor

HuatuoGPT: combining real doctors' data with distilled data

HuatuoGPT is a large model for medical consultation. Its core idea is to use two kinds of data in supervised fine-tuning: data distilled from ChatGPT, with detailed and well-formed answers, and real data from doctors, closer to how doctors actually take histories and make diagnoses. The team then trained a reward model and used RLAIF (Reinforcement Learning from AI Feedback) to align the strengths of both kinds of data in the model.

The paper reports that HuatuoGPT achieved the best medical consultation performance among open-source models at the time on GPT-4 evaluation, human evaluation and medical benchmarks, and in most cases outperformed its "teacher", ChatGPT. It was published in Findings of EMNLP 2023.

HuatuoGPT-II: medical adaptation in a single stage

The usual way to turn a general model into a medical expert is a two-step process of "continued pre-training + supervised fine-tuning", but the data in the two stages differs greatly in language, genre and format. HuatuoGPT-II converts the heterogeneous data from both stages into simple "input–output" pairs and completes medical domain adaptation in a single stage.

The team validated this approach in fields where ChatGPT was relatively weak, such as traditional Chinese medicine: HuatuoGPT-II led on several Chinese medical benchmarks, including medical licensing exams, and achieved the best result on a brand-new national medical licensing exam. It was published at COLM 2024.

Data and evaluation: Huatuo-26M and CMB

In the same period, the team released Huatuo-26M, a Chinese medical question-answering dataset of about 26 million question–answer pairs. It can be used to evaluate retrieval and generation, as external knowledge for retrieval-augmented generation, and for continued training (the paper was later published in Findings of NAACL 2025).

On evaluation, the team proposed CMB (Comprehensive Medical Benchmark in Chinese). The authors argue that directly translating English medical benchmarks fits poorly with the local healthcare context, so CMB is designed entirely around the Chinese language and cultural setting; traditional Chinese medicine is an important part of it, but not the whole. It was published at NAACL 2024.

2024: understanding images and learning to reason

HuatuoGPT-Vision: 1.3 million samples of medical visual knowledge

Constrained by privacy and annotation costs, high-quality medical image–text data has long been scarce. The team refined medical image–text pairs from PubMed and had GPT-4V, which can see the images, denoise and reorganize the data, building PubMedVision with 1.3 million medical visual question-answering samples.

Experiments show that PubMedVision significantly improves the medical abilities of existing multimodal models, including on the Health & Medicine track of MMMU, and manual checks by medical experts confirmed the data quality. HuatuoGPT-Vision, a 34B medical multimodal model trained on this data, outperformed other open-source models in medical multimodal scenarios. It was published at EMNLP 2024.

HuatuoGPT-o1: training complex reasoning with verifiable problems

Math problems can be checked against an answer; medical reasoning is hard to verify. HuatuoGPT-o1 rewrites challenging multiple-choice questions from medical exams into open-ended "verifiable medical problems" with a single answer that can be checked objectively, and introduces a medical verifier to check whether the model's output is correct.

Training has two steps (Figure 1): first, guided by the verifier, the model searches for correct complex reasoning trajectories, which are used for fine-tuning; then reinforcement learning (PPO) with rewards from the verifier further improves complex reasoning. The paper shows that with only 40,000 verifiable medical problems, HuatuoGPT-o1 outperforms general and medical open-source baselines; complex reasoning improves medical problem solving and benefits more from reinforcement learning. It was published in Findings of ACL 2025 as Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems.

Figure 1: HuatuoGPT-o1's two-stage training. Left: searching for complex reasoning trajectories guided by the verifier and using them for fine-tuning; right: reinforcement learning with verifier rewards. Source: the paperFigure 1: HuatuoGPT-o1's two-stage training. Left: searching for complex reasoning trajectories guided by the verifier and using them for fine-tuning; right: reinforcement learning with verifier rewards. Source: the paper

2026: HuatuoGPT-3, adapting with reinforcement learning and no supervised fine-tuning

"Supervised fine-tuning (SFT) first, then reinforcement learning (RL)" is the mainstream pipeline for domain adaptation. It makes cold starts easy but can reduce the model's exploratory diversity, and multi-stage optimization adds complexity. Pure on-policy reinforcement learning, on the other hand, struggles with cold starts. Mixed-policy reinforcement learning, which mixes a teacher model's outputs into training, has two problems of its own: early in training, valuable tokens in the teacher's outputs are learned too slowly, which the authors call "Gradient Starvation"; late in training, outdated teacher outputs pull the model backwards, which they call "Teacher-Distribution Anchoring".

To solve this, the team proposes OnePO (One-stage Policy Optimization), which treats teacher outputs as a "temporary guide" (Figure 2):

  • Adaptive Objective Evolution: strengthens the learning signal for informative teacher tokens to which the current model assigns low probability, easing gradient starvation;
  • Teacher Retirement: a teacher output is kept in training only when its reward is strictly higher than the best reward among the current model's outputs in the same group; once the model catches up, the teacher output automatically "retires" and training returns to standard on-policy reinforcement learning.

Figure 2: Overview of OnePO. Early in training, teacher outputs speed up the learning of domain knowledge; the model then gradually returns to exploring on its own. Source: the paperFigure 2: Overview of OnePO. Early in training, teacher outputs speed up the learning of domain knowledge; the model then gradually returns to exploring on its own. Source: the paper

In the medical domain, OnePO reaches 67.2 on HealthBench (Total) with only 20,000 training samples, 2.7 points above SFT+RL and 7.4 points above pure RL. With more training data, the team trained the open-source medical model series HuatuoGPT-3, in 9B and 27B versions based on Qwen3.5-9B and Qwen3.8-27B. The 27B version scores 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, above the 57.8 and 62.5 of GPT-6 Astra (high), the comparison model in the paper (Figure 3).

Figure 3: HealthBench Professional scores versus model size. Source: the paperFigure 3: HealthBench Professional scores versus model size. Source: the paper

The HuatuoGPT-3 technical report was released on arXiv in October 2026 as an extended version of the ICML 2026 paper on OnePO, and the models and code are open source.

From HuatuoGPT to DentalGPT

Dentistry is a field that relies heavily on imaging and requires highly specialized knowledge, and the methods developed in the HuatuoGPT series carry over to it:

  • How reasoning data is built: DentalGPT builds its complex reasoning (chain-of-thought) data directly with the HuatuoGPT-o1 method.
  • From general medical data to specialist data: the DentalGPT preprint found that images involving teeth make up only about 0.3% of PubMedVision, so the team built a dedicated dataset of more than 120,000 dental images.
  • Specialist evaluation: in DentalGPT's comparative experiments, the medical multimodal model HuatuoGPT-Vision averages 47.0% accuracy on five dental benchmarks, while DentalGPT, a 7B model trained on dental data with reinforcement learning, reaches 67.1%. The DentalGPT paper concludes that domain-specific data and training strategies are essential for dental AI.
  • Reinforcement learning: from PPO in HuatuoGPT-o1 to GRPO in DentalGPT and OnePO in HuatuoGPT-3, reinforcement learning runs through the second half of this line of research.

Team and impact

These results come from researchers at The Chinese University of Hong Kong, Shenzhen and their collaborators, with FreedomAI core team members taking part in many of them. FreedomAI Chief AI Scientist Benyou Wang is one of the authors of the papers above; FreedomAI CTO Zhenyang Cai is the second author of HuatuoGPT-o1 and one of the authors of HuatuoGPT-Vision. The first author of HuatuoGPT-II, HuatuoGPT-Vision, HuatuoGPT-o1 and HuatuoGPT-3 is Junying Chen.

Papers and resources

#HuatuoGPT#medical large models#reinforcement learning#medical multimodal AI
← Back to News