NeurIPS 2026 | Coécrit par Zhenyang Cai, CTO de FreedomAI : un modèle de fondation convolutif en pathologie qui n'entraîne que 7 millions de paramètres surpasse les grands modèles ViT en segmentation cellulaire

Cet article n'est pas encore disponible en français. La version anglaise est présentée ci-dessous.

FreedomAI2026.11.16Article
NeurIPS 2026 | Coécrit par Zhenyang Cai, CTO de FreedomAI : un modèle de fondation convolutif en pathologie qui n'entraîne que 7 millions de paramètres surpasse les grands modèles ViT en segmentation cellulaire

In computational pathology, nuclear segmentation, inflammatory cell detection and analysis of the tissue microenvironment all depend on "cell-level dense prediction", that is, fine-grained pixel-by-pixel judgments on an image. The difficulty is that pathological structures are often small, varied in shape and blurry at the edges; staining, scanners, tissue types and institutions differ widely; and dense pixel-level annotation is expensive.

Most current pathology foundation models are based on ViT (Vision Transformer). A ViT first cuts an image into fixed-size patches and then encodes them. This works well for high-level tasks such as recognizing tissue types, but it can break up continuous tissue structures and lose the shape, texture and boundary details inside each patch, which is exactly the information cell-level prediction needs most.

To address this, researchers from Tsinghua Shenzhen International Graduate School, The Chinese University of Hong Kong, Shenzhen and other institutions propose CMD (ConvNeXt Masked-Diffusion): a fully convolutional backbone pre-trained with masked diffusion in pixel space, with features from a frozen pathology foundation model injected as a condition. With the backbone frozen and only a lightweight segmentation head trained, CMD's cell-level segmentation outperforms mainstream ViT pathology foundation models overall, with an especially clear advantage when annotations are scarce. The paper has been accepted to NeurIPS 2026 and will be presented in the poster session in Sydney on December 10. FreedomAI CTO Zhenyang Cai is the paper's third author, and FreedomAI Chief AI Scientist Benyou Wang is a co-corresponding author.

Paper: Beyond ViT Tokens: Masked-Diffusion Pretrained Convolutional Pathology Foundation Model for Cell-Level Dense Prediction
Conference: NeurIPS 2026 (Poster)
Link: https://openreview.net/forum?id=u0riwBjIls
arXiv: https://arxiv.org/abs/2605.08276
Code: https://github.com/M4A1TasteGood/ConvNeXt-Masked-Diffusion
Model weights: https://huggingface.co/M4A1TasteGood/ConvNeXt_Masked_Diffusion
Authors: Weiming Chen*, Xitong Ling*, Zhenyang Cai, Xidong Wang, Jiawen Li, Tian Guan, Benyou Wang†, Yonghong He† (* co-first authors, † corresponding authors)
Affiliations: Tsinghua Shenzhen International Graduate School; The Chinese University of Hong Kong, Shenzhen; Institute of Tsinghua University in the Pearl River Delta

Background: the "patching" problem of ViT

Existing pathology foundation models, such as UNI/UNI2, Virchow/Virchow2, Prov-GigaPath and Phikon, mostly use a ViT backbone pre-trained on pathology images with self-supervised objectives such as DINOv2, iBOT or masked image modeling; CONCH, MUSK and others add image–text alignment. They excel at global semantics at the image and slide level, but patch-based input makes it hard to preserve fine-grained local continuity.

Convolutional networks have a natural inductive bias toward locality and spatial continuity, which suits fine-grained pathology representation learning. The problem is that they have lacked a scalable pre-training paradigm comparable to what masked image modeling is for ViT. Inspired by generative models, the authors ask: could generative modeling serve as a self-supervised pre-training scheme for convolutional pathology foundation models?

Method: using a diffusion model as a representation learner

CMD's idea can be summed up as "generative pre-training, then frozen reuse". During pre-training, the model receives partially masked pathology patches and reconstructs the original image through a diffusion denoising objective, learning cell morphology, texture continuity and cell boundaries without any dense annotation. After pre-training, the diffusion sampling process is discarded and the network serves as a frozen dense feature extractor: multi-scale features are taken from the decoder's intermediate layers, and each downstream task trains only a lightweight prediction head.

Figure 1: The CMD framework. (A) Masked-diffusion pre-training on large-scale unlabeled pathology patches, with the timestep embedding and pathology foundation model features jointly modulating a ConvNeXt-UNet; (B) after pre-training the backbone is frozen and multi-scale decoder features feed a lightweight segmentation head; (C) the conditioned ConvNeXt block injects the fused features through adaptive normalization (adaLN) (source: the paper)Figure 1: The CMD framework. (A) Masked-diffusion pre-training on large-scale unlabeled pathology patches, with the timestep embedding and pathology foundation model features jointly modulating a ConvNeXt-UNet; (B) after pre-training the backbone is frozen and multi-scale decoder features feed a lightweight segmentation head; (C) the conditioned ConvNeXt block injects the fused features through adaptive normalization (adaLN) (source: the paper)

Around this framework, the paper answers four design questions in turn. The following ablations were all pre-trained on 55,000 unlabeled pathology images, with frozen features evaluated by linear probing:

  • Why a ConvNeXt-UNet backbone? Comparing DiT, Attention U-Net and ConvNeXt-UNet under the same diffusion setup, the ConvNeXt-UNet learns representations that are more sensitive to boundaries. On CPM-15, for example, its boundary F1 (BF1), which measures boundary agreement, reaches 33.66, against only 19.65 for DiT and 19.31 for Attention U-Net.
  • Why masked diffusion instead of standard DDPM? Borrowing from masked diffusion, the authors turn the diffusion timestep from "controlling Gaussian noise strength" into "controlling the proportion of structure masked", so the model recovers masked tissue structures from the visible regions and learns representations that follow morphology.
  • Pixel space or VAE latent space? VAE compression is efficient but can lose high-frequency details such as nuclear boundaries, chromatin texture and small gaps between cells. In the experiments, pixel-space pre-training was clearly better: on CPM-17, for example, Dice rose from 55.20 in VAE latent space to 76.74.
  • Should a pathology foundation model be used as a condition? With frozen UNI features as semantic guidance, Dice on CPM-15 rose from 55.17 to 63.25. The authors see the foundation model features as a complementary semantic prior, while the dense representations are still learned by the masked-diffusion backbone.

In the main experiments, CMD was pre-trained for 80,000 steps on about 1.05 million unlabeled 512×512 pathology patches (from the public HistAI collection), with the conditioning encoder switched to a frozen H0-mini. The frozen backbone of the CMD-L version has about 600 million parameters (602M), and downstream inputs are uniformly 256×256.

Results

The following figures come from the arXiv version of the paper. A higher Dice means the segmentation agrees more closely with the ground truth.

Frozen backbone versus 16 ViT pathology foundation models

On three cell-level segmentation datasets, CPM-15, CPM-17 and TNBC, the authors compared CMD-L with 16 frozen ViT pathology foundation models and three end-to-end segmentation models: nnUNet, TransUNet and ViT-Adapter.

Table 1: Cell-level segmentation Dice (excerpt from Table 4 of the paper; four representative ViT pathology foundation models are shown, including the best on each dataset; CMD-L uses 256×256 input, the other models 384–512)

Method Type Trainable parameters CPM-15 CPM-17 TNBC
Hibou Frozen ViT pathology foundation model 7.06M 0.794 0.875 0.841
H-Optimus-0 Frozen ViT pathology foundation model 9.42M 0.781 0.881 0.843
Kaiko Frozen ViT pathology foundation model 7.06M 0.758 0.882 0.838
UNI2-h Frozen ViT pathology foundation model 9.42M 0.767 0.873 0.833
nnUNet End-to-end segmentation 12.19M 0.791 0.875 0.803
TransUNet End-to-end segmentation 105.28M 0.809 0.897 0.843
ViT-Adapter End-to-end segmentation 133.52M 0.876 0.872 0.854
CMD-L + lightweight head Frozen convolutional backbone 7.15M 0.864 0.865 0.854
CMD-L + enhanced head Frozen convolutional backbone 21.21M 0.889 0.880 0.854

Key takeaways:

  • Training only the segmentation head in both cases, CMD-L beats the 16 ViT pathology foundation models on most metrics. With a 7.15M-parameter lightweight head, CMD-L's Dice and Precision on CPM-15 and TNBC are higher than those of all 16 frozen ViT pathology foundation models; on CPM-15 its Dice is about 0.07 above the best of them, Hibou (0.864 vs 0.794). On CPM-17, the lightweight-head version has the highest Precision in the table (0.901), but its Dice (0.865) is below several ViT models; with the enhanced head, Dice reaches 0.880, about level with the best, Kaiko (0.882).
  • Competitive with end-to-end segmentation networks too. With a 21.21M-parameter enhanced head, CMD-L achieves the highest Dice in the table on CPM-15 (0.889), ahead of ViT-Adapter (0.876), which trains 133.52M parameters; it ties ViT-Adapter on TNBC and trails TransUNet (0.897) on CPM-17.
  • Less dependent on input resolution. CMD uses 256×256 input throughout, while ViT-type and end-to-end models need dataset-specific resolutions of 384–512. The authors see this as evidence that convolutional masked-diffusion representations capture local morphology more efficiently.

Figure 2: Segmentation results with frozen backbones, on TNBC, CPM-17 and CPM-15 from top to bottom; red boxes mark crowded cells or blurry boundaries, where CMD-L's cell masks are more complete and its boundaries more consistent (source: the paper)Figure 2: Segmentation results with frozen backbones, on TNBC, CPM-17 and CPM-15 from top to bottom; red boxes mark crowded cells or blurry boundaries, where CMD-L's cell masks are more complete and its boundaries more consistent (source: the paper)

Few-shot: the biggest advantage with just one annotated image

In 1-shot, 5-shot and 10-shot settings on CPM-17 and TNBC, the CMD-L backbone stays frozen and only a segmentation head of about 7M parameters is trained; by comparison, ViT-Adapter trains 364M parameters end to end.

Table 2: Few-shot segmentation Dice (Table 5 of the paper; TransUNet uses frozen UNI2-h features)

Setting Frozen UNI2-h Frozen H-Optimus-1 TransUNet ViT-Adapter Frozen CMD-L
Trainable / total parameters 9M / 690M 8M / 1144M 23M / 704M 364M / 364M 7M / 609M
CPM-17 · 1-shot 0.463 0.468 0.479 0.714 0.769
CPM-17 · 5-shot 0.468 0.467 0.477 0.812 0.811
CPM-17 · 10-shot 0.464 0.472 0.476 0.819 0.820
TNBC · 1-shot 0.497 0.486 0.478 0.498 0.714
TNBC · 5-shot 0.493 0.490 0.477 0.800 0.792
TNBC · 10-shot 0.497 0.492 0.476 0.828 0.819

With a single annotated image, CMD-L reaches a Dice of 0.769 on CPM-17 and 0.714 on TNBC, against 0.714 and 0.498 for ViT-Adapter, while both frozen ViT pathology foundation models stay below 0.5. As annotations increase to 5 and 10 images, ViT-Adapter gradually catches up and the two trade the lead on Dice, but CMD-L's Precision is higher in every few-shot setting (for example 0.896 vs 0.832 on CPM-17 10-shot), with only about one-fiftieth of ViT-Adapter's trainable parameters.

Scaling and visualization

A larger backbone performs better: with 70,000 pre-training steps and the segmentation head trained on the full training set, moving from CMD-B (217M parameters) to CMD-L (602M) raises Dice on CPM-15, CPM-17 and TNBC from 0.852, 0.829 and 0.836 to 0.865, 0.863 and 0.850. Longer pre-training also helps overall; for example, Dice on TNBC rises steadily from 0.820 at 10,000 steps to 0.854 at 80,000 steps. The authors note that this is not yet a full scaling-law analysis.

Visualization makes the difference clearer. For the ViT pathology foundation models, the authors chose an anchor patch on a nucleus and computed its cosine similarity with all other patches; the activation often spreads across large areas of tissue. CMD does not use patch tokens, so the authors applied K-means clustering (K=4) directly to its dense convolutional features, and the resulting clusters follow individual nuclei and fine boundaries much more closely.

Figure 3: Visualization of dense representations: for ViT-type pathology foundation models, the cosine similarity between an anchor patch and the other patches; for CMD, K-means clustering of its dense convolutional features (source: the paper)Figure 3: Visualization of dense representations: for ViT-type pathology foundation models, the cosine similarity between an anchor patch and the other patches; for CMD, K-means clustering of its dense convolutional features (source: the paper)

Significance and outlook

The study shows that, in a landscape of pathology foundation models dominated by ViT, a fully convolutional backbone with masked-diffusion pre-training can also be a competitive foundation model for cell-level dense prediction, while better preserving histological structural priors. The "pre-train once, reuse frozen, adapt with few annotations" approach saves both annotations and trainable parameters, which suits pathology tasks where dense annotation is expensive.

The authors plan to explore larger data and backbones, conditioning mechanisms guided by multimodal inputs or pathology reports, and whole-slide image (WSI) workflows that connect cell-level prediction with slide-level diagnosis and tumor microenvironment analysis. CMD's code, the CMD-L pre-trained weights and the TNBC segmentation head are available on GitHub and Hugging Face.

The team

The co-first authors are Weiming Chen and Xitong Ling (Tsinghua Shenzhen International Graduate School), and the corresponding authors are Benyou Wang (The Chinese University of Hong Kong, Shenzhen) and Yonghong He (Tsinghua Shenzhen International Graduate School; Institute of Tsinghua University in the Pearl River Delta). FreedomAI team members Zhenyang Cai and Benyou Wang took part in the research.

Zhenyang Cai | FreedomAI CTO, third author

A PhD student at The Chinese University of Hong Kong, Shenzhen, he leads multimodal medical model development at FreedomAI. He contributed to models including HuatuoGPT-o1 and DentalGPT and has published first-author papers at NeurIPS and ACL. On this paper his affiliation is The Chinese University of Hong Kong, Shenzhen.

Benyou Wang | FreedomAI Chief AI Scientist, co-corresponding author

An assistant professor and doctoral supervisor at the School of Data Science, The Chinese University of Hong Kong, Shenzhen.

#NeurIPS 2026#pathologie computationnelle#modèles de fondation en pathologie#segmentation cellulaire#modèles de diffusion
← Retour aux actualités