Google's MedGemma 1.5 Brings 3D CT and MRI Analysis to Open-Source Medical AI

Google has upgraded its open-source medical AI model MedGemma 1.5, making it the first publicly available language model capable of interpreting 3D CT and MRI images.

According to Google research, the healthcare industry is adopting generative AI at roughly twice the rate of the overall economy. To support this shift, the company launched MedGemma 1.5 4B, an enhanced version of its open-source medical image interpretation model, alongside MedASR, a new speech recognition model specifically designed for medical dictation.

Google reports that since its launch last year, the original MedGemma has been downloaded millions of times, spawning hundreds of community versions on Hugging Face.

This update signifies a major expansion in capabilities. The previous version was limited to 2D inputs like X-rays or skin images, whereas MedGemma 1.5 can process 3D volumetric data from CT and MRI scans, as well as histopathology slides. This allows developers to input an entire CT scan volume into the model at once, rather than processing it slice by slice.

This approach also applies to histopathology, where the model analyzes multiple slices of a tissue sample simultaneously. By viewing these slices together, the AI can identify correlations that might be missed in individual slices. Google claims MedGemma 1.5 is the first open-source model to handle such 3D medical data while retaining its understanding of standard images and text.

Significant Improvement in Diagnostic Accuracy

Internal benchmarks show notable performance gains over the previous version. CT classification accuracy increased by 3 percentage points to 61%, while MRI classification rates jumped 14 percentage points to nearly 65%.

The model also shows progress on text-based tasks. On the MedQA medical reasoning benchmark, MedGemma 1.5 4B scored 69%, up from 64% in the prior version. The most significant improvement was in extracting information from electronic health records, where accuracy surged from 68% to 90%.

Despite these advances, Google emphasizes the technology remains in its early stages. The company notes the model is not yet complete, but developers may achieve better results by fine-tuning it on their own specific datasets.

Reducing Transcription Errors in Clinical Workflows

Google also introduced MedASR, a speech recognition tool specifically trained on medical vocabulary. According to Google, MedASR reduces errors in X-ray dictation by 58% and in general medical dictation by 82%, outperforming OpenAI's general-purpose Whisper Large v3 model.

This model has a dual function: serving as a transcription tool and as a voice interface for MedGemma, potentially enabling doctors to interact with the AI system entirely through speech.

Early adopters are already testing this technology. Malaysian company Qmed Asia uses MedGemma to provide a conversational interface for clinical treatment guidelines, while Taiwan's National Health Insurance Administration utilized the model to analyze over 30,000 pathology reports related to lung cancer surgeries.

Both models are available free for research and commercial use via Hugging Face and Google Cloud Vertex AI. However, Google stresses these models are starting points for developers, not finished clinical products. They require validation and customization, and their outputs should not be used for direct diagnosis or treatment.

Regulatory Hurdles for Patient Care

Although MedGemma is open-source, it falls under the "Health AI Developer Foundation Terms of Use" license. This license imposes restrictions on the standard Apache 2.0 license used for the source code. Specifically, using the model weights for direct patient diagnosis or treatment requires approval from relevant authorities as a medical device.

Developers building on MedGemma must pass these restrictions down to third parties. Google explicitly disclaims any liability and does not offer medical advice through the model. While the company does not claim copyright over the model's output, users bear full responsibility for how they use that output.

AI Labs Vie for Leadership in Medical AI

Google's launch comes amid intensifying competition in the sector. OpenAI recently acquired startup Torch for approximately $100 million to build an "AI medical memory" and has launched specialized ChatGPT health features, alongside new AI services for healthcare providers.

Anthropic has also introduced Claude for Healthcare, a HIPAA-compliant solution capable of accessing U.S. databases like Medicare and PubMed. These moves reflect the industry's confidence in a vast market opportunity, driven by the pressure from hundreds of millions of health-related queries already posed to chatbots by users worldwide.