AI Powered MediKit for Healthcare Units

Overview

This project is a multimodal AI system built for classification tasks across medical images and other data types. It combines vision, audio, and text-based components into a single pipeline, enabling flexible and extensible AI-driven applications.

Problem Statement

Medical image classification comes with its own set of challenges. The most important ones are the lack of high-quality labeled data, limited hardware resources (especially in constrained environments like Kaggle), and the need to balance multiple tasks within tight time limits. On top of that, medical datasets often show high variability, where samples from the same class can look very different. Despite these challenges, the goal was to build a system that can still generalize well across different domains.

Model Architecture

At the core of the system is a Vision Transformer (ViT) model used for image classification. The model is pretrained on google/vit-base-patch16-224 and then fine-tuned on domain-specific medical datasets. Cross-entropy loss is used for training, and optimization is handled using the Adam optimizer.

Training Pipeline

All input images are resized to 224×224 and normalized using the ViT feature extractor’s statistics. The dataset is loaded using PyTorch’s ImageFolder utility, with a batch size of 16 and training run for 25 epochs.

Performance Tracking

During training, loss is tracked at every epoch to monitor learning progress. The model is saved after each epoch, and evaluation is performed on a separate test dataset. Accuracy is computed at the end of each epoch to measure performance.

Multimodal System Components

Beyond image classification, the system also supports multiple modalities. For images, it includes both CNN-based and transformer-based approaches, along with few-shot learning modules that allow new categories to be added dynamically. For audio, MFCC features are extracted using Librosa for medical signal classification. For text, SentenceTransformers are used to capture semantic meaning for chatbot and retrieval tasks. The system also includes embedding-based few-shot learning for image classification, enabling flexible expansion of classes.

Utility Modules

To support these capabilities, several utility modules were built, including tools for image browsing and loading, audio playback and control, screen-aware rendering using OpenCV, JSON-based metadata handling, embedding storage and retrieval, a simple chatbot interface, and web integration features.

Domain Coverage

The system has been applied across multiple medical domains, including heart disease, brain tumors, bone fractures, breast cancer, eye diseases, colon diseases, ECG classification, kidney diseases, lung diseases, blood cancer, and oral cancer detection.