Deep Learning

Fewshot Classification of Documents with High Intraclass Variance

Document understanding using ResNet-50 and DiT with embedding-based few-shot learning (triplet loss), achieving strong similarity search performance, followed by insights on classification–few-shot overlap and multimodal fusion of ResNet visual features with BERT embeddings for improved generalization.

ComputerVisionDeepLearningImageProcessingNeuralNetworkPyTorchPythonResNETTripletLossViT

Fewshot Classification of Documents with High Intraclass Variance

Problem Statement

The task is to classify document images. Unlike standard image classification, document images have high intra-class variance, where samples within the same class can look very different. The system also needs to support few-shot learning, where new categories should form new clusters without affecting existing ones.

Initial Approach

Since this was my first project intended for internal production at Zoho, I started with a simple CNN and experimented with MNIST-style datasets, achieving good baseline accuracy.

Model Evolution

I then explored ResNet-50 on natural images and later transitioned to DiT, a vision transformer pretrained on document data, to better capture document-specific representations.

Few-Shot Learning

Document embeddings were extracted and trained using Triplet Loss. With well-labeled data, the system achieved up to 100% accuracy in similarity search.

Key Observation

During experiments, I observed that classification models can sometimes behave like few-shot models, and few-shot models can also resemble classifiers depending on the setup.

Multimodal Learning

Finally, I experimented with multimodal fusion by combining visual features from ResNet and textual embeddings from BERT, which improved generalization across document types.