MedSage 🏥 | Memory-First Medical AI with Qdrant

MedSage is a retrieval-first medical AI platform that combines live clinical consultation, medical document Q&A, and multimodal file intelligence. Built on Qdrant vector database and FastEmbed, MedSage prioritizes semantic memory over raw model capacity, delivering context-aware medical reasoning through advanced RAG (Retrieval-Augmented Generation) pipelines.

"Memory over models — real retrieval, real context, and real medical workflows."

This project was built for the Memory Over Models — AI Hackathon (HiDevs × Qdrant × Lamatic × AI Collective), addressing both the Unstructured Data RAG Challenge and Domain-Specific AI Systems (Healthcare) themes.

---

🎯 Project Overview

MedSage delivers three core capabilities:

• Live Medical Consultation – Symptom-based clinical decision support with real-time diagnosis, reasoning, and emergency detection
• Medical Document Q&A – Upload and query PDFs, DOCX, and medical images using semantic search over Qdrant-stored vectors
• Advanced File Intelligence – Multimodal processing for PPTX, XLSX, CSV, JSON, and audio files with OCR and transcription

Architecture Philosophy

• Retrieval-first design – Qdrant vector database serves as the primary knowledge source, not the LLM
• Dual-mode Qdrant – Cloud instances for persistence + in-memory instances for speed
• Unified embedding space – All modalities (text, OCR, audio transcripts) share the same 384-dimensional vector space

---

🏆 Hackathon Context

Event: Memory Over Models — AI Hackathon Organizers: HiDevs (GenAI workforce) × Qdrant (vector DB) × Lamatic (AI automation) × AI Collective (global AI community)

How MedSage Demonstrates the Themes

1. Unstructured Data RAG – Converts messy medical PDFs, scanned reports, handwritten notes (via OCR), and audio dictations into a queryable vector database
2. Domain-Specific Healthcare AI – Medical keyword validation, graduated retrieval strategies, emergency detection, and structured clinical reasoning
3. Memory-First Architecture – Qdrant handles semantic search and context retrieval; LLMs generate answers only after retrieval, not from scratch

---

MedSage UI Screenshots

| Home Page | Features | Language Selector | |--------------|--------------|------------------------| | <img width="300" src="https://github.com/user-attachments/assets/a745d0d4-2199-4cee-a550-4c3fc3472c6e" /> | <img width="300" src="https://github.com/user-attachments/assets/83cf0d3d-79da-4e15-bf4a-2d9f35d20edf" /> | <img width="250" src="https://github.com/user-attachments/assets/7e4d4e41-3e5a-4190-a170-7a01b737e3cd" /> |

| Language Page | Processing Document | Medical File Chat (In-Memory) | |-------------------|-------------------------|----------------------------------| | <img width="300" src="https://github.com/user-attachments/assets/8975ce7d-cdf1-423e-88ea-e41052508733" /> | <img width="300" src="https://github.com/user-attachments/assets/3159c913-1b13-47ee-92e6-cedb096f5469" /> | <img width="300" src="https://github.com/user-attachments/assets/21ee5225-61ab-4b01-9404-670ad5c73b9b" /> |

| Medical Reasoning (Cloud) | Universal File RAG Upload | Technologies Used | |-------------------------------|-------------------------------|------------------------| | <img width="300" src="https://github.com/user-attachments/assets/895293f5-c9e2-46ec-bd3a-00690ad9d9df" /> | <img width="300" src="https://github.com/user-attachments/assets/a95236ee-ca19-49f9-a9de-44427eb60d30" /> | <img width="300" src="https://github.com/user-attachments/assets/f17bd8cb-09c3-482e-bfe5-8b708e031a5a" /> |

| RAG Pipeline | How MedSage Works | Emergency SOS | |------------------|------------------------|-------------------| | <img width="300" src="https://github.com/user-attachments/assets/cea2e521-0377-473a-bdf2-3e9c341d44de" /> | <img width="300" src="https://github.com/user-attachments/assets/c79959f4-e5e9-4928-8eaf-1437bcb01a49" /> | <img width="300" src="https://github.com/user-attachments/assets/bb4f61a9-6497-4b2a-b708-b9188242a381" /> |

🧠 Qdrant-Powered Memory Architecture

Qdrant Cloud (Primary Vector DB)

• Persistent storage for production-grade medical document vectors

client = QdrantClient(url=Qdrant_URL, api_key = Qdrant_API_KEY)

• Powers semantic search in MedFileChatbot and FileChatPro
• Stores rich metadata: file IDs, page numbers, modality type, retrieval scores

Qdrant In-Memory (:memory)

• Fast, ephemeral sessions for interactive experimentation

CLIENT = QdrantClient(":memory")  # Rapid Prototyping

• Used in MedFileChatbot.tsx for quick file indexing and Q&A
• Used in FileChatPro.tsx for high-speed local vector search
• Enables per-session vector stores that reset cleanly

FastEmbed Integration

MedSage uses Qdrant's FastEmbed library with the BAAI/bge-small-en-v1.5 model (384-dimensional vectors). Benefits:

• Native Qdrant Embeddings: Optimized for vector operations
• Model: BAAI/bge-small-en-v1.5 generating 384-dimensional vectors
• Performance: Sub-50ms embedding generation
• Language: Optimized for English medical text
• Lightweight and fast (ONNX-based)
• Optimized for semantic search in Qdrant
• Consistent embedding space across all modalities

Example: Embedding & Upserting to Qdrant

from fastembed import TextEmbedding
from qdrant_client.models import PointStruct

# Initialize embedding model
EMBEDDING = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Generate embeddings
chunks = ["Patient presents with fever...", "Blood pressure 120/80..."]
vectors = [list(EMBEDDING.embed([chunk]))[0].tolist() for chunk in chunks]

# Upsert to Qdrant
points = [
    PointStruct(
        id=str(uuid.uuid4()),
        vector=vec,
        payload={"text": chunk, "file": "report.pdf"}
    )
    for vec, chunk in zip(vectors, chunks)
]
CLIENT.upsert(collection_name="Health_QA_CoT", points=points)

---

📄 Multimodal Content & Embeddings

Text Documents (PDF, DOCX, TXT)

• Extract text using PyPDF2, python-docx, or plain text readers
• Chunk into ~350-word segments with ~300-word overlap for context preservation
• Embed with FastEmbed (BAAI/bge-small-en-v1.5) and store in Qdrant with metadata

Images (PNG, JPG, JPEG, WEBP)

• Use Gemini Vision for OCR to extract medical text from scans, charts, and handwritten notes
• Embed extracted text using the same FastEmbed model to maintain a unified vector space

Audio (MP3, WAV, M4A, OGG)

• Perform speech-to-text transcription for medical dictations and audio notes
• Chunk transcripts and embed with FastEmbed
• Attach timestamps and file IDs in Qdrant payload for precise source attribution

Result: All modalities land in Qdrant with rich metadata, enabling unified semantic retrieval across text, images, and audio.

---

📋 Application Pages Overview

| Page | Location | Description |
| ------------------ | --------------------------------------- | ---------------------------------------------------------------------------------------------- | | MedChatbot | Frontend/src/pages/MedChatbot.tsx | Live clinical consultation with symptom analysis, voice I/O, and emergency detection |
| MedFileChatbot | Frontend/src/pages/MedFileChatbot.tsx | Medical document upload and Q&A over PDFs, DOCX, and images using Qdrant retrieval |
| FileChatPro | Frontend/src/pages/FileChatPro.tsx | Advanced file intelligence over PPTX, XLSX, CSV, JSON, and audio with multimodal vector search |

🏥 MedChatbot – Live Clinical Consultation (Uses Qdrant Cloud DB)

Location: Frontend/src/pages/MedChatbot.tsx

Core Features

• Real-time Symptom Analysis – Users describe symptoms; system returns structured diagnosis, reasoning, and recommendations
• Voice Integration – Uses webkitSpeechRecognition