A text-only RAG can't read the figures and tables in a paper. I walk through how I built a multimodal RAG system that retrieves and answers from figures and tables too — building a paper-reading AI chatbot from scratch.
How did AI go from handling only text to seeing, hearing, and drawing? I walk through the concept of multimodal AI, its history (CLIP), how inputs and outputs work across modalities, and even the trickiest case — video.
When documents multiply and tables or images enter the mix, basic RAG starts to show its limits. I look at how to improve each stage of the pipeline — indexing (parsing, chunking, embedding), answer generation (query transformation, hybrid search, reranking, citations), agentic RAG, security, and evaluation.
RAG answers questions by retrieving external documents instead of touching the weights. I walk through the indexing and retrieval pipeline, then build a RAG system from a PDF by hand.
Fine-tuning adapts an AI model to your domain. I cover how PEFT, LoRA, and QLoRA cut the memory cost, and when you'd actually reach for them in practice.
A walkthrough of the inner loop — forward pass, loss function, backpropagation, update — with a small Python demo to train a model from scratch.
The hardest part of studying AI was attention and the Transformer. Here I work through them by following the paper 'Attention Is All You Need'.
Four years as a developer, and the age of AI.