Teaching Llama 3 to be Polite
Spent ~$3 on A100 (80GB) instance on RunPod
Feb 25, 20268 min read64
Command Palette
Search for a command to run...
#llm
Articles tagged with #llm
I Pretrained a 360M LLaMA-Style Language Model from Scratch on 6B FineWeb Tokens (Single H100)
Pretraining an LLM from scratch usually sounds like “big-lab-only” territory. I wanted to test how far a smaller, practical setup can go while keeping the process transparent and reproducible. This post documents an end-to-end run of training a ~360M...
Feb 8, 20265 min read96I Built a Tiny Vector Database (and Pointed It at FAISS)
Vector DBs are everywhere these days: Pinecone, Weaviate, Qdrant, Chroma, FAISS … you name it! Most of them are full-featured systems with servers, APIs, dashboards, the works. Sometimes the best way to demystify hype is to build it yourself. Here’s ...
Aug 28, 20254 min read80The instability of a softmax function
The softmax, as we know, is numerically unstable when applied to vectors containing very small or very large numbers because of the exponential function involved in its computation. The softmax formula is: \(\text{softmax}(x_{i}) = \frac{e^{x_{i}}}{\...
Mar 5, 20252 min read26