{"id":"2074930760155312172","url":"https://x.com/_raghavdixit_/status/2074930760155312172","text":"","author":{"name":"Raghav Dixit","username":"_raghavdixit_","avatarUrl":"https://pbs.twimg.com/profile_images/1977545824662917120/9JRycJ1m_200x200.jpg"},"createdAt":"Wed Jul 08 18:56:58 +0000 2026","engagement":{"replies":23,"retweets":107,"likes":800,"views":478519},"article":{"title":"Vectors are all you need","previewText":"A computer cannot read. It can only do arithmetic. Every chatbot you have ever talked to is, underneath, arithmetic at unthinkable scale. Which raises a fair question: how does multiplication end up","coverImageUrl":"https://pbs.twimg.com/media/HMt3ojZW0AAn7m1.jpg","content":"A computer cannot read. It can only do arithmetic. Every chatbot you have ever talked to is, underneath, arithmetic at unthinkable scale. Which raises a fair question: how does multiplication end up writing your emails?\n\nAfter spending a while behind these systems, the thing I keep coming back to is that the ideas are simple. Simpler than the notation makes them look, simpler than the people explaining them make them sound. You just have to build them in the right order.\n\nSo that is what this series does. This first piece comes in two parts, and Part A is about the single most important fact in all of modern AI, which fits in one sentence:\n\nEverything becomes a vector, and the machine learns by nudging numbers until it stops being wrong.\n\nBy the end of this post you will know exactly what that sentence means, including the actual equations, symbol by symbol. Part B assembles ChatGPT out of these pieces. We focus on language first, but keep one thing in mind as you read: images, audio, video, all of it enters the machine through the same door. That will pay off later.\n\n## Everything becomes numbers\n\nBefore a machine can do anything with the word \"cat,\" someone has to turn \"cat\" into numbers.\n\nThat is the whole game. Honestly. Every AI breakthrough you have heard of is, underneath, a better answer to one question: which numbers should represent this thing, and how do we adjust them?\n\nThe modern answer is called a vector: a list of numbers treated as a point in space. Or, the way I prefer to picture it, an arrow from the origin out to that point. The word \"cat\" might be 768 numbers (the exact count is a design choice; more numbers hold more nuance). So is \"dog.\" So is \"helicopter.\"\n\nAnd here is the part that matters. The numbers are chosen so that similar meanings end up as nearby arrows. \"Cat\" and \"dog\" point in almost the same direction. \"Helicopter\" points somewhere else entirely. Meaning becomes geometry.\n\nThese arrows are called embeddings, and they are not hand-built. Nobody sat down and decided cat's 768 numbers. They are learned, from a simple pressure: words that show up in similar sentences get pushed toward similar directions. Show the machine enough of the internet and \"cat\" and \"dog\" drift together on their own, because they both chase things and knock stuff off tables in a million sentences.\n\nThe famous demo of this came out of Google in 2013, from a system called word2vec. You can do arithmetic on meaning:\n\nking − man + woman ≈ queen\n\nTake the arrow for king, subtract the arrow for man, add the arrow for woman, and the nearest word to where you land is queen. The step from \"man\" to \"woman\" is a consistent direction in the space, and you can walk it from anywhere. Nobody programmed that. It fell out of the numbers. The first time you see it work it is genuinely a little unsettling.\n\nThis is not a toy, and it is not just words. I once shipped a system that had to follow the same person across a building full of cameras. Different angles, different lighting, half the time you only see their back. The way it works is exactly this: every sighting of a person becomes a vector, and if two sightings produce arrows pointing nearly the same way, it is the same person. The system never knows a name or a face the way you do. It compares arrows. That one idea runs the same whether the input is a word, a face, or a sound, which is why I keep saying the modalities all enter through the same door.\n\n## The dot product, the one operation to rule them all\n\nIf meaning is direction, we need a way to ask: how aligned are two arrows? That operation is the dot product, and it is worth two minutes, because it is the single most-used operation in all of AI. When you hear that GPUs matter for AI, this is mostly what the GPUs are doing, billions of times per second.\n\nTwo vectors a and b. The dot product is:\n\na · b = Σ aᵢ bᵢ = |a| |b| cos(θ)\n\nLeft side first. Σ is just the instruction \"add up.\" aᵢ means the i-th number in the list a. So Σ aᵢbᵢ says: multiply the two lists together, position by position, and total it. That is the entire computation. Multiply and add.\n\nThe right side is why we bother. That same total happens to equal the lengths of the two arrows times cos(θ), where θ is the angle between them. And cosine is just a dial that reads 1 when two arrows point the same way, 0 when they are perpendicular, and −1 when they oppose.\n\nSo one cheap computation, multiply and add, secretly measures agreement:\n\n- Arrows aligned: big positive number. Similar meaning.\n\n- Arrows unrelated: near zero.\n\n- Arrows opposed: negative.\n\nThe dot product is a similarity score. Cat · dog is large, Cat · helicopter is near zero. Hold onto this one tightly, because in Part B, the \"attention\" inside a transformer, the mechanism everyone treats as deep magic, is this exact operation asked at scale.\n\n## The machine that reshapes space\n\nSo words are arrows. Now we need the machine that does something with them. It is called a neural network, and I want to build it up honestly, because it is the LEGO brick that transformers are assembled from. If you get this, Part B is easy.\n\nStart with one neuron, which is a small formula, not a brain cell:\n\nh = f(w · x + b)\n\nFour symbols, four jobs:\n\n- x is the input. A vector. Say, the embedding of a word.\n\n- w is the weights. Another vector, one weight per input number, encoding how much this neuron cares about each one. Notice w · x is our dot product again: this neuron is scoring how much the input matches the pattern it is tuned for.\n\n- b is the bias, one constant that shifts the score up or down. It sets the neuron's threshold, how easily it activates at all.\n\n- f is the activation function, a small nonlinear squiggle applied at the end. It gets its own section next, because the story of which squiggle to use nearly killed deep learning.\n\nOne neuron computes one score. Now stack a whole row of them, each with its own weights, and feed the same input to all of them. Write the whole row at once and you get:\n\nh = f(Wx + b)\n\nSame shape, but W is now a matrix, all the weight vectors stacked into a grid, and h is a vector of all the scores. Then, and this is the move, feed that output in as the input to another layer. And another. That is a multilayer perceptron, an MLP.\n\nHere is the picture I want in your head. Each layer takes the whole space of possible inputs and reshapes it. Stretches it, rotates it, folds it. Things that were hopelessly tangled in the original space, like sarcastic reviews mixed in with sincere ones, become cleanly separable after enough reshaping. Depth is not magic. Depth is folding space more times.\n\nTwo things to remember before we move on. First, the network's entire knowledge lives in W and b, billions of plain numbers. There is no other place for knowledge to hide. Second, when you hear \"the MLP block inside a transformer\" in Part B, it is literally this, two of these layers, sitting inside every block of GPT. You now understand a real chunk of ChatGPT's anatomy.\n\n## how one function choice froze a field\n\nI owe you the story of \"f\". It sounds like a footnote. It decided the fate of deep learning for about a decade.\n\nFirst, why does \"f\" exist at all? Because without it, stacking layers is pointless. Wx + b is a linear operation, and a stack of linear operations collapses into one linear operation. A thousand layers, no squiggle, equals one layer. All that folding of space I described? The squiggle is what makes folding possible. Remove it and every layer just stretches and rotates the same flat sheet.\n\nSo which squiggle? Walk through the museum with me.\n\nThe sigmoid was the classic choice:\n\nσ(x) = 1 / (1 + e^(−x))\n\nRead it as: squash any number into the range 0 to 1 along a smooth S-curve. Very negative inputs land near 0, very positive land near 1. It was smooth, it was differentiable everywhere, and it looked satisfyingly like a neuron switching from off to on. Everyone used it.\n\nAnd it quietly strangled deep networks. Look at the S-curve far from the center: it goes flat. Flat means the derivative, the slope, is nearly zero out there. Why does that matter? Because, as you are about to see, learning works by passing slopes backward through the layers, multiplying them as it goes. Chain ten layers of sigmoid together and you are multiplying ten small numbers: the signal shrinks toward zero before it reaches the early layers. The front of the network simply stops learning. This is the vanishing gradient problem, and it is a big part of why neural networks spent the 1990s and 2000s written off as a dead end.\n\nThe fix, popularized around 2010, is almost insulting:\n\nReLU(x) = max(0, x)\n\nThat is the whole function. Negative input, output zero. Positive input, pass it through untouched. For any positive input the slope is exactly 1, so the learning signal flows backward through a hundred layers without shrinking. It is also nearly free to compute. This one-line swap is a real part of why deep learning suddenly worked in the 2010s. I love this story because the lesson generalizes: sometimes the breakthrough is not adding cleverness, it is removing the thing that was quietly killing you.\n\nThe museum has a modern wing. Today's frontier models use smoother relatives of ReLU: GELU in GPT-class models, which is ReLU with a soft, probabilistic bend near zero instead of a hard corner, and SwiGLU in Llama-class models, which adds a learned gate deciding how much signal passes. The differences are real but incremental, refinements on ReLU's insight rather than revolutions. The point of the museum is not the exhibits. It is that a function you had never heard of ten minutes ago was load-bearing for the entire field, and that its selection was driven by one question: what happens to the slopes?\n\n## What \"wrong\" means: the loss function\n\nWe have a machine full of adjustable numbers. It starts stupid: W and b are random, so its predictions are noise. Learning means adjusting them. But adjusting toward what?\n\nBefore a machine can get better, someone has to define what \"wrong\" means as a single number. That definition is called the loss function, and for language models the standard choice is cross-entropy. Here it is, and then we will take it apart:\n\nL = − log P(correct next word)\n\nThe setup: the model reads some text and outputs a probability for every word that could come next. One of those candidate words is what actually came next in the training text. The loss is just: take the probability the model gave to that correct word, take its log, flip the sign.\n\nWhy the log? Feel the shape of it:\n\n- Model says the right word had probability 1.0, total confidence, dead right: log(1) = 0. Zero loss. No punishment.\n\n- Probability 0.5: loss ≈ 0.69. Mild sting.\n\n- Probability 0.01, the model was confidently wrong: loss ≈ 4.6. Heavy.\n\n- Probability approaching 0: the loss blows up toward infinity.\n\nThe log turns confident wrongness into catastrophe. A model that hedges honestly gets punished gently. A model that is sure and mistaken gets hammered. Every training step, this pressure pushes probability toward the truth.\n\nOne more layer down, because this is where it gets genuinely beautiful. Minimizing this loss is not an arbitrary recipe. Statisticians have asked for a century: given data, how do you pick the model parameters that best explain it?\n\nTheir answer, maximum likelihood, says: choose the parameters under which the observed data was most probable. Work out the algebra and maximizing the probability of the data is identical to minimizing our cross-entropy loss. Same math, two vocabularies. So when GPT trains, it is doing the most classical thing in statistics: finding the parameters under which the actual internet was the most likely internet. The model is not memorizing sentences. It is fitting a probability distribution over language.\n\nThat is also the honest frame for what a language model is, and it will explain a lot in Part B, including why models sound confident when they are wrong. Wrong and right answers come out of the same distribution, wearing the same fluency.\n\n## Learning: roll downhill\n\nNow the finale of Part A. We have a number that measures wrongness. Learning is: make that number go down.\n\nPicture a landscape. Every possible setting of the weights θ is a location. The altitude at each location is the loss. Somewhere in this landscape, altitude is lowest: the weights that make the fewest, mildest mistakes. Training is finding your way there. The catch: the landscape has billions of dimensions and you are standing in fog. You cannot see the valley. You can only feel the ground under your feet.\n\nSo do the obvious thing. Feel which way is downhill. Step that way. Repeat.\n\n\"Which way is downhill\" has a name: the gradient, written ∇L. It is the collection of slopes of the loss with respect to every single weight at once, computed by an algorithm called backpropagation, which for all its reputation is the chain rule from calculus with immaculate bookkeeping. The full update rule, the one line the entire modern world runs on, is:\n\nθ ← θ − η ∇L(θ)\n\nSymbol by symbol, one last time:\n\n- θ (theta): all the weights and biases, billions of numbers, your location in the landscape.\n\n- ← : update. Replace the old weights with what follows.\n\n- ∇L(θ) (the gradient): the direction of steepest uphill, and how steep.\n\n- − : the most important minus sign in computer science. Uphill makes loss worse, so go the other way.\n\n- η (eta, the learning rate): how big a step to take. Too large and you overshoot the valley and ricochet. Too small and training takes geologic time. I have lost more weekends to this one number than I want to admit. Watch the animation again: the steps shrink as the ground flattens, because the step size scales with the slope. The ball brakes into the valley on its own.\n\nRun that line trillions of times over trillions of words and random weights become GPT. That is training. There is no second mechanism hiding behind the curtain.\n\nOne quick mirror image before we close, because you will meet it again. Flip the minus to a plus and you get gradient ascent: step uphill, used when the number measures something you want more of, like a reward, rather than an error. Descent minimizes pain, ascent maximizes gain, same machinery pointed in opposite directions. Remember this flip. In Part B it turns a next-word guesser into a helpful assistant.\n\n## Where we are, and the crack in the wall\n\nLook at what you now hold, because it is most of the machine:\n\n- Words become vectors, and meaning becomes geometry.\n\n- The dot product measures similarity between meanings.\n\n- An MLP reshapes space with f(Wx + b), its knowledge living entirely in the weights.\n\n- Cross-entropy defines wrong, and it is maximum likelihood in disguise.\n\n- Gradient descent rolls the weights downhill until the machine stops being wrong.\n\nAnd three things you can actually use this week:\n\n1. When an AI answer sounds confident, remember where it came from. The model is a probability distribution, and right and wrong answers wear the same fluency. Confidence is a style, not a signal. Verify anything that matters.\n\n1. \"Semantic search\" is dot products. Every RAG system and \"chat with your docs\" tool retrieves by comparing embedding arrows. When retrieval misses, it is because the arrows did not align, so rephrase your query to say what you mean, not the keywords you remember.\n\n1. Embeddings are cheap and you can use them today. Every major AI provider sells them for fractions of a cent. \"Find me things like this one\" across your notes, tickets, or products is one dot product away, no model training required. (Later in this series: how modern embedding models actually get trained, and we build semantic search from scratch together)\n\nBut there is a crack in the foundation, and it is the cliffhanger.\n\nEverything above treats a word as having one arrow. Consider: \"I sat on the river bank\" and \"I withdrew cash from the bank.\" Same word. Same arrow. Completely different meanings. A fixed embedding decided what \"bank\" means before it ever saw your sentence. And an MLP eats its input as one fixed lump. It has no mechanism for a word to look around at its neighbors and adjust.\n\nWhat language actually needs is a machine where the arrow for \"bank\" gets bent by context, pulled toward riversides in one sentence and toward vaults in another. A machine where every word gets to ask every other word: are you relevant to what I mean right now?\n\nYou already know the operation that scores that kind of relevance. It is the dot product. In 2017, a team at Google worked out how to build an entire architecture around asking that question, everywhere, in parallel, and called the mechanism attention.\n\nPart B builds it: the transformer, GPT piece by piece, how a raw model becomes an assistant, how images sneak in through the same vector door, and the new architectures trying to dethrone attention itself. \nSubscribe so you catch it!"},"adhxContext":{"savedByCount":1,"publicTags":[],"previewUrl":"https://adhx.com/_raghavdixit_/status/2074930760155312172"}}