Day 6/50: Building a Small Language Model from Scratch

If you've ever looked under the hood of a Transformer model (like GPT or BERT), you may have noticed a concept called positional embedding. It is crucial—in fact, without positional embeddings, your favorite language model wouldn't even know the difference between "The cat sat on the mat" and "The mat sat on the cat."
Let's understand it step by step.

The Core Problem: Transformers Have No Sense of Order

At the heart of most modern language models lies the Transformer architecture, a structure that processes input as a set of tokens rather than a sequence.
Unlike RNNs (Recurrent Neural Networks), which read input word-by-word in order, Transformers look at all tokens at once, in parallel. That's great for speed, but here's the tradeoff:
Transformers lack a built-in understanding of word order.
To a Transformer, "I love AI" is no different than "AI love I."
And that's a huge problem, because meaning depends on order.
So, how do we fix it?

But First… What Is an Embedding?

Before we jump into positional embeddings, let's take a moment to talk about embeddings in general, because they're everywhere in machine learning, especially in NLP.
An embedding is a method for representing discrete data (such as words, tokens, or entire sentences) as dense, continuous vectors in a high-dimensional space.
Why do we need this? Because neural networks don't understand text. They understand numbers. So we take each word and turn it into a vector, one that captures its meaning, context, and relationships to other words.

• For example, in a good embedding space:
  The vector for "king" minus "man" plus "woman" should land close to "queen."
  Words like "Paris" and "France" will be near each other, as will "Tokyo" and "Japan."
  These word embeddings allow models to reason about relationships, analogies, and meaning far beyond raw text.

Now that we understand what embeddings are, let's move on to the question of where each word appears, because the position matters too.

The Fix: Positional Embeddings

To give Transformers a sense of order, we inject some extra information into the input: positional embeddings.
Imagine we're feeding a sentence into the model. Each word is turned into a vector (thanks to word embeddings), but we also need to tell the model:
"This is the first word, this is the second, this is the third…"
That's where positional embeddings come in; they are learnable (or sometimes fixed) vectors that are added to the word embeddings.
This combo of word meaning + word position is what the model uses to understand the sentence.

You can think of it like this:
Final Input = Word Embedding + Positional Embedding

This simple addition gives the model a powerful clue: not just what each word means, but where it appears in the sentence.

Wait, Can't the Model Just Learn Order by Itself?

That's a great question.
In theory, a sufficiently large model could attempt to learn position solely by examining patterns. However, in practice, that's inefficient and prone to error. Positional embeddings serve as a helpful shortcut, providing the model with positional awareness from the outset.
Without them, models are just guessing order, like reading a book with all the pages shuffled.

How Are Positional Embeddings Represented?

There are two main flavors of positional embeddings you'll come across:

1. Sinusoidal Positional Embeddings (Used in the original Transformer paper)
   These are fixed, not learned during training. They use sine and cosine functions at different frequencies to create a unique position vector for each token.
   Why use sinusoids? Because they allow the model to generalize to longer sequences it hasn't seen before. They're elegant and mathematically clever.
2. Learned Positional Embeddings (Used in models like BERT)
   Here, the model learns the position vectors during training, just like it learns word meanings.
   This offers flexibility, but it means the model may struggle slightly with sequences longer than those it saw during training.

Real-World Example: Why They Matter

Let's say we give a model these two sentences:
"The dog chased the cat."
"The cat chased the dog."
They have the same words, but the order completely changes the meaning.
If you remove positional embeddings from the model, it will struggle to distinguish between them. Every word looks the same, and the sequence is a blur.
With positional embeddings, it knows that "dog" came before "chased," and "cat" came after. That tiny change makes a big impact on the model's output.

What About New Techniques?

Recently, there's been a lot of innovation in this space. For example:
Rotary Positional Embeddings (RoPE) - used in models like DeepSeek and LLaMA. These embed positional information directly into the attention mechanism and are particularly suitable for long-context scenarios.
RoPE approaches aim to address certain limitations of traditional positional embeddings, particularly in handling long sequences or cross-lingual understanding.

Final Thoughts

It's easy to overlook positional embeddings when talking about AI models, but they're absolutely essential. Without them, Transformers would be like a GPS with no sense of direction—plenty of information, but no clue where anything goes.
So next time you're working with a model that feels like magic, remember: part of that magic comes from teaching the model not just what words mean, but where they belong.