Nlp

In this post, I aim to provide an introduction to the architectural innovations—particularly the Mixture of Experts (MoE)—pioneered by DeepSeek in their V3-Base model, which laid the foundation for their state-of-the-art reasoning model, R1.

Learn how to adapt the vanilla Transformer to build a state-of-the-art LLM like Meta's Llama 3, exploring key architectural changes and optimizations.

A full implementation of the Transformer architecture in PyTorch, built entirely from scratch, designed specifically for use in a translation model.

A short introduction to the architecture behind transformers how they can be utilized in NLP tasks.