{"id":"2079874488733438078","url":"https://x.com/0x0SojalSec/status/2079874488733438078","text":"Train your own LLM from scratch!\nA step-by-step repo that walks you through building and training a transformer model from scratch using PyTorch, \n\nFrom downloading training data all the way to generating text.\n\nThe architecture is built from the ground up following the original Attention is All You Need paper. \n\nMLP, single head attention, multi-head attention, transformer blocks, and the full transformer model - \nall coded and explained with detailed diagrams at each step.\n\nTraining data comes from The Pile - a diverse 825GB open-source dataset covering books, articles, code, websites, and more. \nThe repo includes scripts to download it, preprocess and tokenize it using tiktoken, store it in HDF5 format, and feed it into training batches.\n\nYou can train a 13M parameter model on a single Colab T4 GPU. \nAt 13M parameters the model starts generating proper grammar and coherent short sentences. \nFor billion-parameter training you need at least an A100 or RTX 4090. \n\nThe repo includes a full GPU compatibility table so you know exactly what's possible on your hardware, Includes a complete SFT and RLHF guide as a separate notebook for taking your trained model further.\n\ncapabilities \n\n- End-to-end pipeline: data download to preprocessing to training to text generation\n- Full transformer implementation from scratch with PyTorch\n- Trains models from 13M to 2B+ parameters on a single GPU\n- Training data from The Pile (825GB, 22 diverse datasets)\n- Tokenization via tiktoken (r50k_base)\n- SFT and RLHF guide included\n\n- http://github.com/FareedKhan-dev/train-llm-from-scratch","author":{"name":"Md Ismail Šojal 🕷️","username":"0x0SojalSec","avatarUrl":"https://pbs.twimg.com/profile_images/2007035104158482432/yKGFeKJD_200x200.jpg"},"createdAt":"Wed Jul 22 10:21:34 +0000 2026","engagement":{"replies":3,"retweets":31,"likes":141,"views":4350},"media":{"photos":[{"url":"https://pbs.twimg.com/media/HN0y0oEbAAAP6L5.jpg?name=orig","width":1188,"height":1366}],"videos":[]},"externalLink":{"url":"http://github.com/FareedKhan-dev/train-llm-from-scratch","displayUrl":"github.com","title":"GitHub - FareedKhan-dev/train-llm-from-scratch: A straightforward method for training your LLM, from downloading data to generating text.","description":"A straightforward method for training your LLM, from downloading data to generating text. - FareedKhan-dev/train-llm-from-scratch","thumbnailUrl":"https://repository-images.githubusercontent.com/915520679/9a2fcbc3-4753-4550-ab10-dfdc4ec676bf"}}