Grok 2

How it is built

Its Hugging Face card calls Grok 2 "a model trained and used at xAI in 2024".¹¹1xAI, Grok 2 model card on Hugging Face. https://huggingface.co/xai-org/grok-2 xAI announced Grok-2 on 13 August 2024.²²2xAI, Grok-2 Beta Release, 13 August 2024, read in the Internet Archive copy of 28 September 2026. https://x.ai/news/grok-2 and https://web.archive.org/web/20260928085448/https://x.ai/news/grok-2 The weights of Grok 2 followed on 22 August 2025.³³3Hugging Face, metadata of the xai-org/grok-2 repository, created 22 August 2025. https://huggingface.co/api/models/xai-org/grok-2

It is a mixture of experts. Each of its 64 layers holds 8 experts, and the router sends every token to 2 of them.⁴⁴4xAI, Grok 2 config.json: num_hidden_layers 64, num_local_experts 8, num_experts_per_tok 2, intermediate_size 32768, moe_intermediate_size 16384, residual_moe true, num_attention_heads 64, num_key_value_heads 8, hidden_size 8192, vocab_size 131072, max_position_embeddings 131072, original_max_position_embeddings 8192, scaling_factor 16. https://huggingface.co/xai-org/grok-2/blob/main/config.json. Arithmetic: 32,768 / 16,384 = 2; 8,192 x 16 = 131,072. The config also sets a dense feed-forward width of 32,768, twice the 16,384 of one expert, and turns on residual_moe.⁴⁴

Every layer attends over the full context, with 64 query heads sharing 8 key and value heads.⁴⁴ The hidden size is 8,192. The vocabulary holds 131,072 tokens.⁴⁴

xAI does not state the parameter count. The checkpoint holds 539,032,697,512 bytes of bfloat16 weights at two bytes each, about 269.5B parameters.⁵⁵5Hugging Face file listing of xai-org/grok-2: 39 safetensors files with 539,032,697,512 bytes in all; the config gives torch_dtype bfloat16. https://huggingface.co/api/models/xai-org/grok-2/tree/main. Arithmetic: 539,032,697,512 / 2 = 269.52B. From the config, per layer: attention 0.151B, eight experts 8 x 3 x 8,192 x 16,384 = 3.221B, dense block 3 x 8,192 x 32,768 = 0.805B, so 4.178B; 64 x 4.178B = 267.37B, plus 2 x 131,072 x 8,192 = 2.15B for the embeddings and the output head, gives 269.51B. Counting from the config gives the same total if every layer carries that dense block beside its experts.⁵⁵

What is new

Grok-1, whose weights came out in March 2024, had the same 64 layers and the same 8 experts with 2 per token.⁶⁶6xAI, Grok-1 repository README: 314B parameters, 8 experts with 2 per token, 64 layers, 48 query and 8 key and value heads, embedding size 6,144, context 8,192 tokens, Apache 2.0. https://github.com/xai-org/grok-1. The release date, 17 March 2024, is on the xAI company timeline, read in the Internet Archive copy of 21 September 2026: https://web.archive.org/web/20260921183023/https://x.ai/company Grok 2 is wider. The embedding grows from 6,144 to 8,192 and the query heads from 48 to 64.⁶⁶⁴⁴ The context goes from 8,192 tokens to 131,072, with the rotary position embeddings stretched by a factor of 16.⁶⁶⁴⁴

At its announcement xAI reported 87.5% on MMLU, zero-shot with chain of thought, and 56.0% on GPQA. Grok-1.5 had scored 81.3% and 35.9%.²² xAI does not say whether the released checkpoint is the one it tested.

The licence is new too. Grok-1 came under Apache 2.0.⁶⁶ Grok 2 comes under the xAI Community License, which allows commercial use within xAI's Acceptable Use Policy.⁷⁷7xAI, xAI Community License Agreement for Grok 2, last updated 4 November 2025. https://huggingface.co/xai-org/grok-2/blob/main/LICENSE It forbids using the model or its outputs to train other foundation or general-purpose models, apart from fine-tuning xAI's own materials.⁷⁷

Running it

The download is about 500 GB in 42 files. The card warns that it may fail, and says to retry until it succeeds.¹¹ Serving needs SGLang 0.5.1 or later.¹¹ The checkpoint is split for tensor parallelism across 8 GPUs, each with more than 40 GB of memory, and the launch command passes an FP8 quantization flag.¹¹

At one byte per parameter the weights come to about 270 GB. Eight GPUs of 40 GB hold 320 GB, before the KV cache takes its share.⁸⁸8Arithmetic, not a measurement, on the card and the file listing: 269.5B parameters x 1 byte in FP8 = about 270 GB; 8 GPUs x 40 GB = 320 GB. Embeddings kept at higher precision, the KV cache and activations are ignored. https://huggingface.co/xai-org/grok-2 and https://huggingface.co/api/models/xai-org/grok-2/tree/main

The model is post-trained for chat and expects its own chat template.¹¹ Anyone who distributes it, or a product built on it, has to display "Powered by xAI" and include the licence.⁷⁷