stale
concordance-v7-16k
42.35M parameter concordance model — bpe-16k tokenizer, 8L/512D/8H
Overview
42.35M
Parameters
8.0888
Final Loss
8.2507
Best Val Loss
3257.9
Perplexity
26,624,000
Tokens Processed
0.6
Tokens/Param
6,464 tok/s
Avg Throughput
1h 8m
Training Time
Training Progress1,625 / 100,000 steps (1.6%)
Loss reduced by 17.4% from initial 9.7897
Dataset & Training
Domainconcordance
Tokenizerbpe-16k
Total Iterations100,000
Batch Size32
Context Length512 tokens
Tokens per Batch16,384
Dataset Passes~17
Effective Tokens26,624,000
Training Pipeline
Warmupsteps 1–1,000
Learning rate warmup — model weights adjusting to data distribution
Loss: 9.790 → 8.026Linear LR warmup, gradient clipping
Training Metrics
Loss Curve
?
?
?
?
Smoothed Loss
Perplexity
Learning Rate
Gradient Norm
Throughput (tok/s)
Timing Breakdown
No Telemetry
Model Architecture
Model Configuration
ArchitectureGPT (decoder-only transformer)
Parameters42.35M
Layers8
Embedding Dim512
Attention Heads8
Head Dim64
FFN Dim2048
FFN Activationswiglu
Vocab Size16,000
Context Length512 tokens
Dropout0
Training Configuration
Optimizeradamw
Learning Rate0.0006
LR Min0.00006
LR ScheduleCosine decay
Warmup Steps200
Batch Size32
Grad Accum Steps1
Effective Batch32
Grad Clip1
Weight Decay0.1
Backendhelios
Tokenizerbpe-16k
Seed42
Layer Structure
Token Embed
16,000×512
Pos Embed
512×512
Block 0
Attn+FFN
Block 1
Attn+FFN
Block 2
Attn+FFN
Block 3
Attn+FFN
Block 4
Attn+FFN
Block 5
Attn+FFN
LayerNorm
512
LM Head
512×16,000
Generated Samples
Step 0 — Mar 9, 2026 9:40 AM
Prompt: The
The V as Z er ent in the their to Pi or (a a . Iin, the , W an s A, a sMher his a b in is al is r ( a , is is in the in ent or to , and in , s of the le W ( as ac a S - al an in L are is / e, " to any itL Yby E is . LMthe e
Step 1 — Mar 9, 2026 9:40 AM
Prompt: When in the Course of human events
When in the Course of human events in the l-s Pwhich al , and ala is and (ed a an a To ed org, in the at as um a (wsome EI I is C( um and and , , - ) a : i W to C e(in of the n was (ing a ( ) ) ) s S - ; , a ( k d a , 18La , R R it X Oto y or w yDthe L on g
Step 2 — Mar 9, 2026 9:40 AM
Prompt: It was a dark and stormy
It was a dark and stormyinis arythe ona Sin the ed La a K. . The ata to a b"pHthe s es, in , a to al was Fshe her - an a C PKac, as Lis , ( P Lan LPus to (on a s a (. a , L is o , in " is a R Tto , as a a in , a Tal - of the Ts P the Ga y (L, the i Ka-
Checkpoints
No checkpoints saved yet.
Chat with Model
Send a message to chat with this model
Generated Invalid Date Invalid Date — Alpha Training SystemConfig hash: