Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Record: SP8192 + Parallel Residuals + Coprime-Stride Loader

**val_bpb = 1.08459** (3-seed mean, std 0.00069) | 15.99 MB | 8xH100 SXM | ~115s eval

## Results (3-seed)

| Seed | BPB | val_loss (nats) | Artifact |
|------|-----|-----------------|----------|
| 1337 | **1.08414** | 2.80045 | 15,985,531 |
| 42 | **1.08424** | 2.80070 | 15,989,295 |
| 2025 | **1.08538** | 2.80365 | 15,986,932 |
| **Mean** | **1.08459** | **2.80160** | |

Merged SOTA (PR #1019, 3-seed mean): **2.88218 nats** (1.1147 BPB). This run: **2.80160 nats**. Delta: **-0.0806 nats**. Clears the 0.005-nat threshold.

## Changes from Base (PR #1394)

### 1. Parallel Residuals (from layer 7)
Layers 7-10 execute attention and MLP in parallel (PaLM-style) instead of sequential. The normalized input feeds both branches simultaneously, with learned per-channel scales (`attn_scale`, `mlp_scale`) controlling the contribution of each. Zero additional parameters beyond the existing scale vectors. Nearest PR: #1334 (parallel residuals on SP4096). Different: applied to SP8192 stack with depth recurrence, where the parallel execution interacts with the looped layers 4-5 differently than on SP4096.

### 2. Coprime-Stride Data Loader
Replaces standard sequential shard traversal with coprime-stride ordering. For each shard, a stride coprime to the number of sequences is selected, ensuring all sequences are visited exactly once in a pseudo-random order without repetition. This provides better data diversity within each epoch without additional compute cost. Not present in any SP8192 submission.

### Architecture
- SP8192 vocabulary (8192 BPE tokens via SentencePiece)
- 11 transformer layers, dim 512, MLP 4x, 8 heads / 4 KV heads (GQA)
- Depth recurrence: layers 4-5 looped 2x (effective 13 layers)
- XSA-all (exclusive self-attention on all 11 layers)
- Skip gates, RMSNorm, LeakyReLU(0.5)^2 activation
- MuonEq-R optimizer (row-normalized Newton-Schulz)
- GPTQ int6 weights + int8 embeddings + brotli compression
- SDClip (std-dev based quantization clipping)
- EMA (decay 0.997)

### Compression
- Code: lzma+base85 self-extracting (43KB -> 15.8KB)
- Model: GPTQ int6 + brotli-11 (~15.97MB)
- Total artifact: ~15.99MB (under 16MB limit)

## Compliance
- All techniques are training-side architecture changes. No eval-time adaptation.
- No SLOT, no TTT, no n-gram caches.
- Eval uses `torch.inference_mode()` for scoring. Model weights frozen at eval time.
- GPTQ calibration uses AR self-generated training data (not validation data).
- Sliding window evaluation with stride 64, standard BPB calculation.

## Reproduction

```bash
pip install brotli
pip install flash_attn_3 --no-deps --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291/
torchrun --standalone --nproc_per_node=8 train_gpt.py
```

No env vars needed. Code defaults are the submission config. SP8192 data downloads automatically from `kevclark/parameter-golf` on first run.

## Credits
Base: PR #1394 (@clarkkev) — SP8192 + Depth Recurrence + MuonEq-R + SDClip + GPTQ int6.
Parallel residuals pattern: PR #1334 (@aryanbhosale) — first demonstrated on SP4096.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
brotli
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"track": "10min_16mb",
"val_bpb_mean": 1.08459,
"val_bpb_std": 0.00069,
"seeds": [1337, 42, 2025],
"results": {
"1337": {"val_bpb": 1.08414061, "val_loss": 2.80044779, "bytes_total": 15985531},
"42": {"val_bpb": 1.08423848, "val_loss": 2.80070059, "bytes_total": 15989295},
"2025": {"val_bpb": 1.08538096, "val_loss": 2.80365174, "bytes_total": 15986932}
},
"base_pr": 1394,
"hardware": "8xH100 SXM",
"training_time_seconds": 588,
"eval_method": "sliding_window"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# VOCAB_SIZE, 8192 # for evaluate.py auto-detection
import lzma,base64;exec(compile(lzma.decompress(base64.b85decode(b'{Wp48S^xk9=GL@E0stWa8~^|S5YJf5;Hv5{s9gXwn@VT6Qap3bt~@<3h>ok~)Km^%<bI~`7~^@P9dNt*OmJtouSV|m@^}~LJOY#qcoM@9BaGiY8ypPvdJq=NbK}E`t%**OWHq5Yg*w`LcO2`&Ki8P4h{e@}LyJM^e5fwE@n-bff1Ph*mUliW#yCOV*=I+v<n|QZ$y02ziZN=i)3}Qx?Dtm=+{LAgGTC@~>c^ys%R{D_%yAk9-_tV7^coUOo3$w>`(`ci)t`2F7>r>Ltx>>S2CRw|7ov>Wn1e~_!RLQ=%V9g?)G3yPsu%SBy!lj1PaC-x%dDmCDOZ^r^!)+WWz}ejKXTJ#^U6Ra!};QocHHXQC+4UM!QQ!-N5Xd|%~a(9)bTYIO+>B~8~@lqmri%^qEkQUy074Rh6w7V_#^s9J-3BNA`G;qyR$LYcI?e+loZVWi~B$n=TKFp{%SeHYp{oNWh;U@Ahk8M2$OU%K8B$lb*dRQXd-GR_@*KAZdRdwSd#X_bO(lvJ3fp9Otblkh?o!zlDF02+sRjLV6IqG{ieQx44UY<d&*~wW)rT(c@2s{9hl6`y|y8rgmg1R#?1!{D}?K9zb-nK3<LeSkh!LeUzlS8h1WHc?b{wRB8OkBB)(6yMkVv8B?0MGfkcPfl|~~$iXJcP(EqoeNOe4svf$SH_eUIMd3dtb|BSUb1vN`%al^P71leUBh%RAF5AXfk0tT4&iR1x?tv*Fqvv>(f20c)^AD5kE{7_@f9?Q-ePHMY$wCTcn5ij2k?>T>CFcZ<|5Bh`%hA!j2d4G(X-Bbwu<(#drc<T8}FJXTYMR7vF8q3yez=cNGB8F1PdmPmj9VC!WV!^&%dKzWR96*{A7A)N7k}vCp0ML~vVXoZYuA*2FSP01TJ&$T%a|!qZZa~I^2Aiw;pbe|H(ZRgX7ZRW>k2`tR2eo<C*fPF=ua-Gjm$6ej#6$+#oQea}M^CL|C)@Vy#>$wi$p$UEHkQdFiFmlJR#zIG@3*smd<EnlP}3{vNpKXtGa{;0yoE9_J7iup*avf}g|D`*^A5(PhPU5<`XPgyoW{6*wx-$Tw0|x*z(c}SPuGvdSsAz?4>lqZ?s>Cn@<hX23WqFEHc>I!i44iGk>T1KUmKDUWEJXYFF3Mh*&Tbca$esa+z^`enxeV%UmK_#Ex_)>$lBJA(W<ErW)!h>j|4yV%J<~unPL@@@KfP=NTcv-SVPiG3BDdu=*>C1izrS~RvqEe6Re7Xf)zp2fR3F%Ntl(>3N{Nxb8vzZkhK?{<eqK!tMa#OYP1o`Dy)oYNF+~|aGI-K__ESpdyNRoBiL${AF%Y3@3k1X@$hLMe}7%sFB8m}td+;Q`z`$r_9*3{1$UQTe#oFI!vccSiPbRWIFLr-mwXM?#v8qe`<b9)Lt_?xb)W6crB_wD5gh+^W{R@lmx#6zLwV#@1_nD0R04T*I#{4@@S}Gp;1-ERl{@G-{F5lhlG6=tFCYU|eQx?l%p4d5Ezm2`arwE~zPsD}sRUO0xbUP~oyVEPTNl%)&WbZml`mtOy;FDP$3ko_(|CeVf00qEdhx-gvUE+Yuqpc**EoY7l;!+1iCBbOU;F-Ercm^sJyDbEB}@mHjpX-c*1RBrC5^tRH!A}btv6B$#<_X8&xLF~V$ijv(ui=Ox@s$+H{2;CJQ=N%0vq(pWW%*$D2bLU)4DPogD*V*drm>Rpe$4_KYFAQQI8+y6>9jc4wHeJ2^*|#6%ShCfQ3zc=<|0Z`Ja7Y9E|lQy~CwnZkBE%teQ=I^Ddvss1V~h8|jSxW!&||)rKK)$#K{ntoatClG0YVqyY7Qmr3Mv0zyH;LQ+{zu5s@m_NC1q7o?u96Zu_lD}&wKWnBj8K_s&H!?(%4lDiW`^h3XfN^w7^NZ}p;fxjfP(oriMP93@G`Hokf^LPBO_tUOCA9Y?Ycbc|&Ozk{6p6^Be5B~llA6b0L6cbtv^Vkq!Kv#A|GL0_5%6C1U%s;o?{>h@9KI+Dwrzrw4p&PwOoXNEPg+i3?tCZo5gvp{$48`j*%6?1_2lYE_<5L|gF)<j)|JMbThEV?C!lcQV%|4a)_qSzQboup|HY2p&Pp%sW@3hHq4Ii-qG$y6My%5i0h=;68ITg$W4#12{Vjo-z6sj&n!Cu^Tz~sp(W_BP9`OC+uxiGIqMwt!ZN-vAm=9US#pWOB0e8ZQ>I;xo=L^B}fgC34&i7g6#!NR1C%VE$;9A`1jy3(?aL)zptjpu@kuTmhtvwcM@3tw!`sx~L@_mU?cj#>c8IPpDOH&&j7o|Y3p|58R~&?Y{_mut<{Tnf&sxD9f3-X;AF4fUcuMBob=Cwz5gMSe4H<2Y58wc%s`q3kLoo?}Ounl{`xs5`2}>uTeQZf*|u+~MF52p?tSO5E2=N$0BDF4qUB-&k0+hQ`IXxLj6EcKaUYxWA@INm)8lVMY`UnZLg?lsz)JGZI$YMwMVQvy5L6ot48TZW4Ce=QbprK{6#vN}~oe4*NMzv%h|gAIZcf^UIHf460B!F3yBkcE}f8codomJt7D6T&`WwKkaIZ#PdEIRj1iM5Dg~vQfg<~lvTA6%=u3OxWLCehbC(Dz%VW-Li+^3#s3kwr4g#={BM5U>1fk|?Qf_d!cDQy=6c4lYhBnzZ1-ek9??{sRGuzCw85FX?`;NiXC9ULdAYgG#e}}~P}@5dT#m~HCbjF8X|AIyzv}0nI66v8Ecq}knsz5_8>0lb|MfnOUC+?~Wr&&xx8~4p6pCLBs~t;Qo$6t96lqD_4`1C8r}u_VnsLc4r0sjJ&jV#E)9@lk0vwVqwOrRCcvhq<5oC=|3+Zd3K<KW7i-0|cwtox_1^jY1Onfz#bB#BGwK6t~#<m&0NsSM_nakqyCxeNm`WofU$6hQ9G2j&Vltl7$rQ2|LrE4gTs2;U1hyjI^p%=5pAjLM{CzR$ewtBaSWijNUi?I8Sz|XPihEpOoPljg5?kMDSR^4?+xS5KF`@J!ZDR5N2peS*(jh7GAll0huU8`n_@t~g_q_?@Uy53B$?BY-owq?Mk0bbeoVqLQo98PdIb6lourV!Fz%V5$%ylC@j0Lyu!ywr$xBg)>Um>sT9<oYJ?&SN8Ufe6`=x?DZvWWvF>%XHP)CBB^D3Ig#o6U*S;!Z%aY4YjOY3fc)Gamb-As0K^GfP0Nsbx5l1lNhuG%rTB#P|lX*@9z7UHid}4*hF}JV)V<+D&WTX{yEL>{6>$t!_6Y)@np{JD0~EYdiXTrPrR2lcj8v7(##<^bIPz{AjB}2C{v2{gIEe)Th*hFbnnNGv8Nv#<wdC&0k#7pKdF6NEx5=kF|D#<nY6N!9K|TH;94myskw)JJLr^P`8sT7nSQ|a83G5@kUWk16^o9hEYZ8eb<wR0f+Pp}5<cG?S~>WTYJSS=dR|2uOP=cK)I~-xY>D-5d$(_GjRL0xP_S#M@`RCd=2VjVio~hg8ieN@wK?B36~~1aW+G0q0Wh3hmEcaWPODLH*IiCG#AUXWBl|S9XHbc@FH`}8N%pw6h%2Is!STS;YHhUGnpj0i^gM7NXa<iijt6oks+uv#a=dL6npjlK5Scb^rLm$pMj9|QOC}D}$dco~3aC7La+YD;Xqg?A*;la&JZA?Mh__&>G1<5yi^*H~dZP}kH3_R&Sj(vcKtKziSdH7lWGLd3l58x=xJ@V993OlE6X6}Ij^!U-MxCkwG3k*NBmP04c#koXtaqYpo5)>UyIND(hjWT%SN~rP_&e>@k={z#9~$%k&c5u**qU~ji*F!uGYvC=g6aX@!`QXuk{EsEk~x4IL7>2wHbfU*Xq)OneWpwl2whd3W3sC|5Xa96g(n9H;4ym3lmIdMpoY+npex#4`RU}>SD_6Zpxmvz9TsP_Z%pIu5Zqr#f<Xb!py>LfiqRn?_RDKAW=#QNXN@&cG|g5<Nt`xm+{QT<T}vfr>?6WWtUl|L5(9nlvA3dpC9bq3Q{WvS%Vi{2M;WU~6Ym^n{Md_AygF@Z*NByLJX(p>d_0K_Jd-X_oj+r_jhVOzRstpVhVFl|%-9rI6}CFHYE=w7=cQr(Fj`NMl`+Dv9cGO<VGaPlHeibPcr!JC^?^X}4t@}@H+X3v-%ZmUs~~&CC*6DM+!H;lNVqLQ5qERf?q;SvI5O&sBj0!6Yd|V81-OeT`$n|9=gZ~<r0Q#}PYlS6DkLvwK+Tu#1_#0f^k55*w;d>cVvcT>h;~Rm#|={JKl`0#`!Qg-DoL`ZUa?L|kKs?E)GE+h?2I$_p2GX5ycyr`*dn@~$Zy4WplGhJco)tSYMW5P@JTr{H&1Rj&?S(m{17?+5z0)Wv@ZwY6i8l@X^5V;3(#3DB8E73aPg&PY$`qdEag;(uuP(#bpb7@_1W!7Q!GvyD1xQHr;{)QMWNXG-?NPDt>5aWzfoN&yZ{P4`mn6Vs6;_z3rHG?kbRb0h%yklV{Z5_OXKHm=oSSC;RFLbIfK&w=$+2u|C$>a3&YRj-o%mcEwu!=)IHIsxS?H^j<8h<Izn0%>kEosvtfPXmljt^2vgah$4+3IDoN0~i53_}>*f$|#;`KSzpkQcb6p<E*Y@7TLH6zbx@y}Wi<AFe@EbG{yPhj9tbuG~Ab|hj``H5pyl=YaxuGn(UdlxZTp{qLHV_Dg)d!j;jX+H>Knlb{PPojYm|z4PO{MZ$PLvCdbi#t|qz9}%so+#%RSjg70yYd1FTOm#<#LX=l^b+UC8m!1NBt9(wTOIWA#>vs61cqgjx){yf+%?W83JIS`Z0F|S(FKL6*Lq2rT()Ysz*gVQKMaCiTYW+NbsI_Nj$Fmmy)^5RO_wh;A>ZQt1LM#E`~77iAm|H?K9qHE0&W_m2_`0Gq28A5Ktrv#=XM1RgN*{GHx^A@?eADCWum0=6UNNw%?)ew;A0-s*O&%-EF`Rhw$MsPP22C;EkU-W*8vd8Y*(TgI|$lO{!xP`Sa7#JOpt~cQIr1c!Y3cKAnyXyn$8|!f@X_`xCEZPb-3(@Pip`<VjfsF!;(H$M`4VywFAyv9?ccTY;G#UuTGtoyVM?60R7#4V1zb5F+=No66q~`6czjR6Y({do3wUqFqE}GK=sD#;wtslv+|pgg283f~cXt_ID%W-#+6ZLtZb1x@<Te271n~*!Uc+(Z{2p5!pXOesAVFuG(c5F{(~PeD9JO2ozX@<|j=3)frj)<;u%ysWq`?vln=QnFuOTQWcB(pTy-c6eR&K)2f~QG4no_FhTOSbTap80cX#fpW=F1ueCt9vHh`+ntCngD`Y5q_ysw>>b<%K9*{OhVCvbNSsOVm3LOc6&C+-byY7iW$sTa6_?cojL3P756VE4Z^HqE*CCGOy&jsQFfY*D&Oz|J^>W^f7<U(QRi_8<cE!wEU6*F`<(2NZV07jy+?p+v=oH2#UC_fOteV2dF)Wx5PpGhI1<(Zf6Iv@bh(7bxmCFPKqUhhB$G@o{>+YTVC0}&R|&HHgl#X>b50##u2OM?zEC1KDc{GlId7wIvLbB2I*-aSnO$U3N=)hInkyN3xmpYZ2SS@8c^hE|XHuv(9y$^HfS*Mxp`Qf4)fc9+klDQmud-MK?#84L2lx`G4VIpi_4bLu}!)+(kTy|2^99d`=+GUTB{0U9i^-~w%`Xjw_n6y4edygu?S9(_j*g{9Wd{W=UjM#K+aAZeFLKXUz0Xe!*Sp4r(|&)2*#bDTwoLVEvHURF2DkZQ{@L$zV&|J1!ah(qz5$d(prm3wjad;U3`H4$H-PbieNp+6@W)JPafNq1nde4x#<_ix0r8@qmWD+>zJOq|kndW8Yt>Q%+rN5t*X!mH>IuIZH>ZmgJS-xEFxwNtDT%X{@q#Qg)?kn?@HsY@sdXyfdYxSXy&5F+P^2y~n)5~x-#o!R;gJnKBIj(i=%Br3jcv)e-B*S*XE80`!hj*=iGT=@-xpEit)7ML=p&lVIh8TGoM<VGXd6A28ul7+b3UfAgMd+6Bde1^6ZMO0=;5I3w6za>m;d)g0*drnj5V~l9j1kN4gBb2m$_pK9mtvk1S<PnBA0>%Y^6IhL5eHona7g1y0ScdcIew~ipsPw~|Y@rs(!y{5GJYop%2iEk&08gnS*P&=jYb~P<cI{Y!nfp8Z)zZ|Z-Tt3+DE#E7yHEq@l!P)A)P@On)nqEzmRK%&aDu{hCdN&pqNJE$!D^ic`fVseShQI@gY!5}Y`NzTzO9!2bVq-BS8SbL6)nAkXfIb(7_!skpa2J;juq22$saT$|7r@9ic|Dijg2vbn6pSEu$oeo&ZcIADja)D(J(d}R{j$1snQ9%I?18cucbt;C)3j#8sHr;$-CP!FfkKF3MYzjBZSrJC=O#jVju8bcy@vW4tM4nqUGEM5UI>S+mv-(hcYfkr3)u0H(nrdQ)kCx>AJq7#QoW&#`|g=`Rk!&mt;xr-Q_0TCjs-ApiLQmsr*J&mjB)2d`PHXhcH-2HecP_c%7-;Tkrx=Vvc*pKRq!`*<W&73YN9_mC`yc&$wx<eMenMaA;caej@O~y%~OrH&+@{8>wQ7fE&yUmVA5aRE|zzebuW#?9yyOTJQKKYpBLMSoLfw2@eDKaY2G!`^DFJmErP~7IGimPK*!k7<3)Haj9A63Zs$(y?6Bq2Vjq-f(I&vn&@UoRDW&#g}}C6I8_84COJpt0ns@Ap?j(5sjAGCZ1#^7<OBHLM+7|}s@14H>Q8Gh9;p+Od=t{hA*U)jT}BUpAyu2(xwFeAvD@&lxr8QwM1Ho`$+1MDX{o3#H|H54tCDh4JTQ`xu)^(TOf!80W}FQFl;O64;?s#KO{72&Yf$l~hX~|;t2+hEB$u6B37qq6krbxM0Tv|GLIs0tCNG4nekbc^sq-O5N{OUN&+-3jfAW%*aC#f8Lyi7R5sAKEh}^R}42fc^aPRqUWfA(JeGx#U2H`M=SCf@c-5h9$AeW*L>F&!!-R0_zi5k_WQk3q4)}l%P$=^P}io`XM>@ELnE5W6r$5dKh+}F<&hX7!I<H{3%i}!qXpFjx7`?*NiFWgE|k8J&(;EsKeknLhke*t^j3bId>akWv)a1cp(`SUsV_x8CIM(bU9SMG>zc(;3JCo>5@MO;$p%aL+{X{r^tJee#qbnq$BE;--vJ?$}DO9u9Qu9UrfCeD}E{M+&q={z1c`tftmnnm!%M=}w_bHkQ0P!V-;(*}n`6IZKf4<p{*t**zz8rl;~wk?X*1?ZCLH=e|?qpE58z6^OvEOX;>Qn5SuKpPA1Q4fV7f%|qtY;x+{D6Vc17N;D>?}+B_=Z}gBiWn{VXx%xj@u1HS=k&zK`#2cRK)<^5v;kL>e|if5nzu=|>n}f~xQjAG<%N6Q{H@$Udpl#tCj!HT*HnOseR1}`aN3D>XT#*{5hTjn)=yQV8JAIWV6Z2mvZ4|RB<ll<VD4nDrA>rR%edMhqHgD^w-J}LJ>7#C-)TJn#9|a8G1dR<z$>L4Nv4&yzeuUlb?gN<BsrEqURug)Xha)Pe$z?3Xc=mL`COT19>!et`g6lxluH%~y!2N)d^Ag;ZzI+9KPTU|{3XIAVgoNusLGlA+;|HtWPr2JMj_0VSC-2%snOf0pX$lFf~H9@MJoR2O3v=G!!lLc**r?fVTYz{U!%82<}E0I^ATxlD*!F_CqjnDjiHnbjBVh5^QD#_wo(}fTkYjkrOiPyn-)j$q<vc05@M(`MA%yJ;?`COul_bGlJIiSydNXA2(wwG`n5f(lr%gwe3Sy2;z7V)Wc2nd0;utk#`dtp+JRRI*h_`$1l2sn)a3{#cKgVOK1F+7XvRn#6CdE?l|L%NZZg75)_h?Z(+#<(+c(Y|<iSSj6<OWst;V{dz;6aI#Yp9h__#1>rE-cZE=HacnknBb>J-*lSPz~+);iJB$!Dj3X(+*mULY&hKhA-5CR=wT`iZ7&UQ~tvn8gE%HMW2>oM`W!8OP5`(#=AiKI_&%#Q!F%>E}s4-c9hm-=;C?EKD+FEJ5p34D$xUP|-<!BW?I&F&PQ|qbbMt`LEembw=FF?a>`^kjCwX_KpM&Qyt-+U5*^C!E;I?!AB<J`F17C|57EF&LOq7Xkq(?UN5LZC@<!spLQ5CVwb(xJ@-x=U8F4YFFf;n0}UHjxR;eq1G8bg_Q-)<UbZhMDH>p|#ipP}Yh+BHRn3nY62qAN_OV4sOjwOKQ|KKC)qLT`SD*^SP<DASG1OhDR7Og=++Kc>*2i?`49XRfhsCb_@sYT#ns;soSk=T1Rj_!{=lA?0qMFEp`?_`*HbRehw_cgvS%%D)&GF4S`aZ(B$9H|h-n6wExSb$Oj(#){*iphhoUp%{k!dcY2B*>wg~9qwj#*mGKe_m4iLDNUyH=<AY*SlDMS|th4qTN#sjKmVz*D{R1KR8qnFX;s^f{lJ?+Bup9t@UMS1)`-=@&P+FOKRxZ#Ru+9&I!anbS)TH%BK-HoHz#`J5KE>0LOdTP3&Qb1d_0H2Qw=4!^PZH-5|z8=%n}zt_4%jQ!dpMzB+VH_3`B@KC$r<2>3qQkfOSh!JtKuIHpYhz7mUKS_4#Ckk>qoU~9fkUC}KJMroVoy5gRV8?j#bTxO{9SZaNe>c7D1!LCjpQ`&!9~!XWJ{zJCm<vb!s@*2R-F#h;Zyp{J1EX(`?@#t##gFn0HIkfajR_N^%CP|_0(?u2&zix-u?JW0yAh|ib8P3R=gY-8SSS&ia|lf6njq1sKB<o&<Ta7KQ0okK0X^=FH3TQJR~4XmF?V|`%}Bbu2};oT!I&{Q1%>Kee7-SVA%DfYo(BUouqAh8yy-&;!xN6~eY@i8ED`;+JAt>7su=fs`Ul0Hh16--i8n<ohHhZkW(yx52cY9Lkr>&2Z|xsFt6To2xB>7COq2W0+-GmQ5Lrdu#2K07hsUNaCZy&-rzcvJ+}2XlYI(|jqd;m9gYpBay@OvfUT@Jiw+iA*ndR_*U#jp-QTyxwk%bavCEv0IAO<*c20q6IfsQWAQLBJVT;8eGye@joo%AO@UMJGBI8+UXe`Uw2f-qtaRCVKoAqS1mMpQeP#Ul6i-5NN&T!lxv{#@JWm4&zoMvJdlSI3+F2f983YKFHBRSK)v+rysm1!HBF>nj^hPjJ?Vi#L_n#HC8Gi?$J?7u4v#1x-HI2N7@v)(My3uG0^{`>nC5$2lw@zf$q4xM~%c$|{7}2;|CFN9p+V7Ma6?J5qaJU}swEiuLhrS5#)TAb_YIVSOE%vh}uIsn`u&%!;MvEmp@BO`>kUy~@e#TAIaEHh``AiC&K@X5J;F$g?uv;r}IaF)IX%V;j8Z6o_t+t6$@|W5y}!e?)<&BSFHPAn#5uT0>(3z>TpFLFVW&kH?5gqW(c<WHh~sOx1dG=2P$a%Y@{PsC1h)^znocyA0*_SrD7_$o;|LwHs8nA5Fhfcif}!j=3=ZZfw^bHSKj5IqEc*(ITDdK65A?ahx;S5CAax7+nOK&WF%wKM5M^10-gQIxPi6?#|^c_8!zU@9<>uWbkI6+;G<0n81$&T2-DD^?RG7&#C}t*xIO|R0$+zPa}2{E%}*HXLHbc#TqRiVky=^X-W8@)(OYhYK1|iKTKlf(1M#)?X;pWlGi7u%Z)5sfuN9DOdJ;N@9LCzF=<{SdO&c`PgR8Rn>93jJ~0J)51*wKnRZU;S}fgh23U4%knBTHw{K5LZ6<SMJXR^yqsS$&MA}fdWT-Zqgt^4^idO%mVdq2w>0&B(5}s~xRm89Q(U$bZixVUv{mqe5?lj1}W<xnelbmR-ba%ApNuCZB=W9nvr*;p-G;#)yj)I5scYj%rv4l*fNUI?A4th7EwKSAkW4bym6OEc?lK?m=RP3i}MXbvK6<yGSjjEh+;9Sd-4MEn*6|Ql$4~`yo{yt9oXUU&BoUgGqasEmK`IYcXI<&=H;$5AbtL`*YLP<+2`bqH`5IE#oP#HZzEWy#VP^FzM-#+XEyDl_TFURTjpTj*tGWtK(+F7{9{x3;}N(x<XLB${19#(#)$nkPL_p5TDNkBQ@Nx13~o_BA86&ITnSp2;SkJP+bY^OYMMnpxXxsWlkr~b&ZhM<J4gx6JxGJ_JS#!`HoZJvwv5W|L1S<In(KAtEYw=Mw5#Y66LXt&E)Dyq!)Fy+2Crz)#RWc4!f7#q&_pt++A4UrfkP&4GH0f3(?D3y^+qu6b$qq2*&ZSemWP-dnu^S)lM?p{0|4Gx6$_nD-V@G9(G1~t=9&&+6NpBM-bAp5&##^IM-TUV2wAqHGy<v{+NPi9jCf~<4GA?)<Ih0Fj>!Z<fHTy%;_mYo+0CySxY4|^!;o>l^fPCclJy(ltmZ)0pfG1ER@OqM4q#GY6N%N~r8JEfaV^^DWq;;}2obb~nn)UQcTrZazSIi=eLL$FkWu}h=YZZ2PPgjZfTC&ywiDz}Rrl<grrxQkcG=(;adVC~?R)9{c6>Qwy{L>KXu{>^$S5SWC|wjf#MBH3)0yS`}%1mE!=EpVr=uRk8PZep$@aNp-K`}zLR8fV%bf(4ig<>0lCOZVTL(={AXvm~y*w+3+>#(sDe5g=cgV2J>14BJSED;S;^dK59O73v_*|2i5gRM$wU;_L4g91GAICCCTeVh_T-Dt*|GXZ-~DAP!DgyBa4V3ehy4p>}g0J61qhQ%V#}qrH6qo`OTcy54j%e^TZWG+T0HiGnK#W@{<(faTNEv~B)WVK-seaRcU=0y71nuX~5p7!P3GP0%*LQht433Cf*f|6GT~o)OOUjpMlAjpdof&nfn2Lf5X-k1a(1LsT$$t%}Lj*mZyAsZMOc7P+7ss$huz6Hh2_j?oa1dn=Zvjv;|D&Aln59|CbC8R-76Y8Gh)-Kbv*Y3kn9d?H&G5{wok9r|8#Jf(?xyoaAq^!HjaiQ<;_xv@;~BTj?LAvr5UiVf)mc?5BAi8K;jF?2!!H7Rpy?|+McCb|Da_~Fvy!RzzwIuQF6aAo&3VA8UiJR#cHR18lAoC(Ja38$$Lc+I8A++$BN&Y7mDsw%&SpH4Ko+4Mu?>?5Z}b}ZbnXj#5sZ5{DNkp}v*p4E}iB9mgX0I{$hppe9LQ0bKhK5m&iH=HAvV!X+e7OUnTfE^=fC?5u*=jQe}S&>8pioi0?Q}@qg4&#kmEdd?J<^R7}^PnjCWw>7rT%K<7Z{FuzIBgZl>+@z=4hz7ZZ&~tblYH6b-=(=66zWRpq1_O!@5%6(e88M8;@9y#28GlCetzQ;@}w9pq)H*~GJGR|Q>zCA<HjzT0ie4DcX7gEa<hP>WxVytWWuvE%VcMqqXH}s{PGXF$btlAlT52Te>RXWMlQ{%!A!qc0#w6WBL)qFkr}G1VNcbUQ|;R!&~-rqzrW9u^hi;Fh7MMGyi~@alg5Zglb_lPupyb;+4`eX76|puRdv|=PAdIxT1JyI10eLn=?XvRd4tw}fWvEPtVw9wEyooLt&+oG($M;9ZWUl!1sXD)ZNfi@YeKm6M<y;K0>|=79pTiZ3VkZGoGi}e8h<smXeh`yoVCdK-fQYIiA_zI2_O@M>ED1Gqnh@rFA+MIz2wdsXqg0-if9Q}P-xqUAjGZA&TluML8v_OviO>cHxXGT;hD-%_MAq7&Qeh_fv8D3A#`Wtn4Jh7BT?%g@4>R0EDp$W75>&C1;L}5_byyP6IdeR=DSh?V~@^&p2GHqrGai2E6uV8H2$x&aHa32dW6SX*6WLudZA9nYl8tvrYPrOD#?!xm{J#~wrI=Y!-yp|(P;}Y0HB^FmVngimmIarwBG!gr7E}$f2y@~qduVrZi}W2;Aw0vAxSNsB9h9;P1Ce;=ps<T%pN*s`KR}7^X?U`3hML+=?FTr%!0rbJ;0O@h}Tjy%rgg73Q%rzm2Y?2=+HU88%8qj#rXqTRZS)mcN-iF{)h(f!u_7aRFZ*zjRs11x2cju+wfRJ7%Vi>Y*eE)r=9kppXNEC=AyfmqX|J3(FQ#ZY4wd>$O}h5kretP1qm)(bM(s|n#0a0ggvngf+4xNu{CRHwg64@qih*H%=TM8AGuXSa3FYbEy+8x_VJDf<j-sJ#K=B=qp<%R%Lx-Ghv;0venqY4*UX<wx5uC8yCLg#e7IBk<4g8FQN9*5aKNQLulWI!05e3@Et+coiJLw(XdU$fjPtVj!hk2*jG&oQpAj?99BfX~x47&ut?D1Z$fAtQHa&yoAG~D!(~aJttUGxdc6r#%zBtYvm8@2ghMC;Bx3Adw99>l;)ve7YW?HC~S7+Z?e?=Y5y(YTxS4CU?#nmU=$zDUuhTbkiK2tdfN^EILdE6p7R#h}xZNzEip^+XX@Whsz;pH!$@jC+tHr4=iMn?->YwSAKI5u_T&q(bT3fLd0&=9Q`JUwr`6^e+qpVQxTvl!t(hjt6BGR&i=>NterjcB_SjNJ4VVrJwJsx~ou<&TVj%y?4BN@h`zW9MLjo*&l}G3(f`aBq?j(-v91;@d=wOpmk0Z=8c2m$rjo*BU+nyswe^pY@nO8Cm|OADGQ|HZBvAR(W-)-QGf2T#^~E=vUPfN|*n1)6fQqj!$S09g-mE-f3edBTEy%WI=ruBNe+-y+g>qIMYPI>CeIfi0h{rZ*yo&-G4?9L83{eZzd=071Q=V%dbPk{ox;;)H$GPWh*}a0ntF#d^#2o!rtQ#_^!XT$Rlz{V096Qk5q|^ke+t86F3Lhud?pQUiA$%N?RT2xS_HMa#|kePt_oAwGmVi!-uMkOBD@6k<12{!)0aA0Nl8NP=NQWGGk<?nj@$|%|(HZotoNB2YwCq+~~`amq$?l^9_BqQja8JT9a=M_1O#y|Hr;)%;j<K4{Mi|`L{FNC3j#F*AU3SNmh*8juRHct<4o1+o2kguhb;bSBw7vGc);2FUKOg;BxvR0P%}b*iIojUb0;F{Kj1!_UY6lKf=JFB(9r`^&$++ga-rVcJ5l*$Wat$zAg}^X&&((MsYvCbK<3*L{l@Er>8*#X^)Z*?o!x5Q93EwV=Of0hzG!7Dv=?6;?hpigwG589E?wV5?JeDJ$xEXh`l;SFvB_Wm+xUKw(j7^e=x?f5b~OzPgrE7oa6~no{P&$af_+_36!a`lsoOIN~WBfh^p~J#Uo4}@}a#Ne7X7FwP$$)me`Ug^K|VU=2YeCg6)%7v>?#diz)E<-Rc6$hZ_hk5URC^Ts)Xv9K$AAW`lzkxtu<CN7?1EMSETnsp&Fyt=O#Hyv#Iawj7mC80YV6Vo{t`HajV*G=-=B-Bd1OdOI1*cLZ0)o5v;c<x1111SQBFR}x}&8zXNGpeOM_vpqy*j)jljUdgf&7764CZ}zB4-7yn~&(kAjl@caE6!VfFr1HC0&m=DkTSx~ICF5)m%+X~PmJ^6TYMTloFwo?pz)j)OT#+R&%9Q?;kg!-SUeo{1yjY2XbKGq%H+>1t*>C+Z&Fp|Tb}DV>2XhO&^T%$Qyn^$ICcNDW<~--cre;v%4t2`~6yfE@p@LsPxa#u3*RK}n9t8yhx@d-@-^*3<kNiRCyxvELAHF9AT~7r_{`m&q#)edf2#Kjge#$vtXMvCkFjqTkqeJ&JdfikS>n6Jkht$ix;Uf;Z@-9+Ys6m3uLeX^QyS6^(!i4i!=oFlW$d9Dd&<|T7yGiHc*zFdHeSs@S4d@+r43ec(La&al0wCBV0h`n!*rWQ$P<xC^2-F-A=GwqBRCX16xKX}!3LCFy(q|3;CA$0!G*$W)s9W(45*PdtVu2Nyv}zzBD@z=3m2nk2h!Ndxy@k2s+gWJ(fmW$>M|ngAuBIbe4K{y$*m|L|c{<g{W^F8Mb`8%3!M%ukms=vO{1zY^QRjWyfm@Bju>iQL@@RA%DWXF`k;J0Id!^-HV`@nk5SNAN!QaCUPE7h%KCfcV_993z_wcn~!0{z6^>|YjpE|G&?vuz@_uTgEuPx6|G}EqH@OpZfQU0U(SC}iP$cFu>{Uy$ieQrVtyL5+(q;hSCI{e=3QD4(}{A&S2nxrTl$1jI^2<U{rvwrZ}&Rqf-GCFMlo5K>ce>?NJmjFa&solP8<bE~}Anl&!zU<72opB_wIMzEp(^<if^Z%M(Lfq)Y1Yj*cj$RE<Akp@C5IO>n1vf)OMw&0le->%bBgBa1Fp~UM9Otj;3A!sb<95<vyC#24`NOMI3q^R}is`-F+u)Nv|4(&GQ!$W93iG0Lj|?Rr`g07)+%Pf7*(?Tr>Q0w3>_cF?VC(1Mb!G%R+cY<1_HVH%0|7v5<Y!Q4#BgFffuUK%eoHSvvIJaw#Xe8uXi}<-bL>S`Q2dXW+9s;RD%82N7SEv!PE|sI$q$t{`SUK*X3G$4aVqa<+B`VigwLw0=@*?;tq-Xy7@vObTvOK8d6zd)ZdTchhv%)AtuKJSbb+tlGG3KV@!USm8oy0lZLIft#=K*m@>UhPxSqB-7H`U6e;gfC!qhWfbS}Ita2(fIeo(02N|F*%KE*pGAb5Wi#K}6=2F-cC1$PJ*R#+W{-g7%>v?@sys(Li^^|%x;Qls1m0hP{y3iZtcJ;Bg?QO2l%1s{Gac<Mg@$buGaeEfOSP!-I~bgou1wGkH>DIfs1H&$T4rR$GH(U7F9_#nkWsy9N#ixr0#l^MOe=|VfNFy<8dIyg(^w4z4jbX+75U&|DvEN;Oery0Y65r1t@qWv9j1Dpu2$=SwSrooW<o7x<N{R|7YD0>p~`Ad5Wu67}}c(dyaNffcPh%gdY$s-mHDtn;6-BLrB3UQ9rzvnI6Y1!wWzEHkvONe4|@CiZVPC}-N@;=rJEzs!8LAka6Zu+l)$wFlgWtCxto_%d;a8|+AcxrV(bybquFWzIlNDoGDOya9%{09?N%iS>u_hy~>e2$m30Xw4N)H$T`+wht)RVvFfO>Py}&Z$1lW7EnLpJ&uN7%i{@K@o`XwbSJv6C(;&Em^hfZV%PuL@B`*W!AutdUpCzqdt-GJ)-Wze~@AQ)1ni>_u?NRaG@4?-<k(8;VT*s8KTSH+dg!3A3g@bd$c7aLYd(frT;<=-EpI`Jqly_JLXEi+%~vMV<wIHL&O^_i?9F}hJlS454H@VM0l<DUVN+z3b|PCKFKWq_CMr;t3jdA7wAk7)0ZJ~_-*K;#k=A78`|5am!y(xTVE!kQT`~_WO=ZDK0#E)F)(tseCbSajFnz53^Zc%hVelLZ~+f9IR4Omm<cc@W90;>rS_JUd#Iq-L`xY^^|0+Uby+I=FEeR#9llHJDU#H26l5WEa8v?t*n^y?P&mlSZnX`5fH<3s68^q_Hon*=8E}=ff?9tKdGP}!sr#31D+dx12%g8EZklc#xp|8*V#auzpU6jp6nDhqTOz)cr!@}lWZCv6pDCbwK+lPw)!=*0K*|2r$MJ;e2dcW4|7c~12icGykpn=wC#i;WD>D{75|@>Nq`o8D$%r2uT+?Tv{xE2zJQ^n0@(4$h2f+g4)gK1H4LY2}iV2^V2kn(G1PL8h%8hSeaFPff_HjFE3wgrfm{5O~PbMcG(oQBX1oIbdJkP`~Gj1`X|ApkNo0mQ7Z<UQ5h8`IU41F$-H~d}em+INrSFN~AvnAlpO_ihh7~p*^5UNUjVA5-stbJxBg=q$8jU@yVV$9%$g`ZhaKdAFGZkCbkp$?I&PO^|xb6dKPSYu%acMs*$2{-;O62Oe0D|X>PIuoVMhD}jnck|5I5NVzPWU&um`0qY);(4`olc5>(p(4A}cI+hn6<@=Hhe1)t_K$MSMc&u<5)^Q(>oDHx@ZIA~aCTi1hq4#iUs1^6f72Nw0AN!gmqsdUlQT{y-`2hYUP9H{7YA@Dp7o9GndKm}ZcVUScnzf|4og&Kn)Dr%iAEUBh7(0$SGQv<Q>eSp)fd#jkaj_Jy}pYWOtYqis{O|zgnmYRZewjOt*7NNr6&N{k8k0#`kNWQD+KU47&WFJ%7A6_@?Em&M0-&ih3#Y~<Mp<SE{P4n$KPD_V#&?jcJ73+P-?xalkX5RyCIzd@Ws3`Si-SY&4=ahjW~M%s*4W~IT3n800G2d>(v4P;)QfovBYQl0ssI200dcD')),'train_gpt.py','exec'))
Loading