From d041987e9a77a2cf2fe4161f50544ccf40b21d56 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 11 Jun 2024 17:51:10 +0200 Subject: [PATCH 01/16] Add mkdocs-based documentation --- .gitignore | 1 + docs/img/favicon.ico | Bin 0 -> 15406 bytes docs/img/favicon.svg | 332 +++++++++++++ docs/index.md | 1088 ++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 4 + setup.cfg | 2 + 6 files changed, 1427 insertions(+) create mode 100644 docs/img/favicon.ico create mode 100644 docs/img/favicon.svg create mode 100644 docs/index.md create mode 100644 mkdocs.yml diff --git a/.gitignore b/.gitignore index 404c9b86..5cc86b3d 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ __pycache__/ .tox/ *.egg-info/ db.sqlite3 +site/ diff --git a/docs/img/favicon.ico b/docs/img/favicon.ico new file mode 100644 index 0000000000000000000000000000000000000000..9d694282b34d8570e857b94a047b11566a1d44a4 GIT binary patch literal 15406 zcmeHOcUY8H)*m<3s1y+tQACUc3pQely?3!;2Q*m|V`5^m>nFyTNi@I!Lm6ofOrLqD zcZTWkzVtT2FpwD2bWK^;O;n1?(B?aL%=n?J#$=Q1{_)N8%yXIdz2%&H?`gktAdv=; zo*?=7kf04A9ek2Rnnfa!LPGBR4)-OISnzDxw4dIGkw{hkBvLr^fgv#B&U0vf(Z}T? z4XDz^EU!22T!H@nMT4O;F5j}eOta_9YJJRCjiwzdA9uV|N_s*7=JP)WAq_X6R%EeX3f=S>P}US33yRc6DXp z-pMtlxX2di#HT+ijXKhwClzDeN3xukZU|_TmM{~COr`bCQRr zaVSGm_-P}PWqA{*TIQVFCTx|VShh(}q24(XXD@ozGv4KL4Jc85w8(@VdQFuP3{3ys6r2{&St1`y;7zC>h7Rxs-?9V?MUu$A4tkRu%4QIa%eO@`P zqOU3x(Wd+Wbo|In9IWG|4nNL@@v4I9ujvE}?ir8cJPwnMvNaxzqrIID0&%X1p+x%^q`?{$?U2(Sheu?1cED&4OJnQzb96 zNRLGr1TR->7*RFC%*j(234Y6%NnV$Ess0x++L&b7*@#3B|IMZEebh`)ZZr}(OJI)t z212^RZsG-$$l@`R@O4a6uwl4}HgzSR8k)~Y@Ls}A^}iv^7<2dh&|Wm*?;Ww(UVX|@ zQV?m!d6Ul-OVri6Y)7e*40q^tbs=y2WNGe_NOZ^86z?PG&Ux1F-=aI(P)74_g#9>Y zl_b_0nTb}(dqG<4;rF-yc+cMsJbk$xXMfpHBK^u{5QYqJ@)&uD>X zj~eq^y|YT!%>vz5U#{NMU8&pWsx`#9fQFt%%T9ZfdGnm#YMn>TgR7!^N)*xWm8y65 zRusndR_kKlcj$J5#rb1CY|wFL;l`0Aimgtd!HM+V=la9X{Kl)_l?^^^sFx}?JW`$; z5sKuyh_m3&dj4uJi)8)nV)>TaHSl*uVax#}TaAXe5Sw-TjPn&6hj{3Dm`?D#zpov@ zO7=UHn?$<1p90$GC_Rnz4Cv|v#o1{RHFH))A#4772ElTtMYPsgEZuONQ0=U!(8frr zb#V#E1~gb=qCE4q7*`IxP`cqw_DT*2cEf+Z9PwH z`p1Q`sKc(72g`|yz9vGMzv44l&fFd$ZR*#E7O)MW%!I-5(AqeuK8Svj=uGGuEGgJm zUNCJf?1QV2GrzBhvv5s0w$2xqZdi@_fNZ`AvSF*%$Hmv_c89jWnvUYFUw=Yu?zvF5 z@$;{$)_A189(gzW_AMXcwf1b@5V1Gno{hxAH%J?kF0| z%@{dXm^r3fk{#9!cj6`VOv`@hsGhsvhyh!XZ;@>10>7rcOuf6kswhrfr{6cT#kg%~ zgJmbX$r`;6_3!^FmY*2o;n5_BU1*eS8>(ik{YIX*s8^izV!JRiyn~xMx{pm6mV^4i zz3}HzM|5Ht!JSOn2!AAd%7Vxo4Qp;+5qD9CQMj_hDqi0MG<;J9>sA!*S%B6<{m$wB zmH+;<=!LIO(FuGV+)1~T`P19AtT|mpyhT!=VTDz)p{_)_xus0Ab4vd;9`-kzO@0=azkk#s zIdah`IKbD_RzD-jTNEq^)KY71;8 zy4(m~6K`$+TGt463s-Djjre&O?`VCSnY*e+pAYMkpS36x{d8EouSNQ*w~@Esy}p*d zdM3!H27J(Ve%iPzoRr~Lm(e*jdP-9{ot1l_bu}a53k7fb)A%OpJ1rsu`o)Uq~9VKtpe3 zVS0O+>QqOu>{y>!bl5F_dgf|>Rl(X$+1Z7i652Gk{BbCOIm{&AK8OoC*vY;fAfG*0 z=9q7}StI>axe<5ePtKV8FZ!pYl)BGB&^Rqs1w)%k`7@dbiM*ji*>1B4+iMI3T@~7N zCZQm|RV;tct$(!Kn7JUI-Kv6h-75M@xBkJ>#vkRT1Ra61$Ojs_Kt9`nhITAtl$xJ8 zYNn)M;!v<%EH!iXe$QTe+D4ol;=q%gO?bu?2XVH;PH@^GX830#uKLnum4g1gvp2}T z;mTw4&EmIzFk*jp>9{*xYQ~0Zvb<%Bk$pt-A^pTk8cF~SUjhwRdDMV5EMxRd@tNnZ zfP6Ydc~e6?>w4No{B56R;=@)4L1}m3nYWtoJfq!`AJ$kVo@d9ER~t%Xr)@?o-1QS3 zC9-2t7V(ijBX*#-h_`nW%;UB{qMYdyM41tZJZi{QF2#R--l?a@b0|Irh;42O(nqz4 zvO-%?yysch(>4+BM6{H~f7VQpdzy*FuUcBjPdmzUL+v;>-j0iGjiqw8{jD{!SC{K@ zeF@d6wYcJBpH=d%%Zwdbi}Zw!9qqQilI+Mgg|rA8mpTS)H~0$+12lXt zO&j$u_7~BCd}yHI&1T}{KhXLf+8j9hq@x@Ql}e>gKIin zMIpUl#{~)YQiY_USW!`9V7*su$a8{@aF!}RIBu2xrNV?Al<0Z;6Oexj_Vcd)h5Qvx z>hPcSzwW2Q^See*_JXsBbd)ngQOwqaEB^esp6_Kd3vPp+f-lVrSAKW^`I!cGT!?|c zZ-$!jfBq-T^GLtXv(DqTIGg?KIkS2s$fvX3f;rEb8Gt*G@5b$qJI}A}*NA8MIB?GK zMqJ$6P@?EPS1g721G{@(zjofo8P{T^%xWTN^-cKM?k0lc0v+LM#1%a@tHfSoX8$4a zm(50hs-gJIR6Bmw<{&sWn^iocrA2yIULJ>!-=dGbGBv7QXWoM~0IrU!7WsH$xnDE??>Pn+tqK7@J{ec;_#9%p+X(-uD%$6Q zc;8iL*xP3_@9u72E6!v@4@@wWsu>dZMS`64x3Sjq+GWDJ=;GqwA=mQ?Q z>I^aG&zX188Z6tF+qCPS`!#F)5o2t&$fqpwEupw#hqg?!ySH2$i+Ipf8sowd-tkA! z@b9dF7w!|aO0sQWbz$7%{;?i9D~E2K?=QpwJw-D^uK|7jgx2Z*?3ez{OC=3rP`nT1r;^6@AB%i-Vdm(AD7PZd zdSMmBu@)75%4xvd6l&J&WF2RAq=CPbZWONSwusmE0Nz8#WgA~4lsiv>Tq`OI_kC1d zw9i^=h*^bViw47MOuC-}=t>`1%T+8J!;KEzF#L zP>??12R=3A3p$x}kG{($lLm8AhLvOK!#DiI*8ov=*fM$UL|mCa<)DT!lLh(5Rvl-) zOV3~8GGZ%Q0c+wzF7iOJd@E6++#2P%^R>pf)quy^Tkz<6_PI(jXJ8Ay<-&O)>c`K@ zHokKa|9RV|B}YD|Dp>U)U}9eYUhD*{)`ig~)u7xV;wzsra1@g=cs}#=pk07Ne8Eof z%ksp162N67Sz&LWz6cvq6}(sv=itVX^}NOPI>C}LNM5b7jc=Ic>&7F#(VeN##Zkak ze_LaS-Bn}Qv!GlX9gOZi(EF^Tcw5CM_!igs(oIgl+5h%M*@n|!S(ZQ4pVHFTKc{AH zWUKPmv?=nIbVzgNc1yA%FN@DiiAB1GOBp%=Fz18Jgh5q+Yjv@byw7t|yf-5r(4FFD zh6V#!0}#fhJUg{mRS?;yVb1Q;v0t)4PTB2`TcqoQwP3$+@}egJkKPRWwFzuQ7sx|Z zjWPDk%EIlFYb`N;4d$J@?3Nvu0Gsaqq-1mZr^KdKI0v@v5%i-yE@W*x4tRN|hOySE zpf9^7&6#%-;{H|(kOr@h*^=erw(Hr;YyVQ$_r^bRBn3x0b!7OU+oD$ARH zU7Y=5Ct!=`v5YVb@Mb4FC4kwV9z-{id49&omjQ$SMv^tYSDHJ)B}X`X{`6ioW7glH zM&V6_{UQD7Ss2ZM%cFBj)w`U)gA8zaM|DwLXPqJTy3Mq=6?~4Ha1KsK@%EnP;;m_* zhaVbG>#)~-0gtpA`LA3p;_NyDvHnq2!I}~!eR;b)Z+^QtYi1i@K%Icw_d+hGl1B|( zfoSHDNdb(6ft!$=W7NQKen!xTBHGwX;;iTUq&eYzioA%wDd{sPh0Iw~we-m6w7hv6 z)Qp!#N+Exv$5t=0N~20~uyaT^m23C5fqrd=`_xfuh^?$Q?mY%~Da~%(@prp5+FEbe z-2bdS+u&|3qWCbY1hD{PgdD-RUdTnTP%Q#l1n3?DID9GOsBBPU zkq2Vgs>je1mzHTX%V$J*t&vM-Mg${`|g4u>Tz$Ct3>ksY8{& z(xk{+cuA5yr%QZhdI!{=K<*=?lan&?28R+rW~BN6UQBut@J$L}i#`1Gk-dVnQAvm= zetK}AEI0fl)Y@DH?Axhf&S@`X&uiE7mUMxQxMUWsvj7hrCGyQuxKBr*9_TbKz1J>0 zirxpk(@S5QXcQj&7UcSdmb3E|mOI~9mcL|@Ja=I_*oZE$5k11piLHY4aX<1>gHR5q zfsyF5k(D&$6pP~1%}E{B!=w6t#HWp3Da;C~1-XV?%ETLT`qWlceq;||#@#yhyxbze z!iaKgzHhN;ZM;>w;YNvU^LKjs z=PjKGb`IHyYoe?fSB06AZVS@JT?d@|7K`G4otfa(!blwK1RK!}xPLQ`I{XK~yKjqV zq1VLOVYj3?6aFcuPr0VbpV6VBN4n|6;oOY8Wm};J>Ra%czJuJ*%NFh?C`Iwxgl8=pF&18S2lgKlEUvd2X!@+QS9&Q8%Q z=n*r}oJHK3VFv!9lLoQYA7 zO{s$rwvPN2B*&gSl_YJ>K#aa1#%k_m^O+JD?WH z$w?k6!l=U{>EuASKZV93yH2Oy`HgG?@;{J$K=0gkTDo=wE?*yRl&l>3P=fD!VY5iR z>r6bTU*nBMm^@A}tzuo!0Xs|qc!9r>FEI!z|+hD6v zPV*wvrhN^0_IB_&I+;m>6r9t87t$#tw-1Bz8p5p6BN<2i+^J{N9IF+B3f-0L4?i1j;}QqFsgMqCn|Mmhf386@n+@w6VN~-@3pB$ z;j0!se_x!Qw|#?#8I=e5o=br5w@b2Tw~A>~uVESCZCq+d2RCKJbylKJ8#)I@;*iTw zAGv0tYhIr6Tbc0~Yaf@)-FSY$h?YF`AyQZYCx-H9FcwLe+ z%K>>r$SsF=^QmK+*vTXQ3Av+A$Q}L2NE&h*bWP2bL;7o`owv$gxU}LVWcWun`r?+@)*4-#8`BnbYmoue3?0SgAov;T%eU2axH(t-wbsi{jf2aYQ#S zEr7{S3z)-C9~KDpejmy5CSC*o^&4sKjL2U@vu6yBO9waN%JJVp*mq~Lc9Z}>>IgRAOL(k8( zpAebj9XJ)~N4LIhC}aAcE0&LO;rNsFC6YLs6}wSmVD?q(b2}<^89$V3lAMI{gWh6U ze4j=9&LUQ6i{gHdg*hi)x(<|%XZ_m-bMklxkzr{j&TK(? z(?+L1b-~IBYQ{w|c0$nzcj-3dZCmS%td1IeUUx-d#-%b%VkRypn~P<~>nhYo-FH>Z zczKA9y^{+yFW@(?>xHx6N($z<^&cbog&|OvS;D0Tm9i;*nULfE0&GM#o8r|DcJ2n) zxlZuG9gtUjotqgLjQG%kepS*V)~J~?O)B*xyB$}wL0+lTW)=0;nmLtlmsaR$v}Y<6XGD3 zq9Tm!ftwExT`uyUnu!liIEYi7P@miBAkwZk;b(3+h`b+~a0VUha)`Z>7XWw33i zo61zn>P`G@wL10)4-d$8qFP{!nlh6mXGJQq7*al7h_Bn`F5A7H=Y?wH6`c5ok z(kfnRNDDh9u!KwT4bCEyMzSbF@;DUVt0*@P{_C|nJcK%-mgfY_@lXsBVU*2f@$6>e z{kCS}qmE`gsk;eJz1&=UCic_XEWc(VXM&@YeGu~IRd!s~1$ol*jaDcEl1iU;*SKiu z!N|`5`n_@a32U+RXrEd9Rv+Y#ts3Dg?)V)rf?$Lb;6MlH)-4bVHG!S$;-`gHaZ*RU z$w(MJnVUGokCW)N3(nyZ@X*bpj)Yo=(XB#S2wO_=eJK8b=x@Q_J=sDWyUG*f?1H^invQf6+WQ(ke^j-B^>!8b7*=7z@KW{1 zHqcFdP}}VS{c1yXY*KRc(}k=ZZ!7b6xbc6aXCWp{fLgN4{EQbia;U+PNXMdcLUAeN z<}1L?xj?@<`Ji7hh)a?F`su76*tlS9jsCD#OIiH>=F$^?cM#++Z8q{f_!aOm*j~aZ za}d~CGzrR`H&{h$;9FjA;=Wr`bXEYhEp27$BzGMF=-0Ml@w-y1=#94x{C(vH-tK5b zBO0s7n+J7Z^BS}7)AMO*q=8U}bpd4J3Mmf&X%&o_hyBlrZP}gN@L7{94ZCWW5Bmzv;H~@dKTn^ZYcJw)&o<*n zW1watw13|gDf#IpJo^B$4=1YT1)*Bdge#AtdK8-(J6&gD_n)@zd{n)_oX_zvemOtHqc$6rF`< zJkmi;R?&jb_4;R^c7b}XSlk1CQ*WJt0r`uxfrL7F3A&e_{>a|wFn4|TU3=hp_n&If z-XlI5OBA`FUr`N6Z@pQNYtz!7^Z4HXDcT-q-HSN;*~W6tBs(tF+X)$}W9e%smKjhj z2jbyz=J-$aL3Q@cc*bHok(J`W^Bqk%x66)WeRf<5Uu74Z$9eFZXmtiI>UYyA9y)%D7S#{IIZz!qJ-ZhCj^AR= bKYEVKRcQn7_dWh-8vcL1|KEW}?ZE#6M?m4a literal 0 HcmV?d00001 diff --git a/docs/img/favicon.svg b/docs/img/favicon.svg new file mode 100644 index 00000000..8fd02fdc --- /dev/null +++ b/docs/img/favicon.svg @@ -0,0 +1,332 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..d8727ab4 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,1088 @@ +# freezeyt + +Static web page generator created by the Czech Python community. + + +## What this does + +Freezeyt is a static webpage *freezer*. +It takes a Python web application and turns it into a set of files +that can be served by a simple server like [GitHub Pages] or +Python's [http.server]. + +[GitHub Pages]: https://docs.github.com/en/free-pro-team@latest/github/working-with-github-pages/about-github-pages +[http.server]: https://docs.python.org/3/library/http.server.html + +Freezeyt is compatible with all Python web frameworks that use the common +[Web Server Gateway Interface] (WSGI) + +[Web Server Gateway Interface]: https://www.python.org/dev/peps/pep-3333/ + + +## Installation + +Freezeyt requires Python 3.6 or above. + +It is highly recommended to create and activate a separate virtual +environment for this project. +You can use [`venv`], `virtualenv`, Conda, containers or any other kind +of virtual environment. + +[`venv`]: https://docs.python.org/3/library/venv.html?highlight=venv#module-venv + +The tool can be installed using: + +``` +$ python -m pip install . +``` + + +## Usage + +To use freezeyt, you need a Python web application. +You can use the [example Flask app]. + +[example Flask app]: https://flask.palletsprojects.com/en/1.1.x/quickstart/ + +Both the application and Freezeyt must be importable (installed) +in your environment. + +Run freezeyt with the name of your application and the +output directory. For example: + +```shell +$ python -m freezeyt my_app _build +``` + +Freezeyt may overwrite the build directory (here, `_build`), +removing all existing files from it. + +For more options, see Configuration below. + + +### Python API + +Freezeyt also has a Python API, the `freeze` function +that takes an application to freeze and a configuration dict: + +```python +from freezeyt import freeze +freeze(app, config) +``` + +The `config` should be a dict as if read from a YAML configuration +file (see Configuration below). + +From asynchronous code running in an `asyncio` event loop, +you can call `freeze_async` instead of `freeze`. + + +### Middleware + +Some of Freezeyt's functionality is available as a WSGI middleware. +To use it, wrap your application in `freezeyt.Middeleware`. For example: + +```python +from freezeyt import Middleware + +config = {} # use a configuration dict as for `freeze(app, config)` + +app = Middleware(app, config) +``` + + +## Configuration + +While common options can be given on the command line, +you can have full control over the freezing process with a YAML +configuration file or a variable with the configuration. +You can specify a config file using the `-c/--config` option, +for example: + +```shell +$ python -m freezeyt my_app _build -c freezeyt.yaml +``` + +The configuration variable should be a dictionary. +To pass the config variable, use the `-C/--import-config` option, +for example: + +```shell +$ python -m freezeyt my_app _build -C my_app:freezeyt_config +``` + +Here is an example configuration file: + +```yaml +output: ./_build/ # The website will be saved to this directory +prefix: https://mysite.example.com/subpage/ +extra_pages: + # Let freezeyt know about URLs that are not linked from elsewhere + /robots.txt + /easter-egg.html +extra_files: + # Include additional files in the output: + # Static files + static: + copy_from: static/ + # Web host configuration + CNAME: mysite.example.com + ".nojekyll": '' + googlecc704f0f191eda8f.html: + copy_from: google-verification.html +status_handlers: + # If a redirect page (HTTP status 3xx) is found, warn but don't fail + "3xx": warn +``` + +The following options are configurable: + +### App + +The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. +When the module is specified both by the command line and the config file +an error is raised. + +Examples: + +Freezeyt looks for the variable `app` inside the module by default. +```yaml +app: app_module +``` + +If `app` is in a submodule, separate package names with a dot: +```yaml +app: folder1.folder2.app_module +``` + +A different variable name can be specified by using `:`. +```yaml +app: app_module:wsgi_application +``` + +If the variable is an attribute of some namespace, use dots in the variable name: + +```yaml +app: app_module:namespace.wsgi_application +``` + +When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. + +### Output + +To outupt the frozen website to a directory, specify +the directory name: + +```yaml +output: ./_build/ +``` + +Or use the full form – using the `dir` *saver*: + +```yaml +output: + type: dir + dir: ./_build/ +``` + +If output is not specified in the configuration file, +you must specify the output directory on the command line. +There are two ways to specify the output on the command line: either by the `--output` (`-o`) option or as a second positional argument. + +The output must be specified just by one way otherwise is an error. + +If there is any existing content in the output directory, +freezeyt will either remove it (if the content looks like a previously +frozen website) or raise an error. +Best practice is to remove the output directory before freezing. + + +#### Output to dict + +For testing, `freezeyt` can output to a dictionary rather than save +files to the disk. +This can be configured with: + +```yaml +output: + type: dict +``` + +In this case, the `freeze()` function returns a dictionary of filenames +and their contents. +For example, a site with `/`, `/second_page/` and `/images/smile.png` +will be represented as: + +```python +{ + 'index.html': b'...', + 'second_page': { + 'index.html': b'...', + }, + 'images': { + 'smile.png': b'\x89PNG\r\n\x1a\n\x00...', + }, +} +``` + +This is not useful in the CLI, as the return value is lost. + + +### Prefix + +The URL where the application will be deployed can be +specified with: + +```yaml +prefix: http://localhost:8000/ +``` +or +```yaml +prefix: https://mysite.example.com/subpage/ +``` + +Freezeyt will freeze all pages starting with `prefix` that +it finds. + +The prefix can also be specified on thecommand line with e.g.: +`--prefix=http://localhost:8000/`. +The CLI argument has priority over the config file. + + +### Extra pages + +URLs of pages that are not reachable by following links from the homepage +can specified as “extra” pages in the configuration: + +```yaml +extra_pages: + - /extra/ + - /extra2.html +``` + +Freezeyt will handle these pages as if it found them by following links. +(For example, by default it will follow links in extra pages.) + +Extra pages may also be given on the command line, +e.g. `--extra-page /extra/ --extra-page /extra2.html`. +The lists from CLI and the config file are merged together. + +You can also specify extra pages using a Python generator, +specified using a module name and function name as follows: + +```yaml +extra_pages: + - generator: my_app:generate_extra_pages +``` + +The `generate_extra_pages` function should take the application +as argument and return an iterable of URLs. + +When using the Python API, a generator for extra pages can be specified +directly as a Python object, for example: + +```python +config = { + ... + 'extra_pages': [{'generator': my_generator_function}], +} +another_config = { + ... + 'extra_pages': [my_generator_function], +} +``` + + +### Extra files + +Extra files to be included in the output can be specified, +along with their content. + +These files are not considered part of the app; freezeyt will not try to find +links in them. + +This is useful for configuration of your static server. +(For pages that are part of your website, we recommend +adding them to your application rather than as extra files.) + +If you specify backslashes in `url part`, `freezeyt` convert them to forward slashes. + +For example, the following config will add 3 files to +the output: + +```yaml +extra_files: + CNAME: mysite.example.com + ".nojekyll": '' + config/xyz: abc +``` + +You can also specify extra files using Base64 encoding or +a file path, like so: + + +```yaml +extra_files: + config.dat: + base64: "YWJjZAASNA==" + config2.dat: + copy_from: included/config2.dat +``` + +It's possible to recursively copy an entire directory using `copy_from`, +as in: + + +```yaml +extra_files: + static: + copy_from: static/ +``` + +Extra files cannot be specified on the CLI. + + +### Clean up + +If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. +This prevents, for example, uploading incomplete results to a web hosting by mistake. + +If you want to keep the incomplete directory (for example, +to help debugging), you can use the `--no-cleanup` switch +or the `cleanup` key in the configuration file: + +```yaml +cleanup: False +``` + +The command line switch has priority over the configuration. +Use `--no-cleanup` to override `cleanup: False` from the config. + + +### Fail fast + +Fail fast mode stops the freezing of the app when the first error occurs. + +As with most settings, fail fast can be set both in the configuration file and in the command line using switches (the command line always overrides the configuration file). +The fail fast is defined as `boolean` option. + +If you want to specified fail fast in configuration file, use key `fail_fast` with + `boolean` values `True` or `False`: + +```yaml +fail_fast: True +``` + +If you want to specified fail fast from command line, use switches + `--fail-fast` (short `-x`) resp. `--no-fail-fast` to disable it: + +```shell +$ freezeyt app -o output -x +``` + + +### Github Pages Plugin + +To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. + +By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. + +Configuration example: +```yaml +gh_pages: True +``` + +To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. +You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: +```shell +git fetch output_dir gh-pages +git branch --force gh-pages FETCH_HEAD +``` +Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. + +### Comparison of MIME type and file type + +Freezeyt checks whether the file extensions in its output +correspond to the MIME types served by the app. +If there's a mismatch, freezeyt fails, because this means a server +wouldn't be able to serve the page correctly. + +This funtionality is provided by `freezeyt.Middleware`. + +#### Default MIME type + +It is possible to specify the MIME type used for files without an extension. +For example, if your server of static pages defaults to plain text files, +use: + +```yaml +default_mimetype=text/plain +``` + +If the default MIME type isn't explicitly configured in YAML configuration, +then the `freezeyt` uses value `application/octet-stream`. + +The default mimetype cannot be specified on the CLI. + +#### Recognizing file types from extensions + +There is possibility to modify the way how to determine file type +from file extension. +You can setup your own `get_mimetype` function. + +Freezeyt will register your own function, if you specify it in configuration +YAML file as: + +```yaml +get_mimetype=module:your_function +``` + +If the `get_mimetype` is not defined in configuration file, +then `freezeyt` calls the python function `mimetypes.guess_type` +and uses the mimetype (the first element) it returns. + +`get_mimetype` can be defined as: +* strings in the form `"module:function"`, which name the function to call, +* Python functions (if configuring `freezeyt` from Python, e.g. as a `dict`, + rather than YAML). + +The `get_mimetype`: +* gets one argument - the `filepath` as `str` + +* returns file MIME types as a `list` of MIME types + (e.g. `["text/html"]` or `["audio/wav", "audio/wave"]`). + +If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mimetype` +(see *Default MIME type* above). + +The get_mimetype function cannot be specified on the CLI. + + +#### Using a mime-db database + +There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), +or a database with the same structure. +(This is the database used by GitHub Pages). +The database will be used to get file MIME type from file suffix. + +To use this database, add the path to the JSON file to `freezeyt` configuration: +```yaml +mime_db_file=path/to/mime-db.json +``` +This is equivalent to setting `get_mimetype` to a function that maps +extensions to filetypes according to the database. + +The mime_db file cannot be specified on the CLI. + + +### Progress bar and logging + +The CLI option `--progress` controls what `freezeyt` outputs as it +handles pages: + +* `--progress=log`: Output a message about each frozen page to stdout. +* `--progress=bar`: Draw a status bar in the terminal. Messages about + each frozen page are *also* printed to stdout. +* `--progress=none`: Don't do any of this. + +The default is `bar` if stdout is a terminal, and `log` otherwise. + +It is possible to configure this in the config file using the plugins +`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. +See below on how to enable plugins. + +### Configuration version + +To ensure that your configuration will work unchanged in newer versions of freezeyt, +you should add the current version number, `1`, to your configuration like this: + +```yaml +version: 1 +``` + +This is not mandatory. If the version is not given, the configuration may +not work in future versions of freezeyt. + +The version parameter is not accepted on the command line. + +### Plugins + +It is possible to extend `freezeyt` with *plugins*, either ones that +ship with `freezeyt` or external ones. + +Plugins are added using configuration like: + +```yaml +plugins: + - freezeyt.progressbar:ProgressBar + - mymodule:my_plugin +``` + +#### Custom plugins + +A plugin is a function that `freezeyt` will call before starting to +freeze pages. + +It is passed a `FreezeInfo` object as argument (see the `start` hook below). +Usually, the plugin will call `freeze_info.add_hook` to register additional +functions. + + +### Hooks + +It is possible to register *hooks*, functions that are called when +specific events happen in the freezing process. + +For example, if `mymodule` defines functions `start` and `page_frozen`, +you can make freezeyt call them using this configuration: + +```yaml +hooks: + start: + - mymodule:start + page_frozen: + - mymodule:page_frozen +``` + +When using the Python API, a function can be used instead of a name +like `mymodule:start`. + +#### `start` + +The function will be called when the freezing process starts, +before any other hooks. + +It is passed a `FreezeInfo` object as argument. +The object has the following attributes: + +* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. + If that URL was frozen already, or is outside the `prefix`, does nothing. + If you add a `reason` string, it will be used in error messages as the reason + why the added URL is being handled. +* `add_hook(hook_name, callable)`: Register an additional hook function. +* `total_task_count`: The number of pages `freezeyt` currently “knows about” – + ones that are already frozen plus ones that are scheduled to be frozen. +* `done_task_count`: The number of pages that are done (either successfully + frozen, or failed). +* `failed_task_count`: The number of pages that failed to freeze. + +#### `page_frozen` + +The function will be called whenever a page is processed successfully. +It is passed a `TaskInfo` object as argument. +The object has the following attributes: + +* `get_a_url()`: returns a URL of the page, including `prefix`. + Note that a page may be reachable via several URLs; this function returns + an arbitrary one. +* `path`: the relative path the content is saved to. +* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. +* `exception`: for failed tasks, the exception raised; + `None` otherwise. +* `reasons`: A list of strings explaining why the given page was visited. + (Note that as the freezing progresses, new reasons may be added to + existing tasks.) + + +#### `page_failed` + +The function will be called whenever a page is not saved due to an +exception. +It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). + + +#### `success` + +The function will be called after the app is successfully frozen. +It is passed a `FreezeInfo` object as argument (see the `start` hook). + + +### Freeze actions + +By default, `freezeyt` will save the pages it finds. +You can instruct it to instead ignore certain pages, or treat them as errors. +This is most useful as a response to certain HTTP statuses (e.g. treat all +`404 NOT FOUND` pages as errors), but can be used independently. + +To tell `freezeyt` what to do from within the application (or middleware), +set the `Freezeyt-Action` HTTP header to one of these values: + +* `'save'`: `freezeyt` will save the body of the page. +* `'ignore'`: `freezeyt` will not save any content for the page +* `'warn'`: will save the content and send warn message to stdout +* `'follow'`: `freezeyt` will save content from the redirected location + (this requires a `Location` header, which is usually added for redirects). + Redirects to external pages are not supported. +* `'error'`: fail; the page will not be saved and `freeze()` will raise + an exception. + + +#### Status handling + +If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to +do based on the status. +By default, `200 OK` pages are saved and any others cause errors. +The behavior can be customized using the `status_handlers` setting. +For example, to ignore pages with the `404 NOT FOUND` status, set the +`404` handler to `'ignore'`: + +```yaml +status_handlers: + '404': ignore +``` + +For example, `status_handlers` would be specified as: + +```yaml +status_handlers: + '202': warn + '301': follow + '404': ignore + '418': my_module:custom_action # see below + '429': ignore + '5xx': error +``` + +Note that the status code must be a string, so it needs to be quoted in the YAML file. + +A range of statuses can be specified as a number (`1-5`) followed by lowercase `xx`. +(Other "wildcards" like `50x` are not supported.) + +Status handlers cannot be specified in the CLI. + +#### Custom actions + +You can also define a custom action in `status_handlers` as: +* a string in the form `'my_module:custom_action'`, which names a handler + function to call, +* a Python function (if configuring `freezeyt` from Python rather than from + YAML). + +The action function takes one argument, `task` (TaskInfo): information about the freezing task. +See the `TaskInfo` hook for a description. +Freezeyt's default actions, like `follow`, can be imported from `freezeyt.actions` +(e.g. `freezeyt.actions.follow`). +A custom action should call one of these default actions and return the return value from it. + + +### URL finding + +`freezeyt` discovers new links in the application by URL finders. URL finders +are functions whose goal is to find url of specific MIME type. +`freezeyt` offers different configuration options to use URL finders: + +* use predefined URL finders for `text/html` or `text/css` (default), +* define your own URL finder as your function, +* turn off some of finders (section below) + +Example of configuration: + +```yaml +url_finders: + text/html: get_html_links + text/css: get_css_links +``` + +Keys in the `url_finders` dict are MIME types; + +Values are URL finders, which can be defined as: +* strings in the form `"module:function"`, which name the finder + function to call, +* strings like `get_html_links`, which name a function from the + `freezeyt.url_finders` module, or +* Python functions (if configuring `freezeyt` from Python rather than + YAML). + + +An URL finder gets these arguments: +* page content `BinaryIO`, +* the absolute URL of the page, as a `string`, +* the HTTP headers, as a list of tuples (WSGI). + +The function should return an iterator of all URLs (as strings) found +in the page's contents, as they would appear in `href` or `src` attributes. +Specifically: + +- The URLs can be relative. +- External URLs (i.e. those not beginning with `prefix`) should be included. + +Finder functions may be asynchronous: +- The function can be defined with `async def` (i.e. return a + coroutine). If it is, freezeyt will use the result after `await`. +- The function may be an asynchronous generator (defined with `async def` + and use `yield`). If so, freezeyt will use async iteration to handle it. + +The `freezeyt.url_finders` module includes: +- `get_html_links`, the default finder for HTML +- `get_css_links`, the default finder for CSS +- `get_html_links_async` and `get_css_links_async`, asynchronous variants + of the above +- `none`, a finder that doesn't find any links. + +URL finders cannot be specified in the CLI. + +#### URL finder header + +You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. +If given, it overrides the `url_finders` configuration. + +#### Default `get_html_links` + +The default URL finder for HTML pages looks in `src` and `href` attributes +of all tags in the document. +It currently does not handle other links, such as embedded CSS, but it +may be improved in the future. + +#### Default `get_css_links` + +The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all +links in a stylesheet. + +#### Disabling default URL finders + +If a finder is not explictly specified in the configuration file, `freezeyt` will use the +default for certain MIME type. For example, if you specify +`text/html: my_custom_finder` only, `freezeyt` will use the default finder +for `text/css`. + +You can disable this behaviour: + +```yaml +use_default_url_finders: false +``` + + +#### Finding URLs in Link headers + +By default, `freezeyt` will follow URLs in `Link` HTTP headers. +To disable this, specify: + +```yaml +urls_from_link_headers: false +``` + + +### Path generation + +It is possible to customize the filenames that URLs are saved under +using the `url_to_path` configuration key, for example: + +```yaml +url_to_path: my_module:url_to_path +``` + +The value can be: +* a strings in the form `"module:function"`, which names the + function to call (the function can be omitted along with the colon, + and defaults to `url_to_path`), or +* a Python function (if configuring `freezeyt` from Python rather than + YAML). + +The function receives the *path* of the URL to save, relative to the `prefix`, +and should return a path to the saved file, relative to the build directory. + +The default function, available as `freezeyt.url_to_path`, adds `index.html` +if the URL path ends with `/`. + +`url_to_path` cannot be specified in the CLI. + + +### Middleware static mode + +When using the `freezeyt` middleware, you can enable *static mode*, +which simulates behaviour after the app is saved to static pages: + +```yaml +static_mode: true +``` + +Currently in static mode: +- HTTP methods other than GET and HEAD are disallowed. +- URL parameters are removed +- The request body is discarded + +Other restrictions and features may be added in the future, without regard +to backwards compatibility. +The static mode is intended for interactive use -- testing your app without +having to freeze all of it after each change. + + +## Examples of CLI usage + +```shell +$ python -m freezeyt my_app _build/ +``` + +```shell +$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ +``` + +```shell +$ python -m freezeyt my_app _build/ -c config.yaml +``` + +```shell +$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/ +``` + +```shell +$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/ --config path/to/config.yaml +``` + + +## Contributing + +Are you interested in this project? Awesome! +Anyone who wants to be part of this project and who's willing +to help us is very welcome. +Just started with Python? Good news! +We're trying to target mainly the beginner Pythonistas who +are seeking opportunities to contribute to (ideally open source) +projects and who would like to be part of an open source community +which could give them a head start in their +(hopefully open source :)) programming careers. + +Soo, what if you already have some solid Python-fu? +First, there's always something new to learn, and second, +we'd appreciate if you could guide the “rookies” and pass on +some of the knowledge onto them. + +Contributions, issues and feature requests are welcome. +Feel free to check out the [issues] page if you'd like to +contribute. + +[issues]: https://github.com/encukou/freezeyt/issues + + +## How to contribute + +1. Clone this repository to your local computer: + +```shell +$ git clone https://github.com/encukou/freezeyt +``` + +2. Then fork this repo to your GitHub account +3. Add your forked repo as a new remote to your local computer: + +```shell +$ git remote add https://github.com//freezeyt +``` + +4. Create a new branch at your local computer + +```shell +$ git branch +``` + +5. Switch to your new branch + +```shell +$ git switch +``` + +6. Update the code +7. Push the changes to your forked repo on GitHub + +```shell +$ git push +``` + +8. Finally, make a pull request from your GitHub account to origin + + +### Installing for development + +`freezeyt` can be installed from the current directory: + +```shell +$ python -m pip install -e . +``` + +It also has several groups of extra dependecies: +* `blog` for the project blog +* `dev` for development and running tests +* `typecheck` for mypy type checks + +Each group can be installed separately: + +```shell +$ python -m pip install -e ."[typecheck]" +``` + +or you can install more groups at once: +```shell +$ python -m pip install -e ."[blog, dev, typecheck]" +``` + + +### Using an in-development copy of freezeyt + +* Set `PYTHONPATH` to the directory with `freezeyt`, for example: + * Unix: `export PYTHONPATH="/home/name/freezeyt"` + * Windows: `set PYTHONPATH=C:\Users\Name\freezeyt` + +* Install the web application you want to freeze. Either: + * install the application using `pip`, if possible, or + * install the application's dependencies and `cd` to the app's directory. + +* Run freezeyt, for example: + * `python -m freezeyt demo_app_url_for _build --prefix http://freezeyt.test/foo/` + + + +### Tests + +For testing the project it's necessary to install additional requirements: + +``` +$ python -m pip install .[dev] +``` + +To run tests in your current environment, use pytest: + +``` +$ python -m pytest +``` + +To run tests with multiple Python versions (if you have them installed), +install `tox` using `python -m pip install tox` and run it: + +``` +$ tox +``` + +#### Environ variables for tests + +Some test scenarios compare freezeyt's results with expected output. +When the files with expected output don't exist yet, +they can be created by setting the environment variable +`TEST_CREATE_EXPECTED_OUTPUT` to `1`: + +**Unix** + +```shell +$ export TEST_CREATE_EXPECTED_OUTPUT=1 +``` + +**Windows** + +```shell +> set TEST_CREATE_EXPECTED_OUTPUT=1 +``` + +If you set the variable to any different value or leave it unset +then the files will not be recreated +(tests will fail if the files are not up to date). + +When output changes, you need to first delete the expected output, +regenerate it by running tests with `TEST_CREATE_EXPECTED_OUTPUT=1`, +and check that the difference is correct. + +### Tools and technologies used + +* [PEP 3333 - Python WSGI](https://www.python.org/dev/peps/pep-3333/) +* [flask](https://flask.palletsprojects.com/en/1.1.x/) +* [pytest](https://docs.pytest.org/en/latest/) +* [html5lib](https://html5lib.readthedocs.io/en/latest/) + + +### How to watch progress +Unfortunately our progress of development can be watched only in Czech language. + +Watch the progress: + +* [Youtube playlist](https://www.youtube.com/playlist?list=PLFt-PM7J_H3EU5Oez3ZSVjY5pZJttP2lT) + +Other communication channels and info can be found here: +* [Google doc in Czech](https://tinyurl.com/freezeyt) + + +### Freezeyt Blog + +We keep a blog about the development of Freezeyt. +It is available [here](https://encukou.github.io/freezeyt/). + +**Be warned:** some of it is in the Czech language. + +#### Blog development + +The blog was tested on Python version 3.8. + +The blog is a Flask application. +To run it, install additional dependecies with +`python -m pip install .[blog]`. +Then, set the environment variable `FLASK_APP` to the path of the +blog app. +Also set `FLASK_ENV` to "development" for easier debugging. +Then, run the Flask server. + +1. On Microsoft Windows: + +```shell +> python -m pip install .[blog] +> set FLASK_APP=freezeyt_blog/app.py +> set FLASK_ENV=development +> flask run +``` + +2. On UNIX: + +```shell +$ python -m pip install .[blog] +$ export FLASK_APP=freezeyt_blog/app.py +$ export FLASK_ENV=development +$ flask run +``` + +The URL where your blog is running will be printed on the terminal. + +Once you're satisfied with how the blog looks, you can freeze it with: + +```shell +$ python -m freezeyt freezeyt_blog.app freezeyt_blog/build +``` + +#### Adding new articles to freezeyt blog + +Articles are writen in the `Markdown` language. + +**Article** - save to directory `../freezeyt/freezeyt_blog/articles` + +**Images to articles** - save to directory `../freezeyt/freezeyt_blog/static/images` + +If te files are saved elsewhere, the blog will not work correctly. + + +## History + +### Why did the project start? + +The Czech Python community uses a lot of static web pages that +are generated from a web application for community purposes. +For example, organizing and announcing workshops, courses, +or meetups. + +The community has been so far relying on [Frozen Flask] and [elsa] +in order to generate the static web content. +The new [freezer] ought to be used with any arbitrary Python Web +application framework ([Flask], [Django], [Tornado], etc.). +So the community won't be limited by one web app technology for +generating static pages anymore. + +[Frozen Flask]: https://pythonhosted.org/Frozen-Flask/ +[elsa]: https://github.com/pyvec/elsa/ +[freezer]: https://github.com/encukou/freezeyt +[Flask]: https://flask.palletsprojects.com/en/1.1.x/ +[Django]: https://www.djangoproject.com/ +[Tornado]: https://www.tornadoweb.org/en/stable/ + + +## Authors +See GitHub history for all [contributors](https://github.com/encukou/freezeyt/graphs/contributors). + + +## License + +This project is licensed under the [MIT License](LICENCE.MIT). +May it serve you well. diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 00000000..9fd78617 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,4 @@ +site_name: Freezeyt +nav: + - Freezeyt: index.md +theme: readthedocs diff --git a/setup.cfg b/setup.cfg index 2aca4918..17bbc1f9 100644 --- a/setup.cfg +++ b/setup.cfg @@ -40,6 +40,8 @@ dev = falcon freezegun packaging +docs = + mkdocs blog = flask markdown-it-py From e682944f3c2136fd18ed06a6d2af4f459d24c031 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 11 Jun 2024 19:01:47 +0200 Subject: [PATCH 02/16] Add some highlighting --- docs/css/pygments-friendly.css | 78 ++++++++++++++++++++++++++++++ docs/css/pygments-nord.css | 87 ++++++++++++++++++++++++++++++++++ docs/css/pygments.css | 11 +++++ docs/index.md | 6 ++- mkdocs.yml | 18 ++++++- setup.cfg | 1 + 6 files changed, 198 insertions(+), 3 deletions(-) create mode 100644 docs/css/pygments-friendly.css create mode 100644 docs/css/pygments-nord.css create mode 100644 docs/css/pygments.css diff --git a/docs/css/pygments-friendly.css b/docs/css/pygments-friendly.css new file mode 100644 index 00000000..3c8a979f --- /dev/null +++ b/docs/css/pygments-friendly.css @@ -0,0 +1,78 @@ +[data-bs-theme="light"] { + --string-color: #4070a0; + pre { line-height: 125%; } + td.linenos .normal { color: #666666; background-color: transparent; padding-left: 5px; padding-right: 5px; } + span.linenos { color: #666666; background-color: transparent; padding-left: 5px; padding-right: 5px; } + td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } + span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } + .codehilite .hll { background-color: #ffffcc } + .codehilite { background: #f0f0f0; } + .codehilite .c { color: #60a0b0; font-style: italic } /* Comment */ + .codehilite .err { border: 1px solid #FF0000 } /* Error */ + .codehilite .k { color: #007020; font-weight: bold } /* Keyword */ + .codehilite .o { color: #666666 } /* Operator */ + .codehilite .ch { color: #60a0b0; font-style: italic } /* Comment.Hashbang */ + .codehilite .cm { color: #60a0b0; font-style: italic } /* Comment.Multiline */ + .codehilite .cp { color: #007020 } /* Comment.Preproc */ + .codehilite .cpf { color: #60a0b0; font-style: italic } /* Comment.PreprocFile */ + .codehilite .c1 { color: #60a0b0; font-style: italic } /* Comment.Single */ + .codehilite .cs { color: #60a0b0; background-color: #fff0f0 } /* Comment.Special */ + .codehilite .gd { color: #A00000 } /* Generic.Deleted */ + .codehilite .ge { font-style: italic } /* Generic.Emph */ + .codehilite .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */ + .codehilite .gr { color: #FF0000 } /* Generic.Error */ + .codehilite .gh { color: #000080; font-weight: bold } /* Generic.Heading */ + .codehilite .gi { color: #00A000 } /* Generic.Inserted */ + .codehilite .go { color: #888888 } /* Generic.Output */ + .codehilite .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */ + .codehilite .gs { font-weight: bold } /* Generic.Strong */ + .codehilite .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ + .codehilite .gt { color: #0044DD } /* Generic.Traceback */ + .codehilite .kc { color: #007020; font-weight: bold } /* Keyword.Constant */ + .codehilite .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */ + .codehilite .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */ + .codehilite .kp { color: #007020 } /* Keyword.Pseudo */ + .codehilite .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */ + .codehilite .kt { color: #902000 } /* Keyword.Type */ + .codehilite .m { color: #40a070 } /* Literal.Number */ + .codehilite .s { color: var(--string-color) } /* Literal.String */ + .codehilite .na { color: var(--string-color) } /* Name.Attribute */ + .codehilite .nb { color: #007020 } /* Name.Builtin */ + .codehilite .nc { color: #0e84b5; font-weight: bold } /* Name.Class */ + .codehilite .no { color: #60add5 } /* Name.Constant */ + .codehilite .nd { color: #555555; font-weight: bold } /* Name.Decorator */ + .codehilite .ni { color: #d55537; font-weight: bold } /* Name.Entity */ + .codehilite .ne { color: #007020 } /* Name.Exception */ + .codehilite .nf { color: #06287e } /* Name.Function */ + .codehilite .nl { color: #002070; font-weight: bold } /* Name.Label */ + .codehilite .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */ + .codehilite .nt { color: #062873; font-weight: bold } /* Name.Tag */ + .codehilite .nv { color: #bb60d5 } /* Name.Variable */ + .codehilite .ow { color: #007020; font-weight: bold } /* Operator.Word */ + .codehilite .w { color: #bbbbbb } /* Text.Whitespace */ + .codehilite .mb { color: #40a070 } /* Literal.Number.Bin */ + .codehilite .mf { color: #40a070 } /* Literal.Number.Float */ + .codehilite .mh { color: #40a070 } /* Literal.Number.Hex */ + .codehilite .mi { color: #40a070 } /* Literal.Number.Integer */ + .codehilite .mo { color: #40a070 } /* Literal.Number.Oct */ + .codehilite .sa { color: var(--string-color) } /* Literal.String.Affix */ + .codehilite .sb { color: var(--string-color) } /* Literal.String.Backtick */ + .codehilite .sc { color: var(--string-color) } /* Literal.String.Char */ + .codehilite .dl { color: var(--string-color) } /* Literal.String.Delimiter */ + .codehilite .sd { color: var(--string-color); font-style: italic } /* Literal.String.Doc */ + .codehilite .s2 { color: var(--string-color) } /* Literal.String.Double */ + .codehilite .se { color: var(--string-color); font-weight: bold } /* Literal.String.Escape */ + .codehilite .sh { color: var(--string-color) } /* Literal.String.Heredoc */ + .codehilite .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */ + .codehilite .sx { color: #c65d09 } /* Literal.String.Other */ + .codehilite .sr { color: #235388 } /* Literal.String.Regex */ + .codehilite .s1 { color: var(--string-color) } /* Literal.String.Single */ + .codehilite .ss { color: #517918 } /* Literal.String.Symbol */ + .codehilite .bp { color: #007020 } /* Name.Builtin.Pseudo */ + .codehilite .fm { color: #06287e } /* Name.Function.Magic */ + .codehilite .vc { color: #bb60d5 } /* Name.Variable.Class */ + .codehilite .vg { color: #bb60d5 } /* Name.Variable.Global */ + .codehilite .vi { color: #bb60d5 } /* Name.Variable.Instance */ + .codehilite .vm { color: #bb60d5 } /* Name.Variable.Magic */ + .codehilite .il { color: #40a070 } /* Literal.Number.Integer.Long */ +} diff --git a/docs/css/pygments-nord.css b/docs/css/pygments-nord.css new file mode 100644 index 00000000..51e2ec86 --- /dev/null +++ b/docs/css/pygments-nord.css @@ -0,0 +1,87 @@ +[data-bs-theme="dark"] { + pre { line-height: 125%; } + td.linenos .normal { color: #D8DEE9; background-color: #242933; padding-left: 5px; padding-right: 5px; } + span.linenos { color: #D8DEE9; background-color: #242933; padding-left: 5px; padding-right: 5px; } + td.linenos .special { color: #242933; background-color: #D8DEE9; padding-left: 5px; padding-right: 5px; } + span.linenos.special { color: #242933; background-color: #D8DEE9; padding-left: 5px; padding-right: 5px; } + .codehilite .hll { background-color: #3B4252 } + .codehilite { background: #2E3440; color: #d8dee9 } + .codehilite .c { color: #616e87; font-style: italic } /* Comment */ + .codehilite .err { color: #bf616a } /* Error */ + .codehilite .esc { color: #d8dee9 } /* Escape */ + .codehilite .g { color: #d8dee9 } /* Generic */ + .codehilite .k { color: #81a1c1; font-weight: bold } /* Keyword */ + .codehilite .l { color: #d8dee9 } /* Literal */ + .codehilite .n { color: #d8dee9 } /* Name */ + .codehilite .o { color: #81a1c1; font-weight: bold } /* Operator */ + .codehilite .x { color: #d8dee9 } /* Other */ + .codehilite .p { color: #eceff4 } /* Punctuation */ + .codehilite .ch { color: #616e87; font-style: italic } /* Comment.Hashbang */ + .codehilite .cm { color: #616e87; font-style: italic } /* Comment.Multiline */ + .codehilite .cp { color: #5e81ac; font-style: italic } /* Comment.Preproc */ + .codehilite .cpf { color: #616e87; font-style: italic } /* Comment.PreprocFile */ + .codehilite .c1 { color: #616e87; font-style: italic } /* Comment.Single */ + .codehilite .cs { color: #616e87; font-style: italic } /* Comment.Special */ + .codehilite .gd { color: #bf616a } /* Generic.Deleted */ + .codehilite .ge { color: #d8dee9; font-style: italic } /* Generic.Emph */ + .codehilite .ges { color: #d8dee9; font-weight: bold; font-style: italic } /* Generic.EmphStrong */ + .codehilite .gr { color: #bf616a } /* Generic.Error */ + .codehilite .gh { color: #88c0d0; font-weight: bold } /* Generic.Heading */ + .codehilite .gi { color: #a3be8c } /* Generic.Inserted */ + .codehilite .go { color: #d8dee9 } /* Generic.Output */ + .codehilite .gp { color: #616e88; font-weight: bold } /* Generic.Prompt */ + .codehilite .gs { color: #d8dee9; font-weight: bold } /* Generic.Strong */ + .codehilite .gu { color: #88c0d0; font-weight: bold } /* Generic.Subheading */ + .codehilite .gt { color: #bf616a } /* Generic.Traceback */ + .codehilite .kc { color: #81a1c1; font-weight: bold } /* Keyword.Constant */ + .codehilite .kd { color: #81a1c1; font-weight: bold } /* Keyword.Declaration */ + .codehilite .kn { color: #81a1c1; font-weight: bold } /* Keyword.Namespace */ + .codehilite .kp { color: #81a1c1 } /* Keyword.Pseudo */ + .codehilite .kr { color: #81a1c1; font-weight: bold } /* Keyword.Reserved */ + .codehilite .kt { color: #81a1c1 } /* Keyword.Type */ + .codehilite .ld { color: #d8dee9 } /* Literal.Date */ + .codehilite .m { color: #b48ead } /* Literal.Number */ + .codehilite .s { color: #a3be8c } /* Literal.String */ + .codehilite .na { color: #8fbcbb } /* Name.Attribute */ + .codehilite .nb { color: #81a1c1 } /* Name.Builtin */ + .codehilite .nc { color: #8fbcbb } /* Name.Class */ + .codehilite .no { color: #8fbcbb } /* Name.Constant */ + .codehilite .nd { color: #d08770 } /* Name.Decorator */ + .codehilite .ni { color: #d08770 } /* Name.Entity */ + .codehilite .ne { color: #bf616a } /* Name.Exception */ + .codehilite .nf { color: #88c0d0 } /* Name.Function */ + .codehilite .nl { color: #d8dee9 } /* Name.Label */ + .codehilite .nn { color: #8fbcbb } /* Name.Namespace */ + .codehilite .nx { color: #d8dee9 } /* Name.Other */ + .codehilite .py { color: #d8dee9 } /* Name.Property */ + .codehilite .nt { color: #81a1c1 } /* Name.Tag */ + .codehilite .nv { color: #d8dee9 } /* Name.Variable */ + .codehilite .ow { color: #81a1c1; font-weight: bold } /* Operator.Word */ + .codehilite .pm { color: #eceff4 } /* Punctuation.Marker */ + .codehilite .w { color: #d8dee9 } /* Text.Whitespace */ + .codehilite .mb { color: #b48ead } /* Literal.Number.Bin */ + .codehilite .mf { color: #b48ead } /* Literal.Number.Float */ + .codehilite .mh { color: #b48ead } /* Literal.Number.Hex */ + .codehilite .mi { color: #b48ead } /* Literal.Number.Integer */ + .codehilite .mo { color: #b48ead } /* Literal.Number.Oct */ + .codehilite .sa { color: #a3be8c } /* Literal.String.Affix */ + .codehilite .sb { color: #a3be8c } /* Literal.String.Backtick */ + .codehilite .sc { color: #a3be8c } /* Literal.String.Char */ + .codehilite .dl { color: #a3be8c } /* Literal.String.Delimiter */ + .codehilite .sd { color: #616e87 } /* Literal.String.Doc */ + .codehilite .s2 { color: #a3be8c } /* Literal.String.Double */ + .codehilite .se { color: #ebcb8b } /* Literal.String.Escape */ + .codehilite .sh { color: #a3be8c } /* Literal.String.Heredoc */ + .codehilite .si { color: #a3be8c } /* Literal.String.Interpol */ + .codehilite .sx { color: #a3be8c } /* Literal.String.Other */ + .codehilite .sr { color: #ebcb8b } /* Literal.String.Regex */ + .codehilite .s1 { color: #a3be8c } /* Literal.String.Single */ + .codehilite .ss { color: #a3be8c } /* Literal.String.Symbol */ + .codehilite .bp { color: #81a1c1 } /* Name.Builtin.Pseudo */ + .codehilite .fm { color: #88c0d0 } /* Name.Function.Magic */ + .codehilite .vc { color: #d8dee9 } /* Name.Variable.Class */ + .codehilite .vg { color: #d8dee9 } /* Name.Variable.Global */ + .codehilite .vi { color: #d8dee9 } /* Name.Variable.Instance */ + .codehilite .vm { color: #d8dee9 } /* Name.Variable.Magic */ + .codehilite .il { color: #b48ead } /* Literal.Number.Integer.Long */ +} diff --git a/docs/css/pygments.css b/docs/css/pygments.css new file mode 100644 index 00000000..b4026edd --- /dev/null +++ b/docs/css/pygments.css @@ -0,0 +1,11 @@ +@import "pygments-friendly.css"; /* light theme */ +@import "pygments-nord.css"; /* dark theme */ + +[data-bs-theme="light"] { + --string-color: #4070a0; +} +[data-bs-theme="dark"] { + --string-color: #a3be8c; +} + +.codehilite .s { color: var(--string-color) } /* Literal.String */ diff --git a/docs/index.md b/docs/index.md index d8727ab4..0bbca70b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,4 +1,8 @@ -# freezeyt +--- +title: freezeyt +... + +# freezeyt... Static web page generator created by the Czech Python community. diff --git a/mkdocs.yml b/mkdocs.yml index 9fd78617..7e0796dc 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,18 @@ site_name: Freezeyt +# site_url: ... +repo_url: https://github.com/encukou/freezeyt +edit_uri: edit/main/docs/ +# site_description: nav: - - Freezeyt: index.md -theme: readthedocs + - 'index.md' +theme: + name: mkdocs + color_mode: auto + user_color_mode_toggle: true + # highlightjs: false # disables user_color_mode_toggle?! +markdown_extensions: + - toc: + permalink: true + - codehilite +extra_css: + - css/pygments.css diff --git a/setup.cfg b/setup.cfg index 17bbc1f9..880c0ecb 100644 --- a/setup.cfg +++ b/setup.cfg @@ -42,6 +42,7 @@ dev = packaging docs = mkdocs + pygments blog = flask markdown-it-py From 1fe6e14ca0430ae0fda3eb337c83a267ba50fecf Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 18 Jun 2024 18:07:05 +0200 Subject: [PATCH 03/16] Start breaking out the configuration docs --- docs/config.md | 720 +++++++++++++++++++++++++++++++++++++++++++++++++ docs/index.md | 718 +----------------------------------------------- mkdocs.yml | 5 +- 3 files changed, 725 insertions(+), 718 deletions(-) create mode 100644 docs/config.md diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 00000000..f9eb3fd2 --- /dev/null +++ b/docs/config.md @@ -0,0 +1,720 @@ + +# Configuration + +While common options can be given on the command line, +you can have full control over the freezing process with a YAML +configuration file or a variable with the configuration. +You can specify a config file using the `-c/--config` option, +for example: + +```shell +$ python -m freezeyt my_app _build -c freezeyt.yaml +``` + +The configuration variable should be a dictionary. +To pass the config variable, use the `-C/--import-config` option, +for example: + +```shell +$ python -m freezeyt my_app _build -C my_app:freezeyt_config +``` + +Here is an example configuration file: + +```yaml +output: ./_build/ # The website will be saved to this directory +prefix: https://mysite.example.com/subpage/ +extra_pages: + # Let freezeyt know about URLs that are not linked from elsewhere + /robots.txt + /easter-egg.html +extra_files: + # Include additional files in the output: + # Static files + static: + copy_from: static/ + # Web host configuration + CNAME: mysite.example.com + ".nojekyll": '' + googlecc704f0f191eda8f.html: + copy_from: google-verification.html +status_handlers: + # If a redirect page (HTTP status 3xx) is found, warn but don't fail + "3xx": warn +``` + +The following options are configurable: + +| Option | Meaning | +|--------|---------------------------------------------------------------| +| `app` | [Application to freeze](#Selecting the application to freeze) | + + +## Selecting the application to freeze + +The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. +When the module is specified both by the command line and the config file +an error is raised. + +Examples: + +Freezeyt looks for the variable `app` inside the module by default. +```yaml +app: app_module +``` + +If `app` is in a submodule, separate package names with a dot: +```yaml +app: folder1.folder2.app_module +``` + +A different variable name can be specified by using `:`. +```yaml +app: app_module:wsgi_application +``` + +If the variable is an attribute of some namespace, use dots in the variable name: + +```yaml +app: app_module:namespace.wsgi_application +``` + +When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. + +## Specifying the output + +To outupt the frozen website to a directory, specify +the directory name: + +```yaml +output: ./_build/ +``` + +Or use the full form – using the `dir` *saver*: + +```yaml +output: + type: dir + dir: ./_build/ +``` + +If output is not specified in the configuration file, +you must specify the output directory on the command line. +There are two ways to specify the output on the command line: either by the `--output` (`-o`) option or as a second positional argument. + +The output must be specified just by one way otherwise is an error. + +If there is any existing content in the output directory, +freezeyt will either remove it (if the content looks like a previously +frozen website) or raise an error. +Best practice is to remove the output directory before freezing. + + +### Output to dict + +For testing, `freezeyt` can output to a dictionary rather than save +files to the disk. +This can be configured with: + +```yaml +output: + type: dict +``` + +In this case, the `freeze()` function returns a dictionary of filenames +and their contents. +For example, a site with `/`, `/second_page/` and `/images/smile.png` +will be represented as: + +```python +{ + 'index.html': b'...', + 'second_page': { + 'index.html': b'...', + }, + 'images': { + 'smile.png': b'\x89PNG\r\n\x1a\n\x00...', + }, +} +``` + +This is not useful in the CLI, as the return value is lost. + + +## URL prefix + +The URL where the application will be deployed can be +specified with: + +```yaml +prefix: http://localhost:8000/ +``` +or +```yaml +prefix: https://mysite.example.com/subpage/ +``` + +Freezeyt will freeze all pages starting with `prefix` that +it finds. + +The prefix can also be specified on thecommand line with e.g.: +`--prefix=http://localhost:8000/`. +The CLI argument has priority over the config file. + + +## Extra pages not reachable by links + +URLs of pages that are not reachable by following links from the homepage +can specified as “extra” pages in the configuration: + +```yaml +extra_pages: + - /extra/ + - /extra2.html +``` + +Freezeyt will handle these pages as if it found them by following links. +(For example, by default it will follow links in extra pages.) + +Extra pages may also be given on the command line, +e.g. `--extra-page /extra/ --extra-page /extra2.html`. +The lists from CLI and the config file are merged together. + +You can also specify extra pages using a Python generator, +specified using a module name and function name as follows: + +```yaml +extra_pages: + - generator: my_app:generate_extra_pages +``` + +The `generate_extra_pages` function should take the application +as argument and return an iterable of URLs. + +When using the Python API, a generator for extra pages can be specified +directly as a Python object, for example: + +```python +config = { + ... + 'extra_pages': [{'generator': my_generator_function}], +} +another_config = { + ... + 'extra_pages': [my_generator_function], +} +``` + + +## Extra files not served by the application + +Extra files to be included in the output can be specified, +along with their content. + +These files are not considered part of the app; freezeyt will not try to find +links in them. + +This is useful for configuration of your static server. +(For pages that are part of your website, we recommend +adding them to your application rather than as extra files.) + +If you specify backslashes in `url part`, `freezeyt` convert them to forward slashes. + +For example, the following config will add 3 files to +the output: + +```yaml +extra_files: + CNAME: mysite.example.com + ".nojekyll": '' + config/xyz: abc +``` + +You can also specify extra files using Base64 encoding or +a file path, like so: + + +```yaml +extra_files: + config.dat: + base64: "YWJjZAASNA==" + config2.dat: + copy_from: included/config2.dat +``` + +It's possible to recursively copy an entire directory using `copy_from`, +as in: + + +```yaml +extra_files: + static: + copy_from: static/ +``` + +Extra files cannot be specified on the CLI. + + +## Clean up + +If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. +This prevents, for example, uploading incomplete results to a web hosting by mistake. + +If you want to keep the incomplete directory (for example, +to help debugging), you can use the `--no-cleanup` switch +or the `cleanup` key in the configuration file: + +```yaml +cleanup: False +``` + +The command line switch has priority over the configuration. +Use `--no-cleanup` to override `cleanup: False` from the config. + + +## Fail fast + +Fail fast mode stops the freezing of the app when the first error occurs. + +As with most settings, fail fast can be set both in the configuration file and in the command line using switches (the command line always overrides the configuration file). +The fail fast is defined as `boolean` option. + +If you want to specified fail fast in configuration file, use key `fail_fast` with + `boolean` values `True` or `False`: + +```yaml +fail_fast: True +``` + +If you want to specified fail fast from command line, use switches + `--fail-fast` (short `-x`) resp. `--no-fail-fast` to disable it: + +```shell +$ freezeyt app -o output -x +``` + + +## Github Pages Plugin + +To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. + +By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. + +Configuration example: +```yaml +gh_pages: True +``` + +To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. +You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: +```shell +git fetch output_dir gh-pages +git branch --force gh-pages FETCH_HEAD +``` +Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. + +## Comparison of MIME type and file type + +Freezeyt checks whether the file extensions in its output +correspond to the MIME types served by the app. +If there's a mismatch, freezeyt fails, because this means a server +wouldn't be able to serve the page correctly. + +This funtionality is provided by `freezeyt.Middleware`. + +### Default MIME type + +It is possible to specify the MIME type used for files without an extension. +For example, if your server of static pages defaults to plain text files, +use: + +```yaml +default_mimetype=text/plain +``` + +If the default MIME type isn't explicitly configured in YAML configuration, +then the `freezeyt` uses value `application/octet-stream`. + +The default mimetype cannot be specified on the CLI. + +### Recognizing file types from extensions + +There is possibility to modify the way how to determine file type +from file extension. +You can setup your own `get_mimetype` function. + +Freezeyt will register your own function, if you specify it in configuration +YAML file as: + +```yaml +get_mimetype=module:your_function +``` + +If the `get_mimetype` is not defined in configuration file, +then `freezeyt` calls the python function `mimetypes.guess_type` +and uses the mimetype (the first element) it returns. + +`get_mimetype` can be defined as: +* strings in the form `"module:function"`, which name the function to call, +* Python functions (if configuring `freezeyt` from Python, e.g. as a `dict`, + rather than YAML). + +The `get_mimetype`: +* gets one argument - the `filepath` as `str` + +* returns file MIME types as a `list` of MIME types + (e.g. `["text/html"]` or `["audio/wav", "audio/wave"]`). + +If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mimetype` +(see *Default MIME type* above). + +The get_mimetype function cannot be specified on the CLI. + + +### Using a mime-db database + +There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), +or a database with the same structure. +(This is the database used by GitHub Pages). +The database will be used to get file MIME type from file suffix. + +To use this database, add the path to the JSON file to `freezeyt` configuration: +```yaml +mime_db_file=path/to/mime-db.json +``` +This is equivalent to setting `get_mimetype` to a function that maps +extensions to filetypes according to the database. + +The mime_db file cannot be specified on the CLI. + + +## Progress bar and logging + +The CLI option `--progress` controls what `freezeyt` outputs as it +handles pages: + +* `--progress=log`: Output a message about each frozen page to stdout. +* `--progress=bar`: Draw a status bar in the terminal. Messages about + each frozen page are *also* printed to stdout. +* `--progress=none`: Don't do any of this. + +The default is `bar` if stdout is a terminal, and `log` otherwise. + +It is possible to configure this in the config file using the plugins +`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. +See below on how to enable plugins. + +## Configuration version + +To ensure that your configuration will work unchanged in newer versions of freezeyt, +you should add the current version number, `1`, to your configuration like this: + +```yaml +version: 1 +``` + +This is not mandatory. If the version is not given, the configuration may +not work in future versions of freezeyt. + +The version parameter is not accepted on the command line. + +## Plugins + +It is possible to extend `freezeyt` with *plugins*, either ones that +ship with `freezeyt` or external ones. + +Plugins are added using configuration like: + +```yaml +plugins: + - freezeyt.progressbar:ProgressBar + - mymodule:my_plugin +``` + +### Custom plugins + +A plugin is a function that `freezeyt` will call before starting to +freeze pages. + +It is passed a `FreezeInfo` object as argument (see the `start` hook below). +Usually, the plugin will call `freeze_info.add_hook` to register additional +functions. + + +## Hooks + +It is possible to register *hooks*, functions that are called when +specific events happen in the freezing process. + +For example, if `mymodule` defines functions `start` and `page_frozen`, +you can make freezeyt call them using this configuration: + +```yaml +hooks: + start: + - mymodule:start + page_frozen: + - mymodule:page_frozen +``` + +When using the Python API, a function can be used instead of a name +like `mymodule:start`. + +### `start` + +The function will be called when the freezing process starts, +before any other hooks. + +It is passed a `FreezeInfo` object as argument. +The object has the following attributes: + +* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. + If that URL was frozen already, or is outside the `prefix`, does nothing. + If you add a `reason` string, it will be used in error messages as the reason + why the added URL is being handled. +* `add_hook(hook_name, callable)`: Register an additional hook function. +* `total_task_count`: The number of pages `freezeyt` currently “knows about” – + ones that are already frozen plus ones that are scheduled to be frozen. +* `done_task_count`: The number of pages that are done (either successfully + frozen, or failed). +* `failed_task_count`: The number of pages that failed to freeze. + +### `page_frozen` + +The function will be called whenever a page is processed successfully. +It is passed a `TaskInfo` object as argument. +The object has the following attributes: + +* `get_a_url()`: returns a URL of the page, including `prefix`. + Note that a page may be reachable via several URLs; this function returns + an arbitrary one. +* `path`: the relative path the content is saved to. +* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. +* `exception`: for failed tasks, the exception raised; + `None` otherwise. +* `reasons`: A list of strings explaining why the given page was visited. + (Note that as the freezing progresses, new reasons may be added to + existing tasks.) + + +### `page_failed` + +The function will be called whenever a page is not saved due to an +exception. +It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). + + +### `success` + +The function will be called after the app is successfully frozen. +It is passed a `FreezeInfo` object as argument (see the `start` hook). + + +## Freeze actions + +By default, `freezeyt` will save the pages it finds. +You can instruct it to instead ignore certain pages, or treat them as errors. +This is most useful as a response to certain HTTP statuses (e.g. treat all +`404 NOT FOUND` pages as errors), but can be used independently. + +To tell `freezeyt` what to do from within the application (or middleware), +set the `Freezeyt-Action` HTTP header to one of these values: + +* `'save'`: `freezeyt` will save the body of the page. +* `'ignore'`: `freezeyt` will not save any content for the page +* `'warn'`: will save the content and send warn message to stdout +* `'follow'`: `freezeyt` will save content from the redirected location + (this requires a `Location` header, which is usually added for redirects). + Redirects to external pages are not supported. +* `'error'`: fail; the page will not be saved and `freeze()` will raise + an exception. + + +### Status handling + +If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to +do based on the status. +By default, `200 OK` pages are saved and any others cause errors. +The behavior can be customized using the `status_handlers` setting. +For example, to ignore pages with the `404 NOT FOUND` status, set the +`404` handler to `'ignore'`: + +```yaml +status_handlers: + '404': ignore +``` + +For example, `status_handlers` would be specified as: + +```yaml +status_handlers: + '202': warn + '301': follow + '404': ignore + '418': my_module:custom_action # see below + '429': ignore + '5xx': error +``` + +Note that the status code must be a string, so it needs to be quoted in the YAML file. + +A range of statuses can be specified as a number (`1-5`) followed by lowercase `xx`. +(Other "wildcards" like `50x` are not supported.) + +Status handlers cannot be specified in the CLI. + +### Custom actions + +You can also define a custom action in `status_handlers` as: +* a string in the form `'my_module:custom_action'`, which names a handler + function to call, +* a Python function (if configuring `freezeyt` from Python rather than from + YAML). + +The action function takes one argument, `task` (TaskInfo): information about the freezing task. +See the `TaskInfo` hook for a description. +Freezeyt's default actions, like `follow`, can be imported from `freezeyt.actions` +(e.g. `freezeyt.actions.follow`). +A custom action should call one of these default actions and return the return value from it. + + +## URL finding + +`freezeyt` discovers new links in the application by URL finders. URL finders +are functions whose goal is to find url of specific MIME type. +`freezeyt` offers different configuration options to use URL finders: + +* use predefined URL finders for `text/html` or `text/css` (default), +* define your own URL finder as your function, +* turn off some of finders (section below) + +Example of configuration: + +```yaml +url_finders: + text/html: get_html_links + text/css: get_css_links +``` + +Keys in the `url_finders` dict are MIME types; + +Values are URL finders, which can be defined as: +* strings in the form `"module:function"`, which name the finder + function to call, +* strings like `get_html_links`, which name a function from the + `freezeyt.url_finders` module, or +* Python functions (if configuring `freezeyt` from Python rather than + YAML). + + +An URL finder gets these arguments: +* page content `BinaryIO`, +* the absolute URL of the page, as a `string`, +* the HTTP headers, as a list of tuples (WSGI). + +The function should return an iterator of all URLs (as strings) found +in the page's contents, as they would appear in `href` or `src` attributes. +Specifically: + +- The URLs can be relative. +- External URLs (i.e. those not beginning with `prefix`) should be included. + +Finder functions may be asynchronous: +- The function can be defined with `async def` (i.e. return a + coroutine). If it is, freezeyt will use the result after `await`. +- The function may be an asynchronous generator (defined with `async def` + and use `yield`). If so, freezeyt will use async iteration to handle it. + +The `freezeyt.url_finders` module includes: +- `get_html_links`, the default finder for HTML +- `get_css_links`, the default finder for CSS +- `get_html_links_async` and `get_css_links_async`, asynchronous variants + of the above +- `none`, a finder that doesn't find any links. + +URL finders cannot be specified in the CLI. + +### URL finder header + +You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. +If given, it overrides the `url_finders` configuration. + +### Default `get_html_links` + +The default URL finder for HTML pages looks in `src` and `href` attributes +of all tags in the document. +It currently does not handle other links, such as embedded CSS, but it +may be improved in the future. + +### Default `get_css_links` + +The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all +links in a stylesheet. + +### Disabling default URL finders + +If a finder is not explictly specified in the configuration file, `freezeyt` will use the +default for certain MIME type. For example, if you specify +`text/html: my_custom_finder` only, `freezeyt` will use the default finder +for `text/css`. + +You can disable this behaviour: + +```yaml +use_default_url_finders: false +``` + + +### Finding URLs in Link headers + +By default, `freezeyt` will follow URLs in `Link` HTTP headers. +To disable this, specify: + +```yaml +urls_from_link_headers: false +``` + + +## Path generation + +It is possible to customize the filenames that URLs are saved under +using the `url_to_path` configuration key, for example: + +```yaml +url_to_path: my_module:url_to_path +``` + +The value can be: +* a strings in the form `"module:function"`, which names the + function to call (the function can be omitted along with the colon, + and defaults to `url_to_path`), or +* a Python function (if configuring `freezeyt` from Python rather than + YAML). + +The function receives the *path* of the URL to save, relative to the `prefix`, +and should return a path to the saved file, relative to the build directory. + +The default function, available as `freezeyt.url_to_path`, adds `index.html` +if the URL path ends with `/`. + +`url_to_path` cannot be specified in the CLI. + + +## Middleware static mode + +When using the `freezeyt` middleware, you can enable *static mode*, +which simulates behaviour after the app is saved to static pages: + +```yaml +static_mode: true +``` + +Currently in static mode: +- HTTP methods other than GET and HEAD are disallowed. +- URL parameters are removed +- The request body is discarded + +Other restrictions and features may be added in the future, without regard +to backwards compatibility. +The static mode is intended for interactive use -- testing your app without +having to freeze all of it after each change. diff --git a/docs/index.md b/docs/index.md index 0bbca70b..a85498e2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,7 +2,7 @@ title: freezeyt ... -# freezeyt... +# freezeyt Static web page generator created by the Czech Python community. @@ -95,722 +95,6 @@ app = Middleware(app, config) ``` -## Configuration - -While common options can be given on the command line, -you can have full control over the freezing process with a YAML -configuration file or a variable with the configuration. -You can specify a config file using the `-c/--config` option, -for example: - -```shell -$ python -m freezeyt my_app _build -c freezeyt.yaml -``` - -The configuration variable should be a dictionary. -To pass the config variable, use the `-C/--import-config` option, -for example: - -```shell -$ python -m freezeyt my_app _build -C my_app:freezeyt_config -``` - -Here is an example configuration file: - -```yaml -output: ./_build/ # The website will be saved to this directory -prefix: https://mysite.example.com/subpage/ -extra_pages: - # Let freezeyt know about URLs that are not linked from elsewhere - /robots.txt - /easter-egg.html -extra_files: - # Include additional files in the output: - # Static files - static: - copy_from: static/ - # Web host configuration - CNAME: mysite.example.com - ".nojekyll": '' - googlecc704f0f191eda8f.html: - copy_from: google-verification.html -status_handlers: - # If a redirect page (HTTP status 3xx) is found, warn but don't fail - "3xx": warn -``` - -The following options are configurable: - -### App - -The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. -When the module is specified both by the command line and the config file -an error is raised. - -Examples: - -Freezeyt looks for the variable `app` inside the module by default. -```yaml -app: app_module -``` - -If `app` is in a submodule, separate package names with a dot: -```yaml -app: folder1.folder2.app_module -``` - -A different variable name can be specified by using `:`. -```yaml -app: app_module:wsgi_application -``` - -If the variable is an attribute of some namespace, use dots in the variable name: - -```yaml -app: app_module:namespace.wsgi_application -``` - -When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. - -### Output - -To outupt the frozen website to a directory, specify -the directory name: - -```yaml -output: ./_build/ -``` - -Or use the full form – using the `dir` *saver*: - -```yaml -output: - type: dir - dir: ./_build/ -``` - -If output is not specified in the configuration file, -you must specify the output directory on the command line. -There are two ways to specify the output on the command line: either by the `--output` (`-o`) option or as a second positional argument. - -The output must be specified just by one way otherwise is an error. - -If there is any existing content in the output directory, -freezeyt will either remove it (if the content looks like a previously -frozen website) or raise an error. -Best practice is to remove the output directory before freezing. - - -#### Output to dict - -For testing, `freezeyt` can output to a dictionary rather than save -files to the disk. -This can be configured with: - -```yaml -output: - type: dict -``` - -In this case, the `freeze()` function returns a dictionary of filenames -and their contents. -For example, a site with `/`, `/second_page/` and `/images/smile.png` -will be represented as: - -```python -{ - 'index.html': b'...', - 'second_page': { - 'index.html': b'...', - }, - 'images': { - 'smile.png': b'\x89PNG\r\n\x1a\n\x00...', - }, -} -``` - -This is not useful in the CLI, as the return value is lost. - - -### Prefix - -The URL where the application will be deployed can be -specified with: - -```yaml -prefix: http://localhost:8000/ -``` -or -```yaml -prefix: https://mysite.example.com/subpage/ -``` - -Freezeyt will freeze all pages starting with `prefix` that -it finds. - -The prefix can also be specified on thecommand line with e.g.: -`--prefix=http://localhost:8000/`. -The CLI argument has priority over the config file. - - -### Extra pages - -URLs of pages that are not reachable by following links from the homepage -can specified as “extra” pages in the configuration: - -```yaml -extra_pages: - - /extra/ - - /extra2.html -``` - -Freezeyt will handle these pages as if it found them by following links. -(For example, by default it will follow links in extra pages.) - -Extra pages may also be given on the command line, -e.g. `--extra-page /extra/ --extra-page /extra2.html`. -The lists from CLI and the config file are merged together. - -You can also specify extra pages using a Python generator, -specified using a module name and function name as follows: - -```yaml -extra_pages: - - generator: my_app:generate_extra_pages -``` - -The `generate_extra_pages` function should take the application -as argument and return an iterable of URLs. - -When using the Python API, a generator for extra pages can be specified -directly as a Python object, for example: - -```python -config = { - ... - 'extra_pages': [{'generator': my_generator_function}], -} -another_config = { - ... - 'extra_pages': [my_generator_function], -} -``` - - -### Extra files - -Extra files to be included in the output can be specified, -along with their content. - -These files are not considered part of the app; freezeyt will not try to find -links in them. - -This is useful for configuration of your static server. -(For pages that are part of your website, we recommend -adding them to your application rather than as extra files.) - -If you specify backslashes in `url part`, `freezeyt` convert them to forward slashes. - -For example, the following config will add 3 files to -the output: - -```yaml -extra_files: - CNAME: mysite.example.com - ".nojekyll": '' - config/xyz: abc -``` - -You can also specify extra files using Base64 encoding or -a file path, like so: - - -```yaml -extra_files: - config.dat: - base64: "YWJjZAASNA==" - config2.dat: - copy_from: included/config2.dat -``` - -It's possible to recursively copy an entire directory using `copy_from`, -as in: - - -```yaml -extra_files: - static: - copy_from: static/ -``` - -Extra files cannot be specified on the CLI. - - -### Clean up - -If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. -This prevents, for example, uploading incomplete results to a web hosting by mistake. - -If you want to keep the incomplete directory (for example, -to help debugging), you can use the `--no-cleanup` switch -or the `cleanup` key in the configuration file: - -```yaml -cleanup: False -``` - -The command line switch has priority over the configuration. -Use `--no-cleanup` to override `cleanup: False` from the config. - - -### Fail fast - -Fail fast mode stops the freezing of the app when the first error occurs. - -As with most settings, fail fast can be set both in the configuration file and in the command line using switches (the command line always overrides the configuration file). -The fail fast is defined as `boolean` option. - -If you want to specified fail fast in configuration file, use key `fail_fast` with - `boolean` values `True` or `False`: - -```yaml -fail_fast: True -``` - -If you want to specified fail fast from command line, use switches - `--fail-fast` (short `-x`) resp. `--no-fail-fast` to disable it: - -```shell -$ freezeyt app -o output -x -``` - - -### Github Pages Plugin - -To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. - -By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. - -Configuration example: -```yaml -gh_pages: True -``` - -To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. -You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: -```shell -git fetch output_dir gh-pages -git branch --force gh-pages FETCH_HEAD -``` -Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. - -### Comparison of MIME type and file type - -Freezeyt checks whether the file extensions in its output -correspond to the MIME types served by the app. -If there's a mismatch, freezeyt fails, because this means a server -wouldn't be able to serve the page correctly. - -This funtionality is provided by `freezeyt.Middleware`. - -#### Default MIME type - -It is possible to specify the MIME type used for files without an extension. -For example, if your server of static pages defaults to plain text files, -use: - -```yaml -default_mimetype=text/plain -``` - -If the default MIME type isn't explicitly configured in YAML configuration, -then the `freezeyt` uses value `application/octet-stream`. - -The default mimetype cannot be specified on the CLI. - -#### Recognizing file types from extensions - -There is possibility to modify the way how to determine file type -from file extension. -You can setup your own `get_mimetype` function. - -Freezeyt will register your own function, if you specify it in configuration -YAML file as: - -```yaml -get_mimetype=module:your_function -``` - -If the `get_mimetype` is not defined in configuration file, -then `freezeyt` calls the python function `mimetypes.guess_type` -and uses the mimetype (the first element) it returns. - -`get_mimetype` can be defined as: -* strings in the form `"module:function"`, which name the function to call, -* Python functions (if configuring `freezeyt` from Python, e.g. as a `dict`, - rather than YAML). - -The `get_mimetype`: -* gets one argument - the `filepath` as `str` - -* returns file MIME types as a `list` of MIME types - (e.g. `["text/html"]` or `["audio/wav", "audio/wave"]`). - -If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mimetype` -(see *Default MIME type* above). - -The get_mimetype function cannot be specified on the CLI. - - -#### Using a mime-db database - -There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), -or a database with the same structure. -(This is the database used by GitHub Pages). -The database will be used to get file MIME type from file suffix. - -To use this database, add the path to the JSON file to `freezeyt` configuration: -```yaml -mime_db_file=path/to/mime-db.json -``` -This is equivalent to setting `get_mimetype` to a function that maps -extensions to filetypes according to the database. - -The mime_db file cannot be specified on the CLI. - - -### Progress bar and logging - -The CLI option `--progress` controls what `freezeyt` outputs as it -handles pages: - -* `--progress=log`: Output a message about each frozen page to stdout. -* `--progress=bar`: Draw a status bar in the terminal. Messages about - each frozen page are *also* printed to stdout. -* `--progress=none`: Don't do any of this. - -The default is `bar` if stdout is a terminal, and `log` otherwise. - -It is possible to configure this in the config file using the plugins -`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. -See below on how to enable plugins. - -### Configuration version - -To ensure that your configuration will work unchanged in newer versions of freezeyt, -you should add the current version number, `1`, to your configuration like this: - -```yaml -version: 1 -``` - -This is not mandatory. If the version is not given, the configuration may -not work in future versions of freezeyt. - -The version parameter is not accepted on the command line. - -### Plugins - -It is possible to extend `freezeyt` with *plugins*, either ones that -ship with `freezeyt` or external ones. - -Plugins are added using configuration like: - -```yaml -plugins: - - freezeyt.progressbar:ProgressBar - - mymodule:my_plugin -``` - -#### Custom plugins - -A plugin is a function that `freezeyt` will call before starting to -freeze pages. - -It is passed a `FreezeInfo` object as argument (see the `start` hook below). -Usually, the plugin will call `freeze_info.add_hook` to register additional -functions. - - -### Hooks - -It is possible to register *hooks*, functions that are called when -specific events happen in the freezing process. - -For example, if `mymodule` defines functions `start` and `page_frozen`, -you can make freezeyt call them using this configuration: - -```yaml -hooks: - start: - - mymodule:start - page_frozen: - - mymodule:page_frozen -``` - -When using the Python API, a function can be used instead of a name -like `mymodule:start`. - -#### `start` - -The function will be called when the freezing process starts, -before any other hooks. - -It is passed a `FreezeInfo` object as argument. -The object has the following attributes: - -* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. - If that URL was frozen already, or is outside the `prefix`, does nothing. - If you add a `reason` string, it will be used in error messages as the reason - why the added URL is being handled. -* `add_hook(hook_name, callable)`: Register an additional hook function. -* `total_task_count`: The number of pages `freezeyt` currently “knows about” – - ones that are already frozen plus ones that are scheduled to be frozen. -* `done_task_count`: The number of pages that are done (either successfully - frozen, or failed). -* `failed_task_count`: The number of pages that failed to freeze. - -#### `page_frozen` - -The function will be called whenever a page is processed successfully. -It is passed a `TaskInfo` object as argument. -The object has the following attributes: - -* `get_a_url()`: returns a URL of the page, including `prefix`. - Note that a page may be reachable via several URLs; this function returns - an arbitrary one. -* `path`: the relative path the content is saved to. -* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. -* `exception`: for failed tasks, the exception raised; - `None` otherwise. -* `reasons`: A list of strings explaining why the given page was visited. - (Note that as the freezing progresses, new reasons may be added to - existing tasks.) - - -#### `page_failed` - -The function will be called whenever a page is not saved due to an -exception. -It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). - - -#### `success` - -The function will be called after the app is successfully frozen. -It is passed a `FreezeInfo` object as argument (see the `start` hook). - - -### Freeze actions - -By default, `freezeyt` will save the pages it finds. -You can instruct it to instead ignore certain pages, or treat them as errors. -This is most useful as a response to certain HTTP statuses (e.g. treat all -`404 NOT FOUND` pages as errors), but can be used independently. - -To tell `freezeyt` what to do from within the application (or middleware), -set the `Freezeyt-Action` HTTP header to one of these values: - -* `'save'`: `freezeyt` will save the body of the page. -* `'ignore'`: `freezeyt` will not save any content for the page -* `'warn'`: will save the content and send warn message to stdout -* `'follow'`: `freezeyt` will save content from the redirected location - (this requires a `Location` header, which is usually added for redirects). - Redirects to external pages are not supported. -* `'error'`: fail; the page will not be saved and `freeze()` will raise - an exception. - - -#### Status handling - -If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to -do based on the status. -By default, `200 OK` pages are saved and any others cause errors. -The behavior can be customized using the `status_handlers` setting. -For example, to ignore pages with the `404 NOT FOUND` status, set the -`404` handler to `'ignore'`: - -```yaml -status_handlers: - '404': ignore -``` - -For example, `status_handlers` would be specified as: - -```yaml -status_handlers: - '202': warn - '301': follow - '404': ignore - '418': my_module:custom_action # see below - '429': ignore - '5xx': error -``` - -Note that the status code must be a string, so it needs to be quoted in the YAML file. - -A range of statuses can be specified as a number (`1-5`) followed by lowercase `xx`. -(Other "wildcards" like `50x` are not supported.) - -Status handlers cannot be specified in the CLI. - -#### Custom actions - -You can also define a custom action in `status_handlers` as: -* a string in the form `'my_module:custom_action'`, which names a handler - function to call, -* a Python function (if configuring `freezeyt` from Python rather than from - YAML). - -The action function takes one argument, `task` (TaskInfo): information about the freezing task. -See the `TaskInfo` hook for a description. -Freezeyt's default actions, like `follow`, can be imported from `freezeyt.actions` -(e.g. `freezeyt.actions.follow`). -A custom action should call one of these default actions and return the return value from it. - - -### URL finding - -`freezeyt` discovers new links in the application by URL finders. URL finders -are functions whose goal is to find url of specific MIME type. -`freezeyt` offers different configuration options to use URL finders: - -* use predefined URL finders for `text/html` or `text/css` (default), -* define your own URL finder as your function, -* turn off some of finders (section below) - -Example of configuration: - -```yaml -url_finders: - text/html: get_html_links - text/css: get_css_links -``` - -Keys in the `url_finders` dict are MIME types; - -Values are URL finders, which can be defined as: -* strings in the form `"module:function"`, which name the finder - function to call, -* strings like `get_html_links`, which name a function from the - `freezeyt.url_finders` module, or -* Python functions (if configuring `freezeyt` from Python rather than - YAML). - - -An URL finder gets these arguments: -* page content `BinaryIO`, -* the absolute URL of the page, as a `string`, -* the HTTP headers, as a list of tuples (WSGI). - -The function should return an iterator of all URLs (as strings) found -in the page's contents, as they would appear in `href` or `src` attributes. -Specifically: - -- The URLs can be relative. -- External URLs (i.e. those not beginning with `prefix`) should be included. - -Finder functions may be asynchronous: -- The function can be defined with `async def` (i.e. return a - coroutine). If it is, freezeyt will use the result after `await`. -- The function may be an asynchronous generator (defined with `async def` - and use `yield`). If so, freezeyt will use async iteration to handle it. - -The `freezeyt.url_finders` module includes: -- `get_html_links`, the default finder for HTML -- `get_css_links`, the default finder for CSS -- `get_html_links_async` and `get_css_links_async`, asynchronous variants - of the above -- `none`, a finder that doesn't find any links. - -URL finders cannot be specified in the CLI. - -#### URL finder header - -You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. -If given, it overrides the `url_finders` configuration. - -#### Default `get_html_links` - -The default URL finder for HTML pages looks in `src` and `href` attributes -of all tags in the document. -It currently does not handle other links, such as embedded CSS, but it -may be improved in the future. - -#### Default `get_css_links` - -The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all -links in a stylesheet. - -#### Disabling default URL finders - -If a finder is not explictly specified in the configuration file, `freezeyt` will use the -default for certain MIME type. For example, if you specify -`text/html: my_custom_finder` only, `freezeyt` will use the default finder -for `text/css`. - -You can disable this behaviour: - -```yaml -use_default_url_finders: false -``` - - -#### Finding URLs in Link headers - -By default, `freezeyt` will follow URLs in `Link` HTTP headers. -To disable this, specify: - -```yaml -urls_from_link_headers: false -``` - - -### Path generation - -It is possible to customize the filenames that URLs are saved under -using the `url_to_path` configuration key, for example: - -```yaml -url_to_path: my_module:url_to_path -``` - -The value can be: -* a strings in the form `"module:function"`, which names the - function to call (the function can be omitted along with the colon, - and defaults to `url_to_path`), or -* a Python function (if configuring `freezeyt` from Python rather than - YAML). - -The function receives the *path* of the URL to save, relative to the `prefix`, -and should return a path to the saved file, relative to the build directory. - -The default function, available as `freezeyt.url_to_path`, adds `index.html` -if the URL path ends with `/`. - -`url_to_path` cannot be specified in the CLI. - - -### Middleware static mode - -When using the `freezeyt` middleware, you can enable *static mode*, -which simulates behaviour after the app is saved to static pages: - -```yaml -static_mode: true -``` - -Currently in static mode: -- HTTP methods other than GET and HEAD are disallowed. -- URL parameters are removed -- The request body is discarded - -Other restrictions and features may be added in the future, without regard -to backwards compatibility. -The static mode is intended for interactive use -- testing your app without -having to freeze all of it after each change. - - ## Examples of CLI usage ```shell diff --git a/mkdocs.yml b/mkdocs.yml index 7e0796dc..956957e9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,7 +4,8 @@ repo_url: https://github.com/encukou/freezeyt edit_uri: edit/main/docs/ # site_description: nav: - - 'index.md' + - 'Home': 'index.md' + - 'Configuration': 'config.md' theme: name: mkdocs color_mode: auto @@ -13,6 +14,8 @@ theme: markdown_extensions: - toc: permalink: true + toc_depth: 3 - codehilite + - tables extra_css: - css/pygments.css From fcf20bb238b015cf3040a47ce7825424de168cfc Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 23 Jul 2024 18:41:52 +0200 Subject: [PATCH 04/16] Split out the contributing guide --- docs/contrib.md | 179 +++++++++++++++++++++++++++++++++ docs/css/pygments.css | 4 + docs/index.md | 223 ------------------------------------------ mkdocs.yml | 1 + 4 files changed, 184 insertions(+), 223 deletions(-) create mode 100644 docs/contrib.md diff --git a/docs/contrib.md b/docs/contrib.md new file mode 100644 index 00000000..c2e04217 --- /dev/null +++ b/docs/contrib.md @@ -0,0 +1,179 @@ + +# Contributing + +Freezeyt is developed on [GitHub](https://github.com/encukou/freezeyt). + +Contributions, issues and feature requests are welcome. +Feel free to check out the [issues page] if you'd like to +contribute. + +[issues page]: https://github.com/encukou/freezeyt/issues + + +## Quick guide + +1. Clone this repository to your local + computer: + + $ git clone https://github.com/encukou/freezeyt + +2. Then fork this repo to your GitHub account +3. Add your forked repo as a new remote to your local computer: + + $ git remote add https://github.com//freezeyt + +4. Create a new branch at your local computer + + $ git branch + +5. Switch to your new branch + + $ git switch + +6. Update the code +7. Push the changes to your forked repo on GitHub + + $ git push + +8. Finally, make a pull request from your GitHub account to origin + + +## Installing for development + +Freezeyt can be installed from the current directory: + +```console +$ python -m pip install -e . +``` + +It also has several groups of extra dependecies: + +* `blog` for the project blog +* `dev` for development and running tests +* `typecheck` for [mypy] type checks + +Each group can be installed separately: + +```console +$ python -m pip install -e ."[typecheck]" +``` + +or you can install more groups at once: +```console +$ python -m pip install -e ."[blog, dev, typecheck]" +``` + +[mypy]: https://www.mypy-lang.org/ + + +## Using an in-development copy of freezeyt + +* Set `PYTHONPATH` to the directory with Freezeyt, for example: + * Unix: `export PYTHONPATH="/home/name/freezeyt"` + * Windows: `set PYTHONPATH=C:\Users\Name\freezeyt` + +* Install the web application you want to freeze. Either: + * install the application using `pip`, if possible, or + * install the application's dependencies and `cd` to the app's directory. + +* Run freezeyt, for example: + * `python -m freezeyt demo_app_url_for _build --prefix http://freezeyt.test/foo/` + + + +## Tests + +For testing the project it's necessary to install additional requirements: + +```console +$ python -m pip install .[dev] +``` + +To run tests in your current environment, use pytest: + +```console +$ python -m pytest +``` + +To run tests with multiple Python versions (if you have them installed), +install `tox` using `python -m pip install tox` and run it: + +```console +$ tox +``` + +### Environ variables for tests + +Some test scenarios compare freezeyt's results with expected output. +When the files with expected output don't exist yet, +they can be created by setting the environment variable +`TEST_CREATE_EXPECTED_OUTPUT` to `1`: + +**Unix** + +```console +$ export TEST_CREATE_EXPECTED_OUTPUT=1 +``` + +**Windows** + +```doscon +> set TEST_CREATE_EXPECTED_OUTPUT=1 +``` + +If you set the variable to any different value or leave it unset +then the files will not be recreated +(tests will fail if the files are not up to date). + +When output changes, you need to first delete the expected output, +regenerate it by running tests with `TEST_CREATE_EXPECTED_OUTPUT=1`, +and check that the difference is correct. + + +## How to watch progress + +Unfortunately our progress of development can be watched only in Czech language. + +Watch the progress on our [Youtube playlist](https://www.youtube.com/playlist?list=PLFt-PM7J_H3EU5Oez3ZSVjY5pZJttP2lT). + +Other communication channels and info can be found +in a [Google doc](https://tinyurl.com/freezeyt) (in Czech). + + +## Freezeyt Blog + +We keep a blog about the development of Freezeyt. +It is available [here](https://encukou.github.io/freezeyt/). + +**Be warned:** some of it is in the Czech language. + +### Blog development + +The blog was tested on Python version 3.8. + +The blog is a Flask application. +To run it, install additional dependecies with +`python -m pip install .[blog]` and run the Flask server: + +```console +$ python -m pip install .[blog] +$ flask --app freezeyt_blog/app.py run --debug +``` + +The URL where your blog is running will be printed on the terminal. + +Once you're satisfied with how the blog looks, you can freeze it with: + +```console +$ python -m freezeyt freezeyt_blog.app freezeyt_blog/build +``` + +### Adding new articles to the blog + +Articles are writen in the `Markdown` language. + +**Article** - save to directory `../freezeyt/freezeyt_blog/articles` + +**Images to articles** - save to directory `../freezeyt/freezeyt_blog/static/images` + +If the files are saved elsewhere, the blog will not work correctly. diff --git a/docs/css/pygments.css b/docs/css/pygments.css index b4026edd..eb4c32ab 100644 --- a/docs/css/pygments.css +++ b/docs/css/pygments.css @@ -9,3 +9,7 @@ } .codehilite .s { color: var(--string-color) } /* Literal.String */ + +.codehilite .gp { + user-select: none; /* prompts are non-selectable */ +} diff --git a/docs/index.md b/docs/index.md index a85498e2..39d13458 100644 --- a/docs/index.md +++ b/docs/index.md @@ -118,229 +118,6 @@ $ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page / ``` -## Contributing - -Are you interested in this project? Awesome! -Anyone who wants to be part of this project and who's willing -to help us is very welcome. -Just started with Python? Good news! -We're trying to target mainly the beginner Pythonistas who -are seeking opportunities to contribute to (ideally open source) -projects and who would like to be part of an open source community -which could give them a head start in their -(hopefully open source :)) programming careers. - -Soo, what if you already have some solid Python-fu? -First, there's always something new to learn, and second, -we'd appreciate if you could guide the “rookies” and pass on -some of the knowledge onto them. - -Contributions, issues and feature requests are welcome. -Feel free to check out the [issues] page if you'd like to -contribute. - -[issues]: https://github.com/encukou/freezeyt/issues - - -## How to contribute - -1. Clone this repository to your local computer: - -```shell -$ git clone https://github.com/encukou/freezeyt -``` - -2. Then fork this repo to your GitHub account -3. Add your forked repo as a new remote to your local computer: - -```shell -$ git remote add https://github.com//freezeyt -``` - -4. Create a new branch at your local computer - -```shell -$ git branch -``` - -5. Switch to your new branch - -```shell -$ git switch -``` - -6. Update the code -7. Push the changes to your forked repo on GitHub - -```shell -$ git push -``` - -8. Finally, make a pull request from your GitHub account to origin - - -### Installing for development - -`freezeyt` can be installed from the current directory: - -```shell -$ python -m pip install -e . -``` - -It also has several groups of extra dependecies: -* `blog` for the project blog -* `dev` for development and running tests -* `typecheck` for mypy type checks - -Each group can be installed separately: - -```shell -$ python -m pip install -e ."[typecheck]" -``` - -or you can install more groups at once: -```shell -$ python -m pip install -e ."[blog, dev, typecheck]" -``` - - -### Using an in-development copy of freezeyt - -* Set `PYTHONPATH` to the directory with `freezeyt`, for example: - * Unix: `export PYTHONPATH="/home/name/freezeyt"` - * Windows: `set PYTHONPATH=C:\Users\Name\freezeyt` - -* Install the web application you want to freeze. Either: - * install the application using `pip`, if possible, or - * install the application's dependencies and `cd` to the app's directory. - -* Run freezeyt, for example: - * `python -m freezeyt demo_app_url_for _build --prefix http://freezeyt.test/foo/` - - - -### Tests - -For testing the project it's necessary to install additional requirements: - -``` -$ python -m pip install .[dev] -``` - -To run tests in your current environment, use pytest: - -``` -$ python -m pytest -``` - -To run tests with multiple Python versions (if you have them installed), -install `tox` using `python -m pip install tox` and run it: - -``` -$ tox -``` - -#### Environ variables for tests - -Some test scenarios compare freezeyt's results with expected output. -When the files with expected output don't exist yet, -they can be created by setting the environment variable -`TEST_CREATE_EXPECTED_OUTPUT` to `1`: - -**Unix** - -```shell -$ export TEST_CREATE_EXPECTED_OUTPUT=1 -``` - -**Windows** - -```shell -> set TEST_CREATE_EXPECTED_OUTPUT=1 -``` - -If you set the variable to any different value or leave it unset -then the files will not be recreated -(tests will fail if the files are not up to date). - -When output changes, you need to first delete the expected output, -regenerate it by running tests with `TEST_CREATE_EXPECTED_OUTPUT=1`, -and check that the difference is correct. - -### Tools and technologies used - -* [PEP 3333 - Python WSGI](https://www.python.org/dev/peps/pep-3333/) -* [flask](https://flask.palletsprojects.com/en/1.1.x/) -* [pytest](https://docs.pytest.org/en/latest/) -* [html5lib](https://html5lib.readthedocs.io/en/latest/) - - -### How to watch progress -Unfortunately our progress of development can be watched only in Czech language. - -Watch the progress: - -* [Youtube playlist](https://www.youtube.com/playlist?list=PLFt-PM7J_H3EU5Oez3ZSVjY5pZJttP2lT) - -Other communication channels and info can be found here: -* [Google doc in Czech](https://tinyurl.com/freezeyt) - - -### Freezeyt Blog - -We keep a blog about the development of Freezeyt. -It is available [here](https://encukou.github.io/freezeyt/). - -**Be warned:** some of it is in the Czech language. - -#### Blog development - -The blog was tested on Python version 3.8. - -The blog is a Flask application. -To run it, install additional dependecies with -`python -m pip install .[blog]`. -Then, set the environment variable `FLASK_APP` to the path of the -blog app. -Also set `FLASK_ENV` to "development" for easier debugging. -Then, run the Flask server. - -1. On Microsoft Windows: - -```shell -> python -m pip install .[blog] -> set FLASK_APP=freezeyt_blog/app.py -> set FLASK_ENV=development -> flask run -``` - -2. On UNIX: - -```shell -$ python -m pip install .[blog] -$ export FLASK_APP=freezeyt_blog/app.py -$ export FLASK_ENV=development -$ flask run -``` - -The URL where your blog is running will be printed on the terminal. - -Once you're satisfied with how the blog looks, you can freeze it with: - -```shell -$ python -m freezeyt freezeyt_blog.app freezeyt_blog/build -``` - -#### Adding new articles to freezeyt blog - -Articles are writen in the `Markdown` language. - -**Article** - save to directory `../freezeyt/freezeyt_blog/articles` - -**Images to articles** - save to directory `../freezeyt/freezeyt_blog/static/images` - -If te files are saved elsewhere, the blog will not work correctly. - ## History diff --git a/mkdocs.yml b/mkdocs.yml index 956957e9..ebc2f643 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,6 +6,7 @@ edit_uri: edit/main/docs/ nav: - 'Home': 'index.md' - 'Configuration': 'config.md' + - 'Contributing': 'contrib.md' theme: name: mkdocs color_mode: auto From 62d3c5cdfd69253918838b40a741c2fbbf761440 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 23 Jul 2024 19:03:04 +0200 Subject: [PATCH 05/16] Work on the main page --- docs/index.md | 73 +++++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 34 deletions(-) diff --git a/docs/index.md b/docs/index.md index 39d13458..ea453be0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,7 +4,7 @@ title: freezeyt # freezeyt -Static web page generator created by the Czech Python community. +Freezeyt turns Python web applications into static websites. ## What this does @@ -18,7 +18,7 @@ Python's [http.server]. [http.server]: https://docs.python.org/3/library/http.server.html Freezeyt is compatible with all Python web frameworks that use the common -[Web Server Gateway Interface] (WSGI) +[Web Server Gateway Interface] (WSGI). [Web Server Gateway Interface]: https://www.python.org/dev/peps/pep-3333/ @@ -36,7 +36,7 @@ of virtual environment. The tool can be installed using: -``` +```console $ python -m pip install . ``` @@ -54,28 +54,55 @@ in your environment. Run freezeyt with the name of your application and the output directory. For example: -```shell +```console $ python -m freezeyt my_app _build ``` -Freezeyt may overwrite the build directory (here, `_build`), -removing all existing files from it. +The output directory (here, `_build`), should either not exist yet +or contain output from a previous run of Freezeyt. +Any existing files in it will be removed. + + +## More examples of CLI usage + +You can tell Freezeyt where the application will be hosted, +so it can generate correct URLs: + +```console +$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ +``` + +You can save options like the *prefix* in a file (see [Configuration]), +and then use the `--config` (`-c`) option to use the file: + +```console +$ python -m freezeyt my_app _build/ --config config.yaml +``` -For more options, see Configuration below. +If you use both a configuration file and CLI options like `--prefix`, +the options override settings from the file: + +```console +$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --config path/to/config.yaml +``` ### Python API -Freezeyt also has a Python API, the `freeze` function -that takes an application to freeze and a configuration dict: +Freezeyt also has a Python API: the `freeze` function +that takes an application to freeze and a configuration dict. +For example: ```python from freezeyt import freeze + +config = {'prefix': 'https://pyladies.cz/'} + freeze(app, config) ``` The `config` should be a dict as if read from a YAML configuration -file (see Configuration below). +file (see [Configuration]). From asynchronous code running in an `asyncio` event loop, you can call `freeze_async` instead of `freeze`. @@ -89,35 +116,13 @@ To use it, wrap your application in `freezeyt.Middeleware`. For example: ```python from freezeyt import Middleware -config = {} # use a configuration dict as for `freeze(app, config)` +config = {'prefix': 'https://pyladies.cz/'} app = Middleware(app, config) ``` -## Examples of CLI usage - -```shell -$ python -m freezeyt my_app _build/ -``` - -```shell -$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ -``` - -```shell -$ python -m freezeyt my_app _build/ -c config.yaml -``` - -```shell -$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/ -``` - -```shell -$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/ --config path/to/config.yaml -``` - - +[Configuration]: config.md ## History From 1964bff63519fca94a84eae11c040617d17e61ef Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 30 Jul 2024 18:11:14 +0200 Subject: [PATCH 06/16] Reword the main docs page --- docs/index.md | 81 ++++++++++++++++++++++++++++++++++++++------------- mkdocs.yml | 4 +++ setup.cfg | 1 + 3 files changed, 65 insertions(+), 21 deletions(-) diff --git a/docs/index.md b/docs/index.md index ea453be0..8e7bd47c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -29,38 +29,75 @@ Freezeyt requires Python 3.6 or above. It is highly recommended to create and activate a separate virtual environment for this project. -You can use [`venv`], `virtualenv`, Conda, containers or any other kind +You can use [`venv`][venv], `virtualenv`, Conda, containers or any other kind of virtual environment. -[`venv`]: https://docs.python.org/3/library/venv.html?highlight=venv#module-venv +[venv]: https://docs.python.org/3/library/venv.html?highlight=venv#module-venv The tool can be installed using: ```console -$ python -m pip install . +$ python -m pip install freezeyt ``` +To install a development version of freezeyt, +see [Contributing documentation]. + +[Contributing documentation]: ./contrib.md + + +## Quick usage + +For a Flask app in `hello.py`, run: + +```console +$ python -m freezeyt hello _build +``` + +For detailed instructions, read on. + ## Usage To use freezeyt, you need a Python web application. -You can use the [example Flask app]. +You can use the [example Flask app] to start. -[example Flask app]: https://flask.palletsprojects.com/en/1.1.x/quickstart/ +[example Flask app]: https://flask.palletsprojects.com/en/2.3.x/quickstart/ -Both the application and Freezeyt must be importable (installed) -in your environment. +Specifically, Freezeyt needs a WSGI application, +ideally one named `app` which is the default in [Flask] and [Falcon]. +For other frameworks, search the documentation on how to export a WSGI +application. -Run freezeyt with the name of your application and the -output directory. For example: +Both the application and freezeyt need to be importable (installed) in your +envuronment. + +Run freezeyt with two arguments: the Python module with your `app`, +and an output directory. +Note that freezeyt wants a *module* name (as used in an `import` statement). +Don't use a *file* name with a `.py` suffix. + +For example, if your `app` is defined in the file `my_app.py`, run: ```console $ python -m freezeyt my_app _build ``` +If your application is not named `app`, give its name after a colon. +For example, a [Django WSGI application] is usually in +the `wsgi` submodule and named `application`, so you should run: + +```console +$ python -m freezeyt my_project.wsgi:application _build +``` + The output directory (here, `_build`), should either not exist yet or contain output from a previous run of Freezeyt. Any existing files in it will be removed. +(Freezeyt tries to avoid deleting data it didn't create itself, +but do not rely on this.) + +[WSGI application]: https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/ ## More examples of CLI usage @@ -104,9 +141,11 @@ freeze(app, config) The `config` should be a dict as if read from a YAML configuration file (see [Configuration]). -From asynchronous code running in an `asyncio` event loop, +From asynchronous code running in an [`asyncio`][asyncio] event loop, you can call `freeze_async` instead of `freeze`. +[asyncio]: https://docs.python.org/3/library/asyncio.html + ### Middleware @@ -124,9 +163,9 @@ app = Middleware(app, config) [Configuration]: config.md -## History +## Project info -### Why did the project start? +### History The Czech Python community uses a lot of static web pages that are generated from a web application for community purposes. @@ -135,24 +174,24 @@ or meetups. The community has been so far relying on [Frozen Flask] and [elsa] in order to generate the static web content. -The new [freezer] ought to be used with any arbitrary Python Web -application framework ([Flask], [Django], [Tornado], etc.). -So the community won't be limited by one web app technology for -generating static pages anymore. +The new freezer ought to be used with any arbitrary Python Web +application framework ([Flask], [Django], [Falcon], [Tornado], etc.). +So the community won't be limited by one technology anymore. -[Frozen Flask]: https://pythonhosted.org/Frozen-Flask/ +[Frozen Flask]: https://frozen-flask.readthedocs.io/en/latest/ [elsa]: https://github.com/pyvec/elsa/ [freezer]: https://github.com/encukou/freezeyt -[Flask]: https://flask.palletsprojects.com/en/1.1.x/ [Django]: https://www.djangoproject.com/ [Tornado]: https://www.tornadoweb.org/en/stable/ +[Flask]: https://flask.palletsprojects.com/en/3.0.x/ +[Falcon]: https://falconframework.org/ -## Authors +### Authors See GitHub history for all [contributors](https://github.com/encukou/freezeyt/graphs/contributors). -## License +### License -This project is licensed under the [MIT License](LICENCE.MIT). +This project is licensed under an [MIT License](licence.md). May it serve you well. diff --git a/mkdocs.yml b/mkdocs.yml index ebc2f643..a0da1516 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -7,6 +7,8 @@ nav: - 'Home': 'index.md' - 'Configuration': 'config.md' - 'Contributing': 'contrib.md' +not_in_nav: + licence.md theme: name: mkdocs color_mode: auto @@ -18,5 +20,7 @@ markdown_extensions: toc_depth: 3 - codehilite - tables + - markdown_include.include: + base_path: docs extra_css: - css/pygments.css diff --git a/setup.cfg b/setup.cfg index 880c0ecb..d4729e71 100644 --- a/setup.cfg +++ b/setup.cfg @@ -43,6 +43,7 @@ dev = docs = mkdocs pygments + markdown-include blog = flask markdown-it-py From b09d86595ce92df609a4bf8f45f668a2ec20aee3 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 30 Jul 2024 18:58:01 +0200 Subject: [PATCH 07/16] Start organizing the Configuration page --- docs/config.md | 97 ++++++++++++++++++++++++++++++++++++-------------- mkdocs.yml | 1 + 2 files changed, 72 insertions(+), 26 deletions(-) diff --git a/docs/config.md b/docs/config.md index f9eb3fd2..c3183685 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,6 +1,28 @@ # Configuration +## CLI + +The most common options for freezeyt can be given on the command line. +These are usually shortcuts for the more powerful [YAML-based configuration](#yaml-config), +or ways to load that configuration. + +| CLI option | YAML key | Meaning | +|----------|------------|----------| +| `--help` | --- | Show help and exit | +| APP (positional) | `app` | [Application to freeze](#conf-app) | +| `-o`, `--output`, positional | `output` | [Output directory](#conf-output) | +| `-c`, `--config` | --- | [Configuration file](#conf-cli-config) | +| `-C`, `--import-config` | --- | [Configuration variable](#conf-cli-import-config) | +| `--prefix` | `prefix` | [URL prefix](#conf-prefix) | +| `--extra-page` | `extra_pages` | [Extra pages](#conf-extra_pages) | +| `--progress` | (plugins) | [Progress bar and logging](#conf-cli-progress) | +| `--gh-pages` | `gh_pages` | [Github Pages Plugin](#conf-gh_pages) | +| `--no-cleanup` | `cleanup` | Don't [clean up](#conf-cleanup) | +| `-x`, `--fail-fast` | `fail_fast` | [Fail fast](#conf-fail_fast) | + +## YAML + While common options can be given on the command line, you can have full control over the freezing process with a YAML configuration file or a variable with the configuration. @@ -45,12 +67,35 @@ status_handlers: The following options are configurable: -| Option | Meaning | -|--------|---------------------------------------------------------------| -| `app` | [Application to freeze](#Selecting the application to freeze) | - - -## Selecting the application to freeze +| YAML key | CLI option | Meaning | Example | +|----------|------------|----------|---------| +| `app` | (positional) | [Application to freeze](#conf-app) | `'module:wsgi_app'` | +| `output` | `-o`, `--output`, positional | [Output directory](#conf-output) | `'./_build/'` | +| --- | `-c`, `--config` | [Configuration file](#conf-cli-config) | `'./freezeyt.yaml/'` | +| --- | `-C`, `--inport-config` | [Configuration variable](#conf-cli-import-config) | `'module:conf'` | +| --- | `--help` | [Show help](#conf-cli-help) | --- | +| `prefix` | `--prefix` | [URL prefix](#conf-prefix) | `'https://mysite.example.com/subpage/'` | +| `extra_pages` | `--extra-page` | [Extra pages](#conf-extra_pages) | (list) | +| `extra_files` | --- | [Extra files](#conf-extra_files) | (dict) | +| `cleanup` | `--no-cleanup` | [Clean up](#conf-cleanup) | `False` | +| `fail_fast` | `-x` | [Fail fast](#conf-fail_fast) | `True` | +| `gh_pages` | `--gh-pages` | [Github Pages Plugin](#conf-gh_pages) | `True` | +| `default_mimetype` | --- | [Default MIME type](#conf-default_mimetype) | `text/plain` | +| `get_mimetype` | --- | [MIME type getter](#conf-default_mimetype) | `module:your_function` | +| `mime_db_file` | --- | [MIME type database](#conf-mime_db_file) | `path/to/mime-db.json` | +| (plugins) | `--progress` | [Progress bar and logging](#conf-cli-progress) | `log` | +| `version` | --- | [Configuration version](#conf-version) | `1` | +| `plugins` | --- | [Plugins](#conf-plugins) | (dict) | +| `hooks` | --- | [Hooks](#conf-hooks) | (dict) | +| `status_handlers` | --- | [HTTP Status handling](#conf-status_handlers) | (dict) | +| `url_finders` | --- | [URL finders](#conf-url_finders) | (dict) | +| `use_default_url_finders` | --- | [Use default URL finders](#conf-use_default_url_finders) | `False` | +| `urls_from_link_headers` | --- | [Find URLs in Link headers](#conf-urls_from_link_headers) | `False` | +| `url_to_path` | --- | [Path generation](#conf-url_to_path) | `my_module:url_to_path` | +| `static_mode` | --- | [Middleware static mode](#conf-static_mode) | `True` | + + +## Application to freeze {: #conf-app } The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. When the module is specified both by the command line and the config file @@ -81,7 +126,7 @@ app: app_module:namespace.wsgi_application When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. -## Specifying the output +## Output {: #conf-output } To outupt the frozen website to a directory, specify the directory name: @@ -110,7 +155,7 @@ frozen website) or raise an error. Best practice is to remove the output directory before freezing. -### Output to dict +### Output to dictionary For testing, `freezeyt` can output to a dictionary rather than save files to the disk. @@ -141,7 +186,7 @@ will be represented as: This is not useful in the CLI, as the return value is lost. -## URL prefix +## URL prefix {: #conf-prefix } The URL where the application will be deployed can be specified with: @@ -162,7 +207,7 @@ The prefix can also be specified on thecommand line with e.g.: The CLI argument has priority over the config file. -## Extra pages not reachable by links +## Extra pages {: #conf-extra_pages } URLs of pages that are not reachable by following links from the homepage can specified as “extra” pages in the configuration: @@ -206,7 +251,7 @@ another_config = { ``` -## Extra files not served by the application +## Extra files {: #config-extra_files } Extra files to be included in the output can be specified, along with their content. @@ -255,7 +300,7 @@ extra_files: Extra files cannot be specified on the CLI. -## Clean up +## Clean up {: #conf-cleanup } If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. This prevents, for example, uploading incomplete results to a web hosting by mistake. @@ -272,7 +317,7 @@ The command line switch has priority over the configuration. Use `--no-cleanup` to override `cleanup: False` from the config. -## Fail fast +## Fail fast {: #conf-fail_fast } Fail fast mode stops the freezing of the app when the first error occurs. @@ -294,7 +339,7 @@ $ freezeyt app -o output -x ``` -## Github Pages Plugin +## Github Pages Plugin {: #conf-gh_pages } To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. @@ -322,7 +367,7 @@ wouldn't be able to serve the page correctly. This funtionality is provided by `freezeyt.Middleware`. -### Default MIME type +### Default MIME type {: #conf-default_mimetype } It is possible to specify the MIME type used for files without an extension. For example, if your server of static pages defaults to plain text files, @@ -337,7 +382,7 @@ then the `freezeyt` uses value `application/octet-stream`. The default mimetype cannot be specified on the CLI. -### Recognizing file types from extensions +### MIME type getter {: #conf-get_mimetype } There is possibility to modify the way how to determine file type from file extension. @@ -371,7 +416,7 @@ If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mi The get_mimetype function cannot be specified on the CLI. -### Using a mime-db database +### Using a mime-db database {: #conf-mime_db_file } There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), or a database with the same structure. @@ -388,7 +433,7 @@ extensions to filetypes according to the database. The mime_db file cannot be specified on the CLI. -## Progress bar and logging +## Progress bar and logging {: #conf-cli-progress } The CLI option `--progress` controls what `freezeyt` outputs as it handles pages: @@ -404,7 +449,7 @@ It is possible to configure this in the config file using the plugins `freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. See below on how to enable plugins. -## Configuration version +## Configuration version {: #conf-version } To ensure that your configuration will work unchanged in newer versions of freezeyt, you should add the current version number, `1`, to your configuration like this: @@ -418,7 +463,7 @@ not work in future versions of freezeyt. The version parameter is not accepted on the command line. -## Plugins +## Plugins {: #conf-plugins } It is possible to extend `freezeyt` with *plugins*, either ones that ship with `freezeyt` or external ones. @@ -441,7 +486,7 @@ Usually, the plugin will call `freeze_info.add_hook` to register additional functions. -## Hooks +## Hooks {: #conf-hooks } It is possible to register *hooks*, functions that are called when specific events happen in the freezing process. @@ -530,7 +575,7 @@ set the `Freezeyt-Action` HTTP header to one of these values: an exception. -### Status handling +### HTTP Status handling {: #conf-status_handlers } If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to do based on the status. @@ -578,7 +623,7 @@ Freezeyt's default actions, like `follow`, can be imported from `freezeyt.action A custom action should call one of these default actions and return the return value from it. -## URL finding +## URL finding {: #conf-url_finders } `freezeyt` discovers new links in the application by URL finders. URL finders are functions whose goal is to find url of specific MIME type. @@ -651,7 +696,7 @@ may be improved in the future. The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all links in a stylesheet. -### Disabling default URL finders +### Disabling default URL finders {: #conf-use_default_url_finders } If a finder is not explictly specified in the configuration file, `freezeyt` will use the default for certain MIME type. For example, if you specify @@ -665,7 +710,7 @@ use_default_url_finders: false ``` -### Finding URLs in Link headers +### Finding URLs in Link headers {: #conf-urls_from_link_headers } By default, `freezeyt` will follow URLs in `Link` HTTP headers. To disable this, specify: @@ -700,7 +745,7 @@ if the URL path ends with `/`. `url_to_path` cannot be specified in the CLI. -## Middleware static mode +## Middleware static mode {: #static_mode } When using the `freezeyt` middleware, you can enable *static mode*, which simulates behaviour after the app is saved to static pages: diff --git a/mkdocs.yml b/mkdocs.yml index a0da1516..4e8f0179 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ markdown_extensions: toc_depth: 3 - codehilite - tables + - attr_list - markdown_include.include: base_path: docs extra_css: From e7f99a3b7aec29cfb0d8760bb30ce13d889e7b07 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 13 Aug 2024 18:30:49 +0200 Subject: [PATCH 08/16] Rearrange the config docs --- docs/config.md | 515 ++++++++++++++++++++++++++----------------------- mkdocs.yml | 1 + 2 files changed, 273 insertions(+), 243 deletions(-) diff --git a/docs/config.md b/docs/config.md index c3183685..341b677f 100644 --- a/docs/config.md +++ b/docs/config.md @@ -21,26 +21,36 @@ or ways to load that configuration. | `--no-cleanup` | `cleanup` | Don't [clean up](#conf-cleanup) | | `-x`, `--fail-fast` | `fail_fast` | [Fail fast](#conf-fail_fast) | -## YAML -While common options can be given on the command line, -you can have full control over the freezing process with a YAML -configuration file or a variable with the configuration. -You can specify a config file using the `-c/--config` option, -for example: +### Configuration file (`-c`) {: #conf-cli-config } + +You can specify a config file in YAML (or JSON) format using the `-c/--config` +option, for example: ```shell $ python -m freezeyt my_app _build -c freezeyt.yaml ``` -The configuration variable should be a dictionary. -To pass the config variable, use the `-C/--import-config` option, -for example: +### Configuration variable (`-C`) {: #conf-cli-import-config } + +Instead of a YAML file, you can also put the configuration in a Python +dictionary and tell *freezeyt* to import it using the `-C/--import-config` +option. +Like the application to freeze, the option takes the name of an importable +module and the name of a variable in that module, separated by a colon. +For example: ```shell $ python -m freezeyt my_app _build -C my_app:freezeyt_config ``` + +## Overview + +While common options can be given on the command line, +you can have full control over the freezing process with a YAML +configuration file or a variable with the configuration. + Here is an example configuration file: ```yaml @@ -67,35 +77,52 @@ status_handlers: The following options are configurable: -| YAML key | CLI option | Meaning | Example | -|----------|------------|----------|---------| -| `app` | (positional) | [Application to freeze](#conf-app) | `'module:wsgi_app'` | -| `output` | `-o`, `--output`, positional | [Output directory](#conf-output) | `'./_build/'` | -| --- | `-c`, `--config` | [Configuration file](#conf-cli-config) | `'./freezeyt.yaml/'` | -| --- | `-C`, `--inport-config` | [Configuration variable](#conf-cli-import-config) | `'module:conf'` | -| --- | `--help` | [Show help](#conf-cli-help) | --- | -| `prefix` | `--prefix` | [URL prefix](#conf-prefix) | `'https://mysite.example.com/subpage/'` | -| `extra_pages` | `--extra-page` | [Extra pages](#conf-extra_pages) | (list) | -| `extra_files` | --- | [Extra files](#conf-extra_files) | (dict) | -| `cleanup` | `--no-cleanup` | [Clean up](#conf-cleanup) | `False` | -| `fail_fast` | `-x` | [Fail fast](#conf-fail_fast) | `True` | -| `gh_pages` | `--gh-pages` | [Github Pages Plugin](#conf-gh_pages) | `True` | -| `default_mimetype` | --- | [Default MIME type](#conf-default_mimetype) | `text/plain` | -| `get_mimetype` | --- | [MIME type getter](#conf-default_mimetype) | `module:your_function` | -| `mime_db_file` | --- | [MIME type database](#conf-mime_db_file) | `path/to/mime-db.json` | -| (plugins) | `--progress` | [Progress bar and logging](#conf-cli-progress) | `log` | -| `version` | --- | [Configuration version](#conf-version) | `1` | -| `plugins` | --- | [Plugins](#conf-plugins) | (dict) | -| `hooks` | --- | [Hooks](#conf-hooks) | (dict) | -| `status_handlers` | --- | [HTTP Status handling](#conf-status_handlers) | (dict) | -| `url_finders` | --- | [URL finders](#conf-url_finders) | (dict) | -| `use_default_url_finders` | --- | [Use default URL finders](#conf-use_default_url_finders) | `False` | -| `urls_from_link_headers` | --- | [Find URLs in Link headers](#conf-urls_from_link_headers) | `False` | -| `url_to_path` | --- | [Path generation](#conf-url_to_path) | `my_module:url_to_path` | -| `static_mode` | --- | [Middleware static mode](#conf-static_mode) | `True` | - - -## Application to freeze {: #conf-app } +| YAML key | Meaning | Example | +|----------|---------|---------| +| `app` | [Application to freeze](#conf-app) | `'module:wsgi_app'` | +| `output` | [Output directory](#conf-output) | `'./_build/'` | +| --- | [Configuration file](#conf-cli-config) | `'./freezeyt.yaml/'` | +| --- | [Configuration variable](#conf-cli-import-config) | `'module:conf'` | +| --- | [Show help](#conf-cli-help) | --- | +| `prefix` | [URL prefix](#conf-prefix) | `'https://mysite.example.com/subpage/'` | +| `extra_pages` | [Extra pages](#conf-extra_pages) | (list) | +| `extra_files` | [Extra files](#conf-extra_files) | (dict) | +| `cleanup` | [Clean up](#conf-cleanup) | `False` | +| `fail_fast` | [Fail fast](#conf-fail_fast) | `True` | +| `gh_pages` | [Github Pages Plugin](#conf-gh_pages) | `True` | +| `default_mimetype` | [Default MIME type](#conf-default_mimetype) | `text/plain` | +| `get_mimetype` | [MIME type getter](#conf-default_mimetype) | `module:your_function` | +| `mime_db_file` | [MIME type database](#conf-mime_db_file) | `path/to/mime-db.json` | +| (plugins) | [Progress bar and logging](#conf-cli-progress) | `log` | +| `version` | [Configuration version](#conf-version) | `1` | +| `plugins` | [Plugins](#conf-plugins) | (dict) | +| `hooks` | [Hooks](#conf-hooks) | (dict) | +| `status_handlers` | [HTTP Status handling](#conf-status_handlers) | (dict) | +| `url_finders` | [URL finders](#conf-url_finders) | (dict) | +| `use_default_url_finders` | [Use default URL finders](#conf-use_default_url_finders) | `False` | +| `urls_from_link_headers` | [Find URLs in Link headers](#conf-urls_from_link_headers) | `False` | +| `url_to_path` | [Path generation](#conf-url_to_path) | `my_module:url_to_path` | +| `static_mode` | [Middleware static mode](#conf-static_mode) | `True` | + + +## Basic options + + +### Configuration version {: #conf-version } + +To ensure that your configuration will work unchanged in newer versions of freezeyt, +you should add the current version number, `1`, to your configuration like this: + +```yaml +version: 1 +``` + +This is not mandatory. If the version is not given, the configuration may +not work in future versions of freezeyt. + +The version parameter is not accepted on the command line. + +### Application to freeze {: #conf-app } The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. When the module is specified both by the command line and the config file @@ -126,7 +153,7 @@ app: app_module:namespace.wsgi_application When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. -## Output {: #conf-output } +### Output {: #conf-output } To outupt the frozen website to a directory, specify the directory name: @@ -155,7 +182,7 @@ frozen website) or raise an error. Best practice is to remove the output directory before freezing. -### Output to dictionary +#### Output to dictionary For testing, `freezeyt` can output to a dictionary rather than save files to the disk. @@ -186,7 +213,7 @@ will be represented as: This is not useful in the CLI, as the return value is lost. -## URL prefix {: #conf-prefix } +### URL prefix {: #conf-prefix } The URL where the application will be deployed can be specified with: @@ -207,7 +234,10 @@ The prefix can also be specified on thecommand line with e.g.: The CLI argument has priority over the config file. -## Extra pages {: #conf-extra_pages } +## Extra content + + +### Extra pages {: #conf-extra_pages } URLs of pages that are not reachable by following links from the homepage can specified as “extra” pages in the configuration: @@ -250,8 +280,7 @@ another_config = { } ``` - -## Extra files {: #config-extra_files } +### Extra files {: #config-extra_files } Extra files to be included in the output can be specified, along with their content. @@ -300,7 +329,10 @@ extra_files: Extra files cannot be specified on the CLI. -## Clean up {: #conf-cleanup } +## Debugging options + + +### Clean up {: #conf-cleanup } If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. This prevents, for example, uploading incomplete results to a web hosting by mistake. @@ -317,7 +349,7 @@ The command line switch has priority over the configuration. Use `--no-cleanup` to override `cleanup: False` from the config. -## Fail fast {: #conf-fail_fast } +### Fail fast {: #conf-fail_fast } Fail fast mode stops the freezing of the app when the first error occurs. @@ -339,7 +371,10 @@ $ freezeyt app -o output -x ``` -## Github Pages Plugin {: #conf-gh_pages } +## Built-in plugins + + +### Github Pages Plugin {: #conf-gh_pages } To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. @@ -358,7 +393,28 @@ git branch --force gh-pages FETCH_HEAD ``` Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. -## Comparison of MIME type and file type + +### Progress bar and logging {: #conf-cli-progress } + +The CLI option `--progress` controls what `freezeyt` outputs as it +handles pages: + +* `--progress=log`: Output a message about each frozen page to stdout. +* `--progress=bar`: Draw a status bar in the terminal. Messages about + each frozen page are *also* printed to stdout. +* `--progress=none`: Don't do any of this. + +The default is `bar` if stdout is a terminal, and `log` otherwise. + +It is possible to configure this in the config file using the plugins +`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. +See below on how to enable plugins. + + +## Freezing + + +### Comparison of MIME type and file type Freezeyt checks whether the file extensions in its output correspond to the MIME types served by the app. @@ -367,7 +423,7 @@ wouldn't be able to serve the page correctly. This funtionality is provided by `freezeyt.Middleware`. -### Default MIME type {: #conf-default_mimetype } +#### Default MIME type {: #conf-default_mimetype } It is possible to specify the MIME type used for files without an extension. For example, if your server of static pages defaults to plain text files, @@ -382,7 +438,7 @@ then the `freezeyt` uses value `application/octet-stream`. The default mimetype cannot be specified on the CLI. -### MIME type getter {: #conf-get_mimetype } +#### MIME type getter {: #conf-get_mimetype } There is possibility to modify the way how to determine file type from file extension. @@ -416,7 +472,7 @@ If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mi The get_mimetype function cannot be specified on the CLI. -### Using a mime-db database {: #conf-mime_db_file } +#### Using a mime-db database {: #conf-mime_db_file } There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), or a database with the same structure. @@ -432,130 +488,104 @@ extensions to filetypes according to the database. The mime_db file cannot be specified on the CLI. +### URL finding {: #conf-url_finders } -## Progress bar and logging {: #conf-cli-progress } - -The CLI option `--progress` controls what `freezeyt` outputs as it -handles pages: - -* `--progress=log`: Output a message about each frozen page to stdout. -* `--progress=bar`: Draw a status bar in the terminal. Messages about - each frozen page are *also* printed to stdout. -* `--progress=none`: Don't do any of this. - -The default is `bar` if stdout is a terminal, and `log` otherwise. - -It is possible to configure this in the config file using the plugins -`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. -See below on how to enable plugins. +`freezeyt` discovers new links in the application by URL finders. URL finders +are functions whose goal is to find url of specific MIME type. +`freezeyt` offers different configuration options to use URL finders: -## Configuration version {: #conf-version } +* use predefined URL finders for `text/html` or `text/css` (default), +* define your own URL finder as your function, +* turn off some of finders (section below) -To ensure that your configuration will work unchanged in newer versions of freezeyt, -you should add the current version number, `1`, to your configuration like this: +Example of configuration: ```yaml -version: 1 +url_finders: + text/html: get_html_links + text/css: get_css_links ``` -This is not mandatory. If the version is not given, the configuration may -not work in future versions of freezeyt. - -The version parameter is not accepted on the command line. - -## Plugins {: #conf-plugins } - -It is possible to extend `freezeyt` with *plugins*, either ones that -ship with `freezeyt` or external ones. - -Plugins are added using configuration like: - -```yaml -plugins: - - freezeyt.progressbar:ProgressBar - - mymodule:my_plugin -``` +Keys in the `url_finders` dict are MIME types; -### Custom plugins +Values are URL finders, which can be defined as: +* strings in the form `"module:function"`, which name the finder + function to call, +* strings like `get_html_links`, which name a function from the + `freezeyt.url_finders` module, or +* Python functions (if configuring `freezeyt` from Python rather than + YAML). -A plugin is a function that `freezeyt` will call before starting to -freeze pages. -It is passed a `FreezeInfo` object as argument (see the `start` hook below). -Usually, the plugin will call `freeze_info.add_hook` to register additional -functions. +An URL finder gets these arguments: +* page content `BinaryIO`, +* the absolute URL of the page, as a `string`, +* the HTTP headers, as a list of tuples (WSGI). +The function should return an iterator of all URLs (as strings) found +in the page's contents, as they would appear in `href` or `src` attributes. +Specifically: -## Hooks {: #conf-hooks } +- The URLs can be relative. +- External URLs (i.e. those not beginning with `prefix`) should be included. -It is possible to register *hooks*, functions that are called when -specific events happen in the freezing process. +Finder functions may be asynchronous: +- The function can be defined with `async def` (i.e. return a + coroutine). If it is, freezeyt will use the result after `await`. +- The function may be an asynchronous generator (defined with `async def` + and use `yield`). If so, freezeyt will use async iteration to handle it. -For example, if `mymodule` defines functions `start` and `page_frozen`, -you can make freezeyt call them using this configuration: +The `freezeyt.url_finders` module includes: +- `get_html_links`, the default finder for HTML +- `get_css_links`, the default finder for CSS +- `get_html_links_async` and `get_css_links_async`, asynchronous variants + of the above +- `none`, a finder that doesn't find any links. -```yaml -hooks: - start: - - mymodule:start - page_frozen: - - mymodule:page_frozen -``` +URL finders cannot be specified in the CLI. -When using the Python API, a function can be used instead of a name -like `mymodule:start`. +#### URL finder header -### `start` +You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. +If given, it overrides the `url_finders` configuration. -The function will be called when the freezing process starts, -before any other hooks. +#### Default `get_html_links` -It is passed a `FreezeInfo` object as argument. -The object has the following attributes: +The default URL finder for HTML pages looks in `src` and `href` attributes +of all tags in the document. +It currently does not handle other links, such as embedded CSS, but it +may be improved in the future. -* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. - If that URL was frozen already, or is outside the `prefix`, does nothing. - If you add a `reason` string, it will be used in error messages as the reason - why the added URL is being handled. -* `add_hook(hook_name, callable)`: Register an additional hook function. -* `total_task_count`: The number of pages `freezeyt` currently “knows about” – - ones that are already frozen plus ones that are scheduled to be frozen. -* `done_task_count`: The number of pages that are done (either successfully - frozen, or failed). -* `failed_task_count`: The number of pages that failed to freeze. +#### Default `get_css_links` -### `page_frozen` +The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all +links in a stylesheet. -The function will be called whenever a page is processed successfully. -It is passed a `TaskInfo` object as argument. -The object has the following attributes: +#### Disabling default URL finders {: #conf-use_default_url_finders } -* `get_a_url()`: returns a URL of the page, including `prefix`. - Note that a page may be reachable via several URLs; this function returns - an arbitrary one. -* `path`: the relative path the content is saved to. -* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. -* `exception`: for failed tasks, the exception raised; - `None` otherwise. -* `reasons`: A list of strings explaining why the given page was visited. - (Note that as the freezing progresses, new reasons may be added to - existing tasks.) +If a finder is not explictly specified in the configuration file, `freezeyt` will use the +default for certain MIME type. For example, if you specify +`text/html: my_custom_finder` only, `freezeyt` will use the default finder +for `text/css`. +You can disable this behaviour: -### `page_failed` +```yaml +use_default_url_finders: false +``` -The function will be called whenever a page is not saved due to an -exception. -It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). +#### Finding URLs in Link headers {: #conf-urls_from_link_headers } -### `success` +By default, `freezeyt` will follow URLs in `Link` HTTP headers. +To disable this, specify: -The function will be called after the app is successfully frozen. -It is passed a `FreezeInfo` object as argument (see the `start` hook). +```yaml +urls_from_link_headers: false +``` -## Freeze actions +### Freeze actions By default, `freezeyt` will save the pages it finds. You can instruct it to instead ignore certain pages, or treat them as errors. @@ -575,7 +605,7 @@ set the `Freezeyt-Action` HTTP header to one of these values: an exception. -### HTTP Status handling {: #conf-status_handlers } +#### HTTP Status handling {: #conf-status_handlers } If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to do based on the status. @@ -608,7 +638,7 @@ A range of statuses can be specified as a number (`1-5`) followed by lowercase ` Status handlers cannot be specified in the CLI. -### Custom actions +#### Custom actions You can also define a custom action in `status_handlers` as: * a string in the form `'my_module:custom_action'`, which names a handler @@ -623,143 +653,142 @@ Freezeyt's default actions, like `follow`, can be imported from `freezeyt.action A custom action should call one of these default actions and return the return value from it. -## URL finding {: #conf-url_finders } - -`freezeyt` discovers new links in the application by URL finders. URL finders -are functions whose goal is to find url of specific MIME type. -`freezeyt` offers different configuration options to use URL finders: - -* use predefined URL finders for `text/html` or `text/css` (default), -* define your own URL finder as your function, -* turn off some of finders (section below) +### Path generation -Example of configuration: +It is possible to customize the filenames that URLs are saved under +using the `url_to_path` configuration key, for example: ```yaml -url_finders: - text/html: get_html_links - text/css: get_css_links +url_to_path: my_module:url_to_path ``` -Keys in the `url_finders` dict are MIME types; - -Values are URL finders, which can be defined as: -* strings in the form `"module:function"`, which name the finder - function to call, -* strings like `get_html_links`, which name a function from the - `freezeyt.url_finders` module, or -* Python functions (if configuring `freezeyt` from Python rather than +The value can be: +* a strings in the form `"module:function"`, which names the + function to call (the function can be omitted along with the colon, + and defaults to `url_to_path`), or +* a Python function (if configuring `freezeyt` from Python rather than YAML). +The function receives the *path* of the URL to save, relative to the `prefix`, +and should return a path to the saved file, relative to the build directory. -An URL finder gets these arguments: -* page content `BinaryIO`, -* the absolute URL of the page, as a `string`, -* the HTTP headers, as a list of tuples (WSGI). +The default function, available as `freezeyt.url_to_path`, adds `index.html` +if the URL path ends with `/`. -The function should return an iterator of all URLs (as strings) found -in the page's contents, as they would appear in `href` or `src` attributes. -Specifically: +`url_to_path` cannot be specified in the CLI. -- The URLs can be relative. -- External URLs (i.e. those not beginning with `prefix`) should be included. +### Plugins {: #conf-plugins } -Finder functions may be asynchronous: -- The function can be defined with `async def` (i.e. return a - coroutine). If it is, freezeyt will use the result after `await`. -- The function may be an asynchronous generator (defined with `async def` - and use `yield`). If so, freezeyt will use async iteration to handle it. +It is possible to extend `freezeyt` with *plugins*, either ones that +ship with `freezeyt` or external ones. -The `freezeyt.url_finders` module includes: -- `get_html_links`, the default finder for HTML -- `get_css_links`, the default finder for CSS -- `get_html_links_async` and `get_css_links_async`, asynchronous variants - of the above -- `none`, a finder that doesn't find any links. +Plugins are added using configuration like: -URL finders cannot be specified in the CLI. +```yaml +plugins: + - freezeyt.progressbar:ProgressBar + - mymodule:my_plugin +``` -### URL finder header -You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. -If given, it overrides the `url_finders` configuration. +### Middleware static mode {: #static_mode } -### Default `get_html_links` +When using the `freezeyt` middleware, you can enable *static mode*, +which simulates behaviour after the app is saved to static pages: -The default URL finder for HTML pages looks in `src` and `href` attributes -of all tags in the document. -It currently does not handle other links, such as embedded CSS, but it -may be improved in the future. +```yaml +static_mode: true +``` -### Default `get_css_links` +Currently in static mode: +- HTTP methods other than GET and HEAD are disallowed. +- URL parameters are removed +- The request body is discarded -The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all -links in a stylesheet. +Other restrictions and features may be added in the future, without regard +to backwards compatibility. +The static mode is intended for interactive use -- testing your app without +having to freeze all of it after each change. -### Disabling default URL finders {: #conf-use_default_url_finders } -If a finder is not explictly specified in the configuration file, `freezeyt` will use the -default for certain MIME type. For example, if you specify -`text/html: my_custom_finder` only, `freezeyt` will use the default finder -for `text/css`. -You can disable this behaviour: +## Extending *freezeyt* -```yaml -use_default_url_finders: false -``` +### Custom plugins -### Finding URLs in Link headers {: #conf-urls_from_link_headers } +A plugin is a function that `freezeyt` will call before starting to +freeze pages. -By default, `freezeyt` will follow URLs in `Link` HTTP headers. -To disable this, specify: +It is passed a `FreezeInfo` object as argument (see the `start` hook below). +Usually, the plugin will call `freeze_info.add_hook` to register additional +functions. -```yaml -urls_from_link_headers: false -``` +### Hooks {: #conf-hooks } -## Path generation +It is possible to register *hooks*, functions that are called when +specific events happen in the freezing process. -It is possible to customize the filenames that URLs are saved under -using the `url_to_path` configuration key, for example: +For example, if `mymodule` defines functions `start` and `page_frozen`, +you can make freezeyt call them using this configuration: ```yaml -url_to_path: my_module:url_to_path +hooks: + start: + - mymodule:start + page_frozen: + - mymodule:page_frozen ``` -The value can be: -* a strings in the form `"module:function"`, which names the - function to call (the function can be omitted along with the colon, - and defaults to `url_to_path`), or -* a Python function (if configuring `freezeyt` from Python rather than - YAML). +When using the Python API, a function can be used instead of a name +like `mymodule:start`. -The function receives the *path* of the URL to save, relative to the `prefix`, -and should return a path to the saved file, relative to the build directory. +#### `start` -The default function, available as `freezeyt.url_to_path`, adds `index.html` -if the URL path ends with `/`. +The function will be called when the freezing process starts, +before any other hooks. -`url_to_path` cannot be specified in the CLI. +It is passed a `FreezeInfo` object as argument. +The object has the following attributes: +* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. + If that URL was frozen already, or is outside the `prefix`, does nothing. + If you add a `reason` string, it will be used in error messages as the reason + why the added URL is being handled. +* `add_hook(hook_name, callable)`: Register an additional hook function. +* `total_task_count`: The number of pages `freezeyt` currently “knows about” – + ones that are already frozen plus ones that are scheduled to be frozen. +* `done_task_count`: The number of pages that are done (either successfully + frozen, or failed). +* `failed_task_count`: The number of pages that failed to freeze. -## Middleware static mode {: #static_mode } +#### `page_frozen` -When using the `freezeyt` middleware, you can enable *static mode*, -which simulates behaviour after the app is saved to static pages: +The function will be called whenever a page is processed successfully. +It is passed a `TaskInfo` object as argument. +The object has the following attributes: -```yaml -static_mode: true -``` +* `get_a_url()`: returns a URL of the page, including `prefix`. + Note that a page may be reachable via several URLs; this function returns + an arbitrary one. +* `path`: the relative path the content is saved to. +* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. +* `exception`: for failed tasks, the exception raised; + `None` otherwise. +* `reasons`: A list of strings explaining why the given page was visited. + (Note that as the freezing progresses, new reasons may be added to + existing tasks.) -Currently in static mode: -- HTTP methods other than GET and HEAD are disallowed. -- URL parameters are removed -- The request body is discarded -Other restrictions and features may be added in the future, without regard -to backwards compatibility. -The static mode is intended for interactive use -- testing your app without -having to freeze all of it after each change. +#### `page_failed` + +The function will be called whenever a page is not saved due to an +exception. +It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). + + +#### `success` + +The function will be called after the app is successfully frozen. +It is passed a `FreezeInfo` object as argument (see the `start` hook). diff --git a/mkdocs.yml b/mkdocs.yml index 4e8f0179..114a9d17 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -13,6 +13,7 @@ theme: name: mkdocs color_mode: auto user_color_mode_toggle: true + navigation_depth: 3 # highlightjs: false # disables user_color_mode_toggle?! markdown_extensions: - toc: From aae004577392bbacb9e9f27a2450ead1f7a0d15f Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 13 Aug 2024 19:03:51 +0200 Subject: [PATCH 09/16] Rearranging and rewording --- docs/config.md | 133 ++++++++++++++++++++++++++----------------------- 1 file changed, 70 insertions(+), 63 deletions(-) diff --git a/docs/config.md b/docs/config.md index 341b677f..510c5618 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,15 +1,36 @@ - # Configuration +Freezeyt is primarily configured using a dictionary of options, +usually loaded from a YAML (or JSON) file given on the command line (CLI) +using the `-c/--config` argument, for example: + +```console +$ python -m freezeyt my_app _build -c freezeyt.yaml +``` + +See [below](#example) for an example of what goes in the file. + +Instead of a file, you can also put the configuration in a Python +dictionary and tell *freezeyt* to import it using the `-C/--import-config` +argument. +Like the application to freeze, the argument takes the name of an importable +module and the name of a variable in that module, separated by a colon. +For example: + +```console +$ python -m freezeyt my_app _build -C my_app:freezeyt_config +``` + ## CLI -The most common options for freezeyt can be given on the command line. -These are usually shortcuts for the more powerful [YAML-based configuration](#yaml-config), -or ways to load that configuration. +Most command-line arguments correspond directly to an configuration option. +Unless documented otherwise, CLI arguments will override values +from a file or dictionary. + +Here is a full list of CLI arguments: -| CLI option | YAML key | Meaning | +| CLI argument | Option name | Meaning | |----------|------------|----------| -| `--help` | --- | Show help and exit | | APP (positional) | `app` | [Application to freeze](#conf-app) | | `-o`, `--output`, positional | `output` | [Output directory](#conf-output) | | `-c`, `--config` | --- | [Configuration file](#conf-cli-config) | @@ -20,38 +41,12 @@ or ways to load that configuration. | `--gh-pages` | `gh_pages` | [Github Pages Plugin](#conf-gh_pages) | | `--no-cleanup` | `cleanup` | Don't [clean up](#conf-cleanup) | | `-x`, `--fail-fast` | `fail_fast` | [Fail fast](#conf-fail_fast) | +| `--help` | --- | Show help and exit | +## Example {: #example } -### Configuration file (`-c`) {: #conf-cli-config } - -You can specify a config file in YAML (or JSON) format using the `-c/--config` -option, for example: - -```shell -$ python -m freezeyt my_app _build -c freezeyt.yaml -``` - -### Configuration variable (`-C`) {: #conf-cli-import-config } - -Instead of a YAML file, you can also put the configuration in a Python -dictionary and tell *freezeyt* to import it using the `-C/--import-config` -option. -Like the application to freeze, the option takes the name of an importable -module and the name of a variable in that module, separated by a colon. -For example: - -```shell -$ python -m freezeyt my_app _build -C my_app:freezeyt_config -``` - - -## Overview - -While common options can be given on the command line, -you can have full control over the freezing process with a YAML -configuration file or a variable with the configuration. - -Here is an example configuration file: +Here's an example YAML configuration file. +See below for descriptions of the individual options. ```yaml output: ./_build/ # The website will be saved to this directory @@ -75,15 +70,14 @@ status_handlers: "3xx": warn ``` +## Overview of the options + The following options are configurable: -| YAML key | Meaning | Example | +| Option name | Meaning | Example | |----------|---------|---------| | `app` | [Application to freeze](#conf-app) | `'module:wsgi_app'` | | `output` | [Output directory](#conf-output) | `'./_build/'` | -| --- | [Configuration file](#conf-cli-config) | `'./freezeyt.yaml/'` | -| --- | [Configuration variable](#conf-cli-import-config) | `'module:conf'` | -| --- | [Show help](#conf-cli-help) | --- | | `prefix` | [URL prefix](#conf-prefix) | `'https://mysite.example.com/subpage/'` | | `extra_pages` | [Extra pages](#conf-extra_pages) | (list) | | `extra_files` | [Extra files](#conf-extra_files) | (dict) | @@ -93,7 +87,6 @@ The following options are configurable: | `default_mimetype` | [Default MIME type](#conf-default_mimetype) | `text/plain` | | `get_mimetype` | [MIME type getter](#conf-default_mimetype) | `module:your_function` | | `mime_db_file` | [MIME type database](#conf-mime_db_file) | `path/to/mime-db.json` | -| (plugins) | [Progress bar and logging](#conf-cli-progress) | `log` | | `version` | [Configuration version](#conf-version) | `1` | | `plugins` | [Plugins](#conf-plugins) | (dict) | | `hooks` | [Hooks](#conf-hooks) | (dict) | @@ -120,24 +113,33 @@ version: 1 This is not mandatory. If the version is not given, the configuration may not work in future versions of freezeyt. -The version parameter is not accepted on the command line. ### Application to freeze {: #conf-app } -The module that contains the application must be given on the command line as first argument or in the configuration file. Freezeyt looks for the variable *app* by default. A different variable can be specified using `:`. -When the module is specified both by the command line and the config file +The name of importable Python module that contains the application must be +given in the configuration, or on the command line as first argument. + +Inside the module, *freezeyt* looks for the variable *app* by default. +A different variable can be specified after the module name, separated by +a colon (`:`). +When the module is specified both on the command line and in the config file, an error is raised. -Examples: +When the configuration is a Python dict, `app` can be given directly as +the WSGI application object, rather than a string. + +#### Examples Freezeyt looks for the variable `app` inside the module by default. +In YAML, it looks like this: + ```yaml app: app_module ``` If `app` is in a submodule, separate package names with a dot: ```yaml -app: folder1.folder2.app_module +app: app_package.wsgi ``` A different variable name can be specified by using `:`. @@ -145,34 +147,30 @@ A different variable name can be specified by using `:`. app: app_module:wsgi_application ``` -If the variable is an attribute of some namespace, use dots in the variable name: +In Python, the app can be given directly: -```yaml -app: app_module:namespace.wsgi_application -``` +```python +my_app = Flask(__name__) +... -When configuration is given as a Python dict, `app` can be given as the WSGI application object, rather than a string. +freezeyt_config = {'app': my_app} +``` ### Output {: #conf-output } -To outupt the frozen website to a directory, specify -the directory name: +To output the frozen website to a directory, specify +the directory name as a string: ```yaml output: ./_build/ ``` -Or use the full form – using the `dir` *saver*: - -```yaml -output: - type: dir - dir: ./_build/ -``` +If output is not specified in the configuration, +you must specify it on the command line, either by the `--output` (`-o`) +option or as a second positional argument. -If output is not specified in the configuration file, -you must specify the output directory on the command line. -There are two ways to specify the output on the command line: either by the `--output` (`-o`) option or as a second positional argument. +--- + The output must be specified just by one way otherwise is an error. @@ -181,6 +179,15 @@ freezeyt will either remove it (if the content looks like a previously frozen website) or raise an error. Best practice is to remove the output directory before freezing. +#### The full form + +Or use the full form – using the `dir` *saver*: + +```yaml +output: + type: dir + dir: ./_build/ +``` #### Output to dictionary From 70fbffbd6fbc4ead7e79ab708de4730f51ccfc90 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 20 Aug 2024 18:59:57 +0200 Subject: [PATCH 10/16] Continue rewording & rearranging --- docs/config.md | 360 +++++++++++++++++++++++++++++-------------------- 1 file changed, 211 insertions(+), 149 deletions(-) diff --git a/docs/config.md b/docs/config.md index 510c5618..7d45ea0e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -158,42 +158,32 @@ freezeyt_config = {'app': my_app} ### Output {: #conf-output } -To output the frozen website to a directory, specify -the directory name as a string: +The `output` option conifgures the output directory, where the +result is saved: ```yaml output: ./_build/ ``` -If output is not specified in the configuration, -you must specify it on the command line, either by the `--output` (`-o`) -option or as a second positional argument. +Alternatively, the output directory can be specified on the command line, +either by the `--output` (`-o`) argument or as a second positional argument. ---- - - -The output must be specified just by one way otherwise is an error. +The output must be specified in only one way; providing both the config +option and CLI argument is an error. If there is any existing content in the output directory, freezeyt will either remove it (if the content looks like a previously frozen website) or raise an error. Best practice is to remove the output directory before freezing. -#### The full form - -Or use the full form – using the `dir` *saver*: - -```yaml -output: - type: dir - dir: ./_build/ -``` - #### Output to dictionary -For testing, `freezeyt` can output to a dictionary rather than save -files to the disk. -This can be configured with: +*Freezeyt* can return the result in a dictionary, +rather than save it to disk. +Note that this stores the entire frozen website in memory, +so it is mostly useful for testing. +This can be configured by setting `output` to the dictionary +`{'type': 'dict'}`, rather than to a string: ```yaml output: @@ -217,13 +207,27 @@ will be represented as: } ``` -This is not useful in the CLI, as the return value is lost. +#### Explicitly output to disk + +It is possible to explicitly request *freezeyt* to output to a directory +using a dict like this: + +```yaml +output: + type: dir + dir: ./_build/ +``` + +This is equivalent to the string `./_build/`, or passing `./_build/` as the +CLI argument. + +There are currently no output types other that `dir` and `dict`. ### URL prefix {: #conf-prefix } The URL where the application will be deployed can be -specified with: +specified with `prefix`, for example: ```yaml prefix: http://localhost:8000/ @@ -233,8 +237,22 @@ or prefix: https://mysite.example.com/subpage/ ``` -Freezeyt will freeze all pages starting with `prefix` that -it finds. +The *prefix* URL must end with a slash. + +The page at the *prefix* URL is considered the application's “home page”, +and will always be frozen. + +*freezeyt* considers all pages under the prefix to be part of +the application. +For example, with the second prefix above: + +- `https://mysite.example.com/subpage/blog.html` would be followed, + and the page would be frozen at `/blog.html`; +- `https://mysite.example.com/about.html` would be considered an external link, + and ignored. + +The prefix is also passed to the application as the server and script name, +which should be used whenever the app generates absolute URLs. The prefix can also be specified on thecommand line with e.g.: `--prefix=http://localhost:8000/`. @@ -243,6 +261,19 @@ The CLI argument has priority over the config file. ## Extra content +Usually, *freezeyt* saves pages that are reachable by links from the app's +home page. +There are two cases when this is not enough, and you need to specify +extra content manually: + +- Extra *pages* are part of the application, but not reachable by following + links. For example, a an old URL that redirects to a new location should + be configured as an extra page. + +- Extra *files* are not part of the application. + Typically, these are used to configure the static page server, like + a `CNAME` file GitHub's or `.htaccess` for Apache. + ### Extra pages {: #conf-extra_pages } @@ -251,18 +282,22 @@ can specified as “extra” pages in the configuration: ```yaml extra_pages: - - /extra/ - - /extra2.html + - extra/ + - extra2.html ``` -Freezeyt will handle these pages as if it found them by following links. -(For example, by default it will follow links in extra pages.) +The URLs should be relative (to the [prefix](#conf-prefix)). +Absolute URLs are allowed, but they must start with the prefix. + +Freezeyt will handle these pages as if it found them as links. +For example, by default it will follow links in extra pages. -Extra pages may also be given on the command line, -e.g. `--extra-page /extra/ --extra-page /extra2.html`. +Extra pages may also be given with the `--extra-page` command line argument, +which can be repeated (for example, +`--extra-page extra/ --extra-page extra2.html`). The lists from CLI and the config file are merged together. -You can also specify extra pages using a Python generator, +You can also specify extra pages using a Python function, specified using a module name and function name as follows: ```yaml @@ -270,20 +305,24 @@ extra_pages: - generator: my_app:generate_extra_pages ``` -The `generate_extra_pages` function should take the application -as argument and return an iterable of URLs. +This function should take the application as argument and return an iterable +of URLs as strings. -When using the Python API, a generator for extra pages can be specified +When using the Python API, this function can be specified directly as a Python object, for example: ```python +def generate_extra_pages(app): + yield 'extra/' + yield 'extra2.html' + config = { ... - 'extra_pages': [{'generator': my_generator_function}], + 'extra_pages': [{'generator': generate_extra_pages}], } another_config = { ... - 'extra_pages': [my_generator_function], + 'extra_pages': [generate_extra_pages], } ``` @@ -292,28 +331,33 @@ another_config = { Extra files to be included in the output can be specified, along with their content. -These files are not considered part of the app; freezeyt will not try to find -links in them. - -This is useful for configuration of your static server. -(For pages that are part of your website, we recommend -adding them to your application rather than as extra files.) - -If you specify backslashes in `url part`, `freezeyt` convert them to forward slashes. - -For example, the following config will add 3 files to -the output: +For example, the following config will add 3 files to the output: ```yaml extra_files: - CNAME: mysite.example.com + CNAME: mysite.example.com ".nojekyll": '' - config/xyz: abc + config/xyz: abc ``` -You can also specify extra files using Base64 encoding or -a file path, like so: +The files will be: +- `/CNAME`, with the content `mysite.example.com`; +- `/.nojekyll`, empty; +- `/config/xyz`, with the content `abc`. + +These files are not considered part of the application. +*Freezeyt* will not retrieve them from the app, and it will not try to find +links in them. + +Extra files are mainly useful for configuration of your static server. +For files that are part of the website, +such as a [favicon](https://en.wikipedia.org/wiki/Favicon), we recommend +adding them to your application, and either link to them or add them as +[extra *pages*](#conf-extra_pages). + +You can also specify extra file content using the Base64 encoding (`base64`) or +as a filesystem path to be copied (`copy_from`), like so: ```yaml extra_files: @@ -323,160 +367,138 @@ extra_files: copy_from: included/config2.dat ``` -It's possible to recursively copy an entire directory using `copy_from`, -as in: +If the `copy_from` path names a directory, it will be copied recursively. - -```yaml -extra_files: - static: - copy_from: static/ -``` +In the file name, *freezeyt* treats both backslashes and forward slashes +as path separators. Extra files cannot be specified on the CLI. ## Debugging options +The following options are useful when debugging your application, +or its integration with *freezeyt*. + ### Clean up {: #conf-cleanup } -If an error occurs during the "freeze" process, Freezeyt will delete the incomplete output directory. -This prevents, for example, uploading incomplete results to a web hosting by mistake. +By default, if an error occurs during freezing, *freezeyt* will delete +the incomplete output directory. +This is meant to prevent uploading incomplete results to web hosting by mistake. If you want to keep the incomplete directory (for example, -to help debugging), you can use the `--no-cleanup` switch -or the `cleanup` key in the configuration file: +to help debugging), you can use the `--no-cleanup` command line switch +or the `cleanup` configuration option: + +```shell +$ freezeyt app -o ./build/ --no-cleanup +``` ```yaml cleanup: False ``` The command line switch has priority over the configuration. -Use `--no-cleanup` to override `cleanup: False` from the config. +Use `--cleanup` to override `cleanup: False` from the config. ### Fail fast {: #conf-fail_fast } -Fail fast mode stops the freezing of the app when the first error occurs. - -As with most settings, fail fast can be set both in the configuration file and in the command line using switches (the command line always overrides the configuration file). -The fail fast is defined as `boolean` option. - -If you want to specified fail fast in configuration file, use key `fail_fast` with - `boolean` values `True` or `False`: - -```yaml -fail_fast: True -``` - -If you want to specified fail fast from command line, use switches - `--fail-fast` (short `-x`) resp. `--no-fail-fast` to disable it: +By default, *freezeyt* collects errors it finds on individual pages, +and presents them all when done. +To stop the process early when the first error occurs, use the +the `--fail-fast` (`-x`) command line switch or the `fail_fast` configuration option: ```shell -$ freezeyt app -o output -x +$ freezeyt app -o ./build/ --fail-fast ``` - -## Built-in plugins - - -### Github Pages Plugin {: #conf-gh_pages } - -To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. - -By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. - -Configuration example: ```yaml -gh_pages: True -``` - -To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. -You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: -```shell -git fetch output_dir gh-pages -git branch --force gh-pages FETCH_HEAD +fail_fast: True ``` -Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. - - -### Progress bar and logging {: #conf-cli-progress } -The CLI option `--progress` controls what `freezeyt` outputs as it -handles pages: +The command line switch has priority over the configuration. +Use `--no-fail-fast` to override `fail_fast: True` from the config. -* `--progress=log`: Output a message about each frozen page to stdout. -* `--progress=bar`: Draw a status bar in the terminal. Messages about - each frozen page are *also* printed to stdout. -* `--progress=none`: Don't do any of this. -The default is `bar` if stdout is a terminal, and `log` otherwise. +## Customizing the process -It is possible to configure this in the config file using the plugins -`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. -See below on how to enable plugins. +Here are ways to configure details of how *freezeyt* saves pages. -## Freezing +### MIME type checking +When web pages are saved to files on disk, some information is lost. +The most prominent piece of lost info is the +[`Content-Type`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type) +HTTP header, that is, the document's MIME type. -### Comparison of MIME type and file type +Static page servers typically look at a file's extension to determine +the `Content-Type` -- for example, `index.html` is served as a HTML document +(`text/html`) and `smile.png` is served as a PNG image (`image/png`). -Freezeyt checks whether the file extensions in its output +To ensure that the application will work as intended when frozen and served +with such a server, *freezeyt* verifies that the extensions of saved files correspond to the MIME types served by the app. -If there's a mismatch, freezeyt fails, because this means a server -wouldn't be able to serve the page correctly. -This funtionality is provided by `freezeyt.Middleware`. +This funtionality is provided by [`freezeyt.Middleware`](/#middleware). + +The exact mapping between extensions and `Content-Type` values varies +between servers. +*Freezeyt* uses Python's `mimetypes` module by default, but provides +several ways to customize it. #### Default MIME type {: #conf-default_mimetype } -It is possible to specify the MIME type used for files without an extension. -For example, if your server of static pages defaults to plain text files, +Files without an extension are, by default, served as `application/octet-stream` +(arbitrary binary data). +This can be configured using the `default_mimetype` option. +For example, if your static page server defaults to plain text files, use: ```yaml default_mimetype=text/plain ``` -If the default MIME type isn't explicitly configured in YAML configuration, -then the `freezeyt` uses value `application/octet-stream`. - -The default mimetype cannot be specified on the CLI. +---- + #### MIME type getter {: #conf-get_mimetype } -There is possibility to modify the way how to determine file type -from file extension. -You can setup your own `get_mimetype` function. - -Freezeyt will register your own function, if you specify it in configuration -YAML file as: +The most flexible way to map file extensions to MIME types is with +a custom function, which you can specify using the `get_mimetype` option. +For example: ```yaml get_mimetype=module:your_function ``` -If the `get_mimetype` is not defined in configuration file, -then `freezeyt` calls the python function `mimetypes.guess_type` -and uses the mimetype (the first element) it returns. - -`get_mimetype` can be defined as: -* strings in the form `"module:function"`, which name the function to call, -* Python functions (if configuring `freezeyt` from Python, e.g. as a `dict`, - rather than YAML). +`get_mimetype` can be defined as a string in the form `"module:function"`, +which names the function to call, or as a Python function +(if configuring `freezeyt` using a Python dict). -The `get_mimetype`: -* gets one argument - the `filepath` as `str` +The function will be called with one argument, the file path as a string, and +it should returns a list of corresponding MIME types +(for example, `["text/html"]` or `["audio/wav", "audio/wave"]`). -* returns file MIME types as a `list` of MIME types - (e.g. `["text/html"]` or `["audio/wav", "audio/wave"]`). +If `get_mimetype` instead returns `None`, `freezeyt` will use the +[default MIME type](#conf-default_mimetype). -If `get_mimetype` returns `None`, `freezeyt` will use the configured `default_mimetype` -(see *Default MIME type* above). +By default, `freezeyt` calls the Python function +[`mimetypes.guess_type`](https://docs.python.org/3/library/mimetypes.html#mimetypes.guess_type) +and uses the `type` (the first element) of the result: -The get_mimetype function cannot be specified on the CLI. +```python +def default_mimetype(url: str) -> Optional[List[str]]: + file_mimetype, encoding = guess_type(url) + if file_mimetype is None: + # Freezeyt should use the default + return None + else: + # A one-element list + return [file_mimetype] +``` #### Using a mime-db database {: #conf-mime_db_file } @@ -551,7 +573,7 @@ The `freezeyt.url_finders` module includes: URL finders cannot be specified in the CLI. -#### URL finder header +#### URL finder header {: #conf-header-Freezeyt-URL-Finder } You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. If given, it overrides the `url_finders` configuration. @@ -698,7 +720,47 @@ plugins: ``` -### Middleware static mode {: #static_mode } +## Built-in plugins + + +### Github Pages Plugin {: #conf-gh_pages } + +To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. + +By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. + +Configuration example: +```yaml +gh_pages: True +``` + +To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. +You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: +```shell +git fetch output_dir gh-pages +git branch --force gh-pages FETCH_HEAD +``` +Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. + + +### Progress bar and logging {: #conf-cli-progress } + +The CLI option `--progress` controls what `freezeyt` outputs as it +handles pages: + +* `--progress=log`: Output a message about each frozen page to stdout. +* `--progress=bar`: Draw a status bar in the terminal. Messages about + each frozen page are *also* printed to stdout. +* `--progress=none`: Don't do any of this. + +The default is `bar` if stdout is a terminal, and `log` otherwise. + +It is possible to configure this in the config file using the plugins +`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. +See below on how to enable plugins. + + +## Middleware static mode {: #static_mode } When using the `freezeyt` middleware, you can enable *static mode*, which simulates behaviour after the app is saved to static pages: From 9d694e99bfaf28d5752c627d3688cbd1b281a76b Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 27 Aug 2024 18:45:10 +0200 Subject: [PATCH 11/16] Continue rewording & reorganizing docs --- docs/config.md | 289 ++++++++++++++++++++++++------------------------ docs/index.md | 2 +- docs/licence.md | 4 + docs/pyapi.md | 122 ++++++++++++++++++++ mkdocs.yml | 1 + 5 files changed, 274 insertions(+), 144 deletions(-) create mode 100644 docs/licence.md create mode 100644 docs/pyapi.md diff --git a/docs/config.md b/docs/config.md index 7d45ea0e..068dd044 100644 --- a/docs/config.md +++ b/docs/config.md @@ -461,8 +461,6 @@ use: default_mimetype=text/plain ``` ----- - #### MIME type getter {: #conf-get_mimetype } @@ -503,79 +501,84 @@ def default_mimetype(url: str) -> Optional[List[str]]: #### Using a mime-db database {: #conf-mime_db_file } -There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json), -or a database with the same structure. -(This is the database used by GitHub Pages). -The database will be used to get file MIME type from file suffix. +There is an option to use [the MIME type database from the `jshttp` project](https://github.com/jshttp/mime-db/blob/master/db.json) +(the database used by GitHub Pages), +or a database with the same structure, for mapping file names to MIME types. -To use this database, add the path to the JSON file to `freezeyt` configuration: +To use it, add the path to the JSON file to `freezeyt` configuration: ```yaml mime_db_file=path/to/mime-db.json ``` This is equivalent to setting `get_mimetype` to a function that maps extensions to filetypes according to the database. -The mime_db file cannot be specified on the CLI. ### URL finding {: #conf-url_finders } -`freezeyt` discovers new links in the application by URL finders. URL finders -are functions whose goal is to find url of specific MIME type. -`freezeyt` offers different configuration options to use URL finders: +`freezeyt` discovers new pages in the application by searching for URLs +in pages it processes. The search is done by *URL finders*, +functions that find URLs in a specific page type. -* use predefined URL finders for `text/html` or `text/css` (default), -* define your own URL finder as your function, -* turn off some of finders (section below) - -Example of configuration: +You can configure which finder is used for which MIME type using +the `url_finders` configuration key. +In the default configuration, freezeyt finds links in HTML and CSS files. +The default could be configured like this: ```yaml url_finders: - text/html: get_html_links - text/css: get_css_links + text/html: freezeyt.url_finders:get_html_links + text/css: freezeyt.url_finders:get_css_links ``` -Keys in the `url_finders` dict are MIME types; +Keys in the `url_finders` dict are MIME types. +Values are functions, which can be defined as: + +* Strings in the form `"module:function"`, which name the finder + function to call. +* Python functions (if configuring `freezeyt` from Python). +* Strings without a colon (`:`), which name a function from the + `freezeyt.url_finders` module. Using this shortcut, the default configuration + could also be written as: + + url_finders: + text/html: get_html_links + text/css: get_css_links + + + The `freezeyt.url_finders` module includes these finders: -Values are URL finders, which can be defined as: -* strings in the form `"module:function"`, which name the finder - function to call, -* strings like `get_html_links`, which name a function from the - `freezeyt.url_finders` module, or -* Python functions (if configuring `freezeyt` from Python rather than - YAML). + - `get_html_links`, the default finder for HTML + - `get_css_links`, the default finder for CSS + - `get_html_links_async` and `get_css_links_async`, asynchronous variants + of the above + - `none`, a finder that doesn't find any links. +An URL finder function gets these arguments: -An URL finder gets these arguments: -* page content `BinaryIO`, -* the absolute URL of the page, as a `string`, -* the HTTP headers, as a list of tuples (WSGI). +* The page content, as a binary file open for reading (for example, + `io.BinaryIO`), +* the absolute URL of the page, as a `str`, and +* the HTTP headers, as a list of 2-tuples (as in WSGI). The function should return an iterator of all URLs (as strings) found in the page's contents, as they would appear in `href` or `src` attributes. Specifically: - The URLs can be relative. -- External URLs (i.e. those not beginning with `prefix`) should be included. +- External URLs (i.e. those not beginning with the [`prefix`](#conf-prefix)) + should be included. -Finder functions may be asynchronous: -- The function can be defined with `async def` (i.e. return a - coroutine). If it is, freezeyt will use the result after `await`. -- The function may be an asynchronous generator (defined with `async def` - and use `yield`). If so, freezeyt will use async iteration to handle it. +Finder functions may be asynchronous. +If the function returns a coroutine (for example, if it's defined with +`async def`, freezeyt will use `await` on the result. +If the function returns an asynchronous generator (for example, if it's +defined with `async def` and uses `yield`), freezeyt will use async iteration +to handle it. -The `freezeyt.url_finders` module includes: -- `get_html_links`, the default finder for HTML -- `get_css_links`, the default finder for CSS -- `get_html_links_async` and `get_css_links_async`, asynchronous variants - of the above -- `none`, a finder that doesn't find any links. - -URL finders cannot be specified in the CLI. #### URL finder header {: #conf-header-Freezeyt-URL-Finder } -You can specify a finder in the `Freezeyt-URL-Finder` HTTP header. +You can specify a finder as a string in the `Freezeyt-URL-Finder` HTTP header. If given, it overrides the `url_finders` configuration. #### Default `get_html_links` @@ -587,17 +590,18 @@ may be improved in the future. #### Default `get_css_links` -The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) library to find all -links in a stylesheet. +The default URL finder for CSS uses the [`cssutils`](https://pypi.org/project/cssutils/) +library to find all links in a stylesheet. +This may be changed in the future. #### Disabling default URL finders {: #conf-use_default_url_finders } -If a finder is not explictly specified in the configuration file, `freezeyt` will use the -default for certain MIME type. For example, if you specify +If a finder is not explictly specified in the configuration file, +`freezeyt` will use the default. For example, if you specify `text/html: my_custom_finder` only, `freezeyt` will use the default finder for `text/css`. -You can disable this behaviour: +You can disable this behavior: ```yaml use_default_url_finders: false @@ -616,19 +620,21 @@ urls_from_link_headers: false ### Freeze actions -By default, `freezeyt` will save the pages it finds. -You can instruct it to instead ignore certain pages, or treat them as errors. -This is most useful as a response to certain HTTP statuses (e.g. treat all -`404 NOT FOUND` pages as errors), but can be used independently. +For each page it finds, `freezeyt` will take an *action*: save the page, +ignore it, or treat it as an error. + +By default, `freezeyt` will save the pages with a `200 OK` HTTP status code, +and raise an error for any other status code. -To tell `freezeyt` what to do from within the application (or middleware), -set the `Freezeyt-Action` HTTP header to one of these values: +You can configure the for an individual page from within the application +(or middleware), by setting the `Freezeyt-Action` HTTP header to one of +these strings: * `'save'`: `freezeyt` will save the body of the page. * `'ignore'`: `freezeyt` will not save any content for the page * `'warn'`: will save the content and send warn message to stdout -* `'follow'`: `freezeyt` will save content from the redirected location - (this requires a `Location` header, which is usually added for redirects). +* `'follow'`: `freezeyt` will save content from the redirected location. + This requires a `Location` header, which is usually added for redirects. Redirects to external pages are not supported. * `'error'`: fail; the page will not be saved and `freeze()` will raise an exception. @@ -637,18 +643,17 @@ set the `Freezeyt-Action` HTTP header to one of these values: #### HTTP Status handling {: #conf-status_handlers } If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to -do based on the status. -By default, `200 OK` pages are saved and any others cause errors. -The behavior can be customized using the `status_handlers` setting. -For example, to ignore pages with the `404 NOT FOUND` status, set the -`404` handler to `'ignore'`: +do based on the HTTP status. The behavior can be customized using the +`status_handlers` setting. +For example, to ignore pages with the `404 NOT FOUND` status, rather than +treat them as errors, set the `404` handler to `'ignore'`: ```yaml status_handlers: '404': ignore ``` -For example, `status_handlers` would be specified as: +More varied `status_handlers` could be specified as: ```yaml status_handlers: @@ -660,31 +665,32 @@ status_handlers: '5xx': error ``` -Note that the status code must be a string, so it needs to be quoted in the YAML file. +Note that the status code must be a string. +In a YAML file, it needs to be quoted. -A range of statuses can be specified as a number (`1-5`) followed by lowercase `xx`. +A range of statuses can be specified as one number (`1` to `5`) followed by +lowercase `xx`. (Other "wildcards" like `50x` are not supported.) -Status handlers cannot be specified in the CLI. #### Custom actions You can also define a custom action in `status_handlers` as: + * a string in the form `'my_module:custom_action'`, which names a handler - function to call, -* a Python function (if configuring `freezeyt` from Python rather than from - YAML). - -The action function takes one argument, `task` (TaskInfo): information about the freezing task. -See the `TaskInfo` hook for a description. -Freezeyt's default actions, like `follow`, can be imported from `freezeyt.actions` -(e.g. `freezeyt.actions.follow`). + function to call, or +* a Python function (if configuring `freezeyt` from Python). + +The action function takes one argument a [`TaskInfo`](pyapi.md#TaskInfo) +with information about the page being frozen. +Freezeyt's predefined actions, like `follow`, can be imported from +`freezeyt.actions`. A custom action should call one of these default actions and return the return value from it. ### Path generation -It is possible to customize the filenames that URLs are saved under +It is possible to customize the filenames that pages are saved under using the `url_to_path` configuration key, for example: ```yaml @@ -692,87 +698,104 @@ url_to_path: my_module:url_to_path ``` The value can be: -* a strings in the form `"module:function"`, which names the - function to call (the function can be omitted along with the colon, - and defaults to `url_to_path`), or -* a Python function (if configuring `freezeyt` from Python rather than - YAML). -The function receives the *path* of the URL to save, relative to the `prefix`, -and should return a path to the saved file, relative to the build directory. +* A string in the form `"module:function"`, which names the + function to call. The function can be omitted along with the colon, + and defaults to `url_to_path`. +* A Python function, if configuring `freezeyt` from Python. + +The function receives one string: the *path* portion of the URL to save, +relative to the `prefix`. +It should return a path to the saved file, relative to the build directory, +as a string. The default function, available as `freezeyt.url_to_path`, adds `index.html` -if the URL path ends with `/`. +if the URL ends with `/`. -`url_to_path` cannot be specified in the CLI. ### Plugins {: #conf-plugins } -It is possible to extend `freezeyt` with *plugins*, either ones that -ship with `freezeyt` or external ones. +It is possible to extend `freezeyt` with *plugins*, +either [“built-in” ones](#built-in-plugins) that ship with `freezeyt` +or [external ones](#custom-plugins). Plugins are added using configuration like: ```yaml plugins: - - freezeyt.progressbar:ProgressBar + - freezeyt.plugins:ProgressBarPlugin - mymodule:my_plugin ``` -## Built-in plugins +## Built-in plugins {: #built-in-plugins } ### Github Pages Plugin {: #conf-gh_pages } -To make it easier to upload frozen pages to ([Github Pages service](https://pages.github.com/)), you can also use the `--gh-pages` switch or the `gh_pages` key in the configuration file, which creates a gh-pages git branch in the output directory. +To make it easier to upload frozen pages to the [Github Pages service](https://pages.github.com/), +you can activate the GitHub Pages plugin using the `--gh-pages` CLI argument +or the `gh_pages` key in the configuration. +This creates a `gh-pages` git branch in the output directory. -By default, the Github Pages Plugin is not active, however, if you have activated this plugin in your configuration, you can always override the current configuration with `--no-gh-pages` switch in the CLI. +By default, the Github Pages Plugin is not active. However, if you have +activated it in your configuration, you can override the choice in the CLI with +`--no-gh-pages`. Configuration example: ```yaml gh_pages: True ``` -To deploy a site to Github, you can then work with the git repository directly in the output directory or pull the files into another repository/directory. -You can then pull/fetch files from the newly created gh-pages git branch in many ways, e.g: -```shell -git fetch output_dir gh-pages -git branch --force gh-pages FETCH_HEAD +This is a shortcut for adding `freezeyt.plugins:GHPagesPlugin` +to [`plugins`](#conf-plugins). + +To deploy a site to Github, you can then work with the git repository directly +in the output directory or pull the files into another repository/directory. +You can then pull/fetch files from the newly created `gh-pages` git branch in +many ways, for example: +```console +$ git fetch output_dir gh-pages +$ git branch --force gh-pages FETCH_HEAD ``` -Note: This will overwrite the current contents of the `gh-pages` branch, because of the `--force` switch. +Note that this will overwrite the current contents of the `gh-pages` branch, +because of the `--force` switch. ### Progress bar and logging {: #conf-cli-progress } -The CLI option `--progress` controls what `freezeyt` outputs as it +The CLI argument `--progress` controls what `freezeyt` outputs as it handles pages: * `--progress=log`: Output a message about each frozen page to stdout. * `--progress=bar`: Draw a status bar in the terminal. Messages about - each frozen page are *also* printed to stdout. + each frozen page are *also* printed to stdout, as with `log`. * `--progress=none`: Don't do any of this. The default is `bar` if stdout is a terminal, and `log` otherwise. -It is possible to configure this in the config file using the plugins -`freezeyt.progressbar:ProgressBarPlugin` and `freezeyt.progressbar:LogPlugin`. -See below on how to enable plugins. +Alternately, it is possible to configure logging by adding one of the following +[`plugins`](#conf-plugins): + +* `freezeyt.progressbar:ProgressBarPlugin` +* `freezeyt.progressbar:LogPlugin` ## Middleware static mode {: #static_mode } -When using the `freezeyt` middleware, you can enable *static mode*, +When using the [`freezeyt` middleware](./index.md#middleware), you can enable *static mode*, which simulates behaviour after the app is saved to static pages: ```yaml static_mode: true ``` -Currently in static mode: +Currently, in static mode: + - HTTP methods other than GET and HEAD are disallowed. - URL parameters are removed -- The request body is discarded +- Request bodies are discarded +- Non-default WSGI environ keys are removed Other restrictions and features may be added in the future, without regard to backwards compatibility. @@ -784,14 +807,14 @@ having to freeze all of it after each change. ## Extending *freezeyt* -### Custom plugins +### Custom plugins {: #custom-plugins } A plugin is a function that `freezeyt` will call before starting to freeze pages. -It is passed a `FreezeInfo` object as argument (see the `start` hook below). -Usually, the plugin will call `freeze_info.add_hook` to register additional -functions. +It is passed a [`FreezeInfo`](pyapi.md#FreezeInfo) object as argument. +Usually, the plugin will its [`add_hook`](pyapi.md#FreezeInfo.add_hook) method +to register additional functions. ### Hooks {: #conf-hooks } @@ -810,54 +833,34 @@ hooks: - mymodule:page_frozen ``` -When using the Python API, a function can be used instead of a name -like `mymodule:start`. +When configuring Freezeyt from Python, a function can be used directly +instead of a string. + +The available hooks are: #### `start` -The function will be called when the freezing process starts, -before any other hooks. +Called when the freezing process starts, before any other hooks. -It is passed a `FreezeInfo` object as argument. -The object has the following attributes: +Takes one argument: a [`FreezeInfo`](pyapi.md#FreezeInfo) object. -* `add_url(url, reason=None)`: Add the URL to the set of pages to be frozen. - If that URL was frozen already, or is outside the `prefix`, does nothing. - If you add a `reason` string, it will be used in error messages as the reason - why the added URL is being handled. -* `add_hook(hook_name, callable)`: Register an additional hook function. -* `total_task_count`: The number of pages `freezeyt` currently “knows about” – - ones that are already frozen plus ones that are scheduled to be frozen. -* `done_task_count`: The number of pages that are done (either successfully - frozen, or failed). -* `failed_task_count`: The number of pages that failed to freeze. #### `page_frozen` -The function will be called whenever a page is processed successfully. -It is passed a `TaskInfo` object as argument. -The object has the following attributes: +Called whenever a page is processed successfully. -* `get_a_url()`: returns a URL of the page, including `prefix`. - Note that a page may be reachable via several URLs; this function returns - an arbitrary one. -* `path`: the relative path the content is saved to. -* `freeze_info`: a `FreezeInfo` object. See the `start` hook for details. -* `exception`: for failed tasks, the exception raised; - `None` otherwise. -* `reasons`: A list of strings explaining why the given page was visited. - (Note that as the freezing progresses, new reasons may be added to - existing tasks.) +Takes one argument: a [`TaskInfo`](pyapi.md#TaskInfo) object. #### `page_failed` -The function will be called whenever a page is not saved due to an -exception. -It is passed a `TaskInfo` object as argument (see the `page_frozen` hook). +Called whenever a page is not saved due to an exception. + +Takes one argument: a [`TaskInfo`](pyapi.md#TaskInfo) object. #### `success` -The function will be called after the app is successfully frozen. -It is passed a `FreezeInfo` object as argument (see the `start` hook). +Called after the app is successfully frozen. + +Takes one argument: a [`FreezeInfo`](pyapi.md#FreezeInfo) object. diff --git a/docs/index.md b/docs/index.md index 8e7bd47c..685930ff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -147,7 +147,7 @@ you can call `freeze_async` instead of `freeze`. [asyncio]: https://docs.python.org/3/library/asyncio.html -### Middleware +### Middleware {: #middleware } Some of Freezeyt's functionality is available as a WSGI middleware. To use it, wrap your application in `freezeyt.Middeleware`. For example: diff --git a/docs/licence.md b/docs/licence.md new file mode 100644 index 00000000..1dfd8391 --- /dev/null +++ b/docs/licence.md @@ -0,0 +1,4 @@ + +Freezeyt is available under the following licence. + +# {!../LICENCE.MIT!} diff --git a/docs/pyapi.md b/docs/pyapi.md new file mode 100644 index 00000000..e54b722a --- /dev/null +++ b/docs/pyapi.md @@ -0,0 +1,122 @@ +# Freezeyt's Python API + +## Functions + +### `freezeyt.freeze(app, config)` + +### `freezeyt.freeze_async(app, config)` + + +## WSGI Middleware + +### `freezeyt.Middleware` + + +## Hook Arguments + +### `FreezeInfo` + + +#### `FreezeInfo.add_url(url, reason=None)` + +Add the URL to the set of pages to be frozen. + +If that URL was frozen already, or is outside the `prefix`, does nothing. + +If you add a `reason` string, it will be used in error messages as the reason +why the added URL is being handled. + +#### `FreezeInfo.add_hook(hook_name, callable)` + +Register an additional hook function. + +#### `FreezeInfo.total_task_count` + +The number of pages `freezeyt` currently “knows about” – +ones that are already frozen plus ones that are scheduled to be frozen. + +#### `FreezeInfo.done_task_count` + +The number of pages that are done (either successfully +frozen, or failed). + +#### `FreezeInfo.failed_task_count` + +The number of pages that failed to freeze. + +### `TaskInfo` {: #TaskInfo } + + +### `TaskInfo.get_a_url()`: + +Returns a URL of the page, including `prefix`. + +Note that a page may be reachable via several URLs; this function returns +an arbitrary one. + +### `TaskInfo.path` + +The relative path the content is saved to. + +### `TaskInfo.freeze_info` + +A [`FreezeInfo`]{: #FreezeInfo } object corresponding to the entire +freeze process. + +### `TaskInfo.exception` + +For failed tasks, the exception raised. `None` otherwise. + +### `TaskInfo.reasons` + +A list of strings explaining why the given page was visited. +(Note that as the freezing progresses, new reasons may be added to +existing tasks.) + +## Exceptions + +### `freezeyt.VersionMismatch` + +### `freezeyt.InfiniteRedirection` + +### `freezeyt.ExternalURLError` + +### `freezeyt.RelativeURLError` + +### `freezeyt.UnexpectedStatus` + +### `freezeyt.MultiError` + +### `freezeyt.DirectoryExistsError` + +## Types + +### `freezeyt.Config` + +## Actions + +### `freezeyt.actions.warn` +### `freezeyt.actions.ignore` +### `freezeyt.actions.follow` +### `freezeyt.actions.save` +### `freezeyt.actions.error` + +## URL finders + +### `freezeyt.url_finders.get_html_links` +### `freezeyt.url_finders.get_html_links_async` + +### `freezeyt.url_finders.get_css_links` +### `freezeyt.url_finders.get_css_links_async` + +### `freezeyt.url_finders.none` + +## Plugins + +### `freezeyt.plugins.ProgressBarPlugin` +### `freezeyt.plugins.LogPlugin` +### `freezeyt.plugins.GHPagesPlugin` + +## Utilities + +### `freezeyt.url_to_path` diff --git a/mkdocs.yml b/mkdocs.yml index 114a9d17..84d38fb1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -7,6 +7,7 @@ nav: - 'Home': 'index.md' - 'Configuration': 'config.md' - 'Contributing': 'contrib.md' + - 'API': 'pyapi.md' not_in_nav: licence.md theme: From 3d2a820b7740b6e531de2672d6cef32ad6ade17a Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 27 Aug 2024 18:49:22 +0200 Subject: [PATCH 12/16] Capitalize and format the project name consistently --- docs/config.md | 84 ++++++++++++++++++++++++------------------------- docs/contrib.md | 6 ++-- docs/index.md | 10 +++--- 3 files changed, 50 insertions(+), 50 deletions(-) diff --git a/docs/config.md b/docs/config.md index 068dd044..fa06e1c7 100644 --- a/docs/config.md +++ b/docs/config.md @@ -11,7 +11,7 @@ $ python -m freezeyt my_app _build -c freezeyt.yaml See [below](#example) for an example of what goes in the file. Instead of a file, you can also put the configuration in a Python -dictionary and tell *freezeyt* to import it using the `-C/--import-config` +dictionary and tell Freezeyt to import it using the `-C/--import-config` argument. Like the application to freeze, the argument takes the name of an importable module and the name of a variable in that module, separated by a colon. @@ -52,7 +52,7 @@ See below for descriptions of the individual options. output: ./_build/ # The website will be saved to this directory prefix: https://mysite.example.com/subpage/ extra_pages: - # Let freezeyt know about URLs that are not linked from elsewhere + # Let Freezeyt know about URLs that are not linked from elsewhere /robots.txt /easter-egg.html extra_files: @@ -103,7 +103,7 @@ The following options are configurable: ### Configuration version {: #conf-version } -To ensure that your configuration will work unchanged in newer versions of freezeyt, +To ensure that your configuration will work unchanged in newer versions of Freezeyt, you should add the current version number, `1`, to your configuration like this: ```yaml @@ -111,7 +111,7 @@ version: 1 ``` This is not mandatory. If the version is not given, the configuration may -not work in future versions of freezeyt. +not work in future versions of Freezeyt. ### Application to freeze {: #conf-app } @@ -119,7 +119,7 @@ not work in future versions of freezeyt. The name of importable Python module that contains the application must be given in the configuration, or on the command line as first argument. -Inside the module, *freezeyt* looks for the variable *app* by default. +Inside the module, Freezeyt looks for the variable *app* by default. A different variable can be specified after the module name, separated by a colon (`:`). When the module is specified both on the command line and in the config file, @@ -172,7 +172,7 @@ The output must be specified in only one way; providing both the config option and CLI argument is an error. If there is any existing content in the output directory, -freezeyt will either remove it (if the content looks like a previously +Freezeyt will either remove it (if the content looks like a previously frozen website) or raise an error. Best practice is to remove the output directory before freezing. @@ -209,7 +209,7 @@ will be represented as: #### Explicitly output to disk -It is possible to explicitly request *freezeyt* to output to a directory +It is possible to explicitly request Freezeyt to output to a directory using a dict like this: ```yaml @@ -242,7 +242,7 @@ The *prefix* URL must end with a slash. The page at the *prefix* URL is considered the application's “home page”, and will always be frozen. -*freezeyt* considers all pages under the prefix to be part of +Freezeyt considers all pages under the prefix to be part of the application. For example, with the second prefix above: @@ -261,7 +261,7 @@ The CLI argument has priority over the config file. ## Extra content -Usually, *freezeyt* saves pages that are reachable by links from the app's +Usually, Freezeyt saves pages that are reachable by links from the app's home page. There are two cases when this is not enough, and you need to specify extra content manually: @@ -369,7 +369,7 @@ extra_files: If the `copy_from` path names a directory, it will be copied recursively. -In the file name, *freezeyt* treats both backslashes and forward slashes +In the file name, Freezeyt treats both backslashes and forward slashes as path separators. Extra files cannot be specified on the CLI. @@ -378,12 +378,12 @@ Extra files cannot be specified on the CLI. ## Debugging options The following options are useful when debugging your application, -or its integration with *freezeyt*. +or its integration with Freezeyt. ### Clean up {: #conf-cleanup } -By default, if an error occurs during freezing, *freezeyt* will delete +By default, if an error occurs during freezing, Freezeyt will delete the incomplete output directory. This is meant to prevent uploading incomplete results to web hosting by mistake. @@ -405,7 +405,7 @@ Use `--cleanup` to override `cleanup: False` from the config. ### Fail fast {: #conf-fail_fast } -By default, *freezeyt* collects errors it finds on individual pages, +By default, Freezeyt collects errors it finds on individual pages, and presents them all when done. To stop the process early when the first error occurs, use the the `--fail-fast` (`-x`) command line switch or the `fail_fast` configuration option: @@ -424,7 +424,7 @@ Use `--no-fail-fast` to override `fail_fast: True` from the config. ## Customizing the process -Here are ways to configure details of how *freezeyt* saves pages. +Here are ways to configure details of how Freezeyt saves pages. ### MIME type checking @@ -439,7 +439,7 @@ the `Content-Type` -- for example, `index.html` is served as a HTML document (`text/html`) and `smile.png` is served as a PNG image (`image/png`). To ensure that the application will work as intended when frozen and served -with such a server, *freezeyt* verifies that the extensions of saved files +with such a server, Freezeyt verifies that the extensions of saved files correspond to the MIME types served by the app. This funtionality is provided by [`freezeyt.Middleware`](/#middleware). @@ -474,16 +474,16 @@ get_mimetype=module:your_function `get_mimetype` can be defined as a string in the form `"module:function"`, which names the function to call, or as a Python function -(if configuring `freezeyt` using a Python dict). +(if configuring Freezeyt using a Python dict). The function will be called with one argument, the file path as a string, and it should returns a list of corresponding MIME types (for example, `["text/html"]` or `["audio/wav", "audio/wave"]`). -If `get_mimetype` instead returns `None`, `freezeyt` will use the +If `get_mimetype` instead returns `None`, Freezeyt will use the [default MIME type](#conf-default_mimetype). -By default, `freezeyt` calls the Python function +By default, Freezeyt calls the Python function [`mimetypes.guess_type`](https://docs.python.org/3/library/mimetypes.html#mimetypes.guess_type) and uses the `type` (the first element) of the result: @@ -505,7 +505,7 @@ There is an option to use [the MIME type database from the `jshttp` project](htt (the database used by GitHub Pages), or a database with the same structure, for mapping file names to MIME types. -To use it, add the path to the JSON file to `freezeyt` configuration: +To use it, add the path to the JSON file to Freezeyt configuration: ```yaml mime_db_file=path/to/mime-db.json ``` @@ -515,13 +515,13 @@ extensions to filetypes according to the database. ### URL finding {: #conf-url_finders } -`freezeyt` discovers new pages in the application by searching for URLs +Freezeyt discovers new pages in the application by searching for URLs in pages it processes. The search is done by *URL finders*, functions that find URLs in a specific page type. You can configure which finder is used for which MIME type using the `url_finders` configuration key. -In the default configuration, freezeyt finds links in HTML and CSS files. +In the default configuration, Freezeyt finds links in HTML and CSS files. The default could be configured like this: ```yaml @@ -535,7 +535,7 @@ Values are functions, which can be defined as: * Strings in the form `"module:function"`, which name the finder function to call. -* Python functions (if configuring `freezeyt` from Python). +* Python functions (if configuring Freezeyt from Python). * Strings without a colon (`:`), which name a function from the `freezeyt.url_finders` module. Using this shortcut, the default configuration could also be written as: @@ -570,9 +570,9 @@ Specifically: Finder functions may be asynchronous. If the function returns a coroutine (for example, if it's defined with -`async def`, freezeyt will use `await` on the result. +`async def`, Freezeyt will use `await` on the result. If the function returns an asynchronous generator (for example, if it's -defined with `async def` and uses `yield`), freezeyt will use async iteration +defined with `async def` and uses `yield`), Freezeyt will use async iteration to handle it. @@ -597,8 +597,8 @@ This may be changed in the future. #### Disabling default URL finders {: #conf-use_default_url_finders } If a finder is not explictly specified in the configuration file, -`freezeyt` will use the default. For example, if you specify -`text/html: my_custom_finder` only, `freezeyt` will use the default finder +Freezeyt will use the default. For example, if you specify +`text/html: my_custom_finder` only, Freezeyt will use the default finder for `text/css`. You can disable this behavior: @@ -610,7 +610,7 @@ use_default_url_finders: false #### Finding URLs in Link headers {: #conf-urls_from_link_headers } -By default, `freezeyt` will follow URLs in `Link` HTTP headers. +By default, Freezeyt will follow URLs in `Link` HTTP headers. To disable this, specify: ```yaml @@ -620,20 +620,20 @@ urls_from_link_headers: false ### Freeze actions -For each page it finds, `freezeyt` will take an *action*: save the page, +For each page it finds, Freezeyt will take an *action*: save the page, ignore it, or treat it as an error. -By default, `freezeyt` will save the pages with a `200 OK` HTTP status code, +By default, Freezeyt will save the pages with a `200 OK` HTTP status code, and raise an error for any other status code. You can configure the for an individual page from within the application (or middleware), by setting the `Freezeyt-Action` HTTP header to one of these strings: -* `'save'`: `freezeyt` will save the body of the page. -* `'ignore'`: `freezeyt` will not save any content for the page +* `'save'`: Freezeyt will save the body of the page. +* `'ignore'`: Freezeyt will not save any content for the page * `'warn'`: will save the content and send warn message to stdout -* `'follow'`: `freezeyt` will save content from the redirected location. +* `'follow'`: Freezeyt will save content from the redirected location. This requires a `Location` header, which is usually added for redirects. Redirects to external pages are not supported. * `'error'`: fail; the page will not be saved and `freeze()` will raise @@ -642,7 +642,7 @@ these strings: #### HTTP Status handling {: #conf-status_handlers } -If the `Freezeyt-Action` header is not set, `freezeyt` will determine what to +If the `Freezeyt-Action` header is not set, Freezeyt will determine what to do based on the HTTP status. The behavior can be customized using the `status_handlers` setting. For example, to ignore pages with the `404 NOT FOUND` status, rather than @@ -679,7 +679,7 @@ You can also define a custom action in `status_handlers` as: * a string in the form `'my_module:custom_action'`, which names a handler function to call, or -* a Python function (if configuring `freezeyt` from Python). +* a Python function (if configuring Freezeyt from Python). The action function takes one argument a [`TaskInfo`](pyapi.md#TaskInfo) with information about the page being frozen. @@ -702,7 +702,7 @@ The value can be: * A string in the form `"module:function"`, which names the function to call. The function can be omitted along with the colon, and defaults to `url_to_path`. -* A Python function, if configuring `freezeyt` from Python. +* A Python function, if configuring Freezeyt from Python. The function receives one string: the *path* portion of the URL to save, relative to the `prefix`. @@ -715,8 +715,8 @@ if the URL ends with `/`. ### Plugins {: #conf-plugins } -It is possible to extend `freezeyt` with *plugins*, -either [“built-in” ones](#built-in-plugins) that ship with `freezeyt` +It is possible to extend Freezeyt with *plugins*, +either [“built-in” ones](#built-in-plugins) that ship with Freezeyt or [external ones](#custom-plugins). Plugins are added using configuration like: @@ -764,7 +764,7 @@ because of the `--force` switch. ### Progress bar and logging {: #conf-cli-progress } -The CLI argument `--progress` controls what `freezeyt` outputs as it +The CLI argument `--progress` controls what Freezeyt outputs as it handles pages: * `--progress=log`: Output a message about each frozen page to stdout. @@ -783,7 +783,7 @@ Alternately, it is possible to configure logging by adding one of the following ## Middleware static mode {: #static_mode } -When using the [`freezeyt` middleware](./index.md#middleware), you can enable *static mode*, +When using the [Freezeyt middleware](./index.md#middleware), you can enable *static mode*, which simulates behaviour after the app is saved to static pages: ```yaml @@ -804,12 +804,12 @@ having to freeze all of it after each change. -## Extending *freezeyt* +## Extending Freezeyt ### Custom plugins {: #custom-plugins } -A plugin is a function that `freezeyt` will call before starting to +A plugin is a function that Freezeyt will call before starting to freeze pages. It is passed a [`FreezeInfo`](pyapi.md#FreezeInfo) object as argument. @@ -823,7 +823,7 @@ It is possible to register *hooks*, functions that are called when specific events happen in the freezing process. For example, if `mymodule` defines functions `start` and `page_frozen`, -you can make freezeyt call them using this configuration: +you can make Freezeyt call them using this configuration: ```yaml hooks: diff --git a/docs/contrib.md b/docs/contrib.md index c2e04217..e0e15425 100644 --- a/docs/contrib.md +++ b/docs/contrib.md @@ -66,7 +66,7 @@ $ python -m pip install -e ."[blog, dev, typecheck]" [mypy]: https://www.mypy-lang.org/ -## Using an in-development copy of freezeyt +## Using an in-development copy of Freezeyt * Set `PYTHONPATH` to the directory with Freezeyt, for example: * Unix: `export PYTHONPATH="/home/name/freezeyt"` @@ -76,7 +76,7 @@ $ python -m pip install -e ."[blog, dev, typecheck]" * install the application using `pip`, if possible, or * install the application's dependencies and `cd` to the app's directory. -* Run freezeyt, for example: +* Run Freezeyt, for example: * `python -m freezeyt demo_app_url_for _build --prefix http://freezeyt.test/foo/` @@ -104,7 +104,7 @@ $ tox ### Environ variables for tests -Some test scenarios compare freezeyt's results with expected output. +Some test scenarios compare Freezeyt's results with expected output. When the files with expected output don't exist yet, they can be created by setting the environment variable `TEST_CREATE_EXPECTED_OUTPUT` to `1`: diff --git a/docs/index.md b/docs/index.md index 685930ff..fc12e6d2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -40,7 +40,7 @@ The tool can be installed using: $ python -m pip install freezeyt ``` -To install a development version of freezeyt, +To install a development version of Freezeyt, see [Contributing documentation]. [Contributing documentation]: ./contrib.md @@ -59,7 +59,7 @@ For detailed instructions, read on. ## Usage -To use freezeyt, you need a Python web application. +To use Freezeyt, you need a Python web application. You can use the [example Flask app] to start. [example Flask app]: https://flask.palletsprojects.com/en/2.3.x/quickstart/ @@ -69,12 +69,12 @@ ideally one named `app` which is the default in [Flask] and [Falcon]. For other frameworks, search the documentation on how to export a WSGI application. -Both the application and freezeyt need to be importable (installed) in your +Both the application and Freezeyt need to be importable (installed) in your envuronment. -Run freezeyt with two arguments: the Python module with your `app`, +Run Freezeyt with two arguments: the Python module with your `app`, and an output directory. -Note that freezeyt wants a *module* name (as used in an `import` statement). +Note that Freezeyt wants a *module* name (as used in an `import` statement). Don't use a *file* name with a `.py` suffix. For example, if your `app` is defined in the file `my_app.py`, run: From 9a355595b9107a379ae201748a0f30891ba9d3f0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 10 Sep 2024 18:58:45 +0200 Subject: [PATCH 13/16] Work on the API docs --- docs/config.md | 30 +++++++-------- docs/pyapi.md | 84 ++++++++---------------------------------- freezeyt/__init__.py | 3 ++ freezeyt/filesaver.py | 2 +- freezeyt/freezer.py | 19 ++++++++-- freezeyt/hooks.py | 44 +++++++++++++++++++++- freezeyt/middleware.py | 18 +++++++++ freezeyt/util.py | 16 +++++--- mkdocs.yml | 17 +++++++++ setup.cfg | 1 + 10 files changed, 138 insertions(+), 96 deletions(-) diff --git a/docs/config.md b/docs/config.md index fa06e1c7..d977c4b6 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,4 +1,4 @@ -# Configuration +# Configuration {: #configuration } Freezeyt is primarily configured using a dictionary of options, usually loaded from a YAML (or JSON) file given on the command line (CLI) @@ -33,8 +33,8 @@ Here is a full list of CLI arguments: |----------|------------|----------| | APP (positional) | `app` | [Application to freeze](#conf-app) | | `-o`, `--output`, positional | `output` | [Output directory](#conf-output) | -| `-c`, `--config` | --- | [Configuration file](#conf-cli-config) | -| `-C`, `--import-config` | --- | [Configuration variable](#conf-cli-import-config) | +| `-c`, `--config` | --- | [Configuration file](#configuration) | +| `-C`, `--import-config` | --- | [Configuration variable](#configuration) | | `--prefix` | `prefix` | [URL prefix](#conf-prefix) | | `--extra-page` | `extra_pages` | [Extra pages](#conf-extra_pages) | | `--progress` | (plugins) | [Progress bar and logging](#conf-cli-progress) | @@ -326,7 +326,7 @@ another_config = { } ``` -### Extra files {: #config-extra_files } +### Extra files {: #conf-extra_files } Extra files to be included in the output can be specified, along with their content. @@ -442,7 +442,7 @@ To ensure that the application will work as intended when frozen and served with such a server, Freezeyt verifies that the extensions of saved files correspond to the MIME types served by the app. -This funtionality is provided by [`freezeyt.Middleware`](/#middleware). +This funtionality is provided by [`freezeyt.Middleware`][Middleware]. The exact mapping between extensions and `Content-Type` values varies between servers. @@ -681,14 +681,14 @@ You can also define a custom action in `status_handlers` as: function to call, or * a Python function (if configuring Freezeyt from Python). -The action function takes one argument a [`TaskInfo`](pyapi.md#TaskInfo) +The action function takes one argument a [`TaskInfo`][TaskInfo] with information about the page being frozen. Freezeyt's predefined actions, like `follow`, can be imported from `freezeyt.actions`. A custom action should call one of these default actions and return the return value from it. -### Path generation +### Path generation {: #conf-url_to_path } It is possible to customize the filenames that pages are saved under using the `url_to_path` configuration key, for example: @@ -781,9 +781,9 @@ Alternately, it is possible to configure logging by adding one of the following * `freezeyt.progressbar:LogPlugin` -## Middleware static mode {: #static_mode } +## Middleware static mode {: #conf-static_mode } -When using the [Freezeyt middleware](./index.md#middleware), you can enable *static mode*, +When using the [Freezeyt middleware][middleware], you can enable *static mode*, which simulates behaviour after the app is saved to static pages: ```yaml @@ -812,8 +812,8 @@ having to freeze all of it after each change. A plugin is a function that Freezeyt will call before starting to freeze pages. -It is passed a [`FreezeInfo`](pyapi.md#FreezeInfo) object as argument. -Usually, the plugin will its [`add_hook`](pyapi.md#FreezeInfo.add_hook) method +It is passed a [`FreezeInfo`][FreezeInfo] object as argument. +Usually, the plugin will its [`add_hook`][FreezeInfo-add_hook] method to register additional functions. @@ -842,25 +842,25 @@ The available hooks are: Called when the freezing process starts, before any other hooks. -Takes one argument: a [`FreezeInfo`](pyapi.md#FreezeInfo) object. +Takes one argument: a [`FreezeInfo`][FreezeInfo] object. #### `page_frozen` Called whenever a page is processed successfully. -Takes one argument: a [`TaskInfo`](pyapi.md#TaskInfo) object. +Takes one argument: a [`TaskInfo`][TaskInfo] object. #### `page_failed` Called whenever a page is not saved due to an exception. -Takes one argument: a [`TaskInfo`](pyapi.md#TaskInfo) object. +Takes one argument: a [`TaskInfo`][TaskInfo] object. #### `success` Called after the app is successfully frozen. -Takes one argument: a [`FreezeInfo`](pyapi.md#FreezeInfo) object. +Takes one argument: a [`FreezeInfo`][FreezeInfo] object. diff --git a/docs/pyapi.md b/docs/pyapi.md index e54b722a..8d1fb260 100644 --- a/docs/pyapi.md +++ b/docs/pyapi.md @@ -2,92 +2,38 @@ ## Functions -### `freezeyt.freeze(app, config)` -### `freezeyt.freeze_async(app, config)` +::: freezeyt.freeze + +::: freezeyt.freeze_async ## WSGI Middleware -### `freezeyt.Middleware` +::: freezeyt.Middleware ## Hook Arguments -### `FreezeInfo` - - -#### `FreezeInfo.add_url(url, reason=None)` - -Add the URL to the set of pages to be frozen. - -If that URL was frozen already, or is outside the `prefix`, does nothing. - -If you add a `reason` string, it will be used in error messages as the reason -why the added URL is being handled. - -#### `FreezeInfo.add_hook(hook_name, callable)` - -Register an additional hook function. - -#### `FreezeInfo.total_task_count` - -The number of pages `freezeyt` currently “knows about” – -ones that are already frozen plus ones that are scheduled to be frozen. - -#### `FreezeInfo.done_task_count` - -The number of pages that are done (either successfully -frozen, or failed). - -#### `FreezeInfo.failed_task_count` - -The number of pages that failed to freeze. +Objects of these classes are passed to custom [hooks][conf-hooks] and +[plugins][conf-plugins]. -### `TaskInfo` {: #TaskInfo } +::: freezeyt.FreezeInfo +::: freezeyt.TaskInfo -### `TaskInfo.get_a_url()`: -Returns a URL of the page, including `prefix`. - -Note that a page may be reachable via several URLs; this function returns -an arbitrary one. - -### `TaskInfo.path` - -The relative path the content is saved to. - -### `TaskInfo.freeze_info` - -A [`FreezeInfo`]{: #FreezeInfo } object corresponding to the entire -freeze process. - -### `TaskInfo.exception` - -For failed tasks, the exception raised. `None` otherwise. - -### `TaskInfo.reasons` - -A list of strings explaining why the given page was visited. -(Note that as the freezing progresses, new reasons may be added to -existing tasks.) ## Exceptions -### `freezeyt.VersionMismatch` - -### `freezeyt.InfiniteRedirection` - -### `freezeyt.ExternalURLError` - -### `freezeyt.RelativeURLError` - -### `freezeyt.UnexpectedStatus` - -### `freezeyt.MultiError` +::: freezeyt.VersionMismatch +::: freezeyt.InfiniteRedirection +::: freezeyt.ExternalURLError +::: freezeyt.RelativeURLError +::: freezeyt.UnexpectedStatus +::: freezeyt.MultiError +::: freezeyt.DirectoryExistsError -### `freezeyt.DirectoryExistsError` ## Types diff --git a/freezeyt/__init__.py b/freezeyt/__init__.py index de38b686..190486c0 100644 --- a/freezeyt/__init__.py +++ b/freezeyt/__init__.py @@ -4,6 +4,7 @@ from freezeyt.freezer import default_url_to_path as url_to_path from freezeyt.middleware import Middleware from freezeyt.types import Config +from freezeyt.hooks import FreezeInfo, TaskInfo __version__ = '1.1.1' @@ -21,4 +22,6 @@ 'MultiError', 'VersionMismatch', 'Config', + "FreezeInfo", + "TaskInfo", ] diff --git a/freezeyt/filesaver.py b/freezeyt/filesaver.py index a2aa9857..46504cba 100644 --- a/freezeyt/filesaver.py +++ b/freezeyt/filesaver.py @@ -10,7 +10,7 @@ class DirectoryExistsError(Exception): - """Attempt to overwrite directory that doesn't contain freezeyt output""" + """Attempt to overwrite directory that doesn't contain freezeyt output.""" class FileSaver(Saver): diff --git a/freezeyt/freezer.py b/freezeyt/freezer.py index 1d8647cd..10ea8abc 100644 --- a/freezeyt/freezer.py +++ b/freezeyt/freezer.py @@ -44,6 +44,15 @@ def freeze(app: Optional[WSGIApplication], config: Config) -> SaverResult: + """Freeze the given *app*. + + + Args: + app: The application to freeze. If `None`, the application + is taken from *config*. + config: The configuration dict. See [Configuration][configuration] + for what goes in. + """ return asyncio_run(freeze_async(app, config)) @@ -51,6 +60,10 @@ async def freeze_async( app: Optional[WSGIApplication], config: Config, ) -> SaverResult: + """Asynchronous version of [freeze][freezeyt.freeze]. + + If an asyncio event loop is active, call (and `await`) this function + rather than [freeze][freezeyt.freeze].""" freezer = Freezer(app, config) try: await freezer.prepare() @@ -175,13 +188,13 @@ def update_status(self, old_status, new_status): new_collection[self.path] = self class IsARedirect(BaseException): - """Raised when a page redirects and freezing it should be postponed""" + """Raised when a page redirects and freezing it should be postponed.""" class IgnorePage(BaseException): - """Raised when freezing a page should be ignored""" + """Raised when freezing a page should be ignored.""" class VersionMismatch(ValueError): - """Raised when major version in config is not correct""" + """Raised when major version in config is not correct.""" def needs_semaphore(func): """Decorator for a "task" method that holds self.semaphore when running""" diff --git a/freezeyt/hooks.py b/freezeyt/hooks.py index e4aa9143..9a8d6373 100644 --- a/freezeyt/hooks.py +++ b/freezeyt/hooks.py @@ -14,7 +14,11 @@ def __init__(self, task: 'Task'): self._freezer = task.freezer def get_a_url(self) -> str: - """Return a URL of this page""" + """Return a URL of this page + + Note that a page may be reachable via several URLs; this function + returns an arbitrary one. + """ return urllib.parse.urlunsplit(self._task.get_a_url()) @property @@ -24,10 +28,16 @@ def path(self) -> str: @property def freeze_info(self) -> 'FreezeInfo': + """ + A [`FreezeInfo`][freezeyt.FreezeInfo] object corresponding to the + entire freeze process. + """ + return self._freezer.freeze_info @property def exception(self) -> Optional[BaseException]: + """For failed tasks, the exception raised.""" aio_task = self._task.asyncio_task if aio_task is None: return None @@ -40,7 +50,8 @@ def exception(self) -> Optional[BaseException]: def reasons(self) -> Iterable[str]: """A list of strings explaining why the given page was visited. - New entries may be added as the freezing goes on. + Note that as the freezing progresses, new reasons may be added to + existing tasks. """ return sorted(self._task.reasons) @@ -52,9 +63,27 @@ def __init__(self, freezer: 'Freezer'): self._freezer = freezer def add_url(self, url: str, reason: Optional[str] = None) -> None: + """Add the URL to the set of pages to be frozen. + + Args: + url: The URL to add. + + If that URL was frozen already or is external + (that is, outside the [prefix][conf-prefix]), + `add_url` does nothing. + + reason: A note that will be used in error messages as + the reason why the added URL is being handled. + """ self._freezer.add_task(parse_absolute_url(url), reason=reason) def add_hook(self, hook_name: str, func: Callable) -> None: + """Register an additional hook function. + + Args: + hook_name: Hook name. See [hook docs][conf-hooks] for a list. + func: Function to call. + """ self._freezer.add_hook(hook_name, func) @property @@ -63,12 +92,22 @@ def fail_fast(self) -> bool: @property def total_task_count(self) -> int: + """Number of pages Freezeyt currently “knows about”. + + This includes pages that are already done plus ones that are + scheduled to be frozen. + """ return sum( len(tasks) for tasks in self._freezer.task_collections.values() ) @property def done_task_count(self) -> int: + """The number of pages that are done. + + This includes both pages that were successfully frozen and failed ones. + """ + # Import TaskStatus here to avoid a circular import # (since freezer imports hooks) from freezeyt.freezer import TaskStatus @@ -79,6 +118,7 @@ def done_task_count(self) -> int: @property def failed_task_count(self) -> int: + """The number of pages that failed to freeze.""" # Import TaskStatus here, see done_task_count from freezeyt.freezer import TaskStatus return len(self._freezer.task_collections[TaskStatus.FAILED]) diff --git a/freezeyt/middleware.py b/freezeyt/middleware.py index 04fa8a6c..068635a5 100644 --- a/freezeyt/middleware.py +++ b/freezeyt/middleware.py @@ -14,6 +14,24 @@ class Middleware: + """WSGI middleware. + + By default, the middleware: + + - serves [extra pages][conf-extra_pages] and [extra files][conf-extra_files] + - checks that [mime types correspond to extensions][mime-type-checking] + + If [static mode][conf-static_mode] is enabled, the middleware gives a + preview of how the application would work when frozen. Specifically: + + - HTTP requests other than `GET` (and `OPTIONS`) are blocked + - Non-essential HTTP headers are removed + - Query strings and request bodies are removed + + Args: + app: The application to wrap + config: The configuration dict, as for [freeze][freezeyt.freeze] + """ def __init__(self, app: WSGIApplication, config: Config): self.app = app self.mimetype_checker = MimetypeChecker(config) diff --git a/freezeyt/util.py b/freezeyt/util.py index d822b840..d49975b7 100644 --- a/freezeyt/util.py +++ b/freezeyt/util.py @@ -20,7 +20,7 @@ class InfiniteRedirection(Exception): - """Infinite redirection was detected with redirect_policy='follow'""" + """Infinite redirection was detected in the `'follow'` [action][freeze-actions].""" def __init__(self, task: 'Task'): redirects_to = task.redirects_to assert redirects_to is not None @@ -30,16 +30,16 @@ def __init__(self, task: 'Task'): ) class ExternalURLError(ValueError): - """Unexpected external URL specified""" + """Unexpected external URL specified.""" class RelativeURLError(ValueError): - """Absolute URL was expected""" + """Unexpected relative URL was expected.""" class UnsupportedSchemeError(ValueError): - """Raised for URLs with unsupported schemes""" + """Raised for URLs with unsupported schemes.""" class UnexpectedStatus(ValueError): - """The application returned an unexpected status code for a page""" + """The application returned an unexpected status code for a page.""" def __init__(self, url: AbsoluteURL, status: str): self.url = urllib.parse.urlunsplit(url) self.status = status @@ -55,7 +55,11 @@ def __init__(self, expected: List[str], got: str, url_path: str): ) class MultiError(_MultiErrorBase): - """Contains multiple errors""" + """Contains multiple errors. + + On Python 3.11 and above, this is a subclass of the built-in + `ExceptionGroup`. + """ tasks: 'Sequence[TaskInfo]' if not HAVE_EXCEPTION_GROUP: diff --git a/mkdocs.yml b/mkdocs.yml index 84d38fb1..991183e9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,6 +16,23 @@ theme: user_color_mode_toggle: true navigation_depth: 3 # highlightjs: false # disables user_color_mode_toggle?! +watch: + - "freezeyt/" +plugins: + - mkdocstrings: + handlers: + python: + options: + paths: ["./"] # actually not needed, default + show_source: false + show_root_heading: true + docstring_section_style: "list" + modernize_annotations: true + heading_level: 3 + members_order: source + group_by_category: false + show_bases: false + - autorefs markdown_extensions: - toc: permalink: true diff --git a/setup.cfg b/setup.cfg index d4729e71..266d817e 100644 --- a/setup.cfg +++ b/setup.cfg @@ -44,6 +44,7 @@ docs = mkdocs pygments markdown-include + mkdocstrings[python] blog = flask markdown-it-py From 7bbd83199071563ec044cecc386ddc8cae83e792 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 1 Oct 2024 17:55:15 +0200 Subject: [PATCH 14/16] Document the rest of the API --- docs/config.md | 25 ++++++++++++------------ docs/pyapi.md | 43 +++++++++++++++++++++++++++-------------- freezeyt/actions.py | 9 +++++++++ freezeyt/freezer.py | 7 +++++++ freezeyt/plugins.py | 7 +++++++ freezeyt/url_finders.py | 9 +++++++++ 6 files changed, 73 insertions(+), 27 deletions(-) diff --git a/docs/config.md b/docs/config.md index d977c4b6..b1e2df51 100644 --- a/docs/config.md +++ b/docs/config.md @@ -442,7 +442,7 @@ To ensure that the application will work as intended when frozen and served with such a server, Freezeyt verifies that the extensions of saved files correspond to the MIME types served by the app. -This funtionality is provided by [`freezeyt.Middleware`][Middleware]. +This funtionality is provided by [`freezeyt.Middleware`][freezeyt.Middleware]. The exact mapping between extensions and `Content-Type` values varies between servers. @@ -618,7 +618,7 @@ urls_from_link_headers: false ``` -### Freeze actions +### Freeze actions {: #freeze-actions } For each page it finds, Freezeyt will take an *action*: save the page, ignore it, or treat it as an error. @@ -673,7 +673,8 @@ lowercase `xx`. (Other "wildcards" like `50x` are not supported.) -#### Custom actions +[](){#custom-actions} +#### Custom freeze actions You can also define a custom action in `status_handlers` as: @@ -681,7 +682,7 @@ You can also define a custom action in `status_handlers` as: function to call, or * a Python function (if configuring Freezeyt from Python). -The action function takes one argument a [`TaskInfo`][TaskInfo] +The action function takes one argument a [`TaskInfo`][freezeyt.TaskInfo] with information about the page being frozen. Freezeyt's predefined actions, like `follow`, can be imported from `freezeyt.actions`. @@ -709,8 +710,8 @@ relative to the `prefix`. It should return a path to the saved file, relative to the build directory, as a string. -The default function, available as `freezeyt.url_to_path`, adds `index.html` -if the URL ends with `/`. +The default function, available as [`freezeyt.url_to_path`][freezeyt.url_to_path], +adds `"index.html"` if the URL ends with `/`. ### Plugins {: #conf-plugins } @@ -812,8 +813,8 @@ having to freeze all of it after each change. A plugin is a function that Freezeyt will call before starting to freeze pages. -It is passed a [`FreezeInfo`][FreezeInfo] object as argument. -Usually, the plugin will its [`add_hook`][FreezeInfo-add_hook] method +It is passed a [`FreezeInfo`][freezeyt.FreezeInfo] object as argument. +Usually, the plugin will its [`add_hook`][freezeyt.FreezeInfo.add_hook] method to register additional functions. @@ -842,25 +843,25 @@ The available hooks are: Called when the freezing process starts, before any other hooks. -Takes one argument: a [`FreezeInfo`][FreezeInfo] object. +Takes one argument: a [`FreezeInfo`][freezeyt.FreezeInfo] object. #### `page_frozen` Called whenever a page is processed successfully. -Takes one argument: a [`TaskInfo`][TaskInfo] object. +Takes one argument: a [`TaskInfo`][freezeyt.TaskInfo] object. #### `page_failed` Called whenever a page is not saved due to an exception. -Takes one argument: a [`TaskInfo`][TaskInfo] object. +Takes one argument: a [`TaskInfo`][freezeyt.TaskInfo] object. #### `success` Called after the app is successfully frozen. -Takes one argument: a [`FreezeInfo`][FreezeInfo] object. +Takes one argument: a [`FreezeInfo`][freezeyt.FreezeInfo] object. diff --git a/docs/pyapi.md b/docs/pyapi.md index 8d1fb260..8491144b 100644 --- a/docs/pyapi.md +++ b/docs/pyapi.md @@ -37,32 +37,45 @@ Objects of these classes are passed to custom [hooks][conf-hooks] and ## Types -### `freezeyt.Config` +::: freezeyt.Config + +A dictionary that holds [configuration][configuration] for Freezeyt. ## Actions -### `freezeyt.actions.warn` -### `freezeyt.actions.ignore` -### `freezeyt.actions.follow` -### `freezeyt.actions.save` -### `freezeyt.actions.error` +Built-in [actions][freeze-actions] are available as functions in +the ``freezeyt.actions`` module. +[Custom actions][custom-actions] should call one of these +functions and return its result. + +::: freezeyt.actions.warn +::: freezeyt.actions.ignore +::: freezeyt.actions.follow +::: freezeyt.actions.save +::: freezeyt.actions.error ## URL finders -### `freezeyt.url_finders.get_html_links` -### `freezeyt.url_finders.get_html_links_async` +Built-in [URL finders][conf-url_finders] are available as +functions in the ``freezeyt.url_finders`` module: -### `freezeyt.url_finders.get_css_links` -### `freezeyt.url_finders.get_css_links_async` +::: freezeyt.url_finders.get_html_links +::: freezeyt.url_finders.get_html_links_async -### `freezeyt.url_finders.none` +::: freezeyt.url_finders.get_css_links +::: freezeyt.url_finders.get_css_links_async + +::: freezeyt.url_finders.none ## Plugins -### `freezeyt.plugins.ProgressBarPlugin` -### `freezeyt.plugins.LogPlugin` -### `freezeyt.plugins.GHPagesPlugin` +[Built-in plugins][built-in-plugins] are available in the +``freezeyt.plugins`` module: + +::: freezeyt.plugins.ProgressBarPlugin +::: freezeyt.plugins.LogPlugin +::: freezeyt.plugins.GHPagesPlugin ## Utilities -### `freezeyt.url_to_path` +::: freezeyt.url_to_path diff --git a/freezeyt/actions.py b/freezeyt/actions.py index a3168560..aa737ae1 100644 --- a/freezeyt/actions.py +++ b/freezeyt/actions.py @@ -7,6 +7,7 @@ def warn(task: TaskInfo) -> str: + """Save the content, but send warn message to stdout.""" url = task.get_a_url() response_status = task._task.response_status if response_status is None: @@ -21,6 +22,11 @@ def warn(task: TaskInfo) -> str: def follow(task: TaskInfo) -> str: + """Save content from the redirected location. + + This requires a Location header, which is usually added for redirects. + Redirects to external pages are not supported. + """ url = task._task.get_a_url() response_headers = task._task.response_headers if response_headers is None: @@ -41,14 +47,17 @@ def follow(task: TaskInfo) -> str: def ignore(task: TaskInfo) -> str: + """Do not save any content for the page.""" return 'ignore' def save(task: TaskInfo) -> str: + """Save the body of the page.""" return 'save' def error(task: TaskInfo) -> str: + """Raise an exception.""" return 'error' diff --git a/freezeyt/freezer.py b/freezeyt/freezer.py index 10ea8abc..8b703bdf 100644 --- a/freezeyt/freezer.py +++ b/freezeyt/freezer.py @@ -115,6 +115,13 @@ def parse_handlers( def default_url_to_path(path: str) -> str: + """Return the filesystem path corresponding to the given URL path. + + Note that the input should only contain the path part of an URL; not, + for example, a hostname. + + This function adds `index.html` to paths ending with a slash. + """ if path.endswith('/') or not path: path = path + 'index.html' return encode_file_path(path) diff --git a/freezeyt/plugins.py b/freezeyt/plugins.py index 525653c8..34041b2b 100644 --- a/freezeyt/plugins.py +++ b/freezeyt/plugins.py @@ -14,6 +14,7 @@ class GitCommandError(ValueError): """An exception occurred while executing git commands.""" class ProgressBarPlugin: + """Plugin to fill a CLI progress par as a site is being frozen.""" bar_format = '{percentage:3.0f}%▕{bar}▏{elapsed}, {rate:.2f} pg/s' def __init__(self, freeze_info: FreezeInfo): self.manager = enlighten.get_manager() @@ -34,6 +35,7 @@ def update_bar(self, task_info: TaskInfo) -> None: self.counter.update(0) class LogPlugin: + """Plugin to log progress messages to stderr.""" def __init__(self, freeze_info: FreezeInfo): freeze_info.add_hook('page_frozen', self.page_frozen) freeze_info.add_hook('page_failed', self.page_failed) @@ -64,6 +66,11 @@ def page_failed(self, task_info: TaskInfo) -> None: traceback.print_exception(type(exc), exc, exc.__traceback__) class GHPagesPlugin: + """Plugin for GitHub Pages integration. + + Saves the output to a Git repository, and adds extra files necessary + for GitHub Pages (`CNAME` and `.nojekyll`). + """ def __init__(self, freeze_info: FreezeInfo): if freeze_info._freezer.prefix.path != "/": raise ValueError("When using the Github Pages plugin, you can't specify a path in the prefix, so github can't handle it.") diff --git a/freezeyt/url_finders.py b/freezeyt/url_finders.py index 67930a2d..4e6ced97 100644 --- a/freezeyt/url_finders.py +++ b/freezeyt/url_finders.py @@ -63,6 +63,7 @@ def get_links_from_node( def get_html_links( html_file: BinaryIO, base_url: str, headers: _Headers=None, ) -> Iterable[str]: + """Yield URLs linked from a HTML page.""" content = html_file.read() return _get_html_links(content, base_url, headers) @@ -70,6 +71,7 @@ def get_html_links( def get_css_links( css_file: BinaryIO, base_url: str, headers: _Headers=None, ) -> Iterable[str]: + """Yield URLs linked from a CSS file.""" content = css_file.read() return _get_css_links(content, base_url, headers) @@ -77,6 +79,7 @@ def get_css_links( async def get_css_links_async( css_file: BinaryIO, base_url: str, headers: _Headers=None, ) -> Iterable[str]: + """Yield URLs linked from a HTML page, asynchronously.""" loop = compat.get_running_loop() content = css_file.read() return await loop.run_in_executor( @@ -87,6 +90,7 @@ async def get_css_links_async( async def get_html_links_async( html_file: BinaryIO, base_url: str, headers: _Headers=None, ) -> Iterable[str]: + """Yield URLs linked from a CSS file, asynchronously.""" loop = compat.get_running_loop() content = html_file.read() return await loop.run_in_executor( @@ -96,6 +100,11 @@ async def get_html_links_async( def none( html_file: BinaryIO, base_url: str, headers: _Headers=None, ) -> Iterable[str]: + """Return an empty sequence. + + Useful for text-based configuration, where you can specify "none" to + disable finding links. + """ return [] if TYPE_CHECKING: From 6ddc1e9d89fefd347fbbd9ffd1737c23709cdb0b Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 1 Oct 2024 19:02:00 +0200 Subject: [PATCH 15/16] Start a Why --- docs/index.md | 2 +- docs/why.md | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 docs/why.md diff --git a/docs/index.md b/docs/index.md index fc12e6d2..a15572b6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -100,7 +100,7 @@ but do not rely on this.) [WSGI application]: https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/ -## More examples of CLI usage +### More examples of CLI usage You can tell Freezeyt where the application will be hosted, so it can generate correct URLs: diff --git a/docs/why.md b/docs/why.md new file mode 100644 index 00000000..735e1367 --- /dev/null +++ b/docs/why.md @@ -0,0 +1,104 @@ +# Why Freezeyt? + +Freezeyt helps you build *static websites*. + + +## Static sites + +According to [Wikipedia](https://en.wikipedia.org/wiki/Static_web_page): + +> A **static web page**, sometimes called a **flat page** or a +> **stationary page**, is a web page that is delivered to a web browser +> exactly as stored, in contrast to *dynamic web pages* which are generated +> by a web application. + +Compared to dynamic websites: +- Static sites can be hosted on any platform that allows Web hosting. +- There is no data or database that would need backups. +- Static sites are easily archived, for example by the Wayback machine. +- Static sites are usually more secure: they need only need a Web server, + not processes like Python or a database. + +However, static sites also have significant limitations. +In particular, users cannot make any changes to a static site. +Adding comments, publishing posts, even adding “likes” is not possible. + +With every change -- for example, adding a blog post -- a static site needs +to be *rebuilt* and re-*uploaded* to a Web server. + + +## Static first + +With Freezeyt, your website *is* generated by an application, +but then stored on disk exactly as it will be delivered to the web browser. + +This means that you can write your app in the traditional way, using a Python +framework like [Flask], [Falcon] or [Django], and then *freeze* it to produce +an easy-to-host, read-only version. + +This works great in the early phases of a project, when you're setting up +the structure of the app and don't *yet* have users that would want to make +changes. Or you want to focus on other things than enabling comments. + +It also works great at the late stages of a project: for example, +after a conference is over, you'll want to keep a read-only archive +of its site as long, and as cheaply, as possible. + +We call this approach *static-first*, and think of it as part of a family of +*graceful degradation* principles, where design starts with simpler, lower-tech +solutions, adding extras -- JavaScript, animations, images, CSS styles -- only +when the basics are in place, and making sure that the site works as well as +possible without these extras. + +In other words, a *static-first* website has a *read-only* mode, which is +cheap and easy to host but can only be updated by the admin. +It can *also* have dynamic features on top, but if those features are not +available, it degrades gracefully. + + +[Django]: https://www.djangoproject.com/ +[Flask]: https://flask.palletsprojects.com/en/3.0.x/ +[Falcon]: https://falconframework.org/ + + +## Considerations for static sites + +Not every website can be converted (“frozen”) to a set of files that can be +served statically. +There are two main concepts to keep in mind when designing a static site: + +### MIME type mapping + +When a Web server sends a Web page, it sends *headers* in addition to +the content. +These headers are lost when a site is converted to simple files. +(There are ways to preserve headers in a static site, but that requires custom +server configuration, making the static site less portable than it could be.) + +The most important lost header is [Content-Type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type), +which encodes the type of a page: `text/html` for a Web page, `image/png` +for a picture, `text/css` for a stylesheet. + +In a static site, this information is typically stored in *file extensions*: +a Web page is named `index.html`; an image is `smile.png`; a stylesheet +is `body.css`. +When the static server serves the file, it will generate a Content-Type header +based on the extension. + +The extension is part of the file name, and thus part of the URL of +the resource. +Authors of static websites must be careful to keep the extension in sync with +the Content-Type. +Otherwise the static server will not guess the type correctly, leading to +broken sites. + +Freezeyt is designed to help you keep Content-Type and file extensions in sync, +whether your app is currently not static-only or not. + + +### URL finding + + + + + From 8be2d3a1b8f0849e8c4d4811eb586ed189e6624f Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 15 Oct 2024 18:02:34 +0200 Subject: [PATCH 16/16] =?UTF-8?q?Add=20a=20=E2=80=9CWhy=E2=80=9D=20page?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/config.md | 14 ++++---------- docs/index.md | 4 ++-- docs/why.md | 25 +++++++++++++++++++++++-- mkdocs.yml | 3 ++- 4 files changed, 31 insertions(+), 15 deletions(-) diff --git a/docs/config.md b/docs/config.md index b1e2df51..09e12a7b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -268,7 +268,7 @@ extra content manually: - Extra *pages* are part of the application, but not reachable by following links. For example, a an old URL that redirects to a new location should - be configured as an extra page. + be configured as an extra page. See [Known URLs][known-urls] for details. - Extra *files* are not part of the application. Typically, these are used to configure the static page server, like @@ -429,19 +429,13 @@ Here are ways to configure details of how Freezeyt saves pages. ### MIME type checking -When web pages are saved to files on disk, some information is lost. -The most prominent piece of lost info is the -[`Content-Type`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type) -HTTP header, that is, the document's MIME type. - -Static page servers typically look at a file's extension to determine -the `Content-Type` -- for example, `index.html` is served as a HTML document -(`text/html`) and `smile.png` is served as a PNG image (`image/png`). +For static sites to be served correctly on most servers, the MIME +`Content-Type` of a page must match the saved file's extension. +See [MIME type mapping][mime-type-mapping] for details. To ensure that the application will work as intended when frozen and served with such a server, Freezeyt verifies that the extensions of saved files correspond to the MIME types served by the app. - This funtionality is provided by [`freezeyt.Middleware`][freezeyt.Middleware]. The exact mapping between extensions and `Content-Type` values varies diff --git a/docs/index.md b/docs/index.md index a15572b6..abe29104 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,8 @@ --- -title: freezeyt +title: Freezeyt ... -# freezeyt +# Freezeyt Freezeyt turns Python web applications into static websites. diff --git a/docs/why.md b/docs/why.md index 735e1367..eb508f9f 100644 --- a/docs/why.md +++ b/docs/why.md @@ -66,8 +66,9 @@ available, it degrades gracefully. Not every website can be converted (“frozen”) to a set of files that can be served statically. There are two main concepts to keep in mind when designing a static site: +MIME type mapping, limited number of pages, and known URLs. -### MIME type mapping +### MIME type mapping {: #mime-type-mapping } When a Web server sends a Web page, it sends *headers* in addition to the content. @@ -96,9 +97,29 @@ Freezeyt is designed to help you keep Content-Type and file extensions in sync, whether your app is currently not static-only or not. -### URL finding +### Limited number of pages {: #limited-num-pages } +For example, a calendar app might have a page for *any* month of any year, past +or future. That many pages wouldn't fit on any reasonable disk. +The static version of such a page would need to be limited, perhaps to one +decade or century. +This needs to be done in the application itself. +To hide a page from Freezeyt only, the app can set the +[`Freezeyt-Action`][freeze-actions] HTTP header to `ignore`. +### Known URLs {: #known-urls } + +Most pages of a typical websites are reachable via hyperlinks from the home +page. +Freezeyt will follow such links (in HTML and CSS documents) to find pages +it needs to save. +However, some types of pages aren't linked this way. For example: + +- Pages redirecting from old, obsolete URLs to new locations. +- Data or script files loaded with JavaScript. + +You will need to tell Freezeyt about such pages using the +[`extra_pages`][conf-extra_pages] mechanism. diff --git a/mkdocs.yml b/mkdocs.yml index 991183e9..c2af471f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,9 +5,10 @@ edit_uri: edit/main/docs/ # site_description: nav: - 'Home': 'index.md' + - 'Why?': 'why.md' - 'Configuration': 'config.md' - - 'Contributing': 'contrib.md' - 'API': 'pyapi.md' + - 'Contributing': 'contrib.md' not_in_nav: licence.md theme: