Update web crawl conditions - #501

Merged

ppinchuk merged 37 commits into

mainfrom

pp/fix_collection

Jul 21, 2026

Commits on Jul 16, 2026

Deprecate input
ppinchuk
committed
dfaeee8
Update docstring
ppinchuk
committed
4f7e124
Allow website collection if user provides model config
ppinchuk
committed
dcd97a1
Update docstring
ppinchuk
committed
687ae63
Deprecate input
ppinchuk
committed
ac27f80
Always validate a website from the search engines
ppinchuk
committed
58f00c5
Fix test
ppinchuk
committed
3e55b01
website input form user now assumed to be correct
ppinchuk
committed
b2c42da
New func signature
ppinchuk
committed
ef1bad1
Add tests
ppinchuk
committed
66b614a
Update func name
ppinchuk
committed
f7f5d9f
Fix a jur
ppinchuk
committed
cf94b6a

Commits on Jul 20, 2026

Stricter de-duplication
ppinchuk
committed
042b6bf
Minor cleanup
ppinchuk
committed
17a8a1f
Don't count inputs that have no docs
ppinchuk
committed
3c130ce

Commits on Jul 21, 2026

Add debug logging
ppinchuk
committed
e0985a2
Fix bug that broke crawl
ppinchuk
committed
b7a0a41
Fix error message
ppinchuk
committed
66cdaff
Log the docs that were found
ppinchuk
committed
b520bfd
Only keep documents where persistence was successful
ppinchuk
committed
3395476
Explicitly pass stems to file moves
ppinchuk
committed
f80c9b9
Fix tests
ppinchuk
committed
0d51d48
Remove periods from file stem
ppinchuk
committed
ea531a9
Add test
ppinchuk
committed
05fd2ab
Add phrases
ppinchuk
committed
f3b6976
HTML docs properly cached
ppinchuk
committed
52162a8
Use fast vs final AFL
ppinchuk
committed
c517db1
Add logging
ppinchuk
committed
5730f00
Add logging
ppinchuk
committed
57cb76e
Revert logging call
ppinchuk
committed
b7213fd
ELM crawl can now reload documents using fast and good file loaders
ppinchuk
committed
11d0682
Fix tests
ppinchuk
committed
adb7884
Fix tests
ppinchuk
committed
c9772f8
FIx tests
ppinchuk
committed
e2e6f88
Add versions
ppinchuk
committed
4516590
Update lockfile
ppinchuk
committed
9c65796
Update linter ignores
ppinchuk
committed
f0e72db