Browsertrix Cloud 1.7.0 Beta 0

Deployment

This release can be deployed with Helm by running:

helm upgrade --install btrix https://github.com/webrecorder/browsertrix-cloud/releases/download/v1.7.0-beta.0/browsertrix-cloud-v1.7.0-beta.0.tgz

See the development guide for more info how to deploy Browsertrix Cloud.

What's Changed

• Add and enforce org maxPagesPerCrawl quota by @tw4l in #1044
• ingress: minor tweaks to ingress to update to latest spec: by @ikreymer in #1096
• Use Shared Services for Crawling, Redis, Profile Browsers by @ikreymer in #1088
• feat: implement collections array with names for archived item details by @Chickensoupwithrice in #1098
• 1.6.3 Fixes - Fix sort order for Latest Crawl + 'Remove From Collection' action menu on archived items in collections by @ikreymer in #1113
• Add max crawl size option to backend and frontend by @tw4l in #1045
• dev: GitHub Issue form update: updates "user story" title by @Shrinks99 in #1112
• Add support for collectionIds to archived item PATCH endpoints by @tw4l in #1121
• publish helm chart on release (fixes #1114) (#1117) by @ikreymer in #1123
• Fix unsetting all collections via PATCH update by @tw4l in #1126
• Add event webhook notifications system to backend by @tw4l in #1061
• fix: Terminology unification "crawls" & "archive data" → "items" by @Shrinks99 in #1127
• fix: add "v" to helm chart release filename by @Chickensoupwithrice in #1141
• feat: init k3s ansible playbook by @Chickensoupwithrice in #1071
• Manage collection from archived item details by @SuaYoo in #1085
• supports overriding the replayweb.page version without having to be r… by @ikreymer in #1122
• Use base10 for sizes in frontend by @tw4l in #1133
• Fix too many request errors in UI by @SuaYoo in #1130
• null check: check before accessing config in 'get_all_crawl_search_values' by @ikreymer in #1144
• fix: "crawls" → "items" in collection add component by @Shrinks99 in #1145
• Better Resource settings per number of browsers by @ikreymer in #1103
• chart: move minio credentials to separate secret by @ikreymer in #1143
• Print crawl log to operator log (mostly for testing) by @ikreymer in #1148
• Add operator logs to nightly tests by @tw4l in #1150
• Add and enforce org storage quota by @tw4l in #1106
• ingress: simplify ingress config: (fixes #1135) by @ikreymer in #1146
• Fix 0015 migration by @ikreymer in #1154
• feat: add SMTP {port, use_tls} config by @Chickensoupwithrice in #1142
• Operator refactor to control pods + pvcs directly instead of statefulsets by @ikreymer in #1149
• Issue all non-upload storage-quota-update events from LiteElement by @tw4l in #1151
• Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached by @ikreymer in #1162
• Expect that crawl deleted response is bool, not int by @tw4l in #1170
• Add event webhook tests by @tw4l in #1155
• Make improvements to reset command by @tw4l in #1160
• type fix: ensure max crawl size and max crawl timeout values are set to 0 instead of null by @ikreymer in #1167
• Fix crawl scope help text by @SuaYoo in #1169
• Refactor / Cleanup: move ops functions back into classes by @ikreymer in #1171
• Improved type checking for backend with mypy by @ikreymer in #1174
• Increase wait to 30 seconds to ensure webhooks are sent by @tw4l in #1173
• Rename "Production" to "Remote" by @Chickensoupwithrice in #1105
• Enable saving individual collection form sections by @SuaYoo in #1166
• Implement sync streaming for finished crawl logs by @tw4l in #1168
• Reset crawl state to running when any crawlers are running (after post-process states) by @ikreymer in #1179
• version: bump RWP version to 1.8.12 by @ikreymer in #1181

Full Changelog: v1.6.2...v1.7.0-beta.0

Browsertrix Cloud 1.6.4

Helm charts now published for each release starting with 1.6.4

What's Changed

• publish helm chart on release (fixes #1114) by @ikreymer in #1117

Full Changelog: v1.6.3...v1.6.4

Deployment

This release can be deployed with Helm by running:

helm upgrade --install btrix https://github.com/webrecorder/browsertrix-cloud/releases/download/v1.6.4/browsertrix-cloud-v1.6.4.tgz

See the development guide for more info how to deploy Browsertrix Cloud.

Browsertrix Cloud 1.6.4 Beta 0

Test publishing helm chart as part of release

What's Changed

• publish helm chart on release (fixes #1114) by @ikreymer in #1117

Full Changelog: v1.6.3...v1.6.4-beta.0

Browsertrix Cloud 1.6.3

Bug fix release

What's Changed

• Fix sorting by 'Latest Crawl': by @ikreymer in #1109
• Fix missing menu for archived items in collection, add 'Remove Collection' by @SuaYoo in #1110

Full Changelog: v1.6.2...v1.6.3

Browsertrix Cloud 1.6.2

What's Changed

• docs: Adds guidelines for using admonitions by @Shrinks99 in #1084
• docs: Adds information about 1.6 features to documentation by @Shrinks99 in #1086
• docs: formatting fixes & minor content updates by @Shrinks99 in #1091
• dev: Adds GitHub feature issue template by @Shrinks99 in #1087
• exclusion regex: show unmodified regex string, avoid dropping the ''… by @ikreymer in #1094
• Paginate Workflows and refactor to use server-side queries by @SuaYoo in #1078

Full Changelog: v1.6.1...v1.6.2

Browsertrix Cloud 1.6.1

Updates

• Redis connection leak fix
• Improved documentation for frontend dev: https://docs.browsertrix.cloud/develop/frontend-dev/
• Logs: Show error details and add download button

What's Changed

• Webpack config improvements by @SuaYoo in #1063
• helm chart tweaks: by @ikreymer in #1067
• Update archived item URLs by @SuaYoo in #1064
• fix redis connection leaks + exclusions error: (fixes #1065) by @ikreymer in #1066
• fix: password check github action constructor error by @Chickensoupwithrice in #1077
• Update frontend local dev guide by @SuaYoo in #1073
• Add MkDocs YAML schema to vscode settings.json by @Shrinks99 in #1075
• frontend: minor fixes for queue / exclusions: by @ikreymer in #1076
• follow-up to #1066: update redis to 5.0.0 which includes full fix for… by @ikreymer in #1081
• Show details in crawl error log by @SuaYoo in #1079
• Add button to download error logs by @SuaYoo in #1080
• chart resources: adjust backend memory to 350Mi, as 200Mi was too low by @ikreymer in #1082

Full Changelog: v1.6.0...v1.6.1

Browsertrix Cloud 1.6.0

Major Features

• Support for uploading WACZ files via UI and API (including API integration with ArchiveWeb.page)
• Support for Collections for dynamically grouping crawls and uploads, adding metadata and browsing archived data as a single object.
• Support for downloading collection as single WACZ and sharing collections publicly
• Improvements to docs and user guide at: docs.browsertrix.cloud

Breaking Change

• Resource limits has been adjusted to use unified value for request/limits for memory and only requests for CPU, see #1019. Overridden Helm chart values may need to be adjusted to the latest values.yaml

What's Changed

• Use collection uuid as id (instead of name) by @tw4l in #855
• Upgrade Shoelace to stable version (v2) by @SuaYoo in #856
• Add collection sorting and filtering by @tw4l in #863
• Improve sorting workflows by lastUpdated by @tw4l in #826
• Add crawlSuccessfulCount to workflows by @tw4l in #871
• tests: fixes for crawl cancel + crawl stopped by @ikreymer in #864
• Fix crawl stopping tests by @tw4l in #875
• Fix copy tags button disabling when no tags on Crawl Details page by @Shrinks99 in #877
• Show org creation form if there are no orgs by @SuaYoo in #883
• Update backend local development docs by @SuaYoo in #884
• Wait for DB init for healthcheck + settings by @ikreymer in #885
• Add current context and confirmation dialog to btrix helper reset and bootstrap functions by @tw4l in #887
• Rework collections to track collections in Crawl by @tw4l in #878
• Optimize workflow statistics updates by @tw4l in #892
• collection delete: by @ikreymer in #896
• Sort collection by last updated by @SuaYoo in #897
• Fix ResizeObserver loop error by @SuaYoo in #902
• Concurrent Crawl Limit by @ikreymer in #874
• btrix helper: Add -microk8s flag to explicitly use microk8s by @tw4l in #888
• Use helm tpl to include diskUtilization value in crawler_args by @tw4l in #889
• crawls list: unset errors in crawls list response to avoid very large… by @ikreymer in #904
• Nightly Tests Fix by @ikreymer in #905
• Frontend collections beta UI by @SuaYoo in #886
• Adds icon name and tooltip content fields to btrix-copy-button by @Shrinks99 in #879
• config: add overridable user agent suffix + user agent by @ikreymer in #910
• Precompute crawl file stats by @tw4l in #906
• Adds initial version of the documentation style guide by @Shrinks99 in #891
• frontend: configure running with a fixed 'replay.json', auth headers … by @ikreymer in #899
• fix finished workflows incorrectly being displayed as running by @ikreymer in #909
• Fix bug preventing deleting collections with no crawls by @tw4l in #912
• Adds all workflow settings to the user docs with descriptions by @Shrinks99 in #894
• Fix post-crawl collection stats update and add test by @tw4l in #918
• Admin Panel Minor Frontend Style Updates by @Shrinks99 in #915
• Future of the User Guide page & Org Settings docs! by @Shrinks99 in #914
• Fix Extra URLs in Scope by @ikreymer in #913
• Adds documentation link and version copy button to footer by @Shrinks99 in #920
• Add frontend support for auto-adding collections to workflows by @tw4l in #916
• Backend API consistency pass by @tw4l in #921
• Minor grammatical changes to documentation by @Shrinks99 in #919
• Fix tag editor menu z-index by @Shrinks99 in #939
• Unset default scopeType for seeds so they inherit parent scopeType by… by @tw4l in #952
• Display actions in watch crawl tab after workflow crawl completes by @tw4l in #943
• docs: ansible deploy docs reflect expected env var names by @Chickensoupwithrice in #946
• Uploads API: BaseCrawl refactor + Initial support for /uploads endpoint by @ikreymer in #937
• fix: idempotent mongodb creation, with saved facts by @Chickensoupwithrice in #945
• feat: configure kubectl through ansible by @Chickensoupwithrice in #948
• feat: configure mongodb firewall by @Chickensoupwithrice in #949
• operator: ensure finished is always set by @ikreymer in #953
• remove default: similar to #952, remove default extraHops setting as … by @ikreymer in #954
• re-add support for passing in 'crawler_extra_args' for additional/custom by @ikreymer in #957
• List crawls, uploads, and all objects in UI by @SuaYoo in #941
• Add /artifacts/upload/ path for uploads by @ikreymer in #960
• fix: only use db_create when the db is created by @Chickensoupwithrice in #959
• Frontend more upload path fixes by @ikreymer in #961
• fix: pass ansible-lint in DO playbook by @Chickensoupwithrice in #962
• fix: configure_kubectl is the variable name by @Chickensoupwithrice in #963
• fix: mongodb uri password not accessible on second API call by @Chickensoupwithrice in #964
• Reorder Limits section by @Shrinks99 in #966
• Tests: fix test get crawl loop by @ikreymer in #967
• uploads api: support filtering uploads by collectionId by @ikreymer in #969
• uploads api ext: by @ikreymer in #970
• Allow users to add uploads to collections by @SuaYoo in #968
• frontend: fix to collection editor with crawls and uploads by @ikreymer in #971
• fix: resolve ansible pipenv dependencies successfully by @Chickensoupwithrice in #977
• Fix frontend formatting pre-commit by @SuaYoo in #991
• fix issues that caused cronjob container to crash: by @ikreymer in #987
• feat: ansible lint github action by @Chickensoupwithrice in #976
• Add crawler Use Sitemap option to Browsertrix Cloud by @tw4l in #978
• Upgrade lit + shoelace to reduce build size by @SuaYoo in #938
• Fix pull_request syntax in ansible lint GH Action by @tw4l in #995
• Rework models and crawl responses by @tw4l in #983
• Add collections and tags to upload API endpoints by @tw4l in #993
• Upload WACZ via UI by @SuaYoo in #992
• Add totalSize to collections and make it sortable in list endpoint by @tw4l in #1001
• Add migration to replace None with 0 for configmap CRAWL_TIMEOUT by @tw4l in #1008
• load handling: only launch redis pods if crawler pods successfully ...

Browsertrix Cloud 1.6.0 Beta 4

What's Changed

• backend: crawlout: include raw crawnconfig in api details, fixes #1030 by @ikreymer in #1055
• Collection share dialog + copy updates by @SuaYoo in #1056
• Optimize Release Build on CI by @ikreymer in #1057
• Standardize archived item filtering, sorting and labels by @SuaYoo in #1054

Full Changelog: 1.6.0-beta.3...v1.6.0-beta.4

Browsertrix Cloud 1.6.0 Beta 3

What's Changed

• fix: don't break when unable to parse yaml file by @Chickensoupwithrice in #1017
• chart: always pull latest crawler image - since default image is poin… by @ikreymer in #1018
• fix(build): use /usr/bin/env bash instead of /bin/bash by @Chickensoupwithrice in #1020
• Resource Constraints Cleanup: (fixes #895) by @ikreymer in #1019
• fix(build): build frontend was missing required env vars for older versions of Docker by @Chickensoupwithrice in #1021
• Add crawl ending states to crawljob + display in UI by @ikreymer in #1022
• List web captures in Collection by @SuaYoo in #1024
• fix: usr/bin/env bash interpreter for btrix by @Chickensoupwithrice in #1028
• Sort Collections by size by @SuaYoo in #1026
• feat: Display waiting message while backend is initializing by @Chickensoupwithrice in #1035
• Add info bar to Collection detail view by @SuaYoo in #1036
• Support for Publicly Shareable Collections by @ikreymer in #1038
• fix: run yarn format in frontend dir by @Chickensoupwithrice in #1043
• Backend: standardize search values, filters, and sorting for archived items by @tw4l in #1039
• fix public collections endpoint returning 404 when not public by @ikreymer in #1052

Full Changelog: v1.6.0-beta.2...1.6.0-beta.3

Browsertrix Cloud 1.6.0 Beta 2

What's Changed

• Minor gramatical changes to documentation by @Shrinks99 in #919
• Fix tag editor menu z-index by @Shrinks99 in #939
• Unset default scopeType for seeds so they inherit parent scopeType by… by @tw4l in #952
• Display actions in watch crawl tab after workflow crawl completes by @tw4l in #943
• docs: ansible deploy docs reflect expected env var names by @Chickensoupwithrice in #946
• Uploads API: BaseCrawl refactor + Initial support for /uploads endpoint by @ikreymer in #937
• fix: idempotent mongodb creation, with saved facts by @Chickensoupwithrice in #945
• feat: configure kubectl through ansible by @Chickensoupwithrice in #948
• feat: configure mongodb firewall by @Chickensoupwithrice in #949
• operator: ensure finished is always set by @ikreymer in #953
• remove default: similar to #952, remove default extraHops setting as … by @ikreymer in #954
• re-add support for passing in 'crawler_extra_args' for additional/custom by @ikreymer in #957
• List crawls, uploads, and all objects in UI by @SuaYoo in #941
• Add /artifacts/upload/ path for uploads by @ikreymer in #960
• fix: only use db_create when the db is created by @Chickensoupwithrice in #959
• Frontend more upload path fixes by @ikreymer in #961
• fix: pass ansible-lint in DO playbook by @Chickensoupwithrice in #962
• fix: configure_kubectl is the variable name by @Chickensoupwithrice in #963
• fix: mongodb uri password not accessible on second API call by @Chickensoupwithrice in #964
• Reorder Limits section by @Shrinks99 in #966
• Tests: fix test get crawl loop by @ikreymer in #967
• uploads api: support filtering uploads by collectionId by @ikreymer in #969
• uploads api ext: by @ikreymer in #970
• Allow users to add uploads to collections by @SuaYoo in #968
• frontend: fix to collection editor with crawls and uploads by @ikreymer in #971
• fix: resolve ansible pipenv dependencies successfully by @Chickensoupwithrice in #977
• Fix frontend formatting pre-commit by @SuaYoo in #991
• fix issues that caused cronjob container to crash: by @ikreymer in #987
• feat: ansible lint github action by @Chickensoupwithrice in #976
• Add crawler Use Sitemap option to Browsertrix Cloud by @tw4l in #978
• Upgrade lit + shoelace to reduce build size by @SuaYoo in #938
• Fix pull_request syntax in ansible lint GH Action by @tw4l in #995
• Rework models and crawl responses by @tw4l in #983
• Add collections and tags to upload API endpoints by @tw4l in #993
• Upload WACZ via UI by @SuaYoo in #992
• Add totalSize to collections and make it sortable in list endpoint by @tw4l in #1001
• Add migration to replace None with 0 for configmap CRAWL_TIMEOUT by @tw4l in #1008
• load handling: only launch redis pods if crawler pods successfully loaded. by @ikreymer in #1009
• Rename notes to description in frontend and backend by @tw4l in #1011
• feat: add pre-commit to check we don't have real passwords in yml files by @Chickensoupwithrice in #990
• feat: ansible DO teardown by @Chickensoupwithrice in #950
• feat: create DO registry if it doesn't exist by @Chickensoupwithrice in #947
• Streaming Download for Collections by @ikreymer in #1012
• Show only running crawls in superadmin view by @SuaYoo in #1015
• Add proxy settings for crawler and profilebrowser by @vnznznz in #997

New Contributors

• @Chickensoupwithrice made their first contribution in #946
• @vnznznz made their first contribution in #997

Full Changelog: v1.6.0-beta.1...v1.6.0-beta.2