Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/behavior-bug.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ name: Behavior Bug
about: Let us know when a behavior isn't working as expected.
title: Behavior Bug
labels: bug
assignees: ''

assignees: ""
---

**URL**
Expand All @@ -17,6 +16,7 @@ Note: The expected functionality can be found on [Behaviors Overview](https://ar
If the status of one of the test is failing, that may be related and a similar issue.

**What tool do you see this issue in?**

- [ ] ArchiveWeb.page Extension
- [ ] ArchiveWeb.page App
- [ ] Browsertrix Crawler
Expand Down
5 changes: 2 additions & 3 deletions .github/ISSUE_TEMPLATE/new-behavior-request.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
---
name: New Behavior Request
about: Suggest a New Behavior
title: 'New Behavior Request For: '
title: "New Behavior Request For: "
labels: enhancement
assignees: ''

assignees: ""
---

**What website should this before be applied to?**
Expand Down
39 changes: 19 additions & 20 deletions .github/workflows/autoplay-vimeo.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
pull_request:
schedule:
- cron: '12 16 * * *'
- cron: "12 16 * * *"

jobs:
autoscroll_test:
Expand All @@ -15,28 +15,27 @@ jobs:
node-version: [22.x]

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- name: install requirements
run: yarn install
- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build
- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env
- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE
- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/crawls:/crawls $BROWSERTRIX_IMAGE crawl --url https://oembed.link/https://vimeo.com/1084537 --limit 1 --generateCDX --collection test

- name: check mp4 capture
run: grep '"video/mp4"' ./crawls/collections/test/indexes/index.cdxj
- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/crawls:/crawls $BROWSERTRIX_IMAGE crawl --url https://oembed.link/https://vimeo.com/1084537 --limit 1 --generateCDX --collection test

- name: check mp4 capture
run: grep '"video/mp4"' ./crawls/collections/test/indexes/index.cdxj
39 changes: 19 additions & 20 deletions .github/workflows/autoplay-youtube.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
pull_request:
schedule:
- cron: '10 16 * * *'
- cron: "10 16 * * *"

jobs:
autoscroll_test:
Expand All @@ -15,28 +15,27 @@ jobs:
node-version: [22.x]

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- name: install requirements
run: yarn install
- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build
- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env
- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE
- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/crawls:/crawls $BROWSERTRIX_IMAGE crawl --url https://oembed.link/https://www.youtube.com/watch?v=aT-Up5Y4uRI --limit 1 --generateCDX --collection test

- name: check mp4 capture
run: grep '"video/mp4"' ./crawls/collections/test/indexes/index.cdxj
- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/crawls:/crawls $BROWSERTRIX_IMAGE crawl --url https://oembed.link/https://www.youtube.com/watch?v=aT-Up5Y4uRI --limit 1 --generateCDX --collection test

- name: check mp4 capture
run: grep '"video/mp4"' ./crawls/collections/test/indexes/index.cdxj
43 changes: 21 additions & 22 deletions .github/workflows/autoscroll.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
pull_request:
schedule:
- cron: '10 16 * * *'
- cron: "10 16 * * *"

jobs:
autoscroll_test:
Expand All @@ -15,31 +15,30 @@ jobs:
node-version: [22.x]

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- name: install requirements
run: yarn install
- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build
- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env
- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE
- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js $BROWSERTRIX_IMAGE crawl --url https://www.iana.org/numbers --limit 1 --logging debug --context behaviorScript --behaviors autoscroll > ./log
- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js $BROWSERTRIX_IMAGE crawl --url https://www.iana.org/numbers --limit 1 --logging debug --context behaviorScript --behaviors autoscroll > ./log

- name: check for autoscroll debug log line
run: grep 'Skipping autoscroll, page seems to not be responsive to scrolling events' ./log

- name: check that state is logged as well
run: grep '{"state":{"segments":1}' ./log
- name: check for autoscroll debug log line
run: grep 'Skipping autoscroll, page seems to not be responsive to scrolling events' ./log

- name: check that state is logged as well
run: grep '{"state":{"segments":1}' ./log
82 changes: 41 additions & 41 deletions .github/workflows/facebook-page.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
pull_request:
schedule:
- cron: '20 16 * * *'
- cron: "20 16 * * *"

jobs:
facebook_page:
Expand All @@ -16,43 +16,43 @@ jobs:
node-version: [22.x]

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- uses: prewk/s3-cp-action@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
SOURCE: ${{ secrets.PROFILE_URL }}
DEST: './profile.tar.gz'

- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/Capture-Sample-105559135004406 --limit 1 --logging behaviors --profile /profile.tar.gz > ./log

- name: cat log
run: cat ./log

- uses: sergeysova/jq-action@v2
name: compare crawl log to expected
with:
cmd: >
cat log
| jq -Rr -c 'fromjson? | .details | select(type == "object") | {msg,state} | select(.msg != null)'
| diff - ./test/expected-facebook-page.log
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- uses: prewk/s3-cp-action@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
SOURCE: ${{ secrets.PROFILE_URL }}
DEST: "./profile.tar.gz"

- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/Capture-Sample-105559135004406 --limit 1 --logging behaviors --profile /profile.tar.gz > ./log

- name: cat log
run: cat ./log

- uses: sergeysova/jq-action@v2
name: compare crawl log to expected
with:
cmd: >
cat log
| jq -Rr -c 'fromjson? | .details | select(type == "object") | {msg,state} | select(.msg != null)'
| diff - ./test/expected-facebook-page.log
82 changes: 41 additions & 41 deletions .github/workflows/facebook-photos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
pull_request:
schedule:
- cron: '29 16 * * *'
- cron: "29 16 * * *"

jobs:
facebook_test:
Expand All @@ -16,43 +16,43 @@ jobs:
node-version: [22.x]

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- uses: prewk/s3-cp-action@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
SOURCE: ${{ secrets.PROFILE_URL }}
DEST: './profile.tar.gz'

- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/Capture-Sample-105559135004406/photos --limit 1 --logging behaviors --profile /profile.tar.gz > ./log

- name: cat log
run: cat ./log

- uses: sergeysova/jq-action@v2
name: compare crawl log to expected
with:
cmd: >
cat log
| jq -Rr -c 'fromjson? | .details | select(type == "object") | {msg,state} | select(.msg != null)'
| diff - ./test/expected-facebook-photos.log
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}

- uses: prewk/s3-cp-action@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
SOURCE: ${{ secrets.PROFILE_URL }}
DEST: "./profile.tar.gz"

- name: install requirements
run: yarn install

- name: build behaviors
run: yarn run build

- uses: c-py/action-dotenv-to-setenv@v2
with:
env-file: test/test.env

- name: get browsertrix image
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/Capture-Sample-105559135004406/photos --limit 1 --logging behaviors --profile /profile.tar.gz > ./log

- name: cat log
run: cat ./log

- uses: sergeysova/jq-action@v2
name: compare crawl log to expected
with:
cmd: >
cat log
| jq -Rr -c 'fromjson? | .details | select(type == "object") | {msg,state} | select(.msg != null)'
| diff - ./test/expected-facebook-photos.log
Loading
Loading