Skip to content

Implement Mediawiki REST API render#1947

Merged
kelson42 merged 4 commits intomainfrom
mediawiki-rest-api
Dec 4, 2023
Merged

Implement Mediawiki REST API render#1947
kelson42 merged 4 commits intomainfrom
mediawiki-rest-api

Conversation

@VadimKovalenkoSNF
Copy link
Copy Markdown
Contributor

@VadimKovalenkoSNF VadimKovalenkoSNF commented Nov 14, 2023 •
edited
Loading

Fixes #1601
Successor of #1926
Depends on #1929

  1. Added MediawikiRestApiRenderer that extends WikimediaDesktopRenderer.
  2. Added a temporary workaround for resolving article urls till the solution from PR#1929 is merged.
  3. Created mediawiki-rest-api.director.ts which will be used once PR#1929 is applied.

@VadimKovalenkoSNF VadimKovalenkoSNF self-assigned this Nov 14, 2023
@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 14, 2023 •
edited
Loading

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (8c2c311) 74.28% compared to head (900f58d) 74.42%.
Report is 2 commits behind head on main.

Files Patch % Lines
src/renderers/renderer.builder.ts 50.00% 4 Missing and 2 partials ⚠️
src/MediaWiki.ts 90.47% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1947      +/-   ##
==========================================
+ Coverage   74.28%   74.42%   +0.14%     
==========================================
  Files          39       41       +2     
  Lines        3095     3136      +41     
  Branches      679      689      +10     
==========================================
+ Hits         2299     2334      +35     
- Misses        678      683       +5     
- Partials      118      119       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@VadimKovalenkoSNF VadimKovalenkoSNF mentioned this pull request Nov 15, 2023
Closed
@kelson42
Copy link
Copy Markdown
Contributor

kelson42 commented Nov 15, 2023

@VadimKovalenkoSNF This is the last PR, hope you can fix it and put to review today.

VadimKovalenkoSNF
VadimKovalenkoSNF commented Nov 15, 2023
View reviewed changes
Comment thread test/e2e/forceRender.test.ts
}
})

test('Scrape article from bm.wikipedia.org should throw error when using VisualEditor render', async () => {
Copy link
Copy Markdown
Contributor Author

@VadimKovalenkoSNF VadimKovalenkoSNF Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this set of tests should be removed - they caused issues in CI pipeline related to memory corruption. One of the examples of this problem can be found in these logs - https://github.com/openzim/mwoffliner/actions/runs/6854615228/job/18637919124#step:7:1589
I also faced with terminate called after throwing an instance of 'Xapian::DatabaseLockError' Aborted (core dumped) and Node error process completed with exit code 139. Probably, that happened because we forced the application to throw an error here without the right clearing of the Redis state and/or dangling process.exit(1) invocation.

VadimKovalenkoSNF
VadimKovalenkoSNF commented Nov 15, 2023
View reviewed changes
Comment thread src/parameterList.ts
mwActionApiPath: 'Mediawiki API path (per default "/w/api.php")',
mwRestApiPath: 'Mediawiki Rest API path (per default "/api/rest_v1")',
mwRestApiPath: 'Wikimedia Rest API path (per default "/api/rest_v1")',
mwMediaWikiRestApiPath: 'Mediawiki Rest API path (per default "w/rest.php/v1/page/")',
Copy link
Copy Markdown
Contributor Author

@VadimKovalenkoSNF VadimKovalenkoSNF Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably not the best naming, but I have to follow the convention that mw stands for Mediawiki singleton in mwoffliner, and MediaWikiRestApiPath should be named like this because this is how this API was named originally.

VadimKovalenkoSNF
VadimKovalenkoSNF commented Nov 15, 2023
View reviewed changes
Comment thread test/unit/mwApiCapabilities.test.ts
@@ -0,0 +1,85 @@
import MediaWiki from '../../src/MediaWiki.js'

describe('Checking Mediawiki capabilities', () => {
Copy link
Copy Markdown
Contributor Author

@VadimKovalenkoSNF VadimKovalenkoSNF Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test suit should have been introduced a while ago as a convenient way to check the capabilities of different wikis, as well as testing of the specific receipts. Before trying to scrape some wiki, this is a good place to start.

@VadimKovalenkoSNF VadimKovalenkoSNF marked this pull request as ready for review November 15, 2023 14:03
This was referenced Nov 15, 2023
Closed
Closed
@kelson42 kelson42 merged commit ad5dc1d into main Dec 4, 2023
@kelson42 kelson42 deleted the mediawiki-rest-api branch December 4, 2023 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kelson42 kelson42 Awaiting requested review from kelson42

Assignees

@VadimKovalenkoSNF VadimKovalenkoSNF

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support of new Mediawiki REST API

2 participants

@VadimKovalenkoSNF @kelson42