Developer Documentation

Table of Contents

Quickstart
Day to day
Working with OAuth
Frontend - Working with CSS and JS
Localization and Internationalization
Job Queues
Linting
Testing
Staging
Troubleshooting
Chore: Updating dependencies
See also

Quickstart

Clone the repository

git clone git@github.com:wmde/wikidata-mismatch-finder.git
cd wikidata-mismatch-finder

Install dependencies with composer (requires composer 2+)

composer install

OR use docker to install dependencies

docker run --rm \
    -u "$(id -u):$(id -g)" \
    -v $(pwd):/opt \
    -w /opt \
    laravelsail/php80-composer:latest \
    composer install --ignore-platform-reqs

Copy default environment variables
```
cp .env.example .env
```
Start the application server with Laravel sail
```
./vendor/bin/sail up
```
HOT TIP! Add the following alias to your .bashrc or .profile for easier command execution
```
alias sail='bash vendor/bin/sail'
```
Generate a unique app key with artisan
```
sail artisan key:generate
```
Run database migrations to create initial tables
```
sail artisan migrate
```
Install npm dependencies
```
sail npm install
```
Build the frontend code so you can see the javascript content of the app.
```
sail npm run dev
```

For more advanced frontend topics, see Frontend - Working with CSS and JS

Day to day

Start the application server

sail up

OR if you don't want to see live logging, start it in detached mode

sail up -d

Stop the application server

sail down

Destroy all the things

DANGER! Not only does this remove all downloaded images - which will result in some setup time when running sail up again, this will also delete all the data volumes for your application, removing all data from the database.

sail down --rmi all -v

Working with OAuth

In production, this application relies on wikidata.org's OAuth capabilities in order to authorize and identify users. Since it is not ideal to test in the production environment, we recommend creating your own personal OAuth consumer credentials for testing purposes, in order to develop locally.

Register your local testing app against wikidata in the following link:

OAuth Consumer Registration
Fill in the following fields in the form as follows (see screenshot for example):
- Application name: Make sure to include test in the name of the application, so that it is apparent for reviewers in the Wikimedia Foundation (WMF) that your consumer is meant for testing purposes.
- OAuth protocol version: Leave this field as is. This application uses OAuth1.0a.
- Application description: Fill in a description for your application, to make review easier.
- OAuth "callback" URL: Make sure to fill this in with a localhost address, with the port you would be running this application on (if not using the default port). This should be identical to the APP_URL in your .env file. Please also make sure you tick the checkbox next to "Allow consumer to specify a callback in requests and use "callback" URL above as required prefix."
- Applicable project: This field should be set to wikidatawiki.
- Types of grants being requested: Make sure to check the first radio button - "User identity verification only, no ability to read pages or act on a user's behalf". This will speed up the review process.
The rest of the fields can stay on their default values. Don't forget to check the disclaimer before submitting the form.

Once the form is submitted, make sure to write down your consumer key and consumer secret in a safe place and fill in the details in your local .env file:

#...

MEDIAWIKI_OAUTH_CLIENT_ID=<your-consumer-key>
MEDIAWIKI_OAUTH_CLIENT_SECRET=<your-consumer-secret>
MEDIAWIKI_OAUTH_CALLBACK_URL="${APP_URL}/auth/callback"
MEDIAWIKI_OAUTH_BASE_URL=https://www.wikidata.org

#...

As soon as you receive the email from the WMF team that your consumer is approved, you may start testing your application by logging in through your local instance's home page.

Frontend - Working with CSS and JS

Add the JS and CSS code in the resources/js and resources/css folder respectively.

Laravel mix (and webpack in the background) is responsible for compiling all of the frontend assets placed in these resources subfolders.

Before you begin working with frontend assets, please make sure to install npm dependencies:

sail npm install

Important! Please only run npm install using sail, or make sure that your local node.js version is the similar to sail: 16 or above.

To watch for changes in your frontend files, simply run:

sail npm run watch

To manually compile assets for your local dev environment run:

sail npm run dev

Localization and Internationalization

The Mismatch Finder application employs two separate localization systems: one for the server-side Laravel app, and another for the vue based client-side application.

To switch to any other language than English, set the uselang parameter in the URL. For example, to set the language to German:

http://<your-localhost>/?uselang=de

On the server side, we fully employ the default Laravel localization system and syntax, and messages in the server side should be added as or to php files in the resources/lang/en directory.

The client side localization system utilizes the banana-i18n library and format. Messages for the client side application are kept in the public/i18n/ directory, and are served in order to be consumed by the vue client. Each message that is added to the en.json file, should be documented in the qqq.json file as well, to provide more context for translators.

To ensure that your client side localization files are valid, run:

sail npm run i18n:validate

Job Queues

As the Mismatch Finder includes jobs to validate and import mismatch CSV files, it is possible to also configure the job queue runner to run locally.

By Default, the jobs will run synchronously to the requests that dispatch them. However, in order to queue and run jobs from the database, simple follow these two steps:

In your local .env file, change the value of QUEUE_CONNECTION to database (it should be set to sync by default).
Start the job queue runner by typing in the following command:
```
sail artisan queue:listen --timeout=1200
```
The --timeout=1200 flag in the command above increases the job timeout to 20 minutes thus ensuring that the queue worker doesn't abort jobs that take longer than a minute.

Linting

PHP Linting

The application uses phpcs to detect code format violations.

To run phpcs: sail composer run lint

To fix style errors automatically run: sail composer run fix

Note: Laravel uses the PSR2 Standard which expects camel caps method names. So you might get the error: Method name my_method() is not in camel caps if you scaffold your application. The recommendation there is to change the method names to camel case.

Javascript Linting

The application uses ESLint to detect code format violations in the frontend's *.js and *.vue files.

To run eslint: sail npm run lint

To fix style errors automatically run: sail npm run lint:fix

Testing

PHP Testing

The Laravel framework supports two types of testing: unit and feature tests. In contrast to unit tests, feature tests will boot your Laravel application and therefore are able to access your application's database and other framework services.

Mismatch finder uses an in-memory SQLite database for testing, so that the feature tests will leave your mariadb instance untouched. You can find the config settings for sqlite in phpunit.xml:

    <php>
        [...]
        <server name="DB_CONNECTION" value="sqlite"/>
        <server name="DB_DATABASE" value=":memory:"/>
        [...]
    </php>

Simply run # sail artisan test to start both unit and integration tests:

$ sail artisan test

   PASS  Tests\Unit\ExampleTest
  ✓ example

   PASS  Tests\Feature\ExampleTest
  ✓ example

  Tests:  2 passed
  Time:   0.16s

Javascript Testing

To test our Javascript code in general, and any vue components or pages we create in particular, this repository utilizes the jest test runner. In order to run JS tests, use the following command:

sail npm test

Browser Testing

The app uses Laravel Dusk as the Browser testing framework. Dusk uses a ChromeDriver installation, since we are using Laravel Sail, a standalone chrome installation is included in the docker setup.

To create an enviroment variables file for local browser tests, make a copy of the .env.dusk.example file named .env.dusk.local and fill in the APP_KEY with your unique app key. Your APP KEY should be the one created when setting up the project in the quickstart guide. You can find it in your .env file.

A one time setup is required before running the browser tests. This will create the tables needed for dusk:

$ docker container ls # note down the id of the mysql container

$ docker exec -it <mysql container id> mysql -ppassword

MariaDB [(none)]> CREATE DATABASE dusk character set UTF8mb4 collate utf8mb4_bin;

To run all the browser tests:

sail dusk

Staging

When there are changes that need to be tested before being deployed to production, like an UX decision or testing a new feature, the Mismatch Finder staging server can be used.

To deploy to this server, push your commits to a branch starting with the name staging/. For example staging/my_branch_name. The changes in the branch will be deployed to the server after all the checks have passed.

Troubleshooting

Address already in use

MariaDB

Problem: Sail refuses to start with the following error.

ERROR: for wikidata-mismatch-finder_mariadb_1
Cannot start service mariadb: driver failed programming external connectivity on endpoint 
wikidata-mismatch-finder_mariadb_1 (dbd1f278da0a5edb416b7875f224b83d1f2c08feac6e9a31d01d28567b83b4c7):
Error starting userland proxy: listen tcp4 0.0.0.0:3306: bind: address already in use

Possible Explanation: This error most probably occurs when you already have a local system-wide instance of MySQL or MariaDB running on the default port 3306.

Solution: Set the port to something other than 3306 using the FORWARD_DB_PORT environment variable.

While running the app:

FORWARD_DB_PORT=3308 sail up

OR In your .env file:

##...
APP_DEBUG=true
APP_URL=http://localhost

FORWARD_DB_PORT=3308
##...

(You don’t need to adjust the DB_PORT in .env.dusk.local after this – that variable needs to match the .env DB_PORT, which you haven’t changed, not the FORWARD_DB_PORT.)

`laravel.test`

Problem: Sail refuses to start with the following error.

ERROR: for wikidata-mismatch-finder_laravel.test_1
Cannot start service laravel.test: driver failed programming external connectivity on endpoint 
wikidata-mismatch-finder_laravel.test_1 (0fa137072f412614077154ca7927cbb0ca2f3df5474879bb5e8e33f15a1683e3): 
Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use

Possible Explanation: This error most probably occurs when you already have a local web server running on the default port 80.

Solution: Set the port to something other than 80 using the APP_PORT environment variable.

While running the app:

APP_PORT=1337 sail up

OR In your .env file:

##...
APP_DEBUG=true
APP_URL=http://localhost:1337

APP_PORT=1337
##...

OAuth Error retrieving temporary credentials

Problem: The local server returns this error on clicking the log in link.

League\OAuth1\Client\Credentials\CredentialsException
Error in retrieving temporary credentials.
http://localhost/auth/login

Possible Explanation:

If, after following the steps in the Working with OAuth section, you are getting this error - this can be due to either misconfigured consumer keys, a wrong app url, or simply an attempt to authorize a consumer that is still pending WMF review.

Solution:

Make sure your MEDIAWIKI_OAUTH_CLIENT_ID and MEDIAWIKI_OAUTH_CLIENT_SECRET are correctly set to your consumer key and secret in your local .env file.
Make sure that your APP_URL is set correctly to your localhost address and the port your application is running on (if running on a non default port).
Double check to see that the WMF has emailed to approve your localhost OAuth consumer, if they have not responded yet, please wait patiently, as the review process might take a few hours.

Chore: Updating dependencies

Npm dependencies

You can see which dependencies have new releases by first making sure your local dependencies are up-to-date by executing npm ci and then running npm outdated.

Dependencies should generally be updated to the latest version. If you discover that a dependency should not be updated for some reason, please change this section of the documentation to reflect that (even if it's temporary). If a dependency can only be updated with substantial manual work, you can create a new task for it and skip it in the context of the current chore.

The recommended way to update dependencies is to collect related dependency updates into grouped commits; this keeps the number of commits to review manageable (compared to having one commit for every update), while keeping the scope of each commit limited and increasing reviewability and debuggability (compared to combining all updates in a single commit). For example, this can be one commit for each of:

all ESLint-related dependency updates
all Jest-related dependency updates
all Vue-related dependency updates
all PostCSS/Stylelint-related dependency updates
npm update for all other dependency updates

Make sure the app is running: sail up -d. Then make sure that all unit tests still pass and building still works for every local commit using:

sail npm run test for unit tests.
sail dusk for browser tests.

Composer dependencies

Since Mismatch Finder is using Laravel, we need to make sure that we update backend dependencies as well. To do this, run sail composer outdated.

Run sail composer update to install the new packages. Then make sure that all unit tests still pass and building still works for every local commit using sail artisan test.

Don't update to any major version of Laravel since that would require a migration and would be out of the scope of a chore.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Developer Documentation

Quickstart

Day to day

Start the application server

Stop the application server

Destroy all the things

Working with OAuth

Frontend - Working with CSS and JS

Localization and Internationalization

Job Queues

Linting

PHP Linting

Javascript Linting

Testing

PHP Testing

Javascript Testing

Browser Testing

Staging

Troubleshooting

Address already in use

MariaDB

`laravel.test`

OAuth Error retrieving temporary credentials

Chore: Updating dependencies

Npm dependencies

Composer dependencies

See also

Files

README.md

Latest commit

History

README.md

File metadata and controls

Developer Documentation

Quickstart

Day to day

Start the application server

Stop the application server

Destroy all the things

Working with OAuth

Frontend - Working with CSS and JS

Localization and Internationalization

Job Queues

Linting

PHP Linting

Javascript Linting

Testing

PHP Testing

Javascript Testing

Browser Testing

Staging

Troubleshooting

Address already in use

MariaDB

laravel.test

OAuth Error retrieving temporary credentials

Chore: Updating dependencies

Npm dependencies

Composer dependencies

See also

`laravel.test`