This is a node.js app that coordinates online data collection. See the experiment template for an example of how to integrate a lab.js front-end.
When paired with the right front-end logic (see experiment template), it handles:
- managing participant sessions
- preventing repeat takers
- assigning participants to lists/conditions in a balanced way
- storing both incremental and complete experimental data and making it available to download in a JSON format.
Static files for each experiment should be placed in a directory with the name
of the experiment inside web-root. Replace spaces, parens, etc. with -,
so if your experiment is called “Selective adaptation”, you should put the
lab.js export in web-root/selective-adaptation/.
Before deploying the final version of your experiment, do a “test run” in the mechanical turk sandbox. Because the server maintains a record of which users have completed the experiment in order to balance the assignment of users to experimental lists, it’s recommended to use a second instance of the server with its own domain for sandbox testing.
To set up the lists (conditions, etc.) for your experiment, you need to make a PUT request to https://experiments.leap-lab.org/name-of-experiment/lists, with the JSON-encoded array of lists in the body, like so:
PUT https://experiments.leap-lab.org/name-of-experiment/lists Content-Type:
application/json
[
{
"list_id": 1,
"condition": "condition-1",
},
{
"list_id": 2,
"condition": "condition-2",
}
]
There is, alas, currently no convenient way of doing this. Curl or emacs-restclient (like above) is your friend.
The condition field can contain any valid JSON (number, string, array, or
object/dict). It is available on the Lab.js front-end as
this.parameters.session.condition. You could use it to set basic
conditions (via a string/number), block order (via an array of
strings/objects), or some more complicated combination of different values
in a crossed design (via an object or array of objects).
If you want something other than an equal number of subjects assigned to each
list, you can add a count field to each element. The list balancer will
assign subjects to the list with the largest difference between the existing
(non-abandoned) sessions in that list and the count. The default value of
count is zero, which means that the list balancer by default will assign
each new subject to the list with the fewest subjects working/completed.
This is useful mainly if you need to re-balance lists after analyzing the data and dropping some subjects (by pre-defined exclusion criteria of course). Or if you have a strange design where you want many more subjects in some conditions than others.
Configuration information is pulled from .env file. This is not under
version control, but the .env.example file shows the format.
See docker-compose.yml and docker-compose.devel.yml for the components of the app.
The webserver service is an nginx webserver. This has two functions.
First, it serves static files for the experiments from web-root; any
request that doesn’t correspond to a file or subdirectory with index.html
in web-root gets passed to the node.js app.
Second, it handles the https configuration and sets response headers correctly to ensure a secure connection.
This handles participant sessions in the experiments, including assigning subjects to conditions if needed and collecting data.
It also provides an admin interface to retrieve data, manage condition lists, and monitor user sessions.
This records information on each participant session, generated data, and experimental lists.
The components of the experiment server are coordinated by docker-compose.
The general steps are
- Install
dockeranddocker-compose, and start the docker daemon (e.g. with$ sudo systemctl start docker). - Clone this repository.
- Copy
.env.exampleto.envand edit as needed (probably at leastMONGO_USERNAME,MONGO_PASSWORD,MONGO_DB,DOMAIN, andNODE_ENV). - Create and start the necessary containers with
docker-compose up -d.
Depending on where you’re deploying (local vs. remote machine) and to what end (development or production), the specific steps are detailed below. Most of the work is handled by swapping in the appropriate docker-compose file.
A separate docker-compose config is provided for local development:
$ docker-compose -f docker-compose.devel.yml up
This will create a container for the database if needed, and listen on
port 8080. The local app directory is mounted in the countainer (to
/home/node/app) and nodemon listens for changes in the source. This differs
from the production docker compose config which copies the app source and static
assets into the container when it’s built.
Make sure that no node_modules directory is present since it will mask the
volume that’s created by docker-compose.
Live development of the experiment server itself can be done on a remote machine by combining the production and development docker compose configs:
$ docker-compose -f docker-compose.yml -f docker-compose.devel.yml up -d
This combines the production nginx web server to handle HTTP/S requests with
the live-reloading javascript server.
The default docker-compose.yml configuring is set up for remote production and
staging (sandbox) use, so for normal use all that’s necessary is
$ docker-compose up -d
For HTTPS support, read on.
This is necessary to support HTTPS (which is required for MTurk external HITs).
The certificates necessary for SSL are written into the certbot-etc and
certbot-var volumes by certbot. This is accomplished using a separate docker compose
file, which goes on top of the main one like so:
$ docker-compose -f docker-compose.yml -f docker-compose.certbot.yml up certbot
On its own, this will (re-)create the necessary services (webserver) and run certbot. This needs to be done every time the certificate needs to be renewed.
The DOMAIN environment variable is used to set the domain name for
letsencrypt, so make sure the setting in .env matches the actual domain name
you need a certificate for.
Second, make sure the email address you want certificate expiration reminders to
go to is listed in docker-compose.certbot.yml file.
Additional steps are needed for initial certificate acquisition.
First, because there’s no certificates in place, you need to (temporarily)
adjust the nginx configuration (in nginx-conf/nginx.conf). Right now this is
handled awkwardly: you have to manually uncomment the bit in the first server
block (to allow access to files over HTTP), and comment out the entire second
server block (which will block nginx from starting because of the missing
certificates). Then run certbot as before:
$ docker-compose -f docker-compose.yml -f docker-compose.certbot.yml up certbot
Second, once the certificates are in place, the diffie helman parameter needs to be generated, like
$ mkdir dhparam $ sudo openssl dhparam -out "$PWD/dhparam/dhparam-2048.pem" 2048
Every 90 days you must renew the certificates; LetsEncrypt will email you a
reminder at the email address in the dockerfile. Renewal is simple matter of
running certbot again and re-starting the webserver to load the new
certificates (the --no-deps flag keeps docker-compose from recreating all the
other containers, which isn’t necessary when the webserver is already running)
$ docker-compose -f docker-compose.yml -f docker-compose.certbot.yml up --no-deps certbot
Then the new certificates need to be loaded into nginx. You can either re-start
the whole container using up --force-recreate --no-deps webserver or (slighly
more gracefully) send a SIGHUP signal to the nginx process with
$ docker-compose exec webserver nginx -s reload
Which will validate and load the new certificates and restart any worker processes as necessary.
The lists of conditions and number of assignments to put in each condition is
read from the lists database, which stores documents like this:
[
{
"list_id": 1,
"experiment": "a-nice-experiment",
"condition": "good-condition",
"count": 10
},
{
"list_id": 2,
"experiment": "a-nice-experiment",
"condition": "okay-condition",
"count": 5
}
]Note that when updating lists, the experiment is added automatically based on the URL, and in fact any values specified directly in the JSON will be ignored.
count gives the desired number of assignments for this list. Anything
stored under condition will be stored on the session returned to the
client.
PUT http://localhost:8080/a-nice-experiment/lists
Content-Type: application/json
[
{
"list_id": 1,
"condition": "nothign",
"count": 11
},
{
"list_id": 1,
"condition": "nothing",
"count": 11
},
{
"list_id": 2,
"condition": "something",
"count": 10
}
]
GET http://localhost:8080/a-nice-experiment/lists
GET http://localhost:8080/a-nice-experiment/lists?condition=nothign
Only exposed in development mode (when NODE_ENV != "production").
DELETE http://localhost:8080/a-nice-experiment/lists?condition=nothign
We use PUSH to request a new session. If a matching session is not found in the database, a new session is created. The criterion for matching is having the same workerId and experiment.
The body of the PUSH request has the metadata about the session to store (workerId is mandatory, others are optional).
POST http://localhost:8080/a-nice-experiment/session
Content-Type: application/json
{
"assignmendId": 1233445,
"workerId": "dave",
"hello": "world"
}
The session_id is needed for future requests (to get information on a
specific session and to update the status of a session)
During preview, no workerId is assigned, but assignmentId is set to
ASSIGNMENT_ID_NOT_AVAILABLE. In this case, no record is created and
condition is set to preview:
POST http://localhost:8080/a-nice-experiment/session
Content-Type: application/json
{
"assignmentId": "ASSIGNMENT_ID_NOT_AVAILABLE"
}
This is used by the client to update the server on progress of the experiment, or in case the session is abandoned by closing the window. The body of the request is set as the new status (parsed as plain text).
POST http://localhost:8080/a-nice-experiment/session/680c34d8-a2b4-4f53-be82-fb395a9ef884/status
Content-Type: text/plain
okay
GET http://localhost:8080/a-nice-experiment/session/
(This uses the ID returned in the POST call above)
GET http://localhost:8080/a-nice-experiment/session/680c34d8-a2b4-4f53-be82-fb395a9ef884/
The client should send recorded data to the serer using a POST request to
the experiments data endpoint:
POST