Skip to content

3b. Initial Proposal for ALP301 Class Project

Jump to bottom
Molly Offer-Westort edited this page Mar 8, 2021 · 1 revision

Last updated May 7, 2020

Purpose: This page describes the current project design, and addresses current design questions and challenges.

Overview:

Based on discussions with IPA Philippines staff, awareness of disease symptoms and paths of transmission are currently at good levels in the Philippines. People take the pandemic seriously, and are responsive to the regional community quarantines. However, economic and psychological stresses, combined with isolation, have resulted in increased mental health challenges. The project will focus on spreading awareness of and increasing use of remote access mental health resources (phone hotline, online resources).

Objectives:

  • Demonstrate proof of concept for using Facebook advertisements as a data collection method. We may wish to stratify sampling by geographics/demographics (age, education, gender, etc.) and benchmark our results to census distributions.
  • Collect information on the state of COVID-19 awareness/response, particularly as related to mental health in the Philippines through survey questions.
  • Implement experimental interventions focusing on behavioral outcomes
  • Maximize sharing/engagement of informational posts on Facebook, also access to informational resources

The data collected and measured responses to experimental treatments may inform research conducted by IPA in the Philippines.

Recruitment:

Outreach will be conducted through targeted Facebook advertisements, linked to a project-specific Facebook page. Advertisements run through Facebook and Instagram will direct people to Facebook Messenger, where they will interact with a pre-programmed chatbot to complete a survey. Integrating the survey experience on Facebook services means that respondents using the "Free Basics" mobile version of Facebook's services will not be charged data usage fees for participation. (Although note that the "Free Basics" versions of the Messenger and Facebook apps do no allow attachment of images or videos; this should be accounted for in treatment conditions.)

Currently, we are able to achieve ~1300 responses in a single day, with recruitment stratified across four groups of geographic regions with approximately equally balanced population. Further stratification and a larger sample will likely slow the rate of data collection.

Approximately 90% of the sample pilot sample accessed the recruitment advertisements on their phones, and so we will ensure that the survey is appropriately designed for mobile delivery.

Advertising strategy:

Advertisements will optimize for "conversions", here, completed surveys.

Randomization may occur either within the chatbot or at the advertisement level.

  • Individual level random assignment may be preferable if we would like to fix assignment probabilities as the individual level. This would be useful if we intend to develop this into an adaptive study.

  • Facebook Experiments tools for advertisers facilitate A/B testing with balanced assignment across multiple treatment arms or ad campaigns. Using this tool, we can have different advertisements link to chatbots with different versions of treatment. A benefit of this approach is that we can cluster treatment assignment within geographic units, accounting for the possibility of spillover. A challenge of this approach is that it will be more difficult to control treatment assignment probabilities conditional on consent.

Question: what is the sample over which we're determining a treatment effect? People who consent and opt-in to the study? Or people who are reached by the advertisements? For individual level random assignment, assignment probability stays fixed regardless. For advertisement level random assignment, different click-through and survey completion rates across advertisements mean that choice of sample may result in different treatment assignment probabilities, and this could introduce potential for selection bias.

As the population using Facebook will skew educated, wealthy, and urban, we may also sub-target advertising based on geographic location, gender, education, and income, if we wish to ensure that the survey matches to the census population distribution. Smaller cells will, however, be more expensive and will take more time to fill.

Interventions:

Focusing on a viral campaign, we may focus on treatments that encourage subjects to share a post, tag friends, and ask others to do the same.

  • Identifiable victim (similar to IPA Ghana intervention): encourage subjects think about someone who is close to them who may be isolated or going through a difficult time, and could use some help.
  • Pledge: Before outcome measurement, ask subject if they will pledge to take action.
  • Identity: Prime people to think of themselves as someone others look to for support, reliable source of information.
  • Control: No additional messaging

**Potential factorial design proposed by students: **randomize resource SOURCE as well as STYLE presented with informational link:

  • Source: {Federal Gov’t, Local Gov’t, NGO, Mental health service professional} (Reference: WaPo Monkey Cage Blog article)
  • Style:  {Link, infographic, video, etc}

Challenges:

  • As we are dealing with mental health challenges, we should ensure that if anyone gives responses about self-harm in the survey, they are exited from the chatbot and directed to a hotline or other resources. We may also want to avoid any type of open-response questions.

  • We will likely want to translate advertisements, posts, and survey questions into Tagalog to improve reach and coverage.

  • We may want to connect with the mental health/health team at Facebook.

Outcomes:

  • Information seeking: if respondents agree they would like more information about mental health, and click-through informational links;
  • Social media behavior: if respondents like/share on their timeline or tag [#] friends in a post we show to them;
  • Reminders: offer to provide multiple reminders to take action; subject can say how many reminders they would like, and these can be delivered automatically over messenger, outcomes is [#] reminders;
  • Snowball: invite [#] friends to participate in study; have pre-prepared message to send to friends

Challenges:

  • We will have to work with chatfuel to ensure we can implement all of the behavioral measures, such as having pre-prepared messages.
Clone this wiki locally