Predictive performance in Smart Homework application [Paper iteration]

[henrishi]

@susanathey

Following in the applied direction for this paper. I investigated the predictive performance of the model used in the Smart Homework application (two-parameter model).

This application updated the model parameter on a weekly basis. This allows us to define a different test set than the previous sections. We're now using data from production in the week after each training as the test set, which better mirrors how other real-world applications would work.

This is the updated version of the paper, the new section is from pages 23 - 29. I'm primarily looking for some high-level feedback on whether this is a useful addition to keep in the paper before trying to refine it further.

Shi_bayesian_student_model_20211208.pdf

Next steps:

1. Smart Homework used the two-parameter model, but I replicate the training scheme with other models to compare predictive performance using the same test set.
2. Explore "elasticity" between different knowledge points using the best performing factorization model

Also tagging @shanjukta-nath on this. She has been graciously meeting with me to discuss ideas and results for the past months.

[susanathey]

On right track. Comments:

• abstracts shouldn't talk about future work
• avoid non-academic jargon like "application was powered by"
• clarify "same set of recommendations through the week" ambiguous whether the recommendation system trained model was the same, or there were a fixed set of stories, or what exactly that means-it becomes clear later but be clear from the beginning
• Explain more why coverage is a problem and how generalizable coverage problems are, would you expect a similar situation in other applications, and use language to clarify the specific nature of this application versus more generalizable findings