3. Topic Modeling

[tmozgach]

You could start with topic modeling first. Dr. Yang was using LDA with five or six methods like SVM etc. I think you can easily google some guides to do it with R or python. This is quite mature now.

[tmozgach]

Not important:
An explanation and example of topic modeling with R

http://brazenly.blogspot.ca/2016/05/r-text-classification-and-topic_1.html

[tmozgach]

Main guide that I followed:
https://github.com/abhijeet3922/Topic-Modelling-on-Wiki-corpus

[tmozgach]

DONE:
It has visualization stuff:
https://github.com/shichaoji/easyLDA

[tmozgach]

Tips and tuning parameters:
https://markroxor.github.io/gensim/static/notebooks/lda_training_tips.html

[tmozgach]

Based on the following:
convergence_liklihood.pdf
We need 900-1000 iterations.

The graph above I produced using:
How to monitor convergence of Gensim LDA model?
https://stackoverflow.com/questions/37570696/how-to-monitor-convergence-of-gensim-lda-model

Select iterations and passes parameters of LDA model:

I suggest the following way to choose iterations and passes. First, enable logging (as described in many Gensim tutorials), and set eval_every = 1 in LdaModel. When training the model look for a line in the log that looks something like this:

2016-06-21 15:40:06,753 - gensim.models.ldamodel - DEBUG - 68/1566 documents converged within 400 iterations

If you set passes = 20 you will see this line 20 times. Make sure that by the final passes, most of the documents have converged. So you want to choose both passes and iterations to be high enough for this to happen.

[tmozgach]

@neowangkkk
First result from Cedar, just for Title and Post (no COMMENTS), 10 topics, took 8 hours, download all files and run HTML file:
Uploading cedar_data_10topics.zip…
How to interpret visualization (last section):
https://nlpforhackers.io/topic-modeling/

[tmozgach]

@neowangkkk
20 Topics:
20Topics.zip

Working on 40 topics...

[tmozgach]

@neowangkkk
40 Topics:
40Topics.zip

Working on 30

[tmozgach]

@neowangkkk
30Topics.zip

[tmozgach]

@neowangkkk
10, 20, 30, 40 Topics for Titles, Posts and their COMMENTS (10to40TopicsForALL.zip):
https://drive.google.com/drive/folders/1S8iAnnyjH4NxZpXmkX0ppN9c_Hm6XJLQ?usp=sharing

[neowangkkk]

R Code for importing data

set your working driectory

setwd("/Users/Tao/Dropbox/Data/Reddit_data/")

install tydyverse package

install.packages("tidyverse")
library(tidyverse)

read file using read_csv

data<-read_csv(file = "data_full.csv")

check summary stats

summary(data)

[tmozgach]

@neowangkkk
New result on your friend's data:
https://drive.google.com/drive/folders/1S8iAnnyjH4NxZpXmkX0ppN9c_Hm6XJLQ?usp=sharing
NewAllData.zip

[neowangkkk]

##can you please replace these:
ideas = idea
product = products
cars = car
entrepreneurs = entrepreneur

##can you please remove these:
doesnt
didnt
theyre
isnt
business
work
start

[tmozgach]

@neowangkkk
https://drive.google.com/drive/folders/1S8iAnnyjH4NxZpXmkX0ppN9c_Hm6XJLQ
10-30.zip

I am not sure that it was good idea to exclude a lot of words. It seems that it influences and change topics. I got rid of the following words:
(It is ok that some of them are repeated, it doesn't influence on anything.)

awesome cant though theyre yeah around try enough keep way start work busines isnt theyre didnt doesnt i\'ve you\'re that\'s what\'s let\'s i\'d you\'ll aren\'t \"the i\'ll we\'re wont 009 don\'t it\'s nbsp i\'m get make like would want dont\' use one need know good take thank say also see really could much something ive well give first even great things come thats sure help youre lot someone ask best many question etc better still put might actually let love may tell every maybe always never probably anything cant\' doesnt\' ill already able anyone since another theres everything without didn\'t isn\'t youll\' per else ive get would like want hey might may without also make want put etc actually else far definitely youll\' didnt\' isnt\' theres since able maybe without may suggestedsort never isredditmediadomain userreports far appreciate next think know need look please one null take dont dont\' want\' could able ask well best someone sure lot thank also anyone really something give years use make all ago people know many call include part find become