Reporting on tweets captured per "regex"

[ghost]

I think this can be an opportunity to tune the bot.

Would it be possible to create an analysis of the regex's and the number of tweets they captures? With this information we can identify the regex's that are not returning as many tweets, and improve them if they can be made better.

[squeek502]

This is actually already in place. Every retweet gets both the full tweet data and matches info appended as .json to a log file (matches.json or matches.dry-run.json if the --dry-run flag is set).

Example line of a log file ran through a JSON formatter (see the matches key at the bottom):

{
  "tweet":{
    "created_at":"Tue Nov 17 05:31:03 +0000 2015",
    "id":666488701837414400,
    "id_str":"666488701837414400",
    "text":"I really want to go vegan. I know it is better for me if I do. It's just the transitioning is the problem. Its gonna be awhile.😐😉🍌🍌🍊🍓🍉",
    "source":"<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>",
    "truncated":false,
    "in_reply_to_status_id":null,
    "in_reply_to_status_id_str":null,
    "in_reply_to_user_id":null,
    "in_reply_to_user_id_str":null,
    "in_reply_to_screen_name":null,
    "user":{
      // removed user data
    },
    "geo":null,
    "coordinates":null,
    "place":{
      // removed location data
    },
    "contributors":null,
    "is_quote_status":false,
    "retweet_count":0,
    "favorite_count":0,
    "entities":{
      "hashtags":[

      ],
      "urls":[

      ],
      "user_mentions":[

      ],
      "symbols":[

      ]
    },
    "favorited":false,
    "retweeted":false,
    "filter_level":"low",
    "lang":"en",
    "timestamp_ms":"1447738263936"
  },
  "matches":[
    {
      "match":"I really want to go vegan",
      "index":0,
      "filter":"/i([\\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\\s.]+)?(want([\\s.]+)to|wanna|would([\\s.]+)like([\\s.]+)to)([\\s.]+)(be(([\\s.]+)a)?|become(([\\s.]+)a)?|go)([\\s.]+)#?vegan/gi",
      "filterList":"english"
    }
  ]
}

Relevant code: https://github.com/plorry/VegAssist/blob/master/vegassist.js#L28-L36

[squeek502]

Re-opening as I realized that this is probably more about the actual analysis being done than the capability for it. To do the analysis, we'd need to pull the log from the VegAssist instance and then write a script to parse the JSON and analyze it.

[squeek502]

Ok, I did done it.

Match counts by filter since logging started:

english: 1498 matches
 -> 1243 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(want([\s.]+)to|wanna|would([\s.]+)like([\s.]+)to)([\s.]+)(be(([\s.]+)a)?|become(([\s.]+)a)?|go)([\s.]+)#?vegan/gi
 -> 117 matches: /i('|([\s.]+)a)?m([\s.]+)(considering|thinking([\s.]+)(about|of)|mulling([\s.]+)over)([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 59 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(want([\s.]+)to|wanna|would([\s.]+)like([\s.]+)to|should)([\s.]+)(try)([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 30 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(wish)([\s.]+)i([\s.]+)(was|were)([\s.]+)#?vegan/gi
 -> 25 matches: /^(i([\s.]+)think)?([\s.]+)?i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(should)([\s.]+)(go|be)([\s.]+)#?vegan/gi
 -> 23 matches: /help([\s.]+)me([\s.]+)(be(([\s.]+)a)?|become(([\s.]+)a)?|go)([\s.]+)#?vegan/gi
 -> 1 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(will|do)?([\s.]+)?(need|want)([\s.]+)help([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?|staying(([\s.]+)a)?)([\s.]+)#?vegan/gi

spanish: 110 matches
 -> 81 matches: quiero ser vegana,/(no|not|nay) *$/i,
 -> 15 matches: quiero hacerme vegana,/(no|not|nay) *$/i,
 -> 8 matches: quiero ser vegana
 -> 3 matches: quisiera ser vegana,/(no|not|nay) *$/i,
 -> 2 matches: quiero hacerme vegana
 -> 1 matches: quisiera ser vegana

portuguese: 33 matches
 -> 27 matches: /quero (tornar\-me|ser) vegan/gi,/(não|nao|no|not|non|nay|nope|nenhum(a)?) *$/i,
 -> 4 matches: /gosta(va|ria) de (ser|me tornar|tornar\-me) vegan/gi,/(não|nao|no|not|non|nay|nope|nenhum(a)?) *$/i,
 -> 2 matches: /quero (tornar\-me|ser) vegan/gi

And, for fun, the specific text that matched by the number of times it matched (only for texts that matched >= 5 times):

"i want to go vegan": 217 matches
"i wanna go vegan": 211 matches
"i want to be vegan": 176 matches
"i wanna be vegan": 126 matches
"i want to be a vegan": 90 matches
"quiero ser vegana": 89 matches
"i want to become vegan": 74 matches
"i really want to go vegan": 61 matches
"i wanna be a vegan": 56 matches
"i really wanna go vegan": 47 matches
"i want to become a vegan": 38 matches
"i wanna become vegan": 31 matches
"quero ser vegan": 28 matches
"i wish i was vegan": 26 matches
"i really want to be vegan": 25 matches
"i wanna become a vegan": 24 matches
"i'm thinking about going vegan": 22 matches
"i'm considering going vegan": 21 matches
"i should go vegan": 19 matches
"i'm thinking of going vegan": 18 matches
"quiero hacerme vegana": 17 matches
"i really want to become a vegan": 9 matches
"i wanna try going vegan": 9 matches
"i wanna try being vegan": 9 matches
"i want to try going vegan": 9 matches
"i really want to become vegan": 9 matches
"help me go vegan": 8 matches
"i really wanna be vegan": 7 matches
"i'm thinking about becoming vegan": 7 matches
"i really wanna become vegan": 6 matches
"help me become vegan": 6 matches
"i really want to be a vegan": 6 matches
"im considering going vegan": 6 matches
"i'm considering becoming a vegan": 6 matches
"i'm thinking about becoming a vegan": 6 matches
"i want to try being vegan": 5 matches
"i really want to try going vegan": 5 matches

EDIT: Made matching text counts case-insensitive.

[squeek502]

Re-ran my script on the latest log just for fun. Here are the results:

By filter and rule:

english: 18009 matches
 -> 14201 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(want([\s.]+)to|wanna|would([\s.]+)like([\s.]+)to)([\s.]+)(be(([\s.]+)a)?|become(([\s.]+)a)?|go)([\s.]+)#?vegan/gi
 -> 1508 matches: /i('|([\s.]+)a)?m([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(considering|thinking([\s.]+)(about|of)|mulling([\s.]+)over|planning([\s.]+)on)([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 726 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(want([\s.]+)to|wanna|would([\s.]+)like([\s.]+)to|should)([\s.]+)(try)([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 323 matches: /i('|([\s.]+)a)?m([\s.]+)(considering|thinking([\s.]+)(about|of)|mulling([\s.]+)over)([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 280 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(wish)([\s.]+)i([\s.]+)(was|were)([\s.]+)#?vegan/gi
 -> 280 matches: /help([\s.]+)me([\s.]+)(be(([\s.]+)a)?|become(([\s.]+)a)?|go)([\s.]+)#?vegan/gi
 -> 240 matches: /^(i([\s.]+)think)?([\s.]+)?i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(should)([\s.]+)(go|be)([\s.]+)#?vegan/gi
 -> 233 matches: /(being|staying|going)([\s.]+)?#?vegan('|([\s.]+)i)s([\s.]+)(too)?([\s.]+)?(hard|tough|difficult|rough)/gi
 -> 96 matches: /it(('|([\s.]+)i)s)?([\s.]+)?(too)?([\s.]+)?(hard|tough|difficult|rough)([\s.]+)(to([\s.]+)be(([\s.]+)a)?|being(([\s.]+)a)?|staying(([\s.]+)a)?|going)([\s.]+)#?vegan/gi
 -> 42 matches: /i([\s.]+)(can|could)([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(see|picture|imagine)(([\s.]+)myself)?([\s.]+)(going|being(([\s.]+)a)?|becoming(([\s.]+)a)?)([\s.]+)#?vegan/gi
 -> 40 matches: /(being)?([\s.]+)?#?vegan('|([\s.]+)i)s([\s.]+)(too)?([\s.]+)?(hard|tough|difficult|rough)/gi
 -> 23 matches: /((it)?('|([\s.]+)i)s)?([\s.]+)?(too)?([\s.]+)?(hard|tough|difficult|rough)([\s.]+)(to([\s.]+)be|being|staying)([\s.]+)#?vegan/gi
 -> 17 matches: /i([\s.]+)(really|totally|probably|defin[ia]tely|absolutely|actually|certainly|literally|legitimately|genuinely|honestly|truly|undoubtedly|unquestionably)?([\s.]+)?(will|do)?([\s.]+)?(need|want)([\s.]+)help([\s.]+)(going|becoming(([\s.]+)a)?|being(([\s.]+)a)?|staying(([\s.]+)a)?)([\s.]+)#?vegan/gi

spanish: 1452 matches
 -> 742 matches: quiero ser vegana,/(no|not|nay) *$/i,
 -> 512 matches: /(quisiera|quiero|debería|debe|pensando([\s.]+).*)([\s.]+)(ser|hacerme|hacerse|convertirse([\s.]+)en)([\s.]+)?(un|una)?([\s.]+)vegan(a|o)/gi,/(no) *$/i,
 -> 104 matches: quiero hacerme vegana,/(no|not|nay) *$/i,
 -> 78 matches: quisiera ser vegana,/(no|not|nay) *$/i,
 -> 8 matches: quiero ser vegana
 -> 5 matches: quisiera hacerme vegana,/(no|not|nay) *$/i,
 -> 2 matches: quiero hacerme vegana
 -> 1 matches: quisiera ser vegana

portuguese: 7494 matches
 -> 3361 matches: /(me)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)?([\s.]+)(a|para|pra)([\s.]+)(ser|virar|me([\s.]+)virar|virar-me|tornar|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 2233 matches: /(eu)?([\s.]+)?(realmente|provavelmente|definitivamente|absolutamente|honestamente|certamente|literalmente|genuinamente|sem([\s.]+)duvida|sem([\s.]+)dúvida|sem([\s.]+)questao|sem([\s.]+)questão)?([\s.]+)?(quero|queria|gostaria|gostaria([\s.]+)de|me([\s.]+)gostaria([\s.]+)de|vou|preciso|preciso([\s.]+)de)([\s.]+)(ser|virar|me([\s.]+)virar|virar-me|tornar|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 1061 matches: /(eu)?([\s.]+)?(realmente|provavelmente|definitivamente|absolutamente|honestamente|certamente|literalmente|genuinamente|sem([\s.]+)duvida|sem([\s.]+)dúvida|sem([\s.]+)questao|sem([\s.]+)questão)?([\s.]+)?(quero|queria|gostaria|gostaria([\s.]+)de|me([\s.]+)gostaria([\s.]+)de|vou|preciso|preciso([\s.]+)de)([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)(o|ao)?([\s.]+)?#?vegan(ismo)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 376 matches: /(me)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)?([\s.]+)(a|para|pra)([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)(o|ao)?([\s.]+)?#?vegan(ismo)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 132 matches: /(eu)?([\s.]+)?(estou|estive)?([\s.]+)?(pensando|considerando)([\s.]+)(sobre|em)?([\s.]+)(ser|virar|me([\s.]+)virar|virar-me|tornar|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 122 matches: /(eu)?([\s.]+)?(estou|estive)?([\s.]+)?(pensando|considerando)([\s.]+)(sobre|em)?([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)(o|ao)?([\s.]+)?#?vegan(ismo)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 117 matches: /(eu)?([\s.]+)?(estou|estive)?([\s.]+)?(pensando|considerando)([\s.]+)(sobre|em)?([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 58 matches: /quero (tornar\-me|ser) vegan/gi,/(não|nao|no|not|non|nay|nope|nenhum(a)?) *$/i,
 -> 8 matches: /gosta(va|ria) de (ser|me tornar|tornar\-me) vegan/gi,/(não|nao|no|not|non|nay|nope|nenhum(a)?) *$/i,
 -> 8 matches: /(eu)?([\s.]+)?(quero|queria|gostaria|gostaria([\s.]+)de|me([\s.]+)gostaria([\s.]+)de|vou|preciso|preciso([\s.]+)de)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)([\s.]+)(a|para|pra)([\s.]+)(ser|virar|me([\s.]+)virar|virar-me|tornar|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 8 matches: /(eu)?([\s.]+)?(preciso|necessito)?([\s.]+)(de)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)([\s.]+)(a|para|pra)([\s.]+)(ser|virar|me([\s.]+)virar|virar-me|tornar|me([\s.]+)tornar)([\s.]+)#?vegan(a|o)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 4 matches: /(eu)?([\s.]+)?(preciso|necessito)?([\s.]+)(de)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)([\s.]+)(a|para|pra)([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)(o|ao)?([\s.]+)?#?vegan(ismo)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 4 matches: /(eu)?([\s.]+)?(quero|queria|gostaria|gostaria([\s.]+)de|me([\s.]+)gostaria([\s.]+)de|vou|preciso|preciso([\s.]+)de)?([\s.]+)?(ajuda|ajude|auxília|auxílio|auxilio|auxilio|ajuda-me|ajude-me)([\s.]+)(a|para|pra)([\s.]+)(adotar|em([\s.]+)aderir|me([\s.]+)aderir|aderir([\s.]+)ae|aderir|virar|tornar-se|se([\s.]+)tornar|tornar-me|me([\s.]+)tornar)([\s.]+)(o|ao)?([\s.]+)?#?vegan(ismo)?/gi,/(não|nao|n|nunca|nem|jeito nenhum) *$/i,
 -> 2 matches: /quero (tornar\-me|ser) vegan/gi

dutch: 14 matches
 -> 8 matches: ik wil vegan zijn
 -> 6 matches: ik wil vegan worden

french: 450 matches
 -> 110 matches: /Je([\s.]+)deviens([\s.]+)(végétalien|vegan)/gi
 -> 101 matches: /je([\s.]+)vais([\s.]+)(devenir|être)([\s.]+)(végétalien|vegan)/gi
 -> 55 matches: /je([\s.]+)vais([\s.]+)(devenir|être)([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 54 matches: /je([\s.]+)deviens([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 37 matches: /je([\s.]+)veux(([\s.]+)bien)?([\s.]+)(devenir|être)([\s.]+)(végétalien|vegan)/gi
 -> 34 matches: /je([\s.]+)veux([\s.]+)(devenir|être)(([\s.]+)totalement)?([\s.]+)(végétalien|vegan)/gi
 -> 20 matches: /je([\s.]+)veux(([\s.]+)bien)?([\s.]+)(devenir|être)([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 19 matches: /je([\s.]+)veux([\s.]+)(devenir|être)(([\s.]+)totalement)?([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 7 matches: /J'aimerais([\s.]+)(devenir|être)([\s.]+)(végétalien|vegan)/gi
 -> 6 matches: /je([\s.]+)crois([\s.]+)que([\s.]+)je([\s.]+)vais([\s.]+)(devenir|être)([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 6 matches: /j'aimerais([\s.]+)(devenir|être)([\s.]+)(végétalien|végétalienne|vegan|végane)/gi
 -> 1 matches: /j'hésite(([\s.]+)de([\s.]+)plus([\s.]+)en([\s.]+)plus)?([\s.]+)à([\s.]+)devenir([\s.]+)(végétalien|végétalienne|vegan|végane)/gi

german: 23 matches
 -> 13 matches: /ich([\s.]+)(will|möchte)(([\s.]+)endlich|([\s.]+)auch|([\s.]+)gerne|([\s.]+)bald)?([\s.]+)vegan([\s.]+)(werden|sein|leben|essen)/gi
 -> 4 matches: /ich([\s.]+)(will|möchte)([\s.]+)wieder([\s.]+)vegan([\s.]+)(sein|werden|leben|essen)/gi
 -> 3 matches: /ich([\s.]+)(will|möchte|sollte)([\s.]+)mich([\s.]+)vegan([\s.]+)ernähren/gi
 -> 2 matches: /ich([\s.]+)würde(([\s.]+)ja)?([\s.]+)gerne([\s.]+)vegan([\s.]+)(werden|sein|leben|essen)/gi
 -> 1 matches: /ich([\s.]+)würde([\s.]+)mich(([\s.]+)eigentlich)?([\s.]+)gerne([\s.]+)vegan([\s.]+)ernähren/gi

italian: 12 matches
 -> 12 matches: /(voglio|vorrei|dovrei|penso([\s.]+).*|pensando([\s.]+).*)([\s.]+)(diventare|essere)([\s.]+)?(un|una)?([\s.]+)vegan(a|o)?/gi,/(non) *$/i,

handle: 10 matches
 -> 10 matches: @vegassist

Matched phrases with >= 100 matches:

"i wanna go vegan": 2872 matches
"i want to go vegan": 2620 matches
"i want to be vegan": 1827 matches
"i wanna be vegan": 1559 matches
"quiero ser vegana": 983 matches
" a ser vegana": 837 matches
"i want to be a vegan": 727 matches
"i really want to go vegan": 702 matches
"i wanna be a vegan": 682 matches
" para ser vegano": 651 matches
"i want to become vegan": 612 matches
" a ser vegano": 611 matches
"i really wanna go vegan": 465 matches
" vou virar vegan": 391 matches
"i wanna become vegan": 362 matches
"vou virar vegan": 359 matches
"i'm thinking about going vegan": 347 matches
"i want to become a vegan": 344 matches
"queria ser vegana": 300 matches
"i really want to be vegan": 252 matches
"i'm considering going vegan": 250 matches
"i wish i was vegan": 250 matches
" para ser vegana": 236 matches
"i'm thinking of going vegan": 194 matches
" pra ser vegano": 194 matches
"i wanna become a vegan": 191 matches
" vou virar vegana": 188 matches
"i should go vegan": 169 matches
"vou virar vegana": 163 matches
"je deviens vegan": 156 matches
" a ser vegan": 156 matches
" pra ser vegana": 151 matches
"je vais devenir vegan": 148 matches
"i really want to become vegan": 145 matches
"being vegan is hard": 142 matches
"quiero hacerme vegana": 140 matches
"i really wanna be vegan": 131 matches
"i wanna try going vegan": 122 matches
" pra virar vegan": 120 matches
"help me go vegan": 113 matches
"i wanna try being vegan": 113 matches
"quisiera ser vegana": 102 matches
" a virar vegan": 102 matches