1604WQM

1604 WQMProp Model Tuning

Baseline based on preliminary testing: cnn model with cnnsiamese=False. On enttok, trainmodel MRR 0.55, val MRR 0.538, but this was on an older dataset version.

Model comparison using dataset without entity replacement:

Model	trainAllMRR	devMRR	testMAP	testMRR	settings
attn1511	0.894100	0.732434	0.701140	0.728824	inp_e_dropout=0 dropout=0
	±0.024455	±0.010177	±0.007767	±0.004660
cnn	0.834471	0.689337	0.664300	0.694464	inp_e_dropout=0 dropout=0 cnnsiamese=False
	±0.027208	±0.017536	±0.021354	±0.018800
rnn	0.790574	0.676416	0.653450	0.682013	inp_e_dropout=0 dropout=0
	±0.121752	±0.065309	±0.067856	±0.065095
cnn	0.656457	0.597415	0.562771	0.599110	inp_e_dropout=0 dropout=0
	±0.057810	±0.030885	±0.034142	±0.032238
rnncnn	0.584183	0.546255	0.517417	0.555994	inp_e_dropout=0 dropout=0
	±0.077842	±0.048224	±0.051735	±0.048143
avg	0.497539	0.495806	0.456750	0.503249	inp_e_dropout=0 dropout=0 inp_w_dropout=1/3 deep=2 pact='relu'
	±0.008361	±0.006968	±0.006505	±0.007881
avg	0.508851	0.493943	0.462075	0.506396	inp_e_dropout=0 dropout=0
	±0.021817	±0.018339	±0.013255	±0.015074

Baseline performance:

4x R_wqme_2cnnS - 0.490903 (95% [0.476349, 0.505457]):

11173576.arien.ics.muni.cz.R_wqme_2cnnS etc.
[0.481998, 0.481802, 0.497626, 0.502186, ]

Baseline performance on non-enttok:

3x R_wqm_2cnnS - 0.469754 (95% [0.448621, 0.490888]):

11173578.arien.ics.muni.cz.R_wqm_2cnnS etc.
[0.466466, 0.481421, 0.461376, ]

Narrower filters - cdim={1:0.5,2:0.5,3:0.5}:

4x R_wqme_2cnnS_c121212 - 0.504224 (95% [0.481014, 0.527434]):

11173579.arien.ics.muni.cz.R_wqme_2cnnS_c121212 etc.
[0.515580, 0.481513, 0.501481, 0.518323, ]

attn1511:

4x R_wqme_2a51 - 0.508561 (95% [0.477719, 0.539403]):

11172403.arien.ics.muni.cz.R_wqme_2a51 etc.
[0.485712, 0.523578, 0.493444, 0.531510, ]

Smaller batch (80):

4x R_wqme_2cnnS_d0_bs80 - 0.551855 (95% [0.535277, 0.568433]):

11192570.arien.ics.muni.cz.R_wqme_2cnnS_d0_bs80 etc.
[0.549074, 0.542457, 0.546456, 0.569433, ]

Without dropout inp_e_dropout=0 dropout=0:

4x R_wqme_2cnnS_d0 - 0.619681 (95% [0.553024, 0.686339]):

11173581.arien.ics.muni.cz.R_wqme_2cnnS_d0 etc.
[0.594088, 0.675736, 0.641587, 0.567314, ]

Without dropout, epoch count = 24:

4x R_wqme_2cnnS_d0_nb24 - 0.682384 (95% [0.641784, 0.722983]):

11192526.arien.ics.muni.cz.R_wqme_2cnnS_d0_nb24 etc.
[0.699226, 0.656910, 0.658114, 0.715284, ]

Without dropout, epoch count = 32, expoch fract = 0.5:

3x R_wqme_2cnnS_d0_ef05_nb32 - 0.708962 (95% [0.672337, 0.745588]):

11194607.arien.ics.muni.cz.R_wqme_2cnnS_d0_ef05_nb32 etc.
[0.719661, 0.688114, 0.719112, ]

RNN, without dropout, epoch fract = 0.5:

4x R_wqme_2rnn_d0_ef05 - 0.708217 (95% [0.658453, 0.757982]):

11215149.arien.ics.muni.cz.R_wqme_2rnn_d0_ef05 etc.
[0.654794, 0.729644, 0.730822, 0.717610, ]

TODO:

• Try with balance_class=True
• Try with loss='binary_crossentropy'
• Try with dot-product - ptscorer=B.dot_ptscorer.