Train on entire abstracts or single sentences?

[Larsdegroot]

Hey all,

I'm trying to create a pipeline for relationship extraction to extract information out of scientific abstracts. i'm currently annotating entire abstracts at once because it makes the annotation process easier. However i was wondering if this would effect the training results somewhat, in comparison to separating the abstracts into sentences?

[polm]

Do you have any cases where you're finding relations between items in different sentences? If so you should keep the whole abstracts, as there's no way to find relations between documents.

If that's not the case, then it's hard to say whether it would make a significant difference. Any change like that will definitely have an effect on the training results, but it might not have a major influence on accuracy either way. In situations like this, where you're unsure if approach X or Y is better, it's often better to try both if feasible.

[Larsdegroot]

thank you for the reply! there are some instances of relations across sentences boundaries but i'm unsure if we'll be using those. It's definitely possible for me to try both ways so i'll be sure to do that and update here on what the result is.

[Larsdegroot]

I've done the test on a test set of 110 abstracts, here are the results:

Training on sentences:

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00    164.63    1.22    0.94    1.75    0.01
  0     200       1209.34  13524.86   35.85   42.06   31.24    0.36
  0     400        160.48   5837.57   41.45   48.88   35.98    0.41
  0     600        328.77   3926.89   42.31   52.83   35.29    0.42
  1     800        293.75   2870.99   45.21   51.45   40.32    0.45
  1    1000        269.79   1785.50   44.40   52.82   38.29    0.44
  1    1200        332.47   1726.17   45.12   53.74   38.88    0.45
  1    1400        223.78   1399.85   44.78   52.61   38.98    0.45
  2    1600        276.55   1159.65   47.33   55.36   41.34    0.47
  2    1800        295.30   1096.76   45.03   48.66   41.89    0.45
  2    2000        230.19    831.81   43.28   53.54   36.32    0.43
  2    2200        309.62    773.87   45.27   56.95   37.56    0.45
  3    2400        285.03    709.12   45.70   53.20   40.05    0.46
  3    2600        293.57    570.60   46.86   50.86   43.44    0.47
  3    2800        424.54    674.06   46.49   53.38   41.17    0.46
  3    3000        345.02    503.43   44.58   54.25   37.83    0.45
  4    3200        436.32    525.94   44.79   51.90   39.39    0.45
✔ Saved pipeline to output directory
training\model-last

Training on abstracts:

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00    160.90    5.53    3.24   18.72    0.06
  2     200        923.71  12341.18   38.01   48.07   31.43    0.38
  4     400        454.34   6996.97   40.10   46.96   34.99    0.40
  6     600        232.14   4346.14   43.39   49.25   38.78    0.43
  8     800        236.21   2936.00   43.87   53.71   37.07    0.44
 11    1000        244.89   2168.72   45.79   54.28   39.60    0.46
 13    1200        237.29   1646.92   45.91   54.91   39.45    0.46
 15    1400        273.30   1414.07   45.56   46.76   44.43    0.46
 17    1600        271.54   1275.98   45.50   56.44   38.11    0.45
 19    1800        336.06   1096.17   45.32   50.32   41.23    0.45
 22    2000        322.00    893.06   44.36   53.34   37.96    0.44
 24    2200        435.57    970.46   42.20   52.31   35.36    0.42
 26    2400        325.77    731.89   45.34   50.36   41.23    0.45
 28    2600        376.55    690.67   44.35   55.01   37.15    0.44
 31    2800        387.69    567.58   45.08   56.63   37.44    0.45
✔ Saved pipeline to output directory
training\model-last

To me this seems like no significant difference. however this was with a relatively small dataset so i'll rerun this experiment once more once i have more training data.