Replicated MS MARCO passage ranking, doc ranking, and doc2query (MS M…

…ARCO + CAR) experiments (castorini#1229)

docs/experiments-doc2query.md

@@ -1,6 +1,6 @@
 # Anserini: doc2query Experiments
 
-This page describes how to replicate the doc2query document expansion experiments in following paper:
+This page describes how to replicate the doc2query document expansion experiments in the following paper:
 
 + Rodrigo Nogueira, Wei Yang, Jimmy Lin, Kyunghyun Cho. [Document Expansion by Query Prediction.](https://arxiv.org/abs/1904.08375) _arxiv:1904.08375_
 
@@ -20,17 +20,17 @@ Before going through this guide, it is recommended that you [replicate our BM25
 To start, grab the predicted queries:
 
 ```
-wget https://www.dropbox.com/s/709q495d9hohcmh/pred-test_topk10.tar.gz -P msmarco-passage
-tar -xzvf msmarco-passage/pred-test_topk10.tar.gz -C msmarco-passage
+wget https://www.dropbox.com/s/709q495d9hohcmh/pred-test_topk10.tar.gz -P collections/msmarco-passage
+tar -xzvf collections/msmarco-passage/pred-test_topk10.tar.gz -C collections/msmarco-passage
 ```
 
 To confirm, `pred-test_topk10.tar.gz` should have an MD5 checksum of `241608d4d12a0bc595bed2aff0f56ea3`.
 
 Check out the file:
 
 ```
-$ wc msmarco-passage/pred-test_topk10.txt
- 8841823 536446170 2962345659 msmarco-passage/pred-test_topk10.txt
+$ wc collections/msmarco-passage/pred-test_topk10.txt
+  8841823 536425855 2962345659 collections/msmarco-passage/pred-test_topk10.txt
 ```
 
 These are the predicted queries based on our seq2seq model, based on top _k_ sampling with 10 samples for each document in the corpus.
@@ -40,38 +40,38 @@ Now let's create a new document collection by concatenating the predicted querie
 
 ```
 python src/main/python/msmarco/augment_collection_with_predictions.py \
-  --collection_path msmarco-passage/collection.tsv --output_folder msmarco-passage/collection_jsonl_expanded_topk10 \
-  --predictions msmarco-passage/pred-test_topk10.txt --stride 1
+ --collection_path collections/msmarco-passage/collection.tsv --output_folder collections/msmarco-passage/collection_jsonl_expanded_topk10 \
+ --predictions collections/msmarco-passage/pred-test_topk10.txt --stride 1
 ```
 
 We can then reindex the collection:
 
 ```
 sh ./target/appassembler/bin/IndexCollection -collection JsonCollection \
- -generator DefaultLuceneDocumentGenerator -threads 9 -input msmarco-passage/collection_jsonl_expanded_topk10 \
- -index msmarco-passage/lucene-index-msmarco-expanded-topk10 -storePositions -storeDocvectors -storeRaw
+ -generator DefaultLuceneDocumentGenerator -threads 9 -input collections/msmarco-passage/collection_jsonl_expanded_topk10 \
+ -index indexes/msmarco-passage/lucene-index-msmarco-expanded-topk10 -storePositions -storeDocvectors -storeRaw
 ```
 
 And run retrieval (same as above):
 
 ```
-python ./src/main/python/msmarco/retrieve.py --hits 1000 --index msmarco-passage/lucene-index-msmarco-expanded-topk10 \
- --qid_queries msmarco-passage/queries.dev.small.tsv --output msmarco-passage/run.dev.small.expanded-topk10.tsv
+python ./src/main/python/msmarco/retrieve.py --hits 1000 --index indexes/msmarco-passage/lucene-index-msmarco-expanded-topk10 \
+ --qid_queries collections/msmarco-passage/queries.dev.small.tsv --output runs/run.msmarco-passage.dev.small.expanded-topk10.tsv
 ```
 
 Alternatively, we can run the same script implemented in Java, which is a bit faster:
 
 ```
 ./target/appassembler/bin/SearchMsmarco  -hits 1000 -threads 1 \
- -index msmarco-passage/lucene-index-msmarco-expanded-topk10 -qid_queries msmarco-passage/queries.dev.small.tsv \
- -output msmarco-passage/run.dev.small.expanded-topk10.tsv
+ -index indexes/msmarco-passage/lucene-index-msmarco-expanded-topk10 -qid_queries collections/msmarco-passage/queries.dev.small.tsv \
+ -output runs/run.msmarco-passage.dev.small.expanded-topk10.tsv
 ```
 
 Finally, to evaluate:
 
 ```
 python ./src/main/python/msmarco/msmarco_eval.py \
- msmarco-passage/qrels.dev.small.tsv msmarco-passage/run.dev.small.expanded-topk10.tsv
+ collections/msmarco-passage/qrels.dev.small.tsv runs/run.msmarco-passage.dev.small.expanded-topk10.tsv
 ```
 
 The output should be:
@@ -104,13 +104,14 @@ We will now describe how to reproduce the TREC CAR results of our model BM25+doc
 
 To start, download the TREC CAR dataset and the predicted queries:
 ```
-mkdir trec_car
+mkdir collections/trec_car
+mkdir indexes/trec_car
 
-wget http://trec-car.cs.unh.edu/datareleases/v2.0/paragraphCorpus.v2.0.tar.xz -P trec_car
-wget https://storage.googleapis.com/neuralresearcher_data/doc2query/data/aligned5/pred-test_topk10.tar.gz -P trec_car
+wget http://trec-car.cs.unh.edu/datareleases/v2.0/paragraphCorpus.v2.0.tar.xz -P collections/trec_car
+wget https://storage.googleapis.com/neuralresearcher_data/doc2query/data/aligned5/pred-test_topk10.tar.gz -P collections/trec_car
 
-tar -xf trec_car/paragraphCorpus.v2.0.tar.xz -C trec_car
-tar -xf trec_car/pred-test_topk10.tar.gz -C trec_car
+tar -xf collections/trec_car/paragraphCorpus.v2.0.tar.xz -C collections/trec_car
+tar -xf collections/trec_car/pred-test_topk10.tar.gz -C collections/trec_car
 ```
 
 To confirm, `paragraphCorpus.v2.0.tar.xz` should have an MD5 checksum of `a404e9256d763ddcacc3da1e34de466a` and
@@ -123,9 +124,9 @@ Now let's create a new document collection by concatenating the predicted querie
 
 ```
 python src/main/python/trec_car/augment_collection_with_predictions.py \
- --collection_path trec_car/paragraphCorpus/dedup.articles-paragraphs.cbor \
- --output_folder trec_car/collection_jsonl_expanded_topk10 \
- --predictions trec_car/pred-test_topk10.txt --stride 1
+ --collection_path collections/trec_car/paragraphCorpus/dedup.articles-paragraphs.cbor \
+ --output_folder collections/trec_car/collection_jsonl_expanded_topk10 \
+ --predictions collections/trec_car/pred-test_topk10.txt --stride 1
 ```
 
 This augmentation process might take 2-3 hours.
@@ -134,24 +135,24 @@ We can then index the expanded documents:
 
 ```
 sh target/appassembler/bin/IndexCollection -collection JsonCollection \
- -generator DefaultLuceneDocumentGenerator -threads 30 -input trec_car/collection_jsonl_expanded_topk10 \
- -index trec_car/lucene-index.car17v2.0
+ -generator DefaultLuceneDocumentGenerator -threads 30 -input collections/trec_car/collection_jsonl_expanded_topk10 \
+ -index indexes/trec_car/lucene-index.car17v2.0
 ```
 
 And retrieve the test queries:
 
 ```
 sh target/appassembler/bin/SearchCollection -topicreader Car \
- -index trec_car/lucene-index.car17v2.0 \
+ -index indexes/trec_car/lucene-index.car17v2.0 \
  -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt \
- -output trec_car/run.car17v2.0.bm25.topics.car17v2.0.benchmarkY1test.txt -bm25
+ -output runs/run.car17v2.0.bm25.topics.car17v2.0.benchmarkY1test.txt -bm25
 ```
 
 Evaluation is performed with `trec_eval`:
 ```
 eval/trec_eval.9.0.4/trec_eval -c -m map -c -m recip_rank \
  src/main/resources/topics-and-qrels/qrels.car17v2.0.benchmarkY1test.txt \
- trec_car/run.car17v2.0.bm25.topics.car17v2.0.benchmarkY1test.txt
+ runs/run.car17v2.0.bm25.topics.car17v2.0.benchmarkY1test.txt
 ```
 
 With the above commands, you should be able to replicate the following results:
@@ -160,7 +161,7 @@ map                   	all	0.1807
 recip_rank            	all	0.2750
 ```
 
-Note that this MAP is sligtly higher than the arXiv paper (0.178) because we used
+Note that this MAP is slightly higher than the arXiv paper (0.178) because we used
 TREC CAR corpus v2.0 in this experiment instead of corpus v1.5 used in the paper.
 
 ## Replication Log
@@ -169,3 +170,4 @@ TREC CAR corpus v2.0 in this experiment instead of corpus v1.5 used in the paper
 + Results replicated by [@ronakice](https://github.com/ronakice) on 2019-08-13 (commit [`5b29d16`](https://github.com/castorini/anserini/commit/5b29d1654abc5e8a014c2230da990ab2f91fb340))
 + Results replicated by [@edwinzhng](https://github.com/edwinzhng) on 2020-01-08 (commit [`5cc923d`](https://github.com/castorini/anserini/commit/5cc923d5c02777d8b25df32ff2e2a59be5badfdd))
 + Results replicated by [@HangCui0510](https://github.com/HangCui0510) on 2020-04-23 (commit [`0ae567d`](https://github.com/castorini/anserini/commit/0ae567df5c8a70ac211efd958c9ca1ff609ff782))
++ Results replicated by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2020-05-25 (commit [`b6e0367`](https://github.com/castorini/anserini/commit/b6e0367ef4e2b4fce9d81c8397ef1188e35971e7))

docs/experiments-msmarco-doc.md

@@ -138,4 +138,5 @@ As expected, BM25 tuning makes a big difference!
 + Results replicated by [@richard3983](https://github.com/richard3983) on 2020-05-14 (commit [`a65646f`](https://github.com/castorini/anserini/commit/a65646fe203bf5c9c32189a56082d6f4d3bc340d))
 + Results replicated by [@MXueguang](https://github.com/MXueguang) on 2020-05-20 (commit [`3b2751e`](https://github.com/castorini/anserini/commit/3b2751e2d02a9d530e1c3d30b91083faeece8982))
 + Results replicated by [@shaneding](https://github.com/shaneding) on 2020-05-23 (commit [`b6e0367`](https://github.com/castorini/anserini/commit/b6e0367ef4e2b4fce9d81c8397ef1188e35971e7))
-+ Results replicated by [@adamyy](https://github.com/adamyy) on 2020-05-28 (commit [`a1ecfa4`](https://github.com/castorini/anserini/commit/a1ecfa4aa38fb8a0cf41575d47629ba1c69228fb))
++ Results replicated by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2020-05-24 (commit [`b6e0367`](https://github.com/castorini/anserini/commit/b6e0367ef4e2b4fce9d81c8397ef1188e35971e7))
++ Results replicated by [@adamyy](https://github.com/adamyy) on 2020-05-28 (commit [`a1ecfa4`](https://github.com/castorini/anserini/commit/a1ecfa4aa38fb8a0cf41575d47629ba1c69228fb))

docs/experiments-msmarco-passage.md

@@ -167,4 +167,5 @@ Tuned (`k1=0.82`, `b=0.72`) | 0.1875 | 0.1956 | 0.8578
 + Results replicated by [@richard3983](https://github.com/richard3983) on 2020-05-14 (commit [`a65646f`](https://github.com/castorini/anserini/commit/a65646fe203bf5c9c32189a56082d6f4d3bc340d))
 + Results replicated by [@MXueguang](https://github.com/MXueguang) on 2020-05-20 (commit [`3b2751e`](https://github.com/castorini/anserini/commit/3b2751e2d02a9d530e1c3d30b91083faeece8982))
 + Results replicated by [@shaneding](https://github.com/shaneding) on 2020-05-23 (commit [`b6e0367`](https://github.com/castorini/anserini/commit/b6e0367ef4e2b4fce9d81c8397ef1188e35971e7))
-+ Results replicated by [@adamyy](https://github.com/adamyy) on 2020-05-28 (commit [`94893f1`](https://github.com/castorini/anserini/commit/94893f170e047d77c3ef5b8b995d7fbdd13f4298))
++ Results replicated by [@adamyy](https://github.com/adamyy) on 2020-05-28 (commit [`94893f1`](https://github.com/castorini/anserini/commit/94893f170e047d77c3ef5b8b995d7fbdd13f4298))
++ Results replicated by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2020-05-28 (commit [`d55531a`](https://github.com/castorini/anserini/commit/d55531a738d2cf9e14c376d798d2de4bd3020b6b))

docs/regressions-robust04.md

@@ -86,7 +86,7 @@ P30                                     | BM25      | +RM3      | +Ax       | QL
 ## Replication Log
 
 + Results replicated by [@chriskamphuis](https://github.com/chriskamphuis) on 2018-12-18 (commit [`a15235`](https://github.com/castorini/Anserini/commit/a152359435ac6ae694b39f561343bba5eed8fdc9))
-+ Results replicated by [@infinitecold](https://github.com/infinitecold) on 2019-09-08 (commit [`a1892ae`](https://github.com/castorini/anserini/commit/a1892aec726efe55111a7bc501ab0914afab3a30))
++ Results replicated by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2019-09-08 (commit [`a1892ae`](https://github.com/castorini/anserini/commit/a1892aec726efe55111a7bc501ab0914afab3a30))
 + Results replicated by [@JMMackenzie](https://github.com/JMMackenzie) on 2020-01-21 (commit [`f63cd22`](https://github.com/castorini/anserini/commit/f63cd2275fa5a9d4da2d17e5f983a3308e8b50ce))
 + Results replicated by [@nikhilro](https://github.com/nikhilro) on 2020-01-26 (commit [`d5ee069`](https://github.com/castorini/anserini/commit/d5ee069399e6a306d7685bda756c1f19db721156))
 + Results replicated by [@edwinzhng](https://github.com/edwinzhng) on 2020-01-26 (commit [`7b76dfb`](https://github.com/castorini/anserini/commit/7b76dfbea7e0c01a3a5dc13e74f54852c780ec9b))

docs/release-notes/release-notes-v0.7.0.md

@@ -46,7 +46,7 @@ Sorted by number of commits:
 + Zeynep Akkalyoncu Yilmaz ([zeynepakkalyoncu](https://github.com/zeynepakkalyoncu))
 + Weihua Li ([w329li](https://github.com/w329li))
 + Alireza Mirzaeiyan ([amirzaeiyan](https://github.com/amirzaeiyan))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Leonid Boytsov ([searchivarius](https://github.com/searchivarius))
 + Maik Fröbe ([mam10eks](https://github.com/mam10eks))
 + Rodrigo Nogueira ([rodrigonogueira4](https://github.com/rodrigonogueira4))
@@ -95,7 +95,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Petek Yıldız ([ptkyldz](https://github.com/ptkyldz))
 + Maik Fröbe ([mam10eks](https://github.com/mam10eks))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))
 + Andrew Yates ([andrewyates](https://github.com/andrewyates))

docs/release-notes/release-notes-v0.7.1.md

@@ -58,7 +58,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Petek Yıldız ([ptkyldz](https://github.com/ptkyldz))
 + Maik Fröbe ([mam10eks](https://github.com/mam10eks))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))
 + Andrew Yates ([andrewyates](https://github.com/andrewyates))

docs/release-notes/release-notes-v0.7.2.md

@@ -67,7 +67,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Maik Fröbe ([mam10eks](https://github.com/mam10eks))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))

docs/release-notes/release-notes-v0.8.0.md

@@ -72,7 +72,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Maik Fröbe ([mam10eks](https://github.com/mam10eks))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))

docs/release-notes/release-notes-v0.8.1.md

@@ -65,7 +65,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Petek Yıldız ([ptkyldz](https://github.com/ptkyldz))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))

docs/release-notes/release-notes-v0.9.0.md

@@ -72,7 +72,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + Petek Yıldız ([ptkyldz](https://github.com/ptkyldz))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Charles Wu ([charW](https://github.com/charW))
 + Matteo Catena ([catenamatteo](https://github.com/catenamatteo))

docs/release-notes/release-notes-v0.9.1.md

@@ -89,7 +89,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + niazarak ([niazarak](https://github.com/niazarak))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Eiston Wei ([eiston](https://github.com/eiston))
 + Charles Wu ([charW](https://github.com/charW))

docs/release-notes/release-notes-v0.9.2.md

@@ -87,7 +87,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + niazarak ([niazarak](https://github.com/niazarak))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Eiston Wei ([eiston](https://github.com/eiston))
 + egzhbdt ([egzhbdt](https://github.com/egzhbdt))

docs/release-notes/release-notes-v0.9.3.md

@@ -86,7 +86,7 @@ Sorted by number of commits, [according to GitHub](https://github.com/castorini/
 + niazarak ([niazarak](https://github.com/niazarak))
 + Kevin Xu ([kevinxyc1](https://github.com/kevinxyc1))
 + Matt Yang ([justram](https://github.com/justram))
-+ Kelvin Jiang ([infinitecold](https://github.com/infinitecold))
++ Kelvin Jiang ([kelvin-jiang](https://github.com/kelvin-jiang))
 + Guy Rosin ([guyrosin](https://github.com/guyrosin))
 + Eiston Wei ([eiston](https://github.com/eiston))
 + egzhbdt ([egzhbdt](https://github.com/egzhbdt))

src/main/resources/docgen/templates/robust04.template

@@ -45,7 +45,7 @@ ${effectiveness}
 ## Replication Log
 
 + Results replicated by [@chriskamphuis](https://github.com/chriskamphuis) on 2018-12-18 (commit [`a15235`](https://github.com/castorini/Anserini/commit/a152359435ac6ae694b39f561343bba5eed8fdc9))
-+ Results replicated by [@infinitecold](https://github.com/infinitecold) on 2019-09-08 (commit [`a1892ae`](https://github.com/castorini/anserini/commit/a1892aec726efe55111a7bc501ab0914afab3a30))
++ Results replicated by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2019-09-08 (commit [`a1892ae`](https://github.com/castorini/anserini/commit/a1892aec726efe55111a7bc501ab0914afab3a30))
 + Results replicated by [@JMMackenzie](https://github.com/JMMackenzie) on 2020-01-21 (commit [`f63cd22`](https://github.com/castorini/anserini/commit/f63cd2275fa5a9d4da2d17e5f983a3308e8b50ce))
 + Results replicated by [@nikhilro](https://github.com/nikhilro) on 2020-01-26 (commit [`d5ee069`](https://github.com/castorini/anserini/commit/d5ee069399e6a306d7685bda756c1f19db721156))
 + Results replicated by [@edwinzhng](https://github.com/edwinzhng) on 2020-01-26 (commit [`7b76dfb`](https://github.com/castorini/anserini/commit/7b76dfbea7e0c01a3a5dc13e74f54852c780ec9b))