Dataset generation #31

jack89roberts · 2024-06-19T08:22:11Z

No description provided.

…d losses across potential answers

…(defaults to forget)

…ions

… outputs the metrics we want to track. Also hidden away/cleaned up evaluation scripts and moved most functions to the utils file

…, dtype)

…he max() function in table 1 of the tofu paper (it wasn't)

… script if ran as main

…run with differing file paths

…hould take initial data, and save new data

…s with exactly the right answers in it.. There may still be questions with slightly correct answers

…ute/arc-selective-forgetting into 2-new-dataset

src/arcsf/data/generation/name_bank/publisher_names.csv

src/arcsf/data/generation/utils.py

src/arcsf/data/generation/scripts/generate_tofu.py

J-Dymond

Briefly discussed code with JR. Appropriate to merge now.

J-Dymond

Briefly discussed code with JR. Appropriate to merge now. This time an approving review...

jack89roberts linked an issue Jun 19, 2024 that may be closed by this pull request

Extend TOFU Dataset #2

Closed

J-Dymond and others added 29 commits June 19, 2024 11:13

added some preliminary evaluation metrics with some tests

49f346d

added KS-Test to evaluation functions

6e49195

standardising pytest.approx() in tests

e034706

corrected the input for conditional probability: it accepts normalise…

6680d76

…d losses across potential answers

adding the loss function

5d004f8

added a loss and a test for it

b9eaf97

used the dummy model + tokeniser for tests

8d755ad

TODO: write tests and debug eval functions

b17c36a

added tests for the truth ratio, now working as intented

1dc32f9

adding evaluation pipeline

af4c4c2

trying to implement the evaluation pieplline

c30b3a7

added an end to end test in test_eval.py

42596bd

added an end to end test in test_eval.py

db26a3c

added a qualitative evalation function

24d4913

added argparser for the qualitative eval script

8a2d23f

added initial quanititative and qualitative evaluation scripts

dbf0416

commented and provided docstrings for all code

18a958f

added the data split to the input arguments of the evaluation script …

4913c5a

…(defaults to forget)

📌 Update poetry.lock

9eff99d

🔥 delete VSCode config

97d4043

added a third evaluation script which outputs all scores

6e6e032

broken test: fixed, to do with device when running non-tokenized vers…

bdb1094

…ions

added a script which aggregates the evaluation metrics

1b0aaca

added function which evaluates the model against a baseline model and…

b057cf5

… outputs the metrics we want to track. Also hidden away/cleaned up evaluation scripts and moved most functions to the utils file

minor error in tests going from type(var) == dtype --> isinstance(var…

cd8206e

…, dtype)

added a comment in the ks_test

2dba5b2

minor changes: fixed some imports and ensured the function performs t…

cc3ed62

…he max() function in table 1 of the tofu paper (it wasn't)

minor fixes: added max generated tokens, and gave evaluate_model.py a…

7e552b7

… script if ran as main

added base values path to the argparser of the evaluate script

ed10e00

J-Dymond added 2 commits July 24, 2024 16:33

added some scripts that fixed the question base, will refactor to be …

1652d4b

…run with differing file paths

started refactoring some scripts

c20df7a

J-Dymond force-pushed the 2-new-dataset branch from 47e22b4 to c20df7a Compare July 24, 2024 15:36

J-Dymond and others added 17 commits July 25, 2024 15:33

adapted data_module scripts for loading gen_tofu/tofu2.0/mulch

e334994

all scripts now need to be passed directories/files from where they s…

ef43f8b

…hould take initial data, and save new data

renamed split checking script

afd6e90

added a script to analyse question counts for all entities

2584be1

Merge branch 'develop' into 2-new-dataset

ed74ecc

removed unnecessary files in evaluate

e9c6ea2

updated lock file

c1c5d74

added the full dataset to the repo

d2a9652

fix create dataset file, to actually run ...

93fb29b

added same fixes to the new dataset regarding the retain_permutation

ab30ce3

added some changes to the question_check file to fix perturbed answer…

66a4536

…s with exactly the right answers in it.. There may still be questions with slightly correct answers

created dataset with fixed rows

3fb06ca

Add model cache_dir config to Baskerville scripts and related config

2c7d604

switch to using env variables to set cache paths

f600908

Pelican Publishing -> Fixed Flock Press

a459d6f

Pelican Publishing -> Fixed Flock Press

6449deb

updated dataset with pelican publishing -> Fixed Flock

4cc7c66

jack89roberts marked this pull request as ready for review August 2, 2024 13:31

jack89roberts and others added 3 commits August 2, 2024 14:37

assert -> check and raise

d06c1b3

renamed 13 publishers due to real-world overlap

dde3c09

Merge branch '2-new-dataset' of https://github.com/alan-turing-instit…

f3ebace

…ute/arc-selective-forgetting into 2-new-dataset

jack89roberts commented Aug 2, 2024

View reviewed changes

src/arcsf/data/generation/name_bank/publisher_names.csv Outdated Show resolved Hide resolved

src/arcsf/data/generation/utils.py Outdated Show resolved Hide resolved

src/arcsf/data/generation/scripts/generate_tofu.py Show resolved Hide resolved

addressed some comments

7e55c42

J-Dymond reviewed Aug 5, 2024

View reviewed changes

J-Dymond approved these changes Aug 5, 2024

View reviewed changes

jack89roberts merged commit aee2e95 into develop Aug 7, 2024
1 check passed

jack89roberts deleted the 2-new-dataset branch August 7, 2024 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataset generation #31

Dataset generation #31

jack89roberts commented Jun 19, 2024

J-Dymond left a comment

J-Dymond left a comment

Dataset generation #31

Dataset generation #31

Conversation

jack89roberts commented Jun 19, 2024

J-Dymond left a comment

Choose a reason for hiding this comment

J-Dymond left a comment

Choose a reason for hiding this comment