feat: eval framework for benchmarking debate harness

.eval/datasets/warroom.json

@@ -0,0 +1,197 @@
+[
+  {
+    "question": "Complete the following reactions and select the major products by considering the steric hindrance, and the stability of the products.\n2-ethyl-2,6-dimethylcyclohexan-1-one + ethyl acrylate (t-BuOK) ---> A\n1-nitropropane + (KOH, (E)-but-2-enenitrile, H2O) ---> B",
+    "choices": {
+      "A": "A = ethyl 3-(3-ethyl-1,3-dimethyl-2-oxocyclohexyl)propanoate, B = 3-methyl-4-nitrohexanenitrile",
+      "B": "A = ethyl 3-(3-ethyl-1,3-dimethyl-2-oxocyclohexyl)propanoate, B = 2,3-dimethyl-4-nitrobutanenitrile",
+      "C": "A = ethyl 3-(3-ethyl-3,5-dimethyl-4-oxocyclohexyl)propanoate, B = 3-methyl-4-nitrohexanenitrile",
+      "D": "A = ethyl 3-(3-ethyl-3,5-dimethyl-4-oxocyclohexyl)propanoate, B = 2,3-dimethyl-4-nitrobutanenitrile"
+    },
+    "ground_truth": "A",
+    "source": "gpqa",
+    "category": "science",
+    "id": 0
+  },
+  {
+    "question": "A benzene ring has a carboxylic acid a carbaldehyde and a cyano group all meta to one another. Ortho to the carboxylic acid are a hydroxyl and a dimethyl amino and para to the carboxylic acid is a methoxy group. The methoxy and the alcohol are also both ortho to the nitrile. What is the correct IUPAC name for this molecule?",
+    "choices": {
+      "A": "2-hydroxy-3-cyano-4-methoxy-5-formyl-6-(dimethylamino)benzoic acid",
+      "B": "5-cyano-2-(dimethylamino)-3-formyl-6-hydroxy-4-methoxybenzoic acid",
+      "C": "2-(dimethylamino)-3-formyl-4-methoxy-5-cyano-6-hydroxybenzoic acid",
+      "D": "3-cyano-6-(dimethylamino)-5-formyl-2-hydroxy-4-methoxybenzoic acid"
+    },
+    "ground_truth": "D",
+    "source": "gpqa",
+    "category": "science",
+    "id": 1
+  },
+  {
+    "question": "A spin-half particle is in a linear superposition 0.5|\\uparrow\\rangle+sqrt(3)/2|\\downarrow\\rangle of its spin-up and spin-down states. If |\\uparrow\\rangle and |\\downarrow\\rangle are the eigenstates of \\sigma{z} , then what is the expectation value up to one decimal place, of the operator 10\\sigma{z}+5\\sigma_{x} ? Here, symbols have their usual meanings",
+    "choices": {
+      "A": "0.85",
+      "B": "-0.7",
+      "C": "-1.4",
+      "D": "1.65"
+    },
+    "ground_truth": "B",
+    "source": "gpqa",
+    "category": "science",
+    "id": 2
+  },
+  {
+    "question": "There has been an outbreak of an viral infectious disease in your city. You have been asked to design a molecular diagnostic kit for quick detection of this retrovirus. How would you go about designing this kit?",
+    "choices": {
+      "A": "First you identify the virus using the information about the symptoms, provided by the affected patients. You then design a nested PCR kit based on the information for quick diagnosis.",
+      "B": "First you identify the virus by performing cDNA sequencing.  You then develop a real time PCR kit for quick and accurate diagnosis.",
+      "C": "First you identify the IgG antibodies for the infection. You then develop an ELISA kit targeting IgG antibodies for quick \n and accurate diagnosis.",
+      "D": "First you identify the virus by performing DNA sequencing. You then develop a PCR kit for  quick and accurate diagnosis"
+    },
+    "ground_truth": "B",
+    "source": "gpqa",
+    "category": "science",
+    "id": 3
+  },
+  {
+    "question": "Identify compound E in the following reaction scheme.\nCompound A, when reacted with HCN, produces compound B.\nCompound B, upon reduction with H2 in the presence of Pd, forms compound C.\nCompound C, upon reaction with HNO2, forms compound D.\nCompound D, upon elimination of an inert gas, forms compound E.\nHints:\na) Compound A, when reacted with an unknown phosphorous ylide, forms 1,2-dimethyl-4-(propan-2-ylidene)cyclopentane.\nb) The IR spectrum of compound A exhibits a strong peak at around 1750 cm^-1 whereas, IR spectrum of compound E exhibits strong peak at around 1715 cm^-1.",
+    "choices": {
+      "A": "3,4-dimethylcyclohexan-1-one",
+      "B": "2,3,4-trimethylcyclopentan-1-one",
+      "C": "2,2,3,4-tetramethylcyclobutan-1-one",
+      "D": "4-methylcycloheptan-1-one"
+    },
+    "ground_truth": "A",
+    "source": "gpqa",
+    "category": "science",
+    "id": 4
+  },
+  {
+    "question": "Compute $17^{-1}\\pmod{83}$. Express your answer as a residue from $0$ to $82$, inclusive.\n\n(You may find it helpful to consider the fact that $17\\cdot 5=85$.)",
+    "ground_truth": "44",
+    "source": "math",
+    "category": "Number Theory",
+    "id": 5
+  },
+  {
+    "question": "Let $A$ and $B$ be two opposite corners of a unit cube, and let $C$ be another vertex of the cube other than $A$ and $B.$  Find the distance from $C$ to line $AB.$",
+    "ground_truth": "\\frac{\\sqrt{6}}{3}",
+    "source": "math",
+    "category": "Precalculus",
+    "id": 6
+  },
+  {
+    "question": "The matrix\n\\[\\mathbf{A} = \\begin{pmatrix} 4 & 7 \\\\ c & d \\end{pmatrix}\\]has the property that if you are told the value of $\\|\\mathbf{v}\\|,$ then you can derive the value of $\\|\\mathbf{A} \\mathbf{v}\\|.$  Compute $|c - d|,$ assuming that $c$ and $d$ are real numbers.",
+    "ground_truth": "11",
+    "source": "math",
+    "category": "Precalculus",
+    "id": 7
+  },
+  {
+    "question": "The product of the proper positive integer factors of $n$ can be written as $n^{(ax+b)/c}$, where $x$ is the number of positive divisors $n$ has, $c$ is a positive integer, and the greatest common factor of the three integers $a$, $b$, and $c$ is $1$. What is $a+b+c$?",
+    "ground_truth": "1",
+    "source": "math",
+    "category": "Number Theory",
+    "id": 8
+  },
+  {
+    "question": "The parabola $y = x^2 + bx + c$ has the following properties:\n\nThe point on the parabola closest to $(12,3)$ is the $y$-intercept of the parabola.\nThe parabola passes through $(-5,0).$\n\nEnter the ordered pair $(b,c).$",
+    "ground_truth": "(6,5)",
+    "source": "math",
+    "category": "Intermediate Algebra",
+    "id": 9
+  },
+  {
+    "question": "Calculate the spin angular momentum of 43Ca. [I = 7\u20442]",
+    "choices": {
+      "A": "7.853 x 10^-34 J s",
+      "B": "8.370 x 10^-34 J s",
+      "C": "1.256 x 10^-34 J s",
+      "D": "5.493 x 10^-34 J s",
+      "E": "9.424 x 10^-34 J s",
+      "F": "6.283 x 10^-34 J s",
+      "G": "2.166 x 10^-34 J s",
+      "H": "1.047 x 10^-33 J s",
+      "I": "3.691 x 10^-34 J s",
+      "J": "4.185 x 10^-34 J s"
+    },
+    "ground_truth": "J",
+    "source": "mmlu_pro",
+    "category": "chemistry",
+    "id": 10
+  },
+  {
+    "question": "The principal refracting powers of a thin lens are + 4 and - 5dptr. If the refracting power in an oblique normal section is + 2dptr, what will be its refracting power in a normal section at right angle to the first? What is the angle of inclination of the + 2 section to the + 4 section?",
+    "choices": {
+      "A": "-1dptrs, 29\u00b0 8' 40''",
+      "B": "+3dptrs, 33\u00b0 15' 5''",
+      "C": "0dptrs, 27\u00b0 9' 50''",
+      "D": "-4dptrs, 32\u00b0 5' 55''",
+      "E": "-4dptrs, 26\u00b0 5' 15''",
+      "F": "-2dptrs, 30\u00b0 10' 20''",
+      "G": "-6dptrs, 31\u00b0 12' 45''",
+      "H": "-5dptrs, 24\u00b0 6' 40''",
+      "I": "+1dptr, 25\u00b0 3' 10''",
+      "J": "-3dptrs, 28\u00b0 7' 32''"
+    },
+    "ground_truth": "J",
+    "source": "mmlu_pro",
+    "category": "physics",
+    "id": 11
+  },
+  {
+    "question": "Geologists on a plane are attempting to locate the exact position of the iron reserves in a region by measuring the variation of g in the area, since a large mass of iron will exert an appreciable additional gravitational attraction on a body in the vicinity. They hover above selected spots and observe the movement of a mass suspended from a light spring. If the system has a natural period of 2 s, and the smallest movement of the mass which can be detected is 10^-6 m, what is the minimum change in g which they can observe?",
+    "choices": {
+      "A": "9.87 \u00d710^-6 m/s^2",
+      "B": "6.28 \u00d710^-6 m/s^2",
+      "C": "7.54 \u00d710^-6 m/s^2",
+      "D": "11.00 \u00d710^-6 m/s^2",
+      "E": "2.50 \u00d710^-6 m/s^2",
+      "F": "8.88 \u00d710^-6 m/s^2",
+      "G": "4.93 \u00d710^-6 m/s^2",
+      "H": "3.14 \u00d710^-6 m/s^2",
+      "I": "5.00 \u00d710^-6 m/s^2",
+      "J": "1.57 \u00d710^-6 m/s^2"
+    },
+    "ground_truth": "A",
+    "source": "mmlu_pro",
+    "category": "physics",
+    "id": 12
+  },
+  {
+    "question": "What is the de Broglie wavelength of a radium alpha particle (a helium nucleus,He^2+) having an energy of 4.8MeV(million electron volts)?",
+    "choices": {
+      "A": "1.5 \u00d7 10^-11 cm",
+      "B": "6.6 \u00d7 10^-27 cm",
+      "C": "1.0 \u00d7 10^-14 cm",
+      "D": "1.0 \u00d7 10^-28 cm",
+      "E": "7.7 \u00d7 10^-16 cm",
+      "F": "3.2 \u00d7 10^-15 cm",
+      "G": "2.2 \u00d7 10^-13 cm",
+      "H": "5.0 \u00d7 10^-12 cm",
+      "I": "8.4 \u00d7 10^-14 cm",
+      "J": "6.6 \u00d7 10^-13 cm"
+    },
+    "ground_truth": "J",
+    "source": "mmlu_pro",
+    "category": "chemistry",
+    "id": 13
+  },
+  {
+    "question": "If the exhaust gases contain 0.22% of NO by weight, calculate (i) The minimum value of \\delta allowable if a NO reduction rate of0.032 lb/ft^2hr is to be achieved. (ii) The corresponding minimum allowable value of K. Use the following data. The gases are at: T = 1200\u00b0F P = 18.2psia Average molecular wt. of the gases = 30.0 Effective rate constant K = 760 ft/hr Diffusion coefficientD_(_1)m = 4.0 ft^2/hr",
+    "choices": {
+      "A": "\\delta = 0.0028 ft, K = 55 lb/ft^2hr",
+      "B": "\\delta = 0.0040 ft, K = 45 lb/ft^2hr",
+      "C": "\\delta = 0.0050 ft, K = 60 lb/ft^2hr",
+      "D": "\\delta = 0.0032 ft, K = 47 lb/ft^2hr",
+      "E": "\\delta = 0.0045 ft, K = 44 lb/ft^2hr",
+      "F": "\\delta = 0.0024 ft, K = 49 lb/ft^2hr",
+      "G": "\\delta = 0.0032 ft, K = 50 lb/ft^2hr",
+      "H": "\\delta = 0.0018 ft, K = 42 lb/ft^2hr",
+      "I": "\\delta = 0.0026 ft, K = 47 lb/ft^2hr",
+      "J": "\\delta = 0.0035 ft, K = 52 lb/ft^2hr"
+    },
+    "ground_truth": "D",
+    "source": "mmlu_pro",
+    "category": "engineering",
+    "id": 14
+  }
+]