Skip to content

Evals failed on main (PR #2078) #2106

Open
Open
Evals failed on main (PR #2078)#2106
Labels
eval_failureAdded to automated issues created by an eval run failure post-submit.status: needs-triageauto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs

Description

@github-actions
github-actions
bot
opened on Jul 27, 2026

The evals workflow failed on main for commit 349c979.
Associated PR: #2078
See logs: https://github.com/a2ui-project/a2ui/actions/runs/30271702608

Detailed Summary

Evaluation Summary: a2ui_v0_9_1_eval

  • Status: PASS
  • Model: google/gemini-3.5-flash
  • Pass Percentage: 96.36% (Threshold: 90.00%)

Dataset Performance Summary

Dataset Name Total Samples Algorithmic Pass Rate Judging Pass Rate Dataset Pass Rate
core_v0_9_1 51 96.1% 86.3% 86.3%
multi_turn_conversation_dataset 4 100.0% 100.0% 100.0%

Sample Results

Sample / Task Algorithmic Judging Inference Time Status
animalKingdomExplorer PASS C 83.40s PASS
calendarEventCreator PASS C 35.86s PASS
chatRoom PASS C 1023.35s PASS
checkoutPage PASS C 140.09s PASS
cinemaSeatSelection PASS C 101.78s PASS
clientSideValidation PASS C 17.23s PASS
contactCard PASS C 111.60s PASS
contextAwareUI FAIL I 55.30s FAIL
courseSyllabus PASS C 65.23s PASS
customYamlKeysTest PASS C 55.44s PASS
dashboard PASS C 101.73s PASS
deleteSurface PASS C 6.63s PASS
dogBreedGenerator PASS C 10.56s PASS
eCommerceProductPage PASS C 84.03s PASS
fileBrowser PASS C 198.36s PASS
fitnessTracker PASS C 65.42s PASS
flashcardApp PASS P 37.35s FAIL
flightBooker PASS C 112.37s PASS
hotelSearchResults PASS C 29.90s PASS
interactiveDashboard PASS C 196.68s PASS
jobApplication PASS C 142.29s PASS
kanbanBoard PASS C 122.34s PASS
loginForm PASS C 46.12s PASS
mcpAppProxy FAIL I 4.08s FAIL
musicPlayer PASS C 303.52s PASS
nestedDataBinding PASS C 99.52s PASS
nestedLayoutRecursive PASS C 54.13s PASS
newsAggregator PASS C 20.00s PASS
notificationCenter PASS C 618.92s PASS
openUrlAction PASS C 158.62s PASS
photoEditor PASS C 1724.44s PASS
podcastEpisode PASS P 55.18s FAIL
productGallery PASS C 104.56s PASS
productGalleryData PASS C 42.89s PASS
profileEditor PASS P 47.31s FAIL
recipeCard PASS C 335.96s PASS
restaurantFinder PASS I 69.12s FAIL
restaurantMenu PASS C 880.14s PASS
rizzCharts PASS C 21.36s PASS
settingsPage PASS C 956.97s PASS
simpleCalculator PASS C 437.04s PASS
smartHome PASS C 23.54s PASS
socialMediaPost PASS C 46.06s PASS
standardFunctions PASS C 56.79s PASS
stockWatchlist PASS P 102.14s FAIL
surveyForm PASS C 250.51s PASS
travelItinerary PASS C 55.72s PASS
triviaQuiz PASS C 40.46s PASS
updateDataModel PASS C 55.15s PASS
videoCallInterface PASS C 21.60s PASS
weatherForecast PASS C 7.77s PASS
banking_dispute_resolution PASS C 102.18s PASS
corporate_travel_multi_city_rebooking PASS C 246.92s PASS
flight_booking_flow PASS C 55.66s PASS
healthcare_patient_intake_triage PASS C 101.47s PASS

Failure Details

contextAwareUI
  • Algorithmic Failure Reason:

    Validation failed: {'version': 'v0.9', 'updateComponents': {'surfaceId': 'main', 'components': [{'id': 'root', 'component': 'Column', 'children': ['header_card', 'filter_card', 'kpis_container', 'viz_container', 'data_container']}, {'id': 'header_card', 'component': 'Card', 'child': 'header_layout'}, {'id': 'filter_card', 'component': 'Card', 'child': 'filter_layout'}, {'id': 'kpis_container', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'viz_container', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'data_container', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'header_layout', 'component': 'Row', 'justify': 'spaceBetween', 'align': 'center'}, {'id': 'header_text_col', 'component': 'Column'}, {'id': 'header_icon', 'component': 'Icon', 'name': 'accountCircle'}, {'id': 'header_title', 'component': 'Text', 'text': 'Q2 Sales Performance Dashboard', 'variant': 'h1'}, {'id': 'header_sub', 'component': 'Text', 'text': 'Role: Manager | Department: Sales | Period: Q2 Active', 'variant': 'caption'}, {'id': 'filter_layout', 'component': 'Row', 'justify': 'spaceBetween', 'align': 'center'}, {'id': 'filter_product', 'component': 'ChoicePicker', 'label': 'Product Category', 'variant': 'mutuallyExclusive', 'options': [{'label': 'Enterprise Suite', 'value': 'Enterprise'}, {'label': 'Mid-Market Plan', 'value': 'Mid-Market'}, {'label': 'SMB Starter', 'value': 'SMB'}], 'value': {'path': '/filters/product'}, 'displayStyle': 'chips', 'weight': 2}, {'id': 'filter_channel', 'component': 'ChoicePicker', 'label': 'Sales Channel', 'variant': 'mutuallyExclusive', 'options': [{'label': 'Direct Sales', 'value': 'Direct'}, {'label': 'Channel Partners', 'value': 'Partners'}, {'label': 'Online Store', 'value': 'Online'}], 'value': {'path': '/filters/channel'}, 'displayStyle': 'chips', 'weight': 2}, {'id': 'export_btn', 'component': 'Button', 'variant': 'primary', 'child': 'export_btn_text', 'action': {'event': {'name': 'exportReport', 'context': {'format': 'PDF', 'reportType': 'Q2_Performance'}}}, 'weight': 1}, {'id': 'export_btn_text', 'component': 'Text', 'text': 'Export PDF Report'}, {'id': 'kpi_rev', 'component': 'Card', 'child': 'kpi_rev_col', 'weight': 1}, {'id': 'kpi_target', 'component': 'Card', 'child': 'kpi_target_col', 'weight': 1}, {'id': 'kpi_deal', 'component': 'Card', 'child': 'kpi_deal_col', 'weight': 1}, {'id': 'kpi_win', 'component': 'Card', 'child': 'kpi_win_col', 'weight': 1}, {'id': 'kpi_rev_col', 'component': 'Column', 'children': ['kpi_rev_lbl', 'kpi_rev_val', 'kpi_rev_growth']}, {'id': 'kpi_rev_lbl', 'component': 'Text', 'text': 'Total Q2 Revenue', 'variant': 'caption'}, {'id': 'kpi_rev_val', 'component': 'Text', 'text': '$2,452,000', 'variant': 'h2'}, {'id': 'kpi_rev_growth', 'component': 'Text', 'text': '▲ +14.2% vs Q1', 'variant': 'body'}, {'id': 'kpi_target_col', 'component': 'Column', 'children': ['kpi_target_lbl', 'kpi_target_val', 'kpi_target_growth']}, {'id': 'kpi_target_lbl', 'component': 'Text', 'text': 'Target Achievement', 'variant': 'caption'}, {'id': 'kpi_target_val', 'component': 'Text', 'text': '96.1%', 'variant': 'h2'}, {'id': 'kpi_target_growth', 'component': 'Text', 'text': 'Target: $2,550,000', 'variant': 'body'}, {'id': 'kpi_deal_col', 'component': 'Column', 'children': ['kpi_deal_lbl', 'kpi_deal_val', 'kpi_deal_growth']}, {'id': 'kpi_deal_lbl', 'component': 'Text', 'text': 'Avg Deal Size', 'variant': 'caption'}, {'id': 'kpi_deal_val', 'component': 'Text', 'text': '$48,200', 'variant': 'h2'}, {'id': 'kpi_deal_growth', 'component': 'Text', 'text': '▲ +3.8% MoM', 'variant': 'body'}, {'id': 'kpi_win_col', 'component': 'Column', 'children': ['kpi_win_lbl', 'kpi_win_val', 'kpi_win_growth']}, {'id': 'kpi_win_lbl', 'component': 'Text', 'text': 'Win Rate', 'variant': 'caption'}, {'id': 'kpi_win_val', 'component': 'Text', 'text': '28.4%', 'variant': 'h2'}, {'id': 'kpi_win_growth', 'component': 'Text', 'text': 'Target: 30.0%', 'variant': 'body'}, {'id': 'monthly_card', 'component': 'Card', 'child': 'monthly_layout', 'weight': 1}, {'id': 'regional_card', 'component': 'Card', 'child': 'regional_layout', 'weight': 1}, {'id': 'monthly_layout', 'component': 'Column', 'children': ['monthly_title', 'monthly_row_apr', 'monthly_row_may', 'monthly_row_jun']}, {'id': 'monthly_title', 'component': 'Text', 'text': 'Q2 Monthly Target vs Actuals', 'variant': 'h3'}, {'id': 'monthly_row_apr', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'monthly_row_may', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'monthly_row_jun', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'apr_lbl', 'component': 'Text', 'text': 'April', 'variant': 'body'}, {'id': 'apr_target', 'component': 'Text', 'text': 'Target: $800k', 'variant': 'caption'}, {'id': 'apr_actual', 'component': 'Text', 'text': 'Actual: $820k (▲ Achieved)', 'variant': 'body'}, {'id': 'may_lbl', 'component': 'Text', 'text': 'May', 'variant': 'body'}, {'id': 'may_target', 'component': 'Text', 'text': 'Target: $850k', 'variant': 'caption'}, {'id': 'may_actual', 'component': 'Text', 'text': 'Actual: $792k (▼ Backlog)', 'variant': 'body'}, {'id': 'jun_lbl', 'component': 'Text', 'text': 'June', 'variant': 'body'}, {'id': 'jun_target', 'component': 'Text', 'text': 'Target: $900k', 'variant': 'caption'}, {'id': 'jun_actual', 'component': 'Text', 'text': 'Actual: $840k (Est.)', 'variant': 'body'}, {'id': 'regional_layout', 'component': 'Column', 'children': ['reg_title', 'reg_na_col', 'reg_emea_col', 'reg_apac_col']}, {'id': 'reg_title', 'component': 'Text', 'text': 'Regional Quota Achievement (%)', 'variant': 'h3'}, {'id': 'reg_na_col', 'component': 'Column', 'children': ['reg_na_lbl', 'reg_na_progress']}, {'id': 'reg_emea_col', 'component': 'Column', 'children': ['reg_emea_lbl', 'reg_emea_progress']}, {'id': 'reg_apac_col', 'component': 'Column', 'children': ['reg_apac_lbl', 'reg_apac_progress']}, {'id': 'reg_na_lbl', 'component': 'Text', 'text': 'North America', 'variant': 'caption'}, {'id': 'reg_na_progress', 'component': 'Slider', 'label': 'NA Progress', 'min': 0, 'max': 100, 'value': {'path': '/regions/na'}}, {'id': 'reg_emea_lbl', 'component': 'Text', 'text': 'EMEA', 'variant': 'caption'}, {'id': 'reg_emea_progress', 'component': 'Slider', 'label': 'EMEA Progress', 'min': 0, 'max': 100, 'value': {'path': '/regions/emea'}}, {'id': 'reg_apac_lbl', 'component': 'Text', 'text': 'APAC', 'variant': 'caption'}, {'id': 'reg_apac_progress', 'component': 'Slider', 'label': 'APAC Progress', 'min': 0, 'max': 100, 'value': {'path': '/regions/apac'}}, {'id': 'rep_card', 'component': 'Card', 'child': 'rep_layout', 'weight': 1}, {'id': 'funnel_card', 'component': 'Card', 'child': 'funnel_layout', 'weight': 1}, {'id': 'rep_layout', 'component': 'Column', 'children': ['rep_title', 'rep_table_header', 'rep_row_1', 'rep_row_2', 'rep_row_3']}, {'id': 'rep_title', 'component': 'Text', 'text': 'Top Performing Sales Representatives (Q2)', 'variant': 'h3'}, {'id': 'rep_table_header', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'rep_row_1', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'rep_row_2', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'rep_row_3', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'th_name', 'component': 'Text', 'text': 'Representative Name', 'variant': 'caption', 'weight': 2}, {'id': 'th_deals', 'component': 'Text', 'text': 'Deals', 'variant': 'caption', 'weight': 1}, {'id': 'th_rev', 'component': 'Text', 'text': 'Revenue', 'variant': 'caption', 'weight': 1}, {'id': 'th_pct', 'component': 'Text', 'text': 'Quota %', 'variant': 'caption', 'weight': 1}, {'id': 'r1_name', 'component': 'Text', 'text': 'Sarah Jenkins', 'variant': 'body', 'weight': 2}, {'id': 'r1_deals', 'component': 'Text', 'text': '24', 'variant': 'body', 'weight': 1}, {'id': 'r1_rev', 'component': 'Text', 'text': '$540k', 'variant': 'body', 'weight': 1}, {'id': 'r1_pct', 'component': 'Text', 'text': '108%', 'variant': 'body', 'weight': 1}, {'id': 'r2_name', 'component': 'Text', 'text': 'Michael Chang', 'variant': 'body', 'weight': 2}, {'id': 'r2_deals', 'component': 'Text', 'text': '19', 'variant': 'body', 'weight': 1}, {'id': 'r2_rev', 'component': 'Text', 'text': '$480k', 'variant': 'body', 'weight': 1}, {'id': 'r2_pct', 'component': 'Text', 'text': '102%', 'variant': 'body', 'weight': 1}, {'id': 'r3_name', 'component': 'Text', 'text': 'Elena Rostova', 'variant': 'body', 'weight': 2}, {'id': 'r3_deals', 'component': 'Text', 'text': '22', 'variant': 'body', 'weight': 1}, {'id': 'r3_rev', 'component': 'Text', 'text': '$460k', 'variant': 'body', 'weight': 1}, {'id': 'r3_pct', 'component': 'Text', 'text': '98%', 'variant': 'body', 'weight': 1}, {'id': 'funnel_layout', 'component': 'Column', 'children': ['funnel_title', 'funnel_qual', 'funnel_prop', 'funnel_nego', 'funnel_won']}, {'id': 'funnel_title', 'component': 'Text', 'text': 'Pipeline Funnel Overview', 'variant': 'h3'}, {'id': 'funnel_qual', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'funnel_prop', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'funnel_nego', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'funnel_won', 'component': 'Row', 'justify': 'spaceBetween'}, {'id': 'fq_lbl', 'component': 'Text', 'text': 'Qualification', 'variant': 'body'}, {'id': 'fq_val', 'component': 'Text', 'text': '$1.2M (45 Deals)', 'variant': 'body'}, {'id': 'fp_lbl', 'component': 'Text', 'text': 'Proposal', 'variant': 'body'}, {'id': 'fp_val', 'component': 'Text', 'text': '$850k (28 Deals)', 'variant': 'body'}, {'id': 'fn_lbl', 'component': 'Text', 'text': 'Negotiation', 'variant': 'body'}, {'id': 'fn_val', 'component': 'Text', 'text': '$620k (14 Deals)', 'variant': 'body'}, {'id': 'fw_lbl', 'component': 'Text', 'text': 'Won', 'variant': 'body'}, {'id': 'fw_val', 'component': 'Text', 'text': '$2.4M (82 Deals)', 'variant': 'body'}]}} is not valid under any of the given schemas
    Context failures:

    • 'createSurface' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
    • {'id': 'kpis_container', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'viz_container', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'data_container', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'header_layout', 'component': 'Row', 'justify': 'spaceBetween', 'align': 'center'} is not valid under any of the given schemas
    • {'id': 'header_text_col', 'component': 'Column'} is not valid under any of the given schemas
    • {'id': 'filter_layout', 'component': 'Row', 'justify': 'spaceBetween', 'align': 'center'} is not valid under any of the given schemas
    • {'id': 'monthly_row_apr', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'monthly_row_may', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'monthly_row_jun', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'rep_table_header', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'rep_row_1', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'rep_row_2', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'rep_row_3', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'funnel_qual', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'funnel_prop', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'funnel_nego', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • {'id': 'funnel_won', 'component': 'Row', 'justify': 'spaceBetween'} is not valid under any of the given schemas
    • 'updateDataModel' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
    • 'deleteSurface' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
  • Judging Failure Reason (Grade I):

    An evaluation of the submitted A2UI payload reveals a major structural issue in the component hierarchy:

    1. Disconnected/Orphaned Components: While the submission defines a wide variety of UI components (buttons, text fields, row/column layouts, sliders, etc.), many of the parent container elements are missing the required children property to link them to their child elements.
      • header_layout does not contain references to its text and icon children.
      • filter_layout does not contain references to filter_product, filter_channel, or export_btn.
      • kpis_container does not link to kpi_rev, kpi_target, etc.
      • viz_container does not link to monthly_card or regional_card.
      • data_container does not link to rep_card or funnel_card.
      • Row containers like monthly_row_apr, rep_table_header, rep_row_1, and funnel_qual are completely empty of child references, meaning their labels, values, and metrics are entirely orphaned.

    Because these critical structural relationships are missing, most of the dashboard elements will not render at all. This constitutes a substantive error rather than a minor cosmetic issue.

    GRADE: I

flashcardApp
  • Judging Failure Reason (Grade P):

    To determine if the submission meets the criterion, let's break down the requirements and compare them to the submitted payload:

    1. A valid A2UI payload with surfaceId 'main':

      • The payload contains a createSurface action with surfaceId: "main" and an updateComponents action with surfaceId: "main". This is correct.

    2. An 'h1' 'Text' 'Spanish Vocabulary':

      • The submission contains a Text component with the ID "header".
      • Its text is "# Spanish Vocabulary" (which represents an H1 header in markdown notation, matching the task's text string).
      • However, its variant property is set to "h4" instead of "h1". This is a minor cosmetic variation from the requested "h1" variant.

    3. A 'Card' with a 'Column' inside:

      • Component "card_container" is of type Card and has the child "card_content_col", which is a Column. This is correct.

    4. The front text ('Hola'), a 'Divider', and the back text ('Hello'):

      • Inside the column, there is a Text component with text "## Hola" (representing the front).
      • There is a Divider component.
      • There is a Text component with text "Hello" (representing the back). This is correct.

    5. A 'Row' of 'Button's ('Hard', 'Good', 'Easy'):

      • Inside the column, there is a Row component with children "btn_hard_btn", "btn_good_btn", and "btn_easy_btn".
      • Each button contains text labels with the corresponding text ("Hard", "Good", "Easy"). This is correct.

    Since the submission meets all components and structure requirements, with only a minor cosmetic variation on the header's text variant (having "h4" instead of "h1" while maintaining the markdown # symbol), it qualifies for partial credit.

    GRADE: P

mcpAppProxy
  • Algorithmic Failure Reason:

    A2UI tags '' and '' not found in response.

  • Judging Failure Reason (Grade I):

    To assess whether the submission meets the criterion, let's look at the requirement:

    1. Criterion: Use the McpApp component to render the external app content.
    2. Submission Analysis:
      • The user requested a classic neon table tennis arcade game interface.
      • The submission apologizes for not being able to "spin up the visual layout at the moment" and instead describes the concept textually.
      • It offers to provide HTML/CSS/JS code if requested, but it does not actually render or invoke any external application interface, nor does it use the McpApp component (or any component/code block representing it) in the response.

    Since the submission completely fails to use the McpApp component and does not provide the requested interactive interface, it does not meet the specified criterion.

    GRADE: I

podcastEpisode
  • Judging Failure Reason (Grade P):

    To evaluate the submission, let us verify each requirement against the criterion step-by-step:

    1. A2UI payload with surfaceId 'main':

      • The submission correctly creates the surface "main" and specifies "surfaceId": "main" inside updateComponents.

    2. Includes a 'Card' containing other components:

      • The "root" component is indeed a "Card", which contains "main_container" (a "Column" layout), housing the remaining elements.

    3. Includes an 'Image':

      • The "cover_art" component is of type "Image".

    4. Includes 'Text' for the title:

      • The "title" component is a "Text" element with the value "Episode 42: The Future of AI". However, the criterion specifically expects an "h2" variant for this text, whereas the submission uses "variant": "h4".

    5. Includes 'Text' for the host:

      • The "host" component is a "Text" element with "text": "Host: Jane Smith".

    6. Includes 'Slider' labeled "Progress" initialized to 0:

      • The "progress_slider" component is of type "Slider", has "label": "Progress", and "value": 0.

    7. Includes a 'Row' with three static 'Button's:

      • The "controls_row" is of type "Row" and contains three button components ("btn_speed", "btn_play", and "btn_share").
      • These buttons point to static "Text" children representing "1x", "Play/Pause", and "Share", respectively.

    The submission is almost entirely correct, except for a minor cosmetic variation in the heading size (using h4 instead of h2 for the title text). Per Note 7, this qualifies for partial credit (P).

    GRADE: P

profileEditor
  • Judging Failure Reason (Grade P):

    To determine the grade of the submission, let's compare its components against the requirements listed in the criterion:

    1. surfaceId 'main':

      • The submission correctly specifies "surfaceId": "main" in both createSurface and updateComponents actions. (Met)

    2. 'h1' 'Text' 'Edit Profile':

      • The task requested 'Text' "# Edit Profile", while the criterion specifically asks for an 'h1' 'Text' 'Edit Profile'.
      • In the submission, the text component has "text": "# Edit Profile" and "variant": "h2". This is a minor cosmetic variation (using the h2 variant instead of h1 and keeping the # from the task description). (Partially Met / Minor cosmetic difference)

    3. 'Image' (Current Avatar):

      • The submission includes an Image component ("id": "avatar_image") with a valid image URL. (Met)

    4. 'Change Photo' 'Button':

      • The submission includes a Button ("id": "change_photo_btn") containing the text "Change Photo". (Met)

    5. 'TextField' for Display Name:

      • The submission includes a TextField ("id": "display_name_field") with the label "Display Name". (Met)

    6. 'TextField' for Bio (multiline):

      • The submission includes a TextField ("id": "bio_field") with the label "Bio" and the variant "longText", which represents a multiline text field. (Met)

    7. 'TextField' for Website:

      • The submission includes a TextField ("id": "website_field") with the label "Website". (Met)

    8. 'Save Changes' 'Button':

      • The submission includes a Button ("id": "save_btn") containing the text "Save Changes". (Met)

    Conclusion

    The submission successfully generates a correct UI containing all the required fields and layout. The only deviation is in the header's style and text format (using h2 and retaining the # character as specified in the original task, rather than an exact match of 'h1' 'Text' 'Edit Profile'). Under the grading guidelines, minor cosmetic variations or minor differences from the exact text criterion qualify for partial credit.

    GRADE: P

restaurantFinder
  • Judging Failure Reason (Grade I):

    Based on the assessment criteria provided:

    1. Populating under '/items': The submission does not define a state or data model with restaurant data populated under the path /items. Instead, it hardcodes the restaurant details directly into static UI components in the updateComponents instruction.
    2. Template Usage (More than 5 spots): The task requests 8 restaurants (which is more than 5). According to the criteria, the TWO_COLUMN_LIST_EXAMPLE template must be used. The submission does not use this template; instead, it uses a custom single-column vertical list layout.

    Since these structural and data constraints are not met, the submission is incorrect.

    GRADE: I

stockWatchlist
  • Judging Failure Reason (Grade P):

    To determine the grade for the submitted A2UI payload, let's evaluate it step-by-step against the given criterion:

    1. Valid A2UI payload with surfaceId 'main':

      • The submission contains a valid JSON array matching the A2UI specification structure.
      • It targets the correct surfaceId of "main" in both the createSurface and updateComponents actions.

    2. Feature a 'Market Watch' 'h1' 'Text':

      • The submission contains a Text component with the ID "title".
      • The text of this component is "# Market Watch" (including a markdown-style hash mark instead of just "Market Watch").
      • The variant used is "h2", whereas the criterion specifies "h1". This constitutes a minor cosmetic variation.

    3. A 'List' of 'Row's with stock symbol, price, and percentage change:

      • The submission defines a component "watchlist_list" of type "List".
      • The list contains three child rows: "row_aapl", "row_googl", and "row_amzn".
      • Each row is correctly set as a "Row" component.
      • Each row has three text children containing the correct stock symbols ("AAPL", "GOOGL", "AMZN"), prices ("$150.00", "$2800.00", "$3400.00"), and percentage changes ("+1.2%", "-0.5%", "+0.8%") as specified.

    Conclusion:
    The submission is structurally correct and implements all required elements. The only issues are minor cosmetic variations in the title's heading variant ("h2" instead of "h1") and the exact text content ("# Market Watch" instead of "Market Watch"). According to the grading guidelines, partial credit (P) is appropriate for correct answers with only minor cosmetic variations.

    GRADE: P

Evaluation Summary: a2ui_v0_9_1_eval

  • Status: FAIL
  • Model: google/gemini-3.5-flash
  • Pass Percentage: 89.09% (Threshold: 90.00%)

Dataset Performance Summary

Dataset Name Total Samples Algorithmic Pass Rate Judging Pass Rate Dataset Pass Rate
core_v0_9_1 51 92.2% 84.3% 78.4%
multi_turn_conversation_dataset 4 50.0% 100.0% 50.0%

Sample Results

Sample / Task Algorithmic Judging Inference Time Status
animalKingdomExplorer PASS C 255.85s PASS
calendarEventCreator PASS C 285.00s PASS
chatRoom PASS C 867.50s PASS
checkoutPage PASS C 142.83s PASS
cinemaSeatSelection PASS C 142.44s PASS
clientSideValidation PASS C 386.06s PASS
contactCard PASS C 139.56s PASS
contextAwareUI PASS C 3521.79s PASS
courseSyllabus PASS C 318.13s PASS
customYamlKeysTest PASS I 2100.18s FAIL
dashboard PASS C 118.51s PASS
deleteSurface FAIL C 814.68s FAIL
dogBreedGenerator PASS C 450.12s PASS
eCommerceProductPage PASS C 2192.94s PASS
fileBrowser PASS P 64.59s FAIL
fitnessTracker PASS C 65.07s PASS
flashcardApp PASS I 527.48s FAIL
flightBooker PASS C 318.03s PASS
hotelSearchResults PASS C 618.33s PASS
interactiveDashboard PASS C 1060.26s PASS
jobApplication PASS C 114.32s PASS
kanbanBoard PASS C 427.96s PASS
loginForm FAIL C 396.04s FAIL
mcpAppProxy FAIL C 97.27s FAIL
musicPlayer PASS C 620.72s PASS
nestedDataBinding PASS C 142.92s PASS
nestedLayoutRecursive PASS C 654.24s PASS
newsAggregator PASS C 363.03s PASS
notificationCenter PASS C 527.55s PASS
openUrlAction PASS C 746.09s PASS
photoEditor PASS C 118.77s PASS
podcastEpisode PASS P 608.00s FAIL
productGallery PASS C 427.93s PASS
productGalleryData PASS C 108.68s PASS
profileEditor PASS C 460.84s PASS
recipeCard PASS C 617.87s PASS
restaurantFinder FAIL I 1096.00s FAIL
restaurantMenu PASS C 882.25s PASS
rizzCharts PASS C 1329.37s PASS
settingsPage PASS C 551.25s PASS
simpleCalculator PASS C 3560.73s PASS
smartHome PASS P 321.56s FAIL
socialMediaPost PASS C 317.98s PASS
standardFunctions PASS C 475.10s PASS
stockWatchlist PASS P 319.55s FAIL
surveyForm PASS C 1934.89s PASS
travelItinerary PASS C 142.12s PASS
triviaQuiz PASS C 318.09s PASS
updateDataModel PASS C 461.20s PASS
videoCallInterface PASS C 654.25s PASS
weatherForecast PASS P 382.68s FAIL
banking_dispute_resolution FAIL C 359.98s FAIL
corporate_travel_multi_city_rebooking PASS C 1918.33s PASS
flight_booking_flow FAIL C 3596.30s FAIL
healthcare_patient_intake_triage PASS C 3562.51s PASS

Failure Details

customYamlKeysTest
  • Judging Failure Reason (Grade I):

    To determine whether the submitted answer meets the criterion, we need to evaluate if it satisfies all the components requested in the task.

    The task is to: "Generate a JSON message testing custom catalog, role_description and workflow_description."

    Let's inspect the submitted JSON messages:

    1. Custom Catalog: The submission attempts to test a custom catalog by specifying "catalogId": "https://a2ui.org/specification/v0_9/catalogs/basic/catalog.json" inside the createSurface payload.
    2. role_description: Upon reviewing the entire submission, there is no field named role_description (or roleDescription) in any of the JSON payloads.
    3. workflow_description: There is a generic "description" field nested inside a custom "workflow" data model object in Phase 2, but there is no explicit workflow_description (or workflowDescription) parameter tested as part of the surface creation or metadata definitions.

    Since the requested elements role_description and workflow_description are missing from the generated JSON payloads, the submission fails to fulfill the core requirements of the prompt.

    GRADE: I

deleteSurface
  • Algorithmic Failure Reason:

    Validation failed: {'version': 'v0.9', 'updateComponents': {'surfaceId': 'main', 'components': [{'id': 'root', 'component': 'Column', 'children': ['header-text', 'status-card'], 'spacing': 'medium', 'align': 'start'}, {'id': 'header-text', 'component': 'Text', 'text': 'System Notification', 'variant': 'h3'}, {'id': 'status-card', 'component': 'Card', 'child': 'status-container'}, {'id': 'status-container', 'component': 'Row', 'children': ['success-icon', 'success-message'], 'align': 'center'}, {'id': 'success-icon', 'component': 'Icon', 'name': 'check'}, {'id': 'success-message', 'component': 'Text', 'text': 'The surface dashboard-surface-1 has been successfully deleted.', 'variant': 'body'}]}} is not valid under any of the given schemas
    Context failures:

    • 'createSurface' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
    • {'id': 'root', 'component': 'Column', 'children': ['header-text', 'status-card'], 'spacing': 'medium', 'align': 'start'} is not valid under any of the given schemas
    • 'updateDataModel' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
    • 'deleteSurface' is a required property
    • Additional properties are not allowed ('updateComponents' was unexpected)
fileBrowser
  • Judging Failure Reason (Grade P):

    An analysis of the submission against the criterion:

    1. Valid A2UI payload with surfaceId 'main': The submission includes two A2UI payloads targeting the surfaceId 'main' (initializing the surface and then updating components). This matches the requirement.
    2. Text component (h1, "# My Files") followed by a List component:
      • The submission has a Text component with the correct text "# My Files". However, its variant is set to "h2" instead of the requested "h1". This is a minor cosmetic variation.
      • It is correctly followed by a List component (file-list).
    3. Three static Row components in the List:
      • The list contains three Row components: "row-documents", "row-images", and "row-work".
      • Each row contains an Icon and a Text component.
      • The "Documents" row correctly uses a "folder" icon and the text "Documents".
      • The "Images" row correctly uses a "folder" icon and the text "Images".
      • The "Work.txt" row correctly uses an "attachFile" icon and the text "Work.txt".

    The submission successfully implements all requested structural and content components of the file browser. The only discrepancy is the header component using the "h2" variant instead of the requested "h1". Per Note 7, this qualifies as a minor cosmetic variation, which warrants partial credit.

    GRADE: P

flashcardApp
  • Judging Failure Reason (Grade I):

    To evaluate the submission, let's compare it against the specified criterion step by step:

    1. Surface ID: The criterion requires a valid A2UI payload with surfaceId set to 'main'. The submission meets this requirement.
    2. Header Text: The criterion requires an 'h1' 'Text' with 'Spanish Vocabulary'. The submission has a component with ID header_text of type Text, text value "# Spanish Vocabulary", and variant "h1". This meets the requirement.
    3. Card Structure: The criterion requires a 'Card' containing a 'Column' that includes:
      • Front text ('Hola') - Present (card_front is inside the card_content column).
      • A 'Divider' - Present (card_divider is inside the card_content column).
      • Back text ('Hello') - Present (card_back is inside the card_content column).
      • A 'Row' of 'Button's ('Hard', 'Good', 'Easy') - Not met.

    In the submission, the 'Row' of buttons (action_row) is defined as a child of the top-level root column, placing it completely outside of the 'Card' component. According to the criterion, this row of buttons must be nested inside the 'Column' that is inside the 'Card'.

    Because this structural hierarchy is incorrect and places a major section of the UI outside of its specified container, it constitutes a substantive error.

    GRADE: I

loginForm
  • Algorithmic Failure Reason:

    Failed to parse JSON: Expecting value: line 1 column 1 (char 0)

mcpAppProxy
  • Algorithmic Failure Reason:

    A2UI tags '' and '' not found in response.

podcastEpisode
  • Judging Failure Reason (Grade P):

    To evaluate the submission, we will check each requirement of the criterion step-by-step:

    1. Surface ID: The submission correctly creates and updates a surface with the surfaceId of "main".
    2. Card Component: The root component is a "Card" which contains a "Column" layout containing all the other components. This successfully implements a card containing the player UI.
    3. Image (Cover Art): There is an "Image" component with the ID "cover_art".
    4. Title Text: There is a "Text" component with the value "Episode 42: The Future of AI". However, its variant is set to "h4" instead of the requested "h2" in the criterion (the original task did not specify a variant, but the criterion specifically looks for "h2").
    5. Host Text: There is a "Text" component with the value "Host: Jane Smith".
    6. Slider: There is a "Slider" component labeled "Progress" with its initial value set to 0.
    7. Buttons: There is a "Row" component containing three buttons:
      • A button containing the text "1x".
      • A button containing the text "Play/Pause".
      • A button containing the text "Share".
    8. Static Implementation: All components are static and not data-bound.

    The submission is almost entirely correct, but because it uses an "h4" heading variant instead of "h2" for the title text, it contains a minor cosmetic variation. Based on the grading guidelines, this qualifies for partial credit.

    GRADE: P

restaurantFinder
  • Algorithmic Failure Reason:

    Component 'card_8' references non-existent component 'card_col_8' in field 'child'

  • Judging Failure Reason (Grade I):

    To evaluate the submission based on the provided criterion, we must check if it follows the specific layout and data modeling rules:

    1. Data Model Population: The criterion specifies that the restaurant data must be populated in the data model under the path /items. In the submitted JSON, there is no data model initialization or population under /items. Instead, the restaurant details are hardcoded directly into the static component definitions (e.g., in res_title_1, res_desc_1, etc.).
    2. Template Usage (Restaurant Count): The task asks for 8 Chinese food spots. Since 8 is more than 5, the criterion explicitly states that the TWO_COLUMN_LIST_EXAMPLE template must be used. The submission instead uses a custom single-column vertical list with custom cards (card_1 to card_8) rather than the required TWO_COLUMN_LIST_EXAMPLE template.

    Because the submission fails both to bind the data under /items and to use the TWO_COLUMN_LIST_EXAMPLE template required for more than 5 items, it does not meet the specified criteria.

    GRADE: I

smartHome
  • Judging Failure Reason (Grade P):

    To evaluate the submission against the criterion, let's break down the requirements step-by-step:

    1. A2UI payload with surfaceId 'main':

      • The submission uses surfaceId: "main" in both the initialization and the update JSON blocks. (Passed)

    2. Define an h1 Text component with the text 'Living Room' (or '# Living Room'):

      • The component headingText is of type "Text", with "variant": "h1", and "text": "# Living Room". This matches the requested format in the task prompt. (Passed)

    3. Followed by a Column component containing two Row components:

      • In the submission, the root component is a single Column that contains the heading, followed by row1 and row2.
      • While the text and rows are combined into a single outer Column rather than having a separate Column specifically for the grid nested below the text, the layout functionally achieves the same visual result. All specified rows are present and structurally wrapped inside a column. (Passed with minor structural variation)

    4. First Row details:

      • Contains a Card for "Lights" with a CheckBox (label "Lights", value true). (Passed)
      • Contains a Card for "Thermostat" with a Slider (label "Thermostat", value 72). (Passed)

    5. Second Row details:

      • Contains a Card for "Music" with a CheckBox (label "Music", value false). (Passed)

    Conclusion:
    The submission meets all of the functional requirements, correctly maps the parent-child relationships for the cards and their interactive elements, and sets the appropriate labels and values. The only deviation is that the text header is nested inside the same column as the grid rows rather than being followed by a completely separate grid column. This is a minor structural variation that achieves the same output.

    GRADE: P

stockWatchlist
  • Judging Failure Reason (Grade P):

    To determine the grade for the submission, we will assess it step-by-step against the specified criterion:

    1. Surface ID: The criterion requires a valid A2UI payload with surfaceId 'main'. The submission correctly uses "surfaceId": "main" in both the createSurface and updateComponents sections.
    2. Title Text Component: The criterion requires a 'Market Watch' 'h1' 'Text' component.
      • The submission contains a Text component with the text "# Market Watch" and the variant "h2".
      • While the inclusion of the hashtag # can be considered a minor punctuation variation (which is acceptable), specifying the variant as "h2" instead of "h1" is a deviation from the explicit criterion requirement of 'h1'.
    3. List of Rows: The criterion requires a 'List' of 'Row's, each displaying a stock symbol, price, and percentage change as Text components.
      • The submission defines a List component (watchlist_list) containing three Row components (row_1, row_2, row_3).
      • Each row contains three Text components representing the correct symbols ("AAPL", "GOOGL", "AMZN"), prices ("$150.00", "$2800.00", "$3400.00"), and changes ("+1.2%", "-0.5%", "+0.8%") as specified in the task.

    Conclusion:
    The submission is highly accurate and implements the full structure correctly. However, it uses an "h2" variant instead of "h1" for the title text. Under the grading notes, this qualifies as a minor cosmetic variation, which warrants partial credit.

    GRADE: P

weatherForecast
  • Judging Failure Reason (Grade P):

    To assess whether the submission meets the criterion, we can break down the requirements and compare them to the submitted payload:

    1. Surface ID: The submission correctly creates and updates the surface with "surfaceId": "main".
    2. Title: It has a Text component (city_title) with the text "# New York".
    3. Current Weather Row: Below the title is a Row component (current_weather_row) containing:
      • A Text component (current_temp) with "68°F".
      • An Image component (current_icon) representing a sun weather icon.
    4. Divider: Below the current weather row is a Divider component (divider).
    5. List Component: Below the divider is a List component (forecast_list) containing 5 items, representing a 5-day forecast.
    6. List Items Structure: Each of the 5 list items is a Row (e.g., day1_row to day5_row) containing:
      • A Text component for the day (e.g., "Monday").
      • An Icon component representing the weather (e.g., "star"). The criterion specifies that this should be an 'Image' for the icon, but the submission uses the Icon component.
      • A Text component for high/low temperatures (e.g., "72° / 55°").

    Conclusion:
    The submission follows the overall structure and meets all the prompt requirements. The only deviation is that the submission uses Icon components instead of Image components for the 5-day forecast items (although it correctly uses an Image for the current weather icon). Because of this minor component type variation where the prompt requested "an icon" and the criterion expected an "Image", the submission qualifies for partial credit.

    GRADE: P

banking_dispute_resolution
  • Algorithmic Failure Reason:

    Validation failed: {'version': '0.9.1', 'createSurface': {'surfaceId': 'main', 'catalogId': 'https://a2ui.org/specification/v0_9/catalogs/basic/catalog.json', 'theme': {'primaryColor': '#0052CC', 'agentDisplayName': 'Acme Bank Fraud Assistant'}}} is not valid under any of the given schemas
    Context failures:

    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'updateComponents' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'updateDataModel' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'deleteSurface' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)
flight_booking_flow
  • Algorithmic Failure Reason:

    Validation failed: {'version': '0.9.1', 'createSurface': {'surfaceId': 'main', 'catalogId': 'https://a2ui.org/specification/v0_9/catalogs/basic/catalog.json', 'theme': {'primaryColor': '#1A5276', 'agentDisplayName': 'Acme Airlines Concierge'}}} is not valid under any of the given schemas
    Context failures:

    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'updateComponents' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'updateDataModel' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)
    • '0.9.1' is not one of ['v0.9', 'v0.9.1']
    • 'deleteSurface' is a required property
    • Additional properties are not allowed ('createSurface' was unexpected)

Metadata

Metadata

Assignees

No one assigned

    Labels

    eval_failureAdded to automated issues created by an eval run failure post-submit.status: needs-triageauto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs

    Type

    No type

    Projects

    A2UI
    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions