Unable to deploy AWS budget dashboard - Column 'collection_time' cannot be resolved

[work-bw]

Trying to deploy the Budgets dashboard via cid-cmd deploy. I need some help resolving the installation issue please. Thanks.

Brian

=======================

? [dashboard-id] Please select a dashboard to deploy:   [aws-budgets] AWS Budgets
Dashboard "aws-budgets" is not deployed

Required datasets: 
 - budgets_view


Looking by DataSetId defined in template...complete

There are still 1 datasets missing: budgets_view
Creating dataset: budgets_view

? [athena-workgroup] Select Amazon Athena workgroup to use: CID

? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): CID-CMD-Athena <CREATE NEW DATASOURCE>

? [quicksight-datasource-role] Please choose a QuickSight role. It must have access to Athena: CidCmdQuickSightDataSourceRole

? [athena-database] Select AWS Athena database to use: athenacurcfn_nelnet_cudos
Role CidCmdQuickSightDataSourceRole was updated. https://console.aws.amazon.com/iam/home?#/roles/details/CidCmdQuickSightDataSourceRole
ERROR - Data source already exists {'Arn': 'arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena', 'DataSourceId': 'CID-CMD-Athena', 'Name': 'CID-CMD-Athena', 'Type': 'ATHENA', 'Status': 'CREATION_SUCCESSFUL', 'CreatedTime': datetime.datetime(2023, 7, 3, 19, 3, 21, 937000, tzinfo=tzlocal()), 'LastUpdatedTime': datetime.datetime(2023, 7, 3, 19, 3, 23, 202000, tzinfo=tzlocal()), 'DataSourceParameters': {'AthenaParameters': {'WorkGroup': 'primary'}}, 'SslProperties': {'DisableSsl': False}}
athena_datasource <cid.helpers.quicksight.datasource.Datasource object at 0x7fafd68835e0>
Detected views: account_map
Missing views: budgets_view
CRITICAL - Athena query failed: line 12:3: Column 'collection_time' cannot be resolved

====================

When I'm prompted to select the quicksight-datasource-id my only choice is to create new datasource. It doesn't seem to see the existing datasources.

? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): (Use arrow keys)
 » CID-CMD-Athena <CREATE NEW DATASOURCE>

[iakov-aws]

Hello,

Collection time must be there in table : https://github.com/awslabs/cid-framework/blob/main/data-collection/deploy/module-budgets.yaml#L165

Can you check the version of data collection (in description of the stack) and update to the latest https://catalog.workshops.aws/awscid/en-US/data-collection/update

[iakov-aws]

Brian @work-bw
Regarding datasources, can you check if datasource status are ok?

some what like this in cloud shell:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
aws quicksight list-data-sources  --aws-account-id $AWS_ACCOUNT_ID --query 'DataSources[?Type==`ATHENA`]'

[work-bw]

I updated the data collection stack to v3.4.3 as I was on an old stack previously. collection_time is not in the table, is that something I need to manually add? If so, how? Can you send me some instructions please?

I think my data sources are okay but I still get the same error trying to install the Budgets dashboard after updating the data collection stack. However, if I need to wait a day for collection_time to be added, it has not been a day yet.

Brian

[cloudshell-user@ip-10-142-89-131 ~]$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
[cloudshell-user@ip-10-142-89-131 ~]$ aws quicksight list-data-sources  --aws-account-id $AWS_ACCOUNT_ID --query 'DataSources[?Type==`ATHENA`]'
[
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/12693f2e-cd40-4123-ab1d-a0bfadc17ea3",
        "DataSourceId": "12693f2e-cd40-4123-ab1d-a0bfadc17ea3",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-09-25T18:42:08.644000+00:00",
        "LastUpdatedTime": "2019-09-25T18:42:08.644000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/3654057d-3d37-4188-ba9c-82815371ee8e",
        "DataSourceId": "3654057d-3d37-4188-ba9c-82815371ee8e",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-08-15T18:55:42.047000+00:00",
        "LastUpdatedTime": "2019-08-15T18:55:42.047000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/7b112161-8cc9-451a-8b9e-37130c693252",
        "DataSourceId": "7b112161-8cc9-451a-8b9e-37130c693252",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2020-02-06T20:49:19.556000+00:00",
        "LastUpdatedTime": "2020-02-06T20:49:19.556000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/a9344e2a-f64a-4784-8965-558886369da9",
        "DataSourceId": "a9344e2a-f64a-4784-8965-558886369da9",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-09-25T18:43:42.738000+00:00",
        "LastUpdatedTime": "2019-09-25T18:43:42.738000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/4c587d46-19a8-4cf7-91c6-2f25a55207d1",
        "DataSourceId": "4c587d46-19a8-4cf7-91c6-2f25a55207d1",
        "Name": "cod2",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2023-07-03T18:33:23.121000+00:00",
        "LastUpdatedTime": "2023-07-03T18:48:37.334000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena",
        "DataSourceId": "CID-CMD-Athena",
        "Name": "CID-CMD-Athena",
        "Type": "ATHENA",
        "Status": "CREATION_SUCCESSFUL",
        "CreatedTime": "2023-07-03T19:03:21.937000+00:00",
        "LastUpdatedTime": "2023-07-03T19:03:23.202000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    }
]

[work-bw]

I should also add that I'm getting this same error when I tried to reinstall the trusted advisor organizational dashboard. I deleted the dashboard and when I tried to re-deploy it I get the same error. It always prompts to create a new CID-CMD-Athena datasource and it already exists and throws an error. This looks like a bug reading existing datasources. Thanks.

Brian

? [dashboard-id] Please select a dashboard to deploy:   [ta-organizational-view] Trusted Advisor Organizational View

Latest template: arn:aws:quicksight:us-east-1:223485597511:template/ta-organizational-view/version/19
Dashboard "ta-organizational-view" is not deployed

Required datasets: 
 - ta-organizational-view


Looking by DataSetId defined in template...complete

There are still 1 datasets missing: ta-organizational-view
Creating dataset: ta-organizational-view

? [athena-workgroup] Select Amazon Athena workgroup to use: CID

`? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): CID-CMD-Athena <CREATE NEW DATASOURCE>`

? [quicksight-datasource-role] Please choose a QuickSight role. It must have access to Athena: CidCmdQuickSightDataSourceRole

? [athena-database] Select AWS Athena database to use: athenacurcfn_nelnet_cudos
Role CidCmdQuickSightDataSourceRole was updated. https://console.aws.amazon.com/iam/home?#/roles/details/CidCmdQuickSightDataSourceRole
ERROR - Data source already exists {'Arn': 'arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena', 'DataSourceId': 'CID-CMD-Athena', 'Name': 'CID-CMD-Athena', 'Type': 'ATHENA', 'Status': 'CREATION_SUCCESSFUL', 'CreatedTime': datetime.datetime(2023, 7, 3, 19, 3, 21, 937000, tzinfo=tzlocal()), 'LastUpdatedTime': datetime.datetime(2023, 7, 3, 19, 3, 23, 202000, tzinfo=tzlocal()), 'DataSourceParameters': {'AthenaParameters': {'WorkGroup': 'primary'}}, 'SslProperties': {'DisableSsl': False}}
athena_datasource <cid.helpers.quicksight.datasource.Datasource object at 0x7fcdc1d1c610>
Detected views: 
Missing views: ta_org_view
Missing dependency view: ta_organizational_view_reports, creating
{'account_id': '704691252361'}

? [view-ta-organizational-view-reports-s3FolderPath] Required parameter: s3FolderPath (Trusted Advisor Organization View report S3 path): s3://cid-data-704691252361/tr
usted-advisor/trusted-advisor-data

? [timezone] Please select timezone for datasets scheduled refresh.: America/Chicago
Dataset "ta-organizational-view" created
Deploying dashboard ta-organizational-view

#######
####### Congratulations!
####### Trusted Advisor Organizational View is available at: https://us-west-2.quicksight.aws.amazon.com/sn/dashboards/ta-organizational-view
#######

[work-bw]

Can you let me know the status of this issue please or the next steps? I tried checking the latest cid-cmd updates and I still get the same error trying to deploy the TAO or Budget dashboards. Is this an issue you are able to recreate? Is there other data I can provide to help troubleshoot? Thank you.

Brian

[iakov-aws]

Hi Brain,
I contacted your TAM. I suggest one of our team members to have a call with you to make sure all issues got fixed.

[VoicuAWS]

I see you selected the workgroup CID but none of the datasources are created under workgroups CID, that is why none show up. The existing "CID-CMD-Athena" datasource is under "primary" workgroup per your output above, this is why it is prompting you to create one. If you select primary in the workgroup selection step, it should pickup exiting "CID-CMD-Athena" datasource.

What I am not sure though is why the creation of the new datasource fails. When we try to create it it should use the "CID" workgroup, but the error we see in your log says it is using "primary". We need to understand why that happens. Are you using the latest cid-cmd tool? Logs would help here, but we can discuss more in our upcoming call.
I tried to repro but I did not get the error.