Security-Hub Get-Findings returning 'list' Error

[q00td]

Describe the bug

Hello !

I wanted to output security-hub data to a json file.
When i export my findings from the us-east-1 region everything works well,

Command used :
aws securityhub get-findings --region <eu-west-1/us-east-1>

But when i try to export the findings from the eu-west-1 region i get an error :
'list' object has no attribute 'items'.

I tried to debug it but with no success.
For example, i can use --max-items 1 --region eu-west-1 and i can retrieve 1 Entry, but never the full data

Have a nice day.
Q.

Expected Behavior

Output the full data from the eu-west-1 region.

Current Behavior

Function call crash with output : 'list' object has no attribute 'items'.

Reproduction Steps

Prerequisites for reproduction :

• Security-hub with findings, configured in two regions.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.12.1

Environment details (OS name and version, etc.)

Windows 10

[q00td]

Added :

I also managed to extract 1000 entries, i'm guessing at some point it's just too much data

[RyanFitzSimmonsAK]

Hi @Quentin90, thanks for reaching out. Could you provide debug logs of this behavior? You can get debug logs by adding --debug to your input, and redacting any sensitive information. Thanks!

[q00td]

Hello again !

Here is the output with --debug, sorry but i had to redact most of the data inside the body and headers.

content-type;host;x-amz-date;x-amz-security-token
2023-06-21 09:32:48,430 - MainThread - botocore.auth - DEBUG - StringToSign: Redacted
AWS4-HMAC-SHA256
2023-06-21 09:32:48,430 - MainThread - botocore.auth - DEBUG - Signature: Redacted
2023-06-21 09:32:48,430 - MainThread - botocore.endpoint - DEBUG - Sending http request: Redacted
2023-06-21 09:32:48,431 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
2023-06-21 09:32:48,622 - MainThread - urllib3.connectionpool - DEBUG - [https://securityhub.eu-west-1.amazonaws.com:443](https://securityhub.eu-west-1.amazonaws.com/) "POST /findings HTTP/1.1" 200 87179
2023-06-21 09:32:48,627 - MainThread - botocore.parsers - DEBUG - Response headers: Redacted
2023-06-21 09:32:48,628 - MainThread - botocore.parsers - DEBUG - Response body: Redacted

2023-06-21 09:32:48,656 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli\clidriver.py", line 460, in main
  File "awscli\clidriver.py", line 595, in __call__
  File "awscli\clidriver.py", line 798, in __call__
  File "awscli\clidriver.py", line 931, in invoke
  File "awscli\clidriver.py", line 953, in _display_response
  File "awscli\formatter.py", line 77, in __call__
  File "awscli\botocore\paginate.py", line 446, in build_full_result
  File "awscli\botocore\paginate.py", line 252, in __iter__
  File "awscli\botocore\paginate.py", line 329, in _make_request
  File "awscli\botocore\client.py", line 341, in _api_call
  File "awscli\botocore\client.py", line 683, in _make_api_call
  File "awscli\botocore\client.py", line 703, in _make_request
  File "awscli\botocore\endpoint.py", line 101, in make_request
  File "awscli\botocore\endpoint.py", line 133, in _send_request
  File "awscli\botocore\endpoint.py", line 165, in _get_response
  File "awscli\botocore\endpoint.py", line 219, in _do_get_response
  File "awscli\botocore\parsers.py", line 248, in parse
  File "awscli\botocore\parsers.py", line 864, in _do_parse
  File "awscli\botocore\parsers.py", line 873, in _add_modeled_parse
  File "awscli\botocore\parsers.py", line 914, in _parse_payload
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 651, in _handle_structure
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 969, in _handle_list
  File "awscli\botocore\parsers.py", line 330, in _handle_list
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 651, in _handle_structure
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 651, in _handle_structure
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 651, in _handle_structure
  File "awscli\botocore\parsers.py", line 322, in _parse_shape
  File "awscli\botocore\parsers.py", line 660, in _handle_map
AttributeError: 'list' object has no attribute 'items'


'list' object has no attribute 'items'

Using --max-items 18000, we also had this warning :
'charmap' codec can't encode character '\u2192' in position 681: character maps to <undefined>

Thank you for your time.

[q00td]

Update : We tried to export the file and load it using powershell and we were prompted with the error :

'charmap' codec can't encode character '\u2192' in position 681: character maps to <undefined>
ConvertFrom-Json:<file>

Command used :

$file= Get-Content .\findings.json | ConvertFrom-Json

We are guessing that there are some characters not handled properly when outputing to json (which is default i thinki)

[q00td]

We finally got a hold on the bug,

It's not related to the size, only a special charater..
We made a script to dump findings 500 by 500 but we always crash on the same batch, with the error :
'charmap' codec can't encode character '\u2192' in position 681: character maps to <undefined>

We can't bypass this error as the result of this command contain the next "starting-token".
We also can't encode/decode the unicode character as the command crash before giving any results.

Regards

[RyanFitzSimmonsAK]

Hi @Quentin90, thanks for following up. I agree with your conclusion that this is related to a specific character. Could you tell me if that error is found in the response body, or if the response looks fine and the error shows up between response and output? Once I know that, I'll raise this issue with the appropriate team (service team or CLI team).

[q00td]

Hello @RyanFitzSimmonsAK , No problem,

So here is two output for the same command, the first one is a working one, and the second one is where the bug is happening.
From my believe it is just before sending the request :

• Working one :

2023-07-03 08:05:16,868 - MainThread - botocore.hooks - DEBUG - Event needs-retry.securityhub.GetFindings: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000011CBD931B50>>
2023-07-03 08:05:16,869 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-07-03 08:05:16,869 - MainThread - botocore.hooks - DEBUG - Event after-call.securityhub.GetFindings: calling handler <bound method RetryQuotaChecker.release_retry_quota of 
<botocore.retries.standard.RetryQuotaChecker object at 0x0000011CBD931610>>
2023-07-03 08:05:16,869 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'eu-west-1', 'UseDualStack': False, 'UseFIPS': False}    
2023-07-03 08:05:16,870 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://securityhub.eu-west-1.amazonaws.com

after the output above, the command continues and i see requests configs and then the header/body

And for the bugged one :

2023-07-03 08:05:17,184 - MainThread - botocore.hooks - DEBUG - Event needs-retry.securityhub.GetFindings: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000011CBD931B50>>
2023-07-03 08:05:17,185 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-07-03 08:05:17,185 - MainThread - botocore.hooks - DEBUG - Event after-call.securityhub.GetFindings: calling handler <bound method RetryQuotaChecker.release_retry_quota of 
<botocore.retries.standard.RetryQuotaChecker object at 0x0000011CBD931610>>
2023-07-03 08:05:17,251 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli\clidriver.py", line 460, in main
  File "awscli\clidriver.py", line 595, in __call__
  File "awscli\clidriver.py", line 798, in __call__
  File "awscli\clidriver.py", line 931, in invoke
  File "awscli\clidriver.py", line 953, in _display_response
  File "awscli\formatter.py", line 83, in __call__
  File "awscli\formatter.py", line 102, in _format_response
  File "json\__init__.py", line 180, in dump
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position 681: character maps to <undefined>

'charmap' codec can't encode character '\u2192' in position 681: character maps to <undefined

Have a nice day !

[RyanFitzSimmonsAK]

Hi @Quentin90, thanks for following up. Those logs didn't quite get what I'm after though. When the bug happens, take a look at the response body, and tell me if it looks normal, or if the response is an error. Thanks!

[q00td]

Hi again @RyanFitzSimmonsAK, There is no body nor other logs after the error, the CLI stop with this error.

Regards,
Q.

[RyanFitzSimmonsAK]

Hi @Quentin90, thanks for your patience. I was able to reproduce this behavior. It looks like this is a general bug with the CLI, rather than Security Hub specific.

Steps to reproduce

1. Create an EC2 instance with the character '→' in the name.
2. Run aws ec2 describe-instances

I'll bring this issue up with the team and leave this issue open to track any updates. Thanks again for raising this.

[amberkushwaha]

describe the bug in the process of the following log in circuit of the file.Attach files in the process for the log connectivity.also mention the circuit criteria for it.
Command used :
aws securityhub get-findings --region <eu-west-1/us-east-1>

[RyanFitzSimmonsAK]

The AWS CLI uses the encoding provided by the system. From #8283, easiest solution to this is changing the system locale and setting the environment variable PYTHONUTF8=1.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.