Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve feedback when there's a validation error #1469

Open
kabilar opened this issue Jul 24, 2024 · 1 comment
Open

Improve feedback when there's a validation error #1469

kabilar opened this issue Jul 24, 2024 · 1 comment
Labels
UX

Comments

@kabilar
Copy link
Member

kabilar commented Jul 24, 2024

Reposting a suggestion that @dstansby provided on the LINC Client (lincbrain#44):

I'm currently trying to upload a dataset, and if there's a validation error it's a bit tricky to understand what went wrong and how I can fix it. The output looks like this:

PATH                                                      SIZE        ERRORS     PROGRESS STATUS           MESSAGE                  
dandiset.yaml                                             1.1 kB                          skipped          should be edited online  
dataset_description.json                                  98 Bytes       2                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json     65 Bytes       1                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.ome.zarr                2                ERROR            failed validation        
Summary:                                                  1.3 kB   3 with errors          1 skipped        1 should be edited online
                                                                                          3 ERROR          3 failed validation      
2024-04-25 13:44:09,332 [ WARNING] One or more assets failed validation.  Consult the logfile for details.
2024-04-25 13:44:09,861 [ WARNING] Upload failed, and you are not using the latest version of lincbrain-cli.  We suggest upgrading lincbrain-cli to v0.9.0 and trying again.
2024-04-25 13:44:09,864 [    INFO] Logs saved in /Users/dstansby/Library/Logs/lincbrain-cli/20240425124405Z-15907.log
Error: failed validation

If I then go into the log, I see messages like

024-04-25T13:44:09+0100 [WARNING ] dandi 15907:6207451136  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), dataset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), message='File does not match any pattern known to BIDS.', metadata=None, path=PosixPath('/Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json'), path_regex=None)
2024-04-25T13:44:09+0100 [ERROR   ] dandi 15907:6207451136 Error uploading /Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json:
Example full log 
2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 lincbrain v0.5.0+108.g40a8c76b.dirty, hdmf v3.13.0, pynwb v2.6.0, h5py v3.11.0
2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 sys.argv = ['C:\\Users\\dstansby\\miniforge3\\envs\\linc\\Scripts\\lincbrain', 'upload']
2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 os.getcwd() = C:\Users\dstansby\data\linc\000005
2024-05-02T14:40:51+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/linc/linc-cli HTTP/1.1" 404 44
2024-05-02T14:40:52+0100 [DEBUG   ] dandi 23264:16516 Could not check linc/linc-cli for version updates: 404 Client Error: Not Found for url: https://rig.mit.edu/et/projects/linc/linc-cli
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371
2024-05-02T14:40:52+0100 [DEBUG   ] dandi 23264:16516 GET https://api.lincbrain.org/api/info/
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Response: 200
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Using api key from LINCBRAIN_API_KEY environment variable
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 GET https://api.lincbrain.org/api/auth/token
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token HTTP/1.1" 301 0
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token/ HTTP/1.1" 200 42
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Response: 200
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Found identifier DANDI:000005 in top level 'identifier'
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 7 to 5
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 5 to 7
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 7 to 5
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 5 to 7
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'zlib'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'gzip'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'bz2'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'lzma'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'blosc'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'zstd'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'lz4'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'astype'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'delta'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'quantize'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'fixedscaleoffset'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'packbits'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'categorize'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'pickle'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'base64'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'shuffle'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'bitround'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'crc32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'adler32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'jenkins_lookup3'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'json2'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-utf8'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-bytes'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-array'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'fletcher32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'n5_wrapper'
2024-05-02T14:40:54+0100 [INFO    ] dandi 23264:16516 Found 6 files to consider
2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 No BIDS reference root provided.
2024-05-02T14:40:55+0100 [INFO    ] bids-schema 23264:17416 No schema path specified, defaulting to the bundled schema, `C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\bidsschematools\data\schema`.
2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 The `C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr/` file was not matched by any regex schema entry.
2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416 'C:\\Users\\dstansby\\data\\linc\\000005\\dataset_description.json' had 1 validation errors preventing its upload:
2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:40:56+0100 [ERROR   ] dandi 23264:17416 Error uploading C:\Users\dstansby\data\linc\000005\dataset_description.json:
Traceback (most recent call last):
  File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path
    raise UploadError("failed validation")
lincbrain.exceptions.UploadError: failed validation
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:22488 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\samples.tsv
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:30128 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\participants.tsv
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Running original <function get_digest at 0x0000025AF771B7E0> on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json')
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_dandietag at 0x0000025AF77187C0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 Starting new HTTPS connection (2): api.lincbrain.org:443
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Running original <function get_dandietag at 0x0000025AF77187C0> on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json')
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:29792 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 Starting new HTTPS connection (3): api.lincbrain.org:443
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=samples.tsv HTTP/1.1" 200 274
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 Response: 200
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/ HTTP/1.1" 200 1911
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 Response: 200
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=rawdata%2Fsub-S45%2Fsub-S45_sample-fullhemisphere_PC.json HTTP/1.1" 200 52
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=participants.tsv HTTP/1.1" 200 278
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/
2024-05-02T14:40:57+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_dandietag at 0x0000025AF77187C0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Calculated dandi-etag of df515b2ad6855b779e78e4ec58fdd98f-1 for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Beginning upload
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/initialize/
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/ HTTP/1.1" 200 1915
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 200 649
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json in 1 parts
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Uploading part 1/1 (104 bytes)
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:17160 PUT https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:17160 Starting new HTTPS connection (1): linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:17160 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "PUT /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9 HTTP/1.1" 200 0
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:17160 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Part upload finished ETag=6a057fca28c6d756dae00c6160579d31 Content-Length=0
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Completing upload
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/ HTTP/1.1" 200 768
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Announcing completion to https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "POST /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1 HTTP/1.1" 200 None
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Upload completed. Response content: b'<?xml version="1.0" encoding="UTF-8"?>\n\n<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs%2F082%2Fcda%2F082cda3f-a994-4728-bf34-8108ab122bed</Location><Bucket>linc-brain-mit-prod-us-east-2</Bucket><Key>blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed</Key><ETag>"df515b2ad6855b779e78e4ec58fdd98f-1"</ETag></CompleteMultipartUploadResult>'
2024-05-02T14:40:58+0100 [DEBUG   ] charset_normalizer 23264:29792 Encoding detection: utf_8 is most likely the one.
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/ HTTP/1.1" 200 119
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Assigning asset blob to dandiset & version
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/dandisets/000005/versions/draft/assets/ HTTP/1.1" 200 2182
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [INFO    ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Asset successfully uploaded
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616 'C:\\Users\\dstansby\\data\\linc\\000005\\rawdata\\sub-S45\\sub-S45_sample-fullhemisphere_PC.ome.zarr' had 2 validation errors preventing its upload:
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616  Error 2: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:41:05+0100 [ERROR   ] dandi 23264:1616 Error uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.ome.zarr:
Traceback (most recent call last):
  File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path
    raise UploadError("failed validation")
lincbrain.exceptions.UploadError: failed validation
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:16516 One or more assets failed validation.  Consult the logfile for details.
2024-05-02T14:41:05+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443
2024-05-02T14:41:06+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/lincbrain/linc-cli HTTP/1.1" 200 578
2024-05-02T14:41:06+0100 [WARNING ] dandi 23264:16516 Upload failed, and you are not using the latest version of lincbrain-cli.  We suggest upgrading lincbrain-cli to v0.9.0 and trying again.
2024-05-02T14:41:06+0100 [DEBUG   ] dandi 23264:16516 Caught exception failed validation
2024-05-02T14:41:06+0100 [INFO    ] dandi 23264:16516 Logs saved in C:\Users\dstansby\AppData\Local\lincbrain\lincbrain-cli\Logs\20240502134051Z-23264.log

These messages are hard to parse by eye, and even then the error is just "File does not match any pattern known to BIDS". It would be super useful if there was some hint (maybe at least a link to the BIDS spec?) on how the file pattern can be fixed.

Related issue:

  1. Bundle/interface JS validators for bids and zarr #1397

Thank you

cc @dstansby @aaronkanzer
@kabilar kabilar mentioned this issue Jul 24, 2024
Closed
@yarikoptic
Copy link
Member

These messages are hard to parse by eye, and even then the error is just "File does not match any pattern known to BIDS". It would be super useful if there was some hint (maybe at least a link to the BIDS spec?) on how the file pattern can be fixed.

I don't think there is an easy generic way to address this reliably. Only may be for some cases to look at what entities used and either all are known and suggest closest matching legit ones if some nonlegit ones used.

May be if you provide some specific suggestions, we could see if could be coded up.

@yarikoptic yarikoptic added the UX label Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
UX
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yarikoptic @kabilar