You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using TileDB-VCF on Windows 11:
tiledb 2.20.1 conda-forge
tiledbvcf-py 0.29.0 tiledb
when executing: ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"], threads=1) - see 'THREADS=1 - log' below, all fine
when executing: ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"]) - see 'THREADS>1 - log' below, CreateFile exception - file in use
Might have been resolved in version 0.36.0 but I see Windows is still on 0.29.0.
Any clues? Thanks!
THREADS=1 - log:
[2024-12-16 11:14:52.702] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.704] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Memory budget: total=48876 MiB = tiledb=4096 MiB + input=10 MiB + output=22385 MiB + stats=22385 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Input buffers = 1 threads * 10 samples * 1 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Output buffers = 1 threads * 22385 MiB (flush = 16788 MiB)
[2024-12-16 11:14:52.709] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.711] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.716] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Open array 'file:///C:/Play/data/vcf/tiledb/allele_count'
[2024-12-16 11:14:52.717] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Open array 'file:///C:/Play/data/vcf/tiledb/variant_stats'
[2024-12-16 11:14:54.025] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Initialization completed in 1.318 seconds.
[2024-12-16 11:14:54.566] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.100] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.546] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Output buffer records = 34381824
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Contig chr21: task size = 46.709.984 tasks = 1
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Current contig = Next contig =
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] WriteWorker4: parse chr21:0-46709983
[2024-12-16 11:14:56.507] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:14:56.508] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] VariantStats: new contig = chr21
[2024-12-16 11:15:30.625] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Recorded 2324491 cells from chr21:5030082-46699955 (task 1 / 1)
[2024-12-16 11:15:30.626] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: flushing 80602 records from chr21:5030087-46699312
[2024-12-16 11:15:30.664] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: flushing 2363718 records from chr21:5030082-46699946
[2024-12-16 11:15:31.350] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Worker 0: Drained 372 anchors.
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Ingestion rate = 65805.307 records/sec (VmRSS = NA)
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Finalizing last contig batch of [chr21, chr21]
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Finalize query with 80602 records
[2024-12-16 11:15:31.389] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/allele_count/__fragments/__1734344130626_1734344130626_04458c2356dd461cb1b4323eb00b4f3d_21 samples = HG00096
[2024-12-16 11:15:31.390] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Finalize query with 2363718 records
[2024-12-16 11:15:31.426] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/variant_stats/__fragments/__1734344130667_1734344130667_79989b5ec1db47568dddda8d999b03e9_21 samples = HG00096
[2024-12-16 11:15:31.427] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Write 372 anchor records.
[2024-12-16 11:15:31.516] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Making sure all finalize tasks completed...
[2024-12-16 11:15:31.596] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] All finalize tasks successfully completed. Waited for 0.0807968 sec.
[2024-12-16 11:15:31.597] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Close array
[2024-12-16 11:15:31.600] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Close array
[2024-12-16 11:15:31.603] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Done. Ingested 2.324.491 records (+ 372 anchors) from 1 samples in 38.897 seconds = 59760.7 records/sec
[2024-12-16 11:15:31.604] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] QACheck: [PASS] Total records ingested (2324491) == total records in VCF files (2324491)
Traceback (most recent call last):
File "c:\Play\app\vcf\main.py", line 44, in
ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"])
File "c:\Users\Peter Bossier\miniconda3\Lib\site-packages\tiledbvcf\dataset.py", line 858, in ingest_samples
self.writer.ingest_samples()
RuntimeError: TileDB-VCF exception: [TileDB::IO] Error: Cannot write to file 'C:\Play\data\vcf\tiledb\data__fragments__1734344230208_1734344230208_8141b7595cee4f91bea4e46761e1c315_21\a1.tdb'; File opening error CreateFile GetLastError 32 (0x00000020): The process cannot access the file because it is being used by another process.
Using TileDB-VCF on Windows 11:
tiledb 2.20.1 conda-forge
tiledbvcf-py 0.29.0 tiledb
Might have been resolved in version 0.36.0 but I see Windows is still on 0.29.0.
Any clues? Thanks!
THREADS=1 - log:
[2024-12-16 11:14:52.702] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.704] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Memory budget: total=48876 MiB = tiledb=4096 MiB + input=10 MiB + output=22385 MiB + stats=22385 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Input buffers = 1 threads * 10 samples * 1 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Output buffers = 1 threads * 22385 MiB (flush = 16788 MiB)
[2024-12-16 11:14:52.709] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.711] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.716] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Open array 'file:///C:/Play/data/vcf/tiledb/allele_count'
[2024-12-16 11:14:52.717] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Open array 'file:///C:/Play/data/vcf/tiledb/variant_stats'
[2024-12-16 11:14:54.025] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Initialization completed in 1.318 seconds.
[2024-12-16 11:14:54.566] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.100] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.546] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Output buffer records = 34381824
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Contig chr21: task size = 46.709.984 tasks = 1
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Current contig = Next contig =
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] WriteWorker4: parse chr21:0-46709983
[2024-12-16 11:14:56.507] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:14:56.508] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] VariantStats: new contig = chr21
[2024-12-16 11:15:30.625] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Recorded 2324491 cells from chr21:5030082-46699955 (task 1 / 1)
[2024-12-16 11:15:30.626] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: flushing 80602 records from chr21:5030087-46699312
[2024-12-16 11:15:30.664] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: flushing 2363718 records from chr21:5030082-46699946
[2024-12-16 11:15:31.350] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Worker 0: Drained 372 anchors.
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Ingestion rate = 65805.307 records/sec (VmRSS = NA)
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Finalizing last contig batch of [chr21, chr21]
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Finalize query with 80602 records
[2024-12-16 11:15:31.389] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/allele_count/__fragments/__1734344130626_1734344130626_04458c2356dd461cb1b4323eb00b4f3d_21 samples = HG00096
[2024-12-16 11:15:31.390] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Finalize query with 2363718 records
[2024-12-16 11:15:31.426] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/variant_stats/__fragments/__1734344130667_1734344130667_79989b5ec1db47568dddda8d999b03e9_21 samples = HG00096
[2024-12-16 11:15:31.427] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Write 372 anchor records.
[2024-12-16 11:15:31.516] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Making sure all finalize tasks completed...
[2024-12-16 11:15:31.596] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] All finalize tasks successfully completed. Waited for 0.0807968 sec.
[2024-12-16 11:15:31.597] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Close array
[2024-12-16 11:15:31.600] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Close array
[2024-12-16 11:15:31.603] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Done. Ingested 2.324.491 records (+ 372 anchors) from 1 samples in 38.897 seconds = 59760.7 records/sec
[2024-12-16 11:15:31.604] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] QACheck: [PASS] Total records ingested (2324491) == total records in VCF files (2324491)
THREADS>1 - log:
[2024-12-16 11:16:45.659] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.660] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.660] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.667] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.668] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.668] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.669] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Memory budget: total=48876 MiB = tiledb=4096 MiB + input=320 MiB + output=22230 MiB + stats=22230 MiB
[2024-12-16 11:16:45.670] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Input buffers = 32 threads * 10 samples * 1 MiB
[2024-12-16 11:16:45.670] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Output buffers = 32 threads * 694 MiB (flush = 521 MiB)
[2024-12-16 11:16:45.673] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.674] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.674] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.675] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.676] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.676] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.679] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Open array 'file:///C:/Play/data/vcf/tiledb/allele_count'
[2024-12-16 11:16:45.680] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Open array 'file:///C:/Play/data/vcf/tiledb/variant_stats'
[2024-12-16 11:16:46.983] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Initialization completed in 1.314 seconds.
[2024-12-16 11:16:47.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:47.886] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:48.334] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:48.786] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:49.243] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:49.706] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:50.158] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:50.614] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.064] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.524] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.985] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:52.450] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:52.912] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:53.364] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:53.825] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:54.278] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:54.740] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:55.223] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:55.681] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:56.140] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:56.587] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:57.041] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:57.493] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:58.509] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:59.059] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:59.529] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.000] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.464] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.938] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:01.398] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:01.888] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:02.343] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:02.805] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:03.294] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:03.763] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Output buffer records = 1067008
[2024-12-16 11:17:03.763] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Contig chr21: task size = 16.080.916 tasks = 3
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Current contig = Next contig = chr21
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] WriteWorker4: parse chr21:0-16080915
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] WriteWorker4: parse chr21:16080916-32161831
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] WriteWorker4: parse chr21:32161832-46709983
[2024-12-16 11:17:04.263] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:04.263] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:04.754] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:04.754] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:05.099] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:05.099] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:10.378] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Recorded 386659 cells from chr21:5030082-16080909 (task 1 / 3)
[2024-12-16 11:17:10.379] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: flushing 20775 records from chr21:5030087-16080745
[2024-12-16 11:17:10.387] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: flushing 397915 records from chr21:5030082-16080909
[2024-12-16 11:17:10.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Worker 0: Drained 26 anchors.
[2024-12-16 11:17:10.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Finalize query with 20775 records
[2024-12-16 11:17:10.466] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/allele_count/__fragments/__1734344230379_1734344230379_29df30bd8f634524a08aa5e9fb2e7eea_21 samples = HG00096
[2024-12-16 11:17:10.466] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Finalize query with 397915 records
[2024-12-16 11:17:10.494] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/variant_stats/__fragments/__1734344230388_1734344230388_f8e13ba5ecf14b5ea05a24153431405a_21 samples = HG00096
[2024-12-16 11:17:10.494] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Current contig = chr21 Next contig = chr21
[2024-12-16 11:17:18.917] [tiledb-vcf] [Process: 325032] [Thread: 321184] [error] TileDB-VCF exception: [TileDB::IO] Error: Cannot write to file 'C:\Play\data\vcf\tiledb\data__fragments__1734344230208_1734344230208_8141b7595cee4f91bea4e46761e1c315_21\a1.tdb'; File opening error CreateFile GetLastError 32 (0x00000020): The process cannot access the file because it is being used by another process.
Traceback (most recent call last):
File "c:\Play\app\vcf\main.py", line 44, in
ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"])
File "c:\Users\Peter Bossier\miniconda3\Lib\site-packages\tiledbvcf\dataset.py", line 858, in ingest_samples
self.writer.ingest_samples()
RuntimeError: TileDB-VCF exception: [TileDB::IO] Error: Cannot write to file 'C:\Play\data\vcf\tiledb\data__fragments__1734344230208_1734344230208_8141b7595cee4f91bea4e46761e1c315_21\a1.tdb'; File opening error CreateFile GetLastError 32 (0x00000020): The process cannot access the file because it is being used by another process.
[2024-12-16 11:17:18.929] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Close array
[2024-12-16 11:17:18.933] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Close array
The text was updated successfully, but these errors were encountered: