Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TileDB-VCF exception CreateFile during ingest_samples when threads>1 on Windows version 0.29.0 #795

Open
bossiep1 opened this issue Dec 16, 2024 · 0 comments

Comments

@bossiep1
Copy link

Using TileDB-VCF on Windows 11:
tiledb 2.20.1 conda-forge
tiledbvcf-py 0.29.0 tiledb

  • when executing: ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"], threads=1) - see 'THREADS=1 - log' below, all fine
  • when executing: ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"]) - see 'THREADS>1 - log' below, CreateFile exception - file in use

Might have been resolved in version 0.36.0 but I see Windows is still on 0.29.0.
Any clues? Thanks!

THREADS=1 - log:

[2024-12-16 11:14:52.702] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.703] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.704] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.705] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Memory budget: total=48876 MiB = tiledb=4096 MiB + input=10 MiB + output=22385 MiB + stats=22385 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Input buffers = 1 threads * 10 samples * 1 MiB
[2024-12-16 11:14:52.707] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Output buffers = 1 threads * 22385 MiB (flush = 16788 MiB)
[2024-12-16 11:14:52.709] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.710] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:14:52.711] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.712] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:14:52.716] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Open array 'file:///C:/Play/data/vcf/tiledb/allele_count'
[2024-12-16 11:14:52.717] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Open array 'file:///C:/Play/data/vcf/tiledb/variant_stats'
[2024-12-16 11:14:54.025] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Initialization completed in 1.318 seconds.
[2024-12-16 11:14:54.566] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.100] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:55.546] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Output buffer records = 34381824
[2024-12-16 11:14:56.006] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Contig chr21: task size = 46.709.984 tasks = 1
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Current contig = Next contig =
[2024-12-16 11:14:56.027] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] WriteWorker4: parse chr21:0-46709983
[2024-12-16 11:14:56.507] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:14:56.508] [tiledb-vcf] [Process: 326516] [Thread: 308532] [debug] VariantStats: new contig = chr21
[2024-12-16 11:15:30.625] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Recorded 2324491 cells from chr21:5030082-46699955 (task 1 / 1)
[2024-12-16 11:15:30.626] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: flushing 80602 records from chr21:5030087-46699312
[2024-12-16 11:15:30.664] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: flushing 2363718 records from chr21:5030082-46699946
[2024-12-16 11:15:31.350] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Worker 0: Drained 372 anchors.
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Ingestion rate = 65805.307 records/sec (VmRSS = NA)
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Finalizing last contig batch of [chr21, chr21]
[2024-12-16 11:15:31.351] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Finalize query with 80602 records
[2024-12-16 11:15:31.389] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/allele_count/__fragments/__1734344130626_1734344130626_04458c2356dd461cb1b4323eb00b4f3d_21 samples = HG00096
[2024-12-16 11:15:31.390] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Finalize query with 2363718 records
[2024-12-16 11:15:31.426] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/variant_stats/__fragments/__1734344130667_1734344130667_79989b5ec1db47568dddda8d999b03e9_21 samples = HG00096
[2024-12-16 11:15:31.427] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Write 372 anchor records.
[2024-12-16 11:15:31.516] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] Making sure all finalize tasks completed...
[2024-12-16 11:15:31.596] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] All finalize tasks successfully completed. Waited for 0.0807968 sec.
[2024-12-16 11:15:31.597] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] AlleleCount: Close array
[2024-12-16 11:15:31.600] [tiledb-vcf] [Process: 326516] [Thread: 320468] [debug] VariantStats: Close array
[2024-12-16 11:15:31.603] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] Done. Ingested 2.324.491 records (+ 372 anchors) from 1 samples in 38.897 seconds = 59760.7 records/sec
[2024-12-16 11:15:31.604] [tiledb-vcf] [Process: 326516] [Thread: 320468] [info] QACheck: [PASS] Total records ingested (2324491) == total records in VCF files (2324491)

THREADS>1 - log:

[2024-12-16 11:16:45.659] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.660] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.660] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.667] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.668] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.668] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.669] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Memory budget: total=48876 MiB = tiledb=4096 MiB + input=320 MiB + output=22230 MiB + stats=22230 MiB
[2024-12-16 11:16:45.670] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Input buffers = 32 threads * 10 samples * 1 MiB
[2024-12-16 11:16:45.670] [tiledb-vcf] [Process: 325032] [Thread: 321184] [info] Output buffers = 32 threads * 694 MiB (flush = 521 MiB)
[2024-12-16 11:16:45.673] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='data' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.674] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='data' uri='file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.674] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'data' using group uri 'file:///C:/Play/data/vcf/tiledb/data'
[2024-12-16 11:16:45.675] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Lookup member name='vcf_headers' in group uri='data/vcf/tiledb'
[2024-12-16 11:16:45.676] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Found group member name='vcf_headers' uri='file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.676] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Open 'vcf_headers' using group uri 'file:///C:/Play/data/vcf/tiledb/metadata/vcf_headers'
[2024-12-16 11:16:45.679] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Open array 'file:///C:/Play/data/vcf/tiledb/allele_count'
[2024-12-16 11:16:45.680] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Open array 'file:///C:/Play/data/vcf/tiledb/variant_stats'
[2024-12-16 11:16:46.983] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Initialization completed in 1.314 seconds.
[2024-12-16 11:16:47.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:47.886] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:48.334] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:48.786] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:49.243] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:49.706] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:50.158] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:50.614] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.064] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.524] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:51.985] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:52.450] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:52.912] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:53.364] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:53.825] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:54.278] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:54.740] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:55.223] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:55.681] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:56.140] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:56.587] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:57.041] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:57.493] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:58.509] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:59.059] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:16:59.529] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.000] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.464] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:00.938] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:01.398] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:01.888] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:02.343] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:02.805] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:03.294] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Reading VCF header vfs://s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz
[2024-12-16 11:17:03.763] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Output buffer records = 1067008
[2024-12-16 11:17:03.763] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Contig chr21: task size = 16.080.916 tasks = 3
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Current contig = Next contig = chr21
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] WriteWorker4: parse chr21:0-16080915
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] WriteWorker4: parse chr21:16080916-32161831
[2024-12-16 11:17:03.783] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] WriteWorker4: parse chr21:32161832-46709983
[2024-12-16 11:17:04.263] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:04.263] [tiledb-vcf] [Process: 325032] [Thread: 325108] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:04.754] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:04.754] [tiledb-vcf] [Process: 325032] [Thread: 321988] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:05.099] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] AlleleCount: new contig = chr21
[2024-12-16 11:17:05.099] [tiledb-vcf] [Process: 325032] [Thread: 310844] [debug] VariantStats: new contig = chr21
[2024-12-16 11:17:10.378] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Recorded 386659 cells from chr21:5030082-16080909 (task 1 / 3)
[2024-12-16 11:17:10.379] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: flushing 20775 records from chr21:5030087-16080745
[2024-12-16 11:17:10.387] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: flushing 397915 records from chr21:5030082-16080909
[2024-12-16 11:17:10.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Worker 0: Drained 26 anchors.
[2024-12-16 11:17:10.440] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Finalize query with 20775 records
[2024-12-16 11:17:10.466] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/allele_count/__fragments/__1734344230379_1734344230379_29df30bd8f634524a08aa5e9fb2e7eea_21 samples = HG00096
[2024-12-16 11:17:10.466] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Finalize query with 397915 records
[2024-12-16 11:17:10.494] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: fragment_num = 1 uri = file:///C:/Play/data/vcf/tiledb/variant_stats/__fragments/__1734344230388_1734344230388_f8e13ba5ecf14b5ea05a24153431405a_21 samples = HG00096
[2024-12-16 11:17:10.494] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] Current contig = chr21 Next contig = chr21
[2024-12-16 11:17:18.917] [tiledb-vcf] [Process: 325032] [Thread: 321184] [error] TileDB-VCF exception: [TileDB::IO] Error: Cannot write to file 'C:\Play\data\vcf\tiledb\data__fragments__1734344230208_1734344230208_8141b7595cee4f91bea4e46761e1c315_21\a1.tdb'; File opening error CreateFile GetLastError 32 (0x00000020): The process cannot access the file because it is being used by another process.

Traceback (most recent call last):
File "c:\Play\app\vcf\main.py", line 44, in
ds.ingest_samples(sample_uris=["s3://tiledb-inc-demo-data/examples/notebooks/vcfs/1kg-dragen/HG00096_chr21.gvcf.gz"])
File "c:\Users\Peter Bossier\miniconda3\Lib\site-packages\tiledbvcf\dataset.py", line 858, in ingest_samples
self.writer.ingest_samples()
RuntimeError: TileDB-VCF exception: [TileDB::IO] Error: Cannot write to file 'C:\Play\data\vcf\tiledb\data__fragments__1734344230208_1734344230208_8141b7595cee4f91bea4e46761e1c315_21\a1.tdb'; File opening error CreateFile GetLastError 32 (0x00000020): The process cannot access the file because it is being used by another process.

[2024-12-16 11:17:18.929] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] AlleleCount: Close array
[2024-12-16 11:17:18.933] [tiledb-vcf] [Process: 325032] [Thread: 321184] [debug] VariantStats: Close array

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant