UnicodeDecodeError when writing fromfq output loom file

[marcRDM]

Hi,

I am trying to generate a .loom file out of 10X fastqs using the fromfq command. Everthing seems to run smoothly up to saving the loom file (everything prompted as expected, "INFO - Saving" included) but the file is never written because the following error is thrown:

File "[path_to_loompy]/loompy/normalize.py", line 98, in materialize_attr_values
result = np.array([html.unescape(x) for x in temp.astype(str)], dtype=object)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 45: ordinal not in range(128)

Could you please help me solve this?

Thank you in advance!

[slinnarsson]

Hi - can you paste the full stack trace? It would help me understand where the error comes from. It seems that an invalid ascii character has crept in somewhere.

[marcRDM]

Hi, thanks a lot! See below (this run is one I tried overnight on my own machine and I am getting the same error):

Loompy v3.0.6 by Linnarsson Lab 🌸 (http://linnarssonlab.org & http://loompy.org)

2020-11-20 02:37:22,853 - INFO - Using 4 threads.
2020-11-20 02:37:22,894 - INFO - kallisto bus -i /Volumes/UCL/2020_11_November/human_GRCh38_gencode.v31.600/gencode.v31.fragments.idx -o /var/folders/ly/m076l_vj491_vkxfq0gl53wm0000gq/T/tmp934h381t -x 10xv2 -t 4 /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L001_R1_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L001_R2_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L002_R1_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L002_R2_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L003_R1_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L003_R2_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L004_R1_001.fastq.gz /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L004_R2_001.fastq.gz
2020-11-20 02:37:22,985 - INFO - [index] k-mer length: 31
2020-11-20 02:37:22,986 - INFO - [index] number of targets: 845,338
2020-11-20 02:37:22,986 - INFO - [index] number of k-mers: 271,648,279
2020-11-20 02:38:37,761 - INFO - [index] number of equivalence classes: 4,776,424
2020-11-20 02:39:01,850 - INFO - [quant] will process sample 1: /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L001_R1_001.fastq.gz
2020-11-20 02:39:01,855 - INFO - /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L001_R2_001.fastq.gz
2020-11-20 02:39:01,857 - INFO - [quant] will process sample 2: /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L002_R1_001.fastq.gz
2020-11-20 02:39:01,857 - INFO - /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L002_R2_001.fastq.gz
2020-11-20 02:39:01,858 - INFO - [quant] will process sample 3: /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L003_R1_001.fastq.gz
2020-11-20 02:39:01,858 - INFO - /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L003_R2_001.fastq.gz
2020-11-20 02:39:01,859 - INFO - [quant] will process sample 4: /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L004_R1_001.fastq.gz
2020-11-20 02:39:01,859 - INFO - /Volumes/UCL/2020_11_November/H7NKJBGXH/G7_GEX_S1_L004_R2_001.fastq.gz
2020-11-20 07:23:29,414 - INFO - [quant] finding pseudoalignments for the reads ... done
2020-11-20 07:23:29,441 - INFO - [quant] processed 203,295,446 reads, 130,399,484 reads pseudoaligned
2020-11-20 07:24:12,043 - INFO - Loading gene metadata
2020-11-20 07:24:13,187 - INFO - Loading fragments-to-gene mappings
2020-11-20 07:24:13,933 - INFO - Indexing genes
2020-11-20 07:24:14,382 - INFO - Loading equivalence classes
2020-11-20 07:24:53,319 - INFO - Mapping equivalence classes to genes
2020-11-20 07:25:06,762 - INFO - Loading fragment IDs
2020-11-20 07:25:07,216 - INFO - Loading BUS records
2020-11-20 07:28:45,499 - INFO - Sorting cell IDs
2020-11-20 07:28:54,636 - INFO - Found 130,399,484 records for 60,662 genes and 1,420,742 uncorrected cell barcodes.
2020-11-20 07:28:54,636 - INFO - Correcting cell barcodes
2020-11-20 07:33:47,239 - INFO - Found 251,613 corrected cell barcodes.
2020-11-20 07:33:47,239 - INFO - Removing redundant reads using UMIs
2020-11-20 07:36:50,294 - INFO - 76% sequencing saturation.
2020-11-20 07:36:50,294 - INFO - Counting pseudoalignments for main matrix
2020-11-20 07:37:05,993 - INFO - Found 9,389,041 UMIs.
2020-11-20 07:37:07,340 - INFO - Counting pseudoalignments for layer 'unspliced'
2020-11-20 07:38:58,575 - INFO - Found 1,958,557 UMIs.
2020-11-20 07:38:59,518 - INFO - Counting pseudoalignments for layer 'spliced'
2020-11-20 07:41:37,531 - INFO - Found 8,163,222 UMIs.
2020-11-20 07:41:37,531 - INFO - Calling cells
2020-11-20 07:47:08,140 - INFO - Found 3269 valid cells and ~9677 ambient UMIs.
2020-11-20 07:47:08,140 - INFO - Creating loom file 'G7.loom'
2020-11-20 07:47:08,140 - INFO - Saving
Traceback (most recent call last):
File "/Users/Marc/anaconda3/bin/loompy", line 8, in
sys.exit(cli())
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/commands.py", line 34, in fromfq
create_from_fastq(loomfile, sampleid, list(fastqs), indexdir, metadatafile, threads)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/bus_file.py", line 455, in create_from_fastq
bus.save(out_file, sample_id, samples_metadata_file)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/bus_file.py", line 350, in save
create(out_file, layers, row_attrs, col_attrs, file_attrs=global_attrs)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/loompy.py", line 1073, in create
raise ve
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/loompy.py", line 1065, in create
ds.ra[key] = vals
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/attribute_manager.py", line 129, in setitem
return self.setattr(name, val)
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/attribute_manager.py", line 164, in setattr
self.dict["storage"][name] = loompy.materialize_attr_values(self.ds._file[a][name][:])
File "/Users/Marc/anaconda3/lib/python3.7/site-packages/loompy/normalize.py", line 98, in materialize_attr_values
result = np.array([html.unescape(x) for x in temp.astype(str)], dtype=object)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 45: ordinal not in range(128)

[slinnarsson]

Hm. Perhaps you have a non-standard unicode character in the metadata? Although I think it should still work (unless it's really invalid Unicode), at least with the latest loompy. Could you try with the latest version (not released) by cloning it to your computer?

git clone https://github.com/linnarsson-lab/loompy.git
cd loompy
pip install -e .

(don't miss the dot on the pip install)

[ksenia007]

I also have this error for 10X data, very similar stack trace. Updating to the latest version of loompy did not help. Might be relevant that my fastq is coming from sra-toolkit fastq-dump command. I'd be grateful if you could suggest some possible debugging steps?

[marcRDM]

Same here, my metadata looks fine and the latest loompy did not fix it. I am getting this error both on my personal machine (macOS) and on my university servers (classic academic HPC structure).

[t-whalley]

Hi, I also seem to be having this error on 10X data. I am on macOS. The log is as follows:

(base) SRVs-MBP-2$ loompy fromfq combined.loom combined /full/path/to/transcriptome  metadata.tab fastqs/combined_R1.fastq fastqs/combined_R2.fastq 

Loompy v3.0.6 by Linnarsson Lab 🌸 (http://linnarssonlab.org & http://loompy.org)


2020-12-15 10:50:47,792 - INFO - Using 12 threads.

2020-12-15 10:50:47,794 - INFO - kallisto bus -i /Users/barbaraszomolay/SSC/valentina/new/human_GRCh38_gencode.v31.600/gencode.v31.fragments.idx -o /var/folders/g6/z6sfy45n7xlfjb3zmmppp3nm0000gq/T/tmpf0uc440f -x 10xv2 -t 12 fastqs/combined_R1.fastq fastqs/combined_R2.fastq

2020-12-15 10:50:47,821 - INFO - [index] k-mer length: 31

2020-12-15 10:50:47,821 - INFO - [index] number of targets: 845,338

2020-12-15 10:50:47,821 - INFO - [index] number of k-mers: 271,648,279

2020-12-15 10:51:11,814 - INFO - [index] number of equivalence classes: 4,776,424

2020-12-15 10:51:29,113 - INFO - [quant] will process sample 1: fastqs/combined_R1.fastq

2020-12-15 10:51:29,113 - INFO -                                fastqs/combined_R2.fastq

2020-12-15 10:56:44,815 - INFO - [quant] finding pseudoalignments for the reads ... done

2020-12-15 10:56:44,817 - INFO - [quant] processed 259,909,533 reads, 23,250,908 reads pseudoaligned

2020-12-15 10:57:12,332 - INFO - Loading gene metadata

2020-12-15 10:57:12,631 - INFO - Loading fragments-to-gene mappings

2020-12-15 10:57:13,171 - INFO - Indexing genes

2020-12-15 10:57:13,640 - INFO - Loading equivalence classes

2020-12-15 10:57:40,304 - INFO - Mapping equivalence classes to genes

2020-12-15 10:57:52,483 - INFO - Loading fragment IDs

2020-12-15 10:57:52,885 - INFO - Loading BUS records

2020-12-15 10:58:27,400 - INFO - Sorting cell IDs

2020-12-15 10:58:28,675 - INFO - Found 23,250,908 records for 60,662 genes and 347,392 uncorrected cell barcodes.

2020-12-15 10:58:28,675 - INFO - Correcting cell barcodes

2020-12-15 10:59:20,239 - INFO - Found 191,518 corrected cell barcodes.

2020-12-15 10:59:20,239 - INFO - Removing redundant reads using UMIs

2020-12-15 10:59:42,944 - INFO - 83% sequencing saturation.

2020-12-15 10:59:42,944 - INFO - Counting pseudoalignments for main matrix

2020-12-15 10:59:44,747 - INFO - Found 1,663,207 UMIs.

2020-12-15 10:59:45,717 - INFO - Counting pseudoalignments for layer 'unspliced'

2020-12-15 11:00:37,785 - INFO - Found 700,661 UMIs.

2020-12-15 11:00:38,460 - INFO - Counting pseudoalignments for layer 'spliced'

2020-12-15 11:01:15,244 - INFO - Found 1,056,841 UMIs.

2020-12-15 11:01:15,244 - INFO - Calling cells

2020-12-15 11:01:15,272 - WARNING - No ambient RNA beads were found; maybe sample had too few cells?

2020-12-15 11:01:15,318 - INFO - Found 661 valid cells and ~5 ambient UMIs.

2020-12-15 11:01:15,318 - INFO - Creating loom file 'combined.loom'

2020-12-15 11:01:15,318 - INFO - Saving

Traceback (most recent call last):

  File "/Users/bar/miniconda3/bin/loompy", line 8, in <module>

    sys.exit(cli())

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__

    return self.main(*args, **kwargs)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/click/core.py", line 782, in main

    rv = self.invoke(ctx)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke

    return callback(*args, **kwargs)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/commands.py", line 34, in fromfq

    create_from_fastq(loomfile, sampleid, list(fastqs), indexdir, metadatafile, threads)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/bus_file.py", line 455, in create_from_fastq

    bus.save(out_file, sample_id, samples_metadata_file)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/bus_file.py", line 350, in save

    create(out_file, layers, row_attrs, col_attrs, file_attrs=global_attrs)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/loompy.py", line 1073, in create

    raise ve

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/loompy.py", line 1065, in create

    ds.ra[key] = vals

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/attribute_manager.py", line 129, in __setitem__

    return self.__setattr__(name, val)

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/attribute_manager.py", line 164, in __setattr__

    self.__dict__["storage"][name] = loompy.materialize_attr_values(self.ds._file[a][name][:])

  File "/Users/bar/miniconda3/lib/python3.8/site-packages/loompy/normalize.py", line 98, in materialize_attr_values

    result = np.array([html.unescape(x) for x in temp.astype(str)], dtype=object)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 45: ordinal not in range(128)

I'm not sure if the low number of UMIs is an issue, but I suspected that even if there was little output or none at all it would still write without a codec error. As above, I've tried it with the latest version from this repo. I've also tried export LC_ALL=C with no luck. I also converted the metadata explicitly to ascii.

Thanks

[marcRDM]

Hi @slinnarsson,

Happy new year! Would you have another suggestion to try to solve this issue?

Thanks

[slinnarsson]

Hi - sorry it's difficult to troubleshoot without a reproducible example. Would you be able to share a minimal input that reliable produces the error?

[1000-Monkeys]

Hi @slinnarsson,
I'm also having a very similar problem (MacOS and tested with the 10x 1kPBMC dataset using the directions here: https://linnarssonlab.org/loompy/kallisto/index.html). I've updated loompy and double checked the metadata.tab file and still the same issue. Thank you for your help!

(python) (base) MacBook-Pro-3:Loompy cgantner$ loompy fromfq 1kPBMC.loom 1kPBMC human_GRCh38_gencode.v31 metadata.tab pbmc_1k_v3_S1_L001_R1_001.fastq.gz pbmc_1k_v3_S1_L001_R2_001.fastq.gz pbmc_1k_v3_S1_L002_R1_001.fastq.gz pbmc_1k_v3_S1_L002_R2_001.fastq.gz
Loompy v3.0.6 by Linnarsson Lab 🌸 (http://linnarssonlab.org & http://loompy.org)

2021-01-20 10:09:45,504 - INFO - Using 16 threads.
2021-01-20 10:09:45,505 - INFO - kallisto bus -i human_GRCh38_gencode.v31/gencode.v31.fragments.idx -o /var/folders/21/9x15pwf93jl68p3_wbfnqkgw0000gn/T/tmplif7fkeu -x 10xv3 -t 16 pbmc_1k_v3_S1_L001_R1_001.fastq.gz pbmc_1k_v3_S1_L001_R2_001.fastq.gz pbmc_1k_v3_S1_L002_R1_001.fastq.gz pbmc_1k_v3_S1_L002_R2_001.fastq.gz
2021-01-20 10:09:45,521 - INFO - [index] k-mer length: 31
2021-01-20 10:09:45,521 - INFO - [index] number of targets: 845,338
2021-01-20 10:09:45,521 - INFO - [index] number of k-mers: 271,648,279
2021-01-20 10:10:03,111 - INFO - [index] number of equivalence classes: 4,776,424
2021-01-20 10:10:15,415 - INFO - [quant] will process sample 1: pbmc_1k_v3_S1_L001_R1_001.fastq.gz
2021-01-20 10:10:15,416 - INFO -                                pbmc_1k_v3_S1_L001_R2_001.fastq.gz
2021-01-20 10:10:15,416 - INFO - [quant] will process sample 2: pbmc_1k_v3_S1_L002_R1_001.fastq.gz
2021-01-20 10:10:15,416 - INFO -                                pbmc_1k_v3_S1_L002_R2_001.fastq.gz
2021-01-20 10:12:42,785 - INFO - [quant] finding pseudoalignments for the reads ... done
2021-01-20 10:12:42,785 - INFO - [quant] processed 66,601,887 reads, 47,826,751 reads pseudoaligned
2021-01-20 10:13:01,841 - INFO - Loading gene metadata
2021-01-20 10:13:02,036 - INFO - Loading fragments-to-gene mappings
2021-01-20 10:13:02,428 - INFO - Indexing genes
2021-01-20 10:13:02,804 - INFO - Loading equivalence classes
2021-01-20 10:13:27,790 - INFO - Mapping equivalence classes to genes
2021-01-20 10:13:37,344 - INFO - Loading fragment IDs
2021-01-20 10:13:37,631 - INFO - Loading BUS records
2021-01-20 10:14:47,244 - INFO - Sorting cell IDs
2021-01-20 10:14:49,375 - INFO - Found 47,826,751 records for 60,662 genes and 572,948 uncorrected cell barcodes.
2021-01-20 10:14:49,375 - INFO - Correcting cell barcodes
2021-01-20 10:16:41,054 - INFO - Found 312,447 corrected cell barcodes.
2021-01-20 10:16:41,054 - INFO - Removing redundant reads using UMIs
2021-01-20 10:17:35,538 - INFO - 71% sequencing saturation.
2021-01-20 10:17:35,538 - INFO - Counting pseudoalignments for main matrix
2021-01-20 10:17:41,384 - INFO - Found 5,105,024 UMIs.
2021-01-20 10:17:41,995 - INFO - Counting pseudoalignments for layer 'unspliced'
2021-01-20 10:18:54,570 - INFO - Found 2,503,881 UMIs.
2021-01-20 10:18:55,083 - INFO - Counting pseudoalignments for layer 'spliced'
2021-01-20 10:19:57,419 - INFO - Found 3,278,570 UMIs.
2021-01-20 10:19:57,419 - INFO - Calling cells
2021-01-20 10:22:53,581 - INFO - Found 1173 valid cells and ~14436 ambient UMIs.
2021-01-20 10:22:53,581 - INFO - Creating loom file '1kPBMC.loom'
2021-01-20 10:22:53,581 - INFO - Saving
Traceback (most recent call last):
  File "/Users/cgantner/Documents/R/Loompy/python/bin/loompy", line 8, in <module>
    sys.exit(cli())
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/commands.py", line 34, in fromfq
    create_from_fastq(loomfile, sampleid, list(fastqs), indexdir, metadatafile, threads)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/bus_file.py", line 455, in create_from_fastq
    bus.save(out_file, sample_id, samples_metadata_file)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/bus_file.py", line 350, in save
    create(out_file, layers, row_attrs, col_attrs, file_attrs=global_attrs)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/loompy.py", line 1073, in create
    raise ve
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/loompy.py", line 1065, in create
    ds.ra[key] = vals
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/attribute_manager.py", line 129, in __setitem__
    return self.__setattr__(name, val)
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/attribute_manager.py", line 164, in __setattr__
    self.__dict__["storage"][name] = loompy.materialize_attr_values(self.ds._file[a][name][:])
  File "/Users/cgantner/Documents/R/Loompy/python/lib/python3.8/site-packages/loompy/normalize.py", line 98, in materialize_attr_values
    result = np.array([html.unescape(x) for x in temp.astype(str)], dtype=object)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 45: ordinal not in range(128)
(python) (base) MacBook-Pro-3:Loompy cgantner$

[sam-mccachren]

I'm also having the identical issue, with identical stack trace, while using the 10x 1kPBMC dataset. Has anyone found a solution/workaround for this? Thanks!

[ElinorWing]

Hello,

I am also having this issue using the 10X 1kPBMC dataset. Has anyone managed to find a solution?

Thanks!

[KatarinaYuan]

Hi,
I am also having this issue using the 10X 1kPBMC dataset. Has anyone managed to find a solution?
Thank you!