forked from samtools/htslib
-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathNEWS
2523 lines (1818 loc) · 102 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Noteworthy changes in release a.b
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Noteworthy changes in release 1.21 (12th September 2024)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The primary user-visible changes in this release are updates to the
annot-tsv tool and some speed improvements. Full details of other
changes and bugs fixed are below.
Notice: this is the last SAMtools / HTSlib release where CRAM 3.0 will be
the default CRAM version. From the next we will change to CRAM 3.1
unless the version is explicitly specified, for example using
"samtools view -O cram,version=3.0".
Updates
-------
* Extend annot-tsv with several new command line options.
--delim permits use of other delimiters.
--headers for selection of other header formats.
--no-header-idx to suppress column index numbers in header.
Also removed -h as it is now short for --headers. Note --help
still works. (PR #1779)
* Allow annot-tsv -a to rename annotations. (PR #1709)
* Extend annot-tsv --overlap to be able to specify the overlap
fraction separately for source and target. (PR #1811)
* Added new APIs to facilitate low-level CRAM container manipulations,
used by the new "samtools cat" region filtering code. Functions are:
cram_container_get_coords()
cram_filter_container()
cram_index_extents()
cram_container_num2offset()
cram_container_offset2num()
cram_num_containers()
cram_num_containers_between()
Also improved cram_index_query() to cope with HTS_IDX_NOCOOR regions.
(PR #1771)
* Bgzip now retains file modification and access times when
compressing and decompressing. (PR #1727, fixes #1718. Requested by
Gert Hulselmans.)
* Use FNV1a for string hashing in khash. The old algorithm was
particularly weak with base-64 style strings and lead to a large
number of collisions. (PR #1806. Fixes samtools/samtools#2066,
reported by Hans-Joachim Ruscheweyh)
* Improve the speed of the nibble2base() function on Intel (PR
#1667, PR #1764, PR #1786, PR #1802, thanks to Ruben Vorderman) and
ARM (PR #1795, thanks to John Marshall).
* bgzf_getline() will now warn if it encounters UTF-16 data.
(PR #1487, thanks to John Marshall)
* Speed up bgzf_read(). While this does not reduce CPU significantly,
it does increase the maximum parallelism available permitting 10-15%
faster decoding. (PR #1772, PR #1800, Issue #1798)
* Speed up faidx by use of better isgraph methods (PR #1797) and
whole-line reading (PR #1799, thanks to John Marshall).
* Speed up kputll() function, speeding up BAM -> SAM conversion by
about 5% and also samtools depth. (PR #1805)
* Added more example code, covering fasta/fastq indexing, tabix
indexing and use of the thread pool. (PR #1666)
Build Changes
-------------
* Code warning fixes for pedantic compilers (PR #1777) and avoid
some undefined behaviour (PR #1810, PR #1816, PR #1828).
* Windows based CI has been migrated from AppVeyor to GitHub Actions.
(PR #1796, PR #1803, PR #1808)
* Miscellaneous minor build infrastructure and code fixes.
(PR #1807, PR #1829, both thanks to John Marshall)
* Updated htscodecs submodule to version 1.6.1 (PR #1828)
* Fixed an awk script in the Makefile that only worked with gawk. (PR #1831)
Bug fixes
---------
* Fix small OSS-Fuzz reported issues with CRAM encoding and long
CIGARS and/or illegal positions. (PR #1775, PR #1801, PR #1817)
* Fix issues with on-the-fly indexing of VCF/BCF (bcftools --write-index)
when not using multiple threads. (PR #1837. Fixes samtools/bcftools#2267,
reported by Giulio Genovese)
* Stricter limits on POS / MPOS / TLEN in sam_parse1(). This fixes
a signed overflow reported by OSS-Fuzz and should help prevent other
as-yet undetected bugs. (PR #1812)
* Check that the underlying file open worked for preload: URLs. Fixes
a NULL pointer dereference reported by OSS-Fuzz. (PR #1821)
* Fix an infinite loop in hts_itr_query() when given extremely large
positions which cause integer overflow. Also adds hts_bin_maxpos()
and hts_idx_maxpos() functions.
(PR #1774, thanks to John Marshall and reported by Jesus Alberto
Munoz Mesa)
* Fix an out of bounds read in hts_itr_multi_next() when switching
chromosomes. This bug is present in releases 1.11 to 1.20.
(PR #1788. Fixes samtools/samtools#2063, reported by acorvelo)
* Work around parsing problems with colons in CHROM names.
Fixes samtools/bcftools#2139. (PR #1781, John Marshall / James Bonfield)
* Correct the CPU detection for Mac OS X 10.7. cpuid is used by
htscodecs (see samtools/htscodecs#116), and the corresponding
changes in htslib are PR #1785. Reported by Ryan Carsten Schmidt.
* Make BAM zero-length intervals work the same as CRAM; permitted and
returning overlapping records. (PR #1787. Fixes
samtools/samtools#2060, reported by acorvelo)
* Replace assert() with abort() in BCF synced reader. This is not an
ideal solution, but it gives consistent behaviour when compiling
with or without NDEBUG. (PR #1791, thanks to Martin Pollard)
* Fixed failure to change the write block size on compressed SAM or VCF
files due to an internal type confusion. (PR #1826)
* Fixed an out-of-bounds read in cram_codec_iter_next() (PR #1832)
Noteworthy changes in release 1.20 (15th April 2024)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Updates
-------
* When working on named files, bgzip now sets the modified and access times
of the output files it makes to match those of the corresponding input.
(PR #1727, feature request #1718. Requested by Gert Hulselmans)
* It's now possible to use a -o option to specify the output file name in
bgzip.
(PR #1747, feature request #1726. Requested by Gert Hulselmans)
* Improved error faidx error messages.
(PR #1743, thanks to Nick Moore)
* Faster reading of SAM array (type "B") tags. These often turn up
in ONT and PacBio data.
(PR #1741)
* Improved validity checking of base modification tags.
(PR #1749)
* mpileup overlap removal now works where one read has a deletion.
(PR #1751, fixes samtools/samtools#1992. Reported by Long Tian)
* The S3 plugin can now find buckets via S3 access point aliases.
(PR #1756, thanks to Matt Pawelczyk;
fixes samtools/samtools#1984. Reported by Albert Li)
* Added a --threads option (and -@ short option) to tabix.
(PR #1755, feature request #1735. Requested by Dan Bolser)
* tabix can now index Graph Alignment Format (GAF) files.
(See https://github.com/lh3/gfatools/blob/master/doc/rGFA.md)
(PR #1763, thanks to Adam Novak)
Bug fixes
---------
* Security fix: Prevent possible heap overflow in cram_encode_aux() on
bad RG:Z tags.
(PR #1737)
* Security fix: Prevent attempts to call a NULL pointer if certain URL
schemes are used in CRAM @SQ UR: tags.
(PR #1757)
* Security fix: Fixed a bug where following certain AWS S3 redirects could
downgrade the connection from TLS (i.e. https://) to unencrypted http://.
This could happen when using path-based URLs and AWS_DEFAULT_REGION
was set to a region other that the one where the data was stored.
(PR #1762, fixes #1760. Reported by andaca)
* Fixed arithmetic overflow when loading very long references for CRAM.
(PR #1738, fixes #1738. Reported by Shane McCarthy)
* Fixed faidx and CRAM reference look-ups on compressed fasta where the .fai
index file was present, but the .gzi index of compressed offsets was not.
(PR #1745, fixes #1744. Reported by Theodore Li)
* Fixed BCF indexing on-the-fly bug which produced invalid indexes when
using multiple compression threads.
(PR #1742, fixes #1740. Reported by graphenn)
* Ensure that pileup destructors are called by bam_plp_destroy(), to
prevent memory leaks.
(PR #1749, PR #1754)
* Ensure on-the-fly index timestamps are always older than the data file.
Previously the files could be closed out of order, leading to warnings
being printed when using the index.
(PR #1753, fixes #1732. Reported by Gert Hulselmans)
* To prevent data corruption when reading (strictly invalid) VCF files
with duplicated FORMAT tags, all but the first copy of the data
associated with the tag are now dropped with a warning.
(PR #1752, PR #1761, fixes #1733. Reported by anthakki)
* Fixed a bug introduced in release 1.19 (PR #1689) which broke variant
record data if it tried to remove an over-long tag.
(PR #1752, PR #1761)
* Changed error to warning when complaining about use of the CG tag
in SAM or CRAM files.
(PR #1758, fixes samtools/samtools#2002)
Noteworthy changes in release 1.19.1 (22nd January 2024)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Fixed a regression in release 1.19 that caused all aux records to
be stored uncompressed in CRAM files. The resulting files were
correctly formatted, but bigger than they needed to be.
(PR#1729, fixes samtools#1968. Reported by Clockris)
* Fixed possible out-of-bounds reads due to an incorrect check on
B tag lengths in cram_encode_aux(). (PR#1725)
* Fixed an incorrect check on tag length which could fail to catch a
two byte out-of-bounds read in bam_get_aux(). (PR#1728)
* Made errors reported by hts_open_format() less confusing when it can't
open the reference file. (PR#1724, fixes #1723. Reported by
Alex Leonard)
* Made hts_close() fail more gracefully if it's passed a NULL pointer
(PR#1724)
Noteworthy changes in release 1.19 (12th December 2023)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Updates
-------
* A temporary work-around has been put in the VCF parser so that it is
less likely to fail on rows with a large number of ALT alleles,
where Number=G tags like PL can expand beyond the 2Gb limit enforced
by HTSlib. For now, where this happens the offending tag will be dropped
so the data can be processed, albeit without the likelihood data.
In future work, the library will instead convert such tags into their
local alternatives (see https://github.com/samtools/hts-specs/pull/434).
(PR #1689)
* New program. Adds annot-tsv which annotates regions in a destination file with
texts from overlapping regions in a source file.
(PR#1619)
* Change bam_parse_cigar() so that it can modify existing BAM records. This
makes more useful as public API. Previously it could only handle partially
formed BAM records.
(PR#1651, fixes #1650. Reported by Oleksii Nikolaienko)
* Add "uncompressed" to hts_format_description() where appropriate. This adds
an "uncompressed" description to uncompressed files that would normally be
compressed, such as BAM and BCF.
(PR#1656, in relation to samtools#1884. Thanks to John Marshall)
* Speed up to the VCF parser and writer.
(PR#1644 and PR#1663)
* Add an hclen (hard clip length) SAM filter function.
(PR#1660, with reference to samtools#813)
* Avoid really closing stdin/stdout in hclose()/hts_close()/et al.
See discussion in PR for details.
(PR#1665. Thanks to John Marshall)
* Add support to handle multiple files in bgzip.
(PR#1658, fixes #1642. Requested by bw2)
* Enable auto-vectorisation in CRAM 3.1 codecs. Speeds decoding on some
sequencing platform data.
(PR#1669)
* Speed up removal of lines in large headers.
(PR#1662, fixes #1460. Reported by Anže Starič)
* Apply seqtk PR to improve kseq.h parsing performance. Port of
Fabian Klötzl's (kloetzl) lh3/seqtk#123 and attractivechaos/klib#173 to
HTSlib.
(PR#1674. Thanks to John Marshall)
Build changes
-------------
* Updated htscodecs submodule to 1.6.0.
(PR#1685, PR#1717, PR#1719)
* Apply the packed attribute to uint*_u types for Clang to prevent
-fsanitize=alignment failures.
(PR#1667. Thanks to Fangrui Song)
* Fuzz testing improvements.
(PR#1664)
* Add C++ casts for external headers in klist.h and kseq.h.
(PR#1683. See also PR#1674 and PR#1682)
* Add test case compiling the public headers as C++.
(PR#1682. Thanks to John Marshall)
* Enable optimisation level -O3 for SAM QUAL+33 formatting.
(PR#1679)
* Make compiler flag detection work with zig cc.
(PR#1687)
* Fix unused value warnings when built with NDEBUG.
(PR#1688)
* Remove some disused Makefile variables, fix typos and a warning. Improve
bam_parse_basemod() documentation.
(PR#1705, Thanks to John Marshall)
Bug fixes
---------
* Fail bgzf_useek() when offset is above block limits.
(PR#1668)
* Fix multi-threaded on-the-fly indexing problems.
(PR#1672, fixes samtools#1861 and bcftools#1985. Reported by Mark Ebbert and
lacek)
* Fix hfile_libcurl small seek bug.
(PR#1676, fixes samtools#1918. Also may fix #1037, #1625 and samtools#1622.
Reported by Alex Reynolds, Mark Walker, Arthur Gilly and skatragadda-nygc.
Thanks to John Marshall)
* Fix a minor memory leak in malformed CRAM EXTERNAL blocks. [fuzz]
(PR#1671)
* Fix a cram decode hang from block_resize().
(PR#1680. Reported by Sebastian Deorowicz)
* Cram fuzzing improvements. Fixes a number of cram errors.
(PR#1701, fixes #1691, #1692, #1693, #1696, #1697, #1698, #1699 and #1700.
Thanks to Octavio Galland for finding and reporting all these)
* Fix crypt4gh redirection.
(PR#1675, fixes grbot/crypt4gh-tutorial#2. Reported by hth4)
* Fix PG header linking when records make a loop.
(PR#1702, fixes #1694. Reported by Octavio Galland)
* Prevent issues with no-stored-sequence records in CRAM files, by ensuring
they are accounted for properly in block size calculations, and by limiting
the maximum query length in the CIGAR data. Originally seen as an overflow
by OSS-Fuzz / UBSAN, it turned out this could lead to excessive time and
memory use by HTSlib, and could result in it writing out unreadable CRAM
files.
(PR#1710)
* Fix some illegal shifts and integer overflows found by OSS-Fuzz / UBSAN.
(PR#1707, PR#1712, PR#1713)
Noteworthy changes in release 1.18 (25th July 2023)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Updates
-------
* Using CRAM 3.1 no longer gives a warning about the specification
being draft. Note CRAM 3.0 is still the default output format.
(PR#1583)
* Replaced use of sprintf with snprintf, to silence potential warnings
from Apple's compilers and those who implement similar checks.
(PR#1594, fixes #1586. Reported by Oleksii Nikolaienko)
* Fastq output will now generate empty records for reads with no
sequence data (i.e. sequence is "*" in SAM format). (PR#1576,
fixes samtools/samtools#1576. Reported by Nils Homer)
* CRAM decoding speed-ups. (PR#1580)
* A new MN aux tag can now be used to verify that MM/ML base modification
data has not been broken by hard clipping. (PR#1590, PR#1612. See also
PR samtools/hts-specs#714 and issue samtools/hts-specs#646.
Reported by Jared Simpson)
* The base modification API has been improved to make it easier for callers
to tell unchecked bases from unmodified ones. (PR#1636, fixes #1550.
Requested by Chris Wright)
* A new bam_mods_queryi() API has been added to return additional
data about the i-th base modification returned by bam_mods_recorded().
(PR#1636, fixes #1550 and #1635. Requested by Jared Simpson)
* Speed up index look-ups for whole-chromosome queries. (PR#1596)
* Mpileup now merges adjacent (mis)match CIGAR operations, so CIGARs
using the X/= operators give the same results as if the M operator
was used. (PR#1607, fixes #1597. Reported by Marcel Martin)
* It's now possible to call bcf_sr_set_regions() after adding readers
using bcf_sr_add_reader() (previously this returned an error). Doing so
will discard any unread data, and reset the readers so they iterate over
the new regions. (PR#1624, fixes samtools/bcftools#1918. Reported by
Gregg Thomas)
* The synced BCF reader can now accept regions with reference names including
colons and hyphens, by enclosing them in curly braces. For example,
{chr_part:1-1001}:10-20 will return bases 10 to 20 from reference
"chr_part:1-1001". (PR#1630, fixes #1620. Reported by Bren)
* Add a "samples" directory with code demonstrating usage of HTSlib plus
a tutorial document. (PR#1589)
Build changes
-------------
* Htscodecs has been updated to 1.5.1 (PR#1654)
* Htscodecs SIMD code now works with Apple multiarch binaries.
(PR#1587, HTSlib fix for samtools/htscodecs#76. Reported by John Marshall)
* Improve portability of "expr" usage in version.sh.
(PR#1593, fixes #1592. Reported by John Marshall)
* Improve portability to *BSD targets by ensuring _XOPEN_SOURCE is defined
correctly and that source files properly include "config.h". Perl
scripts also now all use #!/usr/bin/env instead of assuming that
it's in /usr/bin/perl. (PR#1628, fixes #1606.
Reported by Robert Clausecker)
* Fixed NAME entry in htslib-s3-plugin man page so the whatis and apropos
commands find it. (PR#1634, thanks to Étienne Mollier)
* Assorted dependency tracking fixes. (PR#1653, thanks to John Marshall)
Documentation updates
---------------------
* Changed Alpine build instructions as they've switched back to using openssl.
(PR#1609)
* Recommend using -rdynamic when statically linking a libhts.a with
plugins enabled. (PR#1611, thanks to John Marshall. Fixes #1600,
reported by Jack Wimberley)
* Fixed example in docs for sam_hdr_add_line(). (PR#1618, thanks to kojix2)
* Improved test harness for base modifications API. (PR#1648)
Bug fixes
---------
* Fix a major bug when searching against a CRAM index where one container
has start and end coordinates entirely contained within the previous
container. This would occasionally miss data, and sometimes return much
more than required. The bug affected versions 1.11 to 1.17, although the
change in 1.11 was bug-fixing multi-threaded index queries. This bug did
not affect index building. There is no need to reindex your CRAM files.
(PR#1574, PR#1640. Fixes #1569, #1639, samtools/samtools#1808,
samtools/samtools#1819. Reported by xuxif, Jens Reeder and Jared Simpson)
* Prevent CRAM blocks from becoming too big in files with short
sequences but very long aux tags. (PR #1613)
* Fix bug where the CRAM decoder for CONST_INT and CONST_BYTE
codecs may incorrectly look for extra data in the CORE block.
Note that this bug only affected the experimental CRAM v4.0 decoder.
(PR#1614)
* Fix crypt4gh redirection so it works in conjunction with non-file
IO, such as using htsget. (PR#1577)
* Improve error checking for the VCF POS column, when facing invalid
data. (PR#1575, replaces #1570 originally reported and fixed
by Colin Nolan.)
* Improved error checking on VCF indexing to validate the data is BGZF
compressed. (PR#1581)
* Fix bug where bin number calculation could overflow when making iterators
over regions that go to the end of a chromosome. (PR#1595)
* Backport attractivechaos/klib#78 (by Pall Melsted) to HTSlib.
Prevents infinite loops in kseq_read() when reading broken gzip files.
(PR#1582, fixes #1579. Reported by Goran Vinterhalter)
* Backport attractivechaos/klib@384277a (by innoink) to HTSlib.
Fixes the kh_int_hash_func2() macro definition.
(PR#1599, fixes #1598. Reported by fanxinping)
* Remove a compilation warning on systems with newer libcurl releases.
(PR#1572)
* Windows: Fixed BGZF EOF check for recent MinGW releases. (PR#1601,
fixes samtools/bcftools#1901)
* Fixed bug where tabix would not return the correct regions for files
where the column ordering is end, ..., begin instead of begin, ..., end.
(PR#1626, fixes #1622. Reported by Hiruna Samarakoon)
* sam_format_aux1() now always NUL-terminates Z/H tags. (PR#1631)
* Ensure base modification iterator is reset when no MM tag is present.
(PR#1631, PR#1647)
* Fix segfault when attempting to write an uncompressed BAM file opened using
hts_open(name, "wbu"). This was attempting to write BAM data without
wrapping it in BGZF blocks, which is invalid according to the BAM
specification. "wbu" is now internally converted to "wb0" to output
uncompressed data wrapped in BGZF blocks. (PR#1632, fixes #1617.
Reported by Joyjit Daw)
* Fixed over-strict bounds check in probaln_glocal() which caused it to make
sub-optimal alignments when the requested band width was greater than the
query length. (PR#1616, fixes #1605. Reported by Jared Simpson)
* Fixed possible double frees when handling errors in bcf_hdr_add_hrec(),
if particular memory allocations fail. (PR#1637)
* Ensure that bcf_hdr_remove() clears up all pointers to the items removed
from dictionaries. Failing to do this could have resulted in a call
requesting a deleted item via bcf_hdr_get_hrec() returning a stale pointer.
(PR#1637)
* Stop the gzip decompresser from finishing prematurely when an empty
gzip block is followed by more data. (PR#1643, PR#1646)
Noteworthy changes in release 1.17 (21st February 2023)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* A new API for iterating through a BAM record's aux field.
(PR#1354, addresses #1319. Thanks to John Marshall)
* Text mode for bgzip. Allows bgzip to compress lines of text with block breaks
at newlines.
(PR#1493, thanks to Mike Lin for the initial version PR#1369)
* Make tabix support CSI indices with large positions. Unlike SAM and VCF
files, BED files do not set a maximum reference length which hindered CSI
support. This change sets an arbitrary large size of 100G to enable it to
work.
(PR#1506)
* Add a fai_line_length function. Exposes the internal line-wrap length.
(PR#1516)
* Check for invalid barcode tags in fastq output.
(PR#1518, fixes samtools#1728. Reported by Poshi)
* Warn if reference found in a CRAM file is not contained in the specified
reference file.
(PR#1517 and PR#1521, adds diagnostics for #1515. Reported by Wei WeiDeng)
* Add a faidx_seq_len64 function that can return sequence lengths longer than
INT_MAX. At the same time limit faidx_seq_len to INT_MAX output. Also add a
fai_adjust_region to ensure given ranges do not go beyond the end of the
requested sequence.
(PR#1519)
* Add a bcf_strerror function to give text descriptions of BCF errors.
(PR#1510)
* Add CRAM SQ/M5 header checking when specifying a fasta file. This is to
prevent creating a CRAM that cannot be decoded again.
(PR#1522. In response to samtools#1748 though not a direct fix)
* Improve support for very long input lines (> 2Gbyte). This is mostly useful
for tabix which does not do much interpretation of its input.
(PR#1542, a partial fix for #1539)
* Speed up load_ref_portion. This function has been sped up by about 7x, which
speeds up low-depth CRAM decoding by about 10%.
(PR#1551)
* Expand CRAM API to cope with new samtools cram_size command.
(PR#1546)
* Merges neighbouring I and D ops into one op within pileup. This means
4M1D1D1D3M is reported as 4M3D3M. Fixing this in sam.c means not only is
samtools mpileup now looking better, but any tool using the mpileup API will
be getting consistent results.
(PR#1552, fixes the last remaining part of samtools#139)
* Update the API documentation for bgzf_mt as it refered to a previous
iteration.
(PR#1556, fixes #1553. Reported by Raghavendra Padmanabhan)
Build changes
-------------
* Use POSIX grep in testing as egrep and fgrep are considered obsolete.
(PR#1509, thanks to David Seifert)
* Switch to building libdefalte with cmake for Cirris CI.
(PR#1511)
* Ensure strings in config_vars.h are escaped correctly.
(PR#1530, fixes #1527. Reported by Lucas Czech)
* Easier modification of shared library permissions during install.
(PR#1532, fixes #1525. Reported by StephDC)
* Fix build on ancient compilers. Added -std=gnu90 to build tests so older
C compilers will still be happy.
(PR#1524, fixes #1523. Reported by Martin Jakt)
* Switch MacOS CI tests to an ARM-based image.
(PR#1536)
* Cut down the number of embed_ref=2 tests that get run.
(PR#1537)
* Add symbol versions to libhts.so. This is to aid package developers.
(PR#1560 addresses #1505, thanks to John Marshall. Reported by Stefan Bruens)
* htscodecs now updated to v1.4.0.
(PR#1563)
* Cleaned up misleading system error reports in test_bgzf.
(PR#1565)
Bug fixes
---------
* VCF. Fix n-squared complexity in sample line with many adjacent tabs [fuzz].
(PR#1503)
* Improved bcftools detection and reporting of bgzf decode errors.
(PR#1504, thanks to Lilian Janin. PR#1529 thanks to Bergur Ragnarsson, fixes
#1528. PR#1554)
* Prevent crash when the only FASTA entry has no sequence [fuzz].
(PR#1507)
* Fixed typo in sam.h documentation.
(PR#1512, thanks to kojix2)
* Fix buffer read-overrun in bam_plp_insertion_mod.
(PR#1520)
* Fix hash keys being left behind by bcf_hdr_remove.
(PR#1535, fixes #1533. Reported by Giulio Genovese in #842)
* Make bcf_hdr_idinfo_exists more robust by checking id value exists.
(PR#1544, fixes #1538. Reported by Giulio Genovese)
* CRAM improvements. Fixed crash with multi-threaded CRAM. Fixed a bug in the
codec parameter learning for CRAM 3.1 name tokeniser. Fixed Cram compression
container substitution matrix generation,
(PR#1558, PR#1559 and PR#1562)
Noteworthy changes in release 1.16 (18th August 2022)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Make hfile_s3 refresh AWS credentials on expiry in order to make HTSlib work
better with AWS IAM credentials, which have a limited lifespan.
(PR#1462 and PR#1474, addresses #344)
* Allow BAM headers between 2GB and 4GB in size once more. This is not
permitted in the BAM specification but was allowed in an earlier version of
HTSlib. There is now a warning at 2GB and a hard failure at 4GB.
(PR#1421, fixes #1420 and samtools#1613. Reported by John Marshall and
R C Mueller)
* Improve error message when failing to load an index.
(PR#1468, example of the problem samtools#1637)
* Permit MM (base modification) tags containing "." and "?" suffixes. These
define implicit vs explicit coordinates. See the SAM tags specification for
details.
(PR#1423 and PR#1426, fixes #1418. PR#1469, fixes #1466. Reported
by cjw85)
* Warn if spaces instead of tabs are detected in a VCF file to prevent
confusion.
(PR#1328, fixes bcftools#1575. Reported by ketkijoshi278)
* Add an "sclen" filter expression keyword. This is the length of a soft-clip,
both left and right end. It may be combined with qlen (qlen-sclen) to obtain
the number of bases in the query sequence that have been aligned to the genome
ie it provides a way to compare local-alignment vs global-alignment length.
(PR#1441 and PR/samtools#1661, fixes #1436. Requested by Chang Y)
* Improve error messages for CRAM reference mismatches. If the user specifies
the wrong reference, the CRAM slice header MD5sum checks fail. We now report
the SQ line M5 string too so it is possible to validate against the whole
chr in the ref.fa file. The error message has also been improved to report
the reference name instead of #num. Finally, we now hint at the likely cause,
which counters the misleading samtools supplied error of "truncated or
corrupt" file.
(PR#1427, fixes samtools#1640. Reported by Jian-Guo Zhou)
* Expose more of the CRAM API and add new functionality to extract the reference
from a CRAM file.
(PR#1429 and PR#1442)
* Improvements to the implementation of embedded references in CRAM where no
external reference is specified.
(PR#1449, addresses some of the issues in #1445)
* The CRAM writer now allows alignment records with RG:Z: aux tags that
don't have a corresponding @RG ID in the file header. Previously these
tags would have been silently dropped. HTSlib will complain whenever it
has to add one though, as such tags do not conform to recommended practice
for the SAM, BAM and CRAM formats.
(PR#1480, fixes #1479. Reported by Alex Leonard)
* Set tab delimiter in man page for tabix GFF3 sort.
(PR#1457. Thanks to Colin Diesh)
* When using libdeflate, the 1...9 scale of BGZF compression levels is
now remapped to the 1...12 range used by libdeflate instead of being
passed directly. In particular, HTSlib levels 8 and 9 now map to
libdeflate levels 10 and 12, so it is possible to select the highest (but
slowest) compression offered by libdeflate.
(PR#1488, fixes #1477. Reported by Gert Hulselmans)
* The VCF variant API has been extended so that it can return separate flags
for INS and DEL variants as well as the existing INDEL one. These flags
have not been added to the old bcf_get_variant_types() interface as
it could break existing users. To access them, it is necessary to use new
functions bcf_has_variant_type() and bcf_has_variant_types().
(PR#1467)
* The missing, but trivial, `le_to_u8()` function has been added to hts_endian.
(PR#1494, Thanks to John Marshall)
* bcf_format_gt() now works properly on big-endian platforms.
(PR#1495, Thanks to John Marshall)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* Update htscodecs to version 1.3.0 for new SIMD code + various fixes.
Updates the htscodecs submodule and adds changes necessary to make HTSlib
build the new SIMD codec implementations.
(PR#1438, PR#1489, PR#1500)
* Fix clang builds under mingw. Under mingw, clang requires dllexport to be
applied to both function declarations and function definitions.
(PR#1435, PR#1497, PR#1498 fixes #1433. Reported by teepean)
* Fix curl type warning with gcc 12.1 on Windows.
(PR#1443)
* Detect ARM Neon support and only build appropriate SIMD object files.
(PR#1451, fixes #1450. Thanks to John Marshall)
* `make print-config` now reports extra CFLAGS that are needed to build the
SIMD parts of htscodecs. These may be of use to third-party build
systems that don't use HTSlib's or htscodecs' build infrastructure. (PR#1485.
Thanks to John Marshall)
* Fixed some Makefile dependency issues for the "check"/"test" targets
and plugins. In particular, "make check" will now build the "all" target,
if not done already, before running the tests.
(PR#1496)
Bug fixes
---------
* Fix bug when reading position -1 in BCF (0 in VCF), which is used to indicate
telomeric regions. The BCF reader was incorrectly assuming the value stored
in the file was unsigned, so a VCF->BCF->VCF round-trip would change it
from 0 to 4294967296.
(PR#1476, fixes #1475 and bcftools#1753. Reported by Rodrigo Martin)
* Various bugs and quirks have been fixed in the filter expression engine,
mostly related to the handling of absent tags, and the is_true flag.
Note that as a result of these fixes, some filter expressions may give
different results:
- Fixed and-expressions including aux tag values which could give an invalid
true result depending on the order of terms.
- The expression `![NM]` is now true if only `NM` does not exist. In
earlier versions it would also report true for tags like `NM:i:0` which
exist but have a value of zero.
- The expression `[X1] != 0` is now false when `X1` does not exist. Earlier
versions would return true for this comparison when the tag was missing.
- NULL values due to missing tags now propagate through string, bitwise
and mathematical operations. Logical operations always treat them as
false.
(PR#1463, fixes samtools#1670. Reported by Gert Hulselmans;
PR#1478, fixes samtools#1677. Reported by johnsonzcode)
* Fix buffer overrun in bam_plp_insertion_mod. Memory now grows to the proper
size needed for base modification data.
(PR#1430, fixes samtools#1652. Reported by hd2326)
* Remove limit of returned size from fai_retrieve().
(PR#1446, fixes samtools#1660. Reported by Shane McCarthy)
* Cap hts_getline() return value at INT_MAX. Prevents hts_getline() from
returning a negative number (a fail) for very long string length values.
(PR#1448. Thanks to John Marshall)
* Fix breakend detection and test bcf_set_variant_type().
(PR#1456, fixes #1455. Thanks to Martin Pollard)
* Prevent arrays of BCF_BT_NULL values found in BCF files from causing
bcf_fmt_array() to call exit() as the type is unsupported. These are
now tested for and caught by bcf_record_check(), which returns an
error code instead. (PR#1486)
* Improved detection of fasta and fastq files that have very long comments
following identifiers. (PR#1491, thanks to John Marshall.
Fixes samtools/samtools#1689, reported by cjw85)
* Fixed a SEGV triggered by giving a SAM file to `samtools import`.
(PR#1492)
Noteworthy changes in release 1.15.1 (7th April 2022)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Security fix: Fixed broken error reporting in the sam_prob_realn()
function, due to a missing hts_log() parameter. Prior to this fix
(i.e., in HTSlib versions 1.8 to 1.15) it was possible to abuse
the log message format string by passing a specially crafted
alignment record to this function. (PR#1406)
* HTSlib now uses libhtscodecs release 1.2.2. This fixes a number
of bugs where invalid compressed data could trigger usage of
uninitialised values. (PR#1416)
* Fixed excessive memory used by multi-threaded SAM output on
long reads. (Part of PR#1384)
* Fixed a bug where tabix would misinterpret region specifiers
starting at position 0. It will also now warn if the file
being indexed is supposed to be 1-based but has positions
less than or equal to 0. (PR#1411)
* The VCF header parser will now issue a warning if it finds an
INFO header with Type=Flag but Number not equal to 0. It will
also ignore the incorrect Number so the flag can be used. (PR#1415)
Noteworthy changes in release 1.15 (21st February 2022)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* Bgzip now has a --keep option to not remove the input file after
compressing. (PR#1331)
* Improved file format detection so some BED files are no longer
detected as FASTQ or FASTA. (PR#1350, thanks to John Marshall)
* Added xz (lzma), zstd and D4 formats to the file type detection
functions. We don't actively support reading these data types, but
function calls and htsfile can detect them. (PR#1340, thanks to
John Marshall)
* CRAM now also uses libdeflate for read-names if the libdeflate
version is new enough (1.9 onwards). Previously we used zlib for
this due to poor performance of libdeflate. This gives a slight
speed up and reduction in file size. (PR#1383)
* The VCF and BCF readers will now issue a warning if contig, INFO
or FORMAT IDs do not match the formats described in the VCFv4.3
specification. Note that while the invalid names will mostly still
be accepted, future updates will convert the warnings to errors
causing files including invalid names to be rejected. (PR#1389)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* HTSlib now uses libhtscodecs release 1.2.1.
* Improved support for compiling and linking against HTSlib with
Microsoft Visual Studio. (PR#1380, #1377, #1375. Thanks to
Aidan Bickford and John Marshall)
* Various internal CI improvements.
Bug fixes
---------
* Fixed CRAM index queries for HTSJDK output (PR#1388, reported by
Chris Norman). Note this also fixes writing CRAM writing, to match
the specification (and HTSJDK), from version 3.1 onwards.
* Fixed CRAM index queries when required-fields settings are selected
to ignore CIGARs (PR#1372, reported by Giulio Genovese).
* Unmapped but placed (having chr/pos) are now included in the BAM
indices. (PR#1352, thanks to John Marshall)
* CRAM now honours the filename##idx##index nomenclature for
specifying non-standard index locations. (PR#1360, reported by
Michael Cariaso)
* Minor CRAM v1.0 read-group fix (PR#1349, thanks to John Marshall)
* Permit .fa and .fq file type detection as synonyms for FASTA and
FASTQ. (PR#1386).
* Empty VCF format fields are now output ":.:" as instead of "::".
(PR#1370)
* Repeated bcf_sr_seek calls now work. (PR#1363, reported by
Giulio Genovese)
* Bcf_remove_allele_set now works on unpacked BCF records. (PR#1358,
reported by Brent Pedersen).
* The hts_parse_decimal() function used to read numbers in region lists
is now better at rejecting non-numeric values. In particular it
now rejects a lone 'G' instead of interpreting it as '0G', i.e. zero.
(PR#1396, PR#1400, reported by SSSimon Yang; thanks to John Marshall).
* Improve support for GPU issues listed by -Wdouble-promotion.
(PR#1365, reported by David Seisert)
* Fix example code in header file documentation. (PR#1381, Thanks to
Aidan Bickford)
Noteworthy changes in release 1.14 (22nd October 2021)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Features and Updates
--------------------
* Added a keep option to bgzip to leave the original file untouched. This
brings bgzip into line with gzip. (PR #1331, thanks to Alex Petty)
* "endpos" has been added to the filter language, giving the position
of the rightmost mapped base as measured by the CIGAR string. For
unmapped reads it is the same as "pos". (PR #1307, thanks to John Marshall)
* Interfaces have been added to interpret the new base modification tags
added to the SAMtags document in samtools/hts-specs#418. (PR #1132)
* New API functions hts_flush()/sam_flush()/bcf_flush() for flushing output
htsFile/samFile/vcfFile streams. (PR #1326, thanks to John Marshall)
* The synced_bcf_reader now sorts lines with symbolic alleles by END tag as
well as POS. (PR #1321)
* Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and
BCF_SR_TARGETS_OVERLAP for better control of records that start outside
the desired region but overlap it are handled. Fixes samtools/bcftools#1420
and samtools/bcftools#1421 raised by John Marshall. (PR #1327)
* HTSlib will now accept long-cigar CG:B: tags made by htsjdk which don't
quite follow the specification properly (using signed values instead of
unsigned). Thanks to Colin Diesh for reporting an example file. (PR #1317)
* The warning printed when the BGZF reader finds a file with no EOF block
has been changed to be less alarming. Unfortunately some third-party
BGZF encoders don't write EOF blocks at the end of files. Thanks to
Keiran Raine for reporting an example file. (PR #1323)
* The FASTA and FASTQ readers get an option to skip over the first item on
the header line, and use the second as the read name. It allows the original
name to be restored on some of the fastq files served from the European
Nucleotide Archive (ENA). (PR #1325)
* HTSlib is now more strict when parsing the VCF samples line (beginning
#CHROM). It will only accept tabs between the mandatory field names and
sample names must be separated with tabs. (PR #1328)
* HTSlib will now warn if it looks like the header has been corrupted
by diagnostic messages from the program that made it. This can happen when
using `nohup`, which by default mixes stdout and stderr into the same
stream. (PR#1339, thanks to John Marshall)
* File format detection will now recognise signatures for XZ, Zstd and D4
files (note that HTSlib will not read them yet). (PR #1340, thanks to
John Marshall)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* Some redundant tests have been removed from the test harness, speeding it up.
(PR #1308)
* The version.sh script now works better on shallow checkouts. (PR #1324)
* A check-untracked Makefile target has been added to catch untracked files
(mostly) left by the test harness. (PR #1324)