Improve handling of multiline strings

[rmgpanw]

Hi, I was recently reading a table into R using the readr package but discovered that several rows were missing as some cells included a single double quotation mark.

I managed to resolve this by setting quote = "", however I wonder if the read_tsv()/related functions could be updated to at least raise a warning if improper quoting is discovered in the data? The data.table package both raises a warning and deals with this automatically, as per the following reprex.

Many thanks for considering!

library(tibble)
library(readr)
library(data.table)

# create example file
df <- tribble(
  ~code, ~description,
  "1", '"Single quotes',
  "2", "No quotes",
  "3", 'Single quotes"',
  "4", '"Pair of quotes"',
  "5", "No quotes"
)

file_path <- tempfile(pattern = "df_with_quotes", fileext = ".tsv")

write_tsv(df,
  file_path,
  quote = "none",
  escape = "none"
)

# `readr::read_tsv()` reads inappropriately without raising an error with
# default settings
read_tsv(file_path,
  quote = "\""
)
#> Rows: 3 Columns: 2
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: "\t"
#> chr (1): description
#> dbl (1): code
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 3 × 2
#>    code description                                    
#>   <dbl> <chr>                                          
#> 1     1 "Single quotes\n2\tNo quotes\n3\tSingle quotes"
#> 2     4 "Pair of quotes"                               
#> 3     5 "No quotes"

# To read correctly use the `quote` argument
read_tsv(file_path,
  quote = ""
)
#> Rows: 5 Columns: 2
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: "\t"
#> chr (1): description
#> dbl (1): code
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 5 × 2
#>    code description         
#>   <dbl> <chr>               
#> 1     1 "\"Single quotes"   
#> 2     2 "No quotes"         
#> 3     3 "Single quotes\""   
#> 4     4 "\"Pair of quotes\""
#> 5     5 "No quotes"

# `data.table::fread()` raises a warning but reads the table correctly with
# default settings
fread(file_path,
  quote = "\""
)
#> Warning in fread(file_path, quote = "\""): Found and resolved improper quoting
#> in first 100 rows. If the fields are not quoted (e.g. field separator does not
#> appear within any field), try quote="" to avoid this warning.
#>    code    description
#> 1:    1 "Single quotes
#> 2:    2      No quotes
#> 3:    3 Single quotes"
#> 4:    4 Pair of quotes
#> 5:    5      No quotes

# setting `quote=""` produces (almost) the same result, now without a warning
fread(file_path,
  quote = ""
)
#>    code      description
#> 1:    1   "Single quotes
#> 2:    2        No quotes
#> 3:    3   Single quotes"
#> 4:    4 "Pair of quotes"
#> 5:    5        No quotes

sessionInfo()
#> R version 4.2.0 (2022-04-22)
#> Platform: x86_64-apple-darwin17.0 (64-bit)
#> Running under: macOS Big Sur/Monterey 10.16
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] data.table_1.14.6 readr_2.1.4       tibble_3.1.8     
#> 
#> loaded via a namespace (and not attached):
#>  [1] pillar_1.8.1      compiler_4.2.0    R.methodsS3_1.8.2 R.utils_2.12.2   
#>  [5] tools_4.2.0       digest_0.6.31     bit_4.0.5         evaluate_0.20    
#>  [9] lifecycle_1.0.3   R.cache_0.16.0    pkgconfig_2.0.3   rlang_1.0.6      
#> [13] reprex_2.0.2      cli_3.6.0         rstudioapi_0.14   parallel_4.2.0   
#> [17] yaml_2.3.7        xfun_0.37         fastmap_1.1.0     withr_2.5.0      
#> [21] styler_1.9.1      knitr_1.42        fs_1.6.1          vctrs_0.5.2      
#> [25] hms_1.1.2         tidyselect_1.2.0  bit64_4.0.5       glue_1.6.2       
#> [29] R6_2.5.1          fansi_1.0.4       vroom_1.6.1       rmarkdown_2.20   
#> [33] purrr_1.0.1       tzdb_0.3.0        magrittr_2.0.3    htmltools_0.5.4  
#> [37] ellipsis_0.3.2    utf8_1.2.3        crayon_1.5.2      R.oo_1.25.0

[daranzolin]

I experienced the same issue with read_csv(). A single double quote preceding one of the values omitted the remaining records. Only the warning from fread() saved me:

[hadley]

Here's a somewhat more minimal reprex:

library(readr)

lines <- 'code,description
1,"x
2,-
3,y"'
path <- tempfile()
writeLines(lines, path)

# Allows quoted string to span multiple lines
read_csv(path, col_types = list())
#> # A tibble: 1 × 2
#>    code description  
#>   <dbl> <chr>        
#> 1     1 "x\n2,-\n3,y"

# OK: explicit quote works
read_csv(path, quote = "", col_types = list())
#> # A tibble: 3 × 2
#>    code description
#>   <dbl> <chr>      
#> 1     1 "\"x"      
#> 2     2 "-"        
#> 3     3 "y\""

# warns & treats lines as quoted
data.table::fread(path, quote = "\"")
#> Warning in data.table::fread(path, quote = "\""): Found and resolved improper
#> quoting in first 100 rows. If the fields are not quoted (e.g. field separator
#> does not appear within any field), try quote="" to avoid this warning.
#>    code description
#> 1:    1          "x
#> 2:    2           -
#> 3:    3          y"

^{Created on 2023-07-31 with reprex v2.0.2}