transformHeader Called Multiple Times in Papa Parse v5.5.1

[sambit04126]

Bug Report: transformHeader Called Multiple Times in Papa Parse v5.5.1

1. Title
   transformHeader Function Invoked for Data Rows in Papa Parse v5.5.1

2. Description
   In Papa Parse version 5.5.1, the transformHeader function is being called for each data row's fields instead of exclusively for the header row. This results in unexpected transformations of data fields, leading to incorrect parsed output.

3. Expected Behavior
   transformHeader should be invoked only once for each header field in the first row of the CSV.
   Data rows should be processed without invoking transformHeader.

4. Actual Behavior
   transformHeader is invoked for each field in data rows, causing data values to be transformed incorrectly.
   This leads to incorrect and malformed rows in the transformation process.

5. Steps to Reproduce
   Setup the Environment:

Install Papa Parse version 5.5.1.

npm install papaparse@5.5.1

Create a file named index.js and paste the following code:

const Papa = require('papaparse');

const csvContent = `Name,Age,City
John Doe,30,New York
Jane Smith,25,Los Angeles
,abc`;

function mapHeader(header) {
    const transformed = header.trim().toLowerCase();
    console.log(`Transforming header: "${header}" -> "${transformed}"`);
    return transformed;
}

function transformData(data) {
    console.log(`Transforming data: "${data}"`);
    return data;
}

let currentChunk = [];
const chunkSize = 2;

function processChunk(chunk) {
    console.log(chunk);
}

Papa.parse(csvContent, {
    header: true,                
    dynamicTyping: true,        
    skipEmptyLines: true,       
    transformHeader: mapHeader,  
    transform: transformData,   
    step: function (results, parser) {
        console.log('Parsed Row:', results.data);
        currentChunk.push(results.data);

        if (currentChunk.length === chunkSize) {
            processChunk(currentChunk);
            currentChunk = []; 
        }
    },
    complete: function () {
        console.log('Parsing Complete.');
        if (currentChunk.length > 0) {
            processChunk(currentChunk);
        } else {
            console.log('No remaining data to process.');
        }
    },
    error: function (error) {
        console.error('Parsing Error:', error.message);
    }
});

Run the script

node index.js

Observe the Output:

With Papa Parse v5.5.1:

Transforming header: "Name" -> "name"
Transforming header: "Age" -> "age"
Transforming header: "City" -> "city"
Transforming header: "name" -> "name"
Transforming header: "age" -> "age"
Transforming header: "city" -> "city"
Transforming header: "John Doe" -> "john doe"
Transforming header: "30" -> "30"
Transforming header: "New York" -> "new york"
Transforming data: "john doe"
Transforming data: "30"
Transforming data: "new york"
Parsed Row: { name: 'john doe', age: 30, city: 'new york' }
Transforming header: "Jane Smith" -> "jane smith"
Transforming header: "25" -> "25"
Transforming header: "Los Angeles" -> "los angeles"
Transforming data: "jane smith"
Transforming data: "25"
Transforming data: "los angeles"
Parsed Row: { name: 'jane smith', age: 25, city: 'los angeles' }
[ { name: 'john doe', age: 30, city: 'new york' },
  { name: 'jane smith', age: 25, city: 'los angeles' } ]
Transforming header: "" -> ""
Transforming header: "abc" -> "abc"
Transforming data: ""
Transforming data: "abc"
Parsed Row: { name: null, age: 'abc' }
Parsing Complete.
[ { name: null, age: 'abc' } ]

With Papa Parse v5.4.1:

Transforming header: "Name" -> "name"
Transforming header: "Age" -> "age"
Transforming header: "City" -> "city"
Transforming header: "Name" -> "name"
Transforming header: "Age" -> "age"
Transforming header: "City" -> "city"
Transforming data: "John Doe"
Transforming data: "30"
Transforming data: "New York"
Parsed Row: { name: 'John Doe', age: 30, city: 'New York' }
Transforming data: "Jane Smith"
Transforming data: "25"
Transforming data: "Los Angeles"
Parsed Row: { name: 'Jane Smith', age: 25, city: 'Los Angeles' }
[ { name: 'John Doe', age: 30, city: 'New York' },  { name: 'Jane Smith', age: 25, city: 'Los Angeles' } ]
Transforming data: ""
Transforming data: "abc"
Parsed Row: { name: null, age: 'abc' }
Parsing Complete.
[ { name: null, age: 'abc' } ]

The transformHeader function in Papa Parse version 5.5.1 is incorrectly being invoked for data rows, leading to unexpected transformations. This regression affects data integrity during CSV parsing.

[paescuj]

This seems to be fixed with #1086.
With the above reproduction in v5.5.2, I'm getting the same output as it used to be back in v.5.4.1:

Transforming header: "Name" -> "name"
Transforming header: "Age" -> "age"
Transforming header: "City" -> "city"
Transforming header: "name" -> "name"
Transforming header: "age" -> "age"
Transforming header: "city" -> "city"
Transforming data: "John Doe"
Transforming data: "30"
Transforming data: "New York"
Parsed Row: { name: 'John Doe', age: 30, city: 'New York' }
Transforming data: "Jane Smith"
Transforming data: "25"
Transforming data: "Los Angeles"
Parsed Row: { name: 'Jane Smith', age: 25, city: 'Los Angeles' }
[
  { name: 'John Doe', age: 30, city: 'New York' },
  { name: 'Jane Smith', age: 25, city: 'Los Angeles' }
]
Transforming data: ""
Transforming data: "abc"
Parsed Row: { name: null, age: 'abc' }
Parsing Complete.
[ { name: null, age: 'abc' } ]

[oualeo]

Not fixed in 5.5.2: as you can see from the log, transformHeader is still getting called multiple times, and receiving the already-transformed values on the second call.

[jsakaicribl]

I had this same problem with 5.5.2.

[jbool24]

+1 here as well. Error exists with 5.5.2, but its strange and not consistent.

@guseggert After some testing, I found that this behavior always double calls the transfromHeader function BUT does not update the headers (the intended behavior) sometimes on part of the chunks. Also, IF we wrap Papa in a Stream.Transform and call stream.write(//...data) directly where the "stream" is the Duplex returned from Papa.parse(Papa.NODE_STREAM_INPUT, opts) it also does not happen. 🤔 See the Jest test below to see where it occurs.

The following does not cause the errors the OP is talking about but the subsequent example after this does

function toCamelCase(str: string): string {
 return str.replace(/[^a-zA-Z0-9]+(.)/g, (m, chr: string) => chr.toUpperCase()).replace(/^\w/, (c) => c.toLowerCase());
}

export class TransformCsvStreamToObject extends Stream.Transform {
  private parser: Stream.Duplex;

  constructor(options?: TransformStreamOptions) {
    super({ objectMode: true });

    // Create a node stream parser
    this.parser = Papa.parse(Papa.NODE_STREAM_INPUT, {
      delimiter: ',',
      quoteChar: '"',
      header: true,
      fastMode: false,
      transformHeader: (header) => toCamelCase(header),
    });

    // Handle data parsed and push to write buffer
    this.parser.on('data', (data: Record<string, unknown>) => {
        // Do some manual work on data or just pass along what Papa parsed
        // Output to write side stream
        this.push(data);
      }
    });

    // Handle errors
    this.parser.on('error', (err) => {
      this.logger.error(`PAPA STREAM PARSE ERROR: ${err.message}`);
    });

    this.parser.on('end', () => {
      this.logger.verbose('PAPA INPUT STREAM PARSED');
    });
  }

  _transform(chunk: Buffer, _encoding: string, done: () => void) {
    this.parser.write(chunk); // push raw input to papaparser stream
    done();
  }

  _flush(done: () => void) {
    this.parser.end();
    done();
  }

  _final(done: () => void) {
    done();
  }
}

These test causes transformHeader to be called on rows and update the values incorrectly, specifically on the second column I see the value change with the transform header function applied. And it only happens on some of the chunk processed.

# ./fixtures/small_dataFile.csv
Serial Number,Some String Value
200001,CC94E123
200002,CC9C09B3
#  ...plus another 98 rows

# The first half or the file is correct like this
{ serialNumber: '200001', someStringValue: 'CC94E123' }
{ serialNumber: '200002', someStringValue: 'CC9C09B3' }

# but second half of this files parsed values come out like this
{ serialNumber: '200098', someStringValue: 'cC976543' } # it used the camelCase to change the row field value
{ serialNumber: '200099', someStringValue: 'cC976543' }

import fs from 'fs';
import { Writable } from 'stream';
import { pipeline } from 'stream/promises';

import Papa from 'papaparse'


async function asyncLogParsedValues(data): Promise<void> {
  return Promise.resolve(console.log(data));
}

function toCamelCase(str: string): string {
 return str.replace(/[^a-zA-Z0-9]+(.)/g, (m, chr: string) => chr.toUpperCase()).replace(/^\w/, (c) => c.toLowerCase());
}

describe('CSV Transformation Handling', () => {
  it('Uses Papaparse w/custom transformHeaders but it modifies some row values also', async () => {
    const readStream = fs.createReadStream(__dirname + '/fixtures/small_dataFile.csv', 'utf8');

    const writeStream = new Writable({
      objectMode: true,
      write: async (data: Record<string, unknown>, _, done) => {
        try {
          await asyncLogParsedValues(data);
          return done();
        } catch (e) {
          return done(e as Error);
        }
      },
    });

    writeStream.on('finish', () => console.debug(`WRITE STREAM COMPLETED`));

    try {
      const papaParser = Papa.parse(Papa.NODE_STREAM_INPUT, {
        header: true,
        transformHeader: (header) => toCamelCase(header),
      });

      await pipeline(readStream, papaParser, writeStream);
    } catch (err) {
      console.error(err);
    }
  });

  it('Uses Papaparse to transform using pipeline without transforming row values', async () => {
    const readStream = fs.createReadStream(__dirname + '/fixtures/small_dataFile.csv', 'utf8');

    const papaParser = Papa.parse(Papa.NODE_STREAM_INPUT, {
      header: true,
      transformHeader: (header) => toCamelCase(header),
    });

    const writeStream = new Writable({
      objectMode: true,
      write: async (data: Record<string, unknown>, _, done) => {
        try {
          await asyncLogParsedValues(data);
          return done();
        } catch (e) {
          return done(e as Error);
        }
      },
    });

    return pipeline(readStream, papaParser, writeStream);
  });

  it('Uses Papaparse to transform using pipe without transforming any row values', (done) => {
    const readStream = fs.createReadStream(__dirname + '/fixtures/small_dataFile.csv', 'utf8');

    const options = {
      header: true,
      transformHeader: (header) => toCamelCase(header),
    };

    const parsedStream = readStream.pipe(Papa.parse(Papa.NODE_STREAM_INPUT, options));

    parsedStream.on('data', (item) => console.log(item));
    parsedStream.on('finish', () => { console.debug(`INPUT STREAM CONSUMED`); done(); });
  });

});

[jbool24]

Hmmm, Something else strange leads me to believe its with the buffers somehow or Promises are affecting internals...

function notAsyncFunctionUsingValues(data): void {
  return console.log(data);
}

describe('CSV Transformation Handling', () => {
   it('Uses Papaparse w/custom transformHeaders but it modifies some row values also', async () => {
    const readStream = fs.createReadStream(__dirname + '/fixtures/small_dataFile.csv', 'utf8');

    const papaParser = Papa.parse(Papa.NODE_STREAM_INPUT, {
      header: true,
      transformHeader: (header) => toCamelCase(header),
    });

    const writeStream = new Writable({
      objectMode: true,
      write: async (data: Record<string, unknown>, _, done) => {
        try {
          notAsyncFunctionUsingValues(data); <-- if this is not a Async it works correctly; Does async cause something to be split and called separately?
          // await Promise.resolve(notAsyncFunctionUsingValues(data)); <-- But this does not and introduces the error seen 
          return done();
        } catch (e) {
          return done(e as Error);
        }
      },
    });

    return pipeline(readStream, papaParser, writeStream);
  });
});

[jasonpolites]

I am seeing the same problem, specifically related (in my case) to using pause/resume.

If I attempt to pause stream processing, I end up with rows in the CSV being parsed as headers, and corrupting the data due to Duplicate headers found and renamed.

Full repro here:

<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Papa Parse Test</title>
  <script src="https://unpkg.com/papaparse@latest/papaparse.min.js"></script>
  <script type="module">
      document.addEventListener('DOMContentLoaded', async () => {

        const fileInput = document.querySelector('#fileInput');
        const usePauseCheckbox = document.querySelector('#usePause');
        const goButton = document.querySelector('#btnGo');

        let usePause = usePauseCheckbox.checked;
        let file = null;

        // The step callback
        const step = (row, parser) => {
          if(usePause == true) parser.pause();

          console.log(row);

          if(usePause == true) parser.resume();
        };

        goButton.addEventListener('click', _ => {
          if(file) {
            const config = {
              header: true,
              dynamicTyping: true,
              skipEmptyLines: true,
              worker: false,
              step: step,
              transform: (value) => {
                console.log(`Calling transform ${value}`)
                return value;
              },
              transformHeader: (value) => {
                console.log(`Calling transformHeader ${value}`)
                return value;
              }
            }

            Papa.parse(file, config);
          }
        });

        usePauseCheckbox.addEventListener('change', e => {
          usePause = e.target.checked;
        });

        fileInput.addEventListener('change', (event) => {
          file = event.target.files[0];
        });
      });

  </script>
  <style>
    div {
      padding:8px;
    }
  </style>
</head>
<body>
  <div>
    <input type="checkbox" id="usePause" checked>
    <label for="usePause">Use Pause/Resume</label>
  </div>

  <div>
    <input type="file" id="fileInput" accept="*.csv,*.txt">
  </div>

  <div>
    <button id="btnGo">Parse!</button>
  </div>

  Output in console
</body>
</html>

Sample CSV that reproduces the issue (there are some issues with this CSV. It's an excerpt from a real one and the trailing commas were in the original):

Card,Transaction Date,Post Date,Description,Category,Type,Amount,Memo
9538,12/07/2024,12/08/2024,Test,Office & Shipping,Sale,-35.00,
9538,11/15/2024,11/15/2024,AUTOMATIC PAYMENT - THANK,,Payment,35.00,
9538,11/07/2024,11/08/2024,Test,Office & Shipping,Sale,-35.00,
9538,11/07/2024,11/08/2024,Test,Office & Shipping,Sale,-35.00,

Example console output:

CSV data is corrupted due to this apparent header name conflict ("Test_1" does not exist in the CSV as a value):

[carstrb]

+1 also for 5.5.2. Here's my minimal example:

import * as pp from 'papaparse';

export interface Mapping {
    [key: string]: string;
}

const dummyString = 'a,b,c\nid1,12.3,true\nid2,45.6,false';

const dummyHeaderMapping: Mapping = {
    a: 'column_a',
    b: 'column_b',
    c: 'column_c',
}

const parsed = pp.parse(dummyString, {
    header: true,
    transformHeader: (header) => {
            console.log(`transformHeader invoked with header: "${header}"`);
            const mappedHeader = dummyHeaderMapping[header];
            console.log(`  Mapping to: "${mappedHeader}"`);
            return mappedHeader;
        }
});

const { data } = parsed;

console.log(data)

Output is as follows:

transformHeader invoked with header: "a"
  Mapping to: "column_a"
transformHeader invoked with header: "b"
  Mapping to: "column_b"
transformHeader invoked with header: "c"
  Mapping to: "column_c"
transformHeader invoked with header: "column_a"
  Mapping to: "undefined"
transformHeader invoked with header: "column_b"
  Mapping to: "undefined"
transformHeader invoked with header: "column_c"
  Mapping to: "undefined"
[ { undefined: 'true' }, { undefined: 'false' } ]

Hoping this can be resolved so that I can continue using this tool :)

[jasonpolites]

@carstrb FWIW, I spend two days and built my own parser. I think Papa is great, and maintaining a (free) open source project is difficult, so no shade against Papa, but corrupting the data is a bit of a deal-breaker for my use case. Papa is very flexible, and does a lot more than I need in many areas, so if your use case is also narrow, it may be a viable option (building your own, that is).