Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FromCsvReader method to allow building rows with customized csv reader. #339

Open
stayingalivee opened this issue Jun 30, 2024 · 1 comment
Labels
good-first-issue great issue for folks who are first time contributors to this repo

Comments

@stayingalivee
Copy link

stayingalivee commented Jun 30, 2024

Proposal

Under the existing design, user has to manually modify their csv string to match with csv reader defaults.
func (r *Rows) FromCSVString(s string) *Rows function initialize a csv reader with defaults. The user may have a pipe separated csv or some other token. Also, the user cannot enable lazyQoutes (see: https://cs.opensource.google/go/go/+/refs/tags/go1.22.4:src/encoding/csv/reader.go;l=136) which is sometimes needed if you have a json string in the csv file without having to modify the csv extensively to match the defaults.

My proposal is to allow the user to define their csv reader freely with their own custom options.
Easiest way to do it without breaking existing code is the following:

func (r *Rows) FromCSVReader(csvReader *csv.Reader) *Rows {
	for {
		res, err := csvReader.Read()
		if err != nil {
			if errors.Is(err, io.EOF) {
				break
			}
			panic(fmt.Sprintf("Parsing CSV string failed: %s", err.Error()))
		}

		row := make([]driver.Value, len(r.cols))
		for i, v := range res {
			row[i] = CSVColumnParser(strings.TrimSpace(v))
		}
		r.rows = append(r.rows, row)
	}
	return r
}

you can also check this commit that I pushed on my fork: stayingalivee@d0736ed

Use-cases

I added a test to demonstrate the use-case. see: stayingalivee@d0736ed#diff-335b77abfb8789fd51a279c7cbdbdae90b384df153ce1bbac40aa1468d5bbb89R472

for ease of access I will paste the test below:

func TestCSVRowParserFromCsvParser(t *testing.T) {
	t.Parallel()

	line := "a|NULL|[\"a\",\"b\"]"

	r := strings.NewReader(line)
	csvReader := csv.NewReader(r)
	csvReader.Comma = '|'
	csvReader.LazyQuotes = true

	rs := NewRows([]string{"col1", "col2", "col3"}).FromCSVReader(csvReader)
	db, mock, err := New()
	if err != nil {
		t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
	}
	defer db.Close()

	mock.ExpectQuery("SELECT").WillReturnRows(rs)

	rw, err := db.Query("SELECT")
	if err != nil {
		t.Fatalf("unexpected error: %s", err)
	}
	defer rw.Close()
	var col1 string
	var col2 []byte
	var col3 string
	var col3Json []string

	rw.Next()
	if err = rw.Scan(&col1, &col2, &col3); err != nil {
		t.Fatalf("unexpected error: %s", err)
	}
	if col1 != "a" {
		t.Fatalf("expected col1 to be 'a', but got [%T]:%+v", col1, col1)
	}
	if col2 != nil {
		t.Fatalf("expected col2 to be nil, but got [%T]:%+v", col2, col2)
	}

	err = json.Unmarshal([]byte(col3), &col3Json)
	if err != nil {
		t.Fatalf("expected valid json, but got [%T]:%+v", col3, col3)
	}

	if !reflect.DeepEqual(col3Json, []string{"a", "b"}) {
		t.Fatalf("expected col3Json to be [\"a\", \"b\"], but got [%T]:%+v", col3Json, col3Json)
	}
}

If you find this helpful let me know so I can submit a pull request.

@stayingalivee stayingalivee changed the title Add FromCsvParser method to allow building rows with customized csv reader. Add FromCsvReader method to allow building rows with customized csv reader. Jun 30, 2024
@ninadingole
Copy link
Collaborator

CSV rarely have other types of separators, but we shouldn't restrict users to only use , so I don't see any problem with providing an additional method to pass a reader

@ninadingole ninadingole added the good-first-issue great issue for folks who are first time contributors to this repo label Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good-first-issue great issue for folks who are first time contributors to this repo
Projects
None yet
Development

No branches or pull requests

2 participants