Call to /ds/get returns truncated readme text

[chriswhong]

Observed in rosebud after manually editing the readme with several paragraphs of lorem ipsum text.
POST to /upload included the entire string of text.
next GET to /ds/get included truncated text with an ellipsis.

[ramfox]

posting my comment from chat:

Need to look further into this, but this is most likely caused by the way we define and generate previews. We only return a certain number of bytes of the preview readme. I think the idea was that the preview would be truncated, and if you click a “…” or “read more”, we could do a call to the get endpoint with the “selector: readme” to get the full readme. I think in general this is fine, because we want this behavior on the preview page, but we may need a way to just force the get endpoint to return a full readme with the rest of the dataset, so we do t need to do multiple calls for each version in the history

This is definitely the reason this is happening.

Would like feedback on how we thing it's best to solve.

Questions that need to be answered:

1. Is this still the behavior we want from a preview? Specifically, do we still want to truncate the readme? I think yes, since the readme can be arbitrarily long. I'm repeating myself, but the initial idea was that we would send over a truncated readme, and if a user clicked "read more", we would fetch the full readme using the /ds/get endpoint and "selector":"readme" param.
2. We need a way to get the "full" dataset "head" without the body using the /ds/get endpoint. Thinking the sanest path would be to switch back to having the /ds/get endpoint return the "full" dataset head (sans body), and add a Preview flag to /ds/get that returns previews?

type GetParams struct {
   ...
   Preview bool // when `Preview` is true, return a truncated version of the dataset, including the first 100 rows of the body, and first KB of the readme
}

Otherwise, we instead add a... Full flag? Or some other better word, that would indicate getting a non-truncated version of the dataset, without the body.