to_array and to_bytes use-cases

[BramVanroy]

I am reading through the docs and found to_array and to_bytes. I have encountered to_bytes before, when trying to get a deterministic serialization of a model. But to_array is new to me.

I understand the differences in how you use them, and that to_array requires you (or allows you) to only export certain attributes. But what are the typical use-cases for both? Let's say I want to parse a corpus and store the parsed docs in a "one doc per line" format, would I use to_bytes or to_array? (My gut tells me to_bytes.)

In short, what are the use-cases (advantages/disadvantages) of these two methods?

[richardpaulhudson]

Hi @BramVanroy, both these methods generate representations of a document that are designed for consumption by code rather than reading by a human being:

• to_bytes() creates a serialized representation of the document that can be read in by some other process using from_bytes(). The classic situation where you would want serialization is when saving to disk, for which spaCy has extra from_disk() and to_disk() methods, but there are other situations where serialization is required such as passing a document between processes via a queue.

• to_array() creates a table mapping the tokens that make up a document to the values of specific attributes of those tokens; the table is optimized for use as input to a neural network. Especially when using GPUs, it's important that inputs to a neural network have a consistent shape and that the entries have consistent types, and this method ensures that these requirements are met.

I'm not quite sure what you mean by 'storing the parsed docs in a "one doc per line" format':

• if you're looking for an easy and efficient way of storing and manipulating a group of parsed documents, have a look at DocBin.
• if you're looking to create a human-readable representation of the documents with one token on each line, you would need to write a script to do that, but the Doc.to_json() method would probably be of help.

[BramVanroy]

Thanks for the quick reply, Richard. Yeah, I did not explain that very well. I mean that I have a "table" with one sentence per line and some other metadata. I wish to parse each sentence with spaCy once, save that parsed information in the tabel so that I can reuse that later without having to reparse the whole thing. So for every sentence (row) I'd add a parsed spaCy representation.

From reading your components, I think to_bytes is the best way here, so that later I can just call Doc.from_bytes and reconstruct the parsed spaCy docs.

[richardpaulhudson]

Yes, that should work.