Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Layout.read_bytes() method that reads a layout from a byte array #1920

Merged
merged 1 commit into from
Nov 30, 2024

Conversation

rocallahan
Copy link
Contributor

This is much more convenient and a bit more efficient than writing the bytes to a temporary file and reading that back in. It's useful when we have received a layout over the network or embedded in some file.

This makes a copy of the bytes but there doesn't seem to be any way to avoid that currently.

…y in Python or Ruby.

This is much more convenient and a bit more efficient than writing the bytes
to a temporary file and reading that back in. It's useful when we have received
a layout over the network or embedded in some file.

This makes a copy of the bytes but there doesn't seem to be any way to avoid that
currently.
@rocallahan
Copy link
Contributor Author

An even better API (for Python) would be to accept a Python io.BufferedIOBase which we would wrap in a tl::InputStreamBase but that's far more work.

@klayoutmatthias
Copy link
Collaborator

Thanks for this PR. This makes sense.

Version will be 0.29.9 - if you like, I can change that.

Basically, the reader can take data from "data:" stream "paths", but true bytes are easier to work with.

For symmetry, a writer to bytes would be needed too, right?

Matthias

@klayoutmatthias
Copy link
Collaborator

Another comment: Currently, passing the byte array as std::vector<char> involves a memory copy. So this facility is not good for passing huge files.

@rocallahan
Copy link
Contributor Author

For symmetry, a writer to bytes would be needed too, right?

Yes, although we don't need that at the moment.

Currently, passing the byte array as std::vector involves a memory copy. So this facility is not good for passing huge files.

Yes, I guess there is no way around that without extending your bindings infrastructure.

@rocallahan
Copy link
Contributor Author

I should mention that after submitting this PR I worked out another approach: use mkfifo to create a FIFO, and have KLayout read from that FIFO while another Python thread writes bytes into it. Compared to this PR, it would use less memory for large files but requires more system calls and is a lot more complicated for users. Compared to a temporary file, it doesn't require space on the filesystem and is probably a bit faster, but is also more complicated for users. So I still think this PR is useful.

@klayoutmatthias
Copy link
Collaborator

Would you like to add a byte writer too? I feel it would look better that way.

@klayoutmatthias klayoutmatthias changed the base branch from master to pr-1920 November 30, 2024 16:46
@klayoutmatthias klayoutmatthias merged commit 41383e4 into KLayout:pr-1920 Nov 30, 2024
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants