Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add byteLength method and hasState property #258

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open

Add byteLength method and hasState property #258

wants to merge 19 commits into from

Conversation

Meigyoku-Thmn
Copy link

I would like to add the byteLength method and the hasState property.

byteLength method:

  • Guaranteed to be faster than iconv.encode(..).length because it doesn't allocate an entire buffer.
  • Usecase: you might have a very long string and don't want to create a very long buffer to encode it, but you want to write that string into a binary file with a prefix byte length (maybe in 7-bit format likes this method in .NET). So you can use this method, then you encode the string gradually on its' substrings by an encoder.

hasState property:

  • If reading binary data to decode, or encoding to write to binary file, you might want know if the decoder or encoder has any accumulated state inside them so you can decide if there is an error or not.

ashtuchkin and others added 19 commits July 14, 2020 13:13
@ashtuchkin
Introduce the concept of backends
e567849 
 * Add two backends: node & web
 * Convert core lib files to use the backends (and not use Buffer)
 * Convert utf16 codec as an example
 * Add testing for both node side and webpack
 * Bump Node.js minimal supported version to 4.5.0 and modernize some
   existing code. This will allow us to get rid of
   safer-buffer, our only dependency.
@ashtuchkin
Implement UTF-16LE encoding, update tests, adjust codec interface
9aa082f 
Three major reasons for reimplementing UTF-16 and not use native codec:
 1. We want to remove StringDecoder & Buffer references due to ashtuchkin#235.
 2. StringDecoder is inconsistent with handling surrogates on Node v6-9
 3. NPM module string_decoder gives strange results when processing chunks -
    it sometimes prepends '\u0000', likely due to a bug.

Performance was and is a major concern here. Decoder shouldn't be affected because it uses
backend methods directly. Encoder is affected due to introducing character-level loop. It's
still very fast (~450Mb/s), so I'm not too worried. If needed, we can make it about 4x faster
in Node.js by introducing a dedicated backend method. Browser speeds will be the same.
@gyzerok
Convert sbcs codec and some tests to use backend (ashtuchkin#255)
228af9c
@ashtuchkin
Added ESLint and Prettier.
141a8dd
@ashtuchkin
Move generation-specific deps to a separate package.json
0d01f15
@ashtuchkin
Apply ESLint to tests
5da0746
@ashtuchkin
Apply prettier to tests
c16052d
@ashtuchkin
(minor) remove __dirname from require()-s
67f91b9
@ashtuchkin
Add strict mode everywhere.
9bb9d83
@ashtuchkin
(minor) Rename utils.bytesFrom() to bytes() and make it accept hex st…
f49c584 
…rings.
@ashtuchkin
Added more eslint rules and fixed errors
2be15b0
@ashtuchkin
Removed dependency on 'iconv' from sbcs-test.js and added it to web s…
a1bd8f7 
…uite

To do that I've added a generation step and store the data in test/tables/ folder.
@gyzerok
Convert dbcs codec and tests (ashtuchkin#256)
5d99a92
@tran-huynh-minh-ngoc
add byteLength method and hasState property
1f5c89e
@tran-huynh-minh-ngoc
update readme and fix typo, wrong variable and function name
72517be
@tran-huynh-minh-ngoc
add byteLength test for basic encodings
ab95d0a
@tran-huynh-minh-ngoc
add byteLength test for main-test
191aa2a
@tran-huynh-minh-ngoc
format
965b5fb
@tran-huynh-minh-ngoc
implement matchAll for older enviroments
8bbd3e4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Meigyoku-Thmn @ashtuchkin @gyzerok @tran-huynh-minh-ngoc