perf(lexer): use portable-SIMD to speed up whitespace scanning by Boshen · Pull Request #26 · oxc-project/oxc
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the input is 20 bytes, then the first 16 bytes are handled in the for chunk in chunks loop, and the remaining 4 bytes are handled by this code (which adds 12 zero bytes at the end). I assume this is what you mean when you say "remainder isn't aligned".
You can guarantee that remainder is always empty or unimportant (thus you don't need to check for it) if you add enough zeros at the end of bytes. I do this in quick-lint-js when I load the file from the filesystem, so everything can safely assume that the extra bytes are present. (I have a dedicated type for this called padded_string_view which doesn't allow breaking this invariant.)