More performance improvements by jimidle · Pull Request #4210 · antlr/antlr4
Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Jim.Idle <jimi@idle.ws>
undefined Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Eric Vergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Signed-off-by: Jim.Idle <jimi@idle.ws>
Posix threads are available only on Unix (and unix like) platforms. Wrap the dependency accordingly so builds don't fail on other platforms. Signed-off-by: HS <hs@apotell.com> Signed-off-by: Jim.Idle <jimi@idle.ws>
… in to the root (pre v4) files (antlr#4154) - Incorporate an interative tree walker - Expose Reset() and String() in lexer/token and add test Signed-off-by: Jim.Idle <jimi@idle.ws>
* feat: Remove the old version of the source code, as at v4.x.y the source must be under /v4 for modules Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: tidy up old version of go.mod (non v4 version), which is now deprecated Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws>
The badge in the README suggests that ANTLR requires Java 7 or higher, whereas since ANTLR v4.10 Java 11 or higher is required. Signed-off-by: Robert Adam <dev@robert-adam.de> (cherry picked from commit 8188dc5) Signed-off-by: Jim.Idle <jimi@idle.ws>
…BUILD_CPP_TESTS is not set (antlr#4121) Signed-off-by: Leonardo Sarmiento <leonardo@moduleworks.com> Co-authored-by: Leonardo Sarmiento <leonardo@moduleworks.com> Signed-off-by: Jim.Idle <jimi@idle.ws>
* Cpp: Remove code duplication The same function was defined inside multiple different source files (in an anonymous namespace). Not only is this generally bad practice from a maintenance point of view, but it also break unity builds (which can speed up compilation times A LOT). The fix simply consists in introducing a new Utils file that contains the single implementation and every source file that uses this function simply has to be linked against that new file (and include the corresponding header). Signed-off-by: Robert Adam <dev@robert-adam.de> * CI: Enable unity builds for cpp targets This should reduce the build time for those steps significantly Note: Unity builds are enabled only for cmake-based CI builds Signed-off-by: Robert Adam <dev@robert-adam.de> * CI: Perform unity and non-unity cpp builds Signed-off-by: Robert Adam <dev@robert-adam.de> --------- Signed-off-by: Robert Adam <dev@robert-adam.de> Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Signed-off-by: Jim.Idle <jimi@idle.ws>
…4 off the replace option on the go.mod file (antlr#4163) Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Jim.Idle <jimi@idle.ws>
…lashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws>
* fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Cater for the fact that some test rules use start as a label or rule name As a fix for other cvode gen errors when start, end, or exception are used as label names, they are now translated to have a suffix of `_` at code gen time. However, the runtime tests sometimes use start as a rule name and so we must now cater for this in the tests. Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
…ntlr#4169) * doc: Updates to some of the Go doc comments to start a ful ldocumentation cleanup Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes. Using this as a method of forcing myself to read every line of code in the runtime, and therefore discover mistakes in the original implementation. And, of course, actually working docs for the Go runtime, can only be a good thing. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes Also changes the exporet level of a some variables and funcs that were not correct, even though no user has currently needed them it would seem. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Many updates to document exported fuctions correctly and reformat the ingerited Java code It looks like a massive amount of changes, but it is almost all doc or changing exports or renaming unused paramters etc to make the Go linter happy. No actual code changes yet. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More additions and corrections to the Go documentation for the runtime Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Final clean of exported func and type documentation There will be more to do here as there are a lot of things that are hidden internal to the antlr package that probably should not be. There are also a lot of exported funcs and types without any documentation, that will eventually need to be cleaned up. Signed-off-by: Jim.Idle <jimi@idle.ws> * Changed Parser typings (antlr#4149) Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file (antlr#4163) Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * Feature/gotestfix (antlr#4168) * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Cater for the fact that some test rules use start as a label or rule name As a fix for other cvode gen errors when start, end, or exception are used as label names, they are now translated to have a suffix of `_` at code gen time. However, the runtime tests sometimes use start as a rule name and so we must now cater for this in the tests. Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <frank.josua@gmail.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
* feat: Createa n Init routine for BaseATNConfig so we can embed sructs rather than allocate to pointers Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Switch the use of pointers to embedded ATN states to true embeddding Saves many allocations and grbage collections Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <jimi@idle.ws> * Feature/docclean Greatly improve the godoc comments in the runtime (antlr#4169) * doc: Updates to some of the Go doc comments to start a ful ldocumentation cleanup Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes. Using this as a method of forcing myself to read every line of code in the runtime, and therefore discover mistakes in the original implementation. And, of course, actually working docs for the Go runtime, can only be a good thing. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes Also changes the exporet level of a some variables and funcs that were not correct, even though no user has currently needed them it would seem. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Many updates to document exported fuctions correctly and reformat the ingerited Java code It looks like a massive amount of changes, but it is almost all doc or changing exports or renaming unused paramters etc to make the Go linter happy. No actual code changes yet. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More additions and corrections to the Go documentation for the runtime Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Final clean of exported func and type documentation There will be more to do here as there are a lot of things that are hidden internal to the antlr package that probably should not be. There are also a lot of exported funcs and types without any documentation, that will eventually need to be cleaned up. Signed-off-by: Jim.Idle <jimi@idle.ws> * Changed Parser typings (antlr#4149) Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file (antlr#4163) Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * Feature/gotestfix (antlr#4168) * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Cater for the fact that some test rules use start as a label or rule name As a fix for other cvode gen errors when start, end, or exception are used as label names, they are now translated to have a suffix of `_` at code gen time. However, the runtime tests sometimes use start as a rule name and so we must now cater for this in the tests. Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <frank.josua@gmail.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Merge documentation updates Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Rework predictions tructs to use emedding instead of pointers Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: more reworking of PredictionContext for embedding Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Ensure that EmptyPredictionContext is correctly initialized Rework of the variaous PredictionContexts has reduced memory allocations to between 30% and 50% of previous version. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change from use of type casting to using stored type Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Convert CommonToken to true emedding rather than pointers Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <frank.josua@gmail.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Eric Vergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
…omparing two pointers! Fixes: antlr#3967 Closes antlr#3967 - A small error was made whereby the comparison with a newly created merge `M` of two prediction contexts `a` and `b`, was comparing the pointers of `a` and `M` and not using the internal Equals() method. The ATN output trace is now equal. - Also adds a new testrig internal project (so it won't be used by `go get`) that allows quick setup and testing of ATN tracing for a test grammar and test input. - Excludes go performance profiles from git - Corrects a small error in one of the ATN tracing scripts. Signed-off-by: Jim.Idle <jimi@idle.ws>
… fixed Signed-off-by: Jim.Idle <jimi@idle.ws>
- I missed this on my first evaluation of all the collections used by the code I inherited. The PredictionCache was implemented as a map using PredictionContext pointers as the key. This meant that there would never be a cache hit and the cache woudl just accumulate massive numbers of PredictionContext 'objects'! Bloody hell. Sometimes it is difficult to spot such glaring errors. This change vastly reduces memory usage. The next commit will fix the visited context look up which also suffers from the same problem. Fixes: antlr#3934 It probably also fixes a ton of performance and memory usage problems, which I have yet to test. Signed-off-by: Jim.Idle <jimi@idle.ws>
- Note, I think that this still needs work, but it is a lot better memory wise. Signed-off-by: Jim.Idle <jimi@idle.ws>
Some of the legacy code that tried to implement comparisons and hashing like Java was just plain wrong. That stuff is buried deep in there and took a massive effort to work out what was going wrong. In the end, while the hash codes were OK for DFAState, the comparisons inherited from the old code were just plain wrong for certain objects. With those corrected, there is a massive improvement in performance all around. I will push a PR for this stuff now, but I already see some massive potential gains by getting rid of interfaces where we don't need them stopping the use of pointers everywhere indiscrimiately. Next task is going to be pure performance improvements until the Go runtime takes its rightful place as the fastest implementation ;) Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Jim.Idle <jimi@idle.ws>
- I will of course fix this in a follow up PR, but the test DropLoopEntryBranchInLRRule_4 currently hangs. It is also disabled for many other runtimes, so I think it is fine to get this PR in and then follow up with a fix for this test. Signed-off-by: Jim.Idle <jimi@idle.ws>
- Removes the need to perform a `go mod tidy` for every test. This change causes `go mod tidy` to be run once, after which it caches the go.mod and go.sum files for use by the remaining tests. Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Jim.Idle <jimi@idle.ws>
Signed-off-by: Jim.Idle <jimi@idle.ws>
… flowcontrol - 50% performance improvement
- Prior to this change, a recognition error was tracked by performing a panic(),
which the generated code for rules would then use recover() to discover. However,
recover() is not like catch(){} in Java and is expensive to run even if there is
no panic() to find on the execution stack. Eliminating this and doing a simple check
at the end of rule execution brings with it a massive performance improvement up to
50% of CPU execution time. Now that collections and context caching is working correctly
this is a significant improvement in execution time.
Signed-off-by: Jim.Idle <jimi@idle.ws>