Repository navigation
perf: bound single-token lookahead probes - #2829
Conversation
2700c05 to
506b5ce
Compare
Keep complete single-token restrictions at depth one and remove one investigated redundant destination guard. Add a complete decision inventory, content-addressed snapshots and incremental FIRST/FOLLOW and site reuse. Bind full review evidence to caller reachability, transitive grammar and opaque inputs. Keep local token certificates explicit and limited in scope. Preserve the inherited test process environment for Windows Git fixtures and refresh the two review bindings after the watched build input changes. Signed-off-by: 付典 <fudianchn@gmail.com>
|
@fudianchn: I love this very much and I am honestly impressed by it! But I also wonder:
|
506b5ce to
1c729e4
Compare
|
@manticore-projects Thanks! On the cleanup and performance question: this narrows 57 single-token lookaheads across 27 productions and removes one redundant lookahead guard. I reran the parser JMH comparison on the same machine, using the same JDK, JVM settings and 54-statement corpus, with interleaved baseline/candidate runs. Across 12 JVM forks and 120 measurement iterations, the aggregate time for parsing the full corpus went from 17.832 to 17.539 ms/op, a 1.64% reduction. However, the two paired comparisons were -0.05% and -3.21%, and host load was lower toward the end. The fork-bootstrap 95% interval for the aggregate change was [-2.91%, -0.40%], but that does not account for time-dependent host-load drift. The earlier run was also mixed (-1.04% and +0.49%). I would therefore not present this as a meaningful, reliably demonstrated parser speedup. The main value is reducing unnecessary lookahead and making the grammar easier to understand and maintain. The tool provides an evidence-based way to review future LOOKAHEAD changes. From a purely performance perspective, these measurements have not established a substantial practical improvement. |
|
@manticore-projects On your second point, I have opened a draft PR for the reusable audit tooling in your JavaCC core fork: manticore-projects/javacc-8-core#1 . The extraction and JSqlParser integration have been validated locally. The JavaCC PR documents the outstanding parent-build and artifact-publication questions. This JSqlParser PR still contains the original in-repository tooling; it has not yet switched to the extracted implementation or an unpublished JavaCC dependency. I am marking this PR ready for review with that distinction made explicit. |
|
This is fantastic work! |
|
I would like to suggest to move all of this into JavaCC (only) and to only apply the Grammar fixes. Thank you much for all your work and effort. |
|
Would you like me to narrow this PR down to just the grammar fixes, or are you planning to handle that as part of the JavaCC integration? |
Yes, exactly. Thank you much! |
Signed-off-by: 付典 <fudianchn@gmail.com>
fudianchn
changed the title
|
Narrowed this PR to the grammar changes and parser regressions, and merged current master to resolve the conflict. The in-repository analyzer, Gradle task, CI integration and audit documentation are removed. JavaCC integration remains separate in manticore-projects/javacc-8-core#1. |
|
Thank you very much! ~~One question for my understanding though: What is the effective difference between I would have expected (and hoped for) both results in the same parser code? Although not tested myself.~~ Answered it, |
Thank you again, I look into this asap. |
AI disclosure: this change was prepared with AI coding agents. This revision is awaiting my line-by-line review.
What
Bound 57 complete single-token syntactic probes to depth one and remove the redundant depth-three guard in
DBMSDestination. Keep the parser regression tests. Remove the in-repository audit tooling, its build task, CI integration and documentation, as requested.Why
Complete one-token probes need only one token of lookahead. The
DBMSDestinationalternatives start with distinct TABLE and STATEMENT tokens, so their existing dispatch does not need a three-token probe.How
Keep each explicit token restriction as
LOOKAHEAD(1, ...); deleting it could widen the decision to the following production. Preserve semantic predicates and multi-token probes. Merge current master while retaining its newer grammar changes. Reusable audit tooling is tracked separately in manticore-projects/javacc-8-core#1; this PR adds no JavaCC integration or dependency.Root cause
An unbounded syntactic probe generates scan machinery even when its complete restriction is a single token. The DBMS destination guard also probes beyond the token needed to choose its alternative.
Testing
parseSQLStatements, identical corpus and benchmark bytecode, JDK 17.0.20 on an i9-13900KS (32 logical CPUs), interleaved master/candidate twice: each window has 3 forks, 3 three-second warmups and 10 one-second measurements per fork (120 measurement samples overall). Master: 19.447 ± 1.042 and 17.942 ± 0.441 ms/op; candidate: 18.575 ± 1.116 and 18.431 ± 0.712 ms/op (JMH 99.9% confidence intervals). Paired changes are -4.48% and +2.73%; intervals overlap and host load varies. These measurements establish neither a reliable speedup nor a stable slowdown. An initial three-second-total warmup attempt was stopped after observing continued warmup drift and is excluded from these figures.Behavior notes
EXA AT conn TABLEremains invalid. It now reports the missing table name after consuming TABLE, rather than falling through to the STATEMENT alternative. Exact diagnostic text and location are not preserved. No public AST API is changed.