Replace Groovy DSL runtime with ANTLR4 + Javassist for MAL, LAL, and Hierarchy by wu-sheng · Pull Request #13723 · apache/skywalking

@wu-sheng @claude

Document the detailed implementation plan for eliminating Groovy from
OAP runtime via build-time transpilers (MAL/LAL) and v1/v2 module
split (hierarchy), based on Discussion #13716 and skywalking-graalvm-distro.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Add MalExpression, MalFilter, LalExpression functional interfaces and
SampleFamilyFunctions (TagFunction, SampleFilter, ForEachFunction,
DecorateFunction, PropertiesExtractor). Add Java functional interface
overloads alongside existing Groovy Closure methods in SampleFamily,
FilterSpec, ExtractorSpec, and SinkSpec. Change InstanceEntityDescription
to use Function instead of Closure. All 129 existing tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…hase 2)

Ports MalToJavaTranspiler from skywalking-graalvm-distro into a new
mal-transpiler analyzer submodule. The transpiler parses Groovy MAL
expressions/filters via AST at CONVERSION phase and emits equivalent
Java classes implementing MalExpression/MalFilter interfaces from Phase 1.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…hase 3)

Introduces lal-transpiler module that parses LAL Groovy DSL scripts into
AST at Phases.CONVERSION and emits pure Java classes implementing
LalExpression. Handles filter/text/json/yaml/extractor/sink/abort blocks,
parsed property access, safe navigation, cast expressions, GString
interpolation, and SHA-256 deduplication. Makes MalToJavaTranspiler.escapeJava()
public for cross-module reuse. Includes 37 comprehensive tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…ing (Phase 4)

Introduces meter-analyzer-v2 and log-analyzer-v2 modules that provide
same-FQCN replacement classes for DSL.java, Expression.java, and
FilterExpression.java. The v2 classes load transpiled MalExpression/
MalFilter/LalExpression implementations from META-INF manifests via
Class.forName() instead of Groovy GroovyShell/ExpandoMetaClass/
DelegatingScript. Uses maven-shade-plugin to overlay the upstream
Groovy-dependent classes. Includes 7 unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

… (Phase 5)

Extract hierarchy matching rules from HierarchyDefinitionService into
pluggable HierarchyRuleProvider interface. Remove Groovy imports from
server-core by replacing Closure<Boolean> with BiFunction<Service,Service,Boolean>.

- hierarchy-v1: GroovyHierarchyRuleProvider (for CI checker only)
- hierarchy-v2: JavaHierarchyRuleProvider with 4 built-in rules + 12 tests
- HierarchyDefinitionService: add HierarchyRuleProvider interface, DefaultJavaRuleProvider
- HierarchyService: .getClosure().call() → .match()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…ase 6)

Three checker modules verify v1 (Groovy) and v2 (transpiled Java) produce
identical results: hierarchy rules (22 tests), MAL expressions (1187 tests),
MAL filters (29 tests), and LAL scripts (10 tests). Zero behavioral
divergences found when both paths succeed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…L to on-the-fly compilation

- Merge mal-grammar + mal-compiler into meter-analyzer
- Merge lal-grammar + lal-compiler into log-analyzer
- Merge hierarchy-rule-grammar + hierarchy-rule-compiler into hierarchy
- Remove 6 standalone modules (3 grammar + 3 compiler)
- Update DSL.java to compile MAL expressions on-the-fly via MALClassGenerator
  instead of loading from non-existent manifest file
- Add varargs handling for tagEqual/tagNotEqual/tagMatch/tagNotMatch in
  generated Javassist code (wrap String args in new String[]{})
- Update test/script-compiler checker POMs to reference merged module names
- Update CLAUDE.md files with merged file structure and paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

- Fix MAL sample collection regression: skip downsampling() method
  arguments to prevent enum values (MAX, SUM, MIN) from being
  collected as sample names
- Fix MAL safe navigation (?.): parser now correctly propagates
  safeNav flag to chain segments; code generator uses local
  StringBuilder to avoid corrupting parent buffer
- Fix MAL filter grammar: add closureCondition alternatives to
  closureBody rule for bare conditions like { tags -> tags.x == 'v' }
- Fix MAL downsampling detection for bare identifiers parsed as
  ExprArgument wrapping MetricExpr
- Fix MAL sample ordering: use LinkedHashSet for consistent order
- Fix LAL tag() function call: add functionName rule allowing TAG
  token in functionInvocation for if(tag("LOG_KIND") == ...) patterns
- Fix LAL ProcessRegistry support: add PROCESS_REGISTRY to
  valueAccessPrimary grammar rule
- Fix LAL tag statement code generation: wrap single tag entries in
  Collections.singletonMap() since ExtractorSpec.tag() accepts Map
- Fix LAL makeComparison to handle CondFunctionCallContext properly
- Add debug logging to all three code generators (MAL, LAL, Hierarchy)
  showing AST and generated Java source at DEBUG level
- Add generateFilterSource() to MALClassGenerator for testing
- Add error handling unit tests with demo error comments for MAL (5),
  LAL (4), and Hierarchy (4) generators
- All 1248 checker tests pass: MAL 1187, Filter 29, LAL 10, Hierarchy 22

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…ng all four DSL compilers (OAL, MAL, LAL, Hierarchy). Remove Groovy references from docs: LAL code blocks, hierarchy matching rule labels, and stale MeterProcessor comment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Provides a /run-e2e slash command with prerequisites (e2e CLI,
swctl, yq install instructions), rebuild detection, test execution,
and failure debugging workflow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…, interpolated sampler IDs

Address five critical gaps in the LAL v2 compiler that broke shipped production rules:

1. tag("LOG_KIND") in conditions now emits tagValue() helper instead of null
2. Safe navigation (?.) for method calls emits safeCall() helper to prevent NPE
3. Metrics, slowSql, sampledTrace, sampler/rateLimit blocks generate proper
   sub-consumer classes with BindingAware wiring
4. else-if chains build nested IfBlock AST nodes instead of dropping
   intermediate branches
5. GString interpolation in rateLimit IDs (e.g. "${log.service}:${parsed.code}")
   parsed into InterpolationPart segments and emitted as string concatenation

Also fixes ProcessRegistry static calls to pass arguments through, and adds
comprehensive tests (55 total: 35 generator + 20 parser) covering all gaps
including production-like envoy-als, nginx, and k8s-service rule patterns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…cripts and runtime comparison

- Rename test/script-compiler to test/script-cases/script-runtime-with-groovy
- Copy all shipped production configs into test/script-cases/scripts/ as test copies
  (MAL: test-otel-rules, test-meter-analyzer-config, test-log-mal-rules, test-envoy-metrics-rules;
   LAL: test-lal; Hierarchy: test-hierarchy-definition.yml)
- Update all checker tests to load from shared scripts/ directory
- Upgrade LAL checker from compile-only to full runtime execution comparison
  (v1 Groovy vs v2 ANTLR4+Javassist, comparing Binding state: service, layer, tags, abort/save)
- Update Maven coordinates and root pom module path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Move benchmarks from the standalone oap-server/microbench module into
the src/test/ directories of the modules they actually test (server-core
and library-util). Drop AbstractMicrobenchmark base class in favor of
self-contained @test run() methods. Bump JMH 1.21 -> 1.37 and remove
the obsolete -XX:BiasedLockingStartupDelay=0 JVM flag (removed in JDK 18).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

MAL compiler fixes (closes 38 previously failing expressions):
- Add ternary operator (?:) support in closures (grammar, AST, codegen)
- Fix valueEqual() and other primitive-double methods with numeric literal args
- Support double-paren argument syntax: sum((['cluster']))
- Handle NUMBER / SampleFamily via MalRuntimeHelper.divReverse() in v2 package
- Add variable declarations, map literals, forEach/instance closure types
- Add ProcessRegistry class references, improved safe navigation

LAL compiler fixes:
- Fix null-to-string conversion: use null-safe toStr() instead of String.valueOf()
- Add camelToSnake field name fallback for protobuf field access
- Add typed execute(FilterSpec, Binding) method signature
- Reorganize LAL test scripts into oap-cases/ and feature-cases/
- Add data-driven LALExpressionExecutionTest with 27 test cases

MAL checker enhancements:
- Add runtime execution comparison (mock SampleFamily data, execute both
  v1 and v2, compare output samples with labels and values)
- Handle increase()/rate() by priming CounterWindow with initial run
- Extract tagEqual patterns from expressions for matching mock data

All 1,187 MAL + 29 LAL + 22 hierarchy expressions now pass with zero gaps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…L typed signature

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Give v2 (ANTLR4+Javassist) classes distinct FQCNs from v1 (Groovy)
so both can coexist on the classpath without source duplication in
v1-with-groovy test modules.

Package mapping:
- MAL: meter.analyzer.* → meter.analyzer.v2.*
- LAL: log.analyzer.* → log.analyzer.v2.*
- Hierarchy: config.compiler.* → config.v2.compiler.*

Also: remove v2-only files (MalExpression, MalFilter, LalExpression)
from v1-with-groovy modules, add mal-v1-with-groovy dependency to
lal-v1-with-groovy, fix cross-version enum comparison by name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…ime() scalar in MAL compiler

Add ANTLR4 lexer mode for regex literals (=~ /pattern/), def keyword with
type inference from initializer (String[][] for regex, String[] for split),
GString interpolation expansion, .size() to .length translation, decorate()
bean-mode closures, and time() as a scalar function in binary expressions.
Verified with 1,228 v1-v2 checker tests (1,197 MAL + 31 filter).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…v1-v2 checker data

Rewrite MAL run() code generation to use a single reassigned 'sf' variable
instead of multiple intermediate variables, producing cleaner decompiled output.
Add LocalVariableTable attribute so decompilers show 'samples' and 'sf' instead
of 'var1' and 'var2'. Integrate v2 compilers with runtime wiring, add checker
test data files, and clean up unused code across MAL/LAL/Hierarchy modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Add LVT attribute to LAL execute() and consumer accept() methods, and to
Hierarchy apply() method, so decompilers show meaningful variable names
(filterSpec, binding, _t, u, l) instead of var0, var1, etc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Move 8 helper methods (getAt, toLong, toInt, toStr, toBool, isTruthy,
tagValue, safeCall) from being duplicated in every generated class via
addHelperMethods() to a shared LalRuntimeHelper in the rt package.
Generated code now calls LalRuntimeHelper.toStr() etc. via FQCN.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…th typed methods

Fix rateLimit() calls inside if-blocks within sampler generating empty bytecode
by handling the samplerContent grammar alternative in LALScriptParser.visitIfBody().

Replace generic LalRuntimeHelper.safeCall() and isTruthy() with specific typed
methods: isTrue() for Boolean conditions, isNotEmpty() for String non-emptiness,
toString() and trim() for null-safe navigation — making generated code explicit
about intended type semantics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…licitly, add LVT

- Merge consumer sub-classes into single generated class with private methods
- Remove BINDING ThreadLocal from AbstractSpec; all spec methods take ExecutionContext explicitly
- Delete BindingAware.java and Binding.java, replace with ExecutionContext
- Add abort guard before _extractor/_sink calls matching v1 Groovy behavior
- Add LocalVariableTable to all generated methods (execute, _extractor, _sink)
- Rename binding→ctx throughout for consistency
- Add extraLogType to envoy-als.yaml for compile-time proto resolution
- Remove all Consumer callback methods from spec files
- Add finalizeSink abort check in FilterSpec

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…es, filters)

Previously only run() had LVT. Now all generated methods have named locals
in debuggers/decompilers instead of var0/var1/var2:
- metadata(): this, _samples, _scopeLabels, _aggLabels, _pct
- tag/instance apply(Map): this, param name
- tag/instance apply(Object) bridge: this, o
- forEach accept(): this, element, tags
- decorate accept(): this, _arg, param name
- filter test(): this, param name

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…version 10.4.0-SNAPSHOT

Replace CI-friendly ${revision} with hardcoded 10.4.0-SNAPSHOT in all 104 POMs.
This eliminates persistent "Could not find artifact ...pom:${revision}" errors
when building individual modules without -am. Also removes flatten-maven-plugin
(no longer needed), updates release scripts to use versions:set, and wires
LALSourceTypeProvider SPI for envoy-als extraLog type resolution in tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng @claude

…chains

Cache the extraLog cast in a _p local variable and break safe-nav chains
into sequential _tN locals instead of deeply nested ternaries. Repeated
access to the same chain prefix reuses existing variables (dedup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@wu-sheng

@wu-sheng @claude

Remove bind()/evaluate() two-phase pattern from DSL. The mutable
ExecutionContext field made DSL unsafe for concurrent use. Now
evaluate(ExecutionContext) takes ctx as a parameter, matching the
stateless pattern already used by MAL and Hierarchy v2 runtimes.

Update LogFilterListener to store per-request contexts in a list
and pass each to the corresponding DSL.evaluate(ctx) call.
Update LogTestQuery to use the new single-call API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>