Skip to content

feat(dart): Dart 3 grammar bump, parser fixes, and pattern-mode capabilities#11678

Closed
AlexLaroche wants to merge 27 commits into
semgrep:developfrom
AlexLaroche:alexandre/bump-tree-sitter-dart
Closed

feat(dart): Dart 3 grammar bump, parser fixes, and pattern-mode capabilities#11678
AlexLaroche wants to merge 27 commits into
semgrep:developfrom
AlexLaroche:alexandre/bump-tree-sitter-dart

Conversation

@AlexLaroche

@AlexLaroche AlexLaroche commented May 2, 2026

Copy link
Copy Markdown
Contributor

Draft — gated on semgrep/ocaml-tree-sitter-semgrep#590 landing and an r2c admin running ./release dart to publish a real release commit to semgrep/semgrep-dart. See "Before marking ready" below.

Summary

  • Bumps the languages/dart/tree-sitter/semgrep-dart submodule to a regenerated artifact built from upstream UserNobody14/tree-sitter-dart@02e9bb19 (13 commits ahead of the May-2023 pin), and adapts Parse_dart_tree_sitter.ml to the new CST shape.
  • Resolves the headline Dart 3 parsing gap: empty-paren object_pattern (e.g. _MyClass() arms in switch), now common in Flutter (upstream PR #93).
  • Resolves the @annotation + record-return-type method gap (e.g. @override (R, T) method() {…}, the Elm-style architecture pattern in Flutter) — this was the dominant remaining PartialParsing cause on the pub.dev catalog (upstream PR #98).
  • Also picks up: 16 grammar fixes from the upstream parser-improvements PR (#97) — empty record literal (), dot-shorthand patterns, abstract/external fields, null-aware map keys, dotted/operator symbol literals, single-named-field record literals, unnamed library directive, and more.
  • Drops the stale "tree-sitter-dart is currently buggy" segfault gate in Parse_target.ml and the Lang.Dart exclusion in Rule_fetching.ml — the original repro (': string (* filename *)') now exits cleanly.
  • Fixes three parser-side bugs in Parse_dart_tree_sitter.ml discovered during pattern-validation work that reproduce on develop too:
    1. Uninitialized local variable declarations were silently dropped. String z; / late String w; inside a function body never made it into the AST, so patterns like String $X; could not match local declarations. Fixed by emitting (id, None) instead of [] when the optional = expr is absent.
    2. Patterns starting with a Dart statement keyword silently misparsed. if (kDebugMode) { ... } parsed as a function definition (with if as the return type), await foo(); parsed as a bare Call(foo, []) with await stripped. Detect statement-keyword starts up-front and route through the __SEMGREP_EXPRESSION path, bypassing the misleading file-parse.
    3. Function-definition patterns silently degraded to calls. void test() { ... } reinterpreted as Call(test, []) (losing return type, body, and function context), so pattern-inside: void test() { ... } could never match a function definition. Tightened the forward-decl ambiguity heuristic so only truly-ambiguous shapes (no return type, no params, empty body, no attrs) collapse to a Call.
  • Adds three Dart-side metavariable capabilities that other major languages already had:
    1. Typed metavariable $X as T — in pattern mode, the existing Dart as cast is reinterpreted as TypedMetavar when the LHS is a metavariable (TS/Java/Swift use the same trick).
    2. metavariable-type: T filter — Dart registered in Parse_metavariable_type (x as %s wrap, Cast unwrap).
    3. Metavariables inside string interpolationsprint("hello $X") now binds $X to the interpolation expression on Dart targets, matching Semgrep's convention everywhere else. Real Dart $name interpolations (lowercase) keep the literal-text behavior so existing patterns still match.

Validation

Generated the artifact via the standard flow (make && make install && cd lang && ./test-lang dart && ./release dart --dry-run) on ocaml-tree-sitter-semgrep with the bumped submodule, then tested the regenerated parse-dart:

Corpus Old grammar (80e23c07) New grammar (02e9bb19)
199 random .dart files from flutter/flutter@master (1-30 KB) 196 ok / 3 fail 199 ok / 0 fail
5,285 latest-version packages on pub.dev ~102 PartialParsing 56 PartialParsing (1.1%)

The 3 Flutter files that previously failed all live inside developer tooling and exercise Dart 3 patterns:

New tests (8)

Added regression tests covering every distinct fix and capability in this branch:

Test Covers
tests/parsing/dart/switch_object_pattern.dart Empty-paren / field-bearing / generic / guarded object_pattern arms in switch (Dart 3 #93)
tests/parsing/dart/dart3_grammar_features.dart Empty record (), single-named record (name: v,), dot-shorthand .foo/.new, null-aware map element ?key: value, null-assertion ! in cascade, unnamed library directive, dotted/operator symbol literals (#a.b.c, #==), get/set as identifiers
tests/patterns/dart/uninit_local_var.{dart,sgrep} String $X; matches uninitialized locals (parser fix #1)
tests/patterns/dart/function_def_void.{dart,sgrep} void test() { ... } matches function definitions (parser fix #3)
tests/rules/if_pattern_dart.{yaml,dart} if (kDebugMode) { ... } matches real If statements (parser fix #2if keyword routing)
tests/rules/await_pattern_dart.{yaml,dart} await fetch($URL) matches only awaited calls (parser fix #2await keyword routing)
tests/rules/typed_metavar_dart.{yaml,dart} print($X as String) binds against String args/decls only (typed metavariable)
tests/rules/metavar_type_dart.{yaml,dart} metavariable-type: String filter (typed metavariable)
tests/rules/string_interp_metavar_dart.{yaml,dart} print("hello $X") binds the interpolation expression

Commit groups

This branch is 21 commits — grouped by intent:

  1. Submodule bump (commit 1)chore(dart): bump tree-sitter-dart and adapt OCaml mapper for Dart 3 style commit at the base.
  2. CST-shape adaptations (commits 2–10) — one commit per upstream grammar change, each adjusting Parse_dart_tree_sitter.ml to consume the new CST shape: abstract/external var, label/operator identifiers, empty-statement, external instance fields, for-loop/field-initializer changes, null-aware map key, null-assertion in cascades, ffi external get/set fields, full symbol literals, single-named record, record-return method.
  3. Parser bug fixes (commits 11–14) — independent of the bump, surfaced during pattern validation: uninitialized-local-decl drop, statement-keyword pattern routing, await/rethrow keyword routing, function-definition pattern degradation.
  4. Pattern-mode capabilities (commits 17–19) — typed metavariable $X as T + metavariable-type registration, metavariables inside string interpolations, function-def pattern matching.
  5. Cleanup (commit 15) — drop the stale segfault gate in Parse_target.ml and the Lang.Dart exclusion in Rule_fetching.ml.
  6. Validation pointer (commit 16)temporary redirect of .gitmodules so this branch resolves the validation artifact from AlexLaroche/semgrep-dart. Drop before merge.
  7. Style (commit 20)style(dart): apply ocamlformat 0.29.0 to satisfy CI's lint-ocaml hook (CI uses 0.29.0; recent edits were made with a local 0.27.0 whose break-cases differs).
  8. Tests (commits 21–22) — 8 new tests covering every fix and capability above (table in the previous section).

Before marking ready for review

  • fix(dart): bump tree-sitter-dart to upstream master (02e9bb19) ocaml-tree-sitter-semgrep#590 lands (bumps the upstream tree-sitter-dart submodule there to 02e9bb19).
  • An r2c admin runs ./release dart on ocaml-tree-sitter-semgrep to publish a real release commit on semgrep/semgrep-dart.
  • Drop the chore(dart): point semgrep-dart submodule URL at fork for validation commit.
  • Re-point the submodule pointer from c5a5e6c1 (on AlexLaroche/semgrep-dart) to whatever real release SHA appears on semgrep/semgrep-dart.
  • CI green on this repo.

Test plan

  • make core clean against the bumped submodule.
  • Dart lang-parsing tests pass; Dart sgrep-pattern tests pass.
  • Standalone parse-dart: 199/199 files in flutter/flutter@master parse cleanly (was 196/199).
  • pub.dev: 5,229/5,285 fully clean (was ~5,183), 56 PartialParsing (was ~102).
  • Original "segfault" repro (: string (* filename *)) — exits cleanly, no signal, on both parse-dart and semgrep-core -dump_pattern.
  • Pre-commit lint-ocaml hook passes locally with ocamlformat 0.29.0.
  • CI on this repo (semgrep/semgrep).

Bumps the semgrep-dart submodule to 452f0be (regenerated from upstream
tree-sitter-dart 0fc19c3a, 11 commits ahead of the 2023-05 pin), which
fixes parsing of Dart 3 idioms now common in Flutter:
  - empty-paren object patterns in switch arms ('_MyClass()' arms),
  - Dart 3.10 dot-shorthand expressions,
  - extension type declarations,
  - tree-sitter 0.25.x ABI compat,
  - 'set'/'get' contextual keywords usable as identifiers.

Adapts Parse_dart_tree_sitter.ml to the new CST shape:
- declared_identifier/simple_formal_parameter: 4th component is now
  'anon_choice_id_2354d68' (Id | Get | Set).
- map_element: '`Exp' replaced by '`Opt_QMARK_exp' (null-aware
  element); the leading "?" is preserved as
  'OtherExpr ("NullAwareElement", _)' so rules can distinguish it
  from a plain expression.
- '`Pair' gained an optional '?' token before the value.
- map_if_null_expression_: takes 'if_null_expression' (the list)
  directly; outer constructor renamed to '`If_null_exp_'.
- map_object_pattern: middle field is now optional, fixing the
  empty-paren case.
- map_primary: new '`Get'/'`Set'/'`Dot_shor' arms (dot shorthand
  mapped to OtherExpr "DotShorthand"; set/get treated as identifiers).
- map_top_level_definition: stub for '`Exte_type_decl' (extension
  type) emitting OS_Todo carrying the declared identifier so rules
  can still reference the type name.
- map_declaration_: handle '`Exte_and_static_type_id' as a typed
  VarDef.

Adds tests/parsing/dart/switch_object_pattern.dart guarding the
headline fix (empty-paren object patterns + field-bearing patterns +
generic + guarded patterns + null pattern).

Validation: 199/199 .dart files from flutter/flutter@master parse
cleanly with the regenerated parse-dart (was 196/199); all 15 Dart
parsing tests pass.
…tors

Several improvements to Parse_dart_tree_sitter.ml independent of the
grammar bump:

- map_method_signature: pass the actual body to getter/setter mappings
  (was FBNothing) and label methods as Method instead of Function.
  Methods were silently losing their body in the AST, breaking taint
  analysis and any rule that matches inside a method.

- map_string_literal: skip the synthetic ConcatString wrapper for
  single-element string literals. Wrapping inserted a Call node that
  interfered with patterns matching plain string literals.

- map_template_substitution: emit Middle3 (the expression slot) for
  '${expr}' interpolations instead of Right3, and apply the same
  Pattern/Program env distinction used by map_sub_string_test for
  bare '$id' interpolations. This brings interpolation handling in
  line with the rest of the string parser.

- map_selector: use Dart's universal naming convention as a heuristic
  to disambiguate Call from New for '$X(...)'. Identifiers starting
  with an uppercase letter are types, so 'Foo()' becomes a New
  expression; lowercase stays a Call. Improves rule precision on
  taint sources/sinks that target constructors.

- File_type.ml/.mli: add Dart to pl_type and recognize '.dart' so
  non-parser code paths that switch on pl_type classify Dart files
  correctly.

- Unit_engine.ml: enable Dart tainting tests by registering the
  'tests/tainting_rules/dart' directory.

Tests:
- tests/patterns/dart/function_call.{dart,sgrep},
  function_def.{dart,sgrep}: new pattern tests covering Call/New
  disambiguation and method-body presence.
- tests/tainting_rules/dart/arrays_if.{dart,yaml},
  try_return.{dart,yaml}: tainting fixtures exercising arrays and
  try/return paths.
- 5 existing parsing snapshots refreshed (Method label, New for
  uppercase-starting calls, simplified single-element strings).

Validation: 199/199 .dart files from flutter/flutter@master parse
cleanly; 19 Dart tests pass (was 14).
The 'tree-sitter-dart is currently buggy and can generate some
segfaults' note (added 2023-11) referenced an input that no longer
crashes the bundled grammar. The original repro (': string (* filename
*)') now exits cleanly through both the standalone parse-dart and the
semgrep-core dump_pattern path.

- src/parsing/Parse_target.ml: remove the stale comment above the
  Lang.Dart arm.
- src/osemgrep/networking/Rule_fetching.ml: drop the
  'List_.exclude (... =*= Lang.Dart)' so '-e <pat>' (no '-l') tries
  Dart along with every other language.
Temporary: redirects the submodule URL from returntocorp/semgrep-dart
to AlexLaroche/semgrep-dart so a fresh clone of this branch resolves
the regenerated artifact at 452f0be (which only exists on the fork
until r2c runs './release dart' to publish a real release commit).

Drop this commit before opening the upstream PR — the public PR
should use the original returntocorp/semgrep-dart URL with whatever
SHA r2c produces from the regenerate.
@AlexLaroche AlexLaroche force-pushed the alexandre/bump-tree-sitter-dart branch from b2808d9 to 0e9f348 Compare May 2, 2026 15:22
The tree-sitter-dart parser-improvements branch changes the shape of
three CST nodes:

- record_literal_no_const becomes a polymorphic variant with an empty
  `LPAR_RPAR` case for the unit record `()`.
- const_object_expression becomes a polymorphic variant with a new
  `Const_buil_dot_shor_args` case for `const .new(...)` /
  `const .named(...)`.
- initialized_identifier accepts an aliased `get`/`set` token in
  addition to a plain identifier.
- constant_pattern gains `Dot_shor_opt_opt_type_args_args` for
  `case .red:`, `case .new():`, `case .fromRGB(1, 2, 3):`.

Also extracts the previously inlined dot-shorthand handling from
_primary into a reusable map_dot_shorthand helper.

Bumps the semgrep-dart submodule to the corresponding generated CST.
…apes

Tracks the latest tree-sitter-dart parser-improvements commits:

- Adds a `\`Abst_choice_final_buil_opt_type_id_list` arm to
  map_declaration_, handling the three abstract instance field forms
  (`abstract <type> ids;`, `abstract final <type>? ids;`,
  `abstract covariant <type> ids;`) by emitting a VarDef with an
  unhandled `abstract` keyword attribute.
- Adds a `\`Opt_meta_exte_buil_choice_final_buil_opt_type_id_list_semi`
  arm to the top-level definition mapper, handling top-level external
  variable declarations (`external <type> id;`,
  `external final <type>? id;`, `external late <type>? id;`).

Also picks up small CST tag renames (`Show_id_list_`/`Hide_id_list_`,
`Cova_choice_late_buil_choice_final_buil_opt_type_id_list`,
`Late_buil_choice_final_buil_opt_type_id_list`,
`Final_buil_opt_type_id_list`) introduced when the new
`identifier_list` use-site changed the simplification output.

Bumps the semgrep-dart submodule to the corresponding generated CST.
After the latest tree-sitter-dart parser-improvements commit, the
`label` rule accepts `_get`/`_set`/`_function_builtin_identifier` as
identifier aliases, and `initialized_identifier` accepts `_operator`
as an identifier alias. The corresponding CST shape gained
`Func_buil_id` and `Op` tags.

- map_label now matches `\`Id` / `\`Get` / `\`Set` / `\`Func_buil_id`.
- map_initialized_identifier now matches `\`Id` / `\`Get` / `\`Set` /
  `\`Op`.

Bumps the semgrep-dart submodule to the corresponding generated CST.
Adds an `\`Empty_stmt` arm to `map_statement` that emits an empty
`Block` for `;` (Dart's null/empty statement), matching the new
`empty_statement` rule in the grammar.

Bumps the semgrep-dart submodule to the corresponding generated CST.
Adds an `\`Exte_choice_final_buil_opt_type_id_list` arm to
map_declaration_, handling `external final <type>? <ids>` and
`external covariant <var-or-type> <ids>` by emitting a VarDef with
the appropriate keyword attributes.

Bumps the semgrep-dart submodule to the corresponding generated CST.
Tracks the latest tree-sitter-dart parser-improvements commits:

- The for-loop update list now allows a trailing comma; CST gained
  a third tuple component carrying the optional comma. Renames the
  helper `map_anon_arg_rep_COMMA_arg_eb223b2` → `map_for_init_list`
  and adds `map_for_update_list` for the new shape, and updates the
  `Opt_choice_local_var_decl_opt_exp_semi_opt_exp_rep_COMMA_exp_opt_COMMA`
  variant tag.
- The field initializer's right-hand side is now a full
  `_expression` (not `_real_expression` + cascades). Updates the
  `\`Field_init` arm to take 4 components and call `map_expression`.

Bumps the semgrep-dart submodule to the corresponding generated CST.
The `pair` rule now accepts an optional `?` before the key as well as
before the value (Dart 3 null-aware elements), so the CST tuple has
five components instead of four. Updates the `\`Pair` arm of the
element mapper accordingly.

Bumps the semgrep-dart submodule to the corresponding generated CST.
… fields

Bumps semgrep-dart submodule to pick up two upstream tree-sitter-dart
grammar fixes:

- Cascade subsections now accept the postfix `!` null-assertion
  interleaved with argument parts, so code like
  `obj.._parent!.onChildrenChanged(this)` (common in flame, flutter
  widget code) parses cleanly.
- Static external fields can be named with built-in identifiers
  (`get`, `set`, `operator`), used by `dart:ffi` Struct field
  conventions like `external SomeType get;`.

Updates the OCaml mapper for the two new CST shapes:
- New `anon_choice_arg_part_7fd04ec` choice between argument parts and
  the `!` token in cascade contexts (factored into a helper).
- Existing `Exte_and_static_type_id` arm renamed to
  `Exte_and_static_type_choice_id` with a 4-arm choice for the field
  name.
…amed record

Bumps semgrep-dart submodule for three upstream tree-sitter-dart
grammar fixes:

- `library;` (unnamed library directive, Dart 2.19+) — common in files
  that exist solely to carry a doc comment for the library.
- Full symbol literal forms — `#a.b.c` (dotted) and operator symbols
  (`#+`, `#==`, `#[]`, `#[]=`, `#~`, etc.) per the Dart spec.
- Single-named-field record literal with trailing comma
  (`(name: value,)`), required by Dart 3 to disambiguate from a
  labeled parenthesized expression.

Updates the OCaml mapper for the new CST shapes:
- `map_library_name` makes the dotted_identifier_list optional.
- Symbol-literal arms now dispatch on the choice (dotted ids /
  equality / relational / shift / additive / multiplicative / single
  operator tokens) in both expression and pattern positions.
- New `Label_exp_COMMA` arm in record_literal_no_const for the
  single-named-field-with-trailing-comma form.
@CLAassistant

CLAassistant commented May 5, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

The tree-sitter-dart grammar was extended to parse methods whose return
type is a record type when preceded by an annotation (e.g.
`@override (Model, Cmd?) update(Msg msg) { ... }` — Elm-style
architecture pattern). The new grammar introduces hidden rules that, on
the OCaml side, surface as new CST types:

  - bare_annotation                       (just `@name`, no args)
  - record_return_function_signature      (record-type return + name + params)
  - record_return_method_signature        (optional `static` + above)
  - record_return_class_member            (annotations + method + body)

Together with a new `Rep1_bare_anno_record_ret_func_sign_func_body`
top-level variant and a `Record_ret_class_member` variant in
class/extension/enum bodies.

Mapper additions:

  - `map_record_return_function_signature` — produces the same
    `(fkind, fbody) -> stmt` shape as `map_function_signature`. Uses
    `:>` to coerce the inner record-type choice (which excludes the
    empty `()` form intentionally) to `CST.record_type`, so the
    existing `map_record_type` is reused without duplication.
  - `map_record_return_method_signature` — wraps with the optional
    `static` keyword attribute, mirroring the `Opt_static_choice_func_sign`
    arm of `map_method_signature`.
  - `map_bare_annotation` — produces a `NamedAttr` with empty args,
    mirroring `map_annotation` minus the optional argument list.
  - `map_record_return_class_member` — orchestrates annotations +
    method signature + body into a `G.field`.

Mapper updates:

  - `map_extension_body` — added `Record_ret_class_member` arm.
  - `map_top_level_definition` — added
    `Rep1_bare_anno_record_ret_func_sign_func_body` arm.
  - Renamed `map_anon_rep_opt_meta_class_member_defi_cd2fbdb` to
    `map_anon_choice_opt_meta_class_member_defi_44d3600` and rewrote it
    to handle the new variant-based CST type (single element instead of
    list); updated the two callers in `map_class_body` and
    `map_enum_body` to `List.map` over it.

Verified end-to-end: `make core` clean; dart test suite identical to
master baseline (41 pass, 1 pre-existing failure unrelated to the
mapper); previously-failing pub.dev packages (equations, dascade,
medea_flutter_webrtc, password_engine) parse at 99.9% through the full
CLI -> semgrep-core -> mapper -> tree-sitter pipeline.
…word

Dart pattern parser tried the source-file parse first and only fell
back to the expression/statement parse when the file parse produced
errors. That breaks patterns starting with `if`, `for`, `while`,
`do`, `switch`, `try`, `return`, `throw`, `break`, `continue`,
`assert`, or `yield`: Dart's grammar doesn't allow statements at
top level, but `if (cond) { ... }` happens to parse silently as a
function definition (with `if` as the return type, `(cond)` as the
parameter list, `{ ... }` as the body) — no errors are raised, so
the fallback never fires.

Concretely, a pattern like

  pattern-not-inside: |
    if (kDebugMode) {
      ...
    }

was producing a function-definition AST and never matching the
target's actual `If(...)` node.

Detect statement-keyword starts up-front and route them through the
`__SEMGREP_EXPRESSION` fallback path directly, bypassing the
misleading file parse. `for`/`while`/`try`/etc. happened to land on
the fallback path before because their file-parse produced errors,
but `if` slipped through; this makes the behavior uniform across all
statement starters.
map_initialized_variable_definition_unwrapped dropped the variable
when there was no `= expr` initializer and no comma-separated
follow-on declarations: the `None` arm returned `[]` and the entity
list ended up empty.

Concretely, inside a function body
  void m() { String z; late String w; }
neither `z` nor `w` made it into the AST, so patterns like
`String $X;` or `late String $X;` could not match local
declarations. Top-level uninitialized declarations were unaffected
because they go through a different mapper path that uses
map_initialized_identifier_list (always non-empty).

Restructure the optional-init branch to produce a single
(id, init_opt) pair that's prepended to the list of follow-on
declarations, so the declared identifier is always emitted.
Pattern `await foo();` parsed at the file level as just `Call(foo, [])`
— the leading `await` was silently stripped — so the pattern matched
bare `foo()` calls (false positive) and never matched the `await foo()`
it was meant to find. Same root cause as the earlier `if (kDebugMode)`
case: the file parser silently produces the wrong shape, so the
errors-based fallback never fires.

Add `await` (and `rethrow` for symmetry) to the statement-keyword list
so these patterns route through the expression/statement parse path
directly. After this change, `await foo();` parses as
ExprStmt(Await(Call(...))) and matches only actual awaited calls.
@yosefAlsuhaibani yosefAlsuhaibani marked this pull request as ready for review May 6, 2026 05:16
@yosefAlsuhaibani yosefAlsuhaibani marked this pull request as draft May 6, 2026 05:16
Pre-commit's `lint-ocaml` hook in CI runs ocamlformat 0.29.0 (pinned
in `dev/required.opam` and `scripts/lint-ocaml`). Recent edits to this
file used a local 0.27.0, whose break-cases preferences are slightly
different — CI's hook reformatted the file and failed the lint job.

Reformat with the matching 0.29.0 to clear the lint failure: pure
indentation/wrapping changes, no behavior diff.
@AlexLaroche

Copy link
Copy Markdown
Contributor Author

@yosefAlsuhaibani: Can you help me get this PR merged? The first step would be to regenerate Dart's grammar. See semgrep/ocaml-tree-sitter-semgrep#590

Adds two related capabilities:

1. In pattern mode, the existing Dart cast `$X as T` is reinterpreted
   as a TypedMetavar — `$X` only binds against expressions whose
   resolved type is `T`. Same trick TS/Java/Swift use, just keyed off
   the metavariable convention (`is_metavar_name`).
2. Registers Dart in `Parse_metavariable_type` (`x as %s` wrap, `Cast`
   unwrap), so `metavariable-type: T` rules now work for Dart.

Verified: the type infrastructure was already in place — `id_info.id_type`
gets populated by Naming_AST for parameters and use sites — only the
pattern-side syntax was missing. With this commit, typed-metavar
patterns and `metavariable-type` filters distinguish String vs int
arguments correctly on `void f(String s, int n) { print(s); print(n); }`.
Dart's `"hello $name"` interpolation collides with Semgrep's `$NAME`
metavariable convention, so the pattern parser previously kept all
interpolated `$id` fragments as raw text in pattern mode — meaning
`print("hello $X")` could not match `print("hello $name")`.

Special-case the uppercase/underscore form: when the interpolated
identifier matches `is_metavar_name`, emit it as a metavariable
expression (same shape as Program mode produces). Real Dart
interpolations like `$name` (lowercase) keep the raw-text behavior so
patterns like `"value=$count"` against `"value=$count"` still match
literally.

Verified: pattern `print("hello $X")` now matches `print("hello $name")`
and binds `$X` to `name`. Edge case of a Dart identifier whose name
happens to be all-uppercase (legal but very unusual) collides — same
trade-off every Semgrep language already accepts.
`map_function_signature` reinterpreted any pattern-mode signature with
empty parameters as a Call, justified by a forward-declaration
ambiguity (`foo();` looks both like a call and a forward decl). The
heuristic was too coarse: `void test() { ... }` was also rewritten as
`Call(test, [])`, losing the return type, body, and the function
context — so `pattern-inside: void test() { ... }` couldn't match any
function definition.

Tighten the condition: only reinterpret as Call when the construct is
truly ambiguous — no return type, no parameters, an empty body
(`FBDecl`/`FBNothing`/empty `FBStmt`), and no attributes. Anything
that looks like a real function definition (return type, parameters,
non-empty body, or attributes) is now mapped to a `DefStmt(FuncDef)`
as expected.

Verified: `void test() { ... }` matches function definitions in target
classes; the original `foo();` forward-decl pattern still matches Call
expressions (no regression).
…avar, function-def patterns

Adds four targeted regression tests for the recent Dart pattern-mode
fixes:

- `tests/rules/typed_metavar_dart.{yaml,dart}` — `print($X as String)`
  matches String-typed args and binds, but skips `int`/`dynamic`. Also
  covers locally-declared variables (`String name = ...`).
- `tests/rules/metavar_type_dart.{yaml,dart}` — `metavariable-type:
  String` filter restricts a `print($X)` pattern to String-typed
  expressions only.
- `tests/rules/string_interp_metavar_dart.{yaml,dart}` — pattern
  `print("hello $X")` binds the interpolation expression and matches
  multiple targets, while leaving non-matching prefix/literal strings
  untouched.
- `tests/patterns/dart/function_def_void.{dart,sgrep}` — pattern
  `void test() { ... }` matches both top-level and method-level
  function definitions named `test`, ignoring same-shape definitions
  with different names.

All four pass; broader Dart suite goes from 41/42 → 45/46 with the
same single pre-existing `biometric-weak-config` failure unchanged.
Adds tests for the parser-side fixes earlier in this branch and a
smoke parsing test covering the new Dart 3 grammar features pulled in
by the submodule bump:

- `tests/patterns/dart/uninit_local_var.{dart,sgrep}` — pattern
  `String $X;` matches uninitialized local declarations. Locks in the
  fix where `map_initialized_variable_definition_unwrapped` was
  returning [] when the optional `= expr` was absent, silently
  dropping `String z;` from the AST.
- `tests/rules/if_pattern_dart.{yaml,dart}` — pattern
  `if (kDebugMode) { ... }` matches real If statements. Locks in the
  statement-keyword pattern-routing fix that prevents `if`-prefixed
  patterns from silently misparsing as function definitions at the
  file level.
- `tests/rules/await_pattern_dart.{yaml,dart}` — pattern
  `await fetch($URL)` matches only awaited calls, not bare ones.
  Locks in the same routing fix for `await`/`rethrow`, where the
  `await` keyword was previously stripped at the file-parse level
  (the `Call(fetch, [...])` shape) and produced false positives.
- `tests/parsing/dart/dart3_grammar_features.dart` — exercises the
  grammar features brought in by the tree-sitter-dart bump: empty
  record `()`, single-named-field record `(name: v,)`, dot-shorthand
  `.foo`/`.new`, null-aware map element `?key: value`, null-assertion
  `!` inside a cascade chain, unnamed library directive `library;`,
  dotted/operator symbol literals (`#a.b.c`, `#==`), and `get`/`set`
  as identifiers.

All four pass; broader Dart suite goes from 45/46 → 49/50 with the
same single pre-existing `biometric-weak-config` failure unchanged.
@AlexLaroche AlexLaroche marked this pull request as ready for review May 6, 2026 12:07
@AlexLaroche AlexLaroche changed the title fix(dart): bump tree-sitter-dart and adapt OCaml mapper for Dart 3 feat(dart): Dart 3 grammar bump, parser fixes, and pattern-mode capabilities May 6, 2026
@yosefAlsuhaibani yosefAlsuhaibani self-requested a review May 6, 2026 19:42
@yosefAlsuhaibani

Copy link
Copy Markdown
Collaborator

@AlexLaroche can do!

@AlexLaroche

Copy link
Copy Markdown
Contributor Author

@AlexLaroche can do!

@yosefAlsuhaibani Please tell me when the command './release dart' has been executed.

@yosefAlsuhaibani

Copy link
Copy Markdown
Collaborator

@AlexLaroche semgrep/semgrep-dart@0202e42.

Let's

  1. get the submodule point to the upstream submodule
  2. add a changelog to changelog.d/

Switch the submodule URL from the AlexLaroche fork back to
returntocorp/semgrep-dart, and bump the pointer to 0202e42 (yosef's
just-merged PR semgrep#1, regenerated against UserNobody14/tree-sitter-dart
02e9bb19).

Drop the `Labe_stmt` arm from the OCaml mapper: enabling
`$.labeled_statement` was a local patch on top of the upstream grammar
snapshot and isn't on UserNobody14/tree-sitter-dart yet, so it can't
ride along with a canonical-upstream submodule pointer. Labeled
statements weren't supported on develop either, so this is not a
regression; it can be re-introduced in a follow-up once the grammar
change lands upstream.

Also add changelog entries for PR semgrep#11678.
Comment thread .gitmodules Outdated
[submodule "OSS/languages/dart/tree-sitter/semgrep-dart"]
path = languages/dart/tree-sitter/semgrep-dart
url = https://github.com/returntocorp/semgrep-dart.git
url = https://github.com/returntocorp/semgrep-dart

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove this part of the diff?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Match the style used elsewhere in `.gitmodules` (develop has `.git`
on every returntocorp/* submodule URL). Functionally equivalent —
GitHub resolves both forms — but stylistically consistent.
@AlexLaroche

Copy link
Copy Markdown
Contributor Author

@yosefAlsuhaibani:
Is there anything missing for this PR to be merged?

@yosefAlsuhaibani yosefAlsuhaibani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexLaroche Looks good; thanks for taking this on! I'll get to syncing this and merging it to develop.

@semgrep-ci

semgrep-ci Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

@yosefAlsuhaibani has imported this pull request. Semgrep employees can find it at https://github.com/semgrep/semgrep-proprietary/pull/6332.

@AlexLaroche, thank you for your contribution! Once the internal PR has been merged, this PR will be automatically marked as closed. Later, the resulting commit will be synced back out to this repository and will reference this PR.

yosefAlsuhaibani pushed a commit that referenced this pull request May 15, 2026
…ilities (semgrep/semgrep-proprietary#6332)

> **Draft — gated on
semgrep/ocaml-tree-sitter-semgrep#590 landing
and an r2c admin running `./release dart` to publish a real release
commit to `semgrep/semgrep-dart`. See "Before marking ready" below.**

## Summary

- Bumps the `languages/dart/tree-sitter/semgrep-dart` submodule to a
regenerated artifact built from upstream
`UserNobody14/tree-sitter-dart@02e9bb19` (13 commits ahead of the
May-2023 pin), and adapts `Parse_dart_tree_sitter.ml` to the new CST
shape.
- Resolves the headline Dart 3 parsing gap: empty-paren `object_pattern`
(e.g. `_MyClass()` arms in `switch`), now common in Flutter (upstream PR
[#93](UserNobody14/tree-sitter-dart#93)).
- Resolves the `@annotation` + record-return-type method gap (e.g.
`@override (R, T) method() {…}`, the Elm-style architecture pattern in
Flutter) — this was the dominant remaining `PartialParsing` cause on the
pub.dev catalog (upstream PR
[#98](UserNobody14/tree-sitter-dart#98)).
- Also picks up: 16 grammar fixes from the upstream parser-improvements
PR ([#97](UserNobody14/tree-sitter-dart#97)) —
empty record literal `()`, dot-shorthand patterns, abstract/external
fields, null-aware map keys, dotted/operator symbol literals,
single-named-field record literals, unnamed library directive, and more.
- Drops the stale "tree-sitter-dart is currently buggy" segfault gate in
`Parse_target.ml` and the `Lang.Dart` exclusion in `Rule_fetching.ml` —
the original repro (`': string (* filename *)'`) now exits cleanly.
- Fixes three parser-side bugs in `Parse_dart_tree_sitter.ml` discovered
during pattern-validation work that reproduce on `develop` too:
1. **Uninitialized local variable declarations were silently dropped.**
`String z;` / `late String w;` inside a function body never made it into
the AST, so patterns like `String $X;` could not match local
declarations. Fixed by emitting `(id, None)` instead of `[]` when the
optional `= expr` is absent.
2. **Patterns starting with a Dart statement keyword silently
misparsed.** `if (kDebugMode) { ... }` parsed as a function definition
(with `if` as the return type), `await foo();` parsed as a bare
`Call(foo, [])` with `await` stripped. Detect statement-keyword starts
up-front and route through the `__SEMGREP_EXPRESSION` path, bypassing
the misleading file-parse.
3. **Function-definition patterns silently degraded to calls.** `void
test() { ... }` reinterpreted as `Call(test, [])` (losing return type,
body, and function context), so `pattern-inside: void test() { ... }`
could never match a function definition. Tightened the forward-decl
ambiguity heuristic so only truly-ambiguous shapes (no return type, no
params, empty body, no attrs) collapse to a Call.
- Adds three Dart-side metavariable capabilities that other major
languages already had:
1. **Typed metavariable `$X as T`** — in pattern mode, the existing Dart
`as` cast is reinterpreted as `TypedMetavar` when the LHS is a
metavariable (TS/Java/Swift use the same trick).
2. **`metavariable-type: T` filter** — Dart registered in
`Parse_metavariable_type` (`x as %s` wrap, `Cast` unwrap).
3. **Metavariables inside string interpolations** — `print("hello $X")`
now binds `$X` to the interpolation expression on Dart targets, matching
Semgrep's convention everywhere else. Real Dart `$name` interpolations
(lowercase) keep the literal-text behavior so existing patterns still
match.

## Validation

Generated the artifact via the standard flow (`make && make install &&
cd lang && ./test-lang dart && ./release dart --dry-run`) on
`ocaml-tree-sitter-semgrep` with the bumped submodule, then tested the
regenerated `parse-dart`:

| Corpus | Old grammar (`80e23c07`) | New grammar (`02e9bb19`) |
|---|---|---|
| 199 random `.dart` files from
[`flutter/flutter@master`](https://github.com/flutter/flutter) (1-30 KB)
| 196 ok / 3 fail | **199 ok / 0 fail** |
| 5,285 latest-version packages on pub.dev | ~102 `PartialParsing` |
**56 `PartialParsing` (1.1%)** |

The 3 Flutter files that previously failed all live inside developer
tooling and exercise Dart 3 patterns:

-
[`dev/bots/check_tests_cross_imports.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/check_tests_cross_imports.dart)
— `switch (this) { _MaterialLibrary() => ..., _CupertinoLibrary() =>
..., _OtherLibrary() => ... }`
-
[`dev/bots/custom_rules/no_stop_watches.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/custom_rules/no_stop_watches.dart)
— `Element() || null`, nested `InterfaceType(element: ...)`
-
[`dev/bots/custom_rules/render_box_intrinsics.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/custom_rules/render_box_intrinsics.dart)
— `switch (node.parent) { ... }`

## New tests (8)

Added regression tests covering every distinct fix and capability in
this branch:

| Test | Covers |
|---|---|
| `tests/parsing/dart/switch_object_pattern.dart` | Empty-paren /
field-bearing / generic / guarded `object_pattern` arms in switch (Dart
3 `#93`) |
| `tests/parsing/dart/dart3_grammar_features.dart` | Empty record `()`,
single-named record `(name: v,)`, dot-shorthand `.foo`/`.new`,
null-aware map element `?key: value`, null-assertion `!` in cascade,
unnamed library directive, dotted/operator symbol literals (`#a.b.c`,
`#==`), `get`/`set` as identifiers |
| `tests/patterns/dart/uninit_local_var.{dart,sgrep}` | `String $X;`
matches uninitialized locals (parser fix #1) |
| `tests/patterns/dart/function_def_void.{dart,sgrep}` | `void test() {
... }` matches function definitions (parser fix #3) |
| `tests/rules/if_pattern_dart.{yaml,dart}` | `if (kDebugMode) { ... }`
matches real If statements (parser fix #2 — `if` keyword routing) |
| `tests/rules/await_pattern_dart.{yaml,dart}` | `await fetch($URL)`
matches only awaited calls (parser fix #2 — `await` keyword routing) |
| `tests/rules/typed_metavar_dart.{yaml,dart}` | `print($X as String)`
binds against String args/decls only (typed metavariable) |
| `tests/rules/metavar_type_dart.{yaml,dart}` | `metavariable-type:
String` filter (typed metavariable) |
| `tests/rules/string_interp_metavar_dart.{yaml,dart}` | `print("hello
$X")` binds the interpolation expression |

## Commit groups

This branch is 21 commits — grouped by intent:

1. **Submodule bump (commit 1)** — `chore(dart): bump tree-sitter-dart
and adapt OCaml mapper for Dart 3` style commit at the base.
2. **CST-shape adaptations (commits 2–10)** — one commit per upstream
grammar change, each adjusting `Parse_dart_tree_sitter.ml` to consume
the new CST shape: abstract/external var, label/operator identifiers,
empty-statement, external instance fields, for-loop/field-initializer
changes, null-aware map key, null-assertion in cascades, ffi external
get/set fields, full symbol literals, single-named record, record-return
method.
3. **Parser bug fixes (commits 11–14)** — independent of the bump,
surfaced during pattern validation: uninitialized-local-decl drop,
statement-keyword pattern routing, `await`/`rethrow` keyword routing,
function-definition pattern degradation.
4. **Pattern-mode capabilities (commits 17–19)** — typed metavariable
`$X as T` + `metavariable-type` registration, metavariables inside
string interpolations, function-def pattern matching.
5. **Cleanup (commit 15)** — drop the stale segfault gate in
`Parse_target.ml` and the `Lang.Dart` exclusion in `Rule_fetching.ml`.
6. **Validation pointer (commit 16)** — _temporary_ redirect of
`.gitmodules` so this branch resolves the validation artifact from
`AlexLaroche/semgrep-dart`. **Drop before merge.**
7. **Style (commit 20)** — `style(dart): apply ocamlformat 0.29.0` to
satisfy CI's `lint-ocaml` hook (CI uses 0.29.0; recent edits were made
with a local 0.27.0 whose `break-cases` differs).
8. **Tests (commits 21–22)** — 8 new tests covering every fix and
capability above (table in the previous section).

## Before marking ready for review

- [x] semgrep/ocaml-tree-sitter-semgrep#590
lands (bumps the upstream tree-sitter-dart submodule there to
`02e9bb19`).
- [x] An r2c admin runs `./release dart` on `ocaml-tree-sitter-semgrep`
to publish a real release commit on `semgrep/semgrep-dart`.
- [x] Drop the `chore(dart): point semgrep-dart submodule URL at fork
for validation` commit.
- [x] Re-point the submodule pointer from `c5a5e6c1` (on
`AlexLaroche/semgrep-dart`) to whatever real release SHA appears on
`semgrep/semgrep-dart`.
- [ ] CI green on this repo.

## Test plan

- [x] `make core` clean against the bumped submodule.
- [x] Dart lang-parsing tests pass; Dart sgrep-pattern tests pass.
- [x] Standalone `parse-dart`: 199/199 files in `flutter/flutter@master`
parse cleanly (was 196/199).
- [x] pub.dev: 5,229/5,285 fully clean (was ~5,183), 56 `PartialParsing`
(was ~102).
- [x] Original "segfault" repro (`: string (* filename *)`) — exits
cleanly, no signal, on both `parse-dart` and `semgrep-core
-dump_pattern`.
- [x] Pre-commit `lint-ocaml` hook passes locally with ocamlformat
0.29.0.
- [ ] CI on this repo (semgrep/semgrep).

Synced from OSS PR #11678
Author: @AlexLaroche
Imported by: @yosefAlsuhaibani

Closes #11678
OSS-sync-author: Alexandre Laroche <larochealexandre@hotmail.com>

Test plan:

synced from Pro df36084f8b8ee40724eab0322e710c3369efe6f5
@yosefAlsuhaibani

Copy link
Copy Markdown
Collaborator

merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sgrep inside bento.dev integrate tree-sitter java parser into sgrep fix python syntax bugs in sgrep

3 participants