feat(dart): Dart 3 grammar bump, parser fixes, and pattern-mode capabilities#11678
feat(dart): Dart 3 grammar bump, parser fixes, and pattern-mode capabilities#11678AlexLaroche wants to merge 27 commits into
Conversation
Bumps the semgrep-dart submodule to 452f0be (regenerated from upstream
tree-sitter-dart 0fc19c3a, 11 commits ahead of the 2023-05 pin), which
fixes parsing of Dart 3 idioms now common in Flutter:
- empty-paren object patterns in switch arms ('_MyClass()' arms),
- Dart 3.10 dot-shorthand expressions,
- extension type declarations,
- tree-sitter 0.25.x ABI compat,
- 'set'/'get' contextual keywords usable as identifiers.
Adapts Parse_dart_tree_sitter.ml to the new CST shape:
- declared_identifier/simple_formal_parameter: 4th component is now
'anon_choice_id_2354d68' (Id | Get | Set).
- map_element: '`Exp' replaced by '`Opt_QMARK_exp' (null-aware
element); the leading "?" is preserved as
'OtherExpr ("NullAwareElement", _)' so rules can distinguish it
from a plain expression.
- '`Pair' gained an optional '?' token before the value.
- map_if_null_expression_: takes 'if_null_expression' (the list)
directly; outer constructor renamed to '`If_null_exp_'.
- map_object_pattern: middle field is now optional, fixing the
empty-paren case.
- map_primary: new '`Get'/'`Set'/'`Dot_shor' arms (dot shorthand
mapped to OtherExpr "DotShorthand"; set/get treated as identifiers).
- map_top_level_definition: stub for '`Exte_type_decl' (extension
type) emitting OS_Todo carrying the declared identifier so rules
can still reference the type name.
- map_declaration_: handle '`Exte_and_static_type_id' as a typed
VarDef.
Adds tests/parsing/dart/switch_object_pattern.dart guarding the
headline fix (empty-paren object patterns + field-bearing patterns +
generic + guarded patterns + null pattern).
Validation: 199/199 .dart files from flutter/flutter@master parse
cleanly with the regenerated parse-dart (was 196/199); all 15 Dart
parsing tests pass.
…tors
Several improvements to Parse_dart_tree_sitter.ml independent of the
grammar bump:
- map_method_signature: pass the actual body to getter/setter mappings
(was FBNothing) and label methods as Method instead of Function.
Methods were silently losing their body in the AST, breaking taint
analysis and any rule that matches inside a method.
- map_string_literal: skip the synthetic ConcatString wrapper for
single-element string literals. Wrapping inserted a Call node that
interfered with patterns matching plain string literals.
- map_template_substitution: emit Middle3 (the expression slot) for
'${expr}' interpolations instead of Right3, and apply the same
Pattern/Program env distinction used by map_sub_string_test for
bare '$id' interpolations. This brings interpolation handling in
line with the rest of the string parser.
- map_selector: use Dart's universal naming convention as a heuristic
to disambiguate Call from New for '$X(...)'. Identifiers starting
with an uppercase letter are types, so 'Foo()' becomes a New
expression; lowercase stays a Call. Improves rule precision on
taint sources/sinks that target constructors.
- File_type.ml/.mli: add Dart to pl_type and recognize '.dart' so
non-parser code paths that switch on pl_type classify Dart files
correctly.
- Unit_engine.ml: enable Dart tainting tests by registering the
'tests/tainting_rules/dart' directory.
Tests:
- tests/patterns/dart/function_call.{dart,sgrep},
function_def.{dart,sgrep}: new pattern tests covering Call/New
disambiguation and method-body presence.
- tests/tainting_rules/dart/arrays_if.{dart,yaml},
try_return.{dart,yaml}: tainting fixtures exercising arrays and
try/return paths.
- 5 existing parsing snapshots refreshed (Method label, New for
uppercase-starting calls, simplified single-element strings).
Validation: 199/199 .dart files from flutter/flutter@master parse
cleanly; 19 Dart tests pass (was 14).
The 'tree-sitter-dart is currently buggy and can generate some
segfaults' note (added 2023-11) referenced an input that no longer
crashes the bundled grammar. The original repro (': string (* filename
*)') now exits cleanly through both the standalone parse-dart and the
semgrep-core dump_pattern path.
- src/parsing/Parse_target.ml: remove the stale comment above the
Lang.Dart arm.
- src/osemgrep/networking/Rule_fetching.ml: drop the
'List_.exclude (... =*= Lang.Dart)' so '-e <pat>' (no '-l') tries
Dart along with every other language.
Temporary: redirects the submodule URL from returntocorp/semgrep-dart to AlexLaroche/semgrep-dart so a fresh clone of this branch resolves the regenerated artifact at 452f0be (which only exists on the fork until r2c runs './release dart' to publish a real release commit). Drop this commit before opening the upstream PR — the public PR should use the original returntocorp/semgrep-dart URL with whatever SHA r2c produces from the regenerate.
b2808d9 to
0e9f348
Compare
The tree-sitter-dart parser-improvements branch changes the shape of three CST nodes: - record_literal_no_const becomes a polymorphic variant with an empty `LPAR_RPAR` case for the unit record `()`. - const_object_expression becomes a polymorphic variant with a new `Const_buil_dot_shor_args` case for `const .new(...)` / `const .named(...)`. - initialized_identifier accepts an aliased `get`/`set` token in addition to a plain identifier. - constant_pattern gains `Dot_shor_opt_opt_type_args_args` for `case .red:`, `case .new():`, `case .fromRGB(1, 2, 3):`. Also extracts the previously inlined dot-shorthand handling from _primary into a reusable map_dot_shorthand helper. Bumps the semgrep-dart submodule to the corresponding generated CST.
…apes Tracks the latest tree-sitter-dart parser-improvements commits: - Adds a `\`Abst_choice_final_buil_opt_type_id_list` arm to map_declaration_, handling the three abstract instance field forms (`abstract <type> ids;`, `abstract final <type>? ids;`, `abstract covariant <type> ids;`) by emitting a VarDef with an unhandled `abstract` keyword attribute. - Adds a `\`Opt_meta_exte_buil_choice_final_buil_opt_type_id_list_semi` arm to the top-level definition mapper, handling top-level external variable declarations (`external <type> id;`, `external final <type>? id;`, `external late <type>? id;`). Also picks up small CST tag renames (`Show_id_list_`/`Hide_id_list_`, `Cova_choice_late_buil_choice_final_buil_opt_type_id_list`, `Late_buil_choice_final_buil_opt_type_id_list`, `Final_buil_opt_type_id_list`) introduced when the new `identifier_list` use-site changed the simplification output. Bumps the semgrep-dart submodule to the corresponding generated CST.
After the latest tree-sitter-dart parser-improvements commit, the `label` rule accepts `_get`/`_set`/`_function_builtin_identifier` as identifier aliases, and `initialized_identifier` accepts `_operator` as an identifier alias. The corresponding CST shape gained `Func_buil_id` and `Op` tags. - map_label now matches `\`Id` / `\`Get` / `\`Set` / `\`Func_buil_id`. - map_initialized_identifier now matches `\`Id` / `\`Get` / `\`Set` / `\`Op`. Bumps the semgrep-dart submodule to the corresponding generated CST.
Adds an `\`Empty_stmt` arm to `map_statement` that emits an empty `Block` for `;` (Dart's null/empty statement), matching the new `empty_statement` rule in the grammar. Bumps the semgrep-dart submodule to the corresponding generated CST.
Adds an `\`Exte_choice_final_buil_opt_type_id_list` arm to map_declaration_, handling `external final <type>? <ids>` and `external covariant <var-or-type> <ids>` by emitting a VarDef with the appropriate keyword attributes. Bumps the semgrep-dart submodule to the corresponding generated CST.
Tracks the latest tree-sitter-dart parser-improvements commits: - The for-loop update list now allows a trailing comma; CST gained a third tuple component carrying the optional comma. Renames the helper `map_anon_arg_rep_COMMA_arg_eb223b2` → `map_for_init_list` and adds `map_for_update_list` for the new shape, and updates the `Opt_choice_local_var_decl_opt_exp_semi_opt_exp_rep_COMMA_exp_opt_COMMA` variant tag. - The field initializer's right-hand side is now a full `_expression` (not `_real_expression` + cascades). Updates the `\`Field_init` arm to take 4 components and call `map_expression`. Bumps the semgrep-dart submodule to the corresponding generated CST.
The `pair` rule now accepts an optional `?` before the key as well as before the value (Dart 3 null-aware elements), so the CST tuple has five components instead of four. Updates the `\`Pair` arm of the element mapper accordingly. Bumps the semgrep-dart submodule to the corresponding generated CST.
… fields Bumps semgrep-dart submodule to pick up two upstream tree-sitter-dart grammar fixes: - Cascade subsections now accept the postfix `!` null-assertion interleaved with argument parts, so code like `obj.._parent!.onChildrenChanged(this)` (common in flame, flutter widget code) parses cleanly. - Static external fields can be named with built-in identifiers (`get`, `set`, `operator`), used by `dart:ffi` Struct field conventions like `external SomeType get;`. Updates the OCaml mapper for the two new CST shapes: - New `anon_choice_arg_part_7fd04ec` choice between argument parts and the `!` token in cascade contexts (factored into a helper). - Existing `Exte_and_static_type_id` arm renamed to `Exte_and_static_type_choice_id` with a 4-arm choice for the field name.
…amed record Bumps semgrep-dart submodule for three upstream tree-sitter-dart grammar fixes: - `library;` (unnamed library directive, Dart 2.19+) — common in files that exist solely to carry a doc comment for the library. - Full symbol literal forms — `#a.b.c` (dotted) and operator symbols (`#+`, `#==`, `#[]`, `#[]=`, `#~`, etc.) per the Dart spec. - Single-named-field record literal with trailing comma (`(name: value,)`), required by Dart 3 to disambiguate from a labeled parenthesized expression. Updates the OCaml mapper for the new CST shapes: - `map_library_name` makes the dotted_identifier_list optional. - Symbol-literal arms now dispatch on the choice (dotted ids / equality / relational / shift / additive / multiplicative / single operator tokens) in both expression and pattern positions. - New `Label_exp_COMMA` arm in record_literal_no_const for the single-named-field-with-trailing-comma form.
bb92e5f to
45984ee
Compare
The tree-sitter-dart grammar was extended to parse methods whose return
type is a record type when preceded by an annotation (e.g.
`@override (Model, Cmd?) update(Msg msg) { ... }` — Elm-style
architecture pattern). The new grammar introduces hidden rules that, on
the OCaml side, surface as new CST types:
- bare_annotation (just `@name`, no args)
- record_return_function_signature (record-type return + name + params)
- record_return_method_signature (optional `static` + above)
- record_return_class_member (annotations + method + body)
Together with a new `Rep1_bare_anno_record_ret_func_sign_func_body`
top-level variant and a `Record_ret_class_member` variant in
class/extension/enum bodies.
Mapper additions:
- `map_record_return_function_signature` — produces the same
`(fkind, fbody) -> stmt` shape as `map_function_signature`. Uses
`:>` to coerce the inner record-type choice (which excludes the
empty `()` form intentionally) to `CST.record_type`, so the
existing `map_record_type` is reused without duplication.
- `map_record_return_method_signature` — wraps with the optional
`static` keyword attribute, mirroring the `Opt_static_choice_func_sign`
arm of `map_method_signature`.
- `map_bare_annotation` — produces a `NamedAttr` with empty args,
mirroring `map_annotation` minus the optional argument list.
- `map_record_return_class_member` — orchestrates annotations +
method signature + body into a `G.field`.
Mapper updates:
- `map_extension_body` — added `Record_ret_class_member` arm.
- `map_top_level_definition` — added
`Rep1_bare_anno_record_ret_func_sign_func_body` arm.
- Renamed `map_anon_rep_opt_meta_class_member_defi_cd2fbdb` to
`map_anon_choice_opt_meta_class_member_defi_44d3600` and rewrote it
to handle the new variant-based CST type (single element instead of
list); updated the two callers in `map_class_body` and
`map_enum_body` to `List.map` over it.
Verified end-to-end: `make core` clean; dart test suite identical to
master baseline (41 pass, 1 pre-existing failure unrelated to the
mapper); previously-failing pub.dev packages (equations, dascade,
medea_flutter_webrtc, password_engine) parse at 99.9% through the full
CLI -> semgrep-core -> mapper -> tree-sitter pipeline.
…word
Dart pattern parser tried the source-file parse first and only fell
back to the expression/statement parse when the file parse produced
errors. That breaks patterns starting with `if`, `for`, `while`,
`do`, `switch`, `try`, `return`, `throw`, `break`, `continue`,
`assert`, or `yield`: Dart's grammar doesn't allow statements at
top level, but `if (cond) { ... }` happens to parse silently as a
function definition (with `if` as the return type, `(cond)` as the
parameter list, `{ ... }` as the body) — no errors are raised, so
the fallback never fires.
Concretely, a pattern like
pattern-not-inside: |
if (kDebugMode) {
...
}
was producing a function-definition AST and never matching the
target's actual `If(...)` node.
Detect statement-keyword starts up-front and route them through the
`__SEMGREP_EXPRESSION` fallback path directly, bypassing the
misleading file parse. `for`/`while`/`try`/etc. happened to land on
the fallback path before because their file-parse produced errors,
but `if` slipped through; this makes the behavior uniform across all
statement starters.
map_initialized_variable_definition_unwrapped dropped the variable
when there was no `= expr` initializer and no comma-separated
follow-on declarations: the `None` arm returned `[]` and the entity
list ended up empty.
Concretely, inside a function body
void m() { String z; late String w; }
neither `z` nor `w` made it into the AST, so patterns like
`String $X;` or `late String $X;` could not match local
declarations. Top-level uninitialized declarations were unaffected
because they go through a different mapper path that uses
map_initialized_identifier_list (always non-empty).
Restructure the optional-init branch to produce a single
(id, init_opt) pair that's prepended to the list of follow-on
declarations, so the declared identifier is always emitted.
Pattern `await foo();` parsed at the file level as just `Call(foo, [])` — the leading `await` was silently stripped — so the pattern matched bare `foo()` calls (false positive) and never matched the `await foo()` it was meant to find. Same root cause as the earlier `if (kDebugMode)` case: the file parser silently produces the wrong shape, so the errors-based fallback never fires. Add `await` (and `rethrow` for symmetry) to the statement-keyword list so these patterns route through the expression/statement parse path directly. After this change, `await foo();` parses as ExprStmt(Await(Call(...))) and matches only actual awaited calls.
Pre-commit's `lint-ocaml` hook in CI runs ocamlformat 0.29.0 (pinned in `dev/required.opam` and `scripts/lint-ocaml`). Recent edits to this file used a local 0.27.0, whose break-cases preferences are slightly different — CI's hook reformatted the file and failed the lint job. Reformat with the matching 0.29.0 to clear the lint failure: pure indentation/wrapping changes, no behavior diff.
|
@yosefAlsuhaibani: Can you help me get this PR merged? The first step would be to regenerate Dart's grammar. See semgrep/ocaml-tree-sitter-semgrep#590 |
Adds two related capabilities:
1. In pattern mode, the existing Dart cast `$X as T` is reinterpreted
as a TypedMetavar — `$X` only binds against expressions whose
resolved type is `T`. Same trick TS/Java/Swift use, just keyed off
the metavariable convention (`is_metavar_name`).
2. Registers Dart in `Parse_metavariable_type` (`x as %s` wrap, `Cast`
unwrap), so `metavariable-type: T` rules now work for Dart.
Verified: the type infrastructure was already in place — `id_info.id_type`
gets populated by Naming_AST for parameters and use sites — only the
pattern-side syntax was missing. With this commit, typed-metavar
patterns and `metavariable-type` filters distinguish String vs int
arguments correctly on `void f(String s, int n) { print(s); print(n); }`.
Dart's `"hello $name"` interpolation collides with Semgrep's `$NAME`
metavariable convention, so the pattern parser previously kept all
interpolated `$id` fragments as raw text in pattern mode — meaning
`print("hello $X")` could not match `print("hello $name")`.
Special-case the uppercase/underscore form: when the interpolated
identifier matches `is_metavar_name`, emit it as a metavariable
expression (same shape as Program mode produces). Real Dart
interpolations like `$name` (lowercase) keep the raw-text behavior so
patterns like `"value=$count"` against `"value=$count"` still match
literally.
Verified: pattern `print("hello $X")` now matches `print("hello $name")`
and binds `$X` to `name`. Edge case of a Dart identifier whose name
happens to be all-uppercase (legal but very unusual) collides — same
trade-off every Semgrep language already accepts.
`map_function_signature` reinterpreted any pattern-mode signature with
empty parameters as a Call, justified by a forward-declaration
ambiguity (`foo();` looks both like a call and a forward decl). The
heuristic was too coarse: `void test() { ... }` was also rewritten as
`Call(test, [])`, losing the return type, body, and the function
context — so `pattern-inside: void test() { ... }` couldn't match any
function definition.
Tighten the condition: only reinterpret as Call when the construct is
truly ambiguous — no return type, no parameters, an empty body
(`FBDecl`/`FBNothing`/empty `FBStmt`), and no attributes. Anything
that looks like a real function definition (return type, parameters,
non-empty body, or attributes) is now mapped to a `DefStmt(FuncDef)`
as expected.
Verified: `void test() { ... }` matches function definitions in target
classes; the original `foo();` forward-decl pattern still matches Call
expressions (no regression).
…avar, function-def patterns
Adds four targeted regression tests for the recent Dart pattern-mode
fixes:
- `tests/rules/typed_metavar_dart.{yaml,dart}` — `print($X as String)`
matches String-typed args and binds, but skips `int`/`dynamic`. Also
covers locally-declared variables (`String name = ...`).
- `tests/rules/metavar_type_dart.{yaml,dart}` — `metavariable-type:
String` filter restricts a `print($X)` pattern to String-typed
expressions only.
- `tests/rules/string_interp_metavar_dart.{yaml,dart}` — pattern
`print("hello $X")` binds the interpolation expression and matches
multiple targets, while leaving non-matching prefix/literal strings
untouched.
- `tests/patterns/dart/function_def_void.{dart,sgrep}` — pattern
`void test() { ... }` matches both top-level and method-level
function definitions named `test`, ignoring same-shape definitions
with different names.
All four pass; broader Dart suite goes from 41/42 → 45/46 with the
same single pre-existing `biometric-weak-config` failure unchanged.
Adds tests for the parser-side fixes earlier in this branch and a
smoke parsing test covering the new Dart 3 grammar features pulled in
by the submodule bump:
- `tests/patterns/dart/uninit_local_var.{dart,sgrep}` — pattern
`String $X;` matches uninitialized local declarations. Locks in the
fix where `map_initialized_variable_definition_unwrapped` was
returning [] when the optional `= expr` was absent, silently
dropping `String z;` from the AST.
- `tests/rules/if_pattern_dart.{yaml,dart}` — pattern
`if (kDebugMode) { ... }` matches real If statements. Locks in the
statement-keyword pattern-routing fix that prevents `if`-prefixed
patterns from silently misparsing as function definitions at the
file level.
- `tests/rules/await_pattern_dart.{yaml,dart}` — pattern
`await fetch($URL)` matches only awaited calls, not bare ones.
Locks in the same routing fix for `await`/`rethrow`, where the
`await` keyword was previously stripped at the file-parse level
(the `Call(fetch, [...])` shape) and produced false positives.
- `tests/parsing/dart/dart3_grammar_features.dart` — exercises the
grammar features brought in by the tree-sitter-dart bump: empty
record `()`, single-named-field record `(name: v,)`, dot-shorthand
`.foo`/`.new`, null-aware map element `?key: value`, null-assertion
`!` inside a cascade chain, unnamed library directive `library;`,
dotted/operator symbol literals (`#a.b.c`, `#==`), and `get`/`set`
as identifiers.
All four pass; broader Dart suite goes from 45/46 → 49/50 with the
same single pre-existing `biometric-weak-config` failure unchanged.
|
@AlexLaroche can do! |
@yosefAlsuhaibani Please tell me when the command './release dart' has been executed. |
|
@AlexLaroche semgrep/semgrep-dart@0202e42. Let's
|
Switch the submodule URL from the AlexLaroche fork back to returntocorp/semgrep-dart, and bump the pointer to 0202e42 (yosef's just-merged PR semgrep#1, regenerated against UserNobody14/tree-sitter-dart 02e9bb19). Drop the `Labe_stmt` arm from the OCaml mapper: enabling `$.labeled_statement` was a local patch on top of the upstream grammar snapshot and isn't on UserNobody14/tree-sitter-dart yet, so it can't ride along with a canonical-upstream submodule pointer. Labeled statements weren't supported on develop either, so this is not a regression; it can be re-introduced in a follow-up once the grammar change lands upstream. Also add changelog entries for PR semgrep#11678.
| [submodule "OSS/languages/dart/tree-sitter/semgrep-dart"] | ||
| path = languages/dart/tree-sitter/semgrep-dart | ||
| url = https://github.com/returntocorp/semgrep-dart.git | ||
| url = https://github.com/returntocorp/semgrep-dart |
There was a problem hiding this comment.
Could we remove this part of the diff?
Match the style used elsewhere in `.gitmodules` (develop has `.git` on every returntocorp/* submodule URL). Functionally equivalent — GitHub resolves both forms — but stylistically consistent.
|
@yosefAlsuhaibani: |
yosefAlsuhaibani
left a comment
There was a problem hiding this comment.
@AlexLaroche Looks good; thanks for taking this on! I'll get to syncing this and merging it to develop.
|
@yosefAlsuhaibani has imported this pull request. Semgrep employees can find it at https://github.com/semgrep/semgrep-proprietary/pull/6332. @AlexLaroche, thank you for your contribution! Once the internal PR has been merged, this PR will be automatically marked as closed. Later, the resulting commit will be synced back out to this repository and will reference this PR. |
…ilities (semgrep/semgrep-proprietary#6332) > **Draft — gated on semgrep/ocaml-tree-sitter-semgrep#590 landing and an r2c admin running `./release dart` to publish a real release commit to `semgrep/semgrep-dart`. See "Before marking ready" below.** ## Summary - Bumps the `languages/dart/tree-sitter/semgrep-dart` submodule to a regenerated artifact built from upstream `UserNobody14/tree-sitter-dart@02e9bb19` (13 commits ahead of the May-2023 pin), and adapts `Parse_dart_tree_sitter.ml` to the new CST shape. - Resolves the headline Dart 3 parsing gap: empty-paren `object_pattern` (e.g. `_MyClass()` arms in `switch`), now common in Flutter (upstream PR [#93](UserNobody14/tree-sitter-dart#93)). - Resolves the `@annotation` + record-return-type method gap (e.g. `@override (R, T) method() {…}`, the Elm-style architecture pattern in Flutter) — this was the dominant remaining `PartialParsing` cause on the pub.dev catalog (upstream PR [#98](UserNobody14/tree-sitter-dart#98)). - Also picks up: 16 grammar fixes from the upstream parser-improvements PR ([#97](UserNobody14/tree-sitter-dart#97)) — empty record literal `()`, dot-shorthand patterns, abstract/external fields, null-aware map keys, dotted/operator symbol literals, single-named-field record literals, unnamed library directive, and more. - Drops the stale "tree-sitter-dart is currently buggy" segfault gate in `Parse_target.ml` and the `Lang.Dart` exclusion in `Rule_fetching.ml` — the original repro (`': string (* filename *)'`) now exits cleanly. - Fixes three parser-side bugs in `Parse_dart_tree_sitter.ml` discovered during pattern-validation work that reproduce on `develop` too: 1. **Uninitialized local variable declarations were silently dropped.** `String z;` / `late String w;` inside a function body never made it into the AST, so patterns like `String $X;` could not match local declarations. Fixed by emitting `(id, None)` instead of `[]` when the optional `= expr` is absent. 2. **Patterns starting with a Dart statement keyword silently misparsed.** `if (kDebugMode) { ... }` parsed as a function definition (with `if` as the return type), `await foo();` parsed as a bare `Call(foo, [])` with `await` stripped. Detect statement-keyword starts up-front and route through the `__SEMGREP_EXPRESSION` path, bypassing the misleading file-parse. 3. **Function-definition patterns silently degraded to calls.** `void test() { ... }` reinterpreted as `Call(test, [])` (losing return type, body, and function context), so `pattern-inside: void test() { ... }` could never match a function definition. Tightened the forward-decl ambiguity heuristic so only truly-ambiguous shapes (no return type, no params, empty body, no attrs) collapse to a Call. - Adds three Dart-side metavariable capabilities that other major languages already had: 1. **Typed metavariable `$X as T`** — in pattern mode, the existing Dart `as` cast is reinterpreted as `TypedMetavar` when the LHS is a metavariable (TS/Java/Swift use the same trick). 2. **`metavariable-type: T` filter** — Dart registered in `Parse_metavariable_type` (`x as %s` wrap, `Cast` unwrap). 3. **Metavariables inside string interpolations** — `print("hello $X")` now binds `$X` to the interpolation expression on Dart targets, matching Semgrep's convention everywhere else. Real Dart `$name` interpolations (lowercase) keep the literal-text behavior so existing patterns still match. ## Validation Generated the artifact via the standard flow (`make && make install && cd lang && ./test-lang dart && ./release dart --dry-run`) on `ocaml-tree-sitter-semgrep` with the bumped submodule, then tested the regenerated `parse-dart`: | Corpus | Old grammar (`80e23c07`) | New grammar (`02e9bb19`) | |---|---|---| | 199 random `.dart` files from [`flutter/flutter@master`](https://github.com/flutter/flutter) (1-30 KB) | 196 ok / 3 fail | **199 ok / 0 fail** | | 5,285 latest-version packages on pub.dev | ~102 `PartialParsing` | **56 `PartialParsing` (1.1%)** | The 3 Flutter files that previously failed all live inside developer tooling and exercise Dart 3 patterns: - [`dev/bots/check_tests_cross_imports.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/check_tests_cross_imports.dart) — `switch (this) { _MaterialLibrary() => ..., _CupertinoLibrary() => ..., _OtherLibrary() => ... }` - [`dev/bots/custom_rules/no_stop_watches.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/custom_rules/no_stop_watches.dart) — `Element() || null`, nested `InterfaceType(element: ...)` - [`dev/bots/custom_rules/render_box_intrinsics.dart`](https://github.com/flutter/flutter/blob/master/dev/bots/custom_rules/render_box_intrinsics.dart) — `switch (node.parent) { ... }` ## New tests (8) Added regression tests covering every distinct fix and capability in this branch: | Test | Covers | |---|---| | `tests/parsing/dart/switch_object_pattern.dart` | Empty-paren / field-bearing / generic / guarded `object_pattern` arms in switch (Dart 3 `#93`) | | `tests/parsing/dart/dart3_grammar_features.dart` | Empty record `()`, single-named record `(name: v,)`, dot-shorthand `.foo`/`.new`, null-aware map element `?key: value`, null-assertion `!` in cascade, unnamed library directive, dotted/operator symbol literals (`#a.b.c`, `#==`), `get`/`set` as identifiers | | `tests/patterns/dart/uninit_local_var.{dart,sgrep}` | `String $X;` matches uninitialized locals (parser fix #1) | | `tests/patterns/dart/function_def_void.{dart,sgrep}` | `void test() { ... }` matches function definitions (parser fix #3) | | `tests/rules/if_pattern_dart.{yaml,dart}` | `if (kDebugMode) { ... }` matches real If statements (parser fix #2 — `if` keyword routing) | | `tests/rules/await_pattern_dart.{yaml,dart}` | `await fetch($URL)` matches only awaited calls (parser fix #2 — `await` keyword routing) | | `tests/rules/typed_metavar_dart.{yaml,dart}` | `print($X as String)` binds against String args/decls only (typed metavariable) | | `tests/rules/metavar_type_dart.{yaml,dart}` | `metavariable-type: String` filter (typed metavariable) | | `tests/rules/string_interp_metavar_dart.{yaml,dart}` | `print("hello $X")` binds the interpolation expression | ## Commit groups This branch is 21 commits — grouped by intent: 1. **Submodule bump (commit 1)** — `chore(dart): bump tree-sitter-dart and adapt OCaml mapper for Dart 3` style commit at the base. 2. **CST-shape adaptations (commits 2–10)** — one commit per upstream grammar change, each adjusting `Parse_dart_tree_sitter.ml` to consume the new CST shape: abstract/external var, label/operator identifiers, empty-statement, external instance fields, for-loop/field-initializer changes, null-aware map key, null-assertion in cascades, ffi external get/set fields, full symbol literals, single-named record, record-return method. 3. **Parser bug fixes (commits 11–14)** — independent of the bump, surfaced during pattern validation: uninitialized-local-decl drop, statement-keyword pattern routing, `await`/`rethrow` keyword routing, function-definition pattern degradation. 4. **Pattern-mode capabilities (commits 17–19)** — typed metavariable `$X as T` + `metavariable-type` registration, metavariables inside string interpolations, function-def pattern matching. 5. **Cleanup (commit 15)** — drop the stale segfault gate in `Parse_target.ml` and the `Lang.Dart` exclusion in `Rule_fetching.ml`. 6. **Validation pointer (commit 16)** — _temporary_ redirect of `.gitmodules` so this branch resolves the validation artifact from `AlexLaroche/semgrep-dart`. **Drop before merge.** 7. **Style (commit 20)** — `style(dart): apply ocamlformat 0.29.0` to satisfy CI's `lint-ocaml` hook (CI uses 0.29.0; recent edits were made with a local 0.27.0 whose `break-cases` differs). 8. **Tests (commits 21–22)** — 8 new tests covering every fix and capability above (table in the previous section). ## Before marking ready for review - [x] semgrep/ocaml-tree-sitter-semgrep#590 lands (bumps the upstream tree-sitter-dart submodule there to `02e9bb19`). - [x] An r2c admin runs `./release dart` on `ocaml-tree-sitter-semgrep` to publish a real release commit on `semgrep/semgrep-dart`. - [x] Drop the `chore(dart): point semgrep-dart submodule URL at fork for validation` commit. - [x] Re-point the submodule pointer from `c5a5e6c1` (on `AlexLaroche/semgrep-dart`) to whatever real release SHA appears on `semgrep/semgrep-dart`. - [ ] CI green on this repo. ## Test plan - [x] `make core` clean against the bumped submodule. - [x] Dart lang-parsing tests pass; Dart sgrep-pattern tests pass. - [x] Standalone `parse-dart`: 199/199 files in `flutter/flutter@master` parse cleanly (was 196/199). - [x] pub.dev: 5,229/5,285 fully clean (was ~5,183), 56 `PartialParsing` (was ~102). - [x] Original "segfault" repro (`: string (* filename *)`) — exits cleanly, no signal, on both `parse-dart` and `semgrep-core -dump_pattern`. - [x] Pre-commit `lint-ocaml` hook passes locally with ocamlformat 0.29.0. - [ ] CI on this repo (semgrep/semgrep). Synced from OSS PR #11678 Author: @AlexLaroche Imported by: @yosefAlsuhaibani Closes #11678 OSS-sync-author: Alexandre Laroche <larochealexandre@hotmail.com> Test plan: synced from Pro df36084f8b8ee40724eab0322e710c3369efe6f5
|
merged! |
Summary
languages/dart/tree-sitter/semgrep-dartsubmodule to a regenerated artifact built from upstreamUserNobody14/tree-sitter-dart@02e9bb19(13 commits ahead of the May-2023 pin), and adaptsParse_dart_tree_sitter.mlto the new CST shape.object_pattern(e.g._MyClass()arms inswitch), now common in Flutter (upstream PR #93).@annotation+ record-return-type method gap (e.g.@override (R, T) method() {…}, the Elm-style architecture pattern in Flutter) — this was the dominant remainingPartialParsingcause on the pub.dev catalog (upstream PR #98).(), dot-shorthand patterns, abstract/external fields, null-aware map keys, dotted/operator symbol literals, single-named-field record literals, unnamed library directive, and more.Parse_target.mland theLang.Dartexclusion inRule_fetching.ml— the original repro (': string (* filename *)') now exits cleanly.Parse_dart_tree_sitter.mldiscovered during pattern-validation work that reproduce ondeveloptoo:String z;/late String w;inside a function body never made it into the AST, so patterns likeString $X;could not match local declarations. Fixed by emitting(id, None)instead of[]when the optional= expris absent.if (kDebugMode) { ... }parsed as a function definition (withifas the return type),await foo();parsed as a bareCall(foo, [])withawaitstripped. Detect statement-keyword starts up-front and route through the__SEMGREP_EXPRESSIONpath, bypassing the misleading file-parse.void test() { ... }reinterpreted asCall(test, [])(losing return type, body, and function context), sopattern-inside: void test() { ... }could never match a function definition. Tightened the forward-decl ambiguity heuristic so only truly-ambiguous shapes (no return type, no params, empty body, no attrs) collapse to a Call.$X as T— in pattern mode, the existing Dartascast is reinterpreted asTypedMetavarwhen the LHS is a metavariable (TS/Java/Swift use the same trick).metavariable-type: Tfilter — Dart registered inParse_metavariable_type(x as %swrap,Castunwrap).print("hello $X")now binds$Xto the interpolation expression on Dart targets, matching Semgrep's convention everywhere else. Real Dart$nameinterpolations (lowercase) keep the literal-text behavior so existing patterns still match.Validation
Generated the artifact via the standard flow (
make && make install && cd lang && ./test-lang dart && ./release dart --dry-run) onocaml-tree-sitter-semgrepwith the bumped submodule, then tested the regeneratedparse-dart:80e23c07)02e9bb19).dartfiles fromflutter/flutter@master(1-30 KB)PartialParsingPartialParsing(1.1%)The 3 Flutter files that previously failed all live inside developer tooling and exercise Dart 3 patterns:
dev/bots/check_tests_cross_imports.dart—switch (this) { _MaterialLibrary() => ..., _CupertinoLibrary() => ..., _OtherLibrary() => ... }dev/bots/custom_rules/no_stop_watches.dart—Element() || null, nestedInterfaceType(element: ...)dev/bots/custom_rules/render_box_intrinsics.dart—switch (node.parent) { ... }New tests (8)
Added regression tests covering every distinct fix and capability in this branch:
tests/parsing/dart/switch_object_pattern.dartobject_patternarms in switch (Dart 3#93)tests/parsing/dart/dart3_grammar_features.dart(), single-named record(name: v,), dot-shorthand.foo/.new, null-aware map element?key: value, null-assertion!in cascade, unnamed library directive, dotted/operator symbol literals (#a.b.c,#==),get/setas identifierstests/patterns/dart/uninit_local_var.{dart,sgrep}String $X;matches uninitialized locals (parser fix #1)tests/patterns/dart/function_def_void.{dart,sgrep}void test() { ... }matches function definitions (parser fix #3)tests/rules/if_pattern_dart.{yaml,dart}if (kDebugMode) { ... }matches real If statements (parser fix #2 —ifkeyword routing)tests/rules/await_pattern_dart.{yaml,dart}await fetch($URL)matches only awaited calls (parser fix #2 —awaitkeyword routing)tests/rules/typed_metavar_dart.{yaml,dart}print($X as String)binds against String args/decls only (typed metavariable)tests/rules/metavar_type_dart.{yaml,dart}metavariable-type: Stringfilter (typed metavariable)tests/rules/string_interp_metavar_dart.{yaml,dart}print("hello $X")binds the interpolation expressionCommit groups
This branch is 21 commits — grouped by intent:
chore(dart): bump tree-sitter-dart and adapt OCaml mapper for Dart 3style commit at the base.Parse_dart_tree_sitter.mlto consume the new CST shape: abstract/external var, label/operator identifiers, empty-statement, external instance fields, for-loop/field-initializer changes, null-aware map key, null-assertion in cascades, ffi external get/set fields, full symbol literals, single-named record, record-return method.await/rethrowkeyword routing, function-definition pattern degradation.$X as T+metavariable-typeregistration, metavariables inside string interpolations, function-def pattern matching.Parse_target.mland theLang.Dartexclusion inRule_fetching.ml..gitmodulesso this branch resolves the validation artifact fromAlexLaroche/semgrep-dart. Drop before merge.style(dart): apply ocamlformat 0.29.0to satisfy CI'slint-ocamlhook (CI uses 0.29.0; recent edits were made with a local 0.27.0 whosebreak-casesdiffers).Before marking ready for review
02e9bb19)../release dartonocaml-tree-sitter-semgrepto publish a real release commit onsemgrep/semgrep-dart.chore(dart): point semgrep-dart submodule URL at fork for validationcommit.c5a5e6c1(onAlexLaroche/semgrep-dart) to whatever real release SHA appears onsemgrep/semgrep-dart.Test plan
make coreclean against the bumped submodule.parse-dart: 199/199 files influtter/flutter@masterparse cleanly (was 196/199).PartialParsing(was ~102).: string (* filename *)) — exits cleanly, no signal, on bothparse-dartandsemgrep-core -dump_pattern.lint-ocamlhook passes locally with ocamlformat 0.29.0.