Skip to content

allow additional warnings for none unique experiments [AI]#1658

Open
rfbgo wants to merge 2 commits into
GoogleCloudPlatform:developfrom
rfbgo:unique_exps
Open

allow additional warnings for none unique experiments [AI]#1658
rfbgo wants to merge 2 commits into
GoogleCloudPlatform:developfrom
rfbgo:unique_exps

Conversation

@rfbgo

@rfbgo rfbgo commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR adds some explicit warnings that try call out what happens if you have colliding yaml defintions:

1. Matrix/Vector Expansion Collision

This happens when you expand an experiment using vectors/matrices (e.g. var_b and var_c ), but your name template does not use them, resulting in duplicate experiment namespaces.

              experiments:
                collision_matrix:
                  variables:
                    var_b: ['a', 'b']
                    var_c: ['x', 'y']
                  zips:
                    my_zip:
                      - var_b
                      - var_c
                  matrix:
                    - my_zip

yeilds:

    ==> Warning: Two experiments are defined with the name basic.test_wl.collision_matrix
    ==> Warning:   -> Collision Origin: Within the same YAML experiment block: 'collision_matrix' (Matrix/Vector Expansion)
    ==> Warning: Variable differences between experiment definitions:
    ==> Warning:   - var_b: {'previous': 'a', 'new': 'b'}
    ==> Warning:   - var_c: {'previous': 'x', 'new': 'y'}
    ==> Warning:
    ==> Warning: ==> SUGGESTION: Your experiment name template does not distinguish between parallel expansions.
    ==> Warning:     To make the namespaces unique, add one or more of these differing matrix variables into your 'collision_matrix' template:
    ==> Warning:     -> {var_b}
    ==> Warning:     -> {var_c}
    ==> Warning:
    ==> Error: Experiment basic.test_wl.collision_matrix is not unique.
    ──────

2. Collision Across Different YAML Experiment Blocks

This happens when you have two distinct experiment blocks in your YAML (e.g. collision_{my_var1} and collision_{my_var2} ) that end up evaluating to the exact same final experiment name.

            test_wl:
              experiments:
                collision_{my_var1}:
    		  variables:
                    my_var1: 'foo'
                    my_var2: 'bar'
                collision_{my_var2}:
    		  variables:
                    my_var1: 'baz'
                    my_var2: 'foo'
                    unique_to_right: 'yes'

yeilds

    ==> Warning: Two experiments are defined with the name basic.test_wl.collision_foo
    ==> Warning:   -> Collision Origin: Across different YAML experiment blocks: 'collision_{my_var1}' vs 'collision_{my_var2}'
    ==> Warning: Variables unique to newly defined experiment:
    ==> Warning:   - unique_to_right
    ==> Warning: Variable differences between experiment definitions:
    ==> Warning:   - my_var1: {'previous': 'foo', 'new': 'baz'}
    ==> Warning:   - my_var2: {'previous': 'bar', 'new': 'foo'}
    ==> Warning:
    ==> Error: Experiment basic.test_wl.collision_foo is not unique.
    ──────

3. Static Definition Collision

This happens when you have two static (un-expanded) experiment definitions in your YAML using the exact same literal name.

    # Inherited or defined via a template workspace:
    # experiments:
    #   static_collision:
    #     variables:
    #       var: 'original'

    # In your local ramble.yaml:
    ramble:
      variables:
        mpi_command: ''
        batch_submit: '{execute_experiment}'

      applications:
        basic:
          workloads:
            test_wl:
              experiments:
                static_collision:
                  variables:
                    var: 'new'

yields

    ==> Warning: Two experiments are defined with the name basic.test_wl.static_collision
    ==> Warning:   -> Collision Origin: Static definition (No Matrix/Vector expansion active)
    ==> Warning:
    ==> Warning: ==> SUGGESTION: Multiple distinct experiment blocks in your YAML are using the same literal name.
    ==> Warning:     Rename one of the 'static_collision' experiment blocks in your YAML file.
    ==> Warning:
    ==> Error: Experiment basic.test_wl.static_collision is not unique.

@rfbgo rfbgo marked this pull request as draft July 15, 2026 15:14
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances experiment namespace collision diagnostics in Ramble by tracking independent variables to pinpoint the origin of collisions and provide actionable suggestions for resolving them. Feedback on these changes includes replacing direct dictionary access of self.experiments with the safer self.get_experiment helper to prevent potential KeyErrors, and removing an unnecessary check for render_group in locals() since the variable is guaranteed to be defined in the local scope.

Comment thread lib/ramble/ramble/experiment_set.py Outdated
Comment thread lib/ramble/ramble/experiment_set.py Outdated
@ramble-pr-bot

ramble-pr-bot Bot commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Ramble Performance Test Metrics

Results produced with commit: 3abde89

Test Name Outcome Duration (s) Most Recent Run (s) Last 5 Avg (s)
test_analyze_large_file passed 2.9716 3.0625 (48760fd) 2.9483
test_large_template_expansion passed 2.1592 2.1499 (48760fd) 2.1608
test_many_experiments passed 37.5349 37.8478 (48760fd) 37.3594
test_many_objects_defaults passed 20.3720 20.1824 (48760fd) 19.9687
test_matrix_filter_perf passed 1.6294 1.6450 (48760fd) 1.6015

@codecov

codecov Bot commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.33%. Comparing base (3eb8942) to head (3abde89).
⚠️ Report is 6 commits behind head on develop.

Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #1658      +/-   ##
===========================================
+ Coverage    93.30%   93.33%   +0.02%     
===========================================
  Files          352      352              
  Lines        34223    34329     +106     
===========================================
+ Hits         31931    32040     +109     
+ Misses        2292     2289       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rfbgo rfbgo changed the title allow additional warnings for none unique experiments allow additional warnings for none unique experiments [AI] Jul 16, 2026
@rfbgo rfbgo marked this pull request as ready for review July 16, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rfbgo