Skip to content

fix(ci): make publish-chart resilient to a stale base commit#1342

Merged
mitasovr merged 1 commit into
constructorfabric:mainfrom
mitasovr:fix/publish-chart-stale-base-race
Jun 15, 2026
Merged

fix(ci): make publish-chart resilient to a stale base commit#1342
mitasovr merged 1 commit into
constructorfabric:mainfrom
mitasovr:fix/publish-chart-stale-base-race

Conversation

@mitasovr

Copy link
Copy Markdown
Contributor

Problem

The publish-chart job in build-images.yml failed on main (run 27535189783) at the Commit version bumps back to main step:

! [rejected]        HEAD -> main (non-fast-forward)
push failed; rebasing onto refreshed origin/main and retrying
   03af566..28de74f  main       -> origin/main
CONFLICT (content): Merge conflict in charts/insight/values.yaml
rebase failed (conflict on bumped files?); aborting
Process completed with exit code 1.

Re-running the job reproduced the failure identically — confirming this is not a transient race but a structural defect.

Root cause

publish-chart both patch-bumps the umbrella version and rewrites ingestion.toolboxImage in charts/insight/values.yaml, reading from / writing to the tree checked out at the run's trigger SHA.

Two PRs that both touch src/ingestion/** were merged back-to-back:

ae60ce3  feat: Figma connector   (#1306)   merge #1
03af566  feat: Workday connector (#1305)   merge #2  <- trigger SHA of the failing run

Each merge triggers an independent run, and because both touch ingestion, both rebuild the toolbox. The second run's trigger SHA (03af566) does not contain the first run's chore(release) commit, so both runs:

  • compute the same next umbrella version (0.1.67), and
  • rewrite the same ingestion.toolboxImage line to their own build tag.

The first run (28de74f) pushes its release commit; the second run's commit-back is then rejected (non-fast-forward). The retry's git pull --rebase conflicts in charts/insight/values.yaml — specifically on the ingestion.toolboxImage line (the version line auto-merges, both sides wrote 0.1.67) — and aborts → exit 1. A plain job re-run repeats forever because checkout re-pins the same stale SHA.

Secondary symptom: both runs helm push chart 0.1.67 with different contents (different toolbox tags), so the artifact published to GHCR diverges from what main records as 0.1.67.

Why other approaches don't fit

  • concurrency is already set (build-images-${{ github.ref }}, cancel-in-progress: false) and serialises runs — but each run is still pinned to its own trigger SHA, so it doesn't help; if anything it makes the stale-base collision near-certain for close merges.
  • cancel-in-progress: true is incompatible with this pipeline: change detection is per-push (diff vs parent), so cancelling an intermediate run drops that commit's image builds, and the connector descriptor recursion-break relies on the parent-diff — basing it on the last release instead would cause an infinite rebuild loop.
  • Merge queue serialises PR merges, but the release commit-back is a separate async workflow keyed on push to main, outside the queue — so it doesn't gate the racing release runs.

Fix

  • Re-anchor to the live branch tip (git fetch + git reset --hard origin/$GITHUB_REF_NAME) at job start, before any value is computed. Version-compute, the toolboxImage ref, and the commit-back are now all consistent with the branch tip, and a re-run recomputes against the refreshed tip instead of replaying a stale SHA.
  • Replace the fragile rebase-retry with a fail-loud guard. With the re-anchor plus the existing per-ref concurrency serialisation, the final push fast-forwards; the only remaining way the tip moves is a direct human push, where rebasing the bump is unresolvable anyway.

Scope note

bump-descriptors shares the same rebase-retry pattern. Left untouched here to keep this PR scoped to the reproduced failure; can follow up if desired.

Testing

  • python3 -c "import yaml; yaml.safe_load(...)" passes (YAML valid).
  • Logic verified against the failing run's logs; the re-run reproduced the exact failure this fix targets.

🤖 Generated with Claude Code

The publish-chart job both patch-bumps the umbrella `version` and
rewrites `ingestion.toolboxImage` in charts/insight/values.yaml from the
tree checked out at this run's trigger SHA. When two version-bumping PRs
merge back-to-back — especially two that touch src/ingestion/** and so
both rebuild the toolbox — the second run's trigger SHA does not yet
contain the first run's `chore(release)` commit, so both runs:

  * compute the SAME next umbrella version, and
  * rewrite the SAME `ingestion.toolboxImage` line to their own build tag.

The first run pushes its release commit; the second run's commit-back is
then rejected (non-fast-forward). The retry's `git pull --rebase` hits a
conflict in charts/insight/values.yaml — specifically on the
`ingestion.toolboxImage` line (the version line auto-merges because both
runs wrote the same value) — and aborts -> exit 1. A plain job re-run
repeats this forever because `checkout` re-pins the same stale SHA.

Secondary symptom: both runs `helm push` the same chart version with
different contents, so the chart published to GHCR diverges from what
main records as that version.

Fix: re-anchor the working tree to the live branch tip (`git fetch` +
`git reset --hard origin/$GITHUB_REF_NAME`) before any value is computed,
so version-compute, the toolboxImage ref and the commit-back are all
consistent with the branch tip — and a re-run recomputes against the
refreshed tip instead of replaying the stale SHA. Combined with the
existing per-ref `concurrency` serialisation the final push fast-forwards,
so the fragile rebase-retry is replaced with a fail-loud guard.

Reproduced by run 27535189783 (constructorfabric#1306 Figma then constructorfabric#1305 Workday merged
back-to-back, both touching src/ingestion/**).

Note: bump-descriptors shares the same rebase-retry pattern; left
untouched here to keep this fix scoped to the reproduced failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@mitasovr, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 31 minutes and 11 seconds. Learn how PR review limits work.

To continue reviewing without waiting, enable usage-based billing in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3ede29ca-a818-4fa9-a509-16f98f008019

📥 Commits

Reviewing files that changed from the base of the PR and between 3aeefce and d95523a.

📒 Files selected for processing (1)
  • .github/workflows/build-images.yml
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@mitasovr
mitasovr requested a review from cyberantonz June 15, 2026 14:18
@mitasovr
mitasovr merged commit 198a133 into constructorfabric:main Jun 15, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant