fix(ingestion): repair validate-strict gate — pin CDK image, resolve $refs by mitasovr · Pull Request #1311 · constructorfabric/insight

mitasovr · 2026-06-12T12:56:26Z

Problem

The mandatory pre-deploy gate source.sh validate-strict (connector create workflow §5.2) had two defects:

Unpinned validator image. BASE_IMAGE defaulted to airbyte/source-declarative-manifest:latest, so a routine rebuild on 2026-06-11 silently switched the gate to CDK 7.22.0 while the deployed Airbyte 1.8.5 bundles CDK 6.60.9 (airbyte-connector-builder-server/requirements.in, verified inside connector-builder-server:1.8.5).
No $ref resolution in the strict validator. Empirically (tested CDK 6.44.0 → 7.22.0), no schema version accepts instance-side $refs under raw jsonschema validation — yet the real Builder UI resolves all refs with the CDK ManifestReferenceResolver before validating (official Builder-maintained manifest-only connectors, e.g. source-bamboo-hr, use streams: [{$ref}] and $ref-with-sibling merge). So every manifest using $refs — including Builder-authored ones — could never pass the gate, regardless of image version.

Fix

Pin BASE_IMAGE to airbyte/source-declarative-manifest:6.60.9 (= deployed platform's CDK) in source.sh + Dockerfile; documented to bump together with AIRBYTE_VERSION. The local wrapper tag derives from the pin (local-<tag>), so a pin bump auto-rebuilds.
validate_strict.py preprocesses with the CDK's own ManifestReferenceResolver, then validates without the runtime ManifestNormalizer pass — matching Builder UI behavior while still catching what lenient runtime validate forgives (missing type:, templated integer-only slots, unresolvable refs).
Repair four manifests with genuine Builder blockers surfaced by the fixed gate: missing type: AddedFieldDefinition on AddFields.fields[] (claude-admin, cursor, bitbucket-server, confluence) and missing top-level type: DeclarativeSource (bitbucket-server). Mechanical, runtime-neutral additions.
Docs updated: tool README (new §"Validator image pin", corrected strict-validation semantics), source.sh usage text, .env.local.example, create workflow §5.2.

Verification

validate-strict: 16/16 connectors pass
validate (runtime): 16/16 connectors pass

Full matrix run over every connectors/*/*/connector.yaml with the pinned image; the four repaired manifests also re-pass runtime validation.

🤖 Generated with Claude Code

Summary by CodeRabbit

Documentation
- Enhanced validation workflow documentation with clearer image pinning guidance to prevent schema drift and false validation failures.
- Improved validation instructions distinguishing runtime versus strict validation modes.
Chores
- Pinned specific image versions across connector configurations and build tools to ensure consistent schema behavior.
- Updated validation tooling to better handle reference resolution during strict validation checks.

…$refs The pre-deploy strict-validation gate was unusable for any manifest using $refs, and the validator image floated on :latest. - Pin BASE_IMAGE to airbyte/source-declarative-manifest:6.60.9 — the airbyte-cdk version bundled with the deployed Airbyte platform 1.8.5 (airbyte-connector-builder-server/requirements.in). :latest drifts ahead of the deployed Builder UI schema. The local wrapper tag now derives from the pin, so bumping it auto-rebuilds the image. - validate_strict.py now preprocesses manifests with the CDK's own ManifestReferenceResolver — exactly what the Builder UI does — and then validates against declarative_component_schema.yaml without the runtime ManifestNormalizer pass. Raw no-resolution validation rejected every instance-side $ref (definitions.linked and Builder-export styles alike) on every CDK version, so manifests authored via the Builder UI could never pass the gate. - Fix four manifests with genuine Builder blockers surfaced by the repaired gate: missing type: AddedFieldDefinition on AddFields entries (claude-admin, cursor, bitbucket-server, confluence) and a missing top-level type: DeclarativeSource (bitbucket-server). - Update tool README, source.sh usage text, .env.local.example and the connector create workflow (§5.2) accordingly. All 16 declarative connectors now pass both validate-strict and runtime validate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>

coderabbitai · 2026-06-12T12:56:38Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4e6addd7-69fa-4bf2-89ba-b2551aec5d81

📥 Commits

Reviewing files that changed from the base of the PR and between 80ca589 and e4b3beb.

📒 Files selected for processing (10)

cypilot/.core/skills/connector/workflows/create.md
src/ingestion/connectors/ai/claude-admin/connector.yaml
src/ingestion/connectors/ai/cursor/connector.yaml
src/ingestion/connectors/git/bitbucket-server/connector.yaml
src/ingestion/connectors/wiki/confluence/connector.yaml
src/ingestion/tools/declarative-connector/.env.local.example
src/ingestion/tools/declarative-connector/Dockerfile
src/ingestion/tools/declarative-connector/README.md
src/ingestion/tools/declarative-connector/source.sh
src/ingestion/tools/declarative-connector/validate_strict.py

📝 Walkthrough

Walkthrough

This PR standardizes AddFields transformation entries across multiple declarative connectors to use explicit type: AddedFieldDefinition objects, updates container image pinning to version 6.60.9, and implements reference-aware strict validation using Airbyte CDK's ManifestReferenceResolver before JSON Schema checks.

Changes

AddedFieldDefinition Schema Standardization

Layer / File(s)	Summary
Claude Admin connector field definitions `src/ingestion/connectors/ai/claude-admin/connector.yaml`	Shared `tenant_id_injection` and stream-level `AddFields` blocks across messages usage, cost report, code usage, API keys, workspaces, and workspace members are converted to typed `AddedFieldDefinition` objects for all injected and derived fields.
Cursor connector field definitions `src/ingestion/connectors/ai/cursor/connector.yaml`	Shared `mandatory_fields` definition and all five streams' `AddFields` blocks update their field entries to use explicit `type: AddedFieldDefinition` wrappers while preserving path/value logic.
Bitbucket Server connector field definitions `src/ingestion/connectors/git/bitbucket-server/connector.yaml`	Manifest declares `type: DeclarativeSource` and version 6.44.0; all 25\+ streams' `AddFields` transformations restructure enriched fields to typed `AddedFieldDefinition` objects (tenant, source, and stream-specific identifiers).
Confluence connector field definitions `src/ingestion/connectors/wiki/confluence/connector.yaml`	Shared `add_fields` definition and all seven wiki streams' `transformations` blocks update field entries to explicit `AddedFieldDefinition` typed objects with identical path/value mappings.

Image Pinning and Validation Infrastructure

Layer / File(s)	Summary
Docker image and environment configuration `src/ingestion/tools/declarative-connector/Dockerfile`, `.env.local.example`	`BASE_IMAGE` is pinned to `airbyte/source-declarative-manifest:6.60.9` with comments clarifying the pin must match the deployed Airbyte platform's bundled CDK; environment example replaces deprecated `:latest` references.
Strict validation with reference resolution `src/ingestion/tools/declarative-connector/validate_strict.py`	Adds `ManifestReferenceResolver` import and wraps manifest preprocessing in try/except; resolves `$ref` entries (including Builder/runtime merge semantics) before validation; preprocessing failures are reported as strict validation errors.
Validation documentation and shell script updates `src/ingestion/tools/declarative-connector/source.sh`, `README.md`	source.sh pins `BASE_IMAGE` to 6.60.9 and derives local image tag from pinned version; README clarifies `validate` vs `validate-strict` semantics (resolver with/without normalization), expands Builder-UI rules, documents unresolvable `$ref` error interpretation, and updates docker run example with versioned tag.
Connector creation workflow documentation `cypilot/.core/skills/connector/workflows/create.md`	Phase 5.2 validation instructions are updated to document that both validators run inside the pinned `airbyte/source-declarative-manifest` image and warn against `:latest` overrides to prevent schema drift.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

mozhaev-dev

Poem

🐰 Fields wrapped in types, so structured and bright,
Images pinned to versions, no :latest in sight,
References resolved before validation's keen eye,
Consistency blooms where manifests fly!
Our connectors now speak in one voice, unified and true,
With infrastructure steady—much work kindly of you!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and concisely describes the main fix: pinning the CDK image and resolving $refs in the validate-strict gate.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

mitasovr marked this pull request as draft June 12, 2026 12:59

mitasovr mentioned this pull request Jun 12, 2026

feat(ingestion): add Workday HR connector (RaaS report contract, BambooHR parity) #1305

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(ingestion): repair validate-strict gate — pin CDK image, resolve $refs#1311

fix(ingestion): repair validate-strict gate — pin CDK image, resolve $refs#1311
mitasovr wants to merge 1 commit into
constructorfabric:mainfrom
mitasovr:fix/validate-strict-gate

mitasovr commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mitasovr commented Jun 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mitasovr commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading