Skip to content

[fix] Reconcile live StatefulSet drift and preserve HPA replica ownership - #545

Open
Al-assad wants to merge 3 commits into
apache:masterfrom
Al-assad:fix/statefulset-live-state-reconcile
Open

Al-assad wants to merge 3 commits into
apache:masterfrom
Al-assad:fix/statefulset-live-state-reconcile

Conversation

@Al-assad

@Al-assad Al-assad commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

StatefulSet comparison trusted stored desired-state hashes, hiding live image/replica drift and omitting update strategy. Its legacy environment compatibility path could dereference nil replicas. CN template updates could reset HPA-scaled replicas to the initial CR count.

Compare normalized live managed StatefulSet fields, including API-defaulted Pod/container/probe/volume values. Preserve harmless legacy CONFIGMAP_MOUNT_PATH duplicates using actual field comparison. Render the BE init-image fallback before comparison. Patch from the current object with optimistic locking and retain external template metadata. While CN HPA is configured, use CR replicas for creation only and preserve actual replicas on subsequent template updates.

Validation:

  • Reproduced cached-hash image/replica/strategy/environment drift and both omitted/explicit initial-replica HPA update failures before the fix.
  • Regression covers DCR and DDC hash keys, legacy environment replica-only updates, HPA template updates, and actual DDC FE/MS/CG reconcile paths.
  • Kubernetes 1.31 API test verifies defaulting, live drift repair, external template metadata preservation, and no write on repeated reconcile.
  • Full go test ./... passed with Kubernetes 1.31 envtest.

QA validation passed on isolated DCR/DDC workloads. rc07 verified upgrade without Pod/template/PVC changes, native BE and DDC MS replica correction, strategy repair, and HPA template updates retaining 2 actual replicas for both explicit and omitted initial counts. Final combined selectdb/doris-operator:26.1.0-rc10 also passed event-driven drift correction with no CR mutation and retained sentinel data (3 rows, sum 6).

Additional regressions ensure the legacy exception never suppresses newly added configuration environment variables, and unsupported immutable StatefulSet changes are still rejected by the API. The final combined source passed full Go tests, vet, and targeted race checks; both image architectures passed startup smoke tests. Existing immutable PVC handling and CG graceful/decommission workflows remain responsible for their fields. Ownership protection is submitted separately in #542 and should land before broad drift reconciliation.

@Al-assad
Al-assad marked this pull request as ready for review November 4, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant