Repository navigation
CKS: reconcile node capacity after live resize - #14223
andrijapanicsb wants to merge 17 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14223 +/- ##
============================================
+ Coverage 20.45% 20.89% +0.44%
- Complexity 21415 22175 +760
============================================
Files 6487 6503 +16
Lines 589303 592331 +3028
Branches 72245 72685 +440
============================================
+ Hits 120525 123791 +3266
+ Misses 455643 455097 -546
- Partials 13135 13443 +308
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andrijapanicsb
changed the title
|
@ACSHomeBot package G |
Packaging results:
Test packages are available at: |
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 26.03% |
| Branch coverage | 20.43% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
40b50b0 to
7e5a95a
Compare
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
andrijapanicsb
changed the title
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19802 |
Packaging results:
Test packages are available at: |
|
@blueorangutan test |
|
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17329)
|
Defer node preparation until a live CPU or memory change can be dispatched. On a retry of the committed offering, reconcile owned pending work without changing individual VM offerings or operator cordons.
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
Functional and regression testing on the final commit
|
| Area | Tests | Result |
|---|---|---|
| Live increases: worker CPU-only, RAM-only, combined; control CPU-only (API) and RAM-only (UI); both node types in one call; cluster-wide offering; scale-out + offering change in one call | 13 | PASS |
| Jobs executed on either management server (node access and reconciliation from MGMT_B) | – | PASS |
| Same committed offering again, same-capacity offering change, stopped-cluster offering change | – | PASS: no node reads beyond one get node, no markers, no kubelet restart |
| Node preparation only after placement: control VM live-migrated for capacity, then node read, then the single preparation patch | – | PASS |
| Failure paths: control plane unreachable, node SSH unreachable, placement/capacity failure (no pending marker left), placement failure with earlier pending work (kept byte-identical), guest already above target, kubelet failure after resize (pending + CKS cordon kept, second worker untouched, retry finishes), scale timeout, operator cordon with failure, stale ownership marker, foreign marker, committed-offering retry with a mixed-offering node, VM stopped outside CKS, concurrent scale, parameter validation | 15 | PASS |
Regression: scale-out, scale-in (size and nodeids), migrate after resize, stop/start, deleted offering in totals, plain (non-CKS) VM live/stopped scale, UI totals with mixed offerings |
7 | PASS |
| Autoscaling toggle | 1 | Pre-existing issue, same code on main → #14552 |
In every live resize: no VM reboot, no pod restart or UID change, DNS/HTTP service checks passed, the API server answered every readiness poll during control-node resizes, cluster metadata was committed only after all nodes succeeded.
Not tested: VPC tier / Netris and shared (direct-access) networks (node access changed in 306cc2e), multi-control HA, dedicated etcd, hypervisor rejection after dispatch (covered by unit tests), Kubernetes version upgrade after resizes.
Found during testing, pre-existing and independent of this PR:
- CKS: scale requests without an offering (autoscaler, size, enabling autoscaling) resize every node to an outdated cluster-wide offering #14552: scale requests without an offering (cluster-autoscaler,
size, enabling autoscaling) resize every node to an outdated cluster-wide offering - CKS: a single Kubernetes node can be scaled from the Instances view, leaving the cluster inconsistent #14553: a single CKS node can be scaled from the Instances view, leaving the cluster inconsistent
The Trillian smoke run on a8a500f is still to be done.
|
@blueorangutan package |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✖️ suse15. SL-JID 19872 |
|
suse build failed: looks like a generic test so rebuilding in the background |
false positive, i.e. rebuild passed |
|
|
@ACSHomeBot package |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
|
Retested the published packages from Both workers brought the delayed CPU online automatically. The UI retry preserved the operator cordon, and the final CPU and RAM totals matched the API and Kubernetes checks. The attached PDF contains the full test matrix and screenshots. cloudstack-pr-14223-functional-testing-c8e9058956-20261107.pdf |
|
@blueorangutan package |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19931 |
Context
#13226 added live compute-offering changes for CKS nodes, building on #12975. The VM could gain CPU/RAM while Kubernetes still reported the old capacity. Operators had to activate guest resources and restart
kubeletmanually. #14213 tracks that gap.Changes
CKS now activates and verifies the added guest CPU/RAM, restarts
kubeletwhen needed, and waits for the Node to report the new capacity and allocatable resources before completing the scale operation.Nodes are handled one at a time without draining resident Pods. While a node's resize is in progress, the node carries a
cloudstack.apache.org/cks-live-resize-pendingannotation and, unless an operator had already cordoned it, a CKS cordon. Both are set and removed with single conditional Node updates. If a scale fails, the annotation stays, and the CKS cordon stays too if the VM already reached the new offering. A retry of the same scale finishes that node before it changes any other. An existing operator cordon is preserved.QA also exposed a pre-existing cluster CPU/RAM accounting bug, not a regression from this PR. Totals are now calculated from the configured per-node-type offerings and counts, so the API and UI reflect mixed control/worker offerings correctly.
Functional and regression testing
Test environment: Cloud-managed KVM, isolated network, published packages from
f92e52d736b9, before the review changes.The tests will be repeated on the final commit - see comment #14223 (comment)
VPC/direct-access and multi-control HA were not tested. Dedicated etcd VMs are resized the same way as on main; they are not Kubernetes nodes, so nothing is reconciled on them (not live-tested). The separate existing
nodeids-only scale-in issue found during QA is tracked in #14458, with its fix in #14459.UI evidence
Successful scale task and correct aggregate capacity:
Scale request and result after Refresh
Control RAM: 2048 → 3072 MiB; worker unchanged at 3 CPU / 4096 MiB.
After Refresh: 6 CPU / 7168 MiB.