Repository navigation
Don't fail snapshot strategy selection when a snapshot's volume row is gone - #14600
Open
bhouse-nexthop wants to merge 1 commit into
Open
bhouse-nexthop wants to merge 1 commit into
bhouse-nexthop wants to merge 1 commit into
Conversation
…s gone
Picking a snapshot strategy asks every strategy whether it can handle the
snapshot (StorageStrategyFactoryImpl.bestMatch). The Ceph, ScaleIO and
storage-system strategies look up the snapshot's volume with
findByIdIncludingRemoved() and dereference the result without a null check:
VolumeVO volumeVO = volumeDao.findByIdIncludingRemoved(volumeId);
boolean baseVolumeExists = volumeVO.getRemoved() == null; // NPE
findByIdIncludingRemoved() still returns expunged volumes, but returns null
once the volume row itself has been deleted from the volumes table. A
snapshot that still references such a volume then makes canHandle() throw,
and because every strategy is asked, the whole operation fails for every
storage type, not just Ceph or ScaleIO.
The most visible case is listSnapshots: the response's "revertable" flag
comes from SnapshotObject.isRevertable() -> getSnapshotStrategy(REVERT), so
a single such snapshot makes the list call fail with
Cannot invoke "com.cloud.storage.VolumeVO.getRemoved()" because
"volumeVO" is null
Treat a missing volume row the same as a removed volume: Ceph and ScaleIO
return CANT_HANDLE, as they already do for a removed volume, and
StorageSystemSnapshotStrategy declines REVERT. DefaultSnapshotStrategy
already handles a null volume for REVERT.
StorageSystemSnapshotStrategy also unboxed the volume's pool id (pool_id,
else last_pool_id) before its REVERT branch checked whether the volume was
removed. A removed volume can have neither, e.g. a local-storage volume
whose host was deleted (VolumeDaoImpl.updateAndRemoveVolume clears
pool_id), which hit the same NPE on listSnapshots. The pool id is now only
read where it is used, and the REVERT checks skip the volume-pool fallback
when there is none. TAKE and BACKUP still need the volume's pool and are
unchanged.
Signed-off-by: Brad House <bhouse@nexthop.ai>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #14600 +/- ##
============================================
+ Coverage 18.32% 18.33% +0.01%
- Complexity 16644 16664 +20
============================================
Files 5939 5939
Lines 537768 537773 +5
Branches 65929 65930 +1
============================================
+ Hits 98551 98609 +58
+ Misses 428008 427945 -63
- Partials 11209 11219 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
listSnapshotsfailed withCannot invoke "com.cloud.storage.VolumeVO.getRemoved()" because "volumeVO" is nullwhen any listed snapshot references a volume whose row no longer exists in thevolumestable.Fixes #14599
Why it breaks
canHandle()(StorageStrategyFactoryImpl.bestMatch).CephSnapshotStrategy,ScaleIOSnapshotStrategyandStorageSystemSnapshotStrategylook up the snapshot's volume withfindByIdIncludingRemoved()and use the result without a null check. That still finds expunged volumes, but returnsnullonce the row itself is gone.listSnapshotssets each snapshot'srevertableflag throughSnapshotObject.isRevertable()→getSnapshotStrategy(snapshot, REVERT), so a single such snapshot fails the whole list, whatever storage it is on. Other operations that pick a strategy for that snapshot fail the same way.How it is fixed
A missing volume row is treated the same as a removed volume.
baseVolumeExistsis false when the row is missing, so they returnCANT_HANDLE, as they already did for a removed volume.CANT_HANDLEfor a missing row, as it already did for a removed volume.pool_id, elselast_pool_id) before its REVERT branch checked whether the volume was removed. A removed volume can have neither, for example a local-storage volume whose host was deleted (VolumeDaoImpl.updateAndRemoveVolumeclearspool_id), and that failedlistSnapshotsthe same way.DefaultSnapshotStrategyalready handles a null volume for REVERT, solistSnapshotsnow reports such a snapshot as not revertable.Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
How Has This Been Tested?
CephSnapshotStrategyTest.canHandleTestVolumeRowMissing, and newScaleIOSnapshotStrategyTest(missing row, removed volume, and a live PowerFlex volume still gettingHIGHESTfor REVERT/DELETE) andStorageSystemSnapshotStrategyTest(REVERT with a missing row, a removed volume, a live volume without a pool id on both the backend-snapshot and the managed-pool paths, a backend snapshot on a revert-capable pool still gettingHIGHESTwhen the volume has no pool id, a volume on a managedlast_pool_idstill gettingHIGHEST, and TAKE/BACKUP still usinglast_pool_id). They fail with the NPE on the current4.22code and pass with this change.CephSnapshotStrategyTest.canHandleTestReomved*tests passednullas the removed date, so they never tested a removed volume. They now pass a date and expectCANT_HANDLE.engine/storage/snapshotpass, and checkstyle is clean for the module.listSnapshots listall=true. The fixed build has not yet been run against that deployment; the call path above was traced in the code.How did you try to break this feature and the system with this change?
nulllookup result takes the new path.DefaultSnapshotStrategyalready handles a null volume for REVERT.StorPoolSnapshotStrategydoesn't look up the volume.DefaultSnapshotStrategy(TAKE) still dereferences the volume, but TAKE only runs for a snapshot of a volume that exists.createVolume/createTemplatefrom such a snapshot) can still fail on a missing volume row. Those failed before this change as well; this PR only makes the snapshot listable and lets strategy selection decline it.StorageSystemSnapshotStrategystill dereferencesstoragePoolDao.findById()without a null check for a snapshot or volume whose storage pool row is gone. That's a different missing row and is left for a separate change.