Visitar URL original
linstor: fix live migration with storage into Linstor when the target host has no replica by rp- · Pull Request #14344 · apache/cloudstack · GitHub
Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
linstor: make the new resource available on the target host for live …
…migration

Live migration with storage into a Linstor pool spawned the destination
resource through the resource group's auto-placement, which knows nothing
about the target host. PrepareForMigrationCommand still describes the source
disks, so the target agent never connected the new volume either. If the
placement left the target host without a copy (place count 1, no quorum
tiebreaker, 4+ nodes with 2 copies, node filters), the block device was
missing there and libvirt aborted with
"cannot precreate storage for disk type 'block'".

Make the resource available on the target host right after creating it and
take the device path from there. The source is not a Linstor resource of
this pool, so no dual-primary is needed.

A failure before the migration command (creating the resource,
make-available, prepare for migration) now also removes the already created
destination volumes; they used to be left behind. Once the migration
command is sent, the destination volumes are only removed if the source
host reports a failure. If the command times out, the VM may still end up
on the new volumes: like StorageSystemDataMotionStrategy, check whether it
runs on the target host and finish the migration in that case, otherwise
keep the volumes. The error message names the destination pools instead of
the source host.

Unit tests cover make-available before the migration with the device path
from the target host, the cleanup when creating the resource,
make-available, prepare for migration or the migration itself fails, and a
migration timeout with and without the VM running on the target host.
  • Loading branch information
rp- committed Oct 8, 2026
commit dde2377ce329199dd2e29abcf8fbffb97ccb3ae6
8 changes: 8 additions & 0 deletions plugins/storage/volume/linstor/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ All notable changes to Linstor CloudStack plugin will be documented in this file
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2026-10-07]

### Fixed

- Live migration with storage into Linstor failed with "cannot precreate storage for disk type 'block'"
if the auto-placement didn't put a copy of the new resource on the target host; the resource is
now made available on the target host before the migration starts.

## [2026-06-24]

### Fixed
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import com.linbit.linstor.api.model.ApiCallRcList;
import com.linbit.linstor.api.model.ResourceDefinition;
import com.linbit.linstor.api.model.ResourceDefinitionModify;
import com.linbit.linstor.api.model.ResourceMakeAvailable;

import javax.inject.Inject;

Expand All @@ -35,6 +36,8 @@

import com.cloud.agent.AgentManager;
import com.cloud.agent.api.Answer;
import com.cloud.agent.api.CheckVirtualMachineAnswer;
import com.cloud.agent.api.CheckVirtualMachineCommand;
import com.cloud.agent.api.MigrateAnswer;
import com.cloud.agent.api.MigrateCommand;
import com.cloud.agent.api.PrepareForMigrationCommand;
Expand All @@ -54,6 +57,7 @@
import com.cloud.storage.dao.VolumeDao;
import com.cloud.utils.exception.CloudRuntimeException;
import com.cloud.vm.VMInstanceVO;
import com.cloud.vm.VirtualMachine;
import com.cloud.vm.dao.VMInstanceDao;
import org.apache.cloudstack.engine.subsystem.api.storage.CopyCommandResult;
import org.apache.cloudstack.engine.subsystem.api.storage.DataMotionStrategy;
Expand Down Expand Up @@ -169,11 +173,51 @@ private VolumeVO createNewVolumeVO(Volume volume, StoragePoolVO storagePoolVO) {
return _volumeDao.persist(newVol);
}

private DevelopersApi getLinstorAPI(StoragePoolVO storagePool) {
return LinstorUtil.getLinstorAPI(storagePool.getHostAddress(),
LinstorConfigurationManager.ApiToken.valueIn(storagePool.getId()),
Boolean.TRUE.equals(LinstorConfigurationManager.InsecureSsl.valueIn(storagePool.getId())));
}

/**
* Makes the new resource available on the migration target host and returns its device path there.
*
* <p>The resource is spawned by LINSTOR's auto-placement, which does not know about the target host.
* libvirt can only precreate file disks on the target, so if the block device is missing there the
* migration fails with "cannot precreate storage for disk type 'block'". The source disk is not on
* LINSTOR, so a plain (diskless for DRBD) make-available is enough, no dual-primary needed.</p>
*/
private String makeAvailableOnHost(StoragePoolVO storagePool, String rscName, Host host) {
DevelopersApi api = getLinstorAPI(storagePool);
try {
logger.info("Linstor: make resource {} available on migration target {}", rscName, host.getName());
ApiCallRcList answers = api.resourceMakeAvailableOnNode(rscName, host.getName(), new ResourceMakeAvailable());
LinstorUtil.checkLinstorAnswersThrow(answers);
return LinstorUtil.getDevicePath(api, rscName);
} catch (ApiException apiEx) {
logger.error("Linstor: ApiEx - {}", apiEx.getMessage());
throw new CloudRuntimeException(apiEx.getBestMessage(), apiEx);
}
}

/**
* A paused domain (incoming migration still in progress) is reported as PowerUnknown, so this
* is only true once the VM really runs on the host.
*/
private boolean isVmRunningOnHost(VirtualMachineTO vmTO, Host host) {
try {
Answer answer = _agentManager.send(host.getId(), new CheckVirtualMachineCommand(vmTO.getName()));
return answer instanceof CheckVirtualMachineAnswer && answer.getResult() &&
VirtualMachine.PowerState.PowerOn.equals(((CheckVirtualMachineAnswer) answer).getState());
} catch (AgentUnavailableException | OperationTimedoutException e) {
logger.warn("Unable to check if VM [{}] is running on host [{}]", vmTO, host, e);
return false;
}
}

private void removeExactSizeProperty(VolumeInfo volumeInfo) {
StoragePoolVO destStoragePool = _storagePool.findById(volumeInfo.getDataStore().getId());
DevelopersApi api = LinstorUtil.getLinstorAPI(destStoragePool.getHostAddress(),
LinstorConfigurationManager.ApiToken.valueIn(destStoragePool.getId()),
Boolean.TRUE.equals(LinstorConfigurationManager.InsecureSsl.valueIn(destStoragePool.getId())));
DevelopersApi api = getLinstorAPI(destStoragePool);

ResourceDefinitionModify rdm = new ResourceDefinitionModify();
rdm.setDeleteProps(Collections.singletonList(LinstorUtil.LIN_PROP_DRBDOPT_EXACT_SIZE));
Expand Down Expand Up @@ -266,10 +310,10 @@ private void handlePostMigration(boolean success, Map<VolumeInfo, VolumeInfo> sr
_volumeService.expungeVolumeAsync(destVolumeInfo);

if (destroyFuture.get().isFailed()) {
logger.debug("Failed to clean up dest volume on storage");
logger.warn("Failed to clean up dest volume {} on storage", destVolumeInfo);
}
} catch (Exception e) {
logger.debug("Failed to clean up dest volume on storage", e);
logger.warn("Failed to clean up dest volume {} on storage", destVolumeInfo, e);
}
}
}
Expand All @@ -293,9 +337,7 @@ private void handlePostMigration(boolean success, Map<VolumeInfo, VolumeInfo> sr
private boolean needsExactSizeProp(VolumeInfo srcVolumeInfo) {
StoragePoolVO srcStoragePool = _storagePool.findById(srcVolumeInfo.getDataStore().getId());
if (srcStoragePool.getPoolType() == Storage.StoragePoolType.Linstor) {
DevelopersApi api = LinstorUtil.getLinstorAPI(srcStoragePool.getHostAddress(),
LinstorConfigurationManager.ApiToken.valueIn(srcStoragePool.getId()),
Boolean.TRUE.equals(LinstorConfigurationManager.InsecureSsl.valueIn(srcStoragePool.getId())));
DevelopersApi api = getLinstorAPI(srcStoragePool);

String rscName = LinstorUtil.RSC_PREFIX + srcVolumeInfo.getPath();
try {
Expand Down Expand Up @@ -335,6 +377,7 @@ public void copyAsync(Map<VolumeInfo, DataStore> volumeDataStoreMap, VirtualMach

Map<String, MigrateCommand.MigrateDiskInfo> migrateStorage = new HashMap<>();
Map<VolumeInfo, VolumeInfo> srcVolumeInfoToDestVolumeInfo = new HashMap<>();
boolean migrateCommandSent = false;

try {
for (Map.Entry<VolumeInfo, DataStore> entry : volumeDataStoreMap.entrySet()) {
Expand All @@ -351,20 +394,25 @@ public void copyAsync(Map<VolumeInfo, DataStore> volumeDataStoreMap, VirtualMach
VolumeVO destVolume = createNewVolumeVO(srcVolume, destStoragePool);

VolumeInfo destVolumeInfo = _volumeDataFactory.getVolume(destVolume.getId(), destDataStore);
// registered right away, so a failure from here on cleans the new volume up again
srcVolumeInfoToDestVolumeInfo.put(srcVolumeInfo, destVolumeInfo);

destVolumeInfo.processEvent(ObjectInDataStoreStateMachine.Event.MigrationCopyRequested);
destVolumeInfo.processEvent(ObjectInDataStoreStateMachine.Event.MigrationCopySucceeded);
destVolumeInfo.processEvent(ObjectInDataStoreStateMachine.Event.MigrationRequested);

boolean exactSize = needsExactSizeProp(srcVolumeInfo);

String devPath = LinstorUtil.createResource(
destVolumeInfo, destStoragePool, _storagePoolDao, exactSize);
LinstorUtil.createResource(destVolumeInfo, destStoragePool, _storagePoolDao, exactSize);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if creating the linstor resource fails here, does the new volume row still get left behind? it only goes into the cleanup list a few lines later

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also thanks, it is added to the cleanup list now after the DB object was created

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that covers it, thanks


_volumeDao.update(destVolume.getId(), destVolume);
destVolume = _volumeDao.findById(destVolume.getId());

destVolumeInfo = _volumeDataFactory.getVolume(destVolume.getId(), destDataStore);
srcVolumeInfoToDestVolumeInfo.put(srcVolumeInfo, destVolumeInfo);

String devPath = makeAvailableOnHost(
destStoragePool, LinstorUtil.RSC_PREFIX + destVolumeInfo.getUuid(), destHost);

MigrateCommand.MigrateDiskInfo migrateDiskInfo = new MigrateCommand.MigrateDiskInfo(
srcVolumeInfo.getPath(),
Expand All @@ -375,8 +423,6 @@ public void copyAsync(Map<VolumeInfo, DataStore> volumeDataStoreMap, VirtualMach
migrateDiskInfoList.add(migrateDiskInfo);

migrateStorage.put(srcVolumeInfo.getPath(), migrateDiskInfo);

srcVolumeInfoToDestVolumeInfo.put(srcVolumeInfo, destVolumeInfo);
}

PrepareForMigrationCommand pfmc = new PrepareForMigrationCommand(vmTO);
Expand Down Expand Up @@ -410,26 +456,49 @@ public void copyAsync(Map<VolumeInfo, DataStore> volumeDataStoreMap, VirtualMach
boolean kvmAutoConvergence = StorageManager.KvmAutoConvergence.value();
migrateCommand.setAutoConvergence(kvmAutoConvergence);

MigrateAnswer migrateAnswer = (MigrateAnswer) _agentManager.send(srcHost.getId(), migrateCommand);
boolean success = migrateAnswer != null && migrateAnswer.getResult();
// once the migrate command is sent, the VM may end up running on the new volumes,
// so they are only removed if the source host reports the migration as failed
migrateCommandSent = true;
MigrateAnswer migrateAnswer = null;
boolean success;
try {
migrateAnswer = (MigrateAnswer) _agentManager.send(srcHost.getId(), migrateCommand);
success = migrateAnswer != null && migrateAnswer.getResult();
} catch (OperationTimedoutException ex) {
// no answer from the source host, but the migration may still have finished
if (!isVmRunningOnHost(vmTO, destHost)) {
throw ex;
}
logger.info("VM [{}] is running on the destination host [{}], migration was successful", vmTO, destHost);
success = true;
}

handlePostMigration(success, srcVolumeInfoToDestVolumeInfo, vmTO, destHost);

if (migrateAnswer == null) {
throw new CloudRuntimeException("Unable to get an answer to the migrate command");
}
if (!success) {
if (migrateAnswer == null) {
throw new CloudRuntimeException("Unable to get an answer to the migrate command");
}

if (!migrateAnswer.getResult()) {
errMsg = migrateAnswer.getDetails();

throw new CloudRuntimeException(errMsg);
}
} catch (AgentUnavailableException | OperationTimedoutException | CloudRuntimeException ex) {
errMsg = String.format(
"Copy volume(s) of VM [%s] to storage(s) [%s] and VM to host [%s] failed in LinstorDataMotionStrategy.copyAsync. Error message: [%s].",
vmTO, srcHost, destHost, ex.getMessage());
vmTO, volumeDataStoreMap.values(), destHost, ex.getMessage());
logger.error(errMsg, ex);

if (!migrateCommandSent && !srcVolumeInfoToDestVolumeInfo.isEmpty()) {
// failed before the migration was started: remove the already created destination volumes
try {
handlePostMigration(false, srcVolumeInfoToDestVolumeInfo, vmTO, destHost);
} catch (Exception e) {
logger.warn("Failed to clean up the destination volume(s) of VM [{}]", vmTO, e);
}
}

throw new CloudRuntimeException(errMsg);
} finally {
CopyCmdAnswer copyCmdAnswer = new CopyCmdAnswer(errMsg);
Expand Down
Loading