RHEV: A Virtual Machine with a large number of snapshots may fail to start and/or no new snapshots can be created.

Issue

• A VM fails to start and reports the following in the RHEV Admin Portal;

VM <vm-name> is down. Exit message: Unable to read from monitor: Connection reset by peer.

• The vdsm logs on the host that the VM tried to start on contain;

Thread-3743::DEBUG::2014-02-13 10:03:45,861::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 10 level: 2 message: Unable to read from monitor: Connection reset by peer

• The qemu log for the VM in question contains;

-drive file=/rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2,if=none,id=drive-virtio-disk0,format=qcow2,serial=333a1d04-5b14-479a-8716-c753f7e539eb,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2: No such file or directory

• Attempting to create a new snapshot fails with the following sequence reported in the vdsm logs;

b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,142::volume::1058::Storage.Misc.excCmd::(createVolume) '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b ../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7d
fe3fb6 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27' (cwd /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d83
1b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920)

......... etc., etc., .........

b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,291::volume::1058::Storage.Misc.excCmd::(createVolume) FAILED: <err> = '/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: error while creating qcow2: No such file or directory\n'; <rc> = 1

......... etc., etc., .........

b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::286::Storage.Volume::(clone) Volume.clone: can't clone: /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6 to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27
b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::508::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/volume.py", line 466, in create
    srcVolUUID, imgPath, volPath)
  File "/usr/share/vdsm/storage/blockVolume.py", line 197, in _create
    volParent.clone(imgPath, volUUID, volFormat, preallocate)
  File "/usr/share/vdsm/storage/volume.py", line 287, in clone
    raise se.CannotCloneVolume(self.volumePath, dst_path, str(e))
CannotCloneVolume: Cannot clone volume: 'src=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6, dst=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: Error creating a new volume: (["Formatting \'/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27\', fmt=qcow2 size=1073741824 backing_file=\'../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6\' backing_fmt=\'qcow2\' encryption=off cluster_size=65536 "],)'

• After upgrading a RHEV host to vdsm-4.13.2-0.6 and qemu-kvm-rhev-0.12.1.2-2.415 or later versions, a VM with more than 23 snapshots failed to start. Attempting to create a new snapshot also failed.