RHEV VM has been paused due to lack of storage spa

时间 2019-11-08

标签 rhev paused lack storage spa 繁體版

原文原文链接

RHEV VM has been paused due to lack of storage space after RHEV-H upgrade to version 20140112.0.el6

问题

环境
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140112.0.el6)
vdsm 4.13.2-0.6python

问题
VM has been paused due to lack of storage space.
VM has been paused due to no Storage space error.
Error message received in audit_log:sql

2014-02-07 13:39:42.781 | Failed to start VM vmname, because exist snapshot for stateless state. Snapshot will be deleted.  | c1102bb

Since upgrading to RHEVH-6.5 v.20140112.0.el6 /var/log/messages are being flooded by these messages:
因为两个rhevh升级至6.5 v.20140112.0.el6 /是/日志/信息是被flooded这些信息：app

Jan 29 05:17:39 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
#012Traceback (most recent call last):#012  
File "/usr/share/vdsm/sampling.py", line 351, in collect#012  
File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  
File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  
File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012  
File "/usr/share/vdsm/vm.py", line 842, in f#012  
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012  
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

/var/log/vdsm/vdsm.log contains these logs:
/var/log/vdsm/vdsm.log包含如下日志：less

Thread-5949::DEBUG::2014-01-29 05:14:33,027::sampling::285::vm.Vm::(start) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Start statistics collection
Thread-5969::DEBUG::2014-01-29 05:14:33,027::sampling::314::vm.Vm::(run) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats thread started
Thread-5969::DEBUG::2014-01-29 05:14:33,033::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 8 edom: 10 level: 2 message: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
Thread-5969::ERROR::2014-01-29 05:14:33,034::sampling::355::vm.Vm::(collect) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
Traceback (most recent call last):
  File "/usr/share/vdsm/sampling.py", line 351, in collect
  File "/usr/share/vdsm/sampling.py", line 226, in __call__
  File "/usr/share/vdsm/vm.py", line 529, in _highWrite
  File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded
  File "/usr/share/vdsm/vm.py", line 842, in f
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

决议

If this error is received, an error relating to the VM being paused due to lack of storage space. For information on the advisory, and where to find the updated files, follow the link: vdsm 3.3.0-3 bug fix update
若是收到此错误，则与因为缺乏存储空间而暂停的虚拟机相关的错误。有关建议的信息以及在何处查找更新的文件，请访问如下连接：vdsm 3.3.0-3 bug fix updatedom

For more troubleshooting, for upgrades and roll-back issues, see the following guide: RHEV 3.2 to 3.3 Upgrade Guide: Tips, Considerations and Roll-back Issues
有关更多故障排除，有关升级和回滚问题，请参阅如下指南：RHEV 3.2到3.3升级指南：提示、注意事项和回滚问题ide

根源

This seems to occur if a VM is migrated to a hypervisor running vdsm 4.13.2-0.6 from a hypervisor running an earlier version of vdsm.
若是从运行早期版本vdsm的管理程序迁移到运行vdsm 4.13.2-0.6的管理程序，则彷佛会发生这种状况。ui

After the VM is migrated libvirt and vdsm will end up having different path information about the VMs disks.
在迁移VM以后，libvirt和vdsm最终将拥有关于vms磁盘的不一样路径信息。this

During the ExtendDrivesIfNeeded method vdsm asks libvirt for block info about the the disk and passes in a path to the disk, however libvirt has a different path to the disk and reports the invalid path error.
Note that if the VM is started on the vdsm 4.13.2-0.6 hypervisor then the error doesn't occur.
在extenddrivesifrequired方法期间，vdsm要求libvirt提供关于磁盘的块信息，并将路径传递到磁盘，可是libvirt有不一样的磁盘路径，并报告无效的路径错误。
注意，若是VM是在vdsm 4.13.2-0.6 hypervisor上启动的，则不会发生错误。lua

诊断步骤

After migrating the VM to hypervisor running vdsm 4.13.2-0.6, one can see it maintains path information in the format /rhev/data-center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid], like so:
将VM迁移到运行vdsm 4.13.2-0.6的hypervisor后，能够看到它以/rhev/data center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid]的格式维护路径信息，以下:spa

# vdsClient -s 0 list

ed1061e9-a91d-45f2-a62e-e90a57bcd32a
    Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...

But looking at the info libvirt maintains about the VM, the path is specified as /rhev/data-center/[spuuid]/[sduuid]/images/[imguuid]/[voluuid], like so:
但查看libvirt维护的有关虚拟机的信息，路径被指定为/rhev/data center/[spuid]/[sduuid]/images/[imguuid]/[voluuid]，以下所示：

# virsh -r domblklist testvm
Target     Source
------------------------------------------------
hdc        -
vda        /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5

And then the /var/log/messages is flooded with these errors:
而后' /var/log/messages '被这些错误淹没:

Jan 29 07:15:36 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012  File "/usr/share/vdsm/vm.py", line 842, in f#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

But if the VM is started directly on the vdsm 4.13.2-0.6 hypervisor then vsdm and libvirt maintain the same path information:
可是，若是虚拟机直接在vdsm 4.13.2-0.6管理程序上启动，那么vsdm和libvirt会维护相同的路径信息：

# vdsClient -s 0 list

ed1061e9-a91d-45f2-a62e-e90a57bcd32a
    Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...

# virsh -r domblklist testvm
Target     Source
------------------------------------------------
hdc        -
vda        /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5