VM instantiation fails with "Cannot allocate memory"

Issue

• VM instantiation fails with "Cannot allocate memory".

• We are using hugepage and there is enough memory available.

• After nova restart the issue went away.

• nova-compute was using lots of memory before the restart:

[heat-admin@overcloud-controller-0 ~]$ ps aufxg | grep nova
nova      120168  1.3  2.0 23251888 10697348 ?   Ssl  May04 1584:22 /usr/bin/python2 /usr/bin/nova-compute

• After:

[heat-admin@overcloud-controller-0 ~]$ ps aufxg | grep nova-compute
nova      494501  1.6  0.0 3275248 152844 ?      Ssl  13:04   0:37 /usr/bin/python2 /usr/bin/nova-compute

• The following traceback is seen in /var/log/nova/nova-compute.log:

2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c] Traceback (most recent call last):
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2083, in _build_resources
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     yield resources
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1924, in _build_and_run_instance
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     block_device_info=block_device_info)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2649, in spawn
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     admin_pass=admin_password)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3049, in _create_image
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     fallback_from_host)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3162, in _create_and_inject_local_root
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     instance, size, fallback_from_host)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6703, in _try_fetch_image_cache
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     size=size)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 218, in cache
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     *args, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 848, in create_image
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     prepare_template(target=base, *args, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     return f(*args, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 208, in fetch_func_sync
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     fetch_func(target=target, *args, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3156, in clone_fallback_to_fetch
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     libvirt_utils.fetch_image(*args, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 430, in fetch_image
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     images.fetch_to_raw(context, image_id, target)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 129, in fetch_to_raw
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     data = qemu_img_info(path_tmp)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 63, in qemu_img_info
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     out, err = utils.execute(*cmd, prlimit=QEMU_IMG_LIMITS)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 296, in execute
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     return processutils.execute(*cmd, **kwargs)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 368, in execute
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     env=env_variables)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 54, in __init__
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     errread, errwrite)
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]   File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c]     self.pid = os.fork()
2019-07-23 12:46:29.514 120168 ERROR nova.compute.manager [instance: 4413c956-07ed-4b72-b855-eabf8c9add9c] OSError: [Errno 12] Cannot allocate memory