This post is about the errors while getting physical machine metrics with ceilometer specifically Openstack Liberty. Simply errors can be:
– No metrics shown in:
ceilometer meter-list
specifically metrics listed below:
- compute.node.cpu.frequency
- compute.node.cpu.idle.percent
- compute.node.cpu.idle.time
- compute.node.cpu.iowait.percent
- compute.node.cpu.iowait.time
- compute.node.cpu.kernel.percent
- compute.node.cpu.kernel.time
- compute.node.cpu.percent
- compute.node.cpu.user.time
– No samples in:
ceilometer sample-list –meter compute.node.cpu.* where * is the metrics mentioned above.
– Error in nova-compute logs in compute hosts:
Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).
If you have above diagnostics and you are not able to get physical host metrics with ceilometer then I wil try to explain reason and how to solve it. (BTW, I have not summited a bug yet but surely I will)
You probably have configured nova.conf (which are possibley in /etc/nova folder) with:
compute_monitors = ComputeDriverCPUMonitor
Restarted related services (i.e. nova-compute) but apparently you’ll get no outputs for meter-list and sample-list commands of ceilometer.
When you read the docs about Liberty it says about compute_monitors that:
(ListOpt) A list of monitors that can be used for getting compute metrics. You can use the alias/name from the setuptools entry points for nova.compute.monitors.* namespaces. If no namespace is supplied, the “cpu.” namespace is assumed for backwards-compatibility. An example value that would enable both the CPU and NUMA memory bandwidth monitors that used the virt driver variant: [“cpu.virt_driver”, “numa_mem_bw.virt_driver”]
So, first off all its parameter has changed. Additionally it says that it accepts a list. So below configuration should be valid.
compute_monitors = [“cpu.virt_driver”, “numa_mem_bw.virt_driver”]
But, you may not need numa metrics so, in my configuration I changed my configuration to:
compute_monitors = [“cpu.virt_driver”]
then restarted nova-compute service, but no chance I was not able to get any metric as before.
Error in nova-compute logs indicates an interesting situation here. It says there is a virt_driver monitor but it is not enabled in configuration. So I decided to check its values with the help of nova-manage command at which you can run arbitrary python codes. To check the codes I downloaded nova source from git repository. I find the compute_monitor error related source file which is SOURCE_FILE_ROOT/nova/compute/monitors/__init__.py (by simply “grep”ing error message in root directory). So I moved on to nova-manage and run the codes that I interested from __init__.py to python shell.
# get into python shell sudo nova-manage shell python No handlers could be found for logger "oslo_config.cfg" Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from oslo_config import cfg >>> CONF = cfg.CONF >>> dir(CONF) ['GroupAttr', 'StrSubWrapper', 'SubCommandAttr', 'Template', '_ConfigOpts__cache', '_ConfigOpts__clear_cache', '__abstractmethods__', '__call__', '__class__', '__contains__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__getattr__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__len__', '__metaclass__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', '_add_cli_opt', '_all_cli_opts', '_all_opt_infos', '_args', '_check_required_opts', '_cli_opts', '_config_opts', '_convert_value', '_do_get', '_get', '_get_group', '_get_opt_info', '_groups', '_namespace', '_oparser', '_opts', '_parse_cli_opts', '_parse_config_files', '_pre_setup', '_setup', '_substitute', '_unset_defaults_and_overrides', '_validate_cli_options', '_validate_default_values', 'clear', 'clear_default', 'clear_override', 'default_config_files', 'find_file', 'get', 'import_group', 'import_opt', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'list_all_sections', 'log_opt_values', 'print_help', 'print_usage', 'prog', 'project', 'register_cli_opt', 'register_cli_opts', 'register_group', 'register_opt', 'register_opts', 'reload_config_files', 'reset', 'set_default', 'set_override', 'unregister_opt', 'unregister_opts', 'usage', 'values', 'version']
Here I looked at the properties/attributes of CONF variable and I will check them for further commands. First of all I needed to set default configuration files that CONF looks for to “/etc/nova/nova.conf”.
>>> CONF.default_config_files = ["/etc/nova/nova.conf"] >>> CONF.reload_config_files() 2016-03-28 09:37:57.149 28772 WARNING oslo_config.cfg [-] Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT". True
Yes, I could read /etc/nova/nova.conf. Let’s move forward.
>>> cfg_monitors = ['cpu.' + cfg if '.' not in cfg else cfg for cfg in CONF.compute_monitors] >>> print cfg_monitors ['["cpu.virt_driver"]']
Hmmm, that looks interesting, list within list, let’s check more
>>> cfg_monitors = ['nova.compute.monitors.' + cfg if 'nova.compute.monitors.' not in cfg else cfg for cfg in cfg_monitors] >>> print cfg_monitors ['nova.compute.monitors.["cpu.virt_driver"]'] >>>
Ok that shows the error. The parameter supplied to compute_monitors are appended to cfg_monitors as it is. So, I changed /etc/nova/nova.conf with vim to contain:
compute_monitors = cpu.virt_driver
And restarted nova-compute, waited for a while. Then I started to get samples, meters and all other metrics of physical hosts.
That’s all
note to self 