Physical Host Metrics in Ceilometer – An Openstack Liberty Tale

This post is about the errors while getting physical machine metrics with ceilometer specifically  Openstack Liberty. Simply errors can be:

– No metrics shown in:

ceilometer meter-list

specifically metrics listed below:

  • compute.node.cpu.frequency
  • compute.node.cpu.idle.percent
  • compute.node.cpu.idle.time
  • compute.node.cpu.iowait.percent
  • compute.node.cpu.iowait.time

  • compute.node.cpu.kernel.percent
  • compute.node.cpu.kernel.time
  • compute.node.cpu.percent
  • compute.node.cpu.user.time

– No samples in:

ceilometer sample-list –meter compute.node.cpu.* where * is the metrics mentioned above.

– Error in nova-compute logs in compute hosts:

Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).

If you have above diagnostics and you are not able to get physical host metrics with ceilometer then I wil try to explain reason and how to solve it. (BTW, I have not summited a bug yet but surely I will)

You probably have configured nova.conf (which are possibley in /etc/nova folder) with:

compute_monitors = ComputeDriverCPUMonitor

Restarted related services (i.e. nova-compute) but apparently you’ll get no outputs for meter-list and sample-list commands of ceilometer.

When you read the docs about Liberty it says about compute_monitors that:

(ListOpt) A list of monitors that can be used for getting compute metrics. You can use the alias/name from the setuptools entry points for nova.compute.monitors.* namespaces. If no namespace is supplied, the “cpu.” namespace is assumed for backwards-compatibility. An example value that would enable both the CPU and NUMA memory bandwidth monitors that used the virt driver variant: [“cpu.virt_driver”, “numa_mem_bw.virt_driver”]

So, first off all its parameter has changed. Additionally it says that it accepts a list. So below configuration should be valid.

compute_monitors = [“cpu.virt_driver”, “numa_mem_bw.virt_driver”]

But, you may not need numa metrics so, in my configuration I changed my configuration to:

compute_monitors = [“cpu.virt_driver”]

then restarted nova-compute service, but no chance I was not able to get any metric as before.

Error in nova-compute logs indicates an interesting situation here. It says there is a virt_driver monitor but it is not enabled in configuration. So I decided to check its values with the help of nova-manage command at which you can run arbitrary python codes. To check the codes I downloaded nova source from git repository. I find the compute_monitor error related source file which is SOURCE_FILE_ROOT/nova/compute/monitors/ (by simply “grep”ing error message in root directory).  So I moved on to nova-manage and run the codes that I interested from to python shell.

# get into python shell

sudo nova-manage shell python

No handlers could be found for logger "oslo_config.cfg"
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from oslo_config import cfg
>>> CONF = cfg.CONF
>>> dir(CONF)
['GroupAttr', 'StrSubWrapper', 'SubCommandAttr', 'Template', '_ConfigOpts__cache', '_ConfigOpts__clear_cache', '__abstractmethods__', '__call__', '__class__', '__contains__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__getattr__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__len__', '__metaclass__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', '_add_cli_opt', '_all_cli_opts', '_all_opt_infos', '_args', '_check_required_opts', '_cli_opts', '_config_opts', '_convert_value', '_do_get', '_get', '_get_group', '_get_opt_info', '_groups', '_namespace', '_oparser', '_opts', '_parse_cli_opts', '_parse_config_files', '_pre_setup', '_setup', '_substitute', '_unset_defaults_and_overrides', '_validate_cli_options', '_validate_default_values', 'clear', 'clear_default', 'clear_override', 'default_config_files', 'find_file', 'get', 'import_group', 'import_opt', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'list_all_sections', 'log_opt_values', 'print_help', 'print_usage', 'prog', 'project', 'register_cli_opt', 'register_cli_opts', 'register_group', 'register_opt', 'register_opts', 'reload_config_files', 'reset', 'set_default', 'set_override', 'unregister_opt', 'unregister_opts', 'usage', 'values', 'version']

Here I looked at the properties/attributes of CONF variable and I will check them for further commands. First of all I needed to set default configuration files that CONF looks for to “/etc/nova/nova.conf”.

>>> CONF.default_config_files = ["/etc/nova/nova.conf"]
>>> CONF.reload_config_files()
2016-03-28 09:37:57.149 28772 WARNING oslo_config.cfg [-] Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".

Yes, I could read /etc/nova/nova.conf. Let’s move forward.

>>> cfg_monitors = ['cpu.' + cfg if '.' not in cfg else cfg for cfg in CONF.compute_monitors]
>>> print cfg_monitors

Hmmm, that looks interesting, list within list, let’s check more

>>> cfg_monitors = ['nova.compute.monitors.' + cfg if 'nova.compute.monitors.' not in cfg else cfg for cfg in cfg_monitors]
>>> print cfg_monitors

Ok that shows the error. The parameter supplied to compute_monitors are appended to cfg_monitors as it is. So, I changed /etc/nova/nova.conf with vim to contain:

compute_monitors = cpu.virt_driver

And restarted nova-compute, waited for a while. Then I started to get samples, meters and all other metrics of physical hosts.

That’s all


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s