Debugging storage problems


Sometimes you may find some images in failed state. Follow this guide to find problems in your cluster. This guide refers to the NFS storages only (default, netfs transport)

Step one - check mounts

Each storage is mounted in each Compute node and each Agent node in /var/lib/cloudover/storages/[storage name]/ directory. Check by mount command if storage is mounted properly

Step one and half - check permissions

Check if cloudover user could create any files in storage directory. Check its permissions and ownership.

Step two - check tasks

If storage fails to mount and/or tasks related to this storage fail, then check task queue by command:

sudo cc-manage task list

Find in task list any tasks in failed state. You could use grep to find them:

sudo cc-manage task list | grep failed

Then, check each failed task and read log information:

sudo cc-manage task info [task id]

Step three - check logs

First two steps could be done very quickly. Reading logs might take a long time, but it should give you most detailed information about cluster problems. As previously, use cc-manage command to display logs:

sudo cc-manage log 10m

You can define different amount of time to display, or all to show all logs (a lot of text!). To filter logs with any useful information, you can use tag option:

sudo cc-manage log 10m storage

Above command will show you only messages related to storage.

If your cluster is configured to use common database for logging, you can filter particular nodes. Another option is to use graphical interface, in Administrator's panel for viewing messages. On the right side you can also select instance_id, which corresponds to problematic node.

Step four - ask for help

If nothing strange was found in above steps - use our Google Group:!forum/cloudover-users
< Go back     Author: Maciej Nabozny Published: Oct. 14, 2016, 9:35 p.m.