Debugging networking problems


Networking is the most important tool in IaaS clouds. It provides connectivity between instances and Internet. This tutorial will guide you through the network troubleshooting.

Debugging routed networks

Routed networks are handled by Quagga software. Each virtual machine gets one lease, which is an IP network with 29-bit mask. Traffic going from one to another instance is forwarded by instance's node to the other node and finally to the second instance.

If software responsible to forwarding internet traffic is broken, then communication between virtual machines may be impossible

All steps should be applied on both, computing nodes and management node.

Check if Quagga is running

Quagga handles the OSPF protocol in CoreCluster cloud. Check if ospf daemon is running by command:

ps aux | grep ospf
Connect with Quagga OSPF

If Quagga daemon is running, try to connect with it and list neighbors. Check OSPF pasword in file /etc/quagga/ospf.conf. Execute following command:

telnet localhost ospf

If you are able to connect, copy password into the terminal. In Quagga configuration terminal execute following:

ip ospf neighbor list

Above commands should list neighbors of your node. If none is shown, go ahead

Check network config

Check interface names in file /etc/corenetwork/ If you are using interfaces not matching with INTERFACES list, then update this file and restart node. CoreCluster and Quagga uses only interfaces from this list. If interface is missing, then network configuration might be problematic.

This setting is important especially on the new Ubuntu distributions with enpXXX interface names

Check IP forwarding

Check if ip forwarding is enabled in file /proc/sys/net/ipv4/ip_forward

Create virtual machine with lease

Create one virtual machine with routed lease. On the management node you should see new entry in routing table in few seconds, related to the instance's new subnet:

route -n

If no new lease is present, then check your OSPF configuration in file /etc/quagga/ospf.conf. This file is usually overwritten by CoreCluster on each restart, but it might be isefull to debug. If you need to adjust configuration, edit file in /etc/corenetwork/drivers/quagga/

No Internet access

If machines have no Internet access, but communication is fine, then check if Masquerade target is present in NAT table, on Management Node:

iptables -t nat -nL

This rule is necessary for internet access. Other important information is that nodes should be configured with defatult gateway as the Management Node (or any of them with multi-management node installation). Check it with:

route -n

If is especially important if you are using third party DHCP service in your cluster.

< Go back     Author: Maciej Nabozny Published: Nov. 17, 2016, 7:51 a.m.