Advanced

The following section covers advanced troubleshooting techniques. flexiWAN relies on number of underlying components, from Ubuntu Server as OS and all its functionalites such as netplan, to VPP and FRR components.

VPP troubleshooting

Packet capture

As flexiWAN utilizes VPP, regular packet captures using tcpdump will not show all traffic. Instead use the following commands to capture packets. This can also be done from flexiManage Send Command tab.

Baremetal:

  1. vppctl pcap dispatch trace on max 1000 file vrouter.pcap buffer-trace dpdk-input 1000

  2. wait 10-15 seconds

  3. vppctl pcap dispatch trace off

VMware:

  1. vppctl pcap dispatch trace on max 10000 file vrouter.pcap buffer-trace vmxnet3-input 1000

  2. wait 10-15 seconds

  3. vppctl pcap dispatch trace off

After executing the above commands, download the .pcap file from /tmp. In case the file doesn’t appear, simply re-run the packet capture commands again. Inspect the pcap using Wireshark. With the VPP pcap each packet appears multiple times showing the path of the packet through the VPP nodes. This is useful to troubleshoot network issues and tunnel connectivity issues.

VPP CLI

flexiWAN uses VPP network stack. VPP CLI and vppctl commands are available from flexiEdge shell. Learn more about VPP CLI and full list of supported commands`here <https://wiki.fd.io/view/VPP/Command-line_Interface_(CLI)_Guide>`_.

Most commonly used vppctl commnands:

vppctl show int

vppctl show hard

vppctl show ip addr

vppctl show ip fib

vppctl arp

vppctl show ip arp

OSPF troubleshooting

flexiWAN uses FRR for routing, and its component OSPF to share routing information between sites. The way the system works is that OSPF learns possible shortest paths routes. In this section we cover OSPF troubleshooting basics. Commands can be executed from shell or from the Send Command tab per device.

Check if ospf process is running:

ps -ef | grep ospf

Capture OSPF packets using tcpdump on flexiEdge:

tcpdump -n -v -s0 -c 10 -i <Linux i/f>:nnnp proto ospf   - capture 10 ospf packets

To troubleshoot FRR use the vtysh:

Show current FRR configuration:

vtysh -c "show running-config"

Show learned neighbours:

vtysh -c "show ip ospf neighbor"

Show interfaces:

vtysh -c "show ip ospf interface"

Show routes:

vtysh -c "show ip ospf route"

Warning

Manually editing OSPF is not supported. Please make all changes through flexiManage instead.

Netplan configuration

flexiWAN uses netplan.io for network interfaces configuration. Through Netplan YAML files each interface can be configured. Learn more about Netplan here.

During Ubuntu installation user is prompted to select a network interface for internet access. The interface which is selected during setup will be automatically defined in the default Netplan YAML file. This file is used when the vRouter is not running.

/etc/netplan/50-cloud-init.yaml

Interfaces configuration through flexiManage is saved in netplan files once the vRouter is started. flexiManage does not change unassigned interfaces.

UDP Connection Troubleshooting

Checking tunnel network connectivity with UDP test

In case of packet captures showing flexiWAN tunnel traffic is not arriving, run the following test to see if UDP port 4789 is open and not filtered. The test consist of running UDP server script on site A and sending UDP traffic to it from site B. Use the public IP and ports in the case of NAT as seen in the flexiManage public IP on the Device -> Interfaces page.

The test in question uses phython3 so make sure to first install pip3:

apt install python3-pip

Then follow the steps below on each of the flexiEdge devices.

  1. Install the tool

pip3 install udp-test

  1. stop the vRouter from flexiManage devices page on both routers in question.

  2. On site A start the server:

udp-test server -p 4789

  1. Connect to server on site A from site B

udp-test client -h server_IP -p 4789 -l 4789

Where server_IP is IP of the remote site. Try sending messages and confirm it’s passing.

Path Selection Troubleshooting

This section will explain how to check the policies in Flexiwan Edge/Router using basic VPP commands

Once you configure the policies on the Path Selection page you need to install into the respective Flexiwan Edge/Router, Once you install the policies FlexiManage will push the configuration to the respective Flexiwan Edge/Router.

To check whether the policies are pushed we can use the following VPP CLI commands

VPP CLI Commands to check Policy

vppctl show fwabf policy

The above command shows few important counters like matched - Number of packets matched the policy defined applied - Number of packets for which the policy is applied fallback - Number of packets for which the policy is not matched and following fallback action dropped - Number of packets dropped by applying the policy or

vppctl show acl-plugin acl index <acl index number>

acl-index 105 count 2 tag {}

0: ipv4 permit src 0.0.0.0/0 dst 20.1.1.0/24 proto 0 sport 0-65535 dport 0-65535

1: ipv4 permit src 20.1.1.0/24 dst 0.0.0.0/0 proto 0 sport 0-65535 dport 0-65535

used in lookup context index: 0, 1

The above command will show the access control list of a specific ACL index number.

Firewall Troubleshooting

flexiWAN offers several commands to display information about firewall and NAT.

  • vppctl show nat44 sessions - view current WAN NAT sessions

  • vppctl show acl-plugin interface - list interfaces with ACL index values. To be used with the next command.

  • vppctl show acl-plugin acl <index> - view allowed / blocked per ACL rule. Under “index” put the interface from the above command.