Networking and Routing

From Dikapedia
Jump to: navigation, search


This page is mainly about useful networking/routing commands for Linux.


/etc/sysconfig/network

Global default gateway configuration is stored in the /etc/sysconfig/network file. This file specifies gateway and host information for all network interfaces.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-the_sysconfig_directory#s2-sysconfig-network

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-networkscripts-static-routes

https://www.linuxtopia.org/online_books/linux_system_administration/securing_and_optimizing_linux/chap9sec93.html

Routing


  • route command - show the IP routing table on Linux.
  • ip command (or ip route) - show routing, devices, routing policies, and tunnels on Linux.
  • netstat - show network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

To display the routing tables, you can use the 'route' or 'netstat' commands like so: 

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.31.0.1      0.0.0.0         UG        0 0          0 eth0
172.31.0.0      0.0.0.0         255.255.240.0   U         0 0          0 eth0

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.0.1      0.0.0.0         UG    100    0        0 eth0
172.31.0.0      0.0.0.0         255.255.240.0   U     100    0        0 eth0


  • The -n in option means that you want numerical IP addresses displayed, instead of the corresponding host names.
    • Note that the -n option was used with route so it wouldn’t try to resolve any of these IP addresses into hostnames. For one thing, the command runs more quickly, but more important, you don’t want to cloud your troubleshooting with any potential DNS errors.


  • The -r option in 'netstat -rn' specifies that you want the routing table. The -n option is similar to that of the route command.


You can also use 'ip route' to display the main routing table. However, note that it is a significantly different output compared to output of 'route'/'netstat -rn': 

$ ip route 
default via 172.31.0.1 dev eth0 proto dhcp metric 100 
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100 

$ ip route show
default via 172.31.0.1 dev eth0 proto dhcp metric 100 
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100 

$ ip route list
default via 172.31.0.1 dev eth0 proto dhcp metric 100 
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100

The network 172.31.0.0/20 is available on eth0 with a scope of link. That means that the network is valid and reachable through this device, eth0. As long as link remains good on that device, we should be able to reach any IP address inside of 172.31.0.0/20 through eth0 interface. You can see the source is 172.31.0.84, which means traffic leaving eth0 is coming from 172.31.0.84.

Notice that the destination which is reachable through a gateway appears in the routing table output with the keyword via. This line is reflected in the first line of the 'route -n' output as seen above.


Troubleshooting


If you don’t see a default gateway configured here, and the host you want to reach is on a different subnet (say, web1, which is on another network), then that is the likely cause of you not being able to reach the destination host.


To fix this, either be sure to set the gateway in /etc/network/interfaces on Debian-based systems or /etc/sysconfig/network_scripts/ifcfg-<interface> on Red Hat-based systems, or if you get your IP via DHCP, be sure it is set correctly on the DHCP server and then reset your interface with the following on Debian-based systems: 

# Debian-based systems
$ sudo service networking restart

# Red Hat-based systems
# sudo service network restart
Destination host unreachable
root@rhel8:~# ping 192.168.0.1 
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
From 192.168.0.146 icmp_seq=1 Destination Host Unreachable
From 192.168.0.146 icmp_seq=2 Destination Host Unreachable
From 192.168.0.146 icmp_seq=3 Destination Host Unreachable

If you've ensured that networking is up, but your VM cannot ping another host on the same network due to the error: "destination host unreachable," check to see if there is any asymmetric routing going on by looking at the route table by running one of the following:

  • route -n
  • netstat -rn
  • ip route
 $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.0.1      0.0.0.0         UG    100    0        0 eth0
172.31.0.0      0.0.0.0         255.255.240.0   U     100    0        0 eth0
172.31.0.0      0.0.0.0         255.255.240.0   U     100    0        0 virb0
  • As you can see in the above there are two routes for the destination 172.31.0.0 via the gateway 0.0.0.0. You must delete the virb0, or which ever, so that asymmetric routing doesn't occur.

To delete a route:

  • RHEL7:
# route -del -net 172.31.0.0 gw 0.0.0.0 netmask 255.255.240.0 dev virbr0
  • RHEL8:
# route del -net 172.31.0.0 gw 0.0.0.0 netmask 255.255.240.0 dev virbr0


How can I make my secondary network interface work in my CentOS or RHEL EC2 instance?


From k u m o 7874


  • IF YOU ARE USING IPv6 SIMPLY REPLACE EVERYTHING WITH IPv6 PRIVATE IPs etc. Karthik R. was able to do it.


Short Description

Adding a secondary network interface to a non-Amazon Linux instance causes traffic flow issues. This is because both the primary and the secondary network interfaces are in the same subnet, and there is only one routing table with one gateway. Traffic that comes into the secondary network interface will try to leave the instance using the primary network interface. But this is not allowed, because the secondary IP address does not belong to the MAC address of the primary network interface.

To make the secondary interface work, create a secondary network configuration file, add an additional routing table, and then set up rules in the custom routing table policy database so that traffic for the secondary interface uses the new routing table. To be sure that the new secondary route and rules are brought up in every boot, create and configure a secondary static route file.

Here is a summary of the steps for making the secondary interface work:

  1. Create a configuration file
  2. Create a new routing table
  3. Set rules in the Routing Policy Database
  4. Create a static route file


Note: For instructions for Ubuntu instances, click here.


Resolution

All procedures must be done with root user privileges. Either become root with `sudo -i` or execute all commands with `sudo`.


Create a secondary network configuration file

1. Get the name of the primary network interface: (The ^ [ [ : d i g i t : ] ] should be all together, but it doesn't show up right in the code box) 

ip a | grep ^ [ [ : d i g i t : ] ]

You should see something like this: 

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

Important: In the previous example, the primary interfaces are named 'eth0', 'eth1' and so on. However, for instances that support enhanced networking, such as the m4 and m5 family type, you might see a naming inconsistency. For example, the primary might be named 'ens3' while the secondary is named 'eth0'. This naming inconsistency happens when the secondary interface is added while the instance is running.

You can avoid a naming inconsistency by adding the interface at launch time, or rebooting the instance. Or, while the interface is running, you can change the name with: 

ip link set eth0 name ens4 && ip link set ens4 up

2. Create or edit the secondary network interface 'ifcfg-eth1' configuration file. 

vi /etc/sysconfig/network-scripts/ifcfg-eth1

3. Edit the secondary interface file so it is similar to the following. Adapt the edits to your scenario, including your secondary interface MAC address. You can find the secondary interface MAC address using the `ip a` command. 

DEVICE=eth1
NAME=eth1
HWADDR=00:00:00:00:00:00 (not necessary.)
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
NM_CONTROLLED=no
DEFROUTE=no
IPADDR=172.31.0.162
PREFIX=20               # CIDR range of subnet

If you have more than one IP on your secondary interface, configure it as follows: 

DEVICE=eth1
NAME=eth1
HWADDR=00:00:00:00:00:00
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
NM_CONTROLLED=no
DEFROUTE=no
IPADDR=172.31.0.162
PREFIX=20               # CIDR range of subnet
IPADDR1=172.31.13.162
PREFIX1=20
IPADDR2=172.31.1.151
PREFIX2=20

4. For the primary interface to not lose connectivity, be sure that the default gateway remains on the main routing table. To do so, edit the /etc/sysconfig/network file: 

vi /etc/sysconfig/network

Add the following line: 

GATEWAYDEV=eth0

Important: By default, Red Hat instances come with NetworkManager enabled. It can interfere with networking settings, specially the routing table. You can ensure it doesn't happen by adding NM_CONTROLLED=no to every ifcfg-ethX file you have, including ifcfg-eth0, or you can disable it altogether. Please see How to disable NetworkManager in RHEL?

5. Restart the network: 

systemctl restart network


Create a new secondary routing table

The main routing table has ID 254 on Linux. By default, interface traffic is routed based on this table. You must create a new routing table for the secondary interface.

1. Find and take note of your default gateway: 

ip route | grep default

2. Create a new routing table for the secondary interface, and then add the default gateway route using the IP of the gateway you found on the previous step. In this example, the new table is ID 1000, and the IP is "172.31.16.1": 

ip route add default via 172.31.16.1 dev eth1 table 1000

3. Be sure that table 1000 has a route for every IP present on the secondary interface.

Here's an example for two IPs: 

ip route add 172.31.0.162 dev eth1 table 1000
ip route add 172.31.13.162 dev eth1 table 1000

4. Review table 1000 and be sure it is correct: 

ip route show table 1000


Set rules in the routing policy database

1. Set rules for every IP present on the secondary interface in the routing policy database so that traffic coming from these IPs is routed according to table 1000. 

ip rule add from 172.31.0.162 lookup 1000
ip rule add from 172.31.13.162 lookup 1000

Be sure to check the connectivity with these IPs. If you have Elastic IPs pointing to these IPs, they can be accessed from the public network as well.


Create a secondary static route file

1. To bring the new routes and rules up with every boot, create and configure the 'route-eth1' static route file. 

vi /etc/sysconfig/network-scripts/route-eth1

2. Enter the same routes as you entered when creating the secondary routing table at the command line. The difference is that you omit the "ip route add". 

default via 172.31.16.1 dev eth1 table 1000
172.31.0.162 dev eth1 table 1000
172.31.13.162 dev eth1 table 1000

3. Create or edit a rule file for rule-eth1: 

vi /etc/sysconfig/network-scripts/rule-eth1

Put in the same routes as you entered on the command line in when creating the new secondary routing table. The difference is that you omit the "ip rule add". 

from 172.31.0.162 lookup 1000
from 172.31.13.162 lookup 1000

After configuring the CentOS or RHEL IP settings, routing and rules will persist when rebooting the interface. :)

PLEASE NOTE:

  • To test, ping the private IP or the corresponding Elastic IP address to validate the connectivity to the IPs on the secondary network interface.
  • If you have Elastic IPs that are pointing to these IPs, then they can be accessed from the public network as well.
  • The security groups attached to the secondary network interface should allow ICMP traffic for the ping test to return a successful response !!!!!!!.


How can I make my secondary network interface work in my Ubuntu instance?


How can I make my secondary network interface work in my Ubuntu instance?

  • NOTE: for whatever reason, the instructions for Ubuntu 18/20 no longer allows me to connect to both private IP addresses of each NIC if the NICs are in the same subnet... likely because of asymmetric routing. Not sure what changed in the doc, if they removed something or if Ubuntu handles the configuration/instructions differently now. WORKAROUND: create a new subnet in the same AZ, create a new ENI/NIC from that new subnet and attach it to the instance as secondary NIC. Now, following the instructions in the doc above, you should now be able to connect to both private IP addresses of both nics.
ip r show table 1000


Other Random Networking tools/info


Command line interface for testing internet latency using speedtest.net: https://github.com/sivel/speedtest-cli 

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -



If you get "No route to host" after spinning up a VM

Check if ipv6 is enabled. Disable it like so: 

# vi /etc/sysctl.conf
........................................................................
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

# sysctl -p

or 

# reboot


References:


Learn more about ip route. [+] https://learning.oreilly.com/library/view/devops-troubleshooting-linux/9780133035513/ch05.html
[+] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-networkscripts-static-routes

Retrieved from "https://dikapedia.com/wiki/index.php?title=Networking_and_Routing&oldid=2821"