Watchdog

With a system (Remote SDR) running remotely 24 hours a day, we are not immune to a system crash and especially a loss of remote control. To remedy this, it is possible to set up a “Watchdog” which constantly tests the connection to the Internet and restarts the system in the event of no response after a certain time.

Kernel Watchdog

The Linux kernel watchdog is used to monitor if a system is running. It is supposed to automatically restart systems suspended due to unrecoverable software errors. The monitoring module is specific to the hardware or processor used.

To test the possibility of using this functionality on your system, open a terminal window in root mode and type:

echo 1 > /dev/watchdog

Your system will reboot after 15 s in general. It works on Raspberry Pi 4, Orange Pi etc…All you have to do is write a piece of code that every 5 s for example will execute this same command which restarts the 15 s counter and avoids the reboot unless it stops.

In the case below, we will test the presence of the ethernet network and restart the watchdog only if it is present. So in case of blocking of the WIFI no longer allowing to connect remotely, the system will reboot. For this, we will install the ping3 library with:

pip3 install ping3

Below is a small code in python3 which will excite the watchdog every 5s unless there is no internet.

#!/usr/bin/python3        

import os
import time
from ping3 import ping

adr_start = "/home/Last_boot.txt"
periode_to_test_internet=20 #20*5s
internet_ok=True

def myping(host):
    resp = ping(host)
    if resp == False:
        return False
    else:
        return True


m="Last Boot : " +time.asctime() 
file=open(adr_start,"w")
file.write(m+"\r\n") 
file.close()

delay_watchdog = 2*periode_to_test_internet #increase delay at start

try:
    print ("Press CTRL+C to exit")
    while True:
        time.sleep(5)
        if delay_watchdog>0:
            delay_watchdog=delay_watchdog-1
        else :
            internet_ok =False
            delay_watchdog = periode_to_test_internet
            ok1 = myping("www.google.com")
            ok2 = myping("192.168.0.254")
            if ok1 == True or ok2 == True :
                internet_ok = True
        if internet_ok and delay_watchdog < periode_test_internet :
            os.system("echo 1 > /dev/watchdog") # Excite the watchdog 
                
except KeyboardInterrupt:
    print (" Bye.")

The internet test is not done immediately so as not to overload the system, and to give time to regain control if your program crashes, as this can generate an infernal loop where your system keeps rebooting. At each reboot, we write the time in a file to have the date of the last start.

Installation of the “Watchdog” application

An alternative solution is to install a software that monitors the functionality of your single board computer such as the presence of the ethernet network. It will launch a reboot process if necessary. On an Orange Pi or a Raspberry Pi, open a terminal using ssh and download the watchdog application preceded by ‘sudo’ if you are not ‘root’.

sudo apt install watchdog

Once installed, open the configuration file which allows you to perform many periodic tests.

nano /etc/watchdog.conf

Here, we will limit ourselves to testing the ethernet connection to the outside to be sure of always being able to remotely access your Orange PI or Raspberry Pi which runs ‘Remote SDR’. You have to define some parameters.

interval = 10

It is a measurement interval in seconds which must be less than the hardware time_out of your system (generally 15 s). We are not going to set values that are too low either, it is not necessary to test the network every second.

We will measure several times before declaring the system down. Here 360s

retry-timeout = 360

As a measure, we will test 2 ip addresses. For example the google dns and the address of the box internally. You have to choose addresses that are likely to operate permanently.

ping = 8.8.8.8
ping = 192.168.0.254

It is necessary to explain how to activate the system watchdog.

watchdog-device = /dev/watchdog

It is necessary to define if the ethernet connection is made by the wired network in general ‘eth0’. If you use wifi, it’s usually ‘wlan0’. ifconfig gives the details of the connections.

interface = eth0

Save the file with a CTRL-X

Start the service

sudo service watchdog start

Test the service

sudo service watchdog status

All you have to do is cut the Ethernet link for 10 minutes and see how the system reacts.