With a system (Remote SDR) running remotely 24 hours a day, we are not immune to a system crash and especially a loss of remote control. To remedy this, it is possible to set up a “Watchdog” which constantly tests the connection to the Internet and restarts the system in the event of no response after a certain time.
The Linux kernel watchdog is used to monitor if a system is running. It is supposed to automatically restart systems suspended due to unrecoverable software errors. The monitoring module is specific to the hardware or processor used.
To test the possibility of using this functionality on your system, open a terminal window in root mode and type:
echo 1 > /dev/watchdog
Your system will reboot after 15 s in general. It works on Raspberry Pi 4, Orange Pi etc…All you have to do is write a piece of code that every 5 s for example will execute this same command which restarts the 15 s counter and avoids the reboot unless it stops.
In the case below, we will test the presence of the ethernet network and restart the watchdog only if it is present. So in case of blocking of the WIFI no longer allowing to connect remotely, the system will reboot. For this, we will install the ping3 library with:
pip3 install ping3
Below is a small code in python3 which will excite the watchdog every 5s unless there is no internet.
from ping3 import ping
adr_start = "/home/Last_boot.txt"
resp = ping(host)
if resp == False:
m="Last Boot : " +time.asctime()
delay_watchdog = 2*periode_to_test_internet #increase delay at start
print ("Press CTRL+C to exit")
delay_watchdog = periode_to_test_internet
ok1 = myping("www.google.com")
ok2 = myping("192.168.0.254")
if ok1 == True or ok2 == True :
internet_ok = True
if internet_ok and delay_watchdog < periode_test_internet :
os.system("echo 1 > /dev/watchdog") # Excite the watchdog
print (" Bye.")
The internet test is not done immediately so as not to overload the system, and to give time to regain control if your program crashes, as this can generate an infernal loop where your system keeps rebooting. At each reboot, we write the time in a file to have the date of the last start.
Installation of the “Watchdog” application
An alternative solution is to install a software that monitors the functionality of your single board computer such as the presence of the ethernet network. It will launch a reboot process if necessary. On an Orange Pi or a Raspberry Pi, open a terminal using ssh and download the watchdog application preceded by ‘sudo’ if you are not ‘root’.
sudo apt install watchdog
Once installed, open the configuration file which allows you to perform many periodic tests.
Here, we will limit ourselves to testing the ethernet connection to the outside to be sure of always being able to remotely access your Orange PI or Raspberry Pi which runs ‘Remote SDR’. You have to define some parameters.
interval = 10
It is a measurement interval in seconds which must be less than the hardware time_out of your system (generally 15 s). We are not going to set values that are too low either, it is not necessary to test the network every second.
We will measure several times before declaring the system down. Here 360s
retry-timeout = 360
As a measure, we will test 2 ip addresses. For example the google dns and the address of the box internally. You have to choose addresses that are likely to operate permanently.
ping = 18.104.22.168
ping = 192.168.0.254
It is necessary to explain how to activate the system watchdog.
watchdog-device = /dev/watchdog
It is necessary to define if the ethernet connection is made by the wired network in general ‘eth0’. If you use wifi, it’s usually ‘wlan0’. ifconfig gives the details of the connections.
interface = eth0
Save the file with a CTRL-X
Start the service
sudo service watchdog start
Test the service
sudo service watchdog status
All you have to do is cut the Ethernet link for 10 minutes and see how the system reacts.