Automated Server Failover for Remote Stations

 

Overview

Here's a cool trick for anyone with a remote station that wants to have an automated computer failover. This router setup will allow you to have multiple redundant computers on site and if one fails, your remote users will automatically be sent to the backup machine.  When the primary comes back online, users are sent back there.  Any failures can be emailed to you from the router, so you can troubleshoot.

What you need

2 computers - can be Windows, Linux (Raspberry Pi, etc) or Mac

Mikrotik Router - In my case I'm using the hAP ac3 - which goes for about $110 on Amazon

Step By Step Setup (using the Mikrotik Web Gui or Winbox) - this is the IP you give to your users

1. Add a Virtual Service IP

This is the “VIP” your LAN clients will always use (e.g. 192.168.1.100).

    A. Open IP → Addresses

    B. Click Add New

        •  Address: 192.168.1.100/24

        • Interface: bridge (your LAN bridge, sometimes called bridge1)

        • Add a comment: ex. Node-RED VIP

   C. Click OK




2. Create NAT Rules

You’ll make two port-forward rules: one for the primary server (e.g. Node-RED-A), one for the backup (disabled until needed). In this example, the primary server is at 192.168.1.10 and the backup is at 192.168.1.11.

    A. Go to IP → Firewall → NAT

    B. Click Add New

        • Chain: dstnat

        • Dst. Address: 192.168.1.100

        • Protocol: tcp

        • Dst. Port: 1880 (e.g. Node-RED port number)

        • Action: dst-nat → To Addresses: 192.168.1.10

        • Add a comment: e.g. Primary Node-RED

        • Click OK

     C. Add another NAT rule for your second server (e.g. Node-RED-B)

        • Chain: dstnat

        • Dst. Address: 192.168.1.100

        • Protocol: tcp

        • Dst. Port: 1880 (e.g. Node-RED port number)

        • Action: dst-nat → To Addresses: 192.168.1.11

        • Add a comment: e.g. Backup Node-RED

        • Click OK

3. Configure Netwatch - This tells MikroTik router to monitor the primary Node-RED                     (192.168.1.10) and switch NAT rules if it goes down. This will send your users to the backup     server.

    A. Go to Tools → Netwatch

    B. Click Add New

        • Host: 192.168.1.10

        • Interval: 00:00:10 (check every 10 seconds)

    C. In the Down tab (script to run if .10 is unreachable), paste (make sure these match your             comment labels from above):

        /ip firewall nat disable [find comment="Primary Node-RED"];

        /ip firewall nat enable [find comment="Backup Node-RED"];

    D. In the Up tab (script to run when .10 comes back), paste:

        /ip firewall nat enable [find comment="Primary Node-RED"];

        /ip firewall nat disable [find comment="Backup Node-RED"];

    E. Click OK

4. Add Hairpin NAT - This allows clients that are in the same subnet as the servers to route correctly, you’ll need a srcnat masquerade to handle “hairpin NAT” (required when client and server are on the same network and traffic goes through the router).

    A. Open IP → Firewall → NAT

    B. Click Add New

    C. On the General section:

        • Chain: srcnat

        • Src. Address: 192.168.1.0/24

        • Dst. Address: 192.168.1.0/24

    D. Switch to the Action section:

        • Action: masquerade

        • Add a Comment: e.g. Hairpin NAT for Node-RED

     E. Click OK

5. From another computer on the router, go to 192.168.1.100:1880 and then plug and unplug your primary and backup servers from the router to test the failover. You can also see it enable and disable routing rules on the router screen by going to IP → Firewall → NAT.

Setting Up Email Alerts (optional)

When a server goes down, you might want to be notified. The router also has the ability to send emails to you when a server fails.

1. (optional) Configure Email App Password Settings - Gmail (if this is your provider/you have 2-Step Verification on)

          • Go to your Google Account (myaccount.google.com)

          • Select Secuirty (left hand column)

          • Under "How you sign in to Google" select 2-Step Verification

        

          • Scroll down to App passwords

          • At the bottom of the page, enter a name for the App password (e.g. Node-RED)

          • Click Create

          • A screen will not pop up saying - Your app password for your device - with a password

          • Copy this into a file as you will not ever see it again and will need to make a new one if              you lose it

          • Click Done

2. Configure Email Settings

    A. Go to: Tools → Email

    B. Enter in (for the example settings, we'll use Gmail):

        • Server: smtp.gmail.com

        • Port: 587

        • TLS: start tls

        • From: the email address you be sending from (e.g. dave@gmail.com)

        • User: your email address at Gmail (e.g. dave@gmail.com)

        • Password: the application password you set up above

        • Click Apply

3. Create a Netwatch Monitor

     A. Go to Tools → Netwatch

    B. Click Add New

        • (optional) Type: httpd-get (if you want to test for a specific service (e.g., Node-RED) -                 otherwise leave as default

        • Host: Enter the IP/hostname you want to monitor (e.g. 192.168.1.10 from above)

        • Interval: How often to check (e.g., 00:10:00 = every 10 minuntes)

        • Timeout: How long before it’s considered “down” in seconds (e.g., 30.00) 

        • (optional) Port: 1880 (if you want to test for a specific service (e.g., Node-RED)

        • In the Up second (script to run when .10 comes back), paste - adjust what's in quotes to              your preferences:

 /tool e-mail send to="dave@yahoo.com" subject=".10 Host UP" body="The host 192.168.1.10 is now reachable."

        • In the Down section (script to run if .10 is unreachable), paste - adjust what's in quotes              to your preferences:

/tool e-mail send to="dave@yahoo.com" subject=".10 Host Down" body="The host 192.168.1.10 is DOWN."

Do this for each server you want to monitor.

4. Test - plug and unplug servers and you should receive the corresponding up and down emails.

If you've made it this far, you now have a highly available server setup for Node-RED or any other important application.