Saturday 9th November 2024, just after midnight the server was taken down for some routine work. The work in question was a minor tweak to one of the configuration files. The work took less time than I expected, in fact I had not even gotten more than ¼ of the way though my cup of coffee. Happy with the changes the server was re-started, and that’s where the world ended! The server start up flagged two errors, one was an expired security certificate, the other was an error we have seen before and was related to the dashboard system. Of course, the first steps were to roll the change I had done back. This was a simple “Delete the file with the change, and restore the backup taken before any work is done”. It was the certificate that we were concerned about. Undoing the changes made no difference; the system was still failing to start with the same errors as above. Clearly it was not the work undertaken at the start of the evening.

At this point I contacted Mark for both moral support and to share the pain the server was causing. Also, two minds always better than one.

As the hours passed a lot of adjustments and testing was carried out to confirm the security certificate issue was not local to our server. The server is part of a very large complex network that allows not only TG950 but the Dial on Demand system. However, because it is connected to other servers it primarily makes contact with a controller server within the inner workings of the FreeDMR system. It was the certificate on this ‘up stream’ server that had expired. At 03:05 there are not many people about. I did reach out to the international keeper community and took advise from a couple of stations to try and fashion a work around. This sadly could not bypass the problem. It was suggested the planned path of contacting the FreeDMR team once the day had started for normal people.

At 09:30 I reached out to the team, but with it being a weekend and people have real lives away from radio, it did take a while to get a response.

The certificate problem could not be resolved until later in the weekend. All we could do at this point was to message the SWC keepers and update them. Our Facebook page was also updated to reflect the current situation.

Sadly, once the certificate issue was resolved the system would still not start. There was an internal IP clash within the Docker on our server that was causing the system to fail to start. We have seen this before, but none of our normal fixes would work. It required for us to reach out once again to the system programmer. In typical speed they identified the issue and resolved it in what felt like 20 seconds but it was quite a few minutes in reality. However once the problem had been identified and the fix applied our sever booted back to life, and at this point for what felt like the whole of the weekend I could breath again.

What was amazing, even before we could send a message to the keepers to advise the system was back up; people were showing on the dashboard, there was even a QSO!

Sorry everyone for the total loss of the SWC. Thank you all for not ear bending Mark or I. However, seriously if you feel you would like to help support the server then please see the notices on the dashboard page for details how you can donate.