1000² Server

Intro

In my initial thoughts about the smart home system architecture, the server was most likely a raspberry pi:

At that time it seemed the PI was apt for it. And a raspberry pi 2 served me well for a while, and then I moved to a pi 3 and everything was good.

Until it was not enough…

What happened? I had mosquitto and node-red as the main things running. And I added some simple data logging in node-red. Just simple save 1 month of each sensor in a text file every some time. But I wanted a proper database, something that could keep the data for a bit of time. And something for plotting things better, because node-red is quite lacking even at hundreds of data points only.

So I went for the cool team: Grafana and influxdb. Problem was… the Pi 3 started shaking even as I reached barely weeks of data. I was pumping in all the sensors, some 30-40 nodes, but each with multiple streams. A database for the sensors (temperature, humidity, light, motion etc), one for the node health status (battery state, RSSI etc) and one for the server usage (CPU, RAM, temperature).

The raspberry pi 3 is pedestrian in terms of disk access speeds

I mean like 2000 era PCs. So, two things were happening:

1. influx wanted to load databases in the order of 100s MB and running out of RAM when trying to do the continuous queries to down sample the data. That is because I started building the queries a few weeks into the data. So continuous queries never worked for me on the pi 3. (In retrospect, probably should not have pushed all the data of CPU usage and RSSI data points from node red).
2. influx wanted a long time to load databases in the order of 100s MB when I started to request enough data from such a database.

Both of these were caused by insufficient RAM and slow disk access. Ok, so I thought, I can at least make the swapping faster with a SSD? I gave it a try and the resulting server went from a nice closed box to a spread apart contraption:Did it help? Well a bit. Turns out that the limits imposed by the USB 2.0 on the pi don’t make it that fast on an SSD either.

NUC

In a moment of despair, I did the natural thing: I hooked the central node up to the living room computer, a BOXNUC6CAYH (celeron quad core, 4 GB RAM and 120GB SSD SanDisk Plus, Ubuntu 16). Thanks to most of software being compatible with multiple OS’s and having designed the central node to talk over a USB-serial adapter the moving took little time. I later upgraded it to 8 GB of RAM and now it is happily doing both the Home automation tasks and being a living room computer for youtube, movies etc. I put the central node in a separate box:

Afterwards influx managed to run its queries and requesting data was a breeze. It’s not a speed daemon when I ask it for 2 months of plots from 30 sensors at the same, but it does work.

What about the pi 4?

With improved USB 3, proper Gig Ethernet and “desktop replacement” I actually gave a go to the new pi 4 with 4 GB of RAM. In fact, I ran the very useful and nice IOTStack with automatically gets you up and running with plenty of home automation software and runs everything in containers and features automatic backup. Awesome. A good video about it here.

The test consisted only in importing my database in influx and pointing grafana from the NUC at the rpi 4. The results were much better than the pi 3, i could request more data and it did not seem to crash. It’s quite obvious the extra RAM helps, as influx quickly gobbled close to 1.7 GB just for itself. But requesting data is significantly slower than the NUC, albeit usable if only a few charts are displayed in your dashboard at the same time.

So, the pi 4 with 4GB is where I would start at the minimum if you want to deal with databases. Still, I have no idea how it will behave with even larger databases (I don’t have more data yet) or if external SSD will help (no USB boot at the moment). Still…

Cost

If you are starting from scratch, get a 4GB RAM pi 4, maybe a box, some cooler and heasink, the annoying extra micro HDMI adapter, a power supply, a micro SD card or even maybe an SSD you end up quite far away from the 35 Eur the pi foundation seems to cling to. You end up around 130-140 Euro. This is very close to the cost of my NUC, coming in around 180 Eur. And at this price range you can also consider refurbished laptops, other NUCs, refurbished thin clients, brand new mini PCs.

All these, coming around or maybe a bit above the fully equipped pi 4 have one cool thing in common: much faster disk access speeds and possibly more performance, plus a few other advantages. I think they are a much better deal in the long run.