Raspberry Pi cloud data logging on Azure with Fluent Bit
How to log and analyze your Raspberry Pi performance and security on the Azure cloud in a light and efficient way.
Introduction
Do you have a Raspberry Pi running, for example, Pi-hole, OpenVPN, or your own private cloud, and want to keep an eye on security and performance logs in a light and efficient way? Then look no further. This article is for you.
This article covers how to log Raspberry Pi data on Azure Log Analytics using the open-source log processor and forwarder tool Fluent Bit.
On-premises you will use Fluent Bit to collect and forward the logs from the Raspberry Pi to Azure. Fluent Bit, for those already familiar with Fluentd, is also developed by Treasure Data. It is a lightweight (40 kb) and a high-performing log processor and forwarder tool with a pluggable architecture. Also, as opposed to Fluentd, it doesn’t require Ruby to be installed.
I recommend reading this blog post for a more in-depth insight into the differences between Fluent Bit and Fluentd.
On the cloud, you will use Log Analytics in Azure Monitor to analyze your Raspberry Pi logs. Log Analytics is a cloud tool used to write and run log queries via the Kusto query language.
This article will only provide two examples of Kusto queries and will not go much into detail on Log Analytics in Azure Monitor. Thankfully the documentation provided by Microsoft is excellent to get up and running quickly.
You will need:
- A Raspberry Pi running with either Jessie, Stretch, or Buster;
- A Microsoft Azure account.
- Knowledge of basic commands for Linux;
- Familiarity with SQL and/or Kusto (or willingness to learn);
Topics covered:
- Installing Fluent Bit on the Raspberry Pi;
- Creating a Log Analytics workspace on Azure;
- Configuring Fluent Bit;
- Accessing and querying your Raspberry Pi logs in Logs Analytics.
1. Installing Fluent Bit
Fluent Bit is distributed as TD-Agent-bit. To install it, first, open a terminal session and add the Fluent Bit server GPG key to download the packages:
wget -qO — https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -Then, independently of which Raspbian distribution you are running, you need to add the Fluent Bit APT server entry for Raspbian 8 Jessie to your system sources list. Currently, the Jessie repository seems to be the only one containing an ARM-compatible package.
To do this, type:
sudo nano /etc/apt/sources.listAnd then add the following line to the bottom of the file:
deb https://packages.fluentbit.io/raspbian/jessie jessie mainPress CTRL + O and ENTER to save and then press CTRL + X and ENTER to exit nano.
Update your repositories database:
sudo apt-get updateInstall TD-Agent-bit:
sudo apt-get install td-agent-bitStart TD-Agent-bit and check if it is running correctly:
sudo service td-agent-bit start
sudo service td-agent-bit statusIf all went well and Fluent Bit is running as expected you should see something like this:
● td-agent-bit.service - TD Agent Bit
Loaded: loaded (/lib/systemd/system/td-agent-bit.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2020-02-24 21:07:57 CET; 23h ago
Main PID: 12390 (td-agent-bit)
Memory: 3.2M
CGroup: /system.slice/td-agent-bit.service
└─12390 /opt/td-agent-bit/bin/td-agent-bit -c /etc//td-agent-bit/td-agent-bit.confPress CTRL + C and ENTER to return to the terminal.
By default, TD-Agent-bit collects CPU metrics from the host machine and sends them to the standard output. You can access yours in the /var/log/syslog file.
2. Creating a Log Analytics workspace on Azure
If you already have a Log Analytics workspace skip this step and start configuring Fluent Bit.
If not, here is what you need to do to create one.
First, in the Azure portal, in the search bar, type Log Analytics and select Log Analytics workspaces.
Click on Add.
You will then be prompted to complete the form below. To complete it, provide a name for your new Log Analytics workspace, select a Subscription, a Resource Group (or create a new one), and a Location. Once you are done, then click OK.
Wait a couple of minutes and then check that the workspace has been created by accessing Notifications from the Azure Portal menu.
Great! You now have a Log Analytics workspace and can configure Fluent Bit to send logs from your Raspberry Pi to Azure.
3. Configuring Fluent Bit
Fluent Bit can be configured both via the command line and via its configuration file. This article opts for the latter.
Before proceeding with the configuration, I strongly recommend reading the official Fluent Bit documentation, and in particular, the Configuration chapter to learn about the schema of the configuration file.
As a first thing, before proceeding with modifying the Fluent Bit configuration file, you need to stop TD-Agent-bit. To do this, run:
sudo service td-agent-bit stopOnce it stopped, you can then edit the configuration file. In the terminal enter:
sudo nano /etc/td-agent-bit/td-agent-bit.confYou should see the contents of the default Fluent Bit configuration file. Feel free to modify it as you like/need with the input and output plugins described in the official documentation.
To configure the Azure output plugin, you need to input your Azure Customer ID and the associated Primary Key or Secondary Key.
[OUTPUT] Name azure
Match *
Customer_ID <ENTER YOUR OWN ID> Shared_Key <ENTER YOUR OWN KEY>You can find your keys in Azure under Log Analytics -> your Workspace Name -> Advanced Settings -> Connected Sources.
Note that it doesn’t matter whether you use the Windows Servers or the Linux Servers keys.
You can find my configuration file below if you want to have a look at an example. As you can see, I am logging data from the CPU (including temperature), the memory, the network connection, and the Journald daemon.
[SERVICE]
Flush 5
Daemon Off
Log_Level info Parsers_File parsers.conf Plugins_File plugins.conf
HTTP_Server Off HTTP_Listen 0.0.0.0 HTTP_Port 2020[INPUT]
Name cpu Tag cpu.local
Interval_Sec 1[INPUT] Name mem
Tag memory
[INPUT] Name netif Tag netif
Interval_Sec 1 Interval_Nsec 0 Interface eth0
[INPUT] Name thermal
Tag my_thermal[INPUT]
Name systemd Tag host.*
[OUTPUT] Name azure
Match *
Customer_ID <ENTER YOUR OWN ID> Shared_Key <ENTER YOUR OWN KEY>
Once you finished modifying the configuration file, save it by pressing CTRL + O and ENTER and then exit nano by pressing CTRL + X and ENTER.
Now start TD-Agent-bit and check its status by entering:
sudo service td-agent-bit start
sudo service td-agent-bit statusIf no errors are reported, you can then move forward with accessing and querying your Raspberry Pi logs in Log Analytics on Azure.
4. Accessing and querying your Raspberry Pi logs in Logs Analytics
To access your logs, go to your Log Analytics workspace and then on the side-bar on the left click on Logs.
On the right, you will see all the tables associated with that workspace. Scroll down until you see Custom Logs, expand it, and you will see the table containing your Raspberry Pi logs.
Now you can start querying your logs to obtain performance metrics, events, network metrics, etc. For this article, I will provide two query examples based on my Fluent Bit configuration:
- A security-related query;
- A performance-related query.
Security-Related Query
Since my Raspberry Pi is always connected to the internet and running OpenVPN, I am interested in checking for security reasons if there have been failed attempts to access the system.
One way to do this using Log Analytics is by querying the logs for “failed password” messages. To check if it works, SSH into your Pi and input the wrong password. Then go back to Azure and run the following query in Log Analytics.
Note that fluentbit_CL is the name of my table, yours might be different depending on your Fluent Bit configuration.
fluentbit_CL| project TimeGenerated, _HOSTNAME_s, Message| where isnotnull(Message) and Message contains "failed password"| order by TimeGenerated desc nulls last
If all is working correctly, you should see the “failed password” log.
As you can see, it’s quite insightful since we can get information on when the failed login attempt happened, from which IP and on which port.
Performance-Related Query
Another thing I am also interested in is keeping an eye on the CPU temperature to avoid overheating.
One way to keep track of your Raspberry Pi CPU temperature using Log Analytics is by querying the logs for the CPU temperature at any given time during the last 24 hours.
fluentbit_CL| project temp_d, TimeGenerated| where isnotnull(temp_d)
If it works properly, you should see your CPU temperature with the time at which the log was generated.
You can also check for the average CPU temperature during the last 24 hours by running:
fluentbit_CL| project temp_d| where isnotnull(temp_d)| summarize avg(temp_d)
This should return:
There are plenty of other queries that you can run to explore the logs and understand more of what happens within your Raspberry Pi.
You could also connect your Pi to Azure Sentinel, create alert rules to be notified of when a given event happens, and many other things.
Language-wise, Kusto is not hard to learn and use. If you know SQL, this cheat sheet is for you. Alternatively, if you’re new to query languages, you can consult this useful Jumpstart Guide to KQL.
Have fun exploring your logs.
Disclaimer: The opinions and views herein contained are solely mine and neither represent nor express those of my employer.
