Report 8 on Linux Course

March 26, 2020

Posted in:

System monitoring

I've been mostly using Virtual Box during this course with a Ubuntu 16.04 installation running on it. I recently started using Parallels and I find it's much more robust and I no longer have the same headaches with performance I used to have with Virtual Box. I also decided to test Linux Mint this time around instead of Ubuntu because a lot of people have recommended it and it's interesting to see another distro. I've only ever used Ubuntu/Xubuntu thus far.

Sysstat - Performance monitoring tool

This weeks exercises revolve around performance monitoring and log management. As a first thing I installed Sysstat https://github.com/sysstat/sysstat, which is a tool made for monitoring system's performance. I've personally never used any tools like these before on my server so it was interesting to see what this could do. I imagine this could come handy for linux server's with high loads of traffic and bandwith.

Installing was straightforward wit the command:

sudo apt sysstat

There are couple of commands I tried first which were 'sar -u' and 'sar -u 1 5'. The first one prints one line of statistics from the CPU usage and with the latter one you can check the CPU usage from a wider timeframe. As seen on the picture below. So this command is like activity monitor on Mac or other OS's with a GUI application for monitoring the system.

Screenshot 2020-03-27 at 0.59.43.png

Stress testing the system

Stress is a tool that can be installed with the command: 'sudo apt install stress'

It can be used to stress-test the system for a specified amount of time.

The tool can be called on the terminal with command 'stress', followed by different parameters. From what I gathered from documentation, for example giving the tool parameter --cpu 8, as following 'stress --cpu 8', specifies 8 workers going through sqrt(), which is of course the square root function and I imagine it's a good test for the CPU, there is also '-i N' -parameter spawns worker spinning a sync() -function and '-m N' spawns a working spinning on malloc()/free(). For those who've done even little bit of C -programming know that malloc is short for 'memory allocate', so this function allocates a specified amount of memory and free() frees it so this is stressing the memory of the machine.

I was unfamiliar with sync() -function and had to do some digging around in wikipedia (https://en.wikipedia.org/wiki/Sync_(Unix)) and the following "sync is a standard system call in the Unix operating system, which commits all data in the kernel filesystem to non-volatile storage buffers, i.e., data which has been scheduled for writing via low-level I/O system calls. Higher-level I/O layers such as stdio may maintain separate buffers of their own." - So this function seems to be related to low level operations on the filesystem. You don't run into these kind of things when coding with javascript or python.

Seeing how the stress -tool affects performance

On one terminal window I opened processor status with 'top' -command and from other window I ran the Stress tool for 3 rounds with increasing cpu usage

stress --cpu 2

Screenshot 2020-03-27 at 1.48.44.png

stress --cpu 4

Screenshot 2020-03-27 at 1.48.52.png

stress --cpu 8

Screenshot 2020-03-27 at 1.49.01.png

From the pictures we can see that the stress -tools amount of processes corresponds with activity monitoring, each individual worker can be differentiated from each other by the pid, which is the process id. My MacBook Pro's CPU (2,6 GHZ 6-core) got also really heated during this so there was work done on the background.

What was interesting here is how with 2 workers spinning, the CPU usage was the highest. My virtual environment is allocated only 2 cpu's from my Mac's hardware so what I think was here that both of the cores were at full load and every time I added more workers to the stress -tool it just allocated those between the cores so that the total load was at almost 100% per each of the 2 cores and 3 rounds of test.

I/O monitoring with iotop

Iotop is a monitoring tool that can be installed with the command sudo apt install iotop.

It can be used to monitor the I/O (Input/Output) operations. It can be ran with 'sudo iotop' -command and it shows a table of the ongoing processes.

Screenshot 2020-03-27 at 2.05.30.png

This was mostly uninteresting and this could be tested also with 'stress -c 2 -i 1 -m 1 --vm-bytes 128M -t 10s' -command for example which runs the stress tool for 10 seconds.

Then there is also command for iotop 'iotop -oa' which showed more interesting information. Pretty much every action I took on the Linux caused a new event to be logged in the iotop table and it was quite interesting seeing all the small things happening on the background. These seemed to be all I/O operations with even the smallest read or write operation causing it to be shown in the monitoring tool.

Dstat - versatile system monitoring tool

Next I installed dstat on my Linux Mint with sudo apt install dstat.

On default with just running command 'dstat' on terminal the tool looks like this:

Screenshot 2020-03-27 at 2.33.02.png

I needed more info on this particular tool so I ran the command 'man dstat' and it showed me a pretty lenghty list of different options / parameters that I could give to the tool. What was really interesting here that it also allowed to check some network traffic (tcpt and udp for example). This tool doesn't seem to offer that much info on the particular processes but shows a great realtime view of the active processes. I might imagine this tool being useful for a system administrator.

SS -command on linux terminal

Next command I tried was 'ss', this particular command seems to come as default on unix bash so I didn't need to install it from packet manager. This tool offers network monitoring.

The man page for ss shows us the following (there are a lot more parameters, but here's a few):

Screenshot 2020-03-27 at 2.41.54.png

What I found most interesting and also what was part of this report was the --listening option. Which shows all the ports that are currently listening.

The 'ss --listening' command can be given additional parameters to listen for TCP or UDP protocols by adding the parameters in the end, for example: 'ss --listening --tcp'.

Although I'm mostly focused on software/webapp development, for me personally all network administration related is really interesting and it's amazing to see this low level view of what's really going on in the background. This is also the kind of stuff that comes in handy when administrating production servers. It's of course a different thing running a Mickey Mouse operation such as this blog compared to something like Instagram and right now for me personally at least the Linux ecosystem is such a huge beast with hundreds of tools like these that's it's sometimes overwhelming feeling.

grep -i error /var/log/syslog; grep -ir error /var/log/ -command

As an exercise I apparently had to run this particular command and from what I can gather this particular command uses regular expressions to check for lines/events in the log files that have the term 'error' in it.

Here is a screenshot of this:

Screenshot 2020-03-27 at 2.53.45.png

Most errors I actually saw (there was quite a long list) were actually related to Parallels (my virtual enironment) and also had something to do with nameservers (DNS). The error messages were also quite obfustcated, there was hardly anything of interest I was able to gather from these.

Load averages

With the command uptime I was able to gather some info on load averages.

Screenshot 2020-03-27 at 2.59.35.png

The last 3 decimals from left to right are load averages from last minute, last 5 minutes and last 15 minutes.

On a dual-core machine such as the one I'm running now or even if it was single-core this would mean that 0.20 or 0.18 load average wouldn't be nothing to worry about and that the CPU is actually underutilized, the cpu's were mostly idle in this case.

There was some articles about this such as: https://community.tenable.com/s/article/What-is-CPU-Load-Average, but load averages for me are new concept in this context.

Load averages from a larger timeframe

I used 'sar -q' command to gather the load averages from past 4 hours or so and as this was a fresh installation of Linux Mint it of course did not show anything before that.

Screenshot 2020-03-27 at 3.07.36.png

The only really high number I ran in that table was 3.82 around the middle that was from stress testing the cpu with the 'stress' tool.

Return to blog