Dec 152012
 
Article Server Administration

On a production server, the disk usage tends to grow due to the accumulation of historical data in log files, data stored in the database, etc.
The system administrator normally establishes procedures to compress files and backup old data, intended to keep a reasonable amount of disk space free (in most cases, a minimum of 10% disk free is chosen).

But sometimes, the disk usage may grow unexpectedly of other reasons, such as a failure in a mail server that causes outgoing messages to accumulate in the output queue, or a recurring error in the service that makes a large number of error messages being written to the log files, etc.

In this post we will see how to configure an alert that checks periodically the disk usage, and sends an email if one of the filesystems surveyed has a disk usage greater than a configured threshold.

Unix commands to analyze disk usage

df

On a unix/linux system, the “df” (disk free) command output the status of each of the filesystems in the server. It is normally given the “-h” switch to get the information in a human readable form.

In the example above, we can see that the “/” (root) filesystem is 93% full, and the “/c” filesystem is also at 90% of its capacity. The system administrator should analyze the reason for this excessive occupation, and take the required measures to get those filesystems back to a minimum 10% free space.

du

The “du” (disk usage) command outputs the space occupied by single files or whole directory trees. The “-s” (summary) and “-h” (human readable) switches are normally used with this command to get an output easier to read. For instance:

Perl modules to retrieve the disk usage

Filesys::Df, Filesys::DfPortable

When it comes to write a system administration script, it is generally much easier to do it in Perl, rather than in a shell scripting language such as sh, csh or bash. The CPAN library includes some modules, such as “Filesys::Df”, that greatly simplify the development of the script.

Note: Windows systems administrators can also find in CPAN the “Filesys::DfPortable” module, that implements for Windows O.S. the same functionality as “Filesys::Df” does for unix/linux.

A basis perl script can be written to have a look at the information retrieved by this module:

And the result we get is:

We can see that the “df” function returns plenty of information about the filesystem. The data that might be most relevant to out monitoring script are:

  • “per” – percent used disk space
  • “fper” – percent used i-nodes

We can learn the meaning of the other data in the  reference documentation for the module in CPAN.

 

Filesys::DiskUsage

The Filesys::DiskUsage module from CPAN implements a “du” function that returns the total number of bytes used by files and directory trees.

Using this function, we can write a script to monitor disk usage at a finer granularity:

The user running this script must be granted read access to the directories and subdirectories being monitored. If the script is executed by a user without these privileges, we get the following result:

Event if the du() function returns a total number of bytes, the space used by the subdirectories for which access is denied is not computed in that total.

Disk usage monitor script

To finish the script, we can add a parameter “min_free_percent” to specify the threshold disk free that triggers an alert if the actual disk free falls below it.

In the loop that check the filesystems, the code to check this value and print a warning is added:

When the script is run, it outputs:

Automating the execution of the script

Once it is working, we need to automate the periodic execution of the script. On a linux system, we do this by adding an entry to the crontab. For instance, to run it daily at 08:01:

For this to work as expected, the crond daemon must be running on the server.

Besides, crond sends an email to the local user with the output of the script. The mail system must be adequately configure to forward these emails to the intended recipient.

 

 Posted by at 1:26 pm

 Leave a Reply

(required)

(required)