Troubleshooting high server loads on Linux servers

Technical support analysts often receive tickets about high server loads. The cause of high server loads is very rarely attributed to defects in the cPanel software or the applications it installs. High server loads are something that should be initially investigated by the server owner, their system administrator, or server provider.

What causes high server loads?

Excessive usage of any of the following items can typically cause this issue:

• CPU
• memory (including swap)
• disk I/O

How can I check these items?

That depends whether you want to review their current resource usage, or historical resource usage. This tutorial will cover both.

A brief lesson on "sar"

Historical resource usage can be viewed using the "sar" utility, which should exist by default on all cPanel servers from the sysstat package. The stats are collected when sysstat runs from cron (/etc/cron.d/sysstat). If crond is not running, sysstat will not be able to collect historical statistics.

To view resource usage histories from sar, you must provide the path to the file that corresponds with the date of the stats.

For example, if you wanted to view the load averages for your server from the 23rd of the month, you would run this command:

Code:

[user@host ~]$ sar -q -f /var/log/sa/sa23

The command above uses '-q' to obtain the load average information, and '-f' to specify which sar file to obtain the information from. Note that sar may not have historical data going back more than a week or so.

You do not need to specify the date when viewing the statistics for the current day. As such, this command would show the load average for today:

Code:

[user@host ~]$ sar -q

You are strongly encouraged to read the documentation for sar:

Code:

[user@host ~]$ man sar

It provides statistics for many things that can be helpful to know about.


Current CPU usage

Run "top", and on the line that says "Cpu(s)", check the "%id" section which shows the percentage of which your CPUs are idle. The higher the number the better. A 99% idle CPU is not doing much of anything, and a 1% idle CPU is heavily tasked.

Code:

[user@host ~]$ top c

Tip: hit "P" to sort by processes that are currently consuming the most CPU.

Historical CPU usage

Check the "%idle" column:

Code:

[user@host ~]$ sar -p

Current memory usage

Code:

[user@host ~]$ free -m

Tip: run "top c" and hit "M" to see which processes are consuming the most memory.

Historical memory usage

This depends on the version of sar, which used to use '-r' to show %memused and %swpused (swap memory used), but later changed to '-S' to show %swpused.

Check "%memused" and "%swpused":

Code:

[user@host ~]$ sar -r

OR:

Code:

[user@host ~]$ sar -r

Code:

[user@host ~]$ sar -S

A note about memory usage: it is normal to see much of the server's memory being used. Why? Because the OS loves to cache things in memory. Why? Because accessing data from memory is extremely fast and far more efficient than using the server's disk(s).

As such, %memused isn't generally going to be much of an issue (unless perhaps you don't have a swap partition, but that's an issue in and of itself). You should focus on %swpused, which is what gets used when your server's physical memory is full. The lower the number, the better. A %swpused percentage of 0% would mean that your server currently has sufficient physical memory to perform its tasks.

How much %swpused is too much? That depends on your opinion of "too much". Generally speaking, a consistent low percentage of swap usage may not be an issue on your server. If you observe the %swpused increasing over time (e.g., from 1%, to 7%, to 32%), something on your server is consuming too much memory, and it would be wise to determine what that is (rather than just installing more memory). If your server ends up using all of its physical memory and swap memory, it may become unresponsive, requiring a reboot.

Current disk I/O usage

Note: this does not work on OpenVZ/Virtuozzo containers.

This will print the disk usage statistics 10 times, every 1 seconds. Check the %util column:

Code:

[user@host ~]$ iostat -x 1 10

Historial disk I/O usage

Code:

[user@host ~]$ sar -d

Good system administration involves knowing when your server's load is higher than acceptable. The main reason for this (other than preventing your server from becoming unresponsive and requiring a reboot) is tosee what's taking place on the server while the load is high. Fast actions will enable you to troubleshoot the issue while it is occurring.

If your server's load was high from 2AM - 4AM while you were sleeping, you would have missed what took place. While sar can be helpful to show you what specific resources were high during that time, it won't tell you the cause of the high usage. There can be many causes, including DoS attacks, spam attacks, poorly designed php scripts which consume large amounts of memory, web spiders that crawl sites too aggressively, hardware issues, massive amounts of disk writes to a user's MySQL database, and much, much more.

The good news is that you can have much of this information collected and sent to you automatically while the load is high, which you can review later as needed. How? From your process list:

Code:

[user@host ~]$ ps auxwwwf

Read more »

Free Website Monitoring with SMS and Tweet alerts

The one big advantage with Google Docs is that it can be configured to check your website status every minute and thus you’ll instantly know if your site is down. And it’s free.

The Google Docs sheet will monitor your website in the background at all times and will send an email alert whenever it has trouble accessing the website. If the issue is resolved, you’ll get another notification saying “all’s well.” If you are a tech genius you can also make a twitter account to alert you which will enable you to monitor the downtime history.

If you want to configure this for your server you can do it in 4 easy steps.
You can find the steps here.

Read more »

Script to Check the load of webserver and Restart Services

The script below is done in Bash, which can be used across all Linux platforms. The function of the script is to check the web-server load and restart the normal services which reduces the load. It restarts apache and mysql. You can custom this to restart your services.

1. Navigate to /root.
$ cd /root

2. Create a new file
$ vim load.sh
or
$ vi load.sh

3. Insert the following lines to the file
-----------------------------------
LOAD=`uptime|awk '{print $10}'|cut -d, -f1 |cut -d. -f1`
if [ $LOAD -ge 50 ]
then
killall -9 mysqld
killall -9 mysqld
killall -9 mysqld
killall -9 httpd
killall -9 httpd
killall -9 httpd
for SEMID in `ipcs -s httpd| awk '{print $2}'| grep -iv sem`; do ipcrm -s $SEMID; done
/etc/init.d/httpd start
sleep 20
/etc/init.d/mysqld start
echo high load $LOAD at `date`, mysql/apache restarted >> /usr/monitor/mysql.log
fi
-----------------------------------

4.  Give execute permission.
$ chmod 755  load.sh

5. Set a cron to check it at regular intervals.
$ crontab -e
*/5 * * * * sh -x load.sh
(This will check every 5 minutes)

Read more »