Stackvora.tech

Monitor SMART and Predictive Failures

Reliable storage is foundational to any IT infrastructure. Hard disk drives (HDDs) and Solid State Drives (SSDs) are mechanical and electronic devices,…

Reliable storage is foundational to any IT infrastructure. Hard disk drives (HDDs) and Solid State Drives (SSDs) are mechanical and electronic devices, respectively, with finite lifespans and predictable failure modes. Proactive monitoring of Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) attributes can provide early warnings of impending drive failures, allowing for timely replacement and preventing data loss or unexpected downtime. This guide details how to implement robust S.M.A.R.T. monitoring on Linux systems, focusing on common tools and best practices for integrating with enterprise monitoring solutions.

Understanding S.M.A.R.T. Attributes

S.M.A.R.T. is a monitoring system included in computer hard disk drives (HDDs), solid-state drives (SSDs), and eMMC drives. Its primary function is to detect and report various indicators of drive reliability, with the intent of anticipating hardware failures. While attribute interpretation can vary slightly between manufacturers (e.g., Western Digital vs. Seagate), common critical attributes include:

  • Reallocated_Sector_Ct (Attribute ID 5): The count of reallocated sectors. When a drive encounters a read/write/verification error, it attempts to move the data from the problematic sector to a reserved "spare" area. An increasing value here is a strong indicator of degradation.
  • Current_Pending_Sector_Ct (Attribute ID 197): The count of "unstable" sectors waiting to be reallocated. These sectors have read errors but haven't been successfully reallocated yet. A non-zero value often precedes an increase in Reallocated_Sector_Ct.
  • Offline_Uncorrectable_Ct (Attribute ID 198): The count of uncorrectable errors encountered during the drive's offline scan.
  • UDMA_CRC_Error_Count (Attribute ID 199): The count of errors during data transfer via the interface cable. Often indicates a faulty cable or connection issues, not necessarily the drive itself.
  • Power_On_Hours (Attribute ID 9): Total hours the drive has been powered on. Useful for tracking operational age.
  • Temperature_Celsius (Attribute ID 194): Current drive temperature. High temperatures can accelerate degradation.

For SSDs, attributes like Media_Wear_Indicator (ID 177), SSD_Life_Left (ID 231), and Wear_Leveling_Count (ID 173) are crucial for monitoring flash cell degradation.

Accessing S.M.A.R.T. Data with smartctl

The primary tool for interacting with S.M.A.R.T. on Linux is smartctl, part of the smartmontools package. To install it on Debian/Ubuntu:

sudo apt update
sudo apt install smartmontools

On RHEL/CentOS:

sudo yum install smartmontools

To query a drive's S.M.A.R.T. status, use the -a flag for all attributes:

sudo smartctl -a /dev/sda

Replace /dev/sda with the appropriate device identifier. For NVMe drives, the syntax changes slightly:

sudo smartctl -a /dev/nvme0n1

This command outputs a wealth of information, including drive model, serial number, firmware version, and the S.M.A.R.T. attribute table. Pay close attention to the "RAW_VALUE" and "WORST" columns for the critical attributes mentioned above.

S.M.A.R.T. Behind RAID Controllers

Direct access to S.M.A.R.T. data can be challenging when drives are behind a hardware RAID controller. Many controllers abstract the physical drives, presenting a logical volume to the OS. smartctl often requires a special device type flag (-d) to communicate with the controller and then address individual physical drives. Common controller types and their flags include:

  • LSI/Broadcom MegaRAID (perccli, storcli): -d megaraid,N where N is the adapter number (usually 0) followed by the enclosure and slot ID. E.g., sudo smartctl -a -d megaraid,0 /dev/sda. For specific drives, it's often -d sat+megaraid,N or -d scsi,N. You might need to use storcli /c0 show all to map logical disks to physical drive IDs.
  • HP Smart Array (ssacli, hpssacli): -d cciss,N for older controllers or -d sat+hp,N for newer ones. The N represents the drive index within the array.
  • Dell PERC (based on LSI): Similar to MegaRAID.
  • Areca RAID: -d areca,N.

The exact syntax can be arcane and requires consulting the smartctl man page (man smartctl) or smartmontools documentation for your specific controller and drive type. Often, it's easier to use the controller's own CLI tools (e.g., perccli, storcli, ssacli) to query drive status, then parse their output. For example, with storcli:

# List controllers
sudo storcli /c0 show all

# Show physical drive details on controller 0, enclosure 252, slot 0
sudo storcli /c0/e252/s0 show all

This output will usually include critical health indicators that mirror S.M.A.R.T. data.

Automating S.M.A.R.T. Checks with Cron

A common approach is to schedule daily S.M.A.R.T. checks using cron. This involves a simple script that iterates through drives and checks for critical attribute changes. Create a script, for example, /usr/local/bin/check_smart.sh:

#!/bin/bash

# Define critical S.M.A.R.T. attributes and their thresholds
# Attribute ID: (Threshold, Direction - 'inc' for increasing, 'dec' for decreasing)
declare -A SMART_CRITICAL_ATTRIBUTES
SMART_CRITICAL_ATTRIBUTES[5]="1,inc"  # Reallocated_Sector_Ct > 0
SMART_CRITICAL_ATTRIBUTES[197]="1,inc" # Current_Pending_Sector_Ct > 0
SMART_CRITICAL_ATTRIBUTES[198]="1,inc" # Offline_Uncorrectable_Ct > 0
SMART_CRITICAL_ATTRIBUTES[199]="1,inc" # UDMA_CRC_Error_Count > 0 (often cable, but worth alerting)
SMART_CRITICAL_ATTRIBUTES[231]="10,dec" # SSD_Life_Left < 10 (for some SSDs)

# List of devices to monitor
# For RAID controllers, you might need specific -d flags.
# Example: DEVICES="/dev/sda /dev/sdb"
# Example with MegaRAID: DEVICES="/dev/sda -d megaraid,0 /dev/sdb -d megaraid,0"
# Note: For megaraid, you usually only need to specify the adapter, smartctl will iterate drives.
# A simpler approach for RAID is to use the controller's utility and parse it.
# For simplicity, let's assume direct access or the controller passes through.
DEVICES=("/dev/sda" "/dev/sdb" "/dev/sdc") # Adjust as needed for your system

# Email for alerts
ALERT_EMAIL="your_email@example.com"
HOSTNAME=$(hostname -f)
ALERT_MESSAGE=""

for DEVICE_PATH in "${DEVICES[@]}"; do
    # Handle -d flag if present in DEVICES list
    if [[ "$DEVICE_PATH" =~ " -d " ]]; then
        DEVICE=$(echo "$DEVICE_PATH" | awk '{print $1}')
        SMART_CMD="sudo smartctl -a $DEVICE_PATH"
    else
        DEVICE="$DEVICE_PATH"
        SMART_CMD="sudo smartctl -a $DEVICE"
    fi

    OUTPUT=$($SMART_CMD)
    EXIT_CODE=$?

    if [ $EXIT_CODE -ne 0 ]; then
        if echo "$OUTPUT" | grep -q "Smartctl open device: UNKNOWN or missing argument to -d"; then
            ALERT_MESSAGE+="WARNING: S.M.A.T. not supported or failed to open $DEVICE_PATH on $HOSTNAME (perhaps missing -d flag or controller issue).\n"
        else
            ALERT_MESSAGE+="ERROR: smartctl failed for $DEVICE_PATH on $HOSTNAME (Exit Code: $EXIT_CODE).\n$OUTPUT\n"
        fi
        continue
    fi

    # Check overall SMART status (optional but good)
    if echo "$OUTPUT" | grep -q "SMART overall-health self-assessment test result: FAILED"; then
        ALERT_MESSAGE+="CRITICAL: SMART overall health FAILED for $DEVICE_PATH on $HOSTNAME!\n"
    fi

    # Parse attributes
    while IFS= read -r line; do
        if [[ $line =~ ^[[:space:]]*([0-9]+)[[:space:]]+([^[:space:]]+)[[:space:]]+0x[0-9a-fA-F]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+([0-9]+)$ ]]; then
            ATTR_ID=${BASH_REMATCH[1]}
            ATTR_NAME=${BASH_REMATCH[2]}
            RAW_VALUE=${BASH_REMATCH[3]}

            if [[ -v SMART_CRITICAL_ATTRIBUTES[$ATTR_ID] ]]; then
                THRESHOLD_INFO=${SMART_CRITICAL_ATTRIBUTES[$ATTR_ID]}
                THRESHOLD=$(echo "$THRESHOLD_INFO" | cut -d',' -f1)
                DIRECTION=$(echo "$THRESHOLD_INFO" | cut -d',' -f2)

                SHOULD_ALERT=false
                if [[ "$DIRECTION" == "inc" && "$RAW_VALUE" -ge "$THRESHOLD" ]]; then
                    SHOULD_ALERT=true
                elif [[ "$DIRECTION" == "dec" && "$RAW_VALUE" -le "$THRESHOLD" ]]; then
                    SHOULD_ALERT=true
                fi

                if $SHOULD_ALERT; then
                    ALERT_MESSAGE+="CRITICAL: $ATTR_NAME (ID $ATTR_ID) on $DEVICE_PATH has RAW_VALUE $RAW_VALUE (Threshold: $THRESHOLD $DIRECTION) on $HOSTNAME.\n"
                fi
            fi
        fi
    done <<< "$OUTPUT"
done

if [ -n "$ALERT_MESSAGE" ]; then
    echo -e "S.M.A.R.T. Alert from $HOSTNAME\n\n$ALERT_MESSAGE" | mail -s "S.M.A.R.T. Alert: $HOSTNAME" "$ALERT_EMAIL"
fi

Make the script executable: sudo chmod +x /usr/local/bin/check_smart.sh.

Then, add it to cron. For daily execution, add to /etc/cron.daily/check_smart or use sudo crontab -e for the root user:

0 3 * * * /usr/local/bin/check_smart.sh > /dev/null 2>&1

This runs the script daily at 3:00 AM. Ensure your system has mailutils or a similar package installed and configured for email sending.

Integrating with Enterprise Monitoring Systems

For more robust monitoring, integrate S.M.A.R.T. checks into systems like Nagios, Prometheus, Zabbix, or Splunk. This allows for centralized alerting, historical trending, and dashboard visualization.

Nagios/Icinga

Create a custom Nagios plugin. The check_smart.sh script can be modified to output Nagios-compatible exit codes and text. Exit codes 0 (OK), 1 (WARNING), 2 (CRITICAL), 3 (UNKNOWN).

# Example output for a Nagios plugin
echo "DISK_SMART_STATUS OK - All drives healthy."
exit 0

# If an issue is found:
echo "DISK_SMART_STATUS CRITICAL - /dev/sda Reallocated_Sector_Ct is 5!"
exit 2

Then, configure your NRPE agent on the monitored host to execute this script, and define a service check in Nagios.

Prometheus

Use the node_exporter with the smartctl textfile collector. This collector will execute smartctl and expose the metrics in a format Prometheus can scrape. First, enable the textfile collector in node_exporter's configuration. Then, create a script (e.g., /etc/node_exporter/smartctl_metrics.sh) that runs smartctl and formats the output into Prometheus exposition format:

#!/bin/bash

# This script would be run periodically by cron or systemd timer
# and write its output to a file that node_exporter reads.

OUTPUT_FILE="/var/lib/node_exporter/textfile_collector/smart.prom"
TEMP_FILE=$(mktemp)

echo "# HELP smartctl_drive_health_status S.M.A.R.T. overall health status (0=OK, 1=FAIL)" > "$TEMP_FILE"
echo "# TYPE smartctl_drive_health_status gauge" >> "$TEMP_FILE"

# List your drives, handling RAID controllers as needed
# For direct-attached, this usually works
for DRIVE in "/dev/sda" "/dev/sdb"; do
    if sudo smartctl -H "$DRIVE" | grep -q "PASSED"; then
        HEALTH_STATUS=0
    else
        HEALTH_STATUS=1
    fi
    echo "smartctl_drive_health_status{device=\"$(basename "$DRIVE")\"} $HEALTH_STATUS" >> "$TEMP_FILE"

    # You can parse more specific attributes here if needed,
    # e.g., smartctl -A /dev/sda to get raw values for specific attributes.
    # Be careful not to expose too many metrics if not strictly needed.
done

mv "$TEMP_FILE" "$OUTPUT_FILE"

Schedule this script to run every 5-10 minutes via cron or systemd timer. Prometheus will then scrape these metrics from node_exporter, and you can create alerting rules based on smartctl_drive_health_status or specific attribute values.

Best Practices and Troubleshooting

  • Replace at first warning: The existing stub's advice is paramount. Once a drive shows reallocated sectors or pending sectors, its reliability is compromised. Rebuilding a RAID array puts significant stress on remaining drives; a second marginal drive failure during a rebuild can lead to total array failure and data loss. Replace proactively.
  • Keep smartmontools updated: New drives and controllers are constantly released. Updated smartmontools versions (especially smartctl) often include support for new hardware and attribute interpretations.
  • Consider manufacturer tools: Some drive manufacturers provide their own diagnostic tools (e.g., SeaTools for Seagate, Data Lifeguard Diagnostic for Western Digital). These can sometimes provide more in-depth analysis than generic smartctl.
  • Monitor RAID controller logs: In addition to S.M.A.R.T., always monitor your RAID controller's event logs (e.g., via dmesg, /var/log/syslog, or the controller's CLI) for predictive failure warnings or array degradation events.
  • S.M.A.R.T. isn't foolproof: While highly effective, S.M.A.R.T. doesn't predict all failures. Some drives can fail catastrophically without prior warning. Regular backups remain essential.
  • NVMe Drives: For NVMe, smartctl leverages the NVMe-specific health information log. Key attributes to watch include "Critical Warning", "Media and Data Integrity Errors", "Percentage Used", and "Data Units Read/Written".

Common pitfalls:

  • Permissions: smartctl usually requires root privileges. Ensure your cron jobs or monitoring agents run with appropriate permissions (e.g., using sudo or running as root's cron).
  • Incorrect device paths or -d flags: Misidentifying drives or using the wrong controller-specific flags will result in "Device open failed" or "SMART support is not available" errors. Always verify with sudo smartctl -a /dev/sdX first.
  • RAID controller masking: Some RAID controllers completely mask the underlying drives, making S.M.A.R.T. monitoring via smartctl impossible or unreliable. In such cases, rely on the controller's dedicated CLI tools and integrate their output into your monitoring system.
  • Excessive alerting: Start with critical attributes and specific thresholds. Too many alerts (e.g., for minor temperature fluctuations) can lead to alert fatigue.

Back to the knowledge base · Ask the AI assistant