Dell PowerVault ME5 Log Collection (Save Logs, FTP / SFTP Download and Managed Logs)

Storage

Dell PowerVault ME5 Series Administrator's Guide

Managed logs

As the storage system operates, it records diagnostic data in several types of log files. The size of any log file is limited, so over time and during periods of high activity, these logs can fill up and begin overwriting their oldest data. Enabling the managed logs feature (Settings > System > Properties > Managed Logs Properties) allows log data to be transferred to a log-collection system, and store it for later retrieval before any data is lost. The log-collection system is a host computer that is designated to receive the log data transferred from the storage system. The transfer does not remove any data from the logs in the storage system. This feature is disabled by default.

The managed logs feature can be configured to operate in push mode or pull mode:

In push mode, when log data has accumulated to a significant size, the storage system sends notifications with attached log files via email to the log-collection system. The notification will specify the storage-system name, location, contact, and IP address, and will contain a single log segment in a compressed zip file. The log segment will be uniquely named to indicate the log-file type, controller (A or B), and the date/time of creation. The file name format is logtype_yyyy_mm_dd__hh_mm_ss.zip. To activate push mode, select the Include Logs check box in the Settings > System > Properties > Managed Logs Properties panel.
In pull mode, when log data has accumulated to a significant size, the system sends notifications via email or SNMP to the log-collection system, which can then use FTP or SFTP to transfer the appropriate logs from the storage system. The notification will specify the storage-system name, location, contact, and IP address and the log-file type (region) that needs to be transferred. To activate pull mode, clear the Include Logs check box in the Settings > System > Properties > Managed Logs Properties panel.

The managed logs feature monitors the following controller-specific log files:

Expander Controller (EC) log, which includes EC debug data, EC revisions, and PHY statistics
Storage Controller (SC) debug log and controller event log
SC crash logs, which include the SC boot log
Management Controller (MC) log

Each log-file type also contains system-configuration information. The capacity status of each log file is maintained, as well as the status of what data has already been transferred. Three capacity-status levels are defined for each log file:

Need to transfer—The log file has filled to the threshold at which content needs to be transferred. This threshold varies for different log files. When this level is reached:
- In push mode, informational event 400 and all untransferred data is sent to the log-collection system.
- In pull mode, informational event 400 is sent to the log-collection system, which can then request the untransferred log data. The log-collection system can pull log files individually, by controller.
Warning—The log file is nearly full of untransferred data. When this level is reached, warning event 401 is sent to the log-collection system.
Wrapped—The log file has filled with untransferred data and has started to overwrite its oldest data. When this level is reached, informational event 402 is sent to the log-collection system.

Following the transfer of log data in push or pull mode, the log capacity status is reset to zero to indicate that there is no untransferred data.

NOTE: In push mode, if one controller is offline, then its partner will send the logs it has acquired from the offline controller along with its own logs.

Saving log data to a file

To help service personnel diagnose a system problem, you might be asked to provide system log data.

Using the PowerVault Manager, you can save the following log data to a compressed zip file:

Device status summary, which includes basic status and configuration data for the system
The event log from each controller
The debug log from each controller
The boot log, which shows the startup sequence, from each controller
Critical error dumps from each controller, if critical errors have occurred

Logs do not include user data.

NOTE: The controllers share one memory buffer for gathering log data and for loading firmware. Do not perform more than one log saving operation at a time, or to perform a firmware update operation while performing a log saving operation. Also, do not attempt to gather performance metrics while performing a log saving operation.

Alternative methods for obtaining log data are to use the Collect Logs action (Maintenance > Support) or the get logs command in the FTP or SFTP interface. These methods will transfer the entire contents of a log file without changing its capacity-status level. Use of Collect Logs or get logs is expected as part of providing information for a technical support request.

For information about using the FTP or SFTP interface, see Using FTP and SFTP.

Using FTP and SFTP

Although the PowerVault Manager is the preferred interface for downloading log data and historical disk-performance statistics, you can also use FTP and SFTP to do these tasks, and to install security certificates and keys. FTP and SFTP can also be used for updating firmware and installing security certificates and keys.

NOTE: Dell recommends using SFTP rather than FTP because SFTP is a secured protocol.
NOTE: Do not attempt to do more than one of the operations in this appendix at the same time. They can interfere with each other and the operations may fail. Specifically, do not try to do more than one firmware update at the same time or try to download system logs while doing a firmware update.

Downloading system logs

To help service personnel diagnose a system problem, you might be asked to provide system log data.

You can download this data by accessing the system’s FTP or SFTP interface and running the get logs command. When both controllers are online, regardless of operating mode, get logs will download a single, compressed zip file that includes:

Device status summary, which includes basic status and configuration data for the system
MC logs from each controller
Event log from each controller
Debug log from each controller
Boot log from each controller, which shows the startup sequence
Critical error dumps from each controller, if critical errors have occurred
CAPI traces from each controller

Use a command-line-based FTP/SFTP client. A UI-based FTP/SFTP client might not work.

Download system logs

Perform the following steps to download the system logs:

Steps

In the PowerVault Manager, prepare to use FTP/SFTP:
a. Determine the network-port IP addresses of the system controllers. See Configuring controller network ports.
b. Verify that the FTP/SFTP service is enabled on the system. See Enabling or disabling system-management settings.
c. Verify that the user you will log in as has permission to use the FTP interface. The same setting allows a user to transfer files using both FTP and SFTP. See User settings.
Open a Command Prompt (Windows) or a terminal window (UNIX) and go to the destination directory for the log file.
Using the FTP/SFTP port specified in the system services settings, enter:
sftp -P <port. <controller-network-address> or
```
ftp <controller-network-address>
```
For example:
```
sftp -P 1022 10.235.216.152
```
or
```
ftp 10.1.0.9
```
Log in as a user that has permission to use the FTP/SFTP interface.
Make sure the client is in binary transfer mode. Enter: binary
Enter: get logs <filename>.zip
where <filename> is the file that contains the logs. Dell recommends using a filename that identifies the system, controller, and date.
```
get logs Storage2_A_20120126.zip
```
In FTP, wait for the message Operation Complete to appear. No messages are displayed in SFTP; instead, the get command returns once the logs collection is finished.
Quit the FTP/SFTP session.
NOTE: The log files must be extracted from .zip file to view them. To examine diagnostic data, view store_yyyy_mm_dd__hh_mm_ss.logs file first.

Transferring log data to a log-collection system

If the log-management feature is configured in pull mode, a log-collection system can access the FTP or SFTP interface on the storage system and use the get managed-logs command to retrieve untransferred data from a system log file.

This command retrieves the untransferred data from the specified log to a compressed zip file on the log-collection system. Following the transfer of the log data, the log capacity status is reset to zero indicate that there is no untransferred data. Log data is controller specific.

For an overview of the log-management feature, see Managed logs.

Use a command-line-based FTP/SFTP client. A UI-based FTP client might not work.

Transfer log data to a log-collection system

Perform the following steps to transfer log data to a log-collection system:

Steps

In the PowerVault Manager, prepare to use FTP/SFTP:
a. Determine the network-port IP addresses of the system controllers. See Configuring controller network ports.
b. Verify that the FTP/SFTP service is enabled on the system. See Enabling or disabling system-management settings.
c. Verify that the user you will log in as has permission to use the FTP interface. See User settings.
On the log-collection system, open a Command Prompt (Windows) or a terminal window (UNIX) and go to the destination directory for the log file.

Enter:

sftp -P <port> <controller-network-address> or

ftp <controller-network-address>

For example:

sftp -P 1022 10.235.216.152

ftp 10.1.0.9

Log in as a user that has permission to use the FTP/SFTP interface.
Make sure the client is in binary transfer mode. Enter: binary
Enter:
get managed-logs:<log-type> <filename>.zip where:
- log-type specifies the type of log data to transfer:
- crash1, crash2, crash3, or crash4: One of the four crash logs on the storage controller.
- ecdebug: Expander Controller log.
- mc: Management Controller log.
- scdebug: Storage Controller log.
- <filename> is the file that contains the transferred data. Dell recommends using a filename that identifies the system, controller, and date.
```
get managed-logs:scdebug Storage2-A_scdebug_2011_08_22.zip
```
In FTP, wait for the message Operation Complete to appear. No messages are displayed in SFTP; instead, the get command returns once the data transfer is finished.
Quit the FTP/SFTP session.
NOTE: The log files must be extracted from .zip file to view them.

Administering a log-collection system

A log-collection system receives log data that is incrementally transferred from a storage system for which the managed logs feature is enabled, and is used to integrate the data for display and analysis. For information about the managed logs feature, see Managed logs. Over time, a log-collection system can receive many log files from one or more storage systems. The administrator organizes and stores these log files on the log-collection system. Then, if a storage system experiences a problem that needs analysis, that system's current log data can be collected and combined with the stored historical log data to provide a long-term view of the system's operation for analysis. The managed logs feature monitors the following controller-specific log files:

Expander Controller (EC) log, which includes EC debug data, EC revisions, and PHY statistics
Storage Controller (SC) debug log and controller event log
SC crash logs, which include the SC boot log
Management Controller (MC) log

Each log-file type also contains system-configuration information.

How log files are transferred and identified
Log file details
Storing log files

How log files are transferred and identified

Log files can be transferred to the log-collection system in two ways, depending on whether the managed logs feature is configured to operate in push mode or pull mode.

Push mode—when log data has accumulated to a significant size, the storage system sends notification events with attached log files through email to the log-collection system. The notification specifies the storage-system name, location, contact, and IP address, and contains a single log segment in a compressed .zip file. The log segment is uniquely named to indicate the log-file type, the date and time of creation, and the storage system. This information is also in the email subject line. The file name format is logtype_yyyy_mm_dd__hh_mm_ss.zip.
Pull mode—when log data has accumulated to a significant size, the system sends notification events using email or SNMP traps to the log-collection system. The notification specifies the information that is transferred such as the storage-system name, location, contact, IP address, and the log-file type (region). The FTP or SFTP interface on the storage system can be used to transfer the appropriate logs to the log-collection system, as described in Transferring log data to a log-collection system.

Log file details

Log files contain the following information.

SC debug-log records contain date/time stamps of the form mm/dd hh:mm:ss.
SC crash logs (diagnostic dumps) are produced if the firmware fails. Upon restart, such logs are available, and the restart boot log is also included. The four most recent crash logs are retained in the storage system.
When EC debug logs are obtained, EC revision data and SAS PHY statistics are also provided.
MC debug logs transferred by the managed logs feature are for five internal components: appsv, mccli, logc, web, and snmpd. The contained files are log-file segments for these internal components and are numbered sequentially.
The comments field—used when collecting logs—is limited to 256 characters.

Storing log files

It is recommended to store log files hierarchically by storage-system name, log-file type, and date/time. Then, if historical analysis is required, the appropriate log-file segments can easily be located and can be concatenated into a complete record.

For example, assume that the administrator of a log-collection system has created the following hierarchy for logs from two storage systems named Storage1 and Storage2:

In push mode, when the administrator receives an email with an attached ecdebug file from Storage1, the administrator would open the attachment and unzip it into the ecdebug subdirectory of the Storage1 directory.

In pull mode, when the administrator receives notification that an SC debug log needs to be transferred from Storage2, the administrator would use the FTP or SFTP interface on the storage system to get the log and save it into the scdebug subdirectory of the Storage2 directory.