Managing the Storage of Information

SynopsisThe typical business generates huge amounts of data on a daily basis, and until recently the way to deal with this rising amount of stored data seemed to be simple - keep adding hardware, so that the storage limit could be increased. However, this approach has not worked - all that has happened is that organisations have dug deeper and deeper holes into which their data can fall and be lost. Simply adding storage in this manner does not ensure that the data being stored can be searched or retrieved when needed, and it has become obvious that the missing piece of the puzzle is management.

Companies that have blindly added storage without considering its long-term management now find themselves struggling to control their information stores, and are paying far more for the upkeep and maintenance of their storage than is necessary. Because the storage has been deployed in an uncertain, even haphazard, way, it also represents a serious security risk, particularly if mobile workers are involved.

The Managing Storage of Information Report explains the issues involved in controlling storage, and defines best practice in consolidating and optimising resources. It is possible to implement a cost-effective and competitive storage strategy, and the Report establishes the steps that organisations need to take now in order to ensure control of the data that they must maintain as an asset.

The Report should be read by IT Directors, CIOs, and IT managers - individuals with specific responsibilities for storage could fall into any of these groups.

Key FindingsKey Findings

Storage is out of control at the majority of organisations. The increasing gap between storage management ability and the proliferation of storage devices has to be addressed.

The administration overhead of storage is relentlessly escalating. Enterprises must look to do more with less by optimising the storage environment, and instigating monitoring and consolidation initiatives.

There is a huge pool of unmanaged data on laptops, mobile devices and USB memory keys. The management and security of this remote data must be tackled immediately.

Organisations must ensure a data protection strategy is in place that covers the multitude of threats and different storage media.

Regulatory requirements mean that the ability to recover data easily and in a presentable format is more of an imperative.

It is becoming increasingly important to develop a business view of data, which is translated into the type of storage that the data resides on. This tiered approach to storage can bring significant benefits.

Get fundamentals of storage optimisation, compliance, and protection right first before contemplating any sort of Information Lifecycle Management (ILM) vision.

Plans should be put in place to evolve to a utility storage solution over the next five years.

Management Summary

Introduction

The amount of electronic data needing to be held by the enterprise is remorselessly increasing. The Storage Networking Industry Association (SNIA) has estimated that many large organisations had an average compound storage growth rate of 80% from 1999 to 2003. A study conducted by UC Berkeley�s School of Information and Management calculated that the amount of new electronic data stored worldwide has doubled in the last three years. This growth in the amount of data, both in terms of hardware capacity and the drain on resources, has at last forced storage on to the investment agenda for the majority of IT departments.

The storage of data can no longer be ignored by the IT manager, or put on the back burner by simply buying more direct attached disk capacity. The continual reduction in storage hardware prices has caused many organisations to address the symptoms, rather than the root causes. This cannot be allowed to carry on indefinitely, as the growing complexity and size of the storage environment will lead to poor performance, problems with back-up windows, extended recovery times, security issues, and administration costs spiralling out of control. The time has come to address the mounting disparity between storage management capability and the increasing number of storage devices and capacity.

To compound the problems caused by the growth in data volumes, regulatory requirements are increasing the retention period for many types of data and demanding that it is presented in a different way. For example, by insisting that associated data is able to be retrieved, such as all the e-mails relating to a specific topic, or from a particular individual. Data can no longer be archived and forgotten about.

Unfortunately, the storage picture is due to get appreciably worse, even though IT departments are barely managing, and in many cases have lost control of, the current data volumes. Something is on the horizon that will make today�s storage challenges seem like a stroll in the park. Wireless technology, and specifically RFID, has the potential to cause an explosion in the amount of data systems will have to process and store. Those organisations that do not begin the process of storage infrastructure transformation now, could well be completely submerged by the data volumes generated by millions of wireless tags attached to every conceivable entity.

Business Issues

The management of data now is more than just the definition of back-up procedures. Consideration should be given to the data lifecycle and the automation of routine tasks and developing retention policies for data held on the system, enabling tasks to be actioned when certain conditions are met, including defining more capacity to an application when an upper limit is exceeded, or archiving data when a specified age criteria is reached.

The cost of administering storage is around five times the cost of the hardware. The centralisation of storage management and the curtailing of administration overheads must be primary objectives. Vendor-agnostic management tools are allowing organisations to employ software to administer all types of storage equipment from one location. The ability to measure the cost of storage, using financial metrics and where appropriate to charge users for storage consumption can also greatly assist in bringing storage overheads under control.

The workforce is becoming increasingly mobile and in many instances the data can be found outside the organisations direct control. Mobile devices, such as laptops, notebooks, PDAs, and smart phones are becoming essential business tools, providing greater freedom and flexibility. However, the downside is the potential exposure of confidential organisation data to additional risk. The security of data held on mobile devices is a growing concern.

Much of the recent focus has been on securing data passing to and from client terminals and the back-end servers. However, there is an equally serious and increasing risk of remote data being compromised. Many mobile devices lack the necessary protection from the dangers found outside the workplace. It is imperative that policies and software are put in place now to manage and protect remote data stores.

The persistence of the silo approach to storage deployment is leading to an unmanageable situation. Organisations can no longer afford to utilise a tactical and piecemeal approach to storage provision. It is now essential that organisations consider storage as a strategic resource, including its addition into the overall IT infrastructure plans and taking cognisance of business requirements and objectives.

This means the development of a storage strategy and policies, along with a phased plan of action to evolve the storage infrastructure to a common resource, rather than separate islands of storage tied to particular applications or projects. Having this common pool of storage will not only improve utilisation and scalability, but lead to a lower cost of storage and better archiving/recovery provision.

Technology Issues

For many reasons most organisations now find themselves with a complex heterogeneous IT infrastructure containing disparate hardware and software from numerous vendors, with the storage environment being no different. This cannot be allowed to continue; simplification and optimisation must become the watchwords of the IT department. Approaches, such as using industry standards, consolidation, and virtualisation, must be adopted in the storage arena, as well as elsewhere in the infrastructure. The centralisation of data must be a priority to improve storage utilisation that in many cases falls between 20% and 50%, enabling currently available capacity to be used much more effectively and the administration overheads to be reduced.

Industry standards are also important for improving interoperability and easing complexity. The storage environment is no exception and where possible standards should be utilised. IT management must start insisting the Storage Management Initiative Specification (SMI-S) is included on any Request for Proposal (RFP) for new storage solutions. In addition, Linux is now a credible alternative for storage solutions in the data centre, and provides the opportunity to reduce the number of different operating systems in the environment and be hardware agnostic.

In order to meet the challenges facing the IT department there is a need for a step change in the way storage is architected, with storage deployment coordinated with the development of a broader Enterprise Architecture strategy. Organisations should also consider using virtualisation technology to abstract the connectivity from the physical storage, removing the one-to-one link between servers and storage devices. This creates a far more agile storage infrastructure that is flexible enough to allow real-time changes and help to position the IT department as service provider.

Technologies to protect data have always been a prerequisite for efficient operations, as everybody is vulnerable to data loss and corruption. Interestingly, a survey by Ontrack Data Recovery found the major reasons for data loss to be equipment failure and human error, rather than from any malicious intent. Having said that, the threats from internal and external sources continue to increase and it is incumbent on the IT manager to stay one step ahead. A data protection strategy must be developed that employs each storage media and the associated data management services to maximum effect.

Data integrity has become more relevant because of increasing regulatory demands. Not only does this include being able to prevent unauthorised changing of data, but organisations must be capable of substantiating that this is the case through auditing functionality and system features that ensure data cannot be tampered with after being saved to the storage media.

The tiered storage concept is becoming increasingly important for enabling data migration and for exploiting the optimal storage media for the age and value of the data, where policies are created that allow data to be automatically migrated to a lower level, cheaper storage media over time. Disk, tape, and optical technologies should all be used, where most applicable, to provide a cost effective storage solution. Awareness of the data lifecycle and employing a tiered storage approach can significantly reduce costs.

Market Issues

Organisations can do without the lofty visions that many of the storage vendors are peddling in the market place and should be cautious of marketing hype, such as Information Lifecycle Management (ILM). Whilst the ideals and principles are sound there is a danger that such an approach to storage infrastructure does nothing but cloud the issues and confuse the customer. Let�s get the basics right before moving on to tomorrow�s vision. It�s no good being able to take on board business objectives if the storage infrastructure is incapable of reacting quickly to the required changes. The focus must be on optimising the storage environment now, whilst planning to evolve to a utility storage model in the future. Butler Group is particularly impressed with the scalable enterprise philosophy espoused by Dell.

The use of the utility storage model by early adopters is bringing into focus Storage Service Providers (SSPs) as a new market opportunity. All the major storage vendors and some storage consultancies are now able to offer storage on demand solutions that can be tailored to an organisation�s specific requirements. Using managed services in this way enables organisations to exploit the outsourcers� knowledge and expertise to manage the storage efficiently, and offer scalability not usually possible using internal resources.

The whole storage market place is becoming intensively competitive, with the major vendors contending for customers with new products and initiatives. The storage mid-tier is a huge potential market and one that all of the storage vendors are targeting. EMC faces tough competition in this sector, with IBM having recently announced the introduction of its DS6000 and DS8000 storage products, and from Network Appliance and its Network Attached Storage (NAS) devices.

Both the Small to Medium-sized Enterprise (SME) and public sector market segments provide growth opportunities for storage vendors. Storage networks have become easier to deploy and more cost effective, and the public sector is in the throes of e-government initiatives that are large consumers of storage capacity. The issues of data growth, protection, and particularly those of compliance are very pertinent to smaller businesses and Government.

The compliance requirements are relevant to all organisations including smaller enterprises that need to be just as careful with personal information and safeguarding a person�s privacy. Additionally, many SMEs have limited, or no, technical personnel and probably no resources spare to perform the recovery of data as the result of an incident. Whilst data and storage management is of significance to all types of organisation, from multinational to the one-man operation, it is the smaller organisation that is most at risk.

Public sector organisations need to be able to provide flexible and responsive services, despite having to tackle an ever-increasing tide of electronic data. This growing volume of data is putting pressure on the storage infrastructure, where investment has not always been what it should. Not only is the data to be protected, but it has to be easily and safely accessible by different stakeholders, such as employees, individuals, and businesses.

Report Structure

Report Structure

This Report is intended to provide readers with an informative guide to storage management. Butler Group recognises that not every factor will be relevant to all situations, and that some organisations will already have implemented some of the components that make up an effective storage solution. Therefore, the Report has been divided into segments relating to the different guiding principles of information storage, to make it easy for readers to locate the sections that relate to their particular issues.

Section 2 - Introduction

This Section provides an overview of storage concepts and puts the subject into context. It includes a description of the features that storage management should encompass. It also discusses the business drivers, which include the exponential growth in storage needs, the growing compliance issues, and the requirement to quickly retrieve specific documents from archive.

Section 3 - Storage Strategy

A strategy needs to be put into place to manage the data during its retained period, to ensure that it is not altered or deleted, to enable it to be moved automatically as its status changes. The deployment of storage consolidation and the adoption of a more centralised approach can bring significant benefits, including reduction in the system administration overhead, easier data management, and a simpler infrastructure. Engineering storage on demand solutions offers the prospect of flexibility to provide for planned capacity, as well as being able to adjust capability to support changing business needs.

Section 4 - Data and Storage Management

During its life, the value of data changes and so does its status. The lifecycle reflects this and ensures that the data is stored on the appropriate media according to its value and age. This may include on-line disk, near-line optical media, or off-line tape. It is now generally recognised that business continuity and disaster recovery planning are also vital activities. Storage Resource Management (SRM) has evolved from a tool that monitors and reports on storage as an isolated entity, to a set of tools and practices that help companies better serve their stakeholders, and has grown to include management of the system, the fabric, the application, and the storage devices, plus the efficient backup and recovery of data.

Section 5 - Storage Technologies

This Section breaks down the technology components appropriate for managing storage looking at the architectures that can support an efficient storage strategy, and the technologies used to underpin storage resilience, security, and management. This part of the report focuses on the implications of storage management, reviewing the impact in the wider context of storage architecture, and how the different technologies can be used to meet the demands of the business.

Section 6 - Storage Infrastructure Transformation

This section uncovers the issues and advantages of storage management services, along with why storage management is relevant to the public sector and Small to Medium-sized Enterprises ( SMEs). Assistance and practicable advice is also provided for the migration to a next generation storage solution. The Section includes a number of case studies that describe the problems organisations experienced with storage management, the implementation of a solution, and the benefits that the deployment brought.

Section 7 - Storage Futures

This Section outlines Butler Group's opinions as to the future direction of storage technology and markets.

Section 8 - Vendor Profiles

Supplier summary and contact information for the vendors mentioned in the Report. More detailed product profiles can be found within the sections of the Report that are most relevant to the offering. The product and solution profiles are provided as an indication of what is obtainable in the market and should not be used as a comprehensive review of all the available offerings