Optimising storage for complex data needs

Eric Bassier at Quantum explores the difficult issues that surround the efficient storage of unstructured data

 

We’re living in a world where data is everything. The amount we’re producing is increasing exponentially - approximately 2.5 quintillion bytes of data is created globally every day.

 

While data is not only an organisation’s biggest asset, it’s also its biggest challenge to manage. Once businesses have harnessed their data estate, using it to generate business insights is essential to stay relevant and find new revenue opportunities in their industries.

 

To achieve this, organisations must have a clear understanding of what data they create and collect, how often they need to access it, and how long they need to keep it – therefore, how and where it is stored is critical.

 

This means selecting the most appropriate and cost-effective solution for their unique data management needs. It’s impossible to derive value from data if it isn’t stored and secured properly, therefore choosing the right storage options for the organisation is paramount to success.

 

Understanding unstructured data

The difference between structured and unstructured data can have a direct influence on the relevance and effectiveness of storage solutions. Put in context, unstructured data, like large video and imagery files, comprises the majority of data generated by organisations daily, and therefore consumes the most storage capacity. IDC predicts that by 2025, 80% of the world’s data will be unstructured.

 

Additionally, data strategists will need to evaluate how much of the data that they’re generating needs to be archived, along with archiving duration for compliance purposes. A further consideration is what data will be required for analytics that impact business decisions or could be reused for another project – like video footage – in the future.

 

Understanding these two variables will inform the best method of storage – and how much capacity – must be implemented.  

 

Choosing the right data storage

With spiralling amounts of data, organisations must take some key steps to identify their data types, consider the best storage business model – whether on the cloud, on-premises, or a hybrid approach – and look at their unique data storage needs to set up the right storage solution. There are several key considerations for success.

 

Performing a comprehensive data audit. Chief Data Officers, CIOs, and line of business leaders will first need to establish the type of data their organisation generates and collects as well as the amount they should retain for the long term. These needs are best identified by performing a thorough audit of data used and stored across the business functions. Knowing the right combination of storage solutions is essential, as one is never enough.

 

Considering the cloud. Data strategists should then take the time to determine where to store the data based on their specific organisation’s needs. Considerations for storing data in the cloud or on-premises should include elements like cost, compliance, speed, backup, and security.

 

Some workloads are best suited for public cloud infrastructure, and others better suited for on-premises storage. For instance, non-permanent workloads that require very fast uploads and instant sharing are most suited for public cloud storage, from backups that need to be accessible at a moment’s notice to collaboration tools that keep employees connected around the world.

 

However, it’s worth bearing in mind that while it offers great flexibility, public cloud can be a very expensive long-term storage option for hosting files and objects. Usually, a blend of cloud and on-premises – in other words, a hybrid environment – is ideal to support various needs and workloads.

 

Speed, security, and sustainability. With the fact-finding mission accomplished, data leaders should now turn their attention to finding the best storage solution. For on-premises storage, this means choosing the ideal backup solution for the organisation, for instance tape, disk, or a combination of both. The decision will rest on criteria such as data access needs, whether offline solutions are needed, and most obviously, available budget.

 

While disk-based storage is faster than tape, tape storage offers air-gapped backup, which maintains an offline copy of data as an extra layer of security should a successful breach occur. Tape can also help address sustainability concerns, as this system emits less CO2 and requires less power cooling than disk-based storage.

 

Many organisations prefer a combination of both storage solutions which offers the disk-based benefits of fast storage for high-performance combined with the low-cost, secure, long-term archiving of tape.

 

Like fingerprints, no two data files are the same

Every organisation has a unique set of needs, priorities, and goals when it comes to their data storage. By making a thorough data discovery session the first stage in the process, data leaders will gain a full understanding of the types of data being generated and be able to select the best method of storage for that data.

 

In this way, organisations will make the most intelligent decisions for data management and be able to generate meaningful insights from the data that they’re collecting for long-term business success.

 

---

 

Eric Bassier is Senior Director Products at Quantum

 

Main image courtesy of iStockPhoto.com