Dark data: an asset or a liability?

In the physical world, we rarely need to worry about the footprints we leave behind – unless we mean to cover our tracks for well-intentioned or sneaky purposes. Online, however, practically every action taken by humans and machines are like footsteps pressed into wet cement. These permanent traces can both lead to trouble or present unexpected opportunities.

 

In business, the digital trails companies create during their operation can make compliance and auditing much straightforward. But the downside is that businesses must now actively manage the digital wake of their operations too.

 

Is dark data as sinister as it sounds?

 

Dark data is not “dark” in the sense associated with the dark web or illicit activity. The term is closer to dark matter: it’s information that exists but remains undiscovered, inaccessible and, therefore, unused.

 

Coincidentally, scientific estimates suggest that 85 per cent of all the matter in the universe is dark matter, while the share of dark data of all organisational and enterprise data is between 80 to 90 per cent – although a recent global survey by Splunk arrives at a more conservative 55 per cent.

 

The difference between the two is probably down to how dark data is defined. Broadly speaking, any untapped, inaccessible or unknown data qualifies as dark data. But the term can bring radically different sources of data – whether structured, unstructured or semi-structured – under the same umbrella.

 

Much of dark data is made up of raw data, the automatic byproduct generated in the background by digital systems, including servers, firewalls and network monitoring tools. Networking machine data and customer support logs fall under this category too.

 

SaaS platforms and cloud services generate large volumes of such information, but so do legacy mainframes – the high-performance, highly reliable systems used, for example, in financial services, healthcare and aviation.

 

But dark data also includes more visible information than logs and metadata. Old employee emails, raw survey results, document drafts, scanned PDFs and PowerPoint presentations constitute dark data too.

 

Some of these data types have little analytical value and are better described as redundant, obsolete or trivial/ROT data.

 

Duplicated email attachments, backup files, outdated documents and expired session cookies are unlikely to be a goldmine. In these cases, the greatest value may come from identifying and eliminating the data, reducing storage costs and mitigating security and compliance risks.

 

Use cases leveraging dark data

 

Marketing has been exploiting forms of dark data for years, even before the term became widely used. Data companies purchase digital exhaust – unstructured information such as GPS location pings, app usage logs and clickstream data – to improve targeting and predict consumer behaviour.

 

Dark data can also be generated in physical spaces. Even if a shopper isn’t connected to a supermarket’s Wi-Fi, their mobile with the Wi-Fi switched on will broadcast probe requests while looking for available networks, enabling the tracking of its owner as it moves across the shop.

 

The mobile’s “pings” remain dark until they get analysed. Dark data remains dark only so long as it is untapped. Once it is discovered and analysed, it becomes actionable information.

 

Dark data can also be put to good use to improve efficiencies and remove bottlenecks in a number of key areas.

 

Unanalysed network access logs can point to unauthorised access, metadata from internal communication tools can reveal redundant or inefficient processes, and customer service logs can expose emerging fraud patterns that other analytical methods fail to detect.

 

Security and compliance risk

 

Although dark data is much more innocuous than its name at first suggests, it can present significant risks. With storage becoming cheaper and more widely available, many organisations overlook the price they may have to pay for retaining unused information.

 

Yet even a typical small business can generate at least five terabytes of data, at least half of which just sits there. Therefore, over time, the accumulation of digital exhaust and ROT data can become a significant problem even for small businesses.

 

The data invisible to a business can also considerably expand its attack surface.

 

For criminals, it’s much easier to go undetected while sifting through unmanaged data stores in search of some live login data, personally identifiable customer or employee information (PII) or payment card numbers and billing details.

 

Moreover, to amend the old adage, if you can’t see it, you can’t govern it. The EU’s GDPR and global data protection legislation also applies to dark data if it contains names, email and IP addresses, device IDs, location data or other types of PII, even if the data controller is unaware of its existence.

 

Businesses that violate the data minimisation and storage limitation principles of GDPR by the hoarding of haphazardly or automatically collected data can incur hefty fines.

 

The need to manage dark data won’t go away any time soon. With the growing adoption of generative and agentic AI and the proliferation of APIs and MCPs that integrate these systems, the amount of dark data is expected to rise exponentially, while data protection rules are becoming more rigorous.

 

This trend is well demonstrated by the latest developments in the IoT space, where the global inventory of active connected IoT devices reached 21.9 billion endpoints by June 2026.

 

The readings and signals IoT sensors and devices generate is raw data that can either remain dark or can be harnessed to improve predictive maintenance outcomes and operational efficiency.

 

The EU has recognised the economic value of machine-generated information through its EU Data ACT, which regulates access to data generated by connected products, including information which is the by-product of user action, as well as operational logs and metadata associated with the use of the product or service.

 

A growing understanding of the value of raw IoT data will probably shed more light on how dark data can be transformed into an asset in other sectors as well and prompt clarification on how users’ access to it can be guaranteed.