IT managers in many types of firm may have to adjust practices to comply with the EU's Data Retention Directive (DRD), a set of guidelines that were approved on 15 March. The rules will affect "basically every company providing physical access to publicly available communications", according to Patrick Van Eecke, counsel and head of internet law at law firm DLA Piper Rudnick Gray Cary.

Initially the DRD will require telcos and ISPs, and firms such as Google, Yahoo and Skype, to store call traffic and location data (called call detail records – CDRs) for fixed and mobile telephony, internet access and internet email and telephony. These firms will also need systems that can analyse the data. Van Eecke added that the information needed is not the actual content or call transmissions, "It's the traffic data, the telephone numbers, names and addresses of called and calling parties."

Both Van Eecke and Jim Pflaging, chief executive of security analytics vendor SenSage, noted that many companies in addition to telecoms companies and internet services providers will be affected.

"Hotels, internet cafés, public libraries, wireless hotspot operators and wireless community networks – all may eventually need to comply with the directive," said Van Eecke.

Large firms with IP PBXs holding addressing and billing information may also be subject to the mandates of the directive, Pflaging said. "If I were the IT manager of a large firm I'd have this directive on my radar," he added.

Fernando Elizalde of analyst company Frost & Sullivan raised other concerns: "What happens with the small regional ISPs and virtual operators – such as mobile virtual network operators – that are not big enough to have in-house systems? They will still have to comply with the law. Maybe a service hosted by the bigger companies would be the answer."

Elizalde said telecoms providers, ISPs and some other enterprises should start preparing their systems and management procedures now to ensure compliance with the directive, which comes into force in August 2007.

Nigel Ghent, marketing director for storage giant EMC for the UK and Ireland, said the DRD rules had been developed in response to the terrorist threat as well as the dangers of industrial espionage. "The purpose is to enable law enforcement agencies to expedite inquiries concerning terrorism and other serious crimes," he explained.

Frost & Sullivan's Elizalde said, "Telcos need to store CDR data for not less than six months and not more than two years under the current terms of the directive, although some governments are already thinking about moving to a five-year limit."

Elizalde added that telcos would have to store a lot of data and integrate their analysis and retrieval tools with any legacy systems. To comply with the directive and avoid "undue delay" under the rules, firms may need systems that take no more than about 15 minutes to identify required CDR data.

The large number of CDRs that telcos must store means this 15-minute target may be challenging. Pflaging pointed out that a medium-sized telco could easily generate 100 million CDRs a day and the larger ones might each generate about 800 million.

During 9/11 US telecoms carriers were said to have generated over a billion CDRs.
Telcos will need systems that can handle such huge amounts of data to ensure they can quickly answer queries such as: "Who has phoned person X from mobile provider tower X within the last day?" said Pflaging.

EMC's Centera storage specialist director, Andy Stubley, said the main difficulty for managers is not with storage per se. "Storage is cheap and putting CDRs onto disk would be easy – it's the management and access that's the problem." A typical CDR is about 300 bytes, though an IP CDR is around 10 times larger. Although relational databases could be used, Stubley said that the indices needed to store the data could require more space than the data itself, leading to greater storage requirements if such databases are used.

Pflaging said, "The problem is that you're not going to be able to point your Oracle database licence at this to sort it out. Relational databases would be more expensive and performance limited."

A possible solution may result from a recently completed proof-of-concept system, which supports the DRD guidelines. EMC, communications systems vendor Intec and security analysis specialist SenSage said this system can currently process 100 billion CDRs in the timeframe required. The system developed by the three vendors uses Dell servers with the latest dual-core Intel Xeon Woodcrest processors running under 64bit Red Hat server software.