Directories are cool – really!

An interesting thread started off on WebmasterWorld this week. It’s a discussion about directories – in particular the rise and fall in popularity of directories, and whether or not the model still has value in the modern digital age.

In a pre-search world, directories were King!

In the early days of the web, search engines weren’t as efficient at what they did. Websites were rarely created with search engines in mind, web technologies such as Flash acted as barriers to indexing and the technology behind search engines was in its infancy. As was their adoption by users.

Directories on the other hand, were a much simpler approach to organising information (remember, the Internet has deep roots in the academic world) – they were curated by humans who had a genuine interest in the effective categorisation of information and weren’t tainted by commercial concerns.

The Open Directory Project (later known as DMOZ) was essentially an Encyclopedia Galactica in the making and in many ways a precursor to Wikipedia. Its volunteer army moderated and curated tens of thousands of categories of websites and arguably became one of the first websites to be considered to be an ‘authoritative’ resource. Indeed, the idea of human-edited content as a ‘quality signal’ probably owes more to DMOZ than it does to the editorial teams of mainstream media outlets.

DMOZ offered its data for free and many digital giants made good use of it, with Google adopting the data for their own directory (and Yahoo! maintained their own competing directory).

Along came the SEO industry

As search engines grew in prominence and adoption, and Google introduced a link-based algorithm, webmasters began to understand the importance of links. In particular, ‘high quality’ links. And of those, one of the most achievable was a listing in a high PageRank category on DMOZ.

The commercial impact of a DMOZ listing was spectacular. A single link could propel a site’s rankings to the top of much-coveted Google searches and this created a problem that eventually led to the downfall of DMOZ.

The volume of link submissions became a problem for the volunteer moderators, as did the quality of sites being submitted. It became much harder to distinguish a legitimate business from a ‘spam’ site and these issues led to delays in submissions to be approved. At the time, the most common advice given to new SEOs was “submit and forget” as a DMOZ listing could take months to be approved, if at all.

Worse still, individuals began to volunteer edit DMOZ solely to approve their own listings (and block their competitors from getting listed) – such was the power of DMOZ.

The idea of the ‘link directory’ is what finally killed DMOZ

It took several more years for DMOZ to close its doors, but I believe they were the victim of their own generosity.

Free access to DMOZ data allowed webmasters to essentially clone hundreds of thousands of pages of categories and over a million listings. For a time, Google (and other search engines) would rank this content as well, but they quickly introduced countermeasures to prevent this happening.

But the value of those links and the idea of easy user generated content (that is also a source of income) triggered a surge in link directories being produced. For a while, these were a source of easy links for SEOs, but that was never going to last. Over time, Google introduced more countermeasures to dampen the impact of these activities, and with that, the general interest of the ‘directory fad’ began to diminish.

Except it didn’t

And this is the crux of the discussion on WebmasterWorld.

The directory model still has value and is being deployed by lots of digital businesses. It’s the idea of a ‘link directory’ that has diminished. That was, at best, a cheap and sloppy implementation of the directory format. But consider the examples of good implementation:

• Yelp – they’ve carved out a niche and done a great job of dominating it.
• Yell – still going, despite being slow to see past the paper directory format.
• IMDB, IndieDB, ModDB – are all strong resources in their respective industries.
• Trivadvisor – basically a directory with added eCommerce functionality.
• And countless niche directory-esque resources…

These all do something that old school link directories failed to do – they add value to the data.

There are different ways to add value to data

Operating a directory can be a logistical nightmare and a difficult proposition to sell to customers (which is likely why most don’t refer to themselves as directories). Finding different ways to add value and create content that earns a place at the top of search results can be tough job. But there are a number of ways to go about it:

By completion

Either by design or as a consequence of their popularity, some directories have a complete set of data, which adds a lot of value for users. It reduces the likelihood that they will leave the site unsatisfied and means they are more likely to come back.

IMDB for example, falls into this category. There’s no way it would even be on our radar if it didn’t list some of our favourite films and TV shows. It has become the go-to resource for information relating to productions.

By functionality

The hotel industry is dominated by a few big booking engines – these are all directories that offer a fantastic additional feature – the ability to easily compare similar options.

By remuneration

If you can’t earn your way to success, you can certainly buy your way there. The travel industry’s booking engines fall into this category – it’s big business (and the biggest marketing cost for a lot of hotels). They effectively dominate Google results (organic and PPC) and leave hotels, particularly independent hotels, no real choice but to buy in to their proposition.

This isn’t about making money though. Directories need to be sustainable if they are to succeed and for that you need a revenue stream to fund it all.

By exclusivity

I’ve seen several examples of directory type businesses operating within relatively small niches – but they’re the only one doing it and have done for so long it’s near impossible for competition to move in.

They provide a comprehensive resource and become a default tool used by an entire industry. Their members pay a higher annual fee than most SEOs used to pay for even the ‘best’ link directories. And they keep paying every year.

By brand

They approach it differently, but YouTube / Twitch / Instagram ‘influencers’ who promote products are essentially curating information. Perhaps with a much more commercial agenda than the academics behind DMOZ, but effective.

By transformation

Turn data into information – or knowledge! Many review sites do this very well – they became much more than hard data – they offer qualitative information from a variety of sources. This is a great approach to encourage UGC.

By authority

In an age of misinformation and fake news, having any degree of credibility behind your brand is a great thing.

Academic institutions, government bodies, charities and even private organisations can leverage their areas of expertise to create trustworthy and useful resources.

None of this is new

All these components of ‘added value’ are simply aspects of marketing:

• How you are percieved
• Leveraging your strengths
• Outperforming the competition
• Innovative monetisation strategies

Early attempts at ‘link directories’ focused on automation and scaling, going for the quick wins and failing to differentiate their product from the competition. The same could be said for many failed businesses.

The perception of being able to scale big at a low cost encouraged a lot of people to try – the fallout from those failures and countermeasures implemented by search engines stopped others from trying again. However, I think it may simply be the case that the will wasn’t there to commit resource to trying again. Search engines seriously upped their game and became a much more effective way for users to get the information they need.

Perhaps the old-school idea of link directories has faded away, but the effective curation of information is still challenging and has a lot of value for end users.

BRB – off to tinker with a directory idea I’ve been playing around with!