Clarification

According to comments here and on reddit, it was obvious that my intention of this post was somehow misunderstood – partly because of the original misleading title (was: “.. – 5 reasons why social bookmarking doesn’t work”). Maybe these adaptions from an xkcd comic does clarify:

Right tool: Use bookmarks to get things done

I think, derefr sums this up very nice:

I find a GTD approach works well: what next action are you going to apply to this bookmark? If it’s just “well, it was neat!” you have no reason to save it (perhaps share it, but not save it), and can throw it away.

The same goes for using the tag “mycomment” to follow up discussions you’ve partaken or “toread” to know what to read once you’ve got some free time. These bookmarks all serve a purpose that is clear to you while bookmarking. This also helps you picking an appropriate tag. No critique on that one.

Right tool: Sharing links

It is clear that bookmark sharing sites such as reddit, Digg, or Stumbleupon that all focus on link sharing have proven that this concept works. Delicious, Simpy, Magnolia et al. all have features to help you share your bookmarks. No critique on that one.

Wrong tool: Remembering potentially interesting links

This is what this article is dealing about: Saving bookmarks that are not useful to you now but – without yet knowing what you’ll use this bookmark for – you save it because it is potentially interesting in the future. I think that doesn’t work and the 5 points should prove that.

Reason 1: You can’t foresee the future

Deciding which web site will be valuable in the future is a very very hard task. I’m not too good at it. I pile up tons of bookmarks I never look at afterwards and on the other hand I decided to not bookmark sites which I needed afterwards. In fact I’m so unsure about my ability to bookmark the right pages I often don’t try searching for a link in my pile of bookmarks but instead google first because I expect being faster this way. Too often I searched my bookmarks altering tags and search terms and didn’t find the bookmark in the end.

Additionally: Even if I would know which links will be of interest in the future, I can’t decide how I should tag (categorize) my bookmarks. When I tag an article, I normally have skimmed it and while categorizing I look at its title. When I tag I’m in a completely different situation – information wise – from when I search for the link.

Your categories may change when you get
familiar with a product or topic

Your information level when looking at a document
differs from when trying to recall that document

Reason 2: You tear links out of its context

Bookmarking is like cutting passages
from books: you remove information
from the context you originally found it

The word “bookmark” relates to the pretty carton markers you use when reading books. Although the way it is used in the web is far far from what it means in books lets delve into that comparison a bit:
To go sure you will be able to find an important passage once you finished a book, you underline or write a few words into the margin to outline a paragraph. Then, when you recall that great sentence you most certainly know in which book it was written (unless that book is a conglomeration of quotes). Then, you often can remember the way that statement was used in the argumentation and in what topic it was embedded. And finally, amazingly, your brain often tells you where on a page (e.g. bottom left) the searched sentence is written. So you normally get quite a bunch of context information to guide you in your search and you will find the wanted sentence within a short amount of time, even if it wasn’t underlined. And even if you don’t find it, you often have a good time reading through the other amazing statements and end up quoting something you didn’t intend.

The way bookmarks are handled in the web would mean to books that you tear out that sentence out of the book, stick a few colored post-its to it and throw that snippet onto the pile with the 1325 other quotes. Bookmarking means taking information out of the context you originally found the information in. On the web context means how you found that link: Was it on Google or in your feed aggregator? Was it a blog post of one of your colleagues? Was it in an email? I often remember these things. Without being a psychologist or having an education in these things I guess our brain is pretty good in remembering context. So why don’t we use techniques that help our brain instead of trying to replace it?

Reason 3: It takes too much time

Bookmarking should save you time – and frustration. Leaving out the frustration bit: Does it really save you time?
Lets say it takes 10 seconds to categorize a bookmark and lets say you’ll use every 20th of your saved bookmarks (which are rather optimistic guesses). That means that when trying to recall an url from your bookmarking service you need to be 200 seconds faster than when you didn’t bookmark any pages at all (as it took you 200 seconds for bookmarking the 20 bookmarks out of which you used 1).

I’m pretty sure you won’t save over 3 minutes in average searching in your pile of bookmarks compared to thinking for halve a minute where you found that link and then going down that trail. So: Why the hassle?

Reason 4: It didn’t work for me

I tried it. I gathered 3444 bookmarks in 2 years using 3034 tags. I asked myself how I could change my tagging practices to improve the recall. I failed. I gave up. I cannot believe there’s no one out there feeling the same.

I stopped bookmarking nearly two months ago. First, when reading articles that felt so interesting it was hard to not bookmark them. Then, it was kind of liberating not having to think “is this page valuable in the future?” “what tags should I use?”.

I never missed it. I always found that link. I don’t regret.

Reason 5: Social bookmarking won’t improve that soon

You may argue that there soon will be techniques to overcome the problems I just mentioned. But my claim is that social bookmarking sites won’t improve that soon.

In my last post I asked: “Why is tagging stuck?”. Gene Smith argues correctly that tagging isn’t stuck. He continues:

Want to know what is stuck? Del.icio.us

The same is true for all the other social bookmarking sites. RawSugar did a brilliant next step (before it went offline) but the social bookmarking market is quiet ever since. I couldn’t find fresh ideas in delicious’ current redesign. It seems like they moved buttons from here to there. I hoped they wouldn’t just redesign the appearance but would also change the way users interact with their data.

So, I guess these services are just as good as it gets. No improvements to wait for. That means it’s our – the users – turn to change our habits, to find the right tool for the job.

Paolo Valdemarin:

4 years later I’m still wondering when will we get some truly advanced tagging tools.
Where are all these tools to manage all my tags (on Flickr, on del.icio.us, on technorati, in my RSS reader, on my blog, etc), to help me organizing them, to allow me to gain more advantages from tagging? (maybe they are somewhere and I simply have not found them yet…)

Matt Mower:

I have been surprised, that [...] the state of the art in tagging seems firmly wedged in 2003. Surprised because there seemed [...] to be a momentum building in the use of tagging

David Weinberger:

Tagging like it was 2002

Thomas Vander Wal:

In the consumer space thing have been stagnant for a while, but in the enterprise space there is some good forward movement and some innovation taking place
[...]
While there are examples that tagging services have moved forward, there is so much more room to advance and improve. As people’s own collection of tagged pages and objects have grown the tools are needed to better refind them.

Vander Wals post is very very insightful and worth a read: He sums up the tagging history and expresses a few brilliant ideas how to proceed.

The delicious lesson – revisited

The big question remains: Why is tagging stuck?

My suggestion is that we may rethink the delicious lesson: Not in terms of “is it true that personal value precedes network value?” but in terms of “what is the real benefit of the users?” or in other words: “How can we design the itch that causes users to generate valuable metadata?”

Recently I talked with Cédric Huesler, a coworker of mine about his use of del.icio.us: Instead of using delicious for storing his bookmarks for later retrieval he stores them to exchange links with strangers. Indeed he has 19 regular consumers of his bookmarks, 7 of these users he is consumer as well.

He doesn’t store his personal bookmarks at all. He can recall from memory where or how he found a certain website and goes back to his google history.

There are just a few entry points into new information on the web: there is Google, feed aggregators or frontpage sites. When there are good search utilities in those tools who needs bookmarks? I must confess that searching at those entry points feels more natural to me than remembering the exact tag I used.

Let’s put it straight: Using tags to find my bookmarks later just doesn’t work. I give up. And no, it’s not just the lack of good tools that help me going through my bookmarks to reorganize them. I won’t do that for all my 3444 bookmarks. And no, this won’t be solved with better tools to refind my items. What do you want to throw into the mix? Fulltext search and time based drill-down? This has nothing to do with tags.

So, we might have to rephrase the users motivation to tag, as I don’t think Joshua Porter was right when he wrote:

in order to gain more personal value, they use tags to be able to find their bookmarks later

I’m not yet at the point where I could correctly rephrase that statement, but I think Cédrics approach in using tags not for personal recall but for publishing is worth a thought. The value therein is close to the value of blogging: You get attention and you communicate. And that’s what the web is about, isn’t it?

History of tags

Update September, 2007: Thomas Vander Wal wrote a very good roundup on the tag history.

Gartners hype cycles applied to tag history

I think gartners hype cycles prove to be right when applied to the tag history (hype cycle descriptions taken from Floor eTrends):

Technology trigger

A breakthrough, public demonstration, product launch or other event that generates significant
press and industry interest.

The technology trigger most likely was del.icio.us and subsequently flickr adding tagging to their service.

Peak of inflated expectations

A phase of overenthusiasm and unrealistic projections during which a flurry of publicized
activity by technology leaders results in some successes but more failures as the technology is
pushed to its limits. The only enterprises making money at this stage are conference organizers
and magazine publishers.

In this phase there were indeed many blog posts talking about this subject, as Louis Rosenfeld
put it:

Lately, you can’t surf information architecture blogs for five minutes without stumbling on a
discussion of folksonomies

I guess in this phase many people said things they now feel embarassed about.

Trough of disillusionment

The point at which the technology becomes unfashionable and the press abandons the
topic, because the technology did not live up to its overinflated expectations.

This is the phase we’re in now. There are no blog posts any more. Tagging is not really
unfashionable but the topic is “done” à la «if that’s all what’s tagging adds to the web experience, I’m not interested in this technology any more». There isn’t much thinking and innovation going on.

Slope of enlightenment

Focused experimentation and solid hard work by an increasingly diverse range of organizations
lead to a true understanding of the technology’s applicability, risks and benefits. Commercial
off-the-shelf methodologies and tools become available to ease the development process.

Let’s hope gartner is right about the future of folksonomies!

Plateau of productivity

The real-world benefits of the technology are demonstrated and accepted. Tools and
methodologies are increasingly stable as they enter their second and third generation. The final
height of the plateau varies according to whether the technology is broadly applicable or only
benefits a niche market.

It has yet to show if folksonomies such as in del.icio.us or flickr prove themselves for the masses.

Update (September, 2007): Do folksonomies apply to hype cycles at all?

Joe Lamantia raises the question if tagging should be applied at all to Gartners Hype Cycles:

Tagging in fact shows few characteristics of the enterprise technologies that Gartner’s Hype Cycle is built around

Joe argues rightly, that tagging has not yet reached the broad economy, it’s not that Gartner would care to apply folksonomies to their Hype Cycles.

Although: Gartner apply the hype cycle to technologies such as “corporate blogging” or wikis. It seems it does not lie in the nature of tagging that it won’t ever apply to hype cycles, the only fact that hinders Gartner to apply tagging to their hype cycles is that there is no money earned with it. I’m not into business analysis at all so I am grateful for Joes insights which he concludes with:

If it doesn’t cost money, the perceived risks of the technology are lower, and the big analysis firms pay less attention, because their customers see less need to pay for analysis

Local.ch is a local search engine for Switzerland, that means I can now work on information retrieval related stuff full time – which was what I did in my free time already. Being paid for doing the things I like is a gift I don’t take for granted.

The R&D team consists of about 10 people – all very talented and smart. Plus, the atmosphere is friendly yet challenging.

Say bye to tag clouds

Then, I’ll give a talk at webtuesday, Zurich about "Improving navigation in tag spaces": Why tag clouds don’t make much sense, why
tagging lost its ground and what could be done to improve the users experience.

The talk will be based on the few blog posts I wrote about this subject plus some newly gained insights.
If you’re living near Zurich it would be a pleasure to meet you there.

After the presentation at the tagging workshop at www2006 we felt the need to give our paper a more www-friendly, I-don’t-want-to-read-through-those-theoretical-equation-flooded-papers face.

So, here you go: Automated Tag Clustering: Improving search and exploration in the tag space. To read this document you should have a clue what tags are about, you should also know some tag services as delicious or flickr so you can understand the limitations these services currently have. If you don’t want to read through the whole papers, the numerous figures give you a good summary. Finally, to wet your appetite, here a few excerpts of the document:

Currently tagging services still provide a relatively marginal value for information discovery and we claim that with the use of clustering techniques this can be greatly improved [from introduction]

The whole promise of collaborative tagging is that by exploring the tag space you can discover a lot of useful information you would not find with traditional search engines. When your information need is not well defined, the idea that you can explore and see what other people tagged with certain tags is very attractive. We believe that tagging will be able to reach a very wide audience only when exploration techniques will be effective. [from limited exploration]

Although a great visualization paradigm, we believe that with today’s tagclouds it is hard to find more than one or two tags to click on. Tags are not grouped, there is too much information, so that you find lot of related tags scattered on the tag cloud. One or two popular topics and all their related tags tend to dominate the whole cloud. For example, looking at the del.icio.us tagcloud, one would mostly see tags related to web design and technologies. This is because these topics are overwhelmingly more frequent than anything else. [from limited exploration]

Tag web2.0 nowadays is so popular and is combined wildly with anything. In fact this tag is so overused that if you look at tag bookmarks in the del.icio.us dataset, the most used cotag is web2.0[...]. Basing tag similarity on these numbers often doesn’t make sense at all. The similarity measure should be chosen so the popularity of a tag doesn’t affect the set of a tags related tags. Don’t cut the long tail. The success of blogs is driven by the importance of the long tail. We all know that it is crucial to support the niches. Tagging applications should empower the long tail too. If you just sort by popularity, you’d loose all those niches. [from choosing a similarity measure]

We’d be happy to get any kind of feedback on the article. Just post a comment to this blog post.

Edit (4 years later!): A few guys asked me about the source code: Source code with syntax highlighting, download.
You need kmetis to make this run, see usage() to see how it should be used.

I will write more about it afterwards as I’m pretty busy finishing my demo.

If you will attend the conference, leave me a note so we could meet somewhen at the conference.
I’m looking forward to this conference as it will be my first one.

When I first heard about del.icio.us (and after that few days when I didn’t get it..) I thought: “This is revolutionary”. There were many things tags made possible that were just not possible until that day.

Joshua Schachter was the guy that invented tags (or at least that’s how the story is being told). Originally thought as a way to organize ones own bookmarks the social effect became obvious:

If everyone tags, the “community” profits.

Now, we have del.icio.us. Now we organize our bookmarks with tags. And our photos.
And our books, our games, our software, our tagging sites, and also your bulldogs, if you have any.

However, as we have tagged our whole life, what do we do with it? What is it good for?
I fear the tagging-revolution is about to calm. And I believe that’s because many people don’t see the advantages in tagging. I believe that many many things can be made possible by using tag-based systems. If we realized this, tagging would get some fresh air and eventually tagging gets mainstream.

Is it just me, or is the tagging revolution really stuck? I desperately miss new, visionary, inventive articles on tags.

• To all smart people, where are your ideas?
• To all programming geeks: Where are your algorithms, your “proof of concept” web services?

I could stop here with my article, but, hey, I don’t want to be the grumbling guy that sits and waits for new things coming up, so here I am, trying to expose my brain to you.
In this article I want to take a look at what areas tags are already strong in and how tagging could gain ground in these areas.

Tags help you to organize

When Joshua came up with the idea of tags, it was purely meant for organizing. It was only when also other people started organizing by tags, when the whole idea of “folksonomy” came up.
What does organizing mean? It is like tidying ones room: You put every paper and pencil you have at a place you can remember and seems logical to you so you can easily remember where you have put that thing. Now as we are not limited into physical means when we organize data we have many new possibilities. There are already good articles about this so I won’t discuss this in detail here.

At the end of the day, the question arises: Is organizing your bookmarks by tags really that good?

Just to make the point I come up with another way to remember things: While browsing, the browser could save all pages in a cache and when you are searching for a page you have visited (which is why you originally bookmark the page anyway), you make a fulltext search through all your cached pages. It’s a kind of “Google search” over pages you have already visited. I know this would have some downsides but it would have some advantages too. I often searched for a page I bookmarked and couldn’t remember the tag I used. This problem wouldn’t occur in the “searching through cache” system.

What I am trying to say is: If tagging solely would be for organizing your stuff, it wouldn’t be worth the trouble.

Folksonomy – Classification of the masses

Folksonomy is – as I understand it – the distributed classification of data by the big mass of people who tag stuff. Folksonomy often is said as a new system to build a taxonomy. It’s like building the Open Directory by thousands of people tagging stuff.

What is folksonomy good for? Why do we want to put bookmarks into categories?

Folksonomy enables to explore

Where do you go to start building a new expertise? Is it del.icio.us? Is it Google?
Let’s say your boss tells you that the data your software saves in the database should be encoded. You didn’t think much about cryptography, it merely was a topic that you “should know about” but you were never really interested in cryptography (I’m speaking for myself here.. :-) ). You don’t really know where to start. You know you want to know something about cryptography, but you don’t know exactly what.
A good list of articles or even starting points could shorten your learning curve.
Thereafter, you may want to “travel through the cryptography universe”. And to travel means knowing which articles are related to the one you just read and are so enthusiastic about. You need a map of the cryptography universe, you want to know what is left and right, top and bottom, you want to know everything and everyone related to “cryptography”.
Now then: What would you do?

Do tag systems help you to explore?

Delicious on cryptography

Delicious results on «cryptography»

I would go on del.icio.us/tag/cryptography+introduction. There I find a nice article titled “An Overview of Cryptography“. I guess I’m lucky! If I’d read the article, I’d probably find out which subtopics exist, how cryptography is related to similar issues and so on. You kind of get this “map of the cryptography” universe. But, this is done by only one author. Probably I don’t trust him (probably I should do so, after reading his cv), or you simply do not have time and/or energy to read through 44 pages, although the article looks good. I’ll probably go back to delicious and find out, that the related tags of “cryptography” are:

• security
• reference
• encryption
• crypto
• algorithms
• computing
• software
• nsa
• tutorial
• kids
• education

Now this is not very convincing, is it? You argue:

Yeah, but this is far better that what I get on Google

.

Google on cryptography

Google results on «cryptography»

It is. When looking at this Google results I remember that Google is meant for searching when I already know what I search for. But now I am at a different stage. I don’t know exactly what to search for. I don’t know, because I don’t have any expertise in cryptography. BTW: Google does come up with an article that looks like a good introduction into cryptography as well..

Open directory on security

What about open directory? Let’s give it a try: After typing in “cryptography” I find out that this topic is classified in Science > Math > Applications > Communication_Theory > Communication Theory > Cryptography. Clicking this link you get what you were probably looking for.
You get a nice overview:

Google open directory on «cryptography»

• Algorithms
• Books
• Events
• Historical
• Journals
• People
• Programming Libraries
• Research Groups
• Theory

Now you stand at a guidepost. You see the “cryptography universe”. You probably don’t see what is left and right to cryptography, but here you have a “cryptography at a glance”.
Now it’s up to you: Do you want to explore “algorithm land”, take the shortcut and download the programming library of the language of your choice? Or do you even want to get advice from people that are experts on that matter?
Even if the links provided here don’t give you what you are looking for, here you get a clue what you should look for.

Comparing the three

Let’s compare browsing to a reallife quest: Finding out where your next conference will take place. Say you want to go to the next etech conference, you don’t know where it is and you are not an American citizen.

Ouch, nearly bumped my head into horton plaza!

On the conference websites they often put a map showing the conference place like 10 meters above surface. This map is helpful. But only at the point when you are quite next to the conference.

Help, I cannot breathe out there!

Then, when you desperately search for a more general map, you’ll possibly find a map of how it looks from outer space. Yeah, I know that San Diego is in the US, but I’d like to know which airport is next to the conference.

Distances between observer and data

That’s quite similar to the views we have with del.icio.us and open directory.
Delicious would tell you: “the roads nearby are ‘union street’, ‘Broadway circle’ and ‘Broadway’…”, open directory proclaims: “we have five continents in the world: ‘America’, ‘Asia’, ‘Africa’, ‘Australia’ and ‘Europe’…”. Now, I’m exaggerating a bit but you get the point: Sometimes you need a map that lays between the too detailed and the too general map.
Looking for this type of view is like saying: “I want a bit more ontology than tags but not that much taxonomy as open directory”. That’s where I’ve put the question mark. It’s not that you always want to see the data at that distance but sometimes you desperately want to have that viewpoint.

Now, what has this to do with tagging? I believe that this missing in-between view can be won by analyzing tags.
Have you noticed how flickr does this in-between view?
When you search for love, flickr cluster asks you: “What do you mean by ‘love’?”:

flickr cluster results on «love»

• a couple kissing?
• a mother holding it’s baby?
• a red heart?

“Wait: Flickr is a bit different from del.icio.us”, you say. Yup. Flickr uses a narrow, del.icio.us a broad folksonomy system.
But I believe that the data clusters, flickr creates with it’s narrow folksonomy data, can also be generated with delicious’ broad folksonomy data. I am programming an algorithm that computes del.icio.us clusters. I’m still at an early stage but I get clusters like this “shopping cluster”:

«shopping» cluster

I realize that even if the cluster data is available, there’s the question how to navigate through the data. The “zooming in” and “zooming out” won’t be as easy as with Google maps.
But anyway, here is the land no one has explored before. I think this is the area we should talk about. Here is room for improvement.

Folksonomy helps you to stay informed about a certain topic

Back to what folksonomies are good for: If you have built an expertise in cryptography, you want to stay informed. If RSA is hacked, you certainly want to be informed.
Delicious has got an “Inbox” where you can subscribe to a tag, e.g. “cryptography”.
Each bookmark that is tagged “cryptography” gets in your inbox. That’s a great way to stay informed. Alternatively you have a list of of recent popular sites tagged “cryptography”. You can subscribe to this lists using RSS and hopefully you get informed timely if RSA is hacked..

Do tag systems keep you informed?

I think the comparison with the distance to the data applies here too:
If I’d subscribe to cryptography, I’d probably miss some important items, just because the guy who bookmarked it used the tag “crypto”. On the other hand, I do not want to be informed about another Rijndael algorithm, I want to narrow the incoming links to articles or essays that deal with cryptography.
Delicious already offers to narrow results: I could subscribe to «cryptography» and «essay», and, when delicious will support union (and it will, as Joshua promises), I also could have subscribe to (cryptography or crypto) and (essay or article) but you see that it doesn’t really solve the problem.
I imagine that one day you can say:

I want to keep being informed about cryptography

and the service asks you:

Should I keep you informed about

• new implementations
• new articles/essays
• security issues

And I believe this is possible. Flickr already asks you this when you are searching for love pictures. I guess it will be based on clusters again.

Tags help you sharing Lists

Back to what tags are good for: They help you building lists. Let’s name a few examples:

• Wish lists: I know that numerous online shops enable you building whishlists. But I’d like to have a whishlist that’s not bound to a company, that I can arrange and rearrange. Many are already using del.icio.us as a storage of their wish list.
• Share your bookmarks: A friend asked me for some links to javascript WYSIWYG editors. I gave him a list of all my bookmarks tagged javascript and editor
• Offer viewpoints of your data: Let’s say your favourite CMS features tagging (featured in many of those new fancy ruby on rails applications), I’m not speaking about blogs here: To allow “normal” visitors to view your data, you’ll add a navigation providing starting points to your entries; specific locations a visitor can jump in to so he could take bathe in your articles. Probably you would add a link to all items tagged “references” and “networking” to achieve that.

How can tag lists be improved?

I’m often annoyed that I cannot put my del.icio.us links in a specific order. I did a little script that puts my newest bootkmark at the bottom but it doesn’t fully solve the problem.
Actually I’d like being able to compose a view of tagged bookmarks, i.e. I want to offer a list of all firms our company has built the network for:

Networking references

Big firms

• UBS
• Migros
• ABB

Medium-sized firms

• Stadlerrail
• Räber Information Management GmbH

Small firms

• Citrin Informatik GmbH
• Goldschmiedeatelier Thildy Keller
• Mini Fruits Trading

Nowadays, such a list can’t be automatically generated from my bookmarks, but it could be, by letting me configure my view as myView = (references+networking, "Networking References", (big_firms, medium-sized_firms, small_firms)).
I know it’s not a big challenge to program such a thing, but nonetheless it doesn’t exist, as far as I know?

Bottom line

It appears to me that there’s not been much progress being done related to tagging systems lately. What rather became better is the embedding of tagging systems into already existing technologies such as search. It gives the impression that core issues are done and that there’s no much room for improvement. In this article I wanted to disprove this.
I think that there’s much much more than I have written in here, I even believe that todays tagging applications cover just about 5% of all the possible features tagging makes possible. Thus, let’s gain ground.

I am a big fan of delicious. I’ve got 1.5K bookmarks there, I like it’s spirit and how open everything is. This article isn’t meant to criticize, but I think delicious is facing some problems.

Performance scale

You might have read my article about Tag system performance. To summarize my tests: MySQL is just not built for large tag-systems. It just doesn’t scale. It does scale up to 1 Million items but delicious does have far more posts.
I am pretty sure delicious is still on the MySQL train, this strong believe comes from my performance tests: The mysql-schemas I tested really have the same characteristics as delicious has.
I fear delicious faces a performance dead end: They have put more servers in the mix, they cache quite a bit, it still is slow. I strongly believe that for delicious to have a future it must become much faster. For me this is the number one downside of delicious. I dream of a bookmark service that has billions of bookmark-posts yet it still will perform nicely. I think it is time for new tag-systems to come up. On tagdb mailing list, there are very good ideas how large scaled tagging systems should work (e.g. systems powered by Lucene).

Popular link scale

I think one of the coolest feature of delicious is the popular page. When you read this page regularly you are up to date.. wait: you are up to date concerning CSS tips and firefox and live hacks. You all know that if delicious would get mainstream that page wouldn’t be that interesting any more. It already got boring a bit. As someone put it:

I particularly cannot look at that CSS link lists anymore

I think this page doesn’t scale. It is stuck. And moreover it’s a pity that the coolest page on delicious is not about tags. At first glance you don’t even see what tags a popular link has.
IMHO what is needed here are clusters. Bookmarks go into categories: “browsers”, “programming”, “design” but also “health”, “politics”. When delicious gets mainstream there most certainly will be “sports” or “stars”.
One should then have the possibility to subscribe to certain clusters or better make this subscription automatically out of tags in a users bookmarks.

Bottom line

I think there are some fundamental things that must be rearranged at delicious, otherwise there will be

• a) a big competitor (Google? Yahoo? Microsoft?) coming up or
• b) people will spread to different bookmark services that concentrate on certain clusters. Probably some meta-sites will arise where you can have an overview over all the different sites

I think this problems will arise for every bigger tagsystem. I hope that people will not sniff at tagging systems thinking that they don’t perform well enough..

“writing”, “toread”, “productivity”, “language”

Now what instantially pops into my mind is, that the tag “toread” is quite different from the other tags. In fact it is something I want to do with this bookmark further on. I name this type of tag “adjective” (I will come back to that name later on..). The other tags I consider as “categories“.
Now you’ll probably say “ah, this is a rare exception”. This is not true. I often tag items with “blog” because it happens that the interesting page I found about my favourite hobby happens to be a blog. Therefore I named this type of tag as “adjective” as it is rather a description to the item than it is a category to it.
Other tags used often as adjectives are “reference”, “tutorial”, “fun”, “cool”, “news”, “free”..

Now this categorization is not very correct. Sometimes, I use “blog” not as a adjective. This is, if I want to bookmark a blog that has no content that interests me but it just looks good. Then, I’ll probably blog it as “design blog”. In that day when I redesign my blog, I want to search for all design blogs I tagged..
You see: it lays all in the connection between those tags, not in the tags itself. This is IMO pretty important.

What is that for?

Clusters

You probably tried to cluster your bookmarks by using clusty. What this service does: It tries to put your tags into separate clouds. You know the “tag-bundles” of delicious? This is something like a “auto-tag-bundle” feature. Try it out, if you not already did so and see the problems that arise..
I think the key problem in this cluster-service lies in the fact that this service considers all connections (also the adjectives). But it shouldn’t do so! Adjectives aren’t tags I want in my clusters. Adjectives are spread all over my tags, so they should first be cut away from my “tag-tree” (the tree that is built out of your tag-connections you built by tagging bookmarks).

Similar items

This categorization is also important when you search for “similar” items of a bookmark. When I want to search for similar items of that “how to write more clearly”-article, I’ll search for “writing+productivity+language” and will leave out the “toread” tag (adjective).
Probably this made you realize that categorizing tag-connections is an important task.

Tag clouds

Now there are those tag clouds. When I look at my taggloud then the “biggest” tag is “resource”. Now tag clouds are here to easily find bookmarks (I never search my bookmarks for solely “resource”) or to have a map of your main interests (“what is your hobby?” “ah, I am a big fan of resources”.. :-) I am sure you were also annoyed by that. I want those adjective-tags cut away..!

Synonyms

Now back to some therory: There is a third type of tag-connections: Synonyms. “delicious” and “del.icio.us” are classic synonyms. But I consider “ruby” and “rails” as synonyms too (no, they aren’t synonyms but up to now they are used as synonyms). You type in the second tag just to be sure that you won’t search for the second and find nothing.. I don’t think this category is too important for the cluster-task but I just name it here because I’ll use it further on.

Example

Let’s go for an example.
Lets consider tags that are connected to the tag “ajax”. I gathered some tag-connection-data from delicious (via its rss-feed). And I run a query on my statistical data. This is data gathered during the period of one week. It is not complete. But our experiment will work anyway:

Column “tag-connection” is the tag connected to “ajax” (i.e. javascript), column “weight” depicts the number of times this connection occurred in a bookmark-post on delicious. The tags are ordered by weight. In column “type” you see the result of my computations for this tag-connection. Just to make it clear: These are all tags connected to tag “ajax” ordered number by occurrence of the connection. If a bookmark-post somebody did on delicious is tagged with “ajax” and “javascript” that gives one point for the “weight”-column for “ajax-javascript”.
The outcome is quite good, I think (I must admit that I have taken the example that worked out best :-))
There are some errors, sure: xml-ajax should be a “category”-type as well. But we are looking at the usage of these tags not their “real” meaning (whatever that is).

Computation

Synonyms

To compute these categorization I first went for the “synonyms”. The connection “ajax-javascript” is considered as synonym because “ajax-javascript” is “number one connection” of all connections where ajax is a part of. And when considering the connections of “javascript” (the “vice-versa-connection”), ajax is number two.
I consider two tags as synonyms if “in one direction” the other tag is number one and in the other “direction” the other tag is in the top 10. I made up this rule because I think that in most cases there is one “stronger” synonym that is used most of the time when the “weaker” one is used. The fact that the tag “ajax” is mostly used with tag “javascript” could also mean that “javascript” is a supercategory of ajax (which it somehow is). To avoid that this sub-super-categogy-connections are considered as synonyms, we go sure that “ajax” is also important for “javascript” so ajax is not so sub to javascript.. I hope you can follow :-)

Category/Adjective

Then I compute the “category”. Lets put the values of the above table into a graph.

On the x-axis you see the tags: The tick 1 stands for “web”, 2 for “programming”, 3 for “css”, 4=”design”, 5=”php” and so on. You see I removed the synonym-connections “ajax-javascript” and “ajax=xmlhttprequest” as I think they “disturb” the distribution.
The y-axis depicts the weight of the connection: ajax-web has weight “105″, ajax-programming has weight “100″ and so on.
The black line is the “weight”-column of the table above, the red one is the first derivative, the blue one the second derivative of the weight function.
This graph makes it clear that “web” and “programming” are used quite often in combination with “ajax”, then, there is quite a “gap” followed by the “adjective tail”. I consider the “adjective tail” as connections to be categorized as “adjective”. The tags in this tail are used “out of context”: They don’t really belong to the “ajax-cluster”. They sometimes occur together with ajax, but just sometimes. Mostly not. Therefore they are considered as “adjectives”.
Now the task is to find this “gap”. In my experiments I tried to find the last gap. To find the last gap I started at the end of the tail and searched for the first peak of the first derivative (that is when the second derivative goes from positive to negative) and checked if the peak was high enough. If these to conditions were fulfilled, I snipped the connections into two parts the “pre-gap” connections (category) and the “post-gap” connections (adjective).
The same computation has to be made for the “vice-versa” connection. I considered connections as “category” if one of both computations told that it is a “category”.

Further processing: Ambiguous tags

To achieve good clustering results, I think there is a need of checking if the tag is used in different ways. The prominent example hereof is “apple”. Now, when delicious is still restricted to the blogworld, it is clear that apple means Mac-apple. But in future this may change. To recognize if a tag is used in different environments, the algorithm would have to check the “neighbours of neighbours” (as suggested by Pietro Speroni). That is for ajax: check if the neighbours of “javascript” are more or less the same as the neighbours of “web”. You see that it all lays in the connections between tags. The tag per se is not well-defined but the tag in connection with another tag defines it quite well. Therefore for clustering I’m proposing splitting up amiguous tags. That would add much more simplicity to the resulting clusters.

We are onto something

I’m pretty sure we are onto something. I think this is direction it should go. Computations over tag-connection-distributions are cool. Users shouldn’t insert these infos when posting the bookmarks. Posting should stay easy. I’m not that sure about this “synonym”-computation but I think the “category”-computation turned out pretty good. I tried to build some clusters by hand just by considering the category and synonym-connections and I found a completely detached cluster consisting of the tags “cooking”, “health”, “recipes”, “diet” and “food”. As I said, I think we are onto something..

Further reading

• Building tag clusters by hand
• Pietro Speronis different approach to clustering tags (with java-mindmap-visualisation!)