Local.ch is a local search engine for Switzerland, that means I can now work on information retrieval related stuff full time – which was what I did in my free time already. Being paid for doing the things I like is a gift I don’t take for granted.

The R&D team consists of about 10 people – all very talented and smart. Plus, the atmosphere is friendly yet challenging.

Say bye to tag clouds

Then, I’ll give a talk at webtuesday, Zurich about "Improving navigation in tag spaces": Why tag clouds don’t make much sense, why
tagging lost its ground and what could be done to improve the users experience.

The talk will be based on the few blog posts I wrote about this subject plus some newly gained insights.
If you’re living near Zurich it would be a pleasure to meet you there.

After the presentation at the tagging workshop at www2006 we felt the need to give our paper a more www-friendly, I-don’t-want-to-read-through-those-theoretical-equation-flooded-papers face.

So, here you go: Automated Tag Clustering: Improving search and exploration in the tag space. To read this document you should have a clue what tags are about, you should also know some tag services as delicious or flickr so you can understand the limitations these services currently have. If you don’t want to read through the whole papers, the numerous figures give you a good summary. Finally, to wet your appetite, here a few excerpts of the document:

Currently tagging services still provide a relatively marginal value for information discovery and we claim that with the use of clustering techniques this can be greatly improved [from introduction]

The whole promise of collaborative tagging is that by exploring the tag space you can discover a lot of useful information you would not find with traditional search engines. When your information need is not well defined, the idea that you can explore and see what other people tagged with certain tags is very attractive. We believe that tagging will be able to reach a very wide audience only when exploration techniques will be effective. [from limited exploration]

Although a great visualization paradigm, we believe that with today’s tagclouds it is hard to find more than one or two tags to click on. Tags are not grouped, there is too much information, so that you find lot of related tags scattered on the tag cloud. One or two popular topics and all their related tags tend to dominate the whole cloud. For example, looking at the del.icio.us tagcloud, one would mostly see tags related to web design and technologies. This is because these topics are overwhelmingly more frequent than anything else. [from limited exploration]

Tag web2.0 nowadays is so popular and is combined wildly with anything. In fact this tag is so overused that if you look at tag bookmarks in the del.icio.us dataset, the most used cotag is web2.0[...]. Basing tag similarity on these numbers often doesn’t make sense at all. The similarity measure should be chosen so the popularity of a tag doesn’t affect the set of a tags related tags. Don’t cut the long tail. The success of blogs is driven by the importance of the long tail. We all know that it is crucial to support the niches. Tagging applications should empower the long tail too. If you just sort by popularity, you’d loose all those niches. [from choosing a similarity measure]

We’d be happy to get any kind of feedback on the article. Just post a comment to this blog post.

Edit (4 years later!): A few guys asked me about the source code: Source code with syntax highlighting, download.
You need kmetis to make this run, see usage() to see how it should be used.

I will write more about it afterwards as I’m pretty busy finishing my demo.

If you will attend the conference, leave me a note so we could meet somewhen at the conference.
I’m looking forward to this conference as it will be my first one.

When I first heard about del.icio.us (and after that few days when I didn’t get it..) I thought: “This is revolutionary”. There were many things tags made possible that were just not possible until that day.

Joshua Schachter was the guy that invented tags (or at least that’s how the story is being told). Originally thought as a way to organize ones own bookmarks the social effect became obvious:

If everyone tags, the “community” profits.

Now, we have del.icio.us. Now we organize our bookmarks with tags. And our photos.
And our books, our games, our software, our tagging sites, and also your bulldogs, if you have any.

However, as we have tagged our whole life, what do we do with it? What is it good for?
I fear the tagging-revolution is about to calm. And I believe that’s because many people don’t see the advantages in tagging. I believe that many many things can be made possible by using tag-based systems. If we realized this, tagging would get some fresh air and eventually tagging gets mainstream.

Is it just me, or is the tagging revolution really stuck? I desperately miss new, visionary, inventive articles on tags.

• To all smart people, where are your ideas?
• To all programming geeks: Where are your algorithms, your “proof of concept” web services?

I could stop here with my article, but, hey, I don’t want to be the grumbling guy that sits and waits for new things coming up, so here I am, trying to expose my brain to you.
In this article I want to take a look at what areas tags are already strong in and how tagging could gain ground in these areas.

Tags help you to organize

When Joshua came up with the idea of tags, it was purely meant for organizing. It was only when also other people started organizing by tags, when the whole idea of “folksonomy” came up.
What does organizing mean? It is like tidying ones room: You put every paper and pencil you have at a place you can remember and seems logical to you so you can easily remember where you have put that thing. Now as we are not limited into physical means when we organize data we have many new possibilities. There are already good articles about this so I won’t discuss this in detail here.

At the end of the day, the question arises: Is organizing your bookmarks by tags really that good?

Just to make the point I come up with another way to remember things: While browsing, the browser could save all pages in a cache and when you are searching for a page you have visited (which is why you originally bookmark the page anyway), you make a fulltext search through all your cached pages. It’s a kind of “Google search” over pages you have already visited. I know this would have some downsides but it would have some advantages too. I often searched for a page I bookmarked and couldn’t remember the tag I used. This problem wouldn’t occur in the “searching through cache” system.

What I am trying to say is: If tagging solely would be for organizing your stuff, it wouldn’t be worth the trouble.

Folksonomy – Classification of the masses

Folksonomy is – as I understand it – the distributed classification of data by the big mass of people who tag stuff. Folksonomy often is said as a new system to build a taxonomy. It’s like building the Open Directory by thousands of people tagging stuff.

What is folksonomy good for? Why do we want to put bookmarks into categories?

Folksonomy enables to explore

Where do you go to start building a new expertise? Is it del.icio.us? Is it Google?
Let’s say your boss tells you that the data your software saves in the database should be encoded. You didn’t think much about cryptography, it merely was a topic that you “should know about” but you were never really interested in cryptography (I’m speaking for myself here.. :-) ). You don’t really know where to start. You know you want to know something about cryptography, but you don’t know exactly what.
A good list of articles or even starting points could shorten your learning curve.
Thereafter, you may want to “travel through the cryptography universe”. And to travel means knowing which articles are related to the one you just read and are so enthusiastic about. You need a map of the cryptography universe, you want to know what is left and right, top and bottom, you want to know everything and everyone related to “cryptography”.
Now then: What would you do?

Do tag systems help you to explore?

Delicious on cryptography

Delicious results on «cryptography»

I would go on del.icio.us/tag/cryptography+introduction. There I find a nice article titled “An Overview of Cryptography“. I guess I’m lucky! If I’d read the article, I’d probably find out which subtopics exist, how cryptography is related to similar issues and so on. You kind of get this “map of the cryptography” universe. But, this is done by only one author. Probably I don’t trust him (probably I should do so, after reading his cv), or you simply do not have time and/or energy to read through 44 pages, although the article looks good. I’ll probably go back to delicious and find out, that the related tags of “cryptography” are:

• security
• reference
• encryption
• crypto
• algorithms
• computing
• software
• nsa
• tutorial
• kids
• education

Now this is not very convincing, is it? You argue:

Yeah, but this is far better that what I get on Google

.

Google on cryptography

Google results on «cryptography»

It is. When looking at this Google results I remember that Google is meant for searching when I already know what I search for. But now I am at a different stage. I don’t know exactly what to search for. I don’t know, because I don’t have any expertise in cryptography. BTW: Google does come up with an article that looks like a good introduction into cryptography as well..

Open directory on security

What about open directory? Let’s give it a try: After typing in “cryptography” I find out that this topic is classified in Science > Math > Applications > Communication_Theory > Communication Theory > Cryptography. Clicking this link you get what you were probably looking for.
You get a nice overview:

Google open directory on «cryptography»

• Algorithms
• Books
• Events
• Historical
• Journals
• People
• Programming Libraries
• Research Groups
• Theory

Now you stand at a guidepost. You see the “cryptography universe”. You probably don’t see what is left and right to cryptography, but here you have a “cryptography at a glance”.
Now it’s up to you: Do you want to explore “algorithm land”, take the shortcut and download the programming library of the language of your choice? Or do you even want to get advice from people that are experts on that matter?
Even if the links provided here don’t give you what you are looking for, here you get a clue what you should look for.

Comparing the three

Let’s compare browsing to a reallife quest: Finding out where your next conference will take place. Say you want to go to the next etech conference, you don’t know where it is and you are not an American citizen.

Ouch, nearly bumped my head into horton plaza!

On the conference websites they often put a map showing the conference place like 10 meters above surface. This map is helpful. But only at the point when you are quite next to the conference.

Help, I cannot breathe out there!

Then, when you desperately search for a more general map, you’ll possibly find a map of how it looks from outer space. Yeah, I know that San Diego is in the US, but I’d like to know which airport is next to the conference.

Distances between observer and data

That’s quite similar to the views we have with del.icio.us and open directory.
Delicious would tell you: “the roads nearby are ‘union street’, ‘Broadway circle’ and ‘Broadway’…”, open directory proclaims: “we have five continents in the world: ‘America’, ‘Asia’, ‘Africa’, ‘Australia’ and ‘Europe’…”. Now, I’m exaggerating a bit but you get the point: Sometimes you need a map that lays between the too detailed and the too general map.
Looking for this type of view is like saying: “I want a bit more ontology than tags but not that much taxonomy as open directory”. That’s where I’ve put the question mark. It’s not that you always want to see the data at that distance but sometimes you desperately want to have that viewpoint.

Now, what has this to do with tagging? I believe that this missing in-between view can be won by analyzing tags.
Have you noticed how flickr does this in-between view?
When you search for love, flickr cluster asks you: “What do you mean by ‘love’?”:

flickr cluster results on «love»

• a couple kissing?
• a mother holding it’s baby?
• a red heart?

“Wait: Flickr is a bit different from del.icio.us”, you say. Yup. Flickr uses a narrow, del.icio.us a broad folksonomy system.
But I believe that the data clusters, flickr creates with it’s narrow folksonomy data, can also be generated with delicious’ broad folksonomy data. I am programming an algorithm that computes del.icio.us clusters. I’m still at an early stage but I get clusters like this “shopping cluster”:

«shopping» cluster

I realize that even if the cluster data is available, there’s the question how to navigate through the data. The “zooming in” and “zooming out” won’t be as easy as with Google maps.
But anyway, here is the land no one has explored before. I think this is the area we should talk about. Here is room for improvement.

Folksonomy helps you to stay informed about a certain topic

Back to what folksonomies are good for: If you have built an expertise in cryptography, you want to stay informed. If RSA is hacked, you certainly want to be informed.
Delicious has got an “Inbox” where you can subscribe to a tag, e.g. “cryptography”.
Each bookmark that is tagged “cryptography” gets in your inbox. That’s a great way to stay informed. Alternatively you have a list of of recent popular sites tagged “cryptography”. You can subscribe to this lists using RSS and hopefully you get informed timely if RSA is hacked..

Do tag systems keep you informed?

I think the comparison with the distance to the data applies here too:
If I’d subscribe to cryptography, I’d probably miss some important items, just because the guy who bookmarked it used the tag “crypto”. On the other hand, I do not want to be informed about another Rijndael algorithm, I want to narrow the incoming links to articles or essays that deal with cryptography.
Delicious already offers to narrow results: I could subscribe to «cryptography» and «essay», and, when delicious will support union (and it will, as Joshua promises), I also could have subscribe to (cryptography or crypto) and (essay or article) but you see that it doesn’t really solve the problem.
I imagine that one day you can say:

I want to keep being informed about cryptography

and the service asks you:

Should I keep you informed about

• new implementations
• new articles/essays
• security issues

And I believe this is possible. Flickr already asks you this when you are searching for love pictures. I guess it will be based on clusters again.

Tags help you sharing Lists

Back to what tags are good for: They help you building lists. Let’s name a few examples:

• Wish lists: I know that numerous online shops enable you building whishlists. But I’d like to have a whishlist that’s not bound to a company, that I can arrange and rearrange. Many are already using del.icio.us as a storage of their wish list.
• Share your bookmarks: A friend asked me for some links to javascript WYSIWYG editors. I gave him a list of all my bookmarks tagged javascript and editor
• Offer viewpoints of your data: Let’s say your favourite CMS features tagging (featured in many of those new fancy ruby on rails applications), I’m not speaking about blogs here: To allow “normal” visitors to view your data, you’ll add a navigation providing starting points to your entries; specific locations a visitor can jump in to so he could take bathe in your articles. Probably you would add a link to all items tagged “references” and “networking” to achieve that.

How can tag lists be improved?

I’m often annoyed that I cannot put my del.icio.us links in a specific order. I did a little script that puts my newest bootkmark at the bottom but it doesn’t fully solve the problem.
Actually I’d like being able to compose a view of tagged bookmarks, i.e. I want to offer a list of all firms our company has built the network for:

Networking references

Big firms

• UBS
• Migros
• ABB

Medium-sized firms

• Stadlerrail
• Räber Information Management GmbH

Small firms

• Citrin Informatik GmbH
• Goldschmiedeatelier Thildy Keller
• Mini Fruits Trading

Nowadays, such a list can’t be automatically generated from my bookmarks, but it could be, by letting me configure my view as myView = (references+networking, "Networking References", (big_firms, medium-sized_firms, small_firms)).
I know it’s not a big challenge to program such a thing, but nonetheless it doesn’t exist, as far as I know?

Bottom line

It appears to me that there’s not been much progress being done related to tagging systems lately. What rather became better is the embedding of tagging systems into already existing technologies such as search. It gives the impression that core issues are done and that there’s no much room for improvement. In this article I wanted to disprove this.
I think that there’s much much more than I have written in here, I even believe that todays tagging applications cover just about 5% of all the possible features tagging makes possible. Thus, let’s gain ground.

I am a big fan of delicious. I’ve got 1.5K bookmarks there, I like it’s spirit and how open everything is. This article isn’t meant to criticize, but I think delicious is facing some problems.

Performance scale

You might have read my article about Tag system performance. To summarize my tests: MySQL is just not built for large tag-systems. It just doesn’t scale. It does scale up to 1 Million items but delicious does have far more posts.
I am pretty sure delicious is still on the MySQL train, this strong believe comes from my performance tests: The mysql-schemas I tested really have the same characteristics as delicious has.
I fear delicious faces a performance dead end: They have put more servers in the mix, they cache quite a bit, it still is slow. I strongly believe that for delicious to have a future it must become much faster. For me this is the number one downside of delicious. I dream of a bookmark service that has billions of bookmark-posts yet it still will perform nicely. I think it is time for new tag-systems to come up. On tagdb mailing list, there are very good ideas how large scaled tagging systems should work (e.g. systems powered by Lucene).

Popular link scale

I think one of the coolest feature of delicious is the popular page. When you read this page regularly you are up to date.. wait: you are up to date concerning CSS tips and firefox and live hacks. You all know that if delicious would get mainstream that page wouldn’t be that interesting any more. It already got boring a bit. As someone put it:

I particularly cannot look at that CSS link lists anymore

I think this page doesn’t scale. It is stuck. And moreover it’s a pity that the coolest page on delicious is not about tags. At first glance you don’t even see what tags a popular link has.
IMHO what is needed here are clusters. Bookmarks go into categories: “browsers”, “programming”, “design” but also “health”, “politics”. When delicious gets mainstream there most certainly will be “sports” or “stars”.
One should then have the possibility to subscribe to certain clusters or better make this subscription automatically out of tags in a users bookmarks.

Bottom line

I think there are some fundamental things that must be rearranged at delicious, otherwise there will be

• a) a big competitor (Google? Yahoo? Microsoft?) coming up or
• b) people will spread to different bookmark services that concentrate on certain clusters. Probably some meta-sites will arise where you can have an overview over all the different sites

I think this problems will arise for every bigger tagsystem. I hope that people will not sniff at tagging systems thinking that they don’t perform well enough..

“writing”, “toread”, “productivity”, “language”

Now what instantially pops into my mind is, that the tag “toread” is quite different from the other tags. In fact it is something I want to do with this bookmark further on. I name this type of tag “adjective” (I will come back to that name later on..). The other tags I consider as “categories“.
Now you’ll probably say “ah, this is a rare exception”. This is not true. I often tag items with “blog” because it happens that the interesting page I found about my favourite hobby happens to be a blog. Therefore I named this type of tag as “adjective” as it is rather a description to the item than it is a category to it.
Other tags used often as adjectives are “reference”, “tutorial”, “fun”, “cool”, “news”, “free”..

Now this categorization is not very correct. Sometimes, I use “blog” not as a adjective. This is, if I want to bookmark a blog that has no content that interests me but it just looks good. Then, I’ll probably blog it as “design blog”. In that day when I redesign my blog, I want to search for all design blogs I tagged..
You see: it lays all in the connection between those tags, not in the tags itself. This is IMO pretty important.

What is that for?

Clusters

You probably tried to cluster your bookmarks by using clusty. What this service does: It tries to put your tags into separate clouds. You know the “tag-bundles” of delicious? This is something like a “auto-tag-bundle” feature. Try it out, if you not already did so and see the problems that arise..
I think the key problem in this cluster-service lies in the fact that this service considers all connections (also the adjectives). But it shouldn’t do so! Adjectives aren’t tags I want in my clusters. Adjectives are spread all over my tags, so they should first be cut away from my “tag-tree” (the tree that is built out of your tag-connections you built by tagging bookmarks).

Similar items

This categorization is also important when you search for “similar” items of a bookmark. When I want to search for similar items of that “how to write more clearly”-article, I’ll search for “writing+productivity+language” and will leave out the “toread” tag (adjective).
Probably this made you realize that categorizing tag-connections is an important task.

Tag clouds

Now there are those tag clouds. When I look at my taggloud then the “biggest” tag is “resource”. Now tag clouds are here to easily find bookmarks (I never search my bookmarks for solely “resource”) or to have a map of your main interests (“what is your hobby?” “ah, I am a big fan of resources”.. :-) I am sure you were also annoyed by that. I want those adjective-tags cut away..!

Synonyms

Now back to some therory: There is a third type of tag-connections: Synonyms. “delicious” and “del.icio.us” are classic synonyms. But I consider “ruby” and “rails” as synonyms too (no, they aren’t synonyms but up to now they are used as synonyms). You type in the second tag just to be sure that you won’t search for the second and find nothing.. I don’t think this category is too important for the cluster-task but I just name it here because I’ll use it further on.

Example

Let’s go for an example.
Lets consider tags that are connected to the tag “ajax”. I gathered some tag-connection-data from delicious (via its rss-feed). And I run a query on my statistical data. This is data gathered during the period of one week. It is not complete. But our experiment will work anyway:

Column “tag-connection” is the tag connected to “ajax” (i.e. javascript), column “weight” depicts the number of times this connection occurred in a bookmark-post on delicious. The tags are ordered by weight. In column “type” you see the result of my computations for this tag-connection. Just to make it clear: These are all tags connected to tag “ajax” ordered number by occurrence of the connection. If a bookmark-post somebody did on delicious is tagged with “ajax” and “javascript” that gives one point for the “weight”-column for “ajax-javascript”.
The outcome is quite good, I think (I must admit that I have taken the example that worked out best :-))
There are some errors, sure: xml-ajax should be a “category”-type as well. But we are looking at the usage of these tags not their “real” meaning (whatever that is).

Computation

Synonyms

To compute these categorization I first went for the “synonyms”. The connection “ajax-javascript” is considered as synonym because “ajax-javascript” is “number one connection” of all connections where ajax is a part of. And when considering the connections of “javascript” (the “vice-versa-connection”), ajax is number two.
I consider two tags as synonyms if “in one direction” the other tag is number one and in the other “direction” the other tag is in the top 10. I made up this rule because I think that in most cases there is one “stronger” synonym that is used most of the time when the “weaker” one is used. The fact that the tag “ajax” is mostly used with tag “javascript” could also mean that “javascript” is a supercategory of ajax (which it somehow is). To avoid that this sub-super-categogy-connections are considered as synonyms, we go sure that “ajax” is also important for “javascript” so ajax is not so sub to javascript.. I hope you can follow :-)

Category/Adjective

Then I compute the “category”. Lets put the values of the above table into a graph.

On the x-axis you see the tags: The tick 1 stands for “web”, 2 for “programming”, 3 for “css”, 4=”design”, 5=”php” and so on. You see I removed the synonym-connections “ajax-javascript” and “ajax=xmlhttprequest” as I think they “disturb” the distribution.
The y-axis depicts the weight of the connection: ajax-web has weight “105″, ajax-programming has weight “100″ and so on.
The black line is the “weight”-column of the table above, the red one is the first derivative, the blue one the second derivative of the weight function.
This graph makes it clear that “web” and “programming” are used quite often in combination with “ajax”, then, there is quite a “gap” followed by the “adjective tail”. I consider the “adjective tail” as connections to be categorized as “adjective”. The tags in this tail are used “out of context”: They don’t really belong to the “ajax-cluster”. They sometimes occur together with ajax, but just sometimes. Mostly not. Therefore they are considered as “adjectives”.
Now the task is to find this “gap”. In my experiments I tried to find the last gap. To find the last gap I started at the end of the tail and searched for the first peak of the first derivative (that is when the second derivative goes from positive to negative) and checked if the peak was high enough. If these to conditions were fulfilled, I snipped the connections into two parts the “pre-gap” connections (category) and the “post-gap” connections (adjective).
The same computation has to be made for the “vice-versa” connection. I considered connections as “category” if one of both computations told that it is a “category”.

Further processing: Ambiguous tags

To achieve good clustering results, I think there is a need of checking if the tag is used in different ways. The prominent example hereof is “apple”. Now, when delicious is still restricted to the blogworld, it is clear that apple means Mac-apple. But in future this may change. To recognize if a tag is used in different environments, the algorithm would have to check the “neighbours of neighbours” (as suggested by Pietro Speroni). That is for ajax: check if the neighbours of “javascript” are more or less the same as the neighbours of “web”. You see that it all lays in the connections between tags. The tag per se is not well-defined but the tag in connection with another tag defines it quite well. Therefore for clustering I’m proposing splitting up amiguous tags. That would add much more simplicity to the resulting clusters.

We are onto something

I’m pretty sure we are onto something. I think this is direction it should go. Computations over tag-connection-distributions are cool. Users shouldn’t insert these infos when posting the bookmarks. Posting should stay easy. I’m not that sure about this “synonym”-computation but I think the “category”-computation turned out pretty good. I tried to build some clusters by hand just by considering the category and synonym-connections and I found a completely detached cluster consisting of the tags “cooking”, “health”, “recipes”, “diet” and “food”. As I said, I think we are onto something..

Further reading

• Building tag clusters by hand
• Pietro Speronis different approach to clustering tags (with java-mindmap-visualisation!)