As a consequence I just considered solutions that don’t make me decide which page I want to store and which not. Everything has to be stored automatically for me.

Guard your web entry points

Alternative 1: Guard all your web entry points

I guess everyone has about 5 web entry points where he starts 95% of his web browsing. My web entry points are:

• Bloglines
• Google
• programming.reddit.com
• links shared by colleagues over Skype
• links I get via mail

If all those web entry points are searchable I can track down 95% of any web site I ever visited – granted that I remember how I got to the web site I’m trying to recall (which seems no problem to me).

Of all my web entry points only bloglines has no such thing (I know that google reader has it, if only it could store password protected feeds). Google has google search history, reddit has a – although slow – search (Google’s always an alternative) and it allows me to list all my up/downmodded links (which is quite probably also something I remember). Skype has got a good search (if I remember in which channel the link was posted, which is often not the case) and finally there are enough online mail providers (such as Gmail) and server side mail programs (such as Zimbra) available that offer good search facilities.

Pros/Cons:
Searching for things where you found it on seems very natural to me
you miss the 5%: The link your colleague told you, the news site you check too irregularly to remember, the url you typed in from an advertisement
the refind process is often quite cumbersome

Google browser sync

Use Google browser sync to synchronize
firerox browser history between
your different computers

If your problem is that you use different computers – one at home, one at work – and therefore lose your history every time you switch computers Google browser sync might be a solution. This firefox plugin syncs your browser history, (that means when you start typing into the url field it behaves the same on all synced computers) your bookmarks and optionally your passwords. In the end it really feels like all firefox installations have the same state.
You might scream “privacy!” at this point which is solved quite well for my taste: You can encrypt your browser data right from when it leaves your browser and it is stored encrypted on Google’s servers. They promise they don’t decrypt your data using your password which seems quite fair to me.

Pros/Cons:
it solves recalling links within the last few days (which might be just what you want)
you can’t go back more than the number of days back you set your firefox history settings (which you might set to 999 days). This might not be a big disadvantage as the value of a bookmark vanishes over time: As soon as you forgot that you have bookmarked a certain page this bookmark is of no value to you anyway (I quickly skimmed through 100 of my delicious bookmarks from 2 years ago: I remember only 8% of the pages I bookmarked)
your search in firefox history is restricted to just the page titles – which might be not enough (and too many web sites still have meaningless titles)

Google history

When you mark ‘PageRank and Site Info’
then your browsing history is transmitted
to Google and allows you to use ‘Google history’

When you have google firefox toolbar installed and you have pagerank activated in your options your browser sends every loaded url to a Google server. Then on Google history you have a fulltext search over all the websites you ever visited.

Pros/Cons:
having a fulltext search over all visited pages is pretty neat.
it is a privacy nightmare. Google will use this data to improve their services.
your searches are restricted to the public web. Sites from your intranet or any service you need to login (such as your feed aggregator) are not searchable.
the web site you visited might have changed (such as the reddit frontpage or your local newspaper site) and the web site as you saw it is no longer in Googles indexes
the search is quite slow (compared to google web search)

Google desktop search

“I remember I saw it on programming.reddit.com…”
Google Desktop Search shows a history of all
my visits to programming.reddit.com

Another one from Google: Google Desktop is known for it’s capability to search files on your computer. Apart from this, Google Desktop also stores all the web pages you visit with your browser. That is the content of the page you really looked at, not the page that is in the Google cache. And this is quite an improvement to “Google history”:

1. You can search in an index that contains things you actually saw on that website. No matter how short something was on that website, this is now in your index. You remember you saw something on programming.reddit.com but you didn’t click on it? Search in your local google desktop cache and you have access to the html of all your visits to programming.reddit.com.
2. Your search doesn’t end at the “login screen”: Google desktop holds cache of your corporate wiki and your internal bug tracking system.
3. Once you have it, you’ll never lose it: Documents get deleted from the web or they get moved behind a pay-only wall. You won’t lose this valuable information as it is stored safely in your Google Desktop index.

Google desktop lets you access the data in index in many ways:

• In case you remember some exact words of the web page you’re looking for you can simpy search and order by relevance/date
• In case you remember the date you can browse through the history of that day
• In case you remember how you got to that web page you can search for the referring site and visit the browsing history to find the page this way

And it solves the problem with multiple browsers with the ability to sync you indexes via a Google Server (Windows only, Linux is still lacking that feature, you can ‘vote’ for that feature.)

Pros/Cons:
Gogole Desktop is “Google history” without the disadvantages
The software is proprietary, you don’t really know what’s going on, you need to trust Google (there are alternatives: Beagle (Linux only) and Slogger (just saves your browser history, lacks a search frontend)
Your history data is locked in. The cache lives in a binary file on your harddisk

That said I still find Google Desktop very neat. Although I just have tested it for two weeks it greatly served me for accessing web documents more quickly and without the saving effort needed on online bookmarking services.

Clarification

According to comments here and on reddit, it was obvious that my intention of this post was somehow misunderstood – partly because of the original misleading title (was: “.. – 5 reasons why social bookmarking doesn’t work”). Maybe these adaptions from an xkcd comic does clarify:

Right tool: Use bookmarks to get things done

I think, derefr sums this up very nice:

I find a GTD approach works well: what next action are you going to apply to this bookmark? If it’s just “well, it was neat!” you have no reason to save it (perhaps share it, but not save it), and can throw it away.

The same goes for using the tag “mycomment” to follow up discussions you’ve partaken or “toread” to know what to read once you’ve got some free time. These bookmarks all serve a purpose that is clear to you while bookmarking. This also helps you picking an appropriate tag. No critique on that one.

Right tool: Sharing links

It is clear that bookmark sharing sites such as reddit, Digg, or Stumbleupon that all focus on link sharing have proven that this concept works. Delicious, Simpy, Magnolia et al. all have features to help you share your bookmarks. No critique on that one.

Wrong tool: Remembering potentially interesting links

This is what this article is dealing about: Saving bookmarks that are not useful to you now but – without yet knowing what you’ll use this bookmark for – you save it because it is potentially interesting in the future. I think that doesn’t work and the 5 points should prove that.

Reason 1: You can’t foresee the future

Deciding which web site will be valuable in the future is a very very hard task. I’m not too good at it. I pile up tons of bookmarks I never look at afterwards and on the other hand I decided to not bookmark sites which I needed afterwards. In fact I’m so unsure about my ability to bookmark the right pages I often don’t try searching for a link in my pile of bookmarks but instead google first because I expect being faster this way. Too often I searched my bookmarks altering tags and search terms and didn’t find the bookmark in the end.

Additionally: Even if I would know which links will be of interest in the future, I can’t decide how I should tag (categorize) my bookmarks. When I tag an article, I normally have skimmed it and while categorizing I look at its title. When I tag I’m in a completely different situation – information wise – from when I search for the link.

Your categories may change when you get
familiar with a product or topic

Your information level when looking at a document
differs from when trying to recall that document

Reason 2: You tear links out of its context

Bookmarking is like cutting passages
from books: you remove information
from the context you originally found it

The word “bookmark” relates to the pretty carton markers you use when reading books. Although the way it is used in the web is far far from what it means in books lets delve into that comparison a bit:
To go sure you will be able to find an important passage once you finished a book, you underline or write a few words into the margin to outline a paragraph. Then, when you recall that great sentence you most certainly know in which book it was written (unless that book is a conglomeration of quotes). Then, you often can remember the way that statement was used in the argumentation and in what topic it was embedded. And finally, amazingly, your brain often tells you where on a page (e.g. bottom left) the searched sentence is written. So you normally get quite a bunch of context information to guide you in your search and you will find the wanted sentence within a short amount of time, even if it wasn’t underlined. And even if you don’t find it, you often have a good time reading through the other amazing statements and end up quoting something you didn’t intend.

The way bookmarks are handled in the web would mean to books that you tear out that sentence out of the book, stick a few colored post-its to it and throw that snippet onto the pile with the 1325 other quotes. Bookmarking means taking information out of the context you originally found the information in. On the web context means how you found that link: Was it on Google or in your feed aggregator? Was it a blog post of one of your colleagues? Was it in an email? I often remember these things. Without being a psychologist or having an education in these things I guess our brain is pretty good in remembering context. So why don’t we use techniques that help our brain instead of trying to replace it?

Reason 3: It takes too much time

Bookmarking should save you time – and frustration. Leaving out the frustration bit: Does it really save you time?
Lets say it takes 10 seconds to categorize a bookmark and lets say you’ll use every 20th of your saved bookmarks (which are rather optimistic guesses). That means that when trying to recall an url from your bookmarking service you need to be 200 seconds faster than when you didn’t bookmark any pages at all (as it took you 200 seconds for bookmarking the 20 bookmarks out of which you used 1).

I’m pretty sure you won’t save over 3 minutes in average searching in your pile of bookmarks compared to thinking for halve a minute where you found that link and then going down that trail. So: Why the hassle?

Reason 4: It didn’t work for me

I tried it. I gathered 3444 bookmarks in 2 years using 3034 tags. I asked myself how I could change my tagging practices to improve the recall. I failed. I gave up. I cannot believe there’s no one out there feeling the same.

I stopped bookmarking nearly two months ago. First, when reading articles that felt so interesting it was hard to not bookmark them. Then, it was kind of liberating not having to think “is this page valuable in the future?” “what tags should I use?”.

I never missed it. I always found that link. I don’t regret.

Reason 5: Social bookmarking won’t improve that soon

You may argue that there soon will be techniques to overcome the problems I just mentioned. But my claim is that social bookmarking sites won’t improve that soon.

In my last post I asked: “Why is tagging stuck?”. Gene Smith argues correctly that tagging isn’t stuck. He continues:

Want to know what is stuck? Del.icio.us

The same is true for all the other social bookmarking sites. RawSugar did a brilliant next step (before it went offline) but the social bookmarking market is quiet ever since. I couldn’t find fresh ideas in delicious’ current redesign. It seems like they moved buttons from here to there. I hoped they wouldn’t just redesign the appearance but would also change the way users interact with their data.

So, I guess these services are just as good as it gets. No improvements to wait for. That means it’s our – the users – turn to change our habits, to find the right tool for the job.

Paolo Valdemarin:

4 years later I’m still wondering when will we get some truly advanced tagging tools.
Where are all these tools to manage all my tags (on Flickr, on del.icio.us, on technorati, in my RSS reader, on my blog, etc), to help me organizing them, to allow me to gain more advantages from tagging? (maybe they are somewhere and I simply have not found them yet…)

Matt Mower:

I have been surprised, that [...] the state of the art in tagging seems firmly wedged in 2003. Surprised because there seemed [...] to be a momentum building in the use of tagging

David Weinberger:

Tagging like it was 2002

Thomas Vander Wal:

In the consumer space thing have been stagnant for a while, but in the enterprise space there is some good forward movement and some innovation taking place
[...]
While there are examples that tagging services have moved forward, there is so much more room to advance and improve. As people’s own collection of tagged pages and objects have grown the tools are needed to better refind them.

Vander Wals post is very very insightful and worth a read: He sums up the tagging history and expresses a few brilliant ideas how to proceed.

The delicious lesson – revisited

The big question remains: Why is tagging stuck?

My suggestion is that we may rethink the delicious lesson: Not in terms of “is it true that personal value precedes network value?” but in terms of “what is the real benefit of the users?” or in other words: “How can we design the itch that causes users to generate valuable metadata?”

Recently I talked with Cédric Huesler, a coworker of mine about his use of del.icio.us: Instead of using delicious for storing his bookmarks for later retrieval he stores them to exchange links with strangers. Indeed he has 19 regular consumers of his bookmarks, 7 of these users he is consumer as well.

He doesn’t store his personal bookmarks at all. He can recall from memory where or how he found a certain website and goes back to his google history.

There are just a few entry points into new information on the web: there is Google, feed aggregators or frontpage sites. When there are good search utilities in those tools who needs bookmarks? I must confess that searching at those entry points feels more natural to me than remembering the exact tag I used.

Let’s put it straight: Using tags to find my bookmarks later just doesn’t work. I give up. And no, it’s not just the lack of good tools that help me going through my bookmarks to reorganize them. I won’t do that for all my 3444 bookmarks. And no, this won’t be solved with better tools to refind my items. What do you want to throw into the mix? Fulltext search and time based drill-down? This has nothing to do with tags.

So, we might have to rephrase the users motivation to tag, as I don’t think Joshua Porter was right when he wrote:

in order to gain more personal value, they use tags to be able to find their bookmarks later

I’m not yet at the point where I could correctly rephrase that statement, but I think Cédrics approach in using tags not for personal recall but for publishing is worth a thought. The value therein is close to the value of blogging: You get attention and you communicate. And that’s what the web is about, isn’t it?

Address side of the gift. It is remarkable
that this mail got delivered, as it wan’t
wrapped in an envelope or so!

When Julian Cash told me in an email that he was very gateful for the paper Frank, Grigory and I wrote, he asked me for my snail mail address. I wondered what he wanted my address for, then one or two weeks later I found a block of plexiglas in my mail.

Thanks a lot to Julian for showing such appreciation. It showed me that my labour has implications to real life too. Holding a physical gift in my hand, coming from work done in a virtual blogosphere feels special.

Btw: He also thanked Frank for his part by sending him an inflated ball.

History of tags

Update September, 2007: Thomas Vander Wal wrote a very good roundup on the tag history.

Gartners hype cycles applied to tag history

I think gartners hype cycles prove to be right when applied to the tag history (hype cycle descriptions taken from Floor eTrends):

Technology trigger

A breakthrough, public demonstration, product launch or other event that generates significant
press and industry interest.

The technology trigger most likely was del.icio.us and subsequently flickr adding tagging to their service.

Peak of inflated expectations

A phase of overenthusiasm and unrealistic projections during which a flurry of publicized
activity by technology leaders results in some successes but more failures as the technology is
pushed to its limits. The only enterprises making money at this stage are conference organizers
and magazine publishers.

In this phase there were indeed many blog posts talking about this subject, as Louis Rosenfeld
put it:

Lately, you can’t surf information architecture blogs for five minutes without stumbling on a
discussion of folksonomies

I guess in this phase many people said things they now feel embarassed about.

Trough of disillusionment

The point at which the technology becomes unfashionable and the press abandons the
topic, because the technology did not live up to its overinflated expectations.

This is the phase we’re in now. There are no blog posts any more. Tagging is not really
unfashionable but the topic is “done” à la «if that’s all what’s tagging adds to the web experience, I’m not interested in this technology any more». There isn’t much thinking and innovation going on.

Slope of enlightenment

Focused experimentation and solid hard work by an increasingly diverse range of organizations
lead to a true understanding of the technology’s applicability, risks and benefits. Commercial
off-the-shelf methodologies and tools become available to ease the development process.

Let’s hope gartner is right about the future of folksonomies!

Plateau of productivity

The real-world benefits of the technology are demonstrated and accepted. Tools and
methodologies are increasingly stable as they enter their second and third generation. The final
height of the plateau varies according to whether the technology is broadly applicable or only
benefits a niche market.

It has yet to show if folksonomies such as in del.icio.us or flickr prove themselves for the masses.

Update (September, 2007): Do folksonomies apply to hype cycles at all?

Joe Lamantia raises the question if tagging should be applied at all to Gartners Hype Cycles:

Tagging in fact shows few characteristics of the enterprise technologies that Gartner’s Hype Cycle is built around

Joe argues rightly, that tagging has not yet reached the broad economy, it’s not that Gartner would care to apply folksonomies to their Hype Cycles.

Although: Gartner apply the hype cycle to technologies such as “corporate blogging” or wikis. It seems it does not lie in the nature of tagging that it won’t ever apply to hype cycles, the only fact that hinders Gartner to apply tagging to their hype cycles is that there is no money earned with it. I’m not into business analysis at all so I am grateful for Joes insights which he concludes with:

If it doesn’t cost money, the perceived risks of the technology are lower, and the big analysis firms pay less attention, because their customers see less need to pay for analysis

Local.ch is a local search engine for Switzerland, that means I can now work on information retrieval related stuff full time – which was what I did in my free time already. Being paid for doing the things I like is a gift I don’t take for granted.

The R&D team consists of about 10 people – all very talented and smart. Plus, the atmosphere is friendly yet challenging.

Say bye to tag clouds

Then, I’ll give a talk at webtuesday, Zurich about "Improving navigation in tag spaces": Why tag clouds don’t make much sense, why
tagging lost its ground and what could be done to improve the users experience.

The talk will be based on the few blog posts I wrote about this subject plus some newly gained insights.
If you’re living near Zurich it would be a pleasure to meet you there.

I’m sad to hear this as I was a happy user of RawSugar. It was the only tag based service that didn’t just copy delicious to add a new features or polished design.
Apart from a few academic papers and proof of concept applications I never saw anything new in the tagging world until Rawsugar came up with its implementation of tag clustering and auto-tagging. These features probably need some more polish but they were steps into the right direction.

It’s not hard to decide where I should move all my bookmarks: Simpy has nice search features, a friendly user base and a very likeable developer. Otis does care for Rawsugar users and implemented an importer.

Apart from that decision I’m left to ask: "what was wrong with RawSugar?". I think this is an important question as tagging is web 2.0 and web 2.0 most certainly is another bubble going to pop somewhen – the question then is who is going to stay.

what was wrong?

Knowing that I’m not in the best position to analyze, I have a go figuring out why this project failed. Some of these objections I already mentioned, some can be found reading some (sometimes very mean: has techcrunch turned into digg?) comments.

user interface

The web site has always been too cluttered. The page structure even grew with the number of features. Rawsugars biggest competitor, delicious, was always very reluctant in exposing new features to the user.
Then its whole look doesn’t really fit into the web 2.0 world (Ok, delicious isn’t that much better, before the redesign about a year ago it looked like plain xhtml without any styling). It’s not about rounded corners, it’s about freshness. RawSugar always felt a little bit heavy (although it had the best response times).

interaction with community

I always felt like RawSugar doesn’t have the early adopters in mind: For questions it did use a forum, its developers weren’t really involved in the blogosphere. But then they wanted to gain ground in the blogosphere. It very much focused on a feature called "tag search for your blog" (you can see this feature in action when you switch to the index view of this blog, it is a very good feature indeed). This just didn’t really fit. It didn’t feel they were "of us". I hope you get what I mean – I cannot really describe it

ah well..

Maybe these arguments are all in vain. Why did delicious take off? Because it was the first social bookmark service (in fact it wasn’t)? There are many things Rawsugar did right. Why didn’t the user base grow more substantially? I’m keen to hear your thoughts!

Anyway – best luck to all the guys involved in RawSugar. Especially Frank – I did have a great time with you preparing our talk for www2006!

That’s the feeling that arises when I think of all the bookmark services out there that ought to file urls I somehow find notworthy so that I can quickly recall them afterwards: There’s too much clutter and the services are just too slow. Therefore I again begin to save my bookmarks at other places: In firefox or in some text documents lying somewhere on my hard drive (probably I should tag them?).

«I want it now»

As for myself, if I don’t get my data within a fraction of a second, I don’t feel that I have good access to my data. I think one of the best things of Google Web search is that it is incredible fast. Tag services are not. That’s a pity because within tag services I search my lowly amount of bookmarks (around 2000 bookmarks at that moment) whereas with Google I search the whole net. I often find myself at the situation searching for a url on Google rather than looking it up on my bookmark service. Even if I have to do 3 Google searches – I feel at control. If I have to wait for 3 seconds for an answer, that’s simply too much.

An experiment: I ran for ((i=0;i<10;i++)) do wget -q --user-agent="Mozille Firefox 1.9" $url; done; for the three big players in tagging services: 181del.icio.us, 182Simpy and 183Rawsugar. For $url I chose a page displaying all bookmarks saved with a certain tag. I also ran a query on Google for that “tag”. The numbers are number of seconds per query. The queries were run on different hosts with different internet connection, all in Switzerland.

I’m happy to see that with Rawsugar I’m with the fastest bookmark service (2 times faster than Delicious, 3 times faster than Simpy). On the other hand: all the bookmark services are at least 4 times slower than Google. I know that Google has set a high mark – but emotionally that’s the response time I’d like to have when querying for my data.

«I just want my bookmarks»

Have a look at the “result areas” (highlighted in green) of the three bookmark services.

Delicious: Result area in green

Simpy: Result area in green

Rawsugar: Result area in green

To come back to the comparison with the closet: Even if there are 20 magic hands with documents sticking out of the closet, it is crucial that the most important document is the one closest to my face. That means, when using those bookmark services, my eye should first notice the most important link. My opinion: The service that solves this best is Delicious. The results appear on the leftmost part of my screen and the result area covers 64% of the screen. Regarding that matter Rawsugar is far worse: My eye has to search for the first result. I find it natural to start reading at the left of the screen. But at Rawsugar there are so many links in the head of the page and then the left column helps me to refine the result – a great feature but at the wrong place – furthermore: if I want just get a link I know I filed under “important”, I don’t need this at all. Even Delicious’ “saved by xxx other people” – shaped in different colorings – is too much clutter for me. Maybe I’m a puritan but – what the heck – I just want my 9 bookmarks!

Beyond the criticism

Sorry to only criticize. I like those services a lot. They are helping organizing links and they are for free. To turn this post into a constructive comment, two possible solutions to the problems mentioned.

Bookmark services offer a «Minimal mode»

Yes, tagging is about collaboration. But I fear the personal value of tagging is too small – people might leave tagging services because they don’t feel their bookmark problem solved. There’s too much network and too many features. I propose a “minimal mode”, a result page that just shows select * from bookmarks where user="phred" and one_of_its_tags="important". In firefox there are 184Smart Keywords also known as “quick search”. I set up “myr” to search for my rawsugar bookmarks. Typing “myr important” into my location bar I end up on www.rawsugar.com/links/phred/important. If there would be a site on rawsugar that displays the same results just in this minimal mode, I would bend my smart keyword to this result page and would be happy.

Someone codes a «Tag agent»

I imagine a “tag agent” that has incredible response time and no clutter. It could either get the results via rss/api from the bookmark service or it could hold all my tagged bookmarks in it’s own database. Such an agent could be installed on a password-protected part of my website so I can access it from wherever I am. The problem is: I don’t have time to write an application my own. I have thought about such an application, I’ve got plenty of ideas how it should look but I’m afraid I won’t find time to code such an application. I repeatedly get emails from people writing yet another thesis on collaborative categorisation. Instead of writing about insights on the mental actions taking place while tagging – do the tagging world a favour and write such an agent. If there’s such a tag agent already, please leave a comment.

185186187188

After the presentation at the tagging workshop at www2006 we felt the need to give our paper a more www-friendly, I-don’t-want-to-read-through-those-theoretical-equation-flooded-papers face.

So, here you go: Automated Tag Clustering: Improving search and exploration in the tag space. To read this document you should have a clue what tags are about, you should also know some tag services as delicious or flickr so you can understand the limitations these services currently have. If you don’t want to read through the whole papers, the numerous figures give you a good summary. Finally, to wet your appetite, here a few excerpts of the document:

Currently tagging services still provide a relatively marginal value for information discovery and we claim that with the use of clustering techniques this can be greatly improved [from introduction]

The whole promise of collaborative tagging is that by exploring the tag space you can discover a lot of useful information you would not find with traditional search engines. When your information need is not well defined, the idea that you can explore and see what other people tagged with certain tags is very attractive. We believe that tagging will be able to reach a very wide audience only when exploration techniques will be effective. [from limited exploration]

Although a great visualization paradigm, we believe that with today’s tagclouds it is hard to find more than one or two tags to click on. Tags are not grouped, there is too much information, so that you find lot of related tags scattered on the tag cloud. One or two popular topics and all their related tags tend to dominate the whole cloud. For example, looking at the del.icio.us tagcloud, one would mostly see tags related to web design and technologies. This is because these topics are overwhelmingly more frequent than anything else. [from limited exploration]

Tag web2.0 nowadays is so popular and is combined wildly with anything. In fact this tag is so overused that if you look at tag bookmarks in the del.icio.us dataset, the most used cotag is web2.0[...]. Basing tag similarity on these numbers often doesn’t make sense at all. The similarity measure should be chosen so the popularity of a tag doesn’t affect the set of a tags related tags. Don’t cut the long tail. The success of blogs is driven by the importance of the long tail. We all know that it is crucial to support the niches. Tagging applications should empower the long tail too. If you just sort by popularity, you’d loose all those niches. [from choosing a similarity measure]

We’d be happy to get any kind of feedback on the article. Just post a comment to this blog post.

Edit (4 years later!): A few guys asked me about the source code: Source code with syntax highlighting, download.
You need kmetis to make this run, see usage() to see how it should be used.