the Design Experience Weblog

Don Norman says google isn simple, it just hides complexity. I think this is not quite the whole truth. Norman goes on to say that MSN and Yahoo, for example, are simpler because they show you all your options on the home page. I seriously doubt that Norman recommends his clients show every possible option on their web site home pages.

He also says, to get to google news you have to click several times. When I want Google News, I visit http://news.google.com. No additional clicks required. When I want a map, I got to maps.google.com. When I want to check my email I go to gmail.com. There are seperate services, all provided by one company. Google is not trying to be a portal, although you can personlize the home page. This actually refutes Norman's entire premise, since he commends Yahoo for offering a customizable home page where you can get news etc on the main page.

Another way google helps is by "knowing" what you want. If you search for something, it gives you the right thing most of the time, without your having to know if its from the "news" site, or "maps". If you search for an address, the first search result goes to Google Maps. If I search for "news about hurricane katrina" the first result goes to google news. Seems to me that Google is simplfying things by giving me what I want, without my having to choose from hundreds of possible links on one page. This leads me to recommend a book, Don't Make Me Think, by Steve Krug.

Lou Rosenfeld picks up on the Long Tail idea and applies to to search results. He notices that few searches are very common, and there are a large number of less common results. Most often an information artchitect or someone else will optimize the results for those most popular queries. Let's not forget about the rest of the queries. What is missing in those unanalyzed queries and results. It makes sense to get the big picture about what your users are searching for. He says perhaps studying the search queries long tail would help use find hidden users and potential market segements, ongoing stable information needs, as opposed to fads, and learn about rising trends by analyzing the long tail over a period of time.

Sounds like its a good time to do interesting work with search. Luckily I am developing the next (or first in some cases) generation search engine for OpenACS. Adding best bets, and other cool search features is a little way off, but having good tools to look at past search queries, and what sort of results users got when they tried to search seems pretty valuable. Obviously the value depends on what type of site you are running, but overall users needs to find what they are looking for no matter what kind of site it is.

I was reading the Polar Bear Book and just starting thinking about their various definitions of information architecture. I like the definitions about the goals or IA, that is, making it possible to find and use the information you need. This is somehow related to knowledge management, besides being a buzzword, of course.

IA stresses categorization, and metadata, usually of a more formal nature. And KM also has some of those aspects. I would like to see how to take more organic techniques for finding data such as Google, tags, and RSS and make them fit into the concepts of IA. Knowledge management seems to have a little more thought in this direction, in the area of Personal KM, usually related to weblogs, and more now with e-portfolio type applications.

I searched for "information architecure knowledge management" and found a course of study with the same name. It seems to combine library science, computer science, visual design, and communications. I'll have to explore that some more. I like the idea of combining visual design with the other subjects. This reminds me of Edward Tufte and the basics of designing information visually.

John Hiler at Microcontent News gives a great overview of the evolution of hierarchy vs. search for finding information. I have been thinking about this since I read The Inmates are Running the Asylum.

Well, really I have been thinking about it since I have had to organize large amounts of email. When I started filing things in folders, it was a pain to find something. Once I saw gmail, I adapted that model to my IMAP email with mutt. Now I have everything in my inbox, over 3000 items now, and search. It is still working fine. Maybe its not the best solution but it works for me.

Adapting these ideas to content management, organziation, and findability of web site data, I have been using search, combined with categorization, and other metadata filtering of search results. I think this is a good way to go, but the trick is figuring out what metadata is useful, and how to present it in the user interface.

I was reading and had an idea about user entered tags, and other user-derived metadata. I am referring to data such as "users who viewed this object also looked at these objects" or "users who purchased this also purchased that" sort of information. Or "user X says this list of items go together". It is another way to tagging, I supposed, figuring out what items are related by passive means (at least with the first two examples.) Basically just poking around in your database for interesting correlations. Amazon.com is obviously the leader in this space. I can imagine a system that can take advantage of freeform text tags, and user-derived metadata would be very powerful. The focus would be on intelligent analysis of the data. Or maybe not. Perhaps just making the data available is enough to inpsire creative use of it.

"f you want to do something that's going to change the world, build software that people want to use instead of software that managers want to buy." - Jamie Zawinski

Art Rhyno suggests instead of trying to get library backends to work with the we, that there is a public interface that can synch with a web applciation. He suggests that RSS might do the trick to synch the internal catalog with a public web application.

Using RSS or any other noticiation system to synch the interesting catalog data for use by the public would be very exciting. Already my local library can output the books I have checked out, and search results in XML, but its not pretty. There also isn't any sort of standard format that works across platforms. So I could work on an application that shows the books I am currently reading according to my library records, but it would only work with a library with the same platform.

Anyway the post was more about a public inteface to the catalog. I would love that. It would be great to aggreate comments and reviews and link them to a local library catalog. There all all kinds of exicting options. Art says "the trick is to create sustainable web representation of the contents of the catalogue, with dynamic hooks for status information "

Today I read two interesting takes on using tags for categorization and places where they might need to be enhanced or might not work.

Folksonomies: How we can improve the tags from Lars Pind and The Obligatory Folksonomy Post from Michael Feldstein.

I need to dig into these and come up with something useful to say. Right now I'd say that they both mention looking at the resource first, and finding out how different people have tagged it. That is the opposite of finding what other resources have the same tags, and could produce another set of useful metadata.

Robin Good explains Why RSS Search Feeds Based On Web Searches Are Important, partially in resposne to Stephen Cohen's Why RSS Feeds For Search Engine Results Are Useless.

Stephen says that since the top 10 results for a particular query don't change very often RSS is not useful for results from a search engine such as Google or MSN. I don't think you only have to look at the 10 ten results of course. There are probably some interesting items in the top 50 or so. I don't see any reason you can't ask for the second or third page of results as RSS just as well as the first.

This is probably more true if what you are searching for is covered by regular news media, or weblogs, or basically any web site that has regularly updated content. You can't dismiss search based on the fact that a basic query will return fairly static results. You can try an advanced query. If you restrict to pages that have changed in the last number of days or use other criteria to narrow the results I suspect RSS would be more effective.

Robin seems to focus more on the fact that somewhere in Stephen's post he mentioned new sites such as news.google.com. He mentions that there are many valuable sources besides news outlets, and this is true. He does also address the fact that simple queries will not return different results, and of course, you will need a more complex query to get valuable results.

This reminds me of the work I did for a client recently where the entire web site emitted RSS to share data with other affiliated web sites. The RSS feed generation was built on an advanced search concept. Full text indexes of content was one search criteria, but in addition, language, date, and a categorization system were also used to generate RSS feeds.

Why not return search results as RSS? Its as good as any other XML format. And who knows what interesting uses for the data can be imagined once its in a easily machine readable format.

Over at web-graphics.com is another post about user defined tags. Andreas asks, now that we have these tags, how do we associate them with the content. There hasn't been a push to semantically refer to the tags in the HTML of the pages where the tags are used. Usually they just show up as a list of links to categories. It will be interesting how this is solved. I suspect some variant of Dublin Core will become popular similar to the way it is used for defining trackback data for weblog entries.

This is interesting. Suggestions for what tags people have already used. This is pretty much what I was talking about in the last post, comparing documents, but instead of reading the documents, just compare the URL and previous tags. Similar to "people who bought this book also..." idea at Amazon.

Louis Rosenfeld suggests that user supplied tags, popularly mentioned as "folksonomies" lately, would be a good complement to controlled vocabularies. This makes sense. User supplied tags would be useful to figure out how real people think about the categorization of their content. This feedback from users could drive the definition of the controlled vocabulary.

One thing he says is that right now there are 600 photos at flickr tagged with the word "summer" and that after a while there will be 6000, or 60,000 and it will be difficult to pull out the ones that are relevant. Why just use the one word? What if you searched on "summer beach" or some combination of terms? This is how I use del.icio.us to manage links I want to remember.

I am not sure if I use the tags at del.icio.us in the same way as everyone else. I don't make up new tags by combining words, but just put in 3 or 4 words I think will help me find the items in the future. This is some sort of wacky variation of facted classification I think. Where I take out the most important aspects of a item and pick the word that defines that aspect. This means if you look at the list of tags I have, I have a huge number of unique tags, some with only 1 or 2 items in them. But if I search for an intersection of tags I can narrow it down quickly.

Anything that gets people to think about metadata is good. Maybe once you manually tag a bunch of items, the computer could examine those, and compare them to new items and suggest potential tags for the new items.

Tim Bray's series On Search is a reasonably concise collection of the most important concepts in web full text searching. I recently rediscovered it, and felt it was time to read the whole thing.

One particular entry that was relevant to my current thinking was about metadata and search. He mentions that collecting metadata from humans is difficult and if you want to try, don't try to collect too much. I think this is where flickr and del.icio.us are very smart. The freeform tag approach, when used on large enough scale, does seem to allow for easy, useful metadata, particularly on subject on web pages.

It also mentions automatic categorization, and designing an API for managing search related metadata. All areas I plan to work on for OpenACS.

Back to reading for now...

I have so far written two customized advanced search interfaces for the OpenACS search package. With one, it was easy because all searchable items are subtybes of one object_type. In addition it is a content repository content_type so we have one unified view to query for additional search attributes.

So, what I have is a hard coded list of columns that are legal to add in for advanced searching. This works fine, but I imagine I can generalize it a little bit by adding a table to configure the advanced attrbutes. This way a search keyword could be tied to a query fragment to add to the SQL query for the search. It would specify what table to join with, and also any optional order by query fragment.

It makes sense to register a tcl procedure for that, instead of actually storing the query bits in the database. That way the callback could refer to an XQL file for a database specific query. Actually it might end up as a search engine driver specific query.

In this case a table probably isn't needed. Just a magically named tcl procedure, such as search::advanced::${keyword} where the existance of the procedure itself would implicitly "register" the availabilty of a keyword. It could even have a search::advanced::${keyword}::help_text subcommand to return some documentation for users. I think a possible first set of necessary procedures is:

search::advanced::${keyword}::query_fragment
search::advanced::${keyword}::join_fragment
search::advanced::${keyword}::order_by_fragment
search::advanced::${keyword}::help_text

This should also be extended to allow full text indexing of additional columns besides the "main content" of an item. I have tested a setup where title, description, and content each have a sepearate full text index column. Because the query querying of the full text index is abstracted through the search driver, there would need to be a way to abstract it from the advanced search client as well.

I'll keep thinking.