It's been several months since full text indexing was made available for searching diaries, stories, or comments here at Daily Kos.
Yesterday I was reminded that not everyone is aware of the new search system, and that the default settings for new search may not be to everyone's liking.
What I'm hoping to achieve with this diary is better insight into negative experiences with search. Also to convey why I think our new search is worth the effort of learning how to use it, with examples. Along the way I'll mention several newer search utilities or tools that may be of interest in this political/blogging community.
Since becoming available in February over 3600 Kosmopolitans have used new search, on average 2500 times a day.
But of those who've tried, over 800 used it only once. Recently, at my request, ct installed a 5 second "time out" to prevent runaway search jobs due to rapid submit key pounding. This may be a measure of user frustration with search. It seems there is a good likelihood of many disappointing interactions with the new search, along with the more positive ones.
My claim is that the new search system is better than scoop built in search (aka old search), generic web based organic search a la google/ask/yahoo, and even the latest and greatest site focused, blog centered, date range restricted search available (cough ... google blog .. cough). For all sordid details on the new search system, see these diaries: New Supple Search Working, I've been a searching ... for a Heart of Gold, Comment Search Announcement.
The short story is Scoop search was broken and couldn't get up, google et al don't index fast enough to keep up and know nothing about important local metadata like numbers of comments and recommendations received, and even the latest and greatest can't match our local knowledge and frequency of update. Take a look at the search comparison below and see if you agree.
The downside of new search: we have one report (mentioned above) that search is hard to use, and that the documentation is too long and too hard to read. Perhaps the interface may need some changes in order to make it more instantly accessible and useful? Is this others experience as well? Constructive comments welcomed.
New search engines. A couple of these have drifted by lately, and I thought I'd bring them to your attention. I'd like to thank in particular lepermessiah for bringing Kosmix to my attention in a little noticed diary (which I found, of course, using search -- heh). I'll use these in the search comparison as well as the more well known engines.
On to the meat of the matter: search comparisons. The test query is [hayden] as in Michael Hayden, former director of the NSA.
- Daily Kos: most recent (107 hits in last week), highest impact (445 over last year). The "most recent" is the gold standard search, what I expect most people around here are interested in when a search is run. Who recently said something about this topic? The highest impact search is also very important, allowing search for statements within high impact diaries and stories.
- Organic search: Google (66 million hits);
Ask (no numbers);
Yahoo (24 million hits). This is industry standard organic search: ugly! The single word query is not specific enough to focus the results. Also not organized by date, but by everyone else in the worlds idea of what is important (Pagerank for google, not sure for others). Not kos specific.
- Google (restricted to Daily Kos) (19300 hits). It's great you can restrict search to Daily Kos. But you still have the problem of infrequent indexing and ranking, lack of focus by date, and mixing together comments and stories.
- Google blog (24 hits). Now we're getting somewhere. The cool new results interface even let's you restrict your attention to recent items, and you can also restrict by site. But wait. The most recent hit on this subject at daily kos is March 13, 5 days ago. Not so cool. Infrequent indexing, and especially inability to control frequency of indexing, remains a major stumbling block to counting on google search for best results.
- Kosmix US Politics (114). This is a new blog search engine that allows you to split results into various camps. They have a small number of hits, and if you manage to figure out to click "see results in political news" it brings up a fairly nicely sorted list ... of news items. No site restriction I could find. Still in beta, probably worth keeping an eye on.
- Sphere. Another new blog engine, offering time range restriction as well as sorting by time or relevance.
- SphereIt from FDL hayden diary. They provide a system to allow use of a web page itself as the query. Interesting.
But the real question is whether you find the new search system any better or worse than searching daily kos at generic organic search sites like google, ask, or yahoo, or some other search system not listed, not whether I can set up examples in my favor.
Comments in response that include actual search examples sought.