Searching Daily Kos
Quick Search Tips
Search: enter words you want to find in the target collection. By default only targets containing all the words will be returned. Capitalization is ignored. If you want to find several words in a precise order (a phrase) put double quotes around them as in "this is a phrase". Single quotes are not permitted, and are removed. Although "AND" is assumed by default, "OR" and "NOT" are supported, as follows.
Multiple query words are by default understood to be joined by "AND". Certain characters have to be "escaped" by placing a back slash in front of them; to find "/", use "\/", to find "=", use "\=". Complex queries can be constructed using parentheses and restricted queries. Restricted queries: limiting the parts of the document to be searched. The available options are described here. For both stories and comments (documents):
For stories only:
For comments only:
Find: In order to choose what kind of documents to find, use the pull down menu to select one of the following. Each option is identified as either a database search, meaning that the primary database is searched using Scoop directly, or index search, meaning that a specially constructed index of words contained in documents is searched using swish-e. There are three different indices, one for diaries (aka user diaries), one for stories (aka front page stories), and one for comments. User diaries that are elevated to the front page are found in both story and diary indices.
From: and To: pull down menus restrict the time interval of the search. The default interval is from one day ago to most recent. Most recent may be as much as 5 minutes out of date. Sort by: The order of presentation of results. Not all choices are relevant to all searches. Story and Diary Search sort choices.
Comment search sort choices shared with stories and diaries are "Relevance", "Title", and "Time", which are as described above. Those specific to comment are "Cid", "Nrec", and "Ntroll".
More example queries
For comments only Everything past the "#" is a comment, not part of the query!
Searching Daily Kos with election tags (etags)You can now search using election tags as follows. etag=CT-SEN Etags with embedded spaces should be quoted. etag="CA-LT. GOV" You can only use one etag per search. Good election tags have a two letter postal state abbreviation hyphen seperated from either a house district number, or a statewide office abbreviation (SEN, GOV, Lt.GOV, SOS, etc). There is one exception, etag="2006 ELECTION" also works. Good etags are ones we already know about by finding them in the tag list of an existing diary or by appearing in one of the election roundup diaries. The etag is used to build a more complicated query that looks for the tag in tags, title, or body text (stories/diaries) or in subject and comment text (comments). Certain meta dairies are excluded, including open threads, jotter's daily lists, and sidinny's election roundup, and the top comments diaries.
Software used.The search facility currently installed is based on the swish-e search engine: it is fast (compiled C code), open source, and flexible (you can feed it documents out of a database using your own custom preproccessor). It is intended for document collections numbering a million or less, which will do for Daily Kos with under half a million documents (excluding comments). In addition, swish-e offers many of features most desired, including search of phrases and search within configurable document areas (title, author, tags, URLs). What is indexed. All Daily Kos documents from the beginning to within 5 minutes of the present moment are indexed. Older documents are stored in static yearly indices while more recent documents are indexed and reindexed, frequently at first, then less frequently as they age, until they roll over into a growing, static, permenant yearly index. The right mix of number of index files and frequency of update to obtain the optimal combination of ease of maintenance and speed of search may require further experimentation. At present three different series of indices are maintained, corresponding to (in order of increasing total number of documents) front page stories, user diaries, and comments. For 2005 the number of stories, diaries, and comments was 5097, 88943, and 693224, respectively, containing 56993, 371578, and 2945916 distinct words. Design Considerations. The utility of search to a community of users is delivered not simply by the results of the text search, but most critically by the ordering of the results such that highest "value" hits are displayed first. (Most search researchers know that no one looks beyond about the first handful of results; almost no one looks beyond the first page.) The key to arranging for best to come first is to understand, have access to, and use the meta data associated with the text documents searched. Meta Data. At Daily Kos, we have a number of sorts of meta data that are relevant to documents. First among these is the time of posting. What is the latest? When did they say that? Who was first? That sort of thing. So the one thing search has to get right is time, in the sense of being up to date, that is including the latest diaries, as well as in the sense of permitting searches to be organized by reference to the present time. Making it up to date. Because recommendations are only open for 24 hours after a diary is published, it is important that search indices be refreshed sufficiently often during that interval to provide reasonably up to date information on total recommendations and comments. Currently, diaries up to 6 hours old are refreshed every 5 minutes, and diaries 6 to 24 hours old are refreshed every 15 minutes. More Meta Data. Other important meta data associated with stories and diaries includes comments and recommendations. The number of recommendations offers insight, imperfect, but useful, into the degree of approval a diary has received from the Daily Kos community. The number of comments indicates the size of the discussion engendered. The two numbers together give an interesting insight into a diaries history at Daily Kos. Some user diaries are promoted to the front page, and some diarists ("front pagers") are allowed to post directly to the front page. The latter are not open for recommendation. In order to harmonize listings from searches which include both front page stories and user diaries, a plausible value for number of recommendations (83) is assigned to front page stories, but in lists, this number is indicated by "*" to avoid placing too much emphasis on this made up number. The search query. Search queries can contain individual words, all of which must be found in a text document to constitute a hit. Queries can also contain phrases, indicated by double quotes around the phrase. All words are converted to lower case while searching, so letter case is of no consequence. In addition, queries can include boolean terms, such as "and", "or", and "not" which can be used to construct more complex and perhaps more informative results. By default "and" is assumed between individual words in a query. Finally, queries or parts of queries can be restricted to only search within parts of a document or document metadata. The parts available are shown above, along with examples. Certain characters have special meaning to the search engine (open and close paren, double quote, equals sign). Only a restricted list of characters are expected to occur in a word. For the indices here at daily kos, the word characters are as follows.
All other characters (non word, non special) are silently replaced by spaces while searching. For instance searching with "Jerome a Paris", "Jerome.a.Paris", or "Jerome,a,Paris" all give the same result. This is important in certain cases, for example searching for the url http://riverbendblogspot.com is accomplished with the query site=("riverbend.blogspot.com"). The reasonable looking but incorrect query site=riverbendblog.blogspot.com doesn't work as expected because in the absence of the double quotes and parentheses, replacement of "." with " " and the default boolean AND understood between search words makes it equivalent to site=riverbendblog AND blogspot AND com, which returns as hits only documents with riverbendblog in a url, and blogspot and com in the body text. More on Search
If you are intriqued by search and would
like to know more about the basics of how it is done, this is a
good introduction.
On Search. Basic basics. by Tim Bray. Since I've emphasized the importance of meta data, I'll also point to another article from the same series, On Search. SearchMeta, with some further thoughts on things meta. While I'm on the subject of underlying technologies, let me end this section by pointing to an article by Tim Bray on Ajax, the other new addition here at Daily Kos. In it, he also gives credit to Jesse James Garrett for the Ajax acronym in the process of thoroughly describing it.
Daily Kos is an amazing place, community, phenomenon. It has been my
privilege to have the opportunity to contribute.
-- jotter |
Recommended
- I usually wouldn’t post a diary about polling, but this is actually quite important. Typically, I’m not the sort of person who celebrates polls that make me feel good and attacks ones that don’t, as a) I tend to look at the overall average and trending...146 comments 370 Recs
- The pro-Russian prime minister of Slovakia, Robert Fico, was shot and seriously wounded today. A suspected assailant has been arrested but there is no word yet on a possible motive. There are a number of reports that the would-be assassin was...64 comments 411 Recs
- ETA: Thank you, everyone, for getting this to the Rec List. The comments are fantastic and moving, and if anyone ever doubted that the Kos community had any love for Jimmy Carter, they wouldn’t anymore. As of now, the former President is still with us,...228 comments 649 Recs
- NFL kicker's commencement speech slams LGBTQ, feminism, contraception, choice, IVF, the media + JewsHarrison Butker is a placekicker for the Kansas City Chiefs and a miserable human being. NFL fans will probably know him. Everyone else will likely not. However, after his commencement speech at Benedictine College, he will be more famous — or infamous...192 comments 213 Recs
- There’s something fishy at the New York Times. First, what we already know: First, whoever is in charge of the paper’s polls is not doing their job. Second, whoever is choosing what to emphasize in Times coverage of the campaign for the presidency is...179 comments 462 Recs
- From the Guardian: “Had I been President Biden,” Romney said, “when the justice department brought out indictments, I would have immediately pardoned him. I’d have pardoned President Trump.” “Why? Well, because it makes me, President Biden, the big guy...125 comments 177 Recs
- I’m calling it — the mainstream media is done. I just read yet another example of the complete hollowness and irrelevance of the mainstream political media. The Post this morning, following a surprise 10 + point victory by an Angela Alsobrooks (a black...225 comments 407 Recs
- Alabama and the rest of the South are infamous for employee bashing. The political good old boys know there is lucrative swill in the corporate trough. And kissing the CEO’s ass means you do not have to slum it with the people doing the hard work. But...103 comments 235 Recs
- The New York Times released another poll, done with Ipsos, one that shows Biden up 46% to 44% for Trump, but you won’t see it on the front page. If you find and read a story about why it’s hard to measure RFK Jr’s support you will find data on how...90 comments 258 Recs
- The GOP’s favorite phrase when it comes to presidential politics appears to be, “Nobody knew at the time.” Once a president is sworn into office, regardless of how much evidence there is of crimes and irregularities committed to get him there, both the...69 comments 261 Recs
- I am Canadian. Born in Canada, live in Canada. I do have American ancestry going back to colonial Virginia. My middle name is the surname of a greatx10 grandfather or uncle who apparently fought in the Continental Army, my grandparents were married in...189 comments 603 Recs
- You can read a blog post about this cartoon, and a transcript, here. I’ll also post the transcript in comments. We can make these cartoons because of hundreds of supporters pledging low amounts - typically $1-$3 - and that's just how I like it! Pretty...22 comments 140 Recs
Recent Diaries
- Saving democracy in the United States will require the efforts of everyone who can be convinced of the danger and will commit themselves to taking action, As Bernie Sanders explains in an article published yesterday in The Guardian: Today, in 2024, our...2 comments 3 Recs
- On Tuesday, May 14, a pickup truck driven by a man with a history of arrests for driving with a suspended license struck a bus carrying 53 farmworkers, mainly Mexican nationals being transported to their jobs picking watermelons in Marion County,...2 comments 8 Recs
- Welcome to White House Picture of the Day Showcasing a photo from the White House Photography staff, led by Adam Schultz. Feel free to join us anytime and share with us what is happening in ...1 comments 5 Recs
- Its being reported that there will be two Presidential debates www.bbc.com/… Frankly for myself I wonder if this is a sign of desperation by ‘king Trump . Firstly there will be no debate on Trump friendly Fox News. I would have expected ‘king Trump to...4 comments 3 Recs
- There is nothing more critical than saving lives of innocent people, especially the children and women that want nothing more than to live their lives in peace. Please do everything possible you can to end the genocide that is currently going on in...6 comments 8 Recs
- Feeling poorly is crummy.� We all know that.� I can’t really complain because, on the whole, I am pretty healthy.� I whine anyway.� This is just a cold, not really more than body fatigue with the ...10 comments 9 Recs
- The tempo of Russian offensive operations in northern Kharkiv Oblast continues to decrease after Russian forces initially seized areas that Ukrainian officials have now confirmed were less defended. Ukrainian President Volodymyr Zelensky and Ukrainian...19 comments 45 Recs
- Last week Hillary Clinton appeared on MSNBC and expressed her opinions regarding the student protests happening around the country in response to the United States’ backing Israel, even after Israel has committed mass war crimes in the Gaza conflict....19 comments 7 Recs
- SPOILER WARNING: A late night gathering for non serious palaver that does not speak of that night’s show. Posting a spoiler will get you brollywhacked. You don’t want that to happen to you. ...8 comments 6 Recs
- Democracy Docket — US Supreme Court Allows Louisiana to Use Congressional Map With Two Majority-Black Districts The U.S. Supreme Court granted requests from both the state and Black Louisiana voters and civil rights organizations to allow Louisiana’s...6 comments 20 Recs
- Nothing New Here, We’ve Always Been Crooks! Note: I expect to post another 10-15 editorial cartoons in the comments section and possibly some as manual updates if they go online tonight. Thanks for supporting this diary series. In case you missed my...21 comments 46 Recs
- The exact circumstances are not yet clear. But something is afoot at the Miss USA organization. But first, a word from our sponsor! Top Comments appears nightly, at 10:00 pm Eastern / 7:00 pm ...7 comments 24 Recs
- Kitties love to read books, and even do some writing from time to time… But first, the rules… You are all welcome to join us! Here are few not-too-onerous PWB rules Do not troll the ...35 comments 59 Recs
- Guests: Heather Digby Parton of Salon, attorney Keith Barber of Daily Kos Also: MD, NE, WV primary results Biden, Trump agree to early debates... It has been the most important week in the New York criminal trial of Donald J. Trump, as the last witness...1 comments 4 Recs
- Charming. Quick with a joke. People love him. Regards people as ants to be stepped on if it is in his own interest. Would jovially see even small children die for a few bucks. Devoid of ethics and morals.26 comments 6 Recs
- From the Guardian: “Had I been President Biden,” Romney said, “when the justice department brought out indictments, I would have immediately pardoned him. I’d have pardoned President Trump.” “Why? Well, because it makes me, President Biden, the big guy...125 comments 177 Recs
- Today celebrates the opening of a restaurant in 1940 that would go on to become a franchise that has sold billions of burgers, caused an indeterminate number of heart attacks, and without which, a ...48 comments 26 Recs
- I was driving, and the CBC program As It Happens came on, and one of the anchors, Chris Howden said the following: The business at (part of) hand, exploring a centuries old German sport in which two men attempt to pull each other across a table using...9 comments 10 Recs
- Hi, it’s been a while since there was a BookChat, so I threw this together literally this afternoon. It’s barely better than a rough draft, but it’s a BookChat. Seeing lots of talk about Eurovision, and it made me think of books about that and other...24 comments 13 Recs
- The first results (partial and unofficial) in the race for mayor of Anchorage show Suzanne LaFrance with a good lead. She has served on the Assembly and focused on good government at the local level. The incumbent, Bronson, is IMHO an incompetent fool,...2 comments 17 Recs