louisrosenfeld.com logotype

Home > Bloug Archive

Dec 02, 2004: Using Search Log Analysis to Predict the Future

Had a nice chat with Guy Valerio at my seminar in London the other day. Guy has been doing some innovative analysis of search logs at the Financial Times. Guy compiles statistics on the occurrences of company and organization names in FT's logs weekly. He then compares these names to a similar list of name occurrences generated from the FT's most recent articles.

If a company name shows up with great frequency in the search logs but hasn't been covered recently in FT articles, it may indicate a developing story about to hit daylight. Guy finds that this sort of anomaly is often a useful predictor of what the FT should consider covering in the coming days. Conversely, if a company name drops off the search logs, interest in a related story line may be waning, and FT can choose to invest its reporting resources elsewhere.

Using search log analysis to divine emerging stories: Reason #87 to make search log analysis one of keystones of your user research methodology...

email this entry

Comment: Eric Scheid (Dec 2, 2004)

This is the second story i've seen on this idea in the last couple of days. Expect there will be more...

"In Chile, instant Web feedback creates the next day's paper -- Three years ago, under Mr. Edwards's guidance, LUN installed a system whereby all clicks onto its website (www.lun.com) were recorded for all in the newsroom to see. Those clicks - and the changing tastes and desires they represent - drive the entire print content of LUN. If a certain story gets a lot of clicks, for example, that is a signal to Edwards and his team that the story should be followed up, and similar ones should be sought for the next day. If a story gets only a few clicks, it is killed. The system offers a direct barometer of public opinion, much like the TV rating system - but unique to print media."


Comment: Jeff Werness (Dec 11, 2004)

I love this stuff because it relates to the most recent topic on funding ourselves: how do we demonstrate tangible value in the form of hard revenue or business intelligence that justifies our existance (whether or not it goes to our own bottom line)?

More than likely I think there's a volume threshhold at which this sort of IA analysis pays off. At my previous employer, we got on average 700K queries daily. During the holidays this volume doubled to 1.4M to 1.7M daily queries (both numbers roughly translating to 4 queries per site visitor).

By performing a high-level daily overview and a more in-depth weekly analysis (we were on a weekly promotional cycle) my group could deliver information to merchants and marketers based on real consumer interest which resulted in more visibility for those product categories, which not only increased sales, but also improves the overall findability factor for the site, and thus customer satisfaction, which leads to more return customers, who spend more, etc.

In this scenario, the analysis identifies products and brands that are both increasing and decreasing in customers' minds, and allows the merchants to act accordingly. We were getting to the point that if I saw certain brands increasing broadcast advertising activity, I could correlate a certain increase in search queries.

As far as how many queries are needed in order to successfully translate search activity to business intelligence, I can't say having never worked on a smaller site than the one I mentioned. Would be interested in others' experiences with this.

Comment: Lou (Dec 11, 2004)

Indeed. Jeff, just curious: can you tell what kinds of products your former employer sells?

Comment: Jeff Werness (Dec 13, 2004)

Lou, my former employer sells computers, electronics, movies, music, and video games -- the largest offline retailer in those categories :-)

That said, it's easy to see how a site selling such popular product categories gets that much activity (considering that navigation is especially difficult to design when you've got more than a million products in what ends up to be essentially hundreds of categories); search and navigation can be difficult to build and maintain, however the payoff in knowledge gained through real customer feedback is invaluable to the business.

BTW, we met briefly at a conference in Santa Monica several years ago. Really enjoyed your series.

Add a Comment:



URL (optional, but must include http://)

Required: Name, email, and comment.
Want to mention a linked URL? Include http:// before the address.
Want to include bold or italics? Sorry; just use *asterisks* instead.

DAYENU ); } else { // so comments are closed on this entry... print(<<< I_SAID_DAYENU
Comments are now closed for this entry.

Comment spam has forced me to close comment functionality for older entries. However, if you have something vital to add concerning this entry (or its associated comments), please email your sage insights to me (lou [at] louisrosenfeld dot com). I'll make sure your comments are added to the conversation. Sorry for the inconvenience.