louisrosenfeld.com logotype

Home > Bloug Archive

Feb 28, 2008: Site Search Analytics workshop outline

Here's the outline of what I plan to cover in my new workshop. Does this sound like what you'd expect to be covered in a workshop about using site search analytics to improve UX design? Anything missing? Anything seem extraneous?

To entice you to give it some thought, I'll send a copy of Indi Young's new Mental Models book or a signed polar bear v3 to the three most helpful commenters who post feedback before Monday, March 3.

OK, here 'tis...

  1. What is Site Search Analytics (SSA)?
    A brief primer on what the data is and what we can learn from it. Cover search log anatomy, introduce samples of common reports. Explore Zipf distribution (long tail, short head, middle torso).
  2. The Point of SSA
    Overview of methods for determining what users want, and how SSA is different from and complementary with traditional methods. Demonstrate the value of data-driven analysis for designers.
  3. SSA Demonstration
    Introduce generic questions that can be asked of any data set. Live demonstration of basic analytics to indicate sessions, patterns, and failures.
  4. Technical Stuff: the nuts and bolts of SSA
    The three ways queries are captured. Comparison of reports from a variety of tools (search engine, web analytics).
  5. Exercise/Discussion: Pattern Analysis
    Hands-on analysis to determine frequent queries, query categories, metadata values and attributes, and content types. Plumb the impact of seasonality/time.
  6. Improving Navigation and Metadata
    Using SSA to generate metadata attributes and types, determine synonyms, popular terms, major categories. Improve contextual navigation.
  7. Exercise/Discussion: Failure Analysis
    Analyze failures (e.g., common failed queries, navigational failures) to diagnose problems, suggest possible fixes or follow-up research.
  8. Improving Content
    Tuning content and titling to improve findability. Identify and plug content gaps. Determine common content types.
  9. Exercise/Discussion: Session Analysis
    Identifying and sampling sessions. Looking for patterns within sessions, and learning from the changes that commonly take place during sessions.
  10. Improving Search
    Interface design improvements to query entry UI, search results presentation, and search refinement UI.
  11. SSA and UX methodology
    How SSA fills a quantitative hole within typically qualitative UX methodologies. Show how SSA can improve task analysis, personas, and other traditional UX methods.
  12. Advanced Topics/Discussion
    As time permits, discuss SSA's impact on business strategy, when and how to sample the Long Tail, and how to get SSA embedded within your organization.

email this entry

Comment: Russ (Feb 29, 2008)

I really like #7. However, I was thinking (to myself) that what would really be helpful to me would be "how to head off failures at the pass".

So, what I mean is that I'd like to know what I can do to prepare myself for failures--I *know* they're going to happen, so how can I make that less ugly, possibly more helpful?

Make sense?


Comment: Lou (Feb 29, 2008)

Hi Russ; makes sense in general, but I'm not sure it does for SSA. With analytics in general, you're looking at profiling collective behavior, and finding patterns that may help diagnose problems (failures). The fixes usually involve something else (e.g., tuning content, installing a spell checker).

So I'm not sure you can prepare yourself for failures *before* performing SSA. Rather, SSA helps you identify the failures.

That make sense? I have a sneaking suspicion that I might not be understanding your question...

Comment: Russ (Feb 29, 2008)

I may not have been exactly clear, but it would seem to me that there are probably a set of "common" failures that I could prepare myself for, right?

So, let's step outside of SSA for a moment.

If I were designing a website, I would probably link my logo to the home page of the website, even though more than a few people don't know that it is a fairly common design/build effort. In fact, I may (possibly incorrectly) assume that enough users know of this and not provide a Home action anywhere.

So, it would be wise to tell people to please include a method that is clear to users to get back to a home page.

What would be some of the failures that you commonlly see people having--such as installing a spell checker, etc. that we could be aware of and take home with is as small tips with big value?

Make any more sense?

Comment: Lou (Feb 29, 2008)

Russ, this is very interesting for me. For my entire career, I've made design recommendations--like the ones you're suggesting--based on a combination of experience, observation, and intuition.

Perhaps I've spent too much time hanging out with usability people (Keith Instone, are your ears burning?), but for the past few years I've felt increasingly uncomfortable making these recommendations without the benefit of supporting data. That's the main reason I've become so invested in SSA. So sure, I can make plenty of recommendations, but, ultimately, what the heck do I really know other than the stuff that's probably obvious to anyone else with some experience in the field?

SSA can help unearth some of the recommendations that ought to be obvious, and some that are complete surprises.

Anyway, that's my thinking here, for better or for worse...

Comment: Russ (Feb 29, 2008)

Well, by gosh, that's pretty sound thinking.

Except that I'm recommending this to a bunch of people that I know who aren't doing anything in analytics--except that they know that they need to.

I understand where you're coming from and why you might feel uncomfortable, but at the same time, if someone walks away without even those pretty basic things that they can look for and "instantly implement" today...

So, that's just my $.02. I've not really done the circuit, if you will, and I can't say what is successful from that point of view. I know people won't remember everything you say, so when I walk away from any presentation I try to give a couple of things that can hit the wall and stick pretty easily.

I'm definitely the first to claim ignorance here! Maybe I just wanted that book! :-)

Comment: Lou (Feb 29, 2008)

Hmm... Sounds like you're looking for an IA seminar. :-)

Russ, I'll definitely provide a heck of a lot of take-aways, but I'm going to try to teach people to fish, rather than, well you know... Those lessons will hopefully be the most valuable take-aways.

As far as a free book, you're definitely in the running: you're the only one who's commented so far!

Comment: Russ (Feb 29, 2008)

I'm good on the IA seminars for now; Miami will have me all IA'd up.

Sounds like you're already doing what I was asking for--so that's cool.

You could have just said "I do that!", you know!

Comment: Jorge Serrano-Cobos (Mar 1, 2008)

Hi Lou:

One thing interesting is the difference between queries used outside our web that gets traffic, and queries inside the web. Those queries compared, bring lots of useful info in terms of Information Architecture versus SEO (or not versus, integrated?)

And I missed maybe within #3, talking about more complex indicators, KPIs, segments related indicators, comparative indicators between segments, and in the other hand clustering and patterns auto-extraction rules and software applied.

We are now experimenting with data mining tools, maybe with Business Intelligence tools too, to extract the most of the classic "cube" of data: users profiling, web navigation, and time.

I'll try to write down a paper in english, I can send you a copy.


Jorge Serrano-Cobos

Comment: Mark Thristan (Mar 1, 2008)

Hi Lou,

How much of your typical audience is interested in analytics within the firewall? My own experience tends to be on the intranet side of things, so the two elements I propose are solidly intranet-based (although they can certainly be bent to fit the external world also).

Generally speaking, within the confines of the company, users posting queries are identifiable via log-in details, these users often are associated with company metadata tying them to a department or function, and exact locations can be identified via IP addresses, for example. This information can certainly be particularly useful in clustering searches and then analysing to target different audiences. A single search might form part of a user session for example, or indicate an evolution of related searches for an individual, or - when aggregated with searches by other users in the same function - indicate popular or emergent queries, or with IPs by location.

I suppose it's the whole concept of squeezing the pips of the lemon until they squeak: you want to get as much juice out of the queries by combining them with other known sources of data to which they may join. This is typically not pursued by many major corporates (at least not in my experience) - they collect data for one purpose and neglect the other possibilities. I suppose my point is: it's not just the search data which is important, but any other related data with which it can be combined and mined.

This might fit in with #4 and #5 in your presentation. Hope it's of interest.

NB: Lou, if you need any Swiss chocolate, let me know, I've just moved to Geneva.

Comment: Stephen Jones (Mar 1, 2008)

The details listed appear to passively capture what the user *did* - what scope / importance / value is there in soliciting / capturing the *user's* rating (good / bad) of the pages they visit...

From an intranet perspective (i.e. where the user is known), would 'delicious' style features that organise the pages users like / use for later use provide additional data points for results?

Comment: Peter (Mar 2, 2008)

"For the past few years I've felt increasingly uncomfortable making these recommendations without the benefit of supporting data."

You should put that in the description of the course: it explains why we should take it.

At another level, the course seems much more nuts&bolts than, say, the enterprise IA one (which had more "strategic" stuff, although I never attended it). Perhaps you should make nr. 12 not optional and put it earlier in? It seems like the people wanna attend would be dataheads ;) and they'd like some of your "convincing the ceo" experience.

Just some half baked thoughts after a pretty damn relaxing holiday :)

Comment: Peter (Mar 2, 2008)

And: lots of examples with real data, because the dataheads usually only have 1 dataset that they ever work with (the one at work), so being able to compare with others is a great opportunity.

(I use the word dataheads in the best possible way btw.)

And: make a new header picture for http://louisrosenfeld.com/presentations/seminars/site_search_analytics/, it's not very good. The enterprise IA one was much better, and funny too.

Comment: Caroline Jarrett (Mar 2, 2008)

Hi Lou,

Sounds like a great outline to me, but here are a couple of extra suggestions:

1. I'd second Jorge's point about internal compared to external search.

2. In my SSA, I'm seeing a lot of what I'd term 'jump search', i.e. users who are deliberately putting in specialist string (e.g. the code number for a product), which I interpret as meaning they want to jump directly to that product. And I've seen that a lot in usability tests as well. I'd see that as a separate matter from 'navigation search' where the topic in question probably ought to be available through navigation but isn't. Will you be covering this distinction in your 'navigation' section?

3. A question I grapple with a lot is how far down the tail to look. Is it OK to concentrate on the top 10 terms? The top 100? The top 10%? Where do you put the cutoff and why? Do you have any statistical analysis to support your view, or is it more of a pragmatic / gut instinct approach?



Comment: Mari-Carmen Marcos (Mar 3, 2008)

Hi Lou and everybody

I agree with you in the necessity of having supporting data before taking some design decission.

What I want to underline is that clients usually don't have so many data we need to analyse them. Sometimes they only have registered data as browser used, language or times of access, but they cannot track the navigation way.

I think that your course can help us to have more arguments to convince clients about the necessity of having site search analytics tools and let us to use them to improve their website.

Comment: Lou Rosenfeld (Mar 3, 2008)

Everyone: good suggestion to compare web-wide and local queries!

Jorge, it sounds like you're up to some really impressive stuff; I'd love to hear more.

Mark, I have some good data that ties users with their queries. Great suggestion; I definitely plan on getting into this. Lots of pips will squeak. (Heh; never heard that one before.) And yes, I very much do require some Swiss chocolate; how did you know? ;-)

Stephen, that's a cool idea. Though it might push us outside of SSA. Guess covering it would depend (like everything else) on available time and source data to play with.

Peter, you don't like my art? ;-) I love getting into the organizational stuff, but many don't from my experience. Again, I guess it depends on whether we'll have enough time. I also hope to have a couple data sets, but I'm considering inviting attendees to bring their own. Logistically that might be tough, but it's also hard for me to come up with good data samples to share. In fact, that's the thing that makes me most nervous about this workshop; will we have enough data to play with?

Caroline, great suggestion! I'll make sure to address it specialized queries. I've got some suggestions as to how far to go down the tail.

And Mari-Carmen, great suggestion; I've got some research on this that I'll share as the basis of my discussion.

Thanks everyone; fantastic input. Hmmm... now who should get those books?

Comment: Lou (Mar 4, 2008)

Actually, I'll send everyone who responded a book. But some of you are nearly impossible to contact (after trying to find your email addresses on your sites). Mark and Stephen, how do I reach you? (I'm at lou at louisrosenfeld.com )

Add a Comment:



URL (optional, but must include http://)

Required: Name, email, and comment.
Want to mention a linked URL? Include http:// before the address.
Want to include bold or italics? Sorry; just use *asterisks* instead.

DAYENU ); } else { // so comments are closed on this entry... print(<<< I_SAID_DAYENU
Comments are now closed for this entry.

Comment spam has forced me to close comment functionality for older entries. However, if you have something vital to add concerning this entry (or its associated comments), please email your sage insights to me (lou [at] louisrosenfeld dot com). I'll make sure your comments are added to the conversation. Sorry for the inconvenience.