louisrosenfeld.com logotype

Home > Bloug Archive

May 11, 2002: Promoting IA--Survey Results

A few days back, I put out a brief survey to learn a little bit about whom we're trying to promote IA to, what gets in the way, and what might help us do a better job. The results are now available; feel free to peruse the detailed results. Many thanks to the 228 of you who provided feedback!

The results in a nutshell:

  • About 80% of you engage in IA at least half of your work time.
  • 41% of you are in-house IAs, while 35% of you work for agencies or consulting firms.
  • 67% of you are trying to promote IA to your co-workers, more than any other audience. Interestingly you find them the easiest audience to convince.
  • Hardest audience? Your client's decision-makers, followed by the media, and then your own bosses.
  • Hardest part of promoting IA? Timing: 27% don't get a chance until their projects are too deep into the design/development process. Next biggest challenge: IA sounds too "jargony" (17%).
  • What would help? By far, we need better stories to tell: 82% plead for ROI cases, 72% for case studies of successful projects.

Not much surprising here. We want ROI (Return On Investment) cases to make because the people we're selling IA to want ROI cases. I find this scary, as ROI cases are typically BS (Bull Shit). If we have to make up silly numbers to make our clients happy, so be it. But shouldn't we also be educating them about why these numbers are so silly? If we don't, we hurt them, ourselves, and the field in the long run.

Interesting that co-workers are our largest audience. Nice that we feel we have the most success convincing them of IA's value. Why is this? Are we just more experienced trying to convince them because, well, we regularly encounter so many of them?

Also interesting the the biggest challenge is that we don't get a chance to promote what we do until it's too late. I wonder how many of us get to participate in sales calls? Or, maybe a better question: I wonder how many of are willing to participate in sales calls? We've got to get over our aversion to sales if we're going to be at the table when important decisions are getting made.

email this entry

Comment: Celia Romaniuk (May 12, 2002)

I think the reason we're successful at convincing co-workers of our value is because they work directly with us. They see the results of what we do every day. Working with the other groups is harder, because the contact we have with them is far more limited. They tend to get a broad brushstrokes, abstracted, salesy point of view... whereas our coworkers can see the day-to-day detailed thought and its results.

Good IA tends to disappear, and people only notice it when it's not there. I think this goes for our process deliverables as much as for the end product. And so the people who have sweated over the decisions beside us are more likely to appreciate the effort than those who just see the 'obvious' results.

Comment: Dell Whisonant (May 12, 2002)

Our collective need for ROI is interesting, and I keep changing my mind about the value of it. Because it's BS (i.e., easy to pick apart as it applies to OUR project here in this company) you do find yourself saying "just trust me". That's one reason relationships with coworkers/bosses are crucial.

And even if you could prove ROI, I think in many cases companies would decide to forego the time and money required for "good IA" and just take the hit.

In my world, you throw mud up against the wall and fix it later- and everyone from the top down is fine with that. IA is somewhere downstream. I agree with Celia that no one notices IA until it's not there. It's not that you aren't trusted, it's that the organization has other priorities. Making IA a priority in the organization is the long term goal.

What we really do every day is work with slippery mud, within the limits of time and resources, right next to our coworkers. Each coworker has a cause of their own, so compromising/negotiating is how you end up spending the other half of your time. The relationships you have with coworkers influences the outcome of the negotiation- and ultimately, the design.

Comment: Paula Thornton (May 12, 2002)

I agree totally with the comments regarding ROI...because it's the wrong tool for the job Businesses have been 'endoctrinated' to relying on the tool because the distribution of financial resources is predominantly under the control of finance people. The challenge is more grounded in economics: the answers are to be found in the principles of economics rather than finance.

Rather than focusing on return on investment (ROI) we need to establish an understanding of 'exchange of value' (EOV). Companies have been attempting (mostly in vain) to change their companies to be more 'customer focused', but this will fundamentally be improbable until the economic principles of EOV (my own term, see http://www.adaptivepath.com/events/dfw.phtml ) are understood and begin to augment ROI (which is to be applied at 'another level' of planning).

To be better prepared to build these cases, brush up on basic principles of economics and value exchange.

Comment: Dean Esmay (May 12, 2002)

Okay, I feel silly asking, but, short of buying an entire book, where does one get a reasonably tight description of what exactly an Information Architect does?

Comment: Andrew (May 13, 2002)

Dean, actually buying a book is probably the most concise way of getting that description... Otherwise, check out www.boxesandarrows.com and look at Christina Wodke's editorials.

Comment: Victor Lombardi (May 13, 2002)

I fail to understand what fuels the emotional reaction against ROI. If we're so confident that IA is so great then shouldn't we be confident we're providing some value? After all, why bother doing it if you're not getting some benefit at the end???

One of the advantages of working at a consulting firm rather than a less-expensive mom-and-pop firm is that we always have to justify our rates. Ethically, I would feel irresponsible not to do this, because I wouldn't be sure I'm not cheating my client out of their money.

Is the problem vague ROI arguments? Or overly complicated ROI arguments that don't match reality? Maybe someone could clarify more specifically what is bullshit about ROI so I can attack your argument more effectively :)

Comment: Rhonda (May 13, 2002)

My original post was lost when I clicked on http://www.adaptivepath.com/events/dfw.phtml) above, so lucky for you this will be shorter. I would like to see the EOV idea Paula mentined, but the link wasn't working.

Personally, I don't have time to change the way America does business. ROI is a tool, not the goal. Not proving any part of a process of business by using NUMBERS is detrimental to _any_ role. It is a _LUXURY_ to think we don't have to use numbers aka ROI to show that IA brings value. And it shouldn't be hard.

At this point in the development of the IA field, we are maturing enough to see that we are not just one thing, architects. We are/can be valuable parts of the team who makes the decisions. We don't just make beautiful wireframes and site maps (though we can).

I have not worked for boutique firms, but have worked on large scale ecommerce applications, usually involving me having to vacuum vats of information from marketing/technical/business usability/users and arts and analyze it to architect the best user experience. JOB TITLEs aside, I don't need to prove anything about how important this role is. However, if I did have some numbers and case studies, I would surely win over the higher ups who make decisions based on the bottom line.

Comment: Greg Evans (May 13, 2002)

I totally agree, Victor. I was somewhat dismissed at the IA Summit when I brought up this point during the Q/A with Steve Krug. As an in-house IA who has an MBA (yes, we do exist)I consider one of the biggest problems within the field to be the apathy towards trying to put a hard number on the ROI of good IA. By in large, most IAs I have met simply dismiss the notion and respond with "the return is so obvious, we don't need to put a number to it. Besides...its almost impossible to do." As far as I'm concerned, if I can't get a good handle on the amount of time=money that the project can either save or earn...it should never leave the ground.

Comment: RuthK (May 13, 2002)

Most of the IA work I have done has been incidental. In other words, I have done the work as an in-house Producer, Content Manager, and a couple of other titles -- details unimportant. It was actually quite easy for me to convince my superiors that information architecture is important because they hired me to make the dang thing work and work well. I encountered more difficulty in doing this function when I was actually hired to be an IA, formally. Owning the process means you can do whatever it takes, including IA, and bypass the need to convince others of the value of the work. This also means that IA work can be commoditized (as described in previous posts), and dissociating it from other functions makes it more vulnerable to being discounted. I know that many IA's are specialists and don't want to own the larger process of application/web/product development. My point is just that sometimes, for some people, diversification of skills and responsibilities is a viable strategy for incorporating IA into the design process.

Comment: Lou (May 13, 2002)

OK, so we like ROI. ROI is good. We need ROI.

Now does anyone want to tell me how exactly you make an ROI case for (re)designing an information architecture? Show me the numbers.

Come'on, let's see'em!

Comment: Eric Scheid (May 13, 2002)

When it comes to influencing business decisions, ROI is only one of the options, albeit the most common/influential. I've listed some other avenues for selling IA on http://IAwiki.net/SellingIA

To make a ROI case, unfortunately, can require a detailed understanding of the business context, including such things as market size and volatility, supply chain management, and other very-much-not-IA topics. The best we can do is offer hypotheticals like "if there are 1,000 visitors per day, then good IA will mean an extra 100 sales".

The problem though is that although we can say how much the "good IA" will cost for that hypothetical, unless we actually know how many visitors there really are, and how much extra profit those extra 100 sales make, then we can't put forward an ROI argument.

We need more information from the business people as to what the framework for the ROI should be. Once we know that, whatever quantification we come up with can be used to build the ROI case.

In summary: it depends.

Comment: Eric Scheid (May 13, 2002)

Expanding on "show me the numbers"... ROI depends, at it's core, on *quantification*, which means measurement. Can IA be measured?

See http://IAwiki.net/MeasuringIA

Comment: Lou (May 13, 2002)

Eric: my point exactly.

Comment: Victor (May 14, 2002)

Eric says, "To make a ROI case, unfortunately, can require a detailed understanding of the business context". Yeah, but don't we need to do that anyway? Context is one of Lou's circles in the original Venn diagram. Understand the design and the business is what makes us IAs so smart. :)

From my perspective, IA is just the engine in the car. While techies may be concerned with horsepower numbers, what matters to drivers is 0-60 times...how fast it accelerates. So we don't measure IA, we measure the effect of IA (I know that's what the wiki page is getting at, but there seems to be some confusion over this and I want to be clear).

So, measuring the effects of IA. Jakob gives us a fine, specific example:
http://www.useit.com/alertbox/20010121.html

To arrive at a monetary figure, you simply take Jakob's 150% improvement in task time and factor in how often the person uses that application and what their salary is. You end up with a monetary figure, the R in ROI. Calculating how long it took to create the design is the I. Doing the addition tells you if you made money or not.

Not only is it not hard, but it's soooooo exciting when you can look a business owner in the eyes and quantitatively prove the worth of your design.

To pull back a little, Alex Wright penned a good introduction to the topic of ROI for design:
http://www.newarchitectmag.com/print/documentID=22255

And Rashmi has collected a lot more to chew on:
http://www.sims.berkeley.edu/~sinha/UsabilityROI.html

Comment: Lou (May 14, 2002)

Victor, that's a lot of good information, Alex's article especially. But it's important to acknowledge that when a client says to you "prove that IA will provide a positive ROI," you can't. Unless you are able to do some controlled experiments, which is pretty unlikely.

That's not to say that you can't prove ROI of very specific aspects of an information architecture. This you might actually be able to do in a controlled test. But you're still only testing a small portion of something much bigger. And that little thing may be an interface widget or something else that's not necessarily architectural. Rashmi's cites are great, but they're almost all about ROI of usability engineering, and evaluate the ROI of a much more granular set of widgets.

So, I stand by my belief that you can't prove ROI for IA. Scope is too large, and scientific testing is infeasible or unrealistic. That doesn't mean that IA doesn't provide value. But I don't believe you can *prove* it like you might with usability engineering or interaction design.

Comment: Andyed (May 14, 2002)

Try the 50% rule. With every click, you lose 50% of your audiences.

Factor in a reduction in erroroneous clicks and better access to frequently used functions/information, and you've got some nice numbers for estimating the benefit of a particular design iteration.

Alternatively and complementarily, take a look at task fallout or the success rate in a usability test or via logfiles. Make a conservative estimate of the ROI of increasing that conversion rate.

Percents of percents are very subject to small variation. In one case, I was able to eliminate a problem with 50% of a site's products that 25% of users incurred and 25% of those bailed on the site. The result was a 50% increase in sales.

A related observation is that log analysis is an under-rated tool in the IA's toolshed.

Comment: George Olsen (May 14, 2002)

ROI is only one of several ways of measuring value.

And while we shouldn't shy away from ROI calculations -- which I think IAs *do* tend to do -- we should also realize what we do also involves things *are* hard to quantify.

This ain't a new problem. Graphic designers have been dealing with it for years. So has marketing folks and brand strategists. And let's not talk about advertising, where the joke is that half the money is wasted -- it just that no one know which half.

It's why "intangibles" are a recognized line item on a balance sheet -- although obviously intangibles are going to be under intense scrutiny these days.

Also, there are other standards besides ROI. One is "cost of playing/staying in the game." What ROI have computers had for business? There's actually a big debate over quantifying this. But would anyone run a large warehouse these days with a paper and pencil ledger. No.

Value can be added in other hard-to-ROI ways: by creating a competive advantage, by providing better customer service, by creating efficiencies, by avoiding risk, by making your clients/bosses look good to their bosses, etc.

The key thing is we need to be able to provide *concrete* measures, even if they're not necessarily easily quantifiable. But they should be clear and specific, so that it's pretty easy for people to agree whether you achieved them.

(Incidently, this has gotten me thinking about what are good qualities we should be using for this sort of measuring. "Findability" meets the test of being clear, concise and concrete. In constrast, "usabilty" is too broad, since it can be unpacked (at least) into the competiting quality of "learnable" and "efficient.")

Obviously, it's preferable if we can show how our work directly contributes to the bottom line, but there's a lot of other business disciplines that also rely on indirect measures.

But the key is to get your client/boss to agreed to what constitutes "success" *before* you start your work -- and success can be defined in many ways.

Comment: genius (May 15, 2002)

To me, ROI is very straightforward, regardless if it's about IA or an entire site build. There are already many ways and things to measure.

I'm surprised that out of all the posts here, no one has mentioned specific ways of doing ROI.

ROI: quantifiable measure of success, or improvement in performance.

Measure: tangibles or intangibles

Tangibles: units (no. of new customers/subscribers, new purchasers, return purchasers), dollars (profit, revenues, costs, size of purchase), time (speed to task)

Intangibles: pleasure, satisfaction (customer sat. reports, # complaint emails submitted online/by phone, brand impact (awareness, approval, memory)

METHODS:

For tangibles: It's called Site Analytics. There are co.s (web agencies as well as specialists) and tools that doing this. It's based on web logs, yes, but successful implementations (like any successful project) depends on successful collaboration.

We're talking everyone from the client down to the strategy people, tech, database, number crunchers, to IA and creative design working together to set things up, do a baseline analysis, followed by change implementation, then tracking and analysis.

For intangibles: More indirect approaches, familiar to marketing/branding types such as, surveys -- online, phone, immediate or delayed -- as well as focus groups.

'Nuff said.

Comment: jefflash (May 15, 2002)

Lou said "So, I stand by my belief that you can't prove ROI for IA."

I think there are a lot of things you can't prove ROI for. IA may or may not be one of them.

Before we try and figure out how to prove ROI (or how to show that it's not possible to prove it), take a step back and ask why we'd want to prove it.

Most likely, you're trying to show someone your value. That might be a C-level executive inside your organization, or a customer project manager that sent you an RFP.

Do all of their projects require an ROI calculation? Do all parts of their project require ROI calculations? Do they ask for ROI for the graphic design as well? ROI of copywriting? Does each new employee they hire have an ROI calculation tied to them?

If you're competing against other vendors, does each have to come up with their own ROI? If the numbers are just "made up," then there could be huge discrepancies between each vendor. Also, how do you know you can provide more ROI than someone else?

Personally, I like to think more in terms of solving problems rather than justifying ROI. Those on the business side have problems, and they know how much it's worth it (to them) to fix those problems. The problem might be that employees are spending too much time looking for information on the Intranet, or there aren't as many sales on their ecommerce site as they'd like, or too many customers are calling in because they can't find out how to download new drivers. I think if you can guarantee you'll fix those problems (which, in most cases, you should be able to do), then they'll weigh your fee against the artificial value they've placed on the problem getting fixed.

I think most people don't care about IA. They haven't read Lou and Peter's book, they aren't on SIGIA, and they don't look at boxesandarrows. They have business goals, and whether you use IA or AI or voodoo science, they don't care, so long as you can meet those goals at a price they think is reasonable.

Comment: George Olsen (May 16, 2002)

As far as proving our value, it's worth quoting from the action item part of a Forrester report "Get ROI from Design":

"Let UXP experts prove their worth and share in success -- Few usabilty testing and interaction designer experts we interviewed have business measurement practices in place. Companies that hire these vendors should plan measurement strategies when negotiating their next engagement. Push for minimum-improvement guarantees and consider upside in the form of a shared increased net. This practice will accelerate the demise of self-appointed authorities in user experience and make truly scarse resources -- like cultural anthropologists and information architects -- more willing to go fight fires at the client's site during implementation phases."

Metrics are coming. It's a question of whether we define ones that we think are best suited or whether we let others define them for us.

BTW, overall the Forrester report argued strongly in favor of paying more attention to user experience.

Comment: Paula Thornton (May 17, 2002)

Can I go back to my original point on the FLAW of ROI. Fundamentally we are professionals who focus on the 'customer perspective' (otherwise read "stakeholder"). The fundamental perspective of an "investment" is a business focus, not a customer focus. It's trying to justify the customer needs financially. Financial justification and economic justification are different.

Businesses don't fundamentally exist without customers (or other stakeholders). Focusing on things to serve the needs of customers (fundamental to IA activities) is generally justifyable by any means. The real issue comes in the intelligent allocation of resources. For example...how much justification is used to determine the amount of budget spent on sales? Yet sales people run around 'chasing' sales when sales are routinely 'lost' to broken channels. I continue to posture that I can use a portion of monies currently spend on sales to 'chase' customers and help customers that are 'already there...willing and ready' to complete the transactions they are trying to engage in.

These are not financial arguments (ROI-based); these are economic arguments (allocation of resources). That is my fundamental argument against a FOCUS on ROI. Ours is not an ROI-based issue, it's a resource allocation based issue (fundamentally economic).

Comment: Andrew (May 20, 2002)

I've had hte (maybe) somewhat weird experience of pretty much *begging* to include metrics in projects and been told *by the client* "yeah, well, let's not bother." This has been especially bad here in Germany, where pretty much everyone meets the concept of IA with the reaction "that sure sounds important, you just make sure that stuff comes out good on this project."

In fact, it's the biggest client that I worked with seemed to have the least interest in actually proving that improvements were being made (or not). Of course, they spent about a million $ on a Vignette system, so perhaps they didn't really want to look that closely.... But when I suggested things like: "let's test to see where people have the most problems, then we'll promise to improve that by some reasonable amount" or "let's test how well the current search engine works to see if it's worth spending time on that" they just...didn't care.

Comment: Tim Parkin (May 21, 2002)

Just a quick note from someone more on the business end but is still involved in IA.

The big point is "what is the reason that the website we are building exists for".

In some instances, ease of use is not the target. For instance, if I want to get sales prospects, I don't want to give the customer all of the information. I want to the customer to 'register' or to phone me. This makes their task harder but acheives the result.

You need to know the business perspective before you introduce this.

As far as metrics goes, in our case we introduced them as part of our build and told the client it was free. The next time we made changes to the site, we could show the client what these changes had done. The metrics enabled us to quantify our investment into IA or other aspects.

Customers are blinkered and sometimes you have to sneak things in. It won't pay off initially but in the long term you'll benefit

Comment: Bruce Lawson (May 23, 2002)

I was delighted to see that 72% want case-studies of successful projects. A useful companion piece to Louis' new edition would be "Usability: The Site Speaks For Itself" from glasshaus (declaring an interest here: I edited it), which has 6 long case-studies from the IA guys behind eBay, BBC News Online, Economist.com, SynFonts.com (a Flash-driven e-commerce site), MetaFilter and evolt.org. Hopefully, when read with Louis' practical book, it should give better stories to tell, and advance the cause of IA.
(shameless plug over).

Add a Comment:

Name

Email

URL (optional, but must include http://)

Required: Name, email, and comment.
Want to mention a linked URL? Include http:// before the address.
Want to include bold or italics? Sorry; just use *asterisks* instead.

DAYENU ); } else { // so comments are closed on this entry... print(<<< I_SAID_DAYENU
Comments are now closed for this entry.

Comment spam has forced me to close comment functionality for older entries. However, if you have something vital to add concerning this entry (or its associated comments), please email your sage insights to me (lou [at] louisrosenfeld dot com). I'll make sure your comments are added to the conversation. Sorry for the inconvenience.

I_SAID_DAYENU ); } ?>