July 2009

Yes I did just finish In Defense of Food by Michael Pollan, which has been really helpful to me in separating real food from cheap imitations. It's my newest filter for navigating the grocery store, providing an easy to follow set of best practices for avoiding nasty edibles that cause us more long term harm than good. And yes, this book can even help us with software selection.

Michael Pollan reinforces that we must be careful when we enter a unfolding situation, already having decided on certain outcomes. We software experts and users spend a lot of time grouping software into similar features, development models, support system, company backing and more. This work is extremely useful to helping our understanding of software, but must always be taken with a grain (or two) of salt.

So how do we keep our critical eye sharp without getting overwhelmed with the myriad of possibilities when evaluating software?

(1) Frame your context

Understanding your organization processes are key, we software geeks say this all the time. Also important is understanding your organizational capacity to change. Important factors include internal staff expertise, general workload, organizational growth pattern, and system of governance. Defining how you do the work you do helps eliminate features and services that are unnecessary or come into conflict with your organizational context.

(2) Use filters

Filters can help us make sense of complexity by removing some of the detail. Quality software reviews are very helpful filters as they provide both strong context describing who the review is for, and well defined categories to help you group software according to common and critical needs. Internal organizational experience with software are also great filters as they come with institutional knowledge of the organizational context, offering a unique opportunity to combine this knowledge with software experience to shape well fit decisions.

(3) Be discriminating, don't discriminate

Look behind the filter to understand what it does not include. Often filters will look at software from certain perspectives that hide important information. I often run into this problem working with nonprofit associations, for which many vendors have created "association management software". A smartly compiled list of best vendors in this space is a great filter to have. However, it turns out the needs of associations are far more diverse than these association management packages support well. By limiting the focus on association management solutions, nonprofits often miss out compelling offerings provided by donor, membership, constituent management and content management vendors. Vendors generally are one good source of information about their products, however it is much better when verified against independent sources with specific experience using the vendor services.

(4) Test assumptions

Keep a critical eye on your plan for a new system. How many "needs" are supported by assumption versus research? Some nonprofits face difficult change management issues by launching a new system where the core users were not adequately engaged in planning and implementation. Others are disappointed to find that features that claim to be present really are no good for their needs. Beware the salty snacks - some tools make it seem so easy to keep adding fields or new features, but they can make your system very sick with bloat if overindulged. Understand the source of any advise or reviews you receive - what is their context? Who is their intended audience?

Why don’t my report counts show the same numbers on your new system as they did on my old?

Ugh, do I ever hate to have this question come up in a project review meeting. It is surely my candidate for the question consultants least like to hear from a client.

And this question usually follows by a few weeks or months a close runner up question, Can you convert all my old data into the new system. Clients who ask this and consultants who answer yes are looking for trouble for themselves.

This has been on my mind as our team approaches the end of a current and some recent projects whose conclusion has been directly linked to resolving data ‘errors.’ Sometimes you find unexpected joy, such as last winter when a conversion from a really old Raisers Edge version to Giftworks provided historical donation totals that matched entirely. Not to be a pessimist, but yup, I was surprised.

In our hearts if not our experience, we all know that data conversion gaps occur way more often. Here are some thoughts for “both sides” to keep in mind in approaching that next software upgrade.

First, as a background philosophical thought, if an organization’s data looked just right in its old system, would you still need to upgrade or replace? Business practices change and the old ways no longer work. We generally welcome change and we can expect staff to look forward to operating in new forward-looking ways. Yet fitting the old data to the new may be, as a colleague said yesterday may be mixing apples and oranges. The old and the new may never align again.

If you do convert piles of old transaction data or program enrollment data, and the new system has involved strategic planning and incorporation of new business rules, accept that reports will look different. Embrace the new and appreciate it.

Second, conversely on the human or philosophical side of things, you have to make an estimate of how much staff not just “hate the old way” but really are ready to “embrace the new way.” I think of a data conversion some years ago where staff continued to run mailing lists from their old system months after they were doing data entry in their spiffy new system, with its way cooler list generator. Hostage syndrome. They had grown hostage to the features that bore down on them in the old system. So if things ARE going to look different, you need to prepare everyone for it 100% or life will be difficult.

Part of the way to do this is to make sure staff at all levels of the organization are in the room when you discuss data migration. It is not sufficient to just look at reports and lists and to talk with the managers who use them. Let everyone hear how admin and program staff actually enter data, the compromises they make, the guerrilla warfare they may be waging against existing data validations to put what they want where they want it. This may be sobering to senior management and support realism about the conversion.

Third, consider if you can just leave historical data where it is. If you have been paying for a proprietary on-line data system that you want to stop paying for, you may have no choice but to extract old data and put it somewhere it can still be used. If not, why not just continue keep the old system available to run historical reports and start fresh with the new? Two pending conversions to Salesforce to replace a collection of spreadsheets and old Access databases have been way pleasant from a decision to not incorporate years of messy and unaligned old data.

Deciding not to convert may be a major policy decision if senior management or the board is used to a certain kind of review. I don’t underestimate it. But it is worth pursuing the question at the outset, before anything starts.

Fourth, consider converting just top level data, such as all primary demographic and status information for current contacts and leaving aside in the old system. Start with just what you know to be clean and essential.

Fifth, a variation on the third, fill in complex and messy old data by hand over time. As new cases or activity occurs for a contact, fill in or correct the gaps in their data from the old system. Meanwhile, don’t try to run multi-year historical reports on the new system

Sixth, if the data is messy, clean it up parallel to and prior to switching to the new system. My experience: most of the time, it is going to be easier to correct messy data either in its present home or in some convenient way station. In another conversion to Salesforce, where the data now resides in Filemaker, we have agreed to freeze data entry for a bit, and pull the data into a combination of Excel or Access where it will be more susceptible to bulk manipulation and checking.

Seventh, tell the consultant everything you can. If you are relatively new and the system is not well documented, find the staff members who have been there long enough to explain quirky things lost in the mists. In a current conversion from a proprietary undocumented membership system to CiviCRM, we have had one gotcha after another. For example, the current system shows five membership levels. The new system will have the same five membership levels. The paper forms show five membership levels. We checked at the beginning. Great! Easy! Except a faction of the members, in their actual extracted data showed a sixth membership level. And CiviCRM importer chokes on those records so it has to be addressed. OK, we figured this out, but if you add up field by field the consultant detective work to resolve things like this, you suddenly have blown a reasonable data migration budget.

Eighth, agree to leave data conversion out of proposal bids and do it on a time and materials basis only. This will enable everyone as the inevitable problems arise to make pragmatic decisions. Yes, it is worth it to write a script to fix this problem. No, it is not worth it, leave it alone. Or, yes, we need this and organizational staff will fix it ourselves or hire temps to do it.

I regularly resolve not to submit proposals where I have to include data conversion in the cost estimate, and I regularly find myself unable to keep that commitment. Even if you allot a few hours as part of a preproject assessment, it is unlikely you will find all the problems. A current project that is not exactly a system migration, but instead will need to enable staff to use purchased business data in a Drupal-based research and organizing framework. The data is good, used nationally and expensive! It should be clean say the developers! Yet even here we find quirky things. What should we make of a crucial research field which sometimes is stored as an actual dollar amount and sometimes as a coded list of dollar values. Since this is an ongoing data feed and not a one-time data conversion, we need to address it so that data searches will work on both kinds of cost valuation figures. And it’s just not realistic to expect to find all this stuff before you get pretty deeply into it.

Nine, if you are the consultant, you often, not always, do a trial conversion, leave it on a development site for a while for testing, and then do the final conversion to go live. It is really important to carefully document the conversion steps. In one really messy conversion, we ran, tested and refined the 40 or so separate conversion scripts several times before deciding the soup was ready. Different people contributed to the scripts. Having the scripts catalogued and organized lessened the pain of having to rerun them. Plus even a year later, it was possible to go back and find the script that massaged this messed up status category or performed some other task when the client asked about “what happened to the people with interest code x”?

Ten, hmm. I am leaving a slot for you, veterans of data conversions. Your thoughts? I could certainly use them…

Meanwhile, this post derived from a suggestio n from fearless editor Laura Q to write about the question consultants least like to hear from clients. When I posed this to my co-workers here at Database Designs, I got so many suggestions, I first thought of doing a “top ten” and decided to just start with my “favorite.” This could turn into an occasional series, and to balance, my colleague Mimi suggested one on questions consultants MOST like to hear from clients.

I was at a great talk about Trends in Usability Research last night, hosted by MaineUX/ MaineIxDA. Kyle Soucy of Usable Interface - a nationally recognized leader in usability testing - talked about some of the newer approaches that people are taking to user research.

It's apparently been a long two years since I was last doing user research, as there's now whole classes of tools that I didn't even know existed. In particular, there's some really interesting online tools that will help you do unmoderated user testing. For most of them, you'll still need to screen and line up your users, but instead of talking to them one-on-one, you can point them at a website and look at the results later.

You'll of course lose some of the detail that you get from individual conversations and being able to ask the user questions one-on-one. But it can allow you to test more users, and thus get some quantitative heft behind your results - which can be particularly persuasive to decision makers.

Here's some of the tools Kyle covered:

• Loop11: Currently in a free beta stage - request an invite, and you'll get a free login and password. You define the tasks you want your users to conduct, and then send your test participants to your testing site. They see your website (or whatever you're testing) with a header frame that tells them what tasks they're supposed to be doing, and solicits their comments. You can then analyze the results by looking at the success rate for tasks, time to complete, and other useful metrics.
• UserZoom: It's a little harder to get a sense from their website, but I think based on Kyle's talk and their info that this one is an up-market version of Loop11. Kyle estimated it at about $1000 per study, depending on the number of users you're testing.
• Treejack: Free for up to three tasks, or $109/ month otherwise. Very cool in it's simplicity. You upload a set of terms or categories in a hierarchical structure from an Excel spreadsheet, and it creates a simple drill-down interface from them. You then define some tasks (for instance, the term you want your user to look for), and point your test participants at it. They try to find the terms in your structure, and you can see overall success rates and time to complete.
• Chalkmark: By the same folks as Treejack, with the same pricing structure, and even simpler. Upload a screenshot, and define one or more tasks ("Find information about our Executive Director"). It records where they click on the screen, and shows you a heat map of the aggregate results.
• UserTesting.com: Interesting and cheap, but questionable. For $29/user, people will walk through the tasks you define, and you get an actual video of their actions and thoughts, and a write up. However, they're not testing with actual users. Rather, they're paying people to do nothing but sit there and user test sites for all their clients. So it's likely they'll have a somewhat distorted perspective of what it's like to use sites - though better than nothing, certainly, and better than not asking anyone external to your organization at all.
  

So there's some very cool stuff I knew nothing about! Are there others in this realm that you've had good experiences with?

As the internet has progressed from a shared source of information to a primary communications tool, a natural offshoot of the migration has been where the two things meet: people referring internet information. If you're active at all on Facebook, Twitter, MySpace, Friendfeed, or any of the numerous online communities, big or small, then you are regularly seeing links to useful articles and blog posts; cute YouTube videos, and entertaining photos. Much of this information is passed along from online friend to online friend, but where does the first referral originate from? Usually, it's somebody's RSS reader.

The main reason that I'm such an RSS advocate is that I believe that it's the tool that lets me find the strategic and useful needles lost in the haystack of celebrity gossip, prurient content, and corporate promotional materials that they're buried under. But it isn't "RSS", per se, that does the filtering -- it's other people, whom I call "information agents", who do the sifting. If I want to keep up with fundraising trends, a topic that interests me, but, as an IT Director, isn't my primary area of expertise, I'm not going to spend thirty minutes a day doing research. I subscribe to some very pertinent blogs, and I follow a few people on Twitter and in Reader who find the important and insightful articles and share them with me.

Now it appears that Google wants to cut out the social media middlepeople. As I alluded to in my article on RSS, and fleshed out in this post about sharing with reader, the ability to refer information that you find in Reader is one of the things that makes it so powerful. Last week, Google seriously upped the ante by adding Twitter/Facebook/Delicious-like following, "liking" and sharing to the mix.

Here's what the new features do:

Sharing now lets you share with the world, or just those members of the world that you want to share with. Google has always allowed you to share items, but connecting to other people was a bit arcane and limited, as, by default, Google only allowed you to connect to those that you chat with in GMail. If you read up on it, you learned that you could change that to any defined group of associates in your Google Contacts (all of this assuming that you use Google Contacts - many Google Reader users don't). As someone who does use all of the Google stuff, I still found that opening this up to 80 or so people in my contacts didn't make it clear to many of them as to how they could connect with me.

The new Following feature lets you follow anyone who is willing to share, not just people that you personally communicate with. Now my shared items are marked as public, so anyone can follow my shared items feed by clicking on "Sharing Settings" (in the "People You Follow" section) and searching for me by name or email address. Once you locate me (or someone else), you can (and should) browse through their items to make sure that they share things that you'll find useful. For example, I share a lot of things that are on the topics that I blog about here. But I also share items related to civil rights issues and the occasional link that I find funny. Since humor and politics are very subjective topics, you might want to be sure that you're not going to be annoyed or offended you before you subscribe to a feed.

But the internet is not just about who you know. The Like feature allows you to find new people to follow based on common interests. You'll note that certain articles have a new note at the top saying "XX people liked this", where "XX" is the number of people who have indicated that they like the article by checking the option at the bottom of the post. This message is a link, and clicking it expands it into links to each of the people who "liked" it, allowing you to browse their shared items and optionally follow them. This, to me, enables the real power of the social web -- finding people who share your interests, but have better sources. It's what initially was so exciting about social bookmarking service Delicious, and it's about time that Google Reader enabled it.

I'm hoping the Google's next round of Reader updates will improve our ability to not just tag and classify the information that we find, but also share based on those classifications. That will enable me to selectively publish items that I think are of interest to others, perhaps sending nptech links to Friendfeed and the humorous stuff to Facebook. But I welcome these improvements, and I appreciate the way that reader becomes more and more of a single stop for information discovery and distribution. The Internet would be a messier place without it.