December 2008

Agile development and the 70% Rule

I have been wrestling with how the concepts of “agile development” work in smaller software projects. Agile refers to a range of newer methodologies for planning, designing and implementing new web and data systems that include themes like these:
  • Designing and working in short phases, or iterations, of the desired end system.
  • Getting one set of new software features into full use and then moving on to the next
  • Not overreaching your budget or schedule in what you seek to accomplish at any given moment
  • Working in small self-organizing teams that include a client rep as well as developers and that meet regularly and work in person to the extent possible.
  • Focusing on getting software working and in use rather than on extensive documentation and planning.
We have been working through our own adaption of agile for Drupal projects where we see factors that favor it:
  • The client generally can’t predict exactly what they are going to need until they start using the software. Requirements change as new web and data systems get used.
  • Except in large businesses or organizations, the client team may be weighted toward folks with little experience in serious software design and who need time and training to work effectively.
If these problems sound familiar, you might do some more reading. Here are some good starting points:
The original “Agile Manifesto”: http://www.agilemanifesto.org

Wikipedia article

Extreme programming flavor of agile

http://martinfowler.com/articles/newMethodology.html

You can find a lot more starting here, including lots of books by the Agile Manifesto authors. A lot of what is written out there is kind of abstract and geared toward software professionals already familiar with the issues. In fact, we see two types of problems in trying to adapt and stick with agile methodologies.
  • The frameworks often seem too rigid to use in smaller projects where the client-consultant team is new to working with each other. After reading a bunch of books and web articles about it, I sometimes think that agile advocates need to become a bit more agile in their advocacy. It should be easier to implement agile itself in short agile phases or iterations and easier to adapt to small teams and small budgets.
  • The other problem arises in grant-funded technology projects or similar situations. Organizations (and businesses, for that matter) may have a modest annual budget for software support and enhancement and then have to wait a long time for funding to do something new and strategic. When that happens, it may be an all or nothing situation: get everything pre-approved and documented in detail in advanced and done and in use in one cycle. You wait and you get one pass at the funding. This framework also runs smack into the agile point of view.
What can one do about these things?

First, if you are beginning a new software selection and implementation cycle, bring “agile” into the evaluation criteria for software or consultants you are considering. Ask how whichever of the ideas that sound useful apply, and learn from the process.

Second, there is a lot of room and need for educating funders, whether internal or external, on these matters. It would be great to have an extended dialog about how more agile-aware grant-funding could better support technology-infused projects.

Third, developers need to find more popular ways to discuss these concepts. I’ll end with one neat coincidence I have been thinking about. Agile software development is a lot like what we call the 70% rule in the style of Tai Chi I do.

If you have ever done Tai Chi, you may have experienced guidelines that say never aim to learn everything you can or to progress at your full capacity. In the school I go to, this is expressed as the "70% Rule." Make an estimate of your 100% goal level, and then shoot for 70% of that. If you are just getting started or recovering from an injury, reduce to 60%, 50% or less. You aim to achieve what you can comfortably achieve, get used to that. You soon find that your 100% mark has moved up, and therefore you have a new 70% level to work at.

Tai Chi’s Taoist classics have perspectives from which this rule emerges. For example, the Tao de Ching has a passage that says,

Everyone in the world knows that when the beautiful strives to be beautiful, it is repulsive.
Everyone knows that when the good strives to be good, it is no good.

(See more like this on the Energy Arts web site.)

Clients and developers alike frequently have trouble wrapping their minds around such a perspective in software. The pursuit of perfection, of not just 100% of capacity, but 110 or more, is common, whether it’s in the desired features, data conversion or other factors. From trying to satisfy everyone and everything, things get messy and problematic on the fringes.

If you go to 100% to make sure the organization gets everything it hoped for, instead of reducing stress, you may increase stress. Instead of building capacity, you may do damage to staff and developer patience and commitment. Instead of adding resources, you scatter attention.

I’ve been thinking that the 70% rule could offer kind of a corollary to agile development. I’m planning to try it as a more accessible way to get at the same concepts. Keep the long term goals of perfect organizational growth, improved workflow, strenuous contact and program management. Then figure out what the organization can truly absorb right now, and back off to what it would mean to achieve 70% of that in smooth, effortless adoption. Implement that much, get used to it, evaluate it, then set a new higher set of goals, and plan to achieve 70% of that new mark.

Maybe you have experienced some other way to introduce these concepts you'd like to share.

New article: A Few Good Email Discussion List Tools

Rounding out a fair number of email articles from us recently, here's A Few Good Email Discussion List Tools - taking a look at the software that's available to help you facilitate email conversations among your constituents, partners, or staff. Compared the dozens or hundreds of broadcast email tools, the options are surprisingly limited - but we round up the choices recommended by our contributors.

Making a case for upgrading to WordPress 2.7

One of the lesser joys of using open source & free software has been the need to do important security and core feature upgrades manually and sometimes frequently. Although I have been a long time fan of WordPress as a super simple CMS for small sites that need dead easy administration for non-tech types, I have not loved the continual updates it requires. I will admit that this has been known to make me crabby in the past and perhaps not as diligent about keeping all my sites up to date as I should be.

Well, WordPress just released another major upgrade - 2.7 (Coltrain), but this one is clearly worth doing and doing right now. Especially since, from here on out, they are automating the upgrade process and making it available from within the administrative interface. Woo Hoo!

If you are using Word Press and need to upgrade here are a few good reasons to bite the bullet and do it now. These are primarily from the perspective of using WordPress as a lightweight CMS and not as oriented to hardcore blogging, where there are also significant improvements, especially with comment management.

The value adds:

Last manual update ever (almost). As mentioned - they have pretty much automated the upgrade process for future releases. If you have gotten fancy and tweaked core files you will need to save your changes and incorporate them of course but for most users this will make it possible for non technical folks to keep their systems up to date without as much help.

One of my favorite improvements is the integrated plug in browser and installer. This little feature not only lets you see if a plug in exists for what you are trying to do but lets you know if its compatible and automates updates, meaning no more need to download, unzip and FTP. Its very nice, but maybe a bit dangerous since they have made it way too easy to load up your site with a ton of bells and whistles.

Themes also now provide alerts when updates are available but as of yet, no automated one click install.

Most secure version yet. Current WordPress users might remember the onslaught of hacking that occurred last spring and hopefully already had upgraded to at least 2.5 and avoided the pain those of us that had to dig out of a hack experienced. Although more security was introduced with 2.5 and subsequent releases, every new version protects you from more known vulnerabilities, so having the latest is always a good idea.

New and actually improved interface. Through a community wide effort WordPress enlisted some top talent to make the administrative experience so much smoother and shinier and more usable to boot.

The vertical navigation might take a few minutes to get used to but it does make a lot more sense. Another neato but also practical feature is the ability to choose what tools and content display on each administrative screen - if you don't care about the latest theme news you can hide that feed and you won't have to scroll past it anymore.

Faster, easier, writing and editing. Quick edit and bulk editing for pages and posts is really useful for larger sites and much faster than loading individual post editing screen for simple changes to the non-content areas. I'm still waiting for a faster method of ordering pages but this is a big step in the right direction.

And the behind the scenes speed enhancements they have included are a major bonus for those of us on shared bandwidth. The fancy 2.5 "new and improved" editor was a bit buggy and heavy to load but 2.7 seems to have smoothed things out.

The more robust media manager is another nice addition for anyone that wants to offer PDF and zip files for download on their site without the hassle of FTP'ing files to the server.

The downsides.

Lack of plug in or theme compatibility. While I could see clearly that this particular upgrade was a huge step forward and well worth it, I almost decided to wait on one site because it would mean losing a particular plug in what was not compatible. And looking up the author it was clear he wouldn't have time to update it in the near future. In the end I made the leap and will just have to wait for some of my favorite add-ons to catch up. If you use a specific plug in that is very important to your overall site functionality or your theme is heavily customized and you have hacked core files, this can be a tough call.

The good news is that the WordPress community is pretty intent on promoting core improvements to help theme and plug in developers make their work compatible and create new options more easily.

You have already put off upgrades for too long. If you are on an older version of WordPress getting to 2.7 may be a bumpy ride. For anyone using WordPress versions below 2.5 there can be some headaches in making the leap - often around the changes to the way the database works in the newer versions. Making the switch through a series of progressive updates can start to look like shoving a square peg in a round hole. It might be possible to export your existing content and import it into a fresh install of 2.7 but you might lose some information along the way.

Upgrades from later versions (2.5 + 2.6) have been fairly painless but as always, backup before you start making changes, turn off all your plug ins and follow the instructions. I look forward to a pain free upgrade future, let's hope it happens.

There Ain't No Such Thing As a Free Software Package

If I were to group all the questions I get into categories, one category would be far and away the most numerous. I'll call it the "options for people who can't afford software" category. As in, "Well, those databases for $300 or $20/month sound great, but we can't afford to pay anything - what can we get that's free?"

No question riles me up more. This is a really dangerous mind set. Would you think this way for other types of investments? "That Executive Director candidate seems really great, but who can we get that we don't have to pay?" "That office space seems perfect for us, but $100/month is a lot of money - what can we get for free?" Okay, possibly some nonprofits actually would do this for other things... but it's equally dangerous. Things worth having are worth paying for.

And not paying for them up front almost always means that you're paying in some other way. That free office space is great until it starts raining asbestos on your employees, or you're evicted to make way for a paying tenant. That free software might seem great, but if it doesn't do what you need, or your staff can't use it, or it's full of bugs... then it's useless to you, and it doesn't matter how free it is.

Also, I reject the likelihood that your organization has no money of any sort to devote to software. If you can't raise $300 to purchase a donor database that will help you solicit donations more effectively, you need donors much more than you need a database. I'm not saying that you should spend tens of thousands. I'm saying that you should decide the priority of having effective software and assign a budget to it accordingly. Yes, that might mean you'll have to fund raise for it.

Don't get me wrong. Free stuff is nice. I use some free software myself. But it's a BONUS that it's free. You can't start with that as a requirement and expect to end up in a good place. If a software package will help you save time or money, or earn more money, then it's worth paying for....and if it won't, you shouldn't waste your time with it.

Can open source source software save you money?

Next year, given what is likely to be a grim funding year, nonprofit organizations are going to be hunting for ways to save money on technology. There are, of course, arguments that IT budgets should be, at least, level funded during slim times, but the reality is that organizations are going to reduce budgets across the board. One question that will inevitably be asked: can free and open source software save organizations money?

The answer, of course, is a solid maybe, but also a resounding yes. Confusing, huh? Open source software is both free as in "beer" as well as free as in "kittens." There are no license fees, but it takes care and feeding.

The most important part of the equation is what you are implementing, and whether or not you need to factor in migration costs. Nonprofit organizations that did migrations to open source software from proprietary packages with large license fees during relatively fat economic times are reaping the benefits of that change now, and are in good shape to weather the storm. Organizations that haven't been able to do that migration might find those costs to be prohibitive at this time - which is unfortunate.

But if you have a migration planned anyway, now is absolutely the time to look at open source software. At this point in the maturity of most open source packages that nonprofits would want to use, the implementation cost is very much in line with the implementation costs of proprietary software. So that means that you are saving money - no cost to acquire, and no long term license or maintenance fees.

All of the above adds up to that solid maybe - implementing open source software in your organization might save you money depending on what you are implementing, and what the costs are for migration. Where does the resounding yes come from?

This, if any, is the time for organizations to reject the standard "every organization for themselves" mentality of software acquisition and development. Find a solid open source package (like CiviCRM, for instance,) and help fund extensions to that software with other organizations that help make it what you need. Find 5 organizations that do similar work, and collaborate to build an open source application that can work for your part of the sector. Release it so a community can develop around it, make sure to make it modular so that it can be easily extended. Make it full of APIs so you can hook other software to it. Build it with open standards so the data is readable in perpetuity. Doing this will mean you will get far more application for the money you spend. Of course, it all takes effort and work. But it's worth it - and the entire community benefits by an enriched software ecosystem.

It also ends up not just being about saving money. It also ends up being about building community - and community will be an incredibly important asset in the coming years. There is an appropriate popular culture reference: "live together, die alone."

Filling the Communication Gaps

We've come a long way since the Pony Express. It's hard to imagine living in a time when your options for communication were limited to face-to-face, sllooowww mail, and, perhaps, carrier pigeon. Today, we have the opposite problem: there are so many mediums to choose from that a key communication skill is to gleam the method that the person you want to reach prefers. I was taken aback by an Australian ruling that Facebook was an acceptable medium for serving subpoenas, until I read that the defendants had been unreachable by phone or email for months beforehand. At first I thought they were just avoiding the subpoena -- still a big possibility -- but then I reconsidered. How many people have completely abandoned their primary email accounts, assuming that anything in them is spam, in favor of only reading their mail on Facebook or MySpace? Probably a considerable number. I know, just from my day-to-day business dealings, that I will reach some of my coworkers more effectively by phone than I will by email, and vice versa.

So we have postal mail, the telephone, the telegram, facsimile, short wave radio, walkie-talkie and intercom holding up the old guard. And we have email, cell phone, IM, chat, IRC, blogs, Twitter, forums and social networking services charging in as new(er) mediums. And I'm sure I've missed a bunch. The internet has opened up a Pandora's box of communication mediums. So why use one over another? If we break it down to a manageable number of mediums, say, Phone, IM, email and Twitter, there are some intriguing differences. These differences don't imply that one is better than another, but, certainly, one is more practical, courteous or efficient than another in a given circumstance. I evaluate the mediums on a few defining attributes:

Private or Social? While allowing that you can send group emails and IMs, and hold phone conferences, these mediums are primarily suited for one to one or a few conversations, whereas Twitter, and many of the web-based mediums, are social, with a large and partially unknown audience included.

Ambient or Invasive? A phone call is invasive, as is, to some extent, an IM. The sender is sitting there waiting for a response, so the courteous thing to do is to immediately re-prioritize whatever you're doing and respond to them. Email and tweets, on the other hand, are casual mediums. Ignoring either one for an hour is within the bounds of the sender's expectations.

Convenient or In Need of Management? I can send and receive IMs and Tweets and forget about them; phone calls as well, although voicemail needs to be dealt with. Email, on the other hand, is a demanding application. i have to manage it, sort it, categorize it, and clean it up.

Disposable or Archived? Phone calls and IMs, unless I record them, disappear after the conversation is ended. Emails and tweets are saved and searchable, giving me an always available archive of my communications (unless I delete them).

I suggested in a post last week that Twitter bridges the gap between email and IM, just as email bridged the gap between the letter and the phone call. Since then, I've been trying to figure out if a social, ambient, archive-able and convenient medium like microblogging is compelling in my organization. I took a look at Socialcast, one of the many corporate Twitter clones popping up, and I was very impressed with their implementation, which breaks the messages into statuses, ideas, questions and links.

Selling my staff on a tool like this is proving to be a challenge. The argument for it is fairly nuanced, and urging anyone to try something new on faith isn't easy. They're asking why this is better than the Microsoft Messenger chat application, or a more full-featured Sharepoint site? Those are good questions. Micro-messaging software lacks some of the features that these other mediums sport, but it provides a very simple and powerful, approach to information sharing that is far more collegial and less invasive than chat, while it's simpler and quicker to use than Sharepoint. And my bet is that, in the war of communications mediums, it will ultimately be the ones that are easiest to use and least disruptive that win. Or it should be.

New article: Online Fundraising for Year End Procrastinators

Ack! Forgot to link to this one! In the Nonprofit Times, I talk through a some steps and tools for Online Fundraising for Year-End Procrastinators. At this point, you've probably procrastinated too long for even this process (which is likely to take about two weeks to get up and running), but maybe you can get some good tips for next year.

New article: Selecting Software on a Shoestring

Here's a new article that I'm really proud of: Selecting Software on a Shoestring. I know it's not the sexiest topic, but this is the type of thing that we often overlook as technology consultants and service providers - sometimes you don't need that huge, all consuming process to pick a software package. In fact, sometimes that huge, all-consuming process is really silly, and all your really need just a simple set of question that work.

Where's your data, dude?

A recent conversation on a nonprofit technology mailing list got me thinking about what people know about where they want to put their data, and what the risks are of having data in one place or another. Particularly, what are the pros and cons and risks and benefits of software as a service, and cloud computing when it comes to the control and security of your data.

You can think about this on three axes: security, access, and control. First, security; to get it out of the way, I'm going to assume, for the purposes of this post, that the person administering the application and/or the company hosting your data have done everything right to secure the data. If this is true, the security difference boils down to a somewhat increased risk due to increased exposure of your data outside of the firewall, vs. a somewhat decreased risk inside the firewall. These are important considerations, and, really pretty straightforward.

Second: access. What do I mean by access? On one level, it means physical access - can you physically access the data? Ultimately, it really means the ability for you to look at, manipulate, download, remix, integrate, back up, or in any way handle your data as you please. The spectrum goes from the most access in an open source application hosted locally in a box physically in your sight at the moment, to the least access in proprietary applications hosted offsite. Why does the location as well as the license of the software matter? As many of you have noticed with many proprietary applications, it can be very hard to get your data out of them. Open source applications are generally based on open standards, and since the source code is available, it is always possible to get your data out. Of course, it is critical to make sure that any application you choose gives you the level of access that you need at a cost that you can afford. However, there are some interesting examples inside those extremes. If you compare many on-site proprietary fundrasing packages to externally hosted Salesforce.com, Salesforce.com will come out way ahead on data access, because most proprietary fundraising packages make it hard to easily access your data (and many times, you have to pay for APIs and the like,) and Salesforce's basic strategy is to be a completely open platform. As well, some open source applications don't have as mature APIs as some proprietary applications.

Lots of software services don't really give you easy ways to, for instance, create a backup of all of the data that is recreatable in a useful manner. For instance, there is no easy way to backup your google apps data - you need to use third party tools to back up a lot of it, and some of it requires a fair bit of manual labor. If your google account is suspended or deleted by accident (yes, it happens) you are out of luck unless you've taken those precautions.

The third axis is control. By control I mean, to what extent do you actually have control over exactly how your data is used. Again, this ranges from the most control in open source applications in servers hosted physically in your presence, and the least is proprietary applications hosted elsewhere.

There are several aspects to control. One is simply: can the service/company use my data as it pleases? Read the Terms of Service. You'd be surprised how many online services (especially Web 2.0 services) basically say that by using the service, you agree to give them license to use your content as they please.

Another aspect of control has to do with what a service can do with your data under certain situations. You know if the activities of your organization might raise suspicions, or might risk skirting the line of the terms of service (this is true of many activist and human rights organizations.) Companies care way more about their bottom lines than they do about any one particular client's data. Be sure that they will happily close your account, or hand your data over to the authorities if it looks to be in their best interest to do so. And you will be out of luck if that happens.

Another aspect of control is what might happen to the company holding your data. Is it a startup? How's the financial situation? Has the hosting company been around a while? In the current climate, where startups are bleeding staff left and right, and venture capital is drying up, it's even more important than ever to make sure that the company hosting your data is going to be around for a while.

There are lots of benefits to software-as-a-service and "cloud computing." But they are not without risks. Think about security, access and control of your data when you are evaluating your options.

Raising Money in a Down Economy

Eek! It seems like every tech vendor has a marketing piece these days about how we should spend more $$ on tech to raise money in a down economy. One vendor most recently argued that it is perfect time to get that integrated donor management solution and get more targeted asks out there, "fast"!

Well, if you are desperate, maybe fast is your only option. Then all you have left is to choose between cheap, or good. That's not a great place to be when overhauling your critical information management systems.

I would argue that a thoughtful evaluation of existing systems can often yield substantial bang for the buck, while leaping into new systems is fraught with peril. Five strategies that have worked for me in past down cycles:

(1) Talk to your networks. Revisit with your friends and close connections to reinvigorate opportunities on hold, or to discover new collaborations going forward. Call people!
(2) Reach deep into your contacts: Mail lapsed donors. Contact past clients. Reintroduce yourself to folks that have fallen off your radar.
(3) Increase internal collaboration: Discuss with staff opportunities for engagement that may reach across departments such as development, membership, volunteer management, etc. Are there collections of contacts that have fallen through the cracks?
(4) Avoid the "Hail Mary" project: A radical change in systems or focus is adding a lot of unknowns, work and risk in an already uncertain time. Examine your strategies based on longer term trends and in general, stick to what works.

Which leads me to #5. One recent donation solicitation I received had a picture of a crumbling office building, and staff outside with tattered shirts. Having recently visited them, I remembered that everyone seemed to be well clothed. I called them anonymously, and asked about the photo and whether they were accepting donations of used clothes for the staff. They assured me that all was well internally, and that it was the "cause" that is losing its shirt in this economy. I wondered out loud if maybe illustrating the cause might have been a better way of fundraising for it...

(5) Try not to sound desperate: Continue to solicit dollars based on the great work they will support.