So much for testing

My category-based RSS feeds stopped feeding through to Testing Reflections and Test Driven a couple of weeks ago. They had worked fine, and then they just stopped. I assumed that the problem was at their end; either they didn’t think much of my testing posts or their feed polling software had some weirdness in it… But you know what assume does…

I’d got it into my head that because the feeds used to work and now they didn’t, and that I hadn’t changed anything at my end, the problem must be the consumers… Wrong… My feed had crap in it because I don’t really understand how the MT templates work, I’d been sloppy and I hadn’t bothered to test the feed apart from running it through my aggregator and eyeballing it in IE.

Since, as far as I can remember, I didn’t change anything, and it worked to start with, I’m guessing that the consuming sites changed something and the change made the consumer less accommodating of my failure to adhere to the specification… I guess I could blame Postel for his comments on network protocol design; “Be liberal in what you accept, and conservative in what you send”…

Assuming, for a moment, that I’m correct in my guess about the reason for the sudden failure and that the original consumer was happy to be liberal in what it accepted from me; i.e. it recovered from the errors in my feed and managed to work out what I meant to say, but the upgraded consumer was less liberal and therefore it began to reject feeds that had previously accepted. If that’s the case then my gut feeling is that being liberal in what you accept is pants. Yet again, short term happiness wins out over long term stability…

In a world where there are 1000 providers but 400 of them don’t quite provide the correct format of data it can be tempting to be liberal in what you accept so that you can deal with all 1000 of them rather than just the 600 conforming providers. I’d argue that this is the wrong approach and that it’s likely to lead to more work on your end of the contract. Since you decide to accept 400 broken implementations as if they’re correct what happens when the 401st broken implementation comes along that’s broken in a slightly different way to the other 400? To be equally accommodating you might need to adjust your implementation to allow the new provider to interoperate with you, after all, why is this broken implementation any less valid than the other 400 that you already allow? Of course if all of your competitors allow broken interop then there may be a business reason to be liberal, but…

If you were strict in your interpretation of the original specification then you’d initially only be able to interop with 600 out of the 400. If the 400 wanted to be able to work with you then they’d have to fix their non-conformant implementations. If every consumer was conformant then all providers would be forced to be conformant… Win-Win as the business boys like to say…

I know it’s not always possible, but, personally, I think it’s better to be strict in what you accept… Pain now, when you can control it, rather than pain later, when you cant.

Bill Veners has some interesting discussion on this, “Sloppy and Forgiving versus Strict Systems”.