« Source Code | Testing Archives | Way back »
Previously on "Practical Testing"... Having just resurrected this series of blog posts about testing a non-trivial piece of real-world C++ code I've fixed a few bugs and done a bit of refactoring. There's one more step required to bring the code in this article right up to date. The timer queue that is the focus of these blog posts is part of The Server Framework. This body of code has been around since 2001 and has evolved to support new platforms and compilers. One of the things that I do from time to time is remove support for old platforms…

Practical Testing: 36 - Timeout handle wrap

| 0 Comments
Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included two fixes that had been fixed some time ago and which I hadn't documented here. They also lacked unit tests... Last time, I wrote tests for, and fixed, the first bug. This time I fix the final bug. This bug is in the "lock-free" timeout handling code and it would cause the threaded version of the timer queue to do…

Practical Testing: 35 - Heap corruption

| 0 Comments
Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included two fixes that had been fixed some time ago and which I hadn't documented here. They also lacked unit tests... In this episode I find and fix the first of these issues by writing a unit test that triggers the issue. This bug is another edge case that isn't used that often by most of the code that uses the…
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. In the previous update in July 2014 I added custom intrusive…

Bug hunting

| 0 Comments
I've just spent a day tracking down a bug in a pending release of The Server Framework. It was an interesting, and actually quite enjoyable, journey but one that I shouldn't have had to make. The bug was due to a Windows API call being inserted between the previous API call and the call to GetLastError() to retrieve the error code on failure. The new API call overwrote the previous error value and this confused the error handling code for the previous API call. This was a bit of a "school boy error" and it was made worse by the…

Practical Testing: 33 - Intrusive multi-map.

| 0 Comments
Previously on "Practical Testing"... I'm in the process of replacing STL containers with custom intrusive containers in the timer system that I have been developing in this series of articles. The idea is that the intrusive containers do not require memory operations for insertion or deletion as the book-keeping data required to store the data in the container has been added to the data explicitly. This reduces the potential contention between threads in the application and, hopefully, improves overall performance. In the last instalment I showed how my usage of std::set in the timer wheel code could be replaced by…

Practical Testing: 32 - Intrusive containers.

| 0 Comments
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. In 2010 I needed to improve the performance of the code…

The curious case of the missing copy constructor

| 0 Comments
I have a tendency to write unit tests that are a little more invasive than they need to be; these tests make sure that not only are the results as expected but also that as many of the side-effects and interactions with other objects are as expected as well. So, for example, in my current WebSockets development for The Server Framework I have some tests which test that the correct data is delivered to the client of the API that I'm developing and also test that the API interacts with its buffer allocator correctly and doesn't leak memory. The…
I'm nearing the end of my WebSockets implementation for The Server Framework and have been dealing with various protocol compliance issues. Whilst I have decent unit test coverage I haven't, yet, sat down and produced compliance specific unit tests which walk through the various parts of the (ever changing) draft RFC and test each aspect. Looking back, I probably should have taken this approach even though the RFC was fluid. Anyway. One of the people involved in the WebSockets Working Group has a nice set of compliance tests for their implementation and they have made these tests are freely…

In response to @dhanji on unit testing

| 0 Comments
Dhanji over at Rethrick Construction has written an interesting piece on the value of unit testing. I agree with his conclusion; "So the next time someone comes to you saying let's write the tests first, or that we should aim for 80% code coverage, take it with a healthy dose of skepticism." But then I tend to take everything with a dose of skepticism...…

Practical Testing: 31 - A bug in DestroyTimer.

| 0 Comments
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. Whilst doing some development on a new server design I managed…

WinRM/WinRS job memory limits.

| 1 Comment
My tangential testing that began with my problems with commands run via WinRs during some distributed load testing are slowly unravelling back to the start. I now have a better build and test system for the server examples that ship as part of The Server Framework. I have a test runner that runs the examples with memory limits to help spot memory leak bugs and a test runner that checks for lock inversions. My test scripts are also more flexible and I've found a couple of bugs. Today I used a variation on my job object based memory limiting test runner to…
As I mentioned, I've been adjusting my build system and have finally got to the point where my lock inversion detector is suitable to run on all of my example servers during their test phase on the build machines. I'm working my way through the various example server's test scripts and adjusting them so that they use the lock inversion detector, can be easily configured to run the full blown deadlock detector and also can run the servers under the memory profiling test runner that I put together earlier in the week. Note: the deadlock detector mentioned in this blog…

Tangential testing

| 0 Comments
My theorising about the strange memory related failures that I was experiencing with my distributed testing using WinRS have led me to putting together a test runner that can limit the amount of memory available to a process and terminate it if it exceeds the expected amount. With this in place during my server test runs I can spot the kind of memory leak that slipped through the cracks of my testing and made it into release 6.2 of The Server Framework. The idea is that rather than just starting the server we start the server using a monitoring process that first…

Practical Testing: 30 - Reducing contention

| 2 Comments
Previously on "Practical Testing"... I've been looking at the performance of the timer system that I developed and have built a more specialised and higher performance timer system which is more suitable for some high performance reliable UDP work that I'm doing. Whilst developing the new timer wheel I began to consider the thread contention issues that the timer system faced and came up with a notation for talking about contention (see here). Both the general purpose timer queue and the new timer wheel suffered from more potential thread contention that they needed to because of the way the STL…

Practical Testing: 29 - Fixing the timer wheel

| 0 Comments
Previously on "Practical Testing"... I'm writing a timer wheel which matches the interface used by my timer queue. This new implementation is designed for a particular usage scenario with the intention of trading space for speed and improving performance of some reliable UDP code. The last entry completed the development of the timer wheel. This time we fix a couple of the bugs that I've discovered since I started to integrate the code with the system that it was developed for.…
Previously on "Practical Testing"... I'm writing a timer wheel which matches the interface used by my timer queue. This new implementation is designed for a particular usage scenario with the intention of trading space for speed and improving performance of some reliable UDP code. Over the last four entries I've implemented various parts of the timer wheel and adjusted the test code so that I could reuse the tests that I already had for the other implementations with my timer wheel. The tests needed to be tweaked quite a bit to take into account the different behavioural characteristics of the…

Practical Testing: 27 - Fixing things...

| 0 Comments
Previously on "Practical Testing"... To deal with some specific usage scenarios of a piece of general purpose code I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. Last time I left myself with a failing test. The problem is that setting a new timer on the timer wheel sets a timer that's relative to the time that timer wheel thinks is 'now' and the timer wheel's view of the current time could be slightly behind reality; see the previous entry for a diagram…
Previously on "Practical Testing"... To deal with some specific usage scenarios of a piece of general purpose code I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. The timer wheel trades space for speed and so far the development has gone well as I've been able to use the tests that I had already developed for the previous implementations to guide the new development. By the end of last time we'd got to the point where we had four functions left to implement...…

Practical Testing: 25 - Nothing is free

| 0 Comments
I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. The idea being that for certain specific usage scenarios the timer wheel will perform better than the timer queues. Last time I refactored the tests that I was using for the timer queues to remove duplication and I now have a set of failing tests for the new timer wheel. As soon as I started to look at making some of the failing tests pass I realised that having a heap of failing tests…
The most recent articles in the "Practical Testing" series have been discussing the performance of the timer queue that we have built. Once I had got some new, optional, performance tests in place to measure what we were trying to improve I eventually came up with a new approach and began to implement a timer wheel that conforms to the interface used by the other implementations of my timer queue. Whilst doing this it became obvious that there was duplication in my test code and so the tests have been refactored to remove the duplication of the test code between…
The most recent articles in the "Practical Testing" series have been discussing the performance of the timer queue that we have built. As I hinted when I first brought up the performance issues, the particular use case that I have which is causing problems might well be more efficiently dealt with using a different (more specialised and less general purpose) design. The timer queue has adequate performance for general purpose use and can handle timers set within a range of 0ms to 49.7 days with a theoretical granularity of 1ms. It achieves this by using a balanced search tree to…
The previous article in the "Practical Testing" series set things up so that we can measure the performance of the code under test with the intention of trying to improve performance for a specific set of use case scenarios. This time around I'll make a few changes which I hope will improve the performance and measure the effects of the changes with the performance tests that I added last time. One of the things that struck me about the code when I was looking at it during my profiling runs for my client is that when using the CThreadedCallbackTimerQueue in…
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. The tests were written for the pretty scalable timer queue implementation…

Unit testing AVR assembly language

| 0 Comments
My "hobby project" is going well but the AVR assembly language that I'm writing has now got complex enough for me to really miss not having some form of unit testing framework in place. So I decided to remedy that I work out how I could write unit tests for various pieces of assembly language. It turns out that with judicious use of 'mock' implementations of the code residing at various labels and traditional layering and separation of concerns I can build a set of unit tests that run in my AVR simulator and that either end up looping at…

Testing blocking calls

| 0 Comments
I'm developing the simple client server protocol code that I'm harvesting in a test driven manner. Although the code exists, as such, and I'm harvesting it rather than inventing it from scratch the harvesting is taking the "best" ideas from several similar implementations to create the version that will form part of the framework's example code. As such I'm treating this just like a new development and, mostly, writing my tests first. This has worked well, as usual, and has allowed me to mercilessly refactor whilst I go as I decide to incorporate features from different versions of the protocol…

Asserts and testing

| 0 Comments
Miško Hevery over at the Google Testing Blog has a few things to say about Asserts and testing. He's against asserts, specifically ones which perform null checks, as they get in the way of testing. Whilst I agree with his dislike of assertions (see here) I disagree with his dislike of null checks... Miško complains that by having an object check passed in objects for null and assert if they are this causes him problems in testing where he knows that he doesn't need the objects that are being passed in to test the functionality that he's testing. He prefers…

Asynchronous SChannel Servers

| 0 Comments
I'm currently working on an SChannel version of the asynchronous SSL connector code that we use to provide SSL support in The Server Framework. This will eventually be an alternative to the existing OpenSSL support that we currently provide and should prove useful for people who want a tighter integration with Microsoft's Certificate Stores than our current OpenSSL support provides. In many ways the SChannel version of the code is more complex to implement than the OpenSSL version as the OpenSSL SSL_CTX context object deals with data accumulation and record splitting internally; you just push bytes into it and it eventually pushes…

Practical Testing: 20 - Mind the gap

| 0 Comments
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. But... you need to be aware of the tests that you…

Writing testable code

| 0 Comments
There's a nice post by Miško Hevery over on the Google Testing Blog about Writing Testable Code. It pretty much sums up my views on testable code. Go read it!…
The code in the last two articles in the "Practical Testing" series have contained a considerable amount of duplication. This came about for a couple of reasons. Firstly part 17 was a bit rushed and secondly it was useful to compare the CCallbackTimerQueue implementation with the CCallbackTimerQueueEx implementation. I'm also a firm believer that in this kind of situation it's better to get both sets of code working independently and then refactor to remove any duplication rather than attempting to design a duplicate-free solution from the start. Anyway, this time around we'll remove the duplication by creating a base class…
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since then there have been various changes and fixes and redesigns all of which were made considerably easier due to the original tests. There has always been a potential for deadlock when using the timer queue, which is unfortunate but something that you can avoid if…

Another ISO 8583 transaction server

| 3 Comments
It has been a busy couple of weeks for me. I've been working on an ISO 8583 based transaction server for a client and I set myself some fairly tight deadlines. I actually developed the first version of The Server Framework for another ISO 8583 transaction server that I developed for this client back in 2002. It's interesting to look back at the original code and see how the framework has evolved (see here for the latest version), it's also nice to know that the first server is still running nicely and providing customers with the ability to top up their pay…

More on the CLR startup change

| 0 Comments
Last week I mentioned that some of my tests for my Win32 Debug API class had suddenly started failing. It seems that I was right and the changes are due to some .Net fixes that have been rolled out recently. The code runs and the tests pass if I run on a clean instal Vista x64 VM and fail on my day to day development box. It seems that my plan to "stick a breakpoint in mscoree.dll's _CorExeMain()" wasn't such a good idea after all. With the new updates installed and with an x64 process using the Win32 debug API…
Back in October 2007 I sumarised my findings from getting my Win32 DebugAPI based debug engine working on x64. One of the strange things that I found at the time was this: When running a CLR app under the Win32 debug interface you only ever seem to hit the native entry point if you're running under WOW64. In all other situations you don't hit the native entry point ever. If you rely on it to pause your debug tools once the process is completely loaded and ready to roll then you need to stick a break point in _CorExeMain in…

What would I do??

| 0 Comments
There's an entry over on the Dr. Dobbs blog about testing and how you make sure that your tests are testing the right thing; effectively, who tests the test. There's a question at the end "What do you do?" and I think my rather pithy, I've had some wine, answer is, "I think harder". The poster laments the fact that if you're doing TDD then the test fails first and then you write the code and then it works and therefore you know the test is testing the correct thing but if you have existing code then, well, it doesn't…

Practical Testing: 17 - A whole new approach

| 0 Comments
The comments to my last practical testing entry got me thinking. The commenter who had located the bug in part 15, which was fixed in part 16, suggested a new approach to the problem and I've been investigating it. The suggestion is, essentially, to use a timer with a longer range before roll-over rather than GetTickCount() with its 49.7 day roll-over. In Vista and later we could just use GetTickCount64() but on earlier platforms that's not available to us. My commenter's solution was to build a GetTickCount64() on top of GetTickCount() and use that. Given that adjusting the code for…

Practical Testing: 16 - Fixing a timeout bug

| 5 Comments
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Then, in 2005, I adjusted the code to be more scalable and I showed how the tests that had originally been written helped when code needed to be changed for performance purposes. Finally I uploaded a test utility program that I'd been working on, TickShifter, that…
Back in April 2006 I posted a copy of TickShifter, see here for details. It seems that there was a bug in my Win32 debugger code on which TickShifter is built. The bug was that we failed to "forget about" dlls that were unloaded... Because we failed to forget about them it was possible for the debugger code to try and do something with addresses in these dlls that were no longer loaded and this would cause a C++ exception on the debugger thread when our call to ReadProcessMemory() failed and this caused all sorts of problems... The end result…

Mock32, parameterize from below

| 7 Comments
It's funny how potential product ideas beget other potential product ideas and the thing that you eventually end up with as a product for sale is often far from the original product idea and the code that you started to write... I'm not even there yet and I'm on my third wave of product idea focus. Right now I'm working towards getting some developer tools out there to help people write better systems. This is the API hooking stuff that I've blathered on about in the past. The original product idea that started it all was a niche market testing…

Tools, debug thyselves

| 0 Comments
One of the first things that I tried to do with the latest release of my TickShifter tool was to run it on itself. Of course, that didn't work. There are several reasons why the tool might have had problems running on itself but I thought I'd addressed most of them when I dealt with the issues around getting multiple copies of the tool running at the same time. The problems there were mostly around the names that I used for the named kernel objects that were required; the control program communicates with the dll that it injects so that…

TickShifter v0.2

| 0 Comments
As I mentioned a while back, I've been working on adding rudimentary GUIs to my debugging and testing tools. In fact, both the deadlock detection tool and the time shifting tool are both functional enough for real world use but the command line interfaces make them harder to use than they need to be. I'm not yet 100% sure of the form that the final GUI should take, but I've pressed on with my initial GUI development to produce a reasonably complete GUI for the cut down, demonstration tool, TickShifter.…

TickShifter v0.1

| 0 Comments
Well, I figure that I've written about these debug tools that I've been working on for long enough. The time has come to make one available for other people to use and abuse. Given that I hope to sell some of these tools eventually I've decided not to give away early versions at present. Much as I will need feedback I think a properly structured beta program would be better for that. So, what I have done is write a cut down tool that demonstrates and uses the same technology that the other tools use. In essence if this runs…

Look back and shudder

| 0 Comments
I'm currently investigating a memory leak in a complicated piece of multi-threaded code. Unfortunately the code doesn't have any unit tests and the leak only shows up reliably in the release build. Worse, I wrote the code and nobody has fiddled with it since, so it's my leak and my code and I have nobody to complain about and no excuses. I miss the unit tests and loose coupling that would be present if I hadn't written this particular piece of code quite so long ago.…

When your mocks are executable

| 0 Comments
The size of the "units" that I test with my unit tests varies quite considerably. Usually I tend to work at the single class level; the test will provide mocks for the services that the class requires and the tests will ensure that the class behaves correctly and interacts as expected with the its service providers. Sometimes though the units consist of a whole subsystem, still often a single class, but one which wires up and collects many other classes into a single component; again the service providers that it uses may be mocked but they're often only right on…

Implicit Interfaces

| 0 Comments
ImplicitInterfaceImplementation - from Martin Fowler is an interesting piece where Martin suggests that it would be useful to be able to provide alternative implementations of a class's implicit interface. That is, given a class that does not inherit from an interface, it's sometimes useful to be able to split the class into an implicit interface, the public operations that it provides, and treat that as if it were explicitly declared as an interface from which the class derives. This would be useful in testing as it would allow you to mock up things that are currently difficult to mock up.…

Panto season!

| 2 Comments
In case you missed it the first time around... We're doing the old "assert is evil"/"Oh no it isn't" thing over at Ned Batchelder's place.…

I think the weirdness was me

| 0 Comments
A couple of days ago I mentioned that I was having some problems with loading symbols for a common controls dll. I'm now pretty sure that it was my problem, as usual, rather than someone elses. I've reworked my process startup controlling code and now set and clear the breakpoints in a more reliable (i.e. less hacky) way and things are behaving themselves again. I think I was failing to stop one of the threads properly and that was later causing problems.…

DbgHelp weirdness

| 0 Comments
I was using one of my home made debugging tools recently and it kept crashing :( I assumed it was something I was doing but I've eventually tracked it down to where I load the symbols for the loaded modules in the target process. Attempting to load the symbols for x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2180_x-ww_a84f1ff9\\comctl32.dll leads to dbghelp.dll generating a BEX event (which appears to be either buffer overrun or data execution prevention related).…

3 days with JMock

| 2 Comments
As I mentioned last week, I'm currently doing some Java work with an investment banking client. This week I added JMock to their toolset. JMock is a dynamic mock object generation tool that works with JUnit to allow you to create mock objects without writing code. Since most of my TDD work has been in C++, most of the mock objects that I've used in the past have been hand coded. There are similar tools for C++ but, due to the lack of reflection in C++, they can be a bit painful to use. I actually like writing my mocks…

Testing Windows Services

| 6 Comments
Mark Pearce writes about Debugging a .Net Windows Service from within the IDE. We do something similar with our C++ Windows services but, as you've probably come to expect from me, it's slightly more complicated than Mark's approach.…

Kevin Barnes on TDD

| 0 Comments
Kevin Barnes over at Code Craft has just written an interesting piece on TDD. In it he claims that "Excessive unit testing infrastructure hampers your practical ability to refactor things efficiently. People scream at me when I say this. Unit tests are supposed to enable refactoring by spotting unexpected bugs resulting from the refactoring effort. I agree. Unit tests help with this, but only up to a point. When they cross the line into extreme-testing they can be pretty much guaranteed to break every single time you refactor the code" He has a point, as with all things, you need…

Controlling Time, Take 2

| 5 Comments
Recently I finished developing a high performance ISO-8583 financial transaction authorisation server for a client using The Server Framework and whilst I was running the final black-box tests against the server I realised that these particular tests were dependant on the system date of the machine running the server. The server uses the current time and date to make some decisions on whether it can authorise a particular transaction or not. The date is used to look up some information from a database table. The table contains details of whether a particular competition type is available for entry or not; it has…

Profilers and The Perils of Micro-Optimization

| 0 Comments
Ian Griffiths has just written a nice piece on profiling .Net code and why the obvious things that people do are often very wrong: "Unfortunately, lots of developers just love to go off on micro-benchmarking exercises, where they write about 10 lines of code in a variety of different ways, and discover that one runs faster than the rest. (Or worse, one looks like it runs faster than the rest when running in the profiler.)" He advocates measuring performance outside of the profiler as well as within it so that you can be sure you're improving actual real world performance…

What's your worst bug?

| 4 Comments
"Because of a subtle bug called a "race condition," a quick-fingered typist could accidentally configure the Therac-25 so the electron beam would fire in high-power mode but with the metal X-ray target out of position. At least five patients die; others are seriously injured" History's Worst Software Bugs, from Wired. I'm pretty careful about how I write code and I like to have my tests in place and, on the whole, I write pretty reliable code (most of the time) but I'm still glad that I don't work on stuff that's likely to cost lives if it goes wrong... Probably…

Windows TCP/IP Server Performance

| 22 Comments
For the last week or so I've been working on measuring and improving the performance of The Server Framework. The latest release of the free version of my asynchronous, windows, IOCP based, socket server framework can now be obtained from here at ServerFramework.com. This week I've had several potential new clients asking about performance numbers for various aspects of the system and so I thought it best to do some measuring. To be able to measure I first had to build a tool to help me.…

Practical Testing: 15 - Testing payback

| 2 Comments
Last year I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. The code under test was intended to be very scalable, I use the code to provide light weight asynchronous timers for my socket server framework and the idea is that each connection might have several timers associated with it. In the original introduction I wrote this: "The…

Unit testing and accessing external systems

| 0 Comments
There's a lot of talk about how unit tests shouldn't touch the network or the file system or databases whilst they're running. Michael Feathers even has a set of unit test "rules" (A Set of Unit Testing Rules) which go so far as to suggest that: "A test is not a unit test if: It talks to the database It communicates across the network It touches the file system It can't run at the same time as any of your other unit tests You have to do special things to your environment (such as editing config files) to run it."…

Printf debugging when you don't have a console

| 8 Comments
There's a nice story over on "Bug Babble" about debugging a problem with a robot by using various sounds coming out of a speaker to determine where in the code the problem occurred: "Now the robot sounded like a modem trying to connect. We would repro the hang and based on the pitch at 'flatline' we knew the point of the last successful call to the sound driver." Via Google Translate and Radium Software, which also suggests using the CapsLock key, or changing display colours to debug gnarly problems in a printf style when you don't have a console to…

Dependency Injection

| 0 Comments
Jeremy D. Miller writes about The Dependency Injection Pattern; or what I've tended to call "Parameterise from above". He covers the various ways you can inject a dependency into a class rather than having the knowledge of the dependency hard wired into the class itself. I tend to favour the constructor injection pattern myself; it's all about breaking concrete coupling with interfaces and then allowing the creator of the object to specify the concrete type used by the object. As Jeremy says, it's not just for testing; it leads to much more flexible software. It's worth following the link from…

Thoughts on testing COM objects

| 5 Comments
Ben over at Code Circle is thinking about unit testing COM objects. I did quite a lot of this back in 2000 when I was working with an investment bank. The first thing you need to realise is that COM is just an interface layer; so lots of your unit testing should be done on the code that the COM interface layer calls into. You can test pretty much all of the actual functionality of a COM object without bringing COM into the mix at all. This keeps the tests nice and fast. So, rule one for testing COM objects…

How refreshing

| 0 Comments
One of the things that I've always been a bit unsure of is the claim by dynamic languages crowd that static typing buys us nothing as the unit tests solve the same problem. It's a nice idea but I'm a bit scared that the unit tests required would have to be quite a bit better than my unit tests usually are... Anyway, it's nice to see Brian Marick posting about how his usual retort of 'the unit tests will catch the bugs that the static type checker would' was wrong...…

Setup and Teardown, reprise

| 0 Comments
Roy adds a little fine tuning to Brian's advice about avoiding setup and teardown in unit tests. In summary; aim to minimise duplication in your test code...…

Exploring the C++ Unit Testing Framework Jungle

| 5 Comments
Exploring the C++ Unit Testing Framework Jungle over on Games from Within is a really good look at all of the C++ unit testing frameworks out there. It compares the following frameworks: CppUnit Boost.Test CppUnitLite NanoCppUnit Unit++ CxxTest I'm one of those people who has written their own mini framework because none of the existing frameworks seemed to do what I wanted at the time I needed the framework and I wanted to get up to speed quickly with minimal learning time. Had this comparative review been available when I was first starting out with C++ unit testing then I…

Setup and TearDown considered harmful?

| 0 Comments
I'm glad that Brian Button went to the trouble of writing this post and thinking through the implications of shared setup and teardown code in tests. I've been a bit concerned about some of my tests for a while now. You see, I don't use a fancy framework, I just write them myself in C++ and generally there isn't much shared setup code because I don't find it that useful. Brian's insight that the duplication present in tests is often a good thing because it allows the test to communicate more clearly is pretty much what I've thought for a…

Unit testing for speed?

| 1 Comment
Simon is thinking about using unit testing to help with performance testing. Whilst I've found it useful to use unit tests as very focussed ways to run a profiler on a selection of code I don't think it's a good idea to tie the success of your test to it completing within a particular length of time. After all, you might decide to run all of your tests under a tool such as Boundschecker and it would be a pity to get false failures due to the fact that the test is running slower just because you're using some other…

My own legacy code

| 0 Comments
I've just started work on an ISO8583 server for a client. I've done similar work for them in the past and the first thing that I did was to take the basic shell of the last server that I did for them and update it to use all of latest version of The Server Framework. The next was to start to look at the first of the requirements for the new server. It was then that I realised that I was working with legacy code...…
Last week I posted an entry about the simple, hand built, mocks that I use for testing in C++. Whilst I stand by my previous assertion that writing the mocks yourself gives you an further insight into your design from a client perspective I think the discussion that I had with Roy Osherove after last week's posting has made me appreciate the power of having mocks that support the programmatic specification of 'expected' behaviour.…
My Unicode Byte Order Mark hassles yesterday were brought to light by one of my mock objects. It was the expected output log for the object that had been converted from UTF-16 to UTF-8 by CVS without asking... Whilst writing that piece I realised that I probably needed to write this piece. You see, I spend most of my time in C++ and I write most of my Mock objects myself rather than having some framework do it for me and something that I found very useful and simple is the idea that my mock objects create simple logs of…
A long time ago, on Practical Testing: I mentioned that there was a small chance that the way that ResetTimer() operated could mean that you could pass in a handle that you no longer owned and affect someone else's timer by mistake; I said it was unlikely to cause problems... In this posting we fix this problem and in the process show how TDD builds up a set of tests that can support you during redesign...…
Roy Osherove links to Mockpp a mock object framework for C++ and comments on how the framework is painful to use (and it looks like it was painful to write...) He concludes that he's getting on just fine writing his mocks manually... In my experience you'd miss a lot of very useful learning and design work if you didn't write the mocks manually...…

Hoisting test code into production

| 0 Comments
A while ago I mentioned how I had hoisted a mock object up into production code because it only needed a few changes to make it usable as real code rather than just test code. This last week we've done this again...…

The value of malleable systems

| 0 Comments
We're almost at the end of our hand over phase now; my time on the project is almost over. The last couple of weeks have been interesting. We've been working towards a demo of the system and this seems to have helped to get everyone moving in the same direction; suddenly the hand over seems to be happening... I don't think I'll need to be that involved in the demo as everyone else knows what's going on and can explain how the system works... This is a Good Thing. One of the other good things about the last couple of…

So much for testing

| 3 Comments
My category-based RSS feeds stopped feeding through to Testing Reflections and Test Driven a couple of weeks ago. They had worked fine, and then they just stopped. I assumed that the problem was at their end; either they didn't think much of my testing posts or their feed polling software had some weirdness in it... But you know what assume does...…

Looking back from near the end

| 2 Comments
My current consulting gig is coming to an end. We've been in the hand over phase for a while now and I think it's slowly starting to work. Now seems like a good time to look back at what worked and what didn't...…

Practical Testing: 13 - Missing functionality

| 0 Comments
Previously, on Practical Testing: we added a multi-threaded version of our timer queue that manages timeouts automatically. This time we'll integrate our new timer queue into an application that uses the old version of the code and, along the way, discover some functionality that the original version supports but that the new version currently doesn't.…

Measure twice, cut once

| 0 Comments
I'm currently working on a small auction server for a client using The Server Framework. It's a relatively simple server, messages come in, are validated and are then broadcast to interested parties and logged. Yesterday we shipped the first cut of the source to them and I was a bit concerned that our simple test app could thrash the server so easily. Admittedly the first cut of the code hadn't been tuned at all and the logging implementation was fairly Mickey Mouse but I was a little concerned. I woke this morning with a list of perf improvements to try. Some of…

Practical Testing: 12 - Threading is orthogonal

| 0 Comments
Previously, on Practical Testing: we finished our reworking on the code we were left with a simpler timer queue that passed all of our tests, but was single threaded and needed manual intervention from client code to manage timeouts. Today we'll add back the multi-threaded functionality so that the queue manages timeouts automatically.…

11th October is Thanksgiving Day in Canada

| 0 Comments
I've had an interesting week. I'm back with the Refactoring Project and, although things were looking up last time I was here, they've managed to adopt some bad habits in my absence. The latest build that's live with users isn't tagged in CVS; we have 66 other builds that are tagged and the ability to rebuild an arbitrary release has helped on numerous occasions, but this time they decided not to bother. But worse than that, they haven't been running the tests. Monday is Thanksgiving day in Canada, we found that out because they didn't run the tests...…

Tests as tours

| 2 Comments
I've discovered something quite amazing this week; something quite simple that shouldn't really have been much of a surprise, but it was. If you have a substantial set of unit tests then you can use them as the backdrop for documentation that tours the code base in a controlled and manageable way. Kinda like a sing-along, the reader can be directed through several key tests in a debugger whilst the document describes what's going on and reasoning behind some of design decisions.…

Yup, that's where the value is

| 0 Comments
Mats writes about how he finally got unit testing by realising that the value comes not when all your tests pass and you feel happy but when one of your tests fails and you feel sad...…

Code coverage and testing

| 0 Comments
Recently I mentioned that we were in the process of adding additional tests to our code base. We'd been using JITT to reduce the number of tests that we wrote and now it was time to fill in some of the gaps. This week I started to use some code coverage tools to see how well we were doing with our tests...…

Repaying the technical debt

| 1 Comment
The boss is away, we've almost done all the things he asked us to do before he went and, well, there's a bit of spare time so we're looking closely at the code. The last few weeks have been a bit of a push for the finish and now I'm looking at stuff and trying to work out how to repay the technical debt we accrued during the push. Sure, I could be surfing the web and kicking back but, well, it just seems appropriate to work out how much we now owe to the code and how we can…
Previously, on Practical Testing: having decided to rework the code from scratch in TDD style we fixed the tick count bug for the second time - this time the solution was cleaner and simpler. At the end of that episode we were left with a failing test. The test was for multiple timers and it failed because our TDD route was taking a 'simplest thing that could possibly work' approach and that design only supported a single timer. Today we'll fix that problem by moving nearer to a real world solution.…

Just In Time Testing

| 0 Comments
Once we'd integrated the new data provider we were in a position to do some more testing. We configured the code that used the new component to request the same data from the new component and the old code and to save the data to files. Then we wrote some code to compare the files and highlight any changes. Once that was done we located the source of the differences, wrote tests that failed due to the problems and then started to fix the bugs. A while ago I called this process Just In Time Testing (JITT); it's like TDD-lite.…

Singletons and testing

| 15 Comments | 4 TrackBacks
When you need to jump through hoops to write tests you've done it wrong. Jonathan de Halleux writes about testing singletons and how he can subvert the standard C# singleton pattern to create a new singleton for each test he needs to run for the singleton. Omer then subverts the rules another way. Both are wrong.…
Previously, on Practical Testing: Having bolted on some tests to an existing piece of code we're now doing some "agressive refactoring" ;) and rewriting the code from scratch using the testing ideas we developed earlier. The whole point of this exercise was to fix a known bug, we did that in the existing code here, now we have a test that forces us to address the issue in the new code.…
Previously, on Practical Testing: we fought through the pain of writing tests for hard to test code and then we decided to see what it could have been like if we'd developed the code test first... Now we'll add some more tests and some more code, still keeping it all nice and simple...…
I've been writing some blog entries about a piece of code from my 'back catalogue' that didn't have tests and that had a known bug that was reasonably hard to test for. Right at the start I commented that the code was a tad over complicated due to the way it had been developed using HITIW (Hack it till it works). The complexity in the code itself made writing tests for it harder than it should have been and, well, I wouldn't write code like that these days (honest). So, here we are in part 8 and I'm about to…
Previously on Practical Testing: After far too much work we finally got to the point where we had a test for the tick count wrap bug. Now that we have a failing test we can fix the bug.…

Testing Reflections

| 0 Comments
I found this site today from my server log - testingReflections.com I haven't explored it fully yet, but what little I have seen looks good...…

Testing payback

| 0 Comments
I'm building a new data provider for one of my clients. It breaks a huge chunk of their existing codebase out into a new component that hides behind a simple COM interface and abstracts away lots of nasty stuff so that they don't need to worry about it. The inflection point is small; a method or two on a single COM interface. This is good as it means we have a single, easy to test, integration point. Beyond the interface I can do what I like; all I need to worry about is that the stuff I throw back through…

Testing shouldn't be that hard

| 0 Comments
Yesterday's Practical Testing post was a bit of a mammoth testing exercise. The threaded nature of the component under test makes testing it harder than you'd think; and this is just a single worker thread that we have to worry about. The thing is, I expect that the component would have been designed slightly differently if it had been developed test first; using TDD, or even with just in time testing, (JITT?)...…

Practical Testing: 6 - Tests refactored

| 0 Comments
Previously on Practical Testing: The last entry ended with us having two tests, both of which were in need to a good refactoring. The second test had uncovered an unexpected bug... This time around we'll refactor the tests, fix the bug and finally write the test for the tick count wrap bug...…
I'm writing some blog entries that show how to write tests for non trivial pieces of code. This is part 5; the one where we find a bug we weren't expecting...…

Practical Testing: 4 - Taking control of time

| 5 Comments
I'm writing some blog entries that show how to write tests for non trivial pieces of code. This is part 4.…
I'm writing some blog entries that show how to write tests for non trivial pieces of code. This is part 3.…

Practical Testing: 2 - The first test

| 0 Comments
I'm writing some blog entries that show how to write tests for non trivial pieces of code. This is part 2.…

Practical Testing: 1 - Introduction

| 2 Comments | 1 TrackBack
I'm writing some blog entries that show how to write tests for non trivial pieces of code. This is part 1.…

Practical Testing

| 0 Comments | 1 TrackBack
One of the common complaints about TDD and unit testing that I see is that the examples used aren't real. People often say that the examples used are toys and that writing such tests adds little or no value. To be honest, I often find myself agreeing with them. One of the problems of adding unit tests to an existing code-base or driving a new project with TDD is deciding exactly where to spend your testing efforts. This is more of an issue when adding tests to existing code as I personally find that the safety of TDD on new…

Wafer thin testing?

| 0 Comments
I've spent the past month or so helping a corporate client improve code quality in a sprawling application. It's non-trivial, the code-base is huge, the quality is, at best, questionable and the coupling is excessive and made worse by the fact that much of the system is coupled together using a single huge blob of relatively unstructured XML. Fun, fun, fun...…

Testing synchronous communications

| 0 Comments
Today I'm doing some work for a client that involves writing some blocking sockets code to talk to one of our servers. Blocking mode fits well with the architecture of the client application they use; we're moving the app from blocking reads on a serial port to blocking reads on a socket and jiggling the protocol it uses a bit. Testing blocking calls is actually a bit harder than testing non blocking calls because hand cranking the mock objects so that your object under test works correctly is harder to do when your test harness is blocked in a call…

Async Pop

| 3 Comments
A while back I finally started on the async version of the POP3 client. It ended up as a state machine and seemed to work well. The next step was to write an async version of the mail collector that I used to pull all mail from a mailbox on a server and optionally delete it.…

Jimmy on TDD

| 0 Comments
I agree 100% with this piece from Jimmy Nilsson's blog.…

Meanwhile, back with the tests

| 0 Comments
I've seen a lot of blog postings recently that pour scorn on the ideas behind TDD. Ah well, if you don't like it, don't do it. I'm more than happy if our competitors decide that TDD isn't for them. In fact, testing is bad, don't do it, move along now, don't read any of the testing articles here...…

Craig on TDD

| 0 Comments
Craig Andera talks about TDD. I couldn't have said it better myself.…

POPing back

| 0 Comments
Way back in mid November, before we dumped our building's managing agent for being worse than useless and possibly stealing from us, I was working on some POP3 code. I had some down time today so I decided to drop back into it and see if I could move things along a little. In summary, having lots of tests helped...…

Uncontrolled coupling - Singletons, just say NO!

| 0 Comments
We're developing some code for a client. There's a standalone server, which we've completed, and a small stub of code that allows the client's existing system to talk to our new server in place of the old thing they used to talk to... This morning I stubbed out the new stub and put together a test harness project. Unfortunately, due to the way the client's code is coupled it could prove difficult to test the new stub...…

I seem to have lied earlier...

| 0 Comments
Earlier I said "I'll probably keep the code for now, but it's not the code I'd have written first if I was working from a test. I wouldn't need it yet... I may never need it in production..." I lied. It's in CVS if I need it, so why keep it cluttering up the code when I don't need it...…

Test Driven OBEX

| 2 Comments
Way back in June I was playing around with OBEX. I've had a quiet day today and went back to the code to progress it a little more (a client is making interested noises so I need to get back up to speed again). The code I wrote in June was before I'd become test infected...…
Barry suggests that to do meaningful performance tests you need to know a bit out the way the thing that's under test operates.…

Where do the mocks live?

| 0 Comments
During the recent library adjustments the main aim was to add tests. As we write tests we create lots of mock objects. Our libraries are dependant on each other, as libraries tend to be, and this means that often a library uses an interface from another library... When it comes to test the dependant library it's often handy to be able to use the mock object that you created to test the library that you're dependant on... If you see what I mean... The, slightly labored, point being, it's important where you keep your mock objects...…

So, what do these tests look like then?

| 0 Comments
After breaking the socket server into more manageable chunks I started writing the tests. CAsyncSocketConnectionManager is pretty easy to test, it only has one public function; Connect(). So, what exactly do these tests look like?…

Hacking our way to the first test

| 2 Comments
So I have some time on my hands and I've decided that The Server Framework code could do with some tests but before I can do that I need to refactor because the code was written before I became test infected and, well, it's not as good as it could be...…

Edit my points

| 0 Comments
The refactoring project rumbles on but my time with it is drawing to a close. This week the currency traders decided that they wanted to be able to manually override the live data in some circumstances. They wanted to be able to edit a live data point, set its value to a fixed amount and have all dependant displays take this new value into account. Like most things in computing an extra level of indirection saved the day...…

Promote a mock

| 2 Comments
I knew it would happen eventually... As mentioned earlier all of the email filtering code has been developed without any sign of a main(). Now that the time has come to create the actual filter program I found that I didn't actually have a real version of one of the objects that I required, I only had a mock version for testing. The thing is, the mock version is pretty much all I need for the real version, so it looks like it's time to promote it...…

Rolling...

| 0 Comments
The first two filters were pretty easy. I was on a roll and the other filters were implemented just as quickly...…

Just enough RFC822

| 0 Comments
Second on my list of email filters was a filter that splits a 'domain mailbox' into several different mailboxes depending on the username that the email is addressed to. This is basically just an intelligent version of the mailbox writing filter. The problem was, it needed to understand RFC822 addressing...…

The one where I dont use XML

| 0 Comments
So I have an email filter that can write messages to another mailbox. I need to supply it, and all other filters that I might write, with some configuration data. I could use XML but I dont...…

Filtering mail

| 0 Comments
Now that I can retrieve POP3 mail and serve it up again via a POP3 server I want to do stuff to it in between retrieving it and serving it. The idea was to have a series of filters that get passed each message, Do Stuff ™ and either allow the message to be passed on to the next filter or end the filtering process. Most of that works now, here's how I got there.…

Dealing with the simplest things

| 1 Comment
The POP3 client is now complete. It can download messages from POP3 servers and store them in a message store. I've implemented a file system based message store that is compatible with the POP3 server code's file system message store. We can download messages from server's and make them available via our server. As expected the test first approach had driven the design in a simple and decoupled direction; eventually some of the simple decisions were inappropriate so I added some tests and refactored...…

POP3 Client almost complete

| 0 Comments
The test driven development of the POP3 client code is almost complete. The development proceeded in a similar manner to the server code and I'm left with the same thing to write; the message store...…

Dirty Little Secret: Test code is fun to write

| 0 Comments
I've been busy :( but it's paid busy so I suppose I can't complain... This evening I got some time to myself to finally sit down and see how hard it would be to use all of my previous test code plus the real production POP3 server and command parser to act as a test framework for my POP3 client. It took just over an hour to plug it all together and then it just kinda worked...…

James Antill doesn't like TDD

| 0 Comments
And this is why I hate TDD, testing is a great thing. But testing too early is bad, and you are obviuosly doing that. First you need to know what your code has to do in full. For instance even if you wanted to have both sync. and async. APIs (I personally abhore sync. APIs due to non-scalability) the obvious implementation is to have something like... James Antill - in a comment on my Tangled Testing entry. James, I disagree...…

Tangled testing?

| 2 Comments
Last night I started on my POP3 client code. I didn't know where to start; I wasn't really in the mood, so I wrote a test... That got the ball rolling. As usual the test first thing left me with a nicely decoupled class to write. Now I'm in a bit of a quandary about how to mock up the server that I need to test the client against...…

Tests and TDD in anger

| 2 Comments
This week we started to make some changes to the FX code. The existing code made some strange assumptions about some of the edge cases and the resulting display was occasionally inconsistent. We fixed that, and the tests saved us from a couple of embarrassing mistakes. Meanwhile, in my free time, the test driven POP3 server development continues. I now have a working server that can run off of a message store that lives on the file system and can do all the things that a POP3 server is supposed to do. So, did the test first approach help?…

Why objects should keep it on the inside

| 0 Comments
So, I'm integrating this POP3 code with my server and the first thing I do is create null message store. I haven't implemented the message store yet, so in order for me to integrate I need a message store that just says yes to everyone being a user and provides mailboxes that are always empty... The test driven development has made this reasonably easy, I have a message store interface so I just need to stub out the methods appropriately. It's then that I find that the interface is bad...…

Does TDD lead to earlier integration?

| 3 Comments
I'm developing a POP3 server. I've been developing the protocol handler test first and it's nearly done. I haven't started on the mailstore itself yet but I could easilly integrate the POP3 code into the server code and have a functional POP3 server...…

I blame the testing

| 3 Comments
It's amazing what a day or two away can do. I came back to the tests with a fresh mind this morning and dealt with the issues that had caused me problems at the tail end of last week before I got my morning coffee.…

Insufficient coverage

| 0 Comments
Today the FX engine went into UAT. Well, the nearest thing we have to UAT; a user looked at it... 3 bugs, differences between the new code and the current production version. All slipped through our test coverage. :(…

Decoupling the FX GUI

| 0 Comments
The rates engine was now easy to test but the interaction between the engine and the user wasn't. This was unfortunate as the interaction is reasonably complex. We hadn't built and tests for any of the GUI code yet, last week we fixed that...…

FX Testing

| 0 Comments
By Friday our FX test harness was pretty much complete. We had coverage for all the nasty special cases that had caused us problems in the last few weeks. They were the hard things to write tests for so we wrote those tests first. If we can these these things we can test pretty much all of the FX engine's functionality. Now we just need to add back the functionality we removed when we were chainsawing the FX code out of the GUI code. Once that's done we'll merge this branch back into the main tree and test and release…
I was working for my poker game client yesterday. This project now seems to be firmly test first. What was interesting with yesterday's work was how the tests drove the design and how when I finally came to integrate the tested code into the main body of code the required design changes weren't an issue as I had a whole load of tests to support the changes.…

You'll tick when I say so and not before!

| 0 Comments
Today we wrote some complicated FX business logic tests. Things like making sure that the FX library can calculate a EURUSDCAD 1M rate - it can; or a USDCAD ON rate - it can't and nobody had spotted the fact that it was out because it's way down in the 0.00001's of the rate.…

The first FX test

| 0 Comments
On Friday we got to the point where the FX buiness logic code was suitably decoupled from the display logic that we could write our first test for the business logic. In the words of Homer Simpson, "Woo hoo!".…

The onset of infection

| 0 Comments
I've spent the morning doing test driven development, properly; writing tests first and everything. It works, it's faster and it's addictive.…

Untestable 2

| 0 Comments
A couple of days ago I posted some untestable code. I've had a couple of emails saying that people couldn't see why the code was untestable. Here's why, and here's how to fix it.…

Three bugs went into a program

| 0 Comments
Three bugs went into a program; a memory leak, a misunderstood interface and a deadlock...…

Untestable

| 0 Comments
It's easy to write untestable code. It's natural. Most code that we write will be hard to test unless we explicitly think about testing when we write it...…
So, we have currency based limits and they're in millions. Us: "Are they always going to be in millions for all ccys?" Them: "Yup." Coded, tested, delivered. Them: "can we have the JPY limits in billions?" Us: "US or European?" ;)…

Stage complete. Time bonus...

| 0 Comments
This morning I wasted some time tracking down bugs in the multi-threaded online game engine that I'm writing for a client. Now I have tests. Tests are good.…

We came. We saw. We did a little testing.

| 0 Comments
Another week another release. Well, almost. The plan was to release today. The plan ignored the fact that most of the team are at a wedding this weekend and nobody was around today and nobody's around on Monday...…
Kent Beck demonstrates the testing side of XP by separating it out into its own simple methodology. Test-Driven Development is exactly what it says it is. The entire design and development effort is driven by the tests that you write and you're encouraged to write those tests first...…

Our first test

| 0 Comments
The refactoring project reached an exciting new stage on Friday. We were finally able to refactor some code to the point where several classes could be extracted from the application project and built in a library that the application then linked to. This required that we reduce the coupling of the classes and increase the cohesion. Once these classes were extracted we were able to set up a test harness project and write our first test case!…
« Source Code | Testing Archives | Way back »

About this Archive

This page is an archive of all entries in the Testing category.

Source Code is the previous category.

Way back is the next category.

I usually write about C++ development on Windows platforms, but I often ramble on about other less technical stuff...

Find recent content on the main index or look in the archives to find all content.

I have other blogs...

Subscribe to feed The Server Framework - high performance server development
Subscribe to feed Lock Explorer - deadlock detection and multi-threaded performance tools
Subscribe to feed l'Hexapod - embedded electronics and robotics
Subscribe to feed MegèveSki - skiing