Net Objectives

Net Objectives
If you are interested in coaching or training in ATDD or TDD please click here.

Monday, January 30, 2012

Testing Best Practices: Test Categories, Part 2

download the podcast

Continued from Test Categories, Part 1

5. Behavior with related boundaries (“and”) 

Sometimes a behavior varies based on more than one factor.  For example, let’s say that in order for the system to allow someone to be hired (returning a true from the canHire() method, perhaps), the individual in question must both be at least 18 years old and must be a US citizen.

Similar to a boundary within a range, here we need more than one assert to establish the rule.  However, here 2 is not enough.

Let’s further stipulate that there is a constant, System.MINIMUM_HIRING_AGE, and a enumeration named System.Citizenship with members for various countries.  The specifying test code would look like this:

class HiringRuleTest {
   public void testOnlyAppropriateCitizensOfSufficientAgeHired() {
       System.Citizenship anyOtherThanUS =

       HiringRule testHiringRule = new HiringRule();

       Assert.False(testHiringRule(System.MINIMUM_HIRING_AGE - 1,

A typical question that  might occur to you on reading this might be:  

Should we not show that a person under the minimum age and also not a citizen of the US will get a false when we call this method?  We have shown that:

  1. A US citizen of sufficient age will return a true (hire)
  2. A younger US citizen will not be hired (false)
  3. A non-US citizen “of age” will not be hired (false)
But should we not show that 2 and 3 combined will also produce a false?

This is a perfect example of the difference between testing and specification.  If we were to ask an expert in testing this question, they would likely talk about the four possible conditions here as “quadrants” and would, in fact, say that the the test was incomplete without the fourth case.

In TDD, we feel differently.  We don’t want to make the test suite any larger than necessary, ever, because we want the tests to run as fast as possible, and also because we don’t want to create excessive maintenance tasks when things change.  The ideas of “thoroughness” and “rigor” that naturally accompany the testing thought process become “sufficiency” in TDD.  This probably seems like a trivial point, but it becomes less so over the long haul of the development process.

Put another way, we never add a new assert or test unless doing so makes a new distinction about the system that was not already made in the existing assertions and tests.  What makes non US citizens distinct from US citizens is that they will get a false when the age is sufficient.  That a non-US citizen with insufficient age will get a false is no different than a US citizen, it is the same distinction.

Again, TDD does not replace traditional testing.  We still expect that to happen, and still respect the value of testing organizations as much as we ever have.

You may also wonder “why Outer Mongolia to represent ‘some other country’?  Why not something else?”  We don’t really care what country we choose here, hence we named the temporary method variable “anyOtherThanUS”.  Elsewhere in this book, we’ll look at other options for representing “I don’t care” values like this one.

6. Behavior with repeated boundaries (“or”)

Sometimes are have boundaries that are related in a different way.  Sometimes it is a matter of selecting some values across a set, and specifying that some of them must be in a given condition.  If we don’t address this properly, we can seriously explode the size of our tests.

Let us say, for example, that we have a system of rules for assigning employees to teams.  Each team consists of a Salesperson, a Customer Support Representative (CSR), and an Installation Tech. The rule is that no more than one of these people can be a probationary employee (“probie”).  It’s okay if none of them are probies, but if one of them is a probie the other two must not be.  If we think if this in terms of the various possible cases, we might envision something like this:

This would indicate eight individual asserts.  However, if we take a lesson from the previous section on “and” boundaries we note that zero probies is the same behavior (acceptance) as any one probie; it is not a new distinction and we should leave that one off.  Similarly, we don’t want to show that all three being probies is unacceptable, since any two are already not acceptable and so, again, this would not be a new distinction.

That makes for six asserts.  Doesn’t that feel wrong somehow?  Like it s a brute force solution?  Yes, and this instinct is something you should listen to.  Imagine if the teams had 99 different roles and the rule was that no more than 43 of them could be probies?  Would you like to work out the truth table on that?  Me neither, but we can do it with math:

That would be a lot of asserts!  Painful.

Pain is almost always diagnostic.  When something feels wrong, this can be a clue that, perhaps, we are not doing things in the best way.  Here the testing pain is suggesting that perhaps we are not thinking of this problem correctly, and that an alternate way of modeling the problem might be more advantageous, and actually more appropriate.

A clue to this lies in the way we might decide to implement the rule in the production code.  We could, for instance, do something like this:

class TeamBuilder {
    public const MAX_PROBIES = 1;

    public boolean isTeamProperlyConfigured(
        Employee aSalesPerson,
        Employee aCSR,
        Employee aTech) {

        int count = 0;

        if(aSalesperson.type == “probationary”) count++;
        if(aCSR.type == “probationary”) count++;
        if(aTech.type == “probationary”) count++;

        if(count > MAX_PROBIES) return false;

        return true;

This also seems like an inelegant, brute-force approach to the problem, and we might refactor it to be something like this:

class TeamBuilder {
    public const MAX_PROBIES = 1;

    public boolean isTeamProperlyConfigured(
        Employee aSalesPerson,
        Employee aCSR,
        Employee aTech) {

        int count = 0;
        Employee[] emps = new Employee[]
{aSalesPerson, aCDR, aTECH};

        forEach(Employee e in emps)
            if(e.type == “probationary”) count++;

        if(count > MAX_PROBIES) return false;

        return true;

In other words, if we stuff all the team member into a collection, we can just scan it for those who are probies and count them.  This suggests that perhaps we should be using a collection in the first place, in the API of the object:

class TeamBuilder {
    public const MAX_PROBATIONARY = 1;

    public boolean isTeamProperlyConfigured(Employee[] emps) {
        int count = 0;
        forEach(Employee e in emps)
            if(e.type == “probationary”) count++;

        if(count > MAX_PROBATIONARY) return false;

        return true;

Now we can write a test with two asserts: one that uses a collection with just enough probies in it and one with MAX_PROBATIONARY + 1 probies in it.  The boundary is easy to define because we’ve changed the abstraction to that of a collection.

Here is another example of a test being helpful.  The fact that a collection would make the testing easier is the test essentially giving you “design advice”.  Making thing easier for tests is going to tend to make things easier for client code in general since, in test-first, the test is essentially the first client that a given object ever has.

7. Technically-induced boundaries

Addition is an example of a single-behavior function:

val = op1 + op2

As such, there is little that needs to be specified about it. This, however is the mathematical view of the problem. In reality there’s another issue that presents itself in different ways -- the bit-limited internal representation of numbers in a computer or the way a specific library that we consider using is implemented.

Let’s assume, for starters, that op1, op2 and val are 32 bit integers. As such there are a maximal and minimal values that the can take: 231-1 and -231. The tests needs to specify the following: 
  1. What are the largest positive and negative numbers that can be passed as arguments? The boundary in this case is on the “value of the operand
  2. What happens if the sum of the arguments is more, or less than these technical limits?   (231-1) + 1 = ?
The boundary in this case is on the “sum of the operands

When it comes to floating point calculation the problem is further complicated by precision both is the exponent and the mantissa. For example:

                           100000000000.0 + 0.00000000001 = 100000000000.00000000001

In a computer, because of the way floating point processors operate, the calculation may turn out differently:

                            100000000000.0 + 0.00000000001 = 100000000000.0

Note that the boundary in this case is on the “difference between the operands

In many cases the hardware implementation will satisfy the needs of our customer. If it does not, this does not mean we cannot solve the problem. It means that we will not be able to rely on the the system’s software or hardware implementation but would need to roll our own. Surely this is a tidbit of information worthwhile knowing prior to embarking on implementation?

Continued in Test Categories, Part 3...


  1. Sometimes, we have a special case of Range, where both boundaries are configurable and this configuration has to be passed to the object under test because it's used internally. In such cases, I sometimes wonder how to properly document the range. My first try was to do something like:

    leftBoundary = 20;
    rightBoundary = 22;
    Assert.Greater(2, rightBoundary - leftBoundary);

    The last assert is to document that I want to be able to have at least one value that fits in the range.

    The second try was to do something like:
    arbitraryNumberLeavingSomeRoomInsideRange = 2;
    leftBoundary = Any.Integer();
    rightBoundary = leftBoundary + arbitraryNumberLeavingSomeRoomInsideRange;

    What's your take on that? Assuming I have a behaviour description: "if given value fits in the range, X should happen" - do you even consider putting boundary values in the test, or do you encapsulate the logic of checking whether the value is in range (which sounds a bit cumbersome to me)? If you use the range values, what way to specify them do you use?

    1. Just to be more precise - what I mean by "document the range" is "how to document the relationship between the bolundaries of the range to let the test remain as generic and as descriptive as possible". E.g. when I say left = 3 and right = 5, someone may think that this behavior may be applicable if left = 3 and right = 3 when it isn't true.

  2. I do not think I completely understand the issue here, can you give us a concrete example, astral?

    Nonetheless, since when has not understanding something prevented anyone from giving their opinion?

    It does not matter where the definition of the boundaries resides - it could be in a config file, contant, or user configuration outside our control. They may even be random, for all we know. The only thing that matters, that as far as the code respecting the boundaries is concerned they exist.

    (A) So, in your test you could say:
    minValue = Any.Int()
    maxValue = minValue + Any.Int(minRange, maxRange)

    Axis: Value

    where 1,2 and 3 are the different behaviors you're specifying

    where minRange the smallest allowed range and maxRange is the largest. This gives you the four boundary values you need to specify to define the system's behavior.

    (B) What happens is the range itself, as defined by the user is incorrect?

    We need more tests to specify that there is a minimum and a maximum range.
    If minVal == maxVal, what happens?
    If minVal > maxVal, what happens?

    this another range problem :-)

    Axis: RangeSize

    where -- is the behavior when outside of range and + is the behavior specified in (A) above

    E.g.: minVal = maxVal = 6 ==> rangeSize = 0
    minVal = 887, maxVal = 87 ==> rangeSize = -800

    Axis: rangeSize (range cannot be less than 1)
    Axis: rangeSize (range cannot be negative)

    Note - a negative range may well be appropriate, depending on your problem domain.
    Note - if the maxRangeSize = infinity then you do NOT have an upper boundary, but you WILL be limited by the system capabilities. If the users need a larger range than (say) maxInt, a different implementation mechanism will need to be used (e.g., going from byte to short, from the processor's ALU to your home grown math processor, etc.)

    1. Thanks, Amir, you've addressed exactly the issue I meant :-).
      And thanks also for turning my attention to the issue that right boundary = left boundary is actually a separate behavior that should be covered in the specification.