Black, White, and Gray Box Testing

In CS 126, we talk a lot about the benefits of black-box testing. Black-box is one of the approaches that we introduced at the beginning of the semester. Its appeal is being able to test from the perspective of an end user, which we define as a consumer of your library/class, or the user who interacts with the compiled version of your app. This is great because it lets you write tests before writing your first line of implementation code, and it forces you to think about what the user will be interacting with when using your project.

However, black-box testing is not the only approach. White-box (a.k.a. clear-box) is a form of testing where you focus on the internals of the program, ensuring that every little part along the way works. With this approach, you test even the smallest helper functions to make sure that you didn't make any simple programming mistakes. This helps a lot with debugging, as it lets you know early where something might have gone wrong.

It sounds like white-box and black-box testing cover testing aspects that are beneficial but mutually exclusive. White-box testing misses out on the user-oriented tests, while black-box testing is not the most helpful for debugging, since it only tells you when something goes wrong, not where. Luckily, compromise saves us from having to choose. Gray-box testing combines the best of both worlds. When writing tests, you can test for both user interactions and smaller code chunks that you want to ensure work for the user-facing tests.

It seems simple enough to just combine black and white tests to form a more complete "gray" test suite. However, if this combination is not done with care, there are some anti-patterns that can emerge in object oriented languages like Java. The biggest caveat is violating encapsulation conventions.

Encapsulation

Encapsulation is the idea in object-oriented programming that certain items should not be publicly accessible. These restrictions are often very beneficial. It helps keep you, the developer, aware of what methods your users will be able to use. It simplifies maintenance by encouraging you to use those public methods in your own codebase. It ensures a level of consistency by disallowing clients to access internal variables and methods by accident or on purpose. The list goes on, but it is inherently restrictive, so it does add some slight complications to the mix.

Java has four access modifiers that designate levels of encapsulation. You may already be familiar with public and private. There is also protected and package-private (no modifier). Here is what they do:

The reason why this is so relevant is that a private method can not be tested by normal means. You would have to use something called reflection to do this, and that is outside of the scope of this course. Moving up the access level list, the next best option is package-private. As long as your testing suite is in the same package (which it should be), you can test just as you normally would. But should that method have been access-modified just for the sake of testing? This is an open question, and not every situation will yield the same answer. Let's try to clarify these examples with some cases.

What should my access level be?

A note on this: before you even write the code, you should have access levels figured out to some extent. Think about what access you want to give, and if you find that you can't give that access, you may need to iterate on your design. If you are 100% confident in that design, then it is an issue with the requester of said information. For example purposes, we will look at some already written methods, and try and sort out what access modifier they should have.

Consider this method in the Tic Tac Toe assignment:

// TicTacToe.java
xxxxxx static Evaluation checkWinner(String boardState) {
	...
}

This is the checkWinner function that we wanted you to implement. It is the end-all-be-all method for somebody to use your code. Because this is user facing, it should be public.

Next, let's try something a bit more tricky. Say in my TicTacToe.java, I had this helper:

// TicTacToe.java
xxxxxx static int countCharOccurrences(String inputStr, char queryChr) {
    ...
}

This method takes in a string and a character to look for, and returns the number of times that char showed up in that string. Seems like something that is all-purpose and useful, no?

It's tempting to make this a public method. However, why would somebody ever include TicTacToe.java to use countCharOccurrences? There are bound to be better string-focused libraries out there that would have this method. You now have a choice. One correct solution would be to make this a package-private or private method, depending on if you wanted to test it (more on that later). Another correct solution would be to move this to a StringUtils class, where it is more at home. In this case, since a StringUtils class's user-facing methods are string utilities, then you could make it public.

Ok, even trickier:

// TicTacToe.java
private static int xWinCounter = 0;
private static int oWinCounter = 0;
// Gets initialized in evaluateBoard
private static String boardState;

xxxxxx static void checkRowWin() {
    // Do stuff with initialized boardState
    if (x row wins) {
        xWinCounter++;
    } else if (o row wins) {
        oWinCounter++;
    }
}

Considering what we have looked at above: this is a Tic Tac Toe library at the end of the day, so this could be a relevant method that a user of your package would want. So, public, right?

Not so fast! How would anybody from outside this class use this method? It takes no parameters and returns nothing. Sure, it computes a result, but that result is locked in the TicTacToe class, so it's of no use. This method is not an ideal design in the first place, but if you had to have something like this in your code, it would absolutely need to be private.

Finally, let's say you wanted to abstract away some check. Perhaps it's Adventure, and you want to see if the user has met the criteria to beat the game:

// Adventure.java
xxxxxxx boolean checkPlayerWonGame(Player player, Layout layout) {
    return player.currentRoom.equals(layout.endingRoom);
}

That seems kind of trivial to be in its own method, but this gets useful if we use it in multiple places so we can change it later to factor in if player has a certain item and is in the ending room. What should this method be?

The answer is not so clear. On one hand, if this was being called directly by your user facing frontend code, then it should be at least package-private. However, if this was just a small cog in the wheel of several more complicated methods, then it could be argued that it only needs to be private. We couldn't tell you which is correct without seeing it in context of the rest of the implementation. The best approach is to ask yourself whether or not you could foresee yourself or others using this method in any different context. Access modifiers are not one size fits all!

Protected

Where does protected fall in all of that? Protected access is best from when you anticipate somebody else extending your class with a subclass. In that case, you can upgrade to protected from package-private in the examples shown above.

Tie Back to Testing

Hopefully now you have a more solid understanding of what access modifiers you should be using for the methods in your application. Now, if after all of those questions you decide that a method is private, then you have also decided that it should not be tested. That might sound off to you, but it is ok to not test certain methods. You already had to justify the access modifier, so you have in turn justified its testability. Other methods that are more suited to wider access will inherently test this method, and if the larger test fails, it shouldn't be too hard to use debugging techniques to find where the test went wrong.

If you are really on the fence about something being private, and you just want the safest bet, go with package-private, and test it. The moderators in our course understand that this is an incredibly nuanced subject, and we are going to have clear penalties that represent going against the advice of this document. Some anti-patterns that we will penalize:

• Correctly testing a method, but marking the access as public when package-private would be more appropriate.
• Not testing any non-private, non-trivial methods. An example of a trivial method is a getter, since it just returns a value.
• Using a more convoluted solution like reflection when using appropriate access modifiers is cleaner.

Additional Readings

If you are interested in other methods of testing, please check out this article.

Course Policy Statement

Style is by definition a subjective topic, and because we are a style focused course, we sometimes need to pick one way to do a certain thing in order to keep our sections consistent from a grading and feedback perspective. This advice, while trying to find an ideal middle ground, will not always apply to every situation. We encourage you to look through the additional readings if you want a fuller perspective on this topic.