Can You Hack It?

Why wouldn't you just give up?

The system under test has poor testability which means that the testing you'd like to do will take longer and the resolution of your findings will be lower than you'd like.

So do you give up, battle through, wait for someone to add what you need ... or change the product yourself?

There are potential risks to that last option, of course, because (duh!) you're changing the product. But there are potential wins, too, because you're getting the data you want earlier and can give richer feedback to your stakeholders.

I took that route this week. 

The service I'm looking at is essentially a pipeline of steps, each of which calls out to a third-party service and post-processes the result, set up like this:

  result1 = step1.run(input)
  result2 = step2.run(result1)
  result3 = step3.run(result2)

I wanted the response time for a large number of requests against its API, which I can get easily from the client I am using, and also the time for each internal step. While there's a ticket to expose this data, it's not in the product yet.

I thought it was important to capture now, so I made a branch of the source code and added a timer and a log line. I'll use a Kotlin-ish psuedocode for snippets:

(result, time) = Timer {
	call_external_service1(some_data)
}
log.debug("timing, service1, $time, $requestID")

This produces log output that I can parse and visualise easily, for example:

timing, service1, 324, aaaaa
timing, service2, 221, aaaaa
timing, service3, 530, aaaaa

I included requestID, which is also in the API response, so that I can tie the timing data to each request I made in later analysis. I felt reasonably confident that I would not be affecting the behaviour of the product with this, but I showed the change to the developers to check.

For some of the tests I was more concerned about the behaviour of our code over sustained load and didn't care so much about hitting the external services. To facilitate that, I extended my changes:

(result, time) = Timer {
    if (spoof_external) {
	sleep(1000)
	"""
	  spoofed response body
	"""
    }
    else {
	call_external_service1(some_data)
    }
log.debug("timing, service1, $time, $requestID})

In this implementation, if I set a run-time flag spoof_external to true, then there's a short wait and spoofed response instead of an external call. This is enough to simulate the external service and force the service to follow its standard code paths. 

I could have added some variability to the latency and the responses if I'd wanted to, but this was good enough for a first iteration.

Both of those changes worked well. Next I wanted to be able to hit each of the external services individually. Again, our API doesn't provide this option, yet. 

I can do it through unit tests but I wanted something more interactive this time. Both of those approaches have value, and I've written about it in e.g. Exploratory Tooling, Use the Force Multiplier, and The Love of a List and a Loop.

The service under test is written in Kotlin using Spring Boot. I am fluent in neither so I am not about to add any new endpoints, but I could see a cheap way to do what I wanted and it was another iteration on what I'd already done.

In turn, for each step I wanted to explore, I spoofed the external service returning a field from the request my client made instead of some static content:

if (spoof_external) {
    input.some_field
}

This gives me a way to inject data direct to a step. Conceptually, I am telling a step not to take its input from the previous step, but to instead take it from my request, e.g:

result1 = step1.run(input)
result2 = step2.run(result1)
result3 = step3.run(input)

If you're thinking that it's inefficient because other steps still run you'd be right. But that was much less of a concern in this case than being able to test what I wanted to test.

Of course, I still need to see the output, so I added more logging:

log.debug("output, service3, $result")

With this change, I can have Bruno set up on one side of my screen and a shell on the other. In the shell I am doing this:

$ tail -f log.txt  | grep output,

As I submit a request in Bruno, I can see the result of the step I am interested in immediately:

output, service3, resultA
output, service3, resultB
output, service3, resultC

This is instant feedback and now I'm exploring interactively. After each experiment I'm able to go again immediately, with another variant, looking for interesting behaviour.

Was this valuable? 

Yes, I got data that helped us to understand the behaviour of the product in scenarios we think will be important.

Was there risk that I had changed the behaviour of the product sufficiently that the testing was invalid?

Yes, but I was careful to make my changes in such a way that I was happy the risk was very low: in every case I was wrapping or substituting a call I understood well. I also added logging that made it clear what was being called.

So, I made my choice but what would you do in the face of low testability? Just hack it, or just hack it?
Image:  Pawel Czerwinski on Unsplash
Syntax highlighting: pinetools