Offline UI Testing on iOS With Stubs
The original post is published on the JUST EAT tech blog at the following URL http://tech.just-eat.com/2015/11/23/offline-ui-testing-on-ios-with-stubs/
Here at JUST EAT, while we have always used stubs in Unit Tests, we tested against production public APIs for our functional and UI Testing. This always caused us problems with APIs returning different data depending on external factors, such as time of day. We have recently adopted the UI testing framework that Apple introduced at the WWDC 2015 to run functional/automation tests on the iOS UK app and stubs for our APIs along with it. This has enabled us to solve the test failures caused by network requests gone wrong or returning unexpected results.
For out UI Testing we used to rely on KIF but we have never been completely satisfied, for reasons such as:
- The difficulty of reading KIF output because it was mixed in the app logs
- The cumbersome process of taking screenshots of the app upon a test failure
- General issues also reported by the community on the GitHub page
We believe that Apple is providing developers with a full set of development tools and even though some of them are far from being reliable in their initial releases, we trust they will become more and more stable over time.
Another pitfall for us is that our APIs return different values, based on the time of the day, because restaurants might be closed and/or their menu might change. As a consequence, the execution of automation tests against our public APIs was causing some tests not to pass.
Rethinking our functional tests from scratch allowed us to raise the bar and solve outstanding issues with a fresh mind.
We realised we could use the same technology used in our Unit test to add support for offline testing in the automation tests, and therefore we designed around OHHTTPStubs to stub the API calls from the app. Doing this was not as trivial as it might seem at first. OHHTTPStubs works nicely when writing unit tests as stubs can be created and removed during the test, but when it comes to automation tests it simply doesn’t work.
The tests and application run as different instances, meaning that there is no way to inject data directly from the test code. The solution here is to launch the application instance with some launch arguments for enabling a “testing mode” and therefore generating a different data flow.
We pass parameters to the app either in the setup method (per test suite):
1 2 3 4 5 6 7 8
or per single test:
1 2 3 4 5 6 7
In our example we pass two parameters to signal to the app that the automation tests are running. The first parameter is used to stub a particular set of API calls (we’ll come back to the naming later) while the second one is particularly useful to fake the reachability check or the network layer to avoid any kind of outgoing connections. This helps to make sure that the app is fully stubbed, because if not, tests could break in the future due to missing connectivity on the CI machine, API issues or time sensitive events (restaurants are closed etc).
We enable the global stubbing at the end of the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
The launch arguments are retrieved from the application thanks to the
NSProcessInfo class. It should now be clearer why we used the
STUB_API_CALLS_stubsTemplate_addresses argument: the suffix
stubsTemplate_addresses is used to identify a special bundle folder in the app containing the necessary information to stub the API calls involved in the test.
This way the Test Automation Engineers can prepare the bundle and drop it into the project without the hassle of writing code to stub the calls. In our design, each bundle folder contains a
stubsRules.plist file with the relevant information to stub an API call with a given status code, HTTP method and, of course, the response body (provided in a file in the bundle).
This is how the stubs rules are structured:
At this point, there’s nothing more left than showing some code responsible for doing the hard work of stubbing. Here is the
JEHTTPStubManager class previously mentioned in the AppDelegate.
1 2 3 4 5 6 7
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
We created an utility category around OHHTTPStubs:
1 2 3 4 5 6 7 8 9 10 11 12 13
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56
Having our automation tests running offline reduced the majority of red test reports we were seeing with our previous setup. For every non-trivial application, running all the test suites takes several minutes and the last thing you want to see is a red mark in C.I. due to a network request gone wrong. The combination of OHHTTPStubs and Apple’s test framework has enabled us to run the automation tests at anytime during the day and to completely remove the possibility of errors that arise as a result of network requests going wrong.