Offline UI testing on iOS with stubs
The original post is published on the Just Eat tech blog at this URL.
Here at Just Eat, while we have always used stubs in Unit Tests, we tested against production public APIs for our functional and UI Testing. This always caused us problems with APIs returning different data depending on external factors, such as time of day. We have recently adopted the UI testing framework that Apple introduced at the WWDC 2015 to run functional/automation tests on the iOS UK app and stubs for our APIs along with it. This has enabled us to solve the test failures caused by network requests gone wrong or returning unexpected results.
Problem
For out UI Testing we used to rely on KIF but we have never been completely satisfied, for reasons such as:
- The difficulty of reading KIF output because it was mixed in the app logs
- The cumbersome process of taking screenshots of the app upon a test failure
- General issues also reported by the community on the GitHub page
We believe that Apple is providing developers with a full set of development tools and even though some of them are far from being reliable in their initial releases, we trust they will become more and more stable over time.
Another pitfall for us is that our APIs return different values, based on the time of the day, because restaurants might be closed and/or their menu might change. As a consequence, the execution of automation tests against our public APIs was causing some tests not to pass.
Proposed Solution
Rethinking our functional tests from scratch allowed us to raise the bar and solve outstanding issues with a fresh mind.
We realised we could use the same technology used in our Unit test to add support for offline testing in the automation tests, and therefore we designed around OHHTTPStubs to stub the API calls from the app. Doing this was not as trivial as it might seem at first. OHHTTPStubs works nicely when writing unit tests as stubs can be created and removed during the test, but when it comes to automation tests it simply doesn’t work.
The tests and application run as different instances, meaning that there is no way to inject data directly from the test code. The solution here is to launch the application instance with some launch arguments for enabling a “testing mode” and therefore generating a different data flow.
We pass parameters to the app either in the setup method (per test suite):
override func setUp() {
super.setUp()
continueAfterFailure = false
let app = XCUIApplication()
app.launchArguments = ["STUB_API_CALLS_stubsTemplate_addresses",
"RUNNING_AUTOMATION_TESTS"]
app.launch()
}
or per single test:
func test_ApplePayAvailable_UserLoggedIn_ServiceTypeDelivery() {
let app = XCUIApplication()
app.launchArguments = ["STUB_API_CALLS_stubsTemplate_addresses",
"RUNNING_AUTOMATION_TESTS"]
app.launch()
// test code
}
In our example we pass two parameters to signal to the app that the automation tests are running. The first parameter is used to stub a particular set of API calls (we’ll come back to the naming later) while the second one is particularly useful to fake the reachability check or the network layer to avoid any kind of outgoing connections. This helps to make sure that the app is fully stubbed, because if not, tests could break in the future due to missing connectivity on the CI machine, API issues or time sensitive events (restaurants are closed etc).
We enable the global stubbing at the end of the application:didFinishLaunchingWithOptions: method:
#ifndef APP_STORE_BUILD
[self _stubAPICallsIfNeeded];
#endif
//...
- (void)_stubAPICallsIfNeeded
{
// e.g. if 'STUB_API_CALLS_stubsTemplate_addresses' is received as argument
// we globally stub the app using the 'stubsTemplate_addresses.bundle'
NSString *stubPrefix = @"STUB_API_CALLS_";
NSString *bundleName = [[[[NSProcessInfo processInfo].arguments filterUsingBlock:^BOOL(NSString *arg) {
return [arg hasPrefix:stubPrefix];
}] firstObject] stringByReplacingOccurrencesOfString:stubPrefix withString:@""];
if (bundleName)
{
[JEHTTPStubManager applyStubsInBundleWithName:bundleName];
}
}
The launch arguments are retrieved from the application thanks to the NSProcessInfo class. It should now be clearer why we used the STUB_API_CALLS_stubsTemplate_addresses argument: the suffix stubsTemplate_addresses is used to identify a special bundle folder in the app containing the necessary information to stub the API calls involved in the test.
This way the Test Automation Engineers can prepare the bundle and drop it into the project without the hassle of writing code to stub the calls. In our design, each bundle folder contains a stubsRules.plist file with the relevant information to stub an API call with a given status code, HTTP method and, of course, the response body (provided in a file in the bundle).
This is how the stubs rules are structured:
At this point, there's nothing more left than showing some code responsible for doing the hard work of stubbing. Here is the JEHTTPStubManager class previously mentioned in the AppDelegate.
#import <Foundation/Foundation.h>
@interface JEHTTPStubManager : NSObject
+ (void)applyStubsInBundleWithName:(NSString *)bundleName;
@end
#import "JEHTTPStubManager.h"
#import "OHHTTPStubs+JEAdditions.h"
static NSString *je_mappingFilename = @"stubsMapping";
static NSString const *je_matchingURL = @"matching_url";
static NSString const *je_jsonFile = @"json_file";
static NSString const *je_statusCode = @"status_code";
static NSString const *je_httpMethod = @"http_method";
@implementation JEHTTPStubManager
+ (void)applyStubsInBundleWithName:(NSString *)bundleName
{
NSParameterAssert(bundleName);
NSString *bundlePath = [[NSBundle mainBundle] pathForResource:bundleName ofType:@"bundle"];
NSBundle *bundle = [NSBundle bundleWithPath:bundlePath];
NSString *mappingFilePath = [bundle pathForResource:je_mappingFilename ofType:@"plist"];
NSArray *mapping = [NSArray arrayWithContentsOfFile:mappingFilePath];
[mapping enumerateObjectsUsingBlock:^(NSDictionary * _Nonnull stubInfo, NSUInteger idx, BOOL * _Nonnull stop) {
NSString *matchingURL = stubInfo[je_matchingURL];
NSString *jsonFile = stubInfo[je_jsonFile];
NSNumber *statusCode = stubInfo[je_statusCode];
NSString *httpMethod = stubInfo[je_httpMethod];
NSString *inlineResponse = stubInfo[je_inlineResponse];
id stub = [OHHTTPStubs stubURLThatMatchesPattern:matchingURL
withJSONFileName:jsonFile
statusCode:[statusCode integerValue]
HTTPMethod:httpMethod
bundle:bundle];
}];
}
@end
We created an utility category around OHHTTPStubs:
#import "OHHTTPStubs.h"
@interface OHHTTPStubs (JEAdditions)
+ (id)stubURLThatMatchesPattern:(NSString *)regexPattern
withJSONFileName:(NSString *)jsonFileName
statusCode:(NSInteger)statusCode
HTTPMethod:(NSString *)HTTPMethod
bundle:(NSBundle *)bundle;
// ...
@end
#import "OHHTTPStubs+JEAdditions.h"
@implementation OHHTTPStubs (JEAdditions)
#pragma mark - Public
+ (id)stubURLThatMatchesPattern:(NSString *)regexPattern
withJSONFileName:(NSString *)jsonFileName
statusCode:(NSInteger)statusCode
HTTPMethod:(NSString *)HTTPMethod
bundle:(NSBundle *)bundle
{
NSBundle *targetBundle = bundle ?: [NSBundle bundleForClass:[self class]];
NSString *path = [targetBundle pathForResource:jsonFileName ofType:@"json"];
NSString *responseString = [NSString stringWithContentsOfFile:path
encoding:NSUTF8StringEncoding
error:nil];
return [self _stubURLThatMatchesPattern:regexPattern withResponseString:responseString statusCode:statusCode HTTPMethod:HTTPMethod];
}
// ...
#pragma mark - Private
+ (id)_stubURLThatMatchesPattern:(NSString *)regexPattern
withResponseString:(NSString *)responseString
statusCode:(NSInteger)statusCode
HTTPMethod:(NSString *)HTTPMethod
{
NSRegularExpression *regex = [NSRegularExpression regularExpressionWithPattern:regexPattern options:0 error:nil];
return [OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request) {
if (HTTPMethod && ![request.HTTPMethod isEqualToString:HTTPMethod])
{
return NO;
}
NSString *requestURLString = [request.URL absoluteString];
if ([regex firstMatchInString:requestURLString options:kNilOptions range:NSMakeRange(0, [requestURLString length])]) {
return YES;
}
return NO;
} withStubResponse:^OHHTTPStubsResponse*(NSURLRequest *request) {
NSData *response = [responseString dataUsingEncoding:NSUTF8StringEncoding];
return [OHHTTPStubsResponse responseWithData:response
statusCode:(int)statusCode
headers:@{@"Content-Type":@"application/json; charset=utf-8"}];
}];
}
@end
Having our automation tests running offline reduced the majority of red test reports we were seeing with our previous setup. For every non-trivial application, running all the test suites takes several minutes and the last thing you want to see is a red mark in C.I. due to a network request gone wrong. The combination of OHHTTPStubs and Apple’s test framework has enabled us to run the automation tests at anytime during the day and to completely remove the possibility of errors that arise as a result of network requests going wrong.
Update: see also the similar and well-thought-out post Test automation for iOS by Stanislav Pankevich @svpankevich.