Before i say anything else i want to mention that these are my views. So they might not be coherent with yours. Well you just gotta deal with it.
So about testing on iOS. Let me first start with just testing on mobile devices. As a tester, you have a few more responsibilities as compared to testing desktop applications. Let me pin down what i think are the requirements to make up for a comprehensive testing and assuring a stable and optimal product across revisions.
- Feature Testing
- Automation
- Profiling
But when you talk about mobile devices, things unfold in a bit more complex, narrowed way. Also testing has to be done along side development, otherwise it just wouldn’t work. Waterfall model of development is not at all suitable for mobile development. For mobile testing, the list goes like this
- Feature Testing
- Automation (How?)
- Profiling for leaks (We wanna be very memory efficient right?)
- Profiling for CPU Usage (We don’t want to choke the device now do we)
- Profiling for memory usage (Again, cant choke the device out of memory)
- Allocations (Later?)
Now whats new is that in Desktop applications, you don’t need to be so particular about profiling. Fair enough is good. But for mobile apps, you need to be sure you are not doing anything to use extra resources. Reason obviously being we have limited resources and more importantly, there are other apps as well who want those resources. We need to live in harmony remember.
Another thing to remember for a tester on mobile devices. If you go to a developer with “Our app is crashing”, “We are using too much memory”, “We are using too much CPU”. Its not good enough. You need to figure out whats causing that behavior. Is memory the culprit, is CPU or is there some wrong call in the code? And for this you need to profile the builds at the same time you are testing it. You need to get hold of how to use allocations to figure some things. For the testing to be comprehensive, you need to be hand-on with all these things.
Except for manually testing the features, everything else would require tools. This is again something which would be mobile OS dependent. But the bottom line is, we need tools for automation, profiling leaks, CPU, Memory Usage and tracking allocations. Also we need to save the traces (dumps?) of the runs we found problematic, so that developers can look at it and more often then not figure out what the problem is.
Now lets talk iOS testing.
We will keep Automation for the end. The good news is, Apple provides us with tools for all the profiling and allocation tracking in a nifty Mac application called Instruments (remember instrumentation?).It ships with XCode. You have many many tools in Instruments for instrumenting your app. Starting with activity monitor to automation. Lets talk a bit about some of them and how they are useful for mobile testers. You can also go through the list here.
Activity Monitor : Use this to track the system workload and also virtual memory. This is basically a counterpart of Activity monitor for Desktop on iOS. Use it as you use activity monitor.
Allocations : Monitor memory and object allocation pattern.
Leaks : This along with allocations is used to track memory leaks.
CPU Sampler : Use for CPU profiling, you can compare the workload of your app vs the whole system.
And there are many more tools in Instruments which you can use to profile your app. The whole kit is in there. It sure is a pain to start but it makes your life so easy you cant even imagine. Now the best part, you can use these profiling tools on the actual device as well as on simulator. You can create and save the dumps anytime you want, and not to mention, load it later.
Allocations, i want to talk about allocations once again. To start with, it might look like why does a tester need to worry about allocation patterns. Well for mobile devices, you need to help the developers, and in turn the product in finding out whats causing the problem. Allocation lets you do that. Using allocations you can find out which function consumed most amount of time, which function was called most number of times and so on. This information is very useful. Even in case you don’t understand it, remember to track this information for your dump, because it will make life easier for the people debugging the code based on your dumps.
Now we come to automation. This is a bit tricky one because of the requirements of automation systems. Sometimes we want one click automation which would do everything for us and generate the reports, sometimes we want the automation system to be in CI (Continuous Integration). Most of the tools available right now don’t allow these upfront, well not unless you make some tweaks of your own. Most of the tools are UI automation, and given the scenario, i would strongly suggest doing UI automation, because in mobile apps, its the feel that matters. White box automation is always good to have, but in mobile apps, black box automation is a necessity. I have been through some of the tools like FoneMonkey, Frank Framework by ThoughtWorks and the Automation Instruments (shipped in with XCode by Apple). Where Automation Instrument and FoneMonkey are completely UI driven automation, Frank Framework has the capability to do White box automation. Lets talk about each of them.
FoneMonkey : Easy to start with, record and playback, option to change and save the scripts on device(Yay!!), black box testing, found that a few commands got executed without actions (Ouch!!), required to be added in your source code (Eeee!!!) and not very accurate.
Frank Automation : Assume you have a server sitting inside your app and you can communicate with that server. White box testing, Reasonably Hard to start with and maintain, Can very easily be used in CI. Required to be checked in to your source code, with a server sitting in your app, you can basically do anything in the app using white box testing off course.
Automation Instrument : Black Box Testing, One script at a time execution, No interference with code is required, Profiling tools can be used while running automation instruments leading to automatic generation of dumps,
As i said, none of these are a sure shot winner. All of them can be tweaked to be useful. I would recommend frank for white box testing and Automation Instruments for Black Box.
CI with Automation Instruments : Automation Instruments uses javascript as the scripting language and runs one such script at a time, generating a xml output file on the go. We can use apple script to automate Automation Instrument itself. i can say its very much possible having done it myself. After this you can use Automation Instrument for CI as well.
This is a very small outline of what is the minimum required for testing and what are the option to get it done. This is mainly for iOS thought the concept pass the boundary can carry on for android and other mobile platforms as well. Hope this was useful, for any clarifications, mail me at maheshgattani@gmail.com.