Before i say anything else i want to mention that these are my views. So they might not be coherent with yours. Well you just gotta deal with it.

So about testing on iOS. Let me first start with just testing on mobile devices. As a tester, you have a few more responsibilities as compared to testing desktop applications. Let me pin down what i think are the requirements to make up for a comprehensive testing and assuring a stable and optimal product across revisions.

1. Feature Testing
2. Automation
3. Profiling

But when you talk about mobile devices, things unfold in a bit more complex, narrowed way. Also testing has to be done along side development, otherwise it just wouldn’t work. Waterfall model of development is not at all suitable for mobile development. For mobile testing, the list goes like this

1. Feature Testing
2. Automation (How?)
3. Profiling for leaks (We wanna be very memory efficient right?)
4. Profiling for CPU Usage (We don’t want to choke the device now do we)
5. Profiling for memory usage (Again, cant choke the device out of memory)
6. Allocations (Later?)

Now whats new is that in Desktop applications, you don’t need to be so particular about profiling. Fair enough is good. But for mobile apps, you need to be sure you are not doing anything to use extra resources. Reason obviously being we have limited resources and more importantly, there are other apps as well who want those resources. We need to live in harmony remember.

Another thing to remember for a tester on mobile devices. If you go to a developer with “Our app is crashing”, “We are using too much memory”, “We are using too much CPU”. Its not good enough. You need to figure out whats causing that behavior. Is memory the culprit, is CPU or is there some wrong call in the code? And for this you need to profile the builds at the same time you are testing it. You need to get hold of how to use allocations to figure some things. For the testing to be comprehensive, you need to be hand-on with all these things.

Except for manually testing the features, everything else would require tools. This is again something which would be mobile OS dependent. But the bottom line is, we need tools for automation, profiling leaks, CPU, Memory Usage and tracking allocations. Also we need to save the traces (dumps?) of the runs we found problematic, so that developers can look at it and more often then not figure out what the problem is.

Now lets talk iOS testing.

We will keep Automation for the end. The good news is, Apple provides us with tools for all the profiling and allocation tracking in a nifty Mac application called Instruments (remember instrumentation?).It ships with XCode. You have many many tools in Instruments for instrumenting your app. Starting with activity monitor to automation. Lets talk a bit about some of them and how they are useful for mobile testers. You can also go through the list here.

Activity Monitor : Use this to track the system workload and also virtual memory. This is basically a counterpart of Activity monitor for Desktop on iOS. Use it as you use activity monitor.

Allocations : Monitor memory and object allocation pattern.

Leaks : This along with allocations is used to track memory leaks.

CPU Sampler : Use for CPU profiling, you can compare the workload of your app vs the whole system.

And there are many more tools in Instruments which you can use to profile your app. The whole kit is in there. It sure is a pain to start but it makes your life so easy you cant even imagine. Now the best part, you can use these profiling tools on the actual device as well as on simulator. You can create and save the dumps anytime you want, and not to mention, load it later.

Allocations, i want to talk about allocations once again. To start with, it might look like why does a tester need to worry about allocation patterns. Well for mobile devices, you need to help the developers, and in turn the product in finding out whats causing the problem. Allocation lets you do that. Using allocations you can find out which function consumed most amount of time, which function was called most number of times and so on. This information is very useful. Even in case you don’t understand it, remember to track this information for your dump, because it will make life easier for the people debugging the code based on your dumps.

Now we come to automation. This is a bit tricky one because of the requirements of automation systems. Sometimes we want one click automation which would do everything for us and generate the reports, sometimes we want the automation system to be in CI (Continuous Integration). Most of the tools available right now don’t allow these upfront, well not unless you make some tweaks of your own. Most of the tools are UI automation, and given the scenario, i would strongly suggest doing UI automation, because in mobile apps, its the feel that matters. White box automation is always good to have, but in mobile apps, black box automation is a necessity. I have been through some of the tools like FoneMonkey, Frank Framework by ThoughtWorks and the Automation Instruments (shipped in with XCode by Apple). Where Automation Instrument and FoneMonkey are completely UI driven automation, Frank Framework has the capability to do White box automation. Lets talk about each of them.

FoneMonkey : Easy to start with, record and playback, option to change and save the scripts on device(Yay!!), black box testing, found that a few commands got executed without actions (Ouch!!), required to be added in your source code (Eeee!!!) and not very accurate.

Frank Automation : Assume you have a server sitting inside your app and you can communicate with that server. White box testing, Reasonably Hard to start with and maintain, Can very easily be used in CI. Required to be checked in to your source code, with a server sitting in your app, you can basically do anything in the app using white box testing off course.

Automation Instrument : Black Box Testing, One script at a time execution, No interference with code is required, Profiling tools can be used while running automation instruments leading to automatic generation of dumps,

As i said, none of these are a sure shot winner. All of them can be tweaked to be useful. I would recommend frank for white box testing and Automation Instruments for Black Box.

CI with Automation Instruments : Automation Instruments uses javascript as the scripting language and runs one such script at a time, generating a xml output file on the go. We can use apple script to automate Automation Instrument itself. i can say its very much possible having done it myself. After this you can use Automation Instrument for CI as well.

This is a very small outline of what is the minimum required for testing and what are the option to get it done. This is mainly for iOS thought the concept pass the boundary can carry on for android and other mobile platforms as well. Hope this was useful, for any clarifications, mail me at maheshgattani@gmail.com.

I have been having these long discussions with friends and colleagues about android and iOS. Which one is better? Which one has a better future? Which device to buy right now, IPhone or Android device? I have heard so many arguments, arguments from hard core android fans and arguments from apple fans.

I for one, believe iOS is in a very very good state as compared to android. Also, when we say iOS, we can assume the device and the OS as the same because of the integration apple provides. When it come to Android, we have to think of the OS separately and of device separately. But lets see what each of them has to offer.

For Developers?

When we talk about the developer view, we need to take care of 2 things. What all we can do and how easily we can do that?

Android is a clear winner in what all you can do. Being an open source, it provides such an environment that you can essentially do anything if your device permits it.

iOS is a restricted environment. Somethings are not meant to be changed because the OS is designed that way (read Flash, Any app which can be used to run other programs is not allowed). Apple has a screening process, so just because you developed an app doesn’t mean you can put it on App store. You need to follow some strict guidelines only then you can assume apple would approve your app.

iOS is a winner when it comes to how easily can you do something.

To start with, the toughest part being creating the UI experience. iOS development SDK provides you with a tool for interface building. A very easy to use and effective tool. MVC(Model-View-Controller) implementation might be a bit tricky to start with but is very effective as well. The code goes in Objective C which is just another OOP language and you tend to get used to it as you use it. Also since there are only a small set of devices with iOS, building an app on it takes a lot less of thinking as to what all devices you support.

Android lacks heavily in terms of UI experience. Without any standard interface builder, it is a haunting task to get your UI as you want it to look. Writing whole or part of views in XML might not suit all the users. The code goes in Java which is very well known. Also, with so many android devices in market, it might be a painful experience to figure out which all devices you want to support. And since the OS and device are not so well linked, you also have to think about the device specification when you are building an app.

For Users

For users, things boil down to ease of use, User experience and ability to do more.

iOS, for me, is the winner when it comes to ease of use. The devices with iOS gives you an easy feel to everything. Be it accessing things, be it closing things, be it downloading music and videos, be it downloading new apps, almost everything. For a person who is not very technical, iOS is far far easy to use then it android counterpart.

User experience, again iOS. This is true because of the tight integration between the device and the OS. This is an easy one, just try playing a game in android and same on iOS, the difference is visible. Also, android as a platform isnt as stable as iOS is. The apps on android arnt as stable as they are on iOS (read apple`s filtering process).

Ability to do more? This is tricky, mainly because android as a package does provides you more stuff to do. But iOS provides you more app to do search and use from. The difference isn’t very apparent. On the contrary, you can do almost anything on both the OS`s. You would find app for everything on both the platforms. As of now, you have many more devices to select from in android and many more app to select from in iOS. My vote goes to iOS, being more stable and with lot more apps.

Going Ahead

Both the platforms, iOS and android, have great future. iOS 5 has been announced and it includes lot of nice features. Android 3.1 (Ice cream sandwich) is out which is the first in line to supports both tablets and handsets. Though Android is behind iOS right now, i expect it to match iOS sooner than later. With contributions from everywhere, android is developing fast. We have Microsoft “Mango” coming. Mobile field is very exciting right now and rapidly progressing.

What to buy?

My vote goes to iOS mainly because of the tight integration. The iOS devices are high end when it comes to specs. The user experience is amazing as compared to only good for android. Android OS depends a lot on device hardware. Samsung has done some rights with Galaxy series. It basically boils down to

“Freedom of choice OR freedom from choice”

read

“Android OR iOS”