Continuing to my last post. Today i want to discuss how to make android apps which are stable, fast and efficient on resources. By now we should understand, mobile platforms are different and need specific tools as well. But whats common is, the need to optimize resource utilization. Again, like for iOS, we discuss the following topics for Android.

1. Feature Testing
2. Automation
3. Profiling for leaks
4. Profiling for CPU Usage
5. Profiling for memory usage
6. Allocations

Before you start reading further, i recommend reading abut the instrumentation class in Android documentation and DDMS. DDMS is a great tool provided by android specifically to profile your application and is for our good luck, very nicely integrated in Eclipse.

Feature Testing is usual manual testing. It doesn’t require any tools and so is the same as how you do it on Desktop Application or on iOS.

Automation : There are many tools out there to do automation on android. AutoAndroid, SeeTest, Calculon, Robotium and many more may be. This image taken from Vendor Benchmarking depicts nicely the comparison of the first three tools.

I would personally say, Robotium, having used it, is very easy to start with. Being a library so can be imported directly to your project. You can also create a separate test project along side your app. This is the normal way of using it. In order to use it for one click automation, you need it be a totally independent project. And so you need to write a wrapper above Robotium which can be bit of a pain. All the above mentioned 4 automation tools solve different problems and you need to pick one based on your product and your requirement. This goes without saying that all of these are black box testing tools and may or may not require maintenance as you go along developing.

Lets go to profiling, Leaks in your android are hard to find. I couldn’t find any useful tool out there to successfully monitor leaks on android. There is no equivalent of Instrument Leaks of iOS for Android. But there are other ways you can try and monitor. Instrumentation class provides us access to the memory usage of your app at any point of time. You just need to make a call and you can get the memory usage. Now using this function you can track the memory usage before and after the code you want to monitor. I understand its hard to find small leaks through this approach but its the best i got. Let me know if you find something better :).

While we are on memory usage. Lets go through memory utilization as well. There are many many ways to get to know your apps memory utilization. DDMS is the easiest, just select the DDMS view and your app from the left side pane. Then there is Instrumentation class, you can use it to track the memory utilization from your code. Then there is another way, your adb shell. You can use many commands to do so, top, dumpsys meminfo proc, procrank, vmstat. There is another tool named smem but you have to separately install it.The tricky part is, once you run the above commands, we get many memory values and its not very clear which one is useful

• Vss = virtual set size
• Rss = resident set size
• Pss = proportional set size
• Uss = unique set size

Different command line functions would give different outputs.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application’s “fair share” of each shared area to give a realistic measure of the activity of a system during boot. Hence PSS is our best shot for the memory usage by our app. I use procrank as it directly gives me the PSS value. Sometimes if you need more in depth output of dirty pages, you should use dumpsys meminfo.

Profiling for CPU. This is a bad one as i know only one way to do it. Use the top command and see the cpu usage. It works great for me tough because thats all i am looking for :).

Allocations again is something which can be done in multiple ways. I particularly like the DDMS solution for this. You can also use dmtracedump.exe and traceview. Whats good about android allocations is, you can create a dump file for the period you were monitoring your app for, and store it to be viewed later or used later or compared later. its an amazing debugging and testing tool and you should definitely read more about it.

All along the way i have assumed you have the knowledge of what is logcat and how to use it. Its basically a console output of what your device is doing. If you are reading this, you must already understand that for mobile testers, you need to get close to the product from the start and just UI level and feature testing is not enough.

I hope this was helpful. In case i mentioned something wrong or you want to ask anything, drop me a mail at maheshgattani@gmail.com

Before i say anything else i want to mention that these are my views. So they might not be coherent with yours. Well you just gotta deal with it.

So about testing on iOS. Let me first start with just testing on mobile devices. As a tester, you have a few more responsibilities as compared to testing desktop applications. Let me pin down what i think are the requirements to make up for a comprehensive testing and assuring a stable and optimal product across revisions.

1. Feature Testing
2. Automation
3. Profiling

But when you talk about mobile devices, things unfold in a bit more complex, narrowed way. Also testing has to be done along side development, otherwise it just wouldn’t work. Waterfall model of development is not at all suitable for mobile development. For mobile testing, the list goes like this

1. Feature Testing
2. Automation (How?)
3. Profiling for leaks (We wanna be very memory efficient right?)
4. Profiling for CPU Usage (We don’t want to choke the device now do we)
5. Profiling for memory usage (Again, cant choke the device out of memory)
6. Allocations (Later?)

Now whats new is that in Desktop applications, you don’t need to be so particular about profiling. Fair enough is good. But for mobile apps, you need to be sure you are not doing anything to use extra resources. Reason obviously being we have limited resources and more importantly, there are other apps as well who want those resources. We need to live in harmony remember.

Another thing to remember for a tester on mobile devices. If you go to a developer with “Our app is crashing”, “We are using too much memory”, “We are using too much CPU”. Its not good enough. You need to figure out whats causing that behavior. Is memory the culprit, is CPU or is there some wrong call in the code? And for this you need to profile the builds at the same time you are testing it. You need to get hold of how to use allocations to figure some things. For the testing to be comprehensive, you need to be hand-on with all these things.

Except for manually testing the features, everything else would require tools. This is again something which would be mobile OS dependent. But the bottom line is, we need tools for automation, profiling leaks, CPU, Memory Usage and tracking allocations. Also we need to save the traces (dumps?) of the runs we found problematic, so that developers can look at it and more often then not figure out what the problem is.

Now lets talk iOS testing.

We will keep Automation for the end. The good news is, Apple provides us with tools for all the profiling and allocation tracking in a nifty Mac application called Instruments (remember instrumentation?).It ships with XCode. You have many many tools in Instruments for instrumenting your app. Starting with activity monitor to automation. Lets talk a bit about some of them and how they are useful for mobile testers. You can also go through the list here.

Activity Monitor : Use this to track the system workload and also virtual memory. This is basically a counterpart of Activity monitor for Desktop on iOS. Use it as you use activity monitor.

Allocations : Monitor memory and object allocation pattern.

Leaks : This along with allocations is used to track memory leaks.

CPU Sampler : Use for CPU profiling, you can compare the workload of your app vs the whole system.

And there are many more tools in Instruments which you can use to profile your app. The whole kit is in there. It sure is a pain to start but it makes your life so easy you cant even imagine. Now the best part, you can use these profiling tools on the actual device as well as on simulator. You can create and save the dumps anytime you want, and not to mention, load it later.

Allocations, i want to talk about allocations once again. To start with, it might look like why does a tester need to worry about allocation patterns. Well for mobile devices, you need to help the developers, and in turn the product in finding out whats causing the problem. Allocation lets you do that. Using allocations you can find out which function consumed most amount of time, which function was called most number of times and so on. This information is very useful. Even in case you don’t understand it, remember to track this information for your dump, because it will make life easier for the people debugging the code based on your dumps.

Now we come to automation. This is a bit tricky one because of the requirements of automation systems. Sometimes we want one click automation which would do everything for us and generate the reports, sometimes we want the automation system to be in CI (Continuous Integration). Most of the tools available right now don’t allow these upfront, well not unless you make some tweaks of your own. Most of the tools are UI automation, and given the scenario, i would strongly suggest doing UI automation, because in mobile apps, its the feel that matters. White box automation is always good to have, but in mobile apps, black box automation is a necessity. I have been through some of the tools like FoneMonkey, Frank Framework by ThoughtWorks and the Automation Instruments (shipped in with XCode by Apple). Where Automation Instrument and FoneMonkey are completely UI driven automation, Frank Framework has the capability to do White box automation. Lets talk about each of them.

FoneMonkey : Easy to start with, record and playback, option to change and save the scripts on device(Yay!!), black box testing, found that a few commands got executed without actions (Ouch!!), required to be added in your source code (Eeee!!!) and not very accurate.

Frank Automation : Assume you have a server sitting inside your app and you can communicate with that server. White box testing, Reasonably Hard to start with and maintain, Can very easily be used in CI. Required to be checked in to your source code, with a server sitting in your app, you can basically do anything in the app using white box testing off course.

Automation Instrument : Black Box Testing, One script at a time execution, No interference with code is required, Profiling tools can be used while running automation instruments leading to automatic generation of dumps,

As i said, none of these are a sure shot winner. All of them can be tweaked to be useful. I would recommend frank for white box testing and Automation Instruments for Black Box.

CI with Automation Instruments : Automation Instruments uses javascript as the scripting language and runs one such script at a time, generating a xml output file on the go. We can use apple script to automate Automation Instrument itself. i can say its very much possible having done it myself. After this you can use Automation Instrument for CI as well.

This is a very small outline of what is the minimum required for testing and what are the option to get it done. This is mainly for iOS thought the concept pass the boundary can carry on for android and other mobile platforms as well. Hope this was useful, for any clarifications, mail me at maheshgattani@gmail.com.