Development of mobile application like any software development process is requiring a consistent approach to quality assurance. For mobile applications this need is especially actual as number of HW platforms that mobile application has to run is constantly increasing (especially for Android OS). Users are expecting continuously stable and reliable software. Failure to do so may pose an significant damage to user acceptance and loyalty.
Under this circumstances relying only on manual testing as a way for insuring quality is a very risky and cost intensive. Costs are especially exploding then our software is required to run on majority of Android devices available on the market. This is becoming almost impossible to deliver.
How we do it?
Very efficient way to address this problem is partial or maximal usage of automated testing as a pillar of quality assurance. Experience of instinctools EE labs shows that this approach can bring significant economic value as well as ensure constantly high level of quality. How is this archived?
In our projects we use two types of tests:
- Unit test
- UI Automation test
UI Automation tests are end-to-end functional tests that are conducted on pre-release builds and ensure that application is meeting application requirements. In fact UI Automation tests are automating typical test scenarios that otherwise are performed by human. Automated tests can be ran on the local workstation (developer or QA) or remotely using a special farm for mass testing (example: AppThwack). Using such a farm makes possible to run our tests on hundreds of different HW variants, thus archiving a way broader HW coverage than any manual testing approach can achieve.
There are several approaches for developing automated tests for iOS and Android platforms.
For Android platform we have analysed available test techniques and chose a Robotium framework . Robotium tests are based on JUnit and provide ample opportunities to interact with the UI. They also allow monolithic integration of tests directly in the project. In this way we are getting an opportunity to run automated tests directly on a developer workstation, as well as several ways to use remote HW farms like AppThwack .
Development/Quality Assurance process
Typical process for enabling quality assurance based on automated testing is consisting of following stages (happening usually in parallel to main development process):
- Preparing the infrastructure to develop and run automated tests
- Formalization of the application requirements,
- Writing manual test cases and scenarios for functional testing
- Qualification of test scenarios for automation
- Programming and debugging test scripts
- Run Automated Tests (on development, QA environments or HW farms)
- Processing of test results
Multiple HW compatibility
As mentioned above HW testing farms like AppThwack make possible to test our application or hundreds of different HW combinations in parallel. After running the tests these services provide set of reports that show us lists of failed tests and failed HW, as well as screenshots on different HW and resolutions if required. Here are some examples of such a tests :
Downsides of this approach
As usual any benefit comes with disadvantages. These include:
- No 100% automated test coverage. Unfortunately not all test cases can be automated. As well as for some automations some additional test-related infrastructural components have to be developed (test mocks, test infrastructures and s.o.). This means also that certain set of test cases will stay manual.
- Increased cost of initial development. To enable automation test driven approach more efforts have to be invested in test development. This can be 30-70% of development of initial application. Good news here is that this investment is paying back (on manual QA savings) with 2nd or 3rd application release already.
- Automated tests are becoming obsolete and have to be maintained. With every new feature development we need to plan for 30-50% of additional development efforts (developing new test and maintaining old). But again this is paid back generously by consistent SW quality, wide HW coverage guarantee and happy customers.