About Client
The Client is a leading provider of Virtualization and MDM Solutions which enables IT to deal with challenges associated with mobility while empowering users to work the way they want without compromising security.
Application: Mobile Device Management Solution
Mobile devices are proliferating in the enterprise at an exponential rate. With the growing number of device models, platforms and operating system versions available, enterprises are facing new and complex mobility management challenges. Accessing corporate resources from a mobile device can introduce a significant threat to information & IP security. MDM Solution can include over-the-air distribution of applications, data and configuration settings for all types of mobile devices; be it company-owned or employee-owned (BYOD) devices across the organization.
Problems and solutions
Considering the complexity of the software and continuous evolving of the product with addition of new features, QA cycles were taking too long and the client wanted faster turn-around from QA Team covering all the features in order to manage and improve the release deadlines.
Solution
- Reviewed Client requirements and built an efficient team with experience on enterprise centric products and relevant skills. This gave an extra edge in identifying real world scenarios as well as in troubleshooting issues found during testing.
- After initial training, the team focused on core features of the product. In a phased manner the team included other components and gradually covered all the existing features in the regression cycle. Once the team was accustomed to the TCs (test cases), we worked on increasing productivity without affecting the quality.
- Being on a different time-zone also has a strategic advantage. Onsite developers would finish their work and generate a nightly build which was tested overnight (i.e. our day) and a detailed report was sent before they would come to work the next day. This enabled us to make efficient use of full 24 hours, reduce turnaround time, get faster feedback and increase overall productivity.
With the planned efforts, soon we were able to cover entire Test Suite in 40% lesser timeframe.
One of the main client requirements was to use real devices for all the testing as performing tests on real devices would give the most realistic view of test results. However, it is expensive to acquire/manage so many test devices and perform all the tests on such a wide range of devices. Potentially thousands of different devices can be used by consumers and it is not possible to get coverage on all of them.
Solution
- We discussed this with the client and based on the client feedback, existing customer base, and market data, created a matrix of the devices for testing to ensure that we are getting the optimum coverage in terms of OS, Version, and Manufacturer. And if a feature was dependent on devices manufacturer/OS/version, we ensured that all positive and negative scenarios are covered using different devices.
Considering the complexity of the software and continuous evolving of the product with addition of new features, QA cycles were taking too long and the client wanted faster turn-around from QA Team covering all the features in order to manage and improve the release deadlines.
Solution
- To overcome this challenge, we used the available product documentation. In case of queries we discussed with the on-site QA Lead and based on the details we created mind maps, test cases for the features and get them reviewed by QA Managers, Product Managers, Development Managers to ensure that we understood the feature correctly and we are all on same page.
We also used to sync up with on-site teams through weekly/bi-weekly meetings and discuss tasks, issues encountered if any and tried to overcome the hurdles.
Client did not have the bandwidth to setup different complex enterprise setups to ensure that the application functioned properly in various environments.
Solution
- With help of our proficient IT Team having hands-on experience in setting up complex IT infrastructure and in-depth understanding of various aspects of Information Security, we created an in-house setup to replicate enterprise like test-beds. We then performed product testing by integrating the required components like:
LDAP: Microsoft AD
Email : Microsoft Exchange, Lotus Notes
Certificate Enrollment Methods : NDES/ OpenSCEP
VPN : PPTP, L2TP, IPSec
Wi-Fi: WPA, WPA2, WEP (Personal & Enterprise with different types of authentication including User & Cert based Auth using RADIUS server)
We identified that tests needed to be automated to achieve efficiency.
Solution
- After careful analysis, ROBOT Framework was found to satisfy all needed requirements.
It is an open-source Advanced Keyword Driven Framework with Multi-OS support. Benefit of working with Robot Framework is that writing test cases follows natural work flow i.e. Real language is used for keyword description, so it is easy to follow a test case – even for a non-programmer, which together with its simple usage and easy library extension, makes it a great tool for test case automation providing powerful results reporting.
ROBOT Framework was not well-known while we started working on it and hence adequate documentation was not available. Engineers researched to get relevant information in case of any problems. Gradually we were well-versed with it and we started creating test libraries, keywords, & test cases successfully and were able to automate most of the test cases.
Benefits to the Client
Due to us effectively managing and executing regression tasks and handling special assignments the client had the following benefits:
- Thorough insight into the quality of software
- Complete coverage of device/OS combination for testing
- Reduced turn-around time as offshore team worked in a different time zone
- Fewer QA bottlenecks during releases
- Test case management including creating and updating TCs
- Dedicated team members working on the product in sync with onsite team
- Tailored reporting (Daily as well as End of Iteration Reports)
- Ease of availability of test-beds representing enterprise environments
- Scalable test lab with different configurations and adequate devices/tools
