The development of the GoFAST software at CEO-Vision has been structured from the beginning with a set of support tools such as Git for the source management, JIRA for bugs and requirements management, Jenkins for deployment automation and many other tools.
However, on the test side, they were confined to an integration environment on which manual and non-exhaustive tests were carried out for each version, listed in an Excel spreadsheet. These tests were supplemented by automated tests concerning security (vulnerability scan,...).
With the increase in the number of customers and the multiplicity of configurations (browsers, OS, ...), the level of undetected significant bugs before production deployment led to the creation of HOTFIX. Moreover it became increasingly difficult to have a clear view of the test results in the pre-production phase. With a "sponsor" client whishing a better industrialization a major upgrade, we started this important project in the summer of 2018.
First we searched for a tool fitting our testing needs to:
-
be user friendly for most user`s background (technical or business)
-
be extensible (API, ...), in particular to be able to dialogue with our continuous integration platform and to accept tests automation in a second phase.
-
be integrated with JIRA or being able to integrate it (JIRA bugs creation directly in the testing tool )
-
be financially affordable
-
be open: being able to import and export tests cases easily
-
produce reports readable by everyone
So we tested JIRA plugins like XRay or Zephyr and other external ones to JIRA like Inflectra, TestRail and Practitest.
Overall the functional coverage of these tools is quite close. However the ease of use and ease of import / export allowed us to select TestRail very clearly and quickly, TesRail being far superior.
The first step was to import our "Test cases" from our Excel file. These "Tests cases" were grouped into "Tests suites" defining tests that must be performed at the same time and which generally correspond to a specific functional perimeter, in our case :
-
GoFAST "Core" (the common functions in all our versions)
-
GoFAST "Enterprise" (specific functions in this version)
-
GoFAST "Community" (the specific functions in this version)
-
GoFAST "Specific Client A" (customer-specific functions with a custom GoFAST)
-
Mobile & Tablet
-
Technical tests (architecture, ...)
These Tests Suites are grouped into "Tests Plan" and executed :
-
At the customer’s first installation
-
At each launch of a new major or intermediate version
Note that these "Plan Tests" can be executed for any Browser / OS pair. Nevertheless it is too expensive to do it in "manual" test.
When bugs are detected, they can be automatically created in JIRA from the TestRail tool giving immediate visibility to the development team. They are corrected and the relevant "Test Case" relaunched after the fix is produced.
Also note that the test cases are 'live'. We enrich them over time.
Of course spending hundreds of tests is expensive, however we quickly saw the number of bugs (new or regression type) coming down and therefore less patch to produce urgently and a better "quality perception" within our users and customers.
The next big step in 2019 will be testing automation in TestRail with a tool that simulates user navigation in the browser, Selenium.