How to implement a software testing process within the SCRUM framework

Let’s start by listing SCRUM’s key features and aspects for those unfamiliar with it:

• The development is iterative and incremental. Iterative means that the development process is split into several equivalent periods of time called Sprints. One Sprint lasts from one to four weeks. Incremental means that each iteration results in a new, potentially usable product that solves a particular business problem. This product is called «increment».
• The team is self-organizing and does not have a Project Manager (PM);
• A SCRUM Master (SM) is present on the team.
• There is the customer’s representative called Product Owner (PO) on the team.
• The entire development timeframe is split into time periods called Sprints. The length of a Sprint is agreed upon at the start of the project and is changed only if unexpected circumstances do not allow the deadline to be met.
• Various tasks (functional requirements, bug fixes, corrections requested by the customer, etc.) form a work pool called a Backlog. Initially the Backlog contains only the customer’s requirements.
• Backlog Grooming is done at the start of a Sprint (or any later time in a Sprint if necessary) to review the Backlog tasks. This results in the creation of a processed Backlog for the two or three subsequent Sprints. The PO and the SCRUM team then define the Sprint’s goal and the anticipated result, and the team compiles the Sprint’s Backlog.
• After the Sprint has been planned, it is generally advised not to change its contents. However, if new tasks are added, old or equally prioritized tasks are removed from the Sprint. If this leads to a change of the Sprint’s goal, the Sprint is cancelled and planned from scratch.
• The methodology includes «Daily Scrums», or brief everyday meetings that serve to track progress and make sure that the team understands whether they’re getting closer to fulfilling the Sprint’s goal or not.
• At the end of a Sprint, the completed tasks are either approved or rejected and returned into the Backlog.
• Each Sprint results in a product increment.

Live Typing has employed SCRUM when working on several products to a certain degree of success: the team attained discipline without managerial supervision; the tasks were better assessed because their evaluation happened down the line when several pitfalls had already been detected, and the customer was able to provide prompt and thorough feedback. However, successful implementation of SCRUM left us wondering how we would incorporate the already existing testing process into the methodology, so now we share with you the results of our testing department’s experimentation with SCRUM.

»This case study is an apt illustration of how complex SCRUM implementation can be when the methodology is introduced into an existing work sequence, despite how easy it may sound in theory. Failed experiments, mistakes, misrepresentation, bureaucratic hurdles and other problems that pile up during SCRUM implementation are all a tricky obstacle course to conquer, and even though the folks from our testing department haven’t overcome it completely, they’re still making great progress. What counts is that they examine their results, come up with solutions, refine their methods and work hard on their attitude towards the product and their work in general with a shared objective in mind."

Alexey Pimenov, Agile Coach in ScrumTrek

The challenges

Our company’s priority area is mobile app development. As a result, the working conditions are sometimes far from perfect, with tight deadlines, lack of documentation (often the case with startups) and customers who change their requirements frequently and suddenly. This pits our testing department against a whole range of challenges, with shortage of time being the primary concern. While a large enterprise project brings with it a huge amount of requirements, it’s easier to deal with it than with a mobile startup as it provides pre-approved documentation and the schedule is not too limited. The scale of mobile development projects is usually much smaller, but due to the nature of mobile platform testing, distinctive requirements for mobile apps and frequent absence of technical documentation, the workload is comparable with larger projects, while the amount of time dedicated to testing is cut short.

This leads to the following issues with mobile app testing:

• The Terms of Reference or technical specifications are not put together as they get obsolete quickly. Consequently, it is impossible to make an up-to-date test case suite.

Requirements change on the fly, defeating the purpose of TOR/technical specifications and making them a waste of valuable product development time.

Communication and awareness issues within the team are also unheard of. For instance, a developer might ask the customer about a certain detail and overlook to share it with the team, or a tester might miss a Scrum during which important decisions have been made (but not documented due to a lack of time), and so on. As a result, the team might lack a clear idea of the project. This translates into the next problem on our list:

• Difficult entry into the project when its features have been partially implemented.

The work process is often organized in such a way that a build is handed over to testing only when the majority of features are complete. A tester that hasn’t been following the product’s development closely will burn through a lot of time getting to know the project before starting to actually test it.

• If the app interacts with other apps or services, testing difficulty is increased.

If your app supports (for example) calendar integration, uses custom-made chats, obtains and shares social network data or interacts with cloud services like Google Drive, testing difficulty will grow exponentially because of how diverse the app’s interactions are.

Implementation experience

The main problem of implementing testing in SCRUM was the testing routine that was present at the time, which was formed with long development stages in mind, after which the build was to be handed over to testing. It involved bug report acceptance, corrections, more testing, more corrections and so forth until the features that had been planned for this stage were fully implemented. However, SCRUM is designed around short Sprints that each end in a complete and workable product increment. Eventually, we established two scenarios.

Scenario 1. For our purpose, we’ll call it «Single test + fix check».

Key aspects:

• Daily meetings with the customer (who knew about the bugs) were held.
• At the start of each Sprint every developer on the team selected a pool of tasks from the Backlog, including the bugs they would fix.
• The tester was usually put to work somewhere in the middle of the Sprint. This was a misunderstanding of the SCRUM methodology and it led to the team not being always 100% involved in the entire Sprint.
• Discovered bugs were sent to the Backlog.
• After the start of a Sprint, no tasks would be added to the Backlog.
• The final acceptance by the customer was done based on the stage’s result and not the Sprint’s result.
• There was a PM on the team, but no PO or SM.

This way, the bugs found in a current Sprint were corrected only during the next. Nevertheless, because the customer was aware of the project status and the final acceptance was done only at the end of a stage, there was more time put aside for tests and fixes in the Sprint.

Advantages of Scenario 1:

• New tasks were not added to the Sprint;
• The customer could easily understand the reasons for moving stage completion dates;
• The last Sprints of a stage could be dedicated predominantly to fixing and testing.

This approach was very similar to the one that we used before the introduction of SCRUM, which allowed us to familiarize ourselves with the new methodology in a relatively safe way. The features were delivered to the customer fully tested by the end of a stage, leaving our hands (and conscience) clean.

Disadvantages of Scenario 1:

• The resulting approach was «ScrumBut», a cut-down version of SCRUM, especially in terms of team communication. The PM had a great extent of control over the processes, and the team (including the tester) worked under a manager’s supervision just like they did before.

Let’s illustrate this drawback with an actual case. Our tester was used to asking the manager for all kinds of clarifications, but for the manager it was more convenient to examine the project on their own account and send the build to the customer as quickly as possible, telling the customer about possible flaws in the product, instead of handing it to the tester who could reject the build and delay the build shipping date. Communication issues went unresolved and resulted in bugs that the tester would learn about virtually on the last day of the project. On the contrary, SCRUM declares that every team member is equally and willingly responsible for the product. Having learned this lesson, the tester would directly consult the customer afterwards.

• The bugs were fixed only by the next Sprint and sometimes even later.

We did not fully comprehend the SCRUM principle and mindlessly followed the «no new tasks in the current Sprint» rule. We took a similar attitude to bug fixing, eliminating critical bugs only by the next Sprint, putting off low-priority errors for later, up to a point where we would have an entire Sprint dedicated entirely to bug fixing at the end of a stage.

Occasionally the customer would like to test the product on their own, and the team wouldn’t have a fixed list of the tasks that were supposed to be completed in a given Sprint, except for the list of tasks for the stage. This led to the following events:

• The developers would pick a certain number of tasks;
• The completed tasks would then be handed over for testing;
• The tester would find the bugs;
• The developers would fix some of the bugs and pick up new tasks, as the remaining bugs were low priority and there was going to be a fixing / final testing stage anyway;
• The customer would see that several features were partially complete, start their own testing, discover bugs and get upset.

Don’t shoot yourself in the foot like we did and make it a rule to clearly designate tasks for a Sprint and eliminate critical bugs in the current Sprint.

This scenario resulted in a quality product, but the work process was not entirely SCRUM. The customer was somewhat discontented to discover bugs on their own, but was satisfied with the final product.

Breathing in and out, running right along.

Scenario 2. For our purposes, we’ll call it «Test flow».

Key aspects:

• Daily meetings with the PO who followed the project status were held;
• The tester received builds with partially finished features every day;
• All found bugs were sent into the Backlog;
• At the start of a Sprint the team would select a pool of tasks from the Sprint’s Backlog;
• No tasks were added into the Sprint after it started, but the blocker bugs were given higher priority and were fixed and checked in the same Sprint;
• The customer accepted the build at the end of a Sprint, and there was a fixed list (Sprint Backlog) that contained the tasks that absolutely had to be completed during the Sprint;
• In this scenario, the PM became a SCRUM-master (SM) and mostly maintained contact between the developers and the PO or different parts of the team. The SM also served to protect the team from excessive PO pressure, offered work process changes and so on. As a whole, the SM played the role of a facilitator;
• The PO received the build only after it had passed through the tester;
• A Sprint Retrospective was done at the end of each Sprint and a demo was prepared for the customer’s test group (Stakeholders).

This scenario was much closer to actual SCRUM, even though tasks were still sometimes added to Sprints. The duration of Sprints was also altered a few times.

Advantages of Scenario 2:

• New features were tested every day and the scope and complexity of work increased more gradually;
• Brief daily SCRUM meetings allowed the team to understand the product context better. This can be especially beneficial if the team is spread among several cities;
• Blocker bugs were eliminated as quickly as possible;
• The features were delivered as functional parts that each made attainment of the Sprint’s goal closer.

Productivity would indeed increase if all of these conditions were met. Every day at about the same time the tester received the builds and provided reports for the team to fix the critical issues if necessary. This allowed the tester to better plan their runtime and ensured that they wouldn’t be pulled out to work on a different project.

Disadvantages of Scenario 2:

• It wasn’t always possible for builds to be sent every day, which occasionally led to emergency work at the end of a Sprint;
• Adding tasks after the Sprint had been started could lead to delays. This happened when the developer had picked up a nice pace at the beginning of the Sprint, completed a good portion of tasks and decided to pick up even more tasks without fully finishing the ones at hand. It was almost inevitable that some unexpected difficulty would then show up to prevent meeting the deadline.

There is another side to this scenario: if a non-critical, non-blocker bug was discovered, the task it pertained to was marked as «complete» anyway. It was supplemented with a comment that described the bug or had a link to the bug tracker attached, and the PO examined it and made the final decision to accept or reject the task. It was somewhat awkward to knowingly supply the customer with a buggy build.

This scenario turned out to be a better approach, though it wasn’t perfect. To smooth out the rough edges, we set a few additional rules for ourselves.

Time for your second wind to kick in — we’re entering the homestretch!

Hindsight analysis

• Here are some patterns we established for ourselves when managing projects using SCRUM methodology. While we are still refining our practices, we hope you’ll find our methods handy:
• Be sure to pick a fixed number of tasks for a sprint.

The only kind of task that you can and should add to a Sprint is fixing blocker bugs. It is acceptable to add other tasks if and only if the initial tasks are fully complete, 100% tested and the features are absolutely proven to work as they are supposed to.

• Split large tasks into smaller ones and supply builds for testing as frequently as possible. This will save you from tester downtime at the start of a Sprint and emergency work at the end.
• Make several (three would be a good idea) build versions during the course of a Sprint.

Each of these builds has to be a product increment. If everything goes south after you add the latest feature and the solution threatens to be time-consuming and complicated, you can send the PO one of the previous builds, as at least one of them has a decent chance of being operational.

• If you are in mobile development, clearly define the list of devices that guarantee stable performance of your product.

You may successfully test your build on Nexus 5, Galaxy S7 and Xperia Z3 only to hear a less-than-flattering response after your customer tries to run it on their LG G3. The mobile device market (especially if we’re talking about Android) is very diverse. It’s immensely difficult to fine-tune your product for every device, so make sure you agree upon which devices you cater to.

• NEVER arrange the build delivery deadline for Friday, Saturday or Sunday.

It is common for bugs to crawl out at the last possible moment, so don’t condemn yourself to working on weekends and carrying around a bunch of test devices off work. This is sure to take its toll on the quality of your testing, so make sure that your Sprints end on a Tuesday or a Wednesday so that the pre-delivery finishing touches are made as part of your routine.

• Set up the bug tracker so that any created bugs go through your tester.
• This way, bug reports will be consistent and more transparent, and the tester will be aware of the project’s entire error situation.
• Critical decisions should be made collectively.
• If someone has to discuss a significant change with the PO, it is preferable for the entire team to take part so that there’s always an additional perspective. Moreover, the change is likely to influence the work process of the rest of the team, and if this change is discussed as a group, the team will have a better understanding of the project.
• Limit the amount of tasks in work that’s prepared for testing and checks.

We’ve also introduced a kanban-like element into our team’s work procedure. Sometimes, when the list of tasks for testing is growing, the developers won’t hand the build to the tester (because they want to complete as many tasks as possible before sending the build away); or there is a lot of tasks prepared for testing, but the tester won’t send the build over to the PO, wanting to present the PO with the largest possible part of the tested product. This has a negative impact on everyone: the tester has an emergency as there’s a lot of tasks to be tested but not much time until the end of the Sprint; the developers have an emergency as the tester has dumped an entire truckload of bugs at their doorstep and they also haven’t got much time until the Sprint ends; even the PO is in an emergency as the Sprint draws to its end and the team is still deep in it, fixing and testing. To prevent this, we decided that each team member should not have more than (for example) two tasks while the total amount of tasks that are tested or prepared for testing must not exceed (in this case) six if there are three developers on the team.

Problems that SCRUM solves

First and foremost, SCRUM helps if there is no documentation and facilitates communication within the team. The methodology helps clearly define and gradually assign the tasks and maintain ongoing contact with the customer’s representative, and generally makes it easier for everyone to get a grasp of the project. In the meanwhile, you can compile test documentation if you wish.

In SCRUM, testing begins right after development has started, which helps the tester to stay in the know from the very beginning. Abrupt changes in customer requirements are promptly discussed with the team and are accepted or rejected collectively because the PO stays in touch with the team and quickly gets the idea of what can and can’t be implemented.

Conclusion

You’ve crossed the finish line! While these methods can’t guarantee the successful completion of a project, they might at least help you implement SCRUM in your work practice. We hope you find our experience insightful.