I have recently been working with a new client on a greenfield project, where we are a small engineering team (2 Senior devs + 1 part-time QA/PM + 1 part-time Product Strategy) tasked with building a working / viable product in less than 6 months. While our primary goal is to deliver this particular software product, we are also tasked with helping the company improve and learn better ways of doing software development.

We started with chartering, introduced evolutionary design, story mapping, product strategy and ensemble programming. This way of working was entirely new to them, and what we were suggesting seemed to be the opposite of how they had been approaching their work. Up to this point the group had spent a year or so gathering requirements and building a functional prototype. In an effort to help them improve and learn our approach, we invited their developers and testers to join the ensemble as their time allowed. Initially, we had 1 developer join, then a 2nd one.

We completed our first milestone and had a working production version of the application (a walking skeleton). It was ugly and didn’t really do anything exciting but if you squinted you could see where the product was going. We could output the files needed and in a format that the first client could consume, and we verified that with them. It was something we could build on and continuously test; although, the testers didn’t really find any interest in the application yet. Our next milestone was to start making the application functional, where users could actually build something valuable, validate it, and export it.

The Problem

Once we started releasing changes that had a UI, it started getting the tester’s attention. A tester came out of the woodwork, informing the ensemble of “defects” via slack. This caused a bit of disruption with the ensemble. They were stopping what they were doing to look at what was posted in slack (context switching), trying to figure out what the tester was talking about and trying to figure out if this was an actual defect via back and forth messages in slack.

Most of the “defects” being reported were UX things that hadn’t been discussed yet as part of evolutionary design (make it work, make it right, make it fast/pretty), and were assumptions or expectations of the tester. This started happening frequently and was really distracting, and sometimes frustrating, the ensemble. Other than the existing prototype we did not have any official UX designs for the application. We had the UX person in the ensemble who built the prototype, so we had immediate access to our designer and there wasn’t a need for fancy designs. We could build just enough and then elicit user feedback before creating a complex UI.

Evolutionary Design of Guitar

The ensemble was chasing down things that were entirely irrelevant to the current work or were things the tester wanted from their perspective that was not in alignment with product strategy. We started to see a huge waste snake building in front of us that we needed to address quickly to keep us moving forward.

The problems we needed to address quickly:

1. Stop tossing “defects” over the wall
2. Everyone has the same expectations for the work being done

Since we were in the middle of working towards a milestone and needed to get that completed, we couldn’t stop immediately and have a long discussion about this. We also weren’t sure yet how open the tester would be to that type of discussion. What we were able to do immediately is build some trust with the tester. We invited them into the conversation and to join the ensemble to discuss what they were finding. These conversations were time boxed and learnings added to existing stories. Space was given on our mural board for the tester to add notes about what they were finding that the ensemble team could have a conversation about when time allotted. The tester agreed, and we started to see good things happening within the conversations. They also started joining the story mapping sessions and other meetings we had with the product team.

While this was an improvement, it was still a nagging distraction that the ensemble was pulled towards. We still needed to have a bigger discussion on a better way to collaborate.

Collaborating on a Solution

During the course of revisiting our charter for this work (specifically around working agreements), it came up that we needed to have a discussion about bugs/defects and agree on how we wanted to handle them. About a week later, we took a couple of hours and had a good discussion about bugs/defects with the entire ensemble (including our tester). In that conversation there developed an important distinction that the group agreed upon:

1. What should the system do NOW?
2. What should the system do eventually?

Because we were evolving the system, there were things that we knew the system didn’t do that eventually it should, but to spend our time on testing for future things wasn’t worth it. We agreed it was most important to focus on what we are expecting it to do now and preserve the existing expected behavior. The good news is that the ensemble uses test first development and has a number of tests that can be used as feedback around existing behavior. The majority of these are unit tests, but there is also a significant number of integration tests and a handful of UI tests. This gives the ensemble quick feedback on their work regularly.

Testing Pyramid

This led us into another great discussion around defining a bug/defect. In the discussion we discovered there were several categories of things that could be found. The most important one being: “A behavior that surprises the ensemble and prevents the user from doing what they need to do.” We defined this as a blocker (by definition a defect) and was to be fixed immediately as part of our zero defect policy.

So what about all the other things that are found? As we talked about those we realized there were some other categories of issues that we needed to address:

• Defect (by definition a bug) - This was something that needed to be fixed but there was a workaround that the user could use. This would be worked on as soon as the current story was completed and before starting another piece of work.
• Improvements - These were things we didn’t like about the UI or flow/navigation of the system but did not really need to be addressed right now. We could get user feedback and incorporate it into later stories. We would delete these and not keep them around, as they will come out in our user feedback or the problem would reoccur if it was that important.
• Out of our control - These were things that other teams had to fix that we didn’t have control over. We needed to have a conversation with those teams on how to make sure that these were prioritized and communication on the work was happening.
• Investigation - These were things we didn’t quite understand and needed investigation such as looking at logs or getting more details on how to reproduce the issue.

Once we had that defined, we discussed the process; most importantly, how do we prevent context switching, distractions and find these issues as early as possible when they are cheap to fix. Based on the current company culture and the testers experience, our tester was used to doing things in a silo. Testing separately from the developers, reporting issues, filing tickets, retesting when those tickets were marked as resolved. This resulted in misunderstandings of the system expectations, late detection, expensive fixes (sometimes multiple fixes for the same thing). It was really important to the ensemble that everyone understood the expectation of the current story being worked on.

Since our tester was already joining the ensemble, there were a few wins we could implement quickly:

• Triage time to have a conversation about the things that have been discovered and categorize based upon what we defined previously.
  
• Exploratory testing as part of ensemble work When the tester is in the ensemble, they would be part of the rotation. Instead of driving/navigation writing code, they would ask questions or navigate a developer to test a part of the system related to the story that was being worked on.
  

The Outcome

Since introducing this change, the ensemble is working much smoother and addressing things found by the tester quickly. Through conversations, we are able to build a common understanding of what the system should do at this moment in time. The tester has helped bring up scenarios where things might not make sense or might not be intuitive to the user. The interaction with the tester has now changed from “I found all these bugs, let me know when they are fixed or if you have any questions” to “I noticed this behavior since the last deployment and I have a few questions, can we discuss?” In the near future we hope the tester will be even more engaged in the discussion of the work to be done and helping uncover potential bugs/defects before the ensemble starts writing code.