be a digital-first business

Your 16-Point Litmus Test for the Right Agile Development Services Vendor

Thursday January 15, 2015, by Brillio

Finding the right agile development services vendor is a bit like getting a tattoo. You could find a great design but picking the right artist can make all the difference between ending up with an awe-inspiring dragon or a badly drawn garden lizard. In choosing agile development software vendors, as in life, we are our choices. To help you choose wisely, here are sixteen critical questions you must ask of your vendor.

1. QUESTION: How strong is the product knowledge of the team?

How familiar is the team on the domain of your product development/project? Do they know the underlying intricacies of the process/workflow involved?
WHY ASK: 75% of projects fail because the requirements and expectations of the end user are not fully understood. The team’s domain understanding is critical to ensuring that the stories are developed and implemented properly so that reworking can be kept to a minimum. 

2. QUESTION: What is their previous experience on similar projects executed on Agile?

Can the team provide case studies of agile projects with similar complexity in the relevant domain? How effective is the team’s release planning? Do they look at the bigger picture? Is there a possibility that they may mistake the trees for the woods?

WHY ASK: Experience in similar projects or domains can help the team reuse their learnings or components to enhance productivity and deliver a quality product at good speed.

3. QUESTION: What are the credentials of the Scrum Master and Scrum Coach?

Do the Scrum Master and Scrum Coach have demonstrable experience in leading Agile projects in the relevant domain? Is there a focused practice to foster the agile practices? How do they coach the team in times of difficulty and confusion?

WHY ASK: ‘Being’ agile is more important than ‘Doing’ agile. Many teams put too much into the ceremonies and events of agile, and end up missing out on the true essence of agility. Agile teams who want to be high performing teams should move from a fixed mindset to a growth mindset. Agile Coaches and Scrum Masters can enable this transformation by being involved and providing the necessary coaching and guidance.

4. QUESTION: How do they implement governance and reporting? 

What kind of reports/dashboards are going to be generated? How would governance meetings be conducted? Would they be plain status reports or would they contain insights and learning from the retrospectives?  Request for sample reports to evaluate the team’s innovative spirit.

WHY ASK: There are plenty of application lifecycle management and software development tools in the market which can help teams speed up their activities of requirements management, defect management, development, testing and deployment. Each tool has its own interface and reporting.  Teams must orchestrate the smooth integration of these tool sets to derive meaningful, insightful reporting so that they can get the full picture and continuously improve their performance.

5. QUESTION: How disciplined and effective are the Daily Scrum meetings?  

How does the team plan to structure the daily standup? How do they avoid unnecessary discussions and wasteful conversations? How does the team focus on identifying daily hurdles and having these obstacles removed quickly by the Scrum Master?

WHY ASK: Wasteful conversations derail the true purpose of the Daily Scrum meetings – to get updates on the status of the work, to clear obstacles and offer support to team members in resolving any issues with completing their commitments for the sprint. Many teams do not fully understand the agile principles and practices, and end up conducting the ceremony ineffectively – without the right information, progress can be slow and prone to excesses. 

6. QUESTION: How does the team manage requirement maturity?

How would the team go about evaluating the Product backlog items? How do they determine priorities and ensure the required level of detail for completing the business and technical design?

WHY ASK: Product Backlog is a key artefact in Scrum. The PBI(Product Backlog Items) should be kept up-to-date and elaborate. This is important for requirements management and the effective delivery of sprint goals.

7. QUESTION: What is the team’s architectural view (systems approach)?

How would the team manage higher level architecture requirements? How would the team manage any changes required to system components to meet business requirements?

WHY ASK: In agile development, requirements evolve fast, and are unclear at the outset. That’s why the team needs to take a holistic systems approach to architecture. They need to build in a loosely coupled and robust way so that changes can be accommodated without compromising the architectural stability and the non-functional behavior of the system. 

8. QUESTION: What is the team’s approach to Sprint Planning and Preparation? 

Does the team plan to complete a high-level system design sprint before starting on the delivery sprints?

WHY ASK: Agile development is evolving and iterative. The team plans for a sprint on the first day of the sprint and works towards delivering it in a short iteration of 2-4 weeks, as promised to the customer. Without sufficient details, dependency identification, and resolution in the sprint planning, the sprint outcome is at risk. 

9. QUESTION: How does the team handle change management during a sprint?

What happens if there is a request for change in the requirements within a sprint? While planning the sprint, does the team insist on getting a commitment to control changes from the product owner and other stakeholders?

WHY ASK:  As per the Scrum, there should be NO CHANGE during the Sprint. This is not possible unless the team can collaborate effectively with the customer to fully understand the priorities.

While some teams do this well, some teams can flounder due to miscommunications and misunderstood expectations. Mature teams follow a highly disciplined Scrum with an agreed process for Change Management with the customer.

10. QUESTION: How does the team execute integration communications, testing and release management?

How does the team plan to communicate with larger stakeholders to ensure that there is enough integration testing and no blockades during major releases?  What about common components and integration testing with other modules?

WHY ASK:  While following scrum and agile delivery, teams should not lose track of strong engineering practices. It may not be possible to have unit testing and reviews as elaborate as those in a waterfall cycle. However, there should be sufficient engineering checks and balances in place to ensure that the product delivered is of the best quality. 

11. QUESTION: How does the team review effectiveness?

How does the team manage design reviews, code review and unit testing? How do they limit defects during daily builds and final integration testing?

WHY ASK:   The Agile manifesto says “Individuals and Interactions over Process and Tools”. This should not be misinterpreted to affect the discipline of process and tools. It could lower the quality of the deliverables. Teams need to know how to strike this delicate balance in a quick sprint cycle.   

12. QUESTION: How does the team manage Testing Strategy and Test Automation?

What is the testing strategy of team?  In every sprint, how do they ensure that there is sufficient testing before the sprint demo so that functional defects are minimized?

WHY ASK: It is common to see immature delivery teams crunching their testing time to accommodate the shunt effect of development extending beyond the planned time. This leads to insufficient testing and a bug-ridden product. The team needs to know how to manage their requirements in a fast-paced agile delivery cycle. Do they plan for automation from the beginning? How do they ensure that the test cases and automation scripts are current at all times? All these factors become important for high-quality, sustained agile delivery. 

13. QUESTION: What is the team’s approach to estimation and predictability?

What methodology does the team use for software size estimation? The team will need to ensure sufficient bandwidth to come through on the sprint delivery commitments – to this end, how is capacity planning factored into sprint planning?

WHY ASK: Requirements may not always be clear. Estimation, in such cases, can prove to be a real challenge for agile teams. Many teams use relative estimations, such as story points, to overcome this limitation. As the requirements evolve, the story point estimations need to be enhanced to form task-based plans called sprint backlogs. Over a few sprints, teams mature enough to arrive at a predictable velocity. This helps capacity planning to meet the velocity required to achieve the release commitments. 

14. QUESTION: How does the team manage architecture reviews and system integration?

Is there an adequate support system of experienced designers/architects for periodic architecture reviews? Can this support system identify intermediary integration components if there are multiple teams involved?

WHY ASK: If the product has to be delivered by multiple scrum teams, integration can become a challenge. Successful systems integration and quick delivery are the result of proper sprint planning and dependency management. Successful deliveries happen thanks to the appropriate architecture for such integration and testing. Again, planning is critical.

15. QUESTION: Can they scale? 

As business requirements evolve, does the vendor organization have the capability and capacity to scale to larger teams?

WHY ASK: Often, the required product cannot be built by a single, collocated team. This could be due to the unavailability of required skills, existing outsourcing agreements, and distributed teams spanning multiple vendors. In these situations, your vendor must have the capability to scale at short notice. They need to be able to mobilize the required human capital at the right time with the right skills. 

16. QUESTION: What is the team’s approach to retrospectives and continuous improvement?

The team needs to inspect, adapt and continuously improve – does the team have a mechanism in place for effective retrospectives? How does the team measure velocity and set higher targets for improvements?

WHY ASK:  Agile is an iterative process. Inspection and Adoption are at the core of any agile team. The Retrospective is a Scrum ceremony which enables this ‘Inspect and adapt’ process for teams to review their work at the end of every sprint. This allows them to take the required corrective actions to follow a continuous path of improvement. Teams need to be able do this with ease and be highly adaptive to be suitable for agile delivery.

Share Post