Agile analytics<\/em><\/a> isn\u2019t a novel concept in any shape or form. Things like feedback loops<\/a> and process-oriented development<\/a> seem to integrate flawlessly into the analytics paradigm, at least on paper. Heck, there\u2019s even the Build-Measure<\/strong>-Learn<\/a> framework for continuous development. It would be difficult to argue that analytics doesn\u2019t have a role in something with measure<\/em> in the name!<\/p>\n However, past three years of working at Reaktor<\/a>, one of the world\u2019s top agile technology houses, have introduced me to a whole new set of problems with integrating an \u201canalytics mindset\u201d into an agile workflow, or an \u201cagile mindset\u201d into the analytics process.<\/p>\n The crux of the matter is how to negotiate the time-boxed methodology of a sprint<\/em> with the ever-fluctuating and contextual parameters of change<\/em>. To put it succintly: it\u2019s difficult to measure the impact<\/em> of development work in a way that would establish a robust feedback loop.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n One of the main things working against agile analytics is the very thing that fuels sprints<\/em> in general: the Definition of Done<\/strong> (DoD)<\/a>. In Scrum, one of the most popular agile frameworks, DoD is basically a checklist of things that each task\/feature\/sprint must pass in order for the sprint to be deemed a success. DoD has a lot of things working for it:<\/p>\n It\u2019s negotiated by everyone in the team.<\/p>\n<\/li>\n It\u2019s not monolithic.<\/p>\n<\/li>\n It\u2019s adjusted to better respond to an ever-changing business context.<\/p>\n<\/li>\n It quantifies the success of the agile workflow.<\/p>\n<\/li>\n<\/ul>\n In previous posts<\/a> and talks on the topic, I have recommended adding Analytics<\/strong> as part of the DoD in any development project. By having analytics<\/em> as a keyword in the DoD, the idea is that when developing features, the ever-important question of \u201cShould we measure this?”<\/strong> would always be considered. \u201cNo\u201d is a perfectly valid answer to this question – it doesn\u2019t make sense to measure everything<\/em>. But what\u2019s important is that this discussion is had in the first place.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Anyone working in analytics can share anecdotes of times when they wanted to check some usage statistics from the available data, only to discover that the necessary tags were never deployed to begin with. By adjusting the DoD, this can potentially be avoided.<\/p>\n However, having a Definition of Done<\/strong> has proven to be inadequate for establishing goals<\/em> fuelled and validated by measurement. Even though a sprint or a feature branch can be deemed done<\/strong> by the criteria established in the DoD, it still doesn\u2019t mean that the feature was a success<\/strong>.<\/p>\n \n \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n <\/span><\/p>\n It\u2019s tempting to think of an equivalence between something being done<\/strong> and something being completed<\/strong>. However, I argue that there\u2019s a distinction between the two. The first can be formalized with tools such as the Definition of Done. The second one is harder to pin down, because it requires consideration of what is successful<\/em>.<\/p>\n <\/p>\n <\/a><\/p>\n Yeah, I can do inspirational images, too!<\/span><\/p>\n<\/div>\n Consider the following story in a backlog:<\/p>\n Implement a single-page checkout flow.<\/p>\n<\/blockquote>\n A feature like this could have a very clear Definition of Done, and it would be very easy to validate if this feature can be shipped:<\/p>\n Test coverage has been updated to include the new code.<\/p>\n<\/li>\n Documentation has been written to cover how the new checkout works.<\/p>\n<\/li>\n Deployment to production is successful.<\/p>\n<\/li>\n The new checkout is measured in Google Analytics.<\/p>\n<\/li>\n The new checkout is deployed to 50% of visitors (A\/B test).<\/p>\n<\/li>\n<\/ul>\n The new checkout is done<\/em> when it passes these requirements. The team members working on the new checkout can look at these steps and plan their work accordingly.<\/p>\n But what determines if the new checkout flow is a success<\/strong>? It\u2019s a technical<\/em> success once deployed to production, but what actually validates that the new checkout brings added value to the organization?<\/p>\n A new set of criteria need to be devised for this purpose. These criteria need to take into account the following things:<\/p>\n Success is subjective<\/em><\/strong>. Different stakeholders might have different ideas for what qualifies as a success.<\/p>\n<\/li>\n Success is temporal<\/em><\/strong>. What is successful today, might not be successful tomorrow, especially in a constantly evolving and changing business landscape.<\/p>\n<\/li>\n Success is non-binary<\/em><\/strong>. It\u2019s not always possible to pass a true \/ false<\/strong> verdict for the success of a developed feature. Sometimes there are varying degrees<\/em> of success.<\/p>\n<\/li>\n<\/ul>\n Definition of Done is a great tool, because it guides us to do things right<\/em>. However, it lacks the scope to guide us in the prioritization of the tasks at hand. It lacks the ability to determine if we\u2019re doing the right things<\/em>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n A Definition of Success<\/strong> for the example in this chapter could be something like:<\/p>\n This migration is successful for the team<\/em> when the checkout flow has been deployed to production.<\/p>\n<\/li>\n This migration is successful for the client<\/em> when the A\/B test shows statistically significant results that the new checkout increases revenue per user.<\/p>\n<\/li>\n This migration is successful for the end user<\/em> when the new flow decreases checkout abandonment.<\/p>\n<\/li>\n<\/ul>\n If we take this thought experiment to its conclusion, we\u2019ll find that completion<\/strong> stems from the union of a feature being done<\/strong> and a feature being done successfully<\/strong>. It\u2019s possible for these two to overlap, such as in the success criterion for the team<\/em> above, and it\u2019s possible that success<\/em> is impossible to determine for certain features and some particular stakeholders.<\/p>\n Definition of Success is a series of questions you ask when prioritizing (grooming<\/em>) or adding new stories to the backlog. The questions are:<\/p>\n If this feature is developed, what determines whether it is a success<\/em> to the end user<\/strong>?<\/p>\n<\/li>\n If this feature is developed, what determines whether it is a success<\/em> to the project owner \/ client<\/strong>?<\/p>\n<\/li>\n If this feature is developed, what determines whether it is a success<\/em> to the team<\/strong>?<\/p>\n<\/li>\n<\/ul>\n By asking these questions, you are forced to think about the impact<\/em> of your project work, not just the outcome<\/em>. Optimally, impact<\/em> is something you can measure – something that you can use data and analytics to validate with. But it\u2019s perfectly fine to establish success criteria as something more ephemeral.<\/p>\n Let\u2019s take a look at some examples.<\/p>\n In the first example, establishing success criteria for something with such potential for impact as a new checkout flow is fairly easy. We can determine hard-and-fast metrics for success, with clearly defined goals which need to be surpassed for the feature to be a success.<\/p>\n A very technical task, such as migrating a cloud backend from one service partner to another can be more difficult to align with success criteria. How can the end user<\/em> validate success for the migration? Sometimes, it\u2019s perfectly fine to not have an answer for all three vectors. In fact, only when you can\u2019t find success criteria for a single<\/em> vector should you be concerned about the validity of the feature in the first place.<\/p>\n The final example shows that more than one of the three vectors can share the same success criteria. In this case, building a marketing dashboard improves the visibility of marketing efforts, and this is beneficial to the client as well as<\/em> the team that built the dashboard. The success criterion for the end user is again difficult to pin down. In the \u201clong run\u201d, more visibility to the current marketing efforts should result in better campaigns for the benefit of the end users as well.<\/p>\n In the end, we want to build valuable and value-adding things. We want the value of development work to be, at the very least, a sum of its parts. Optimally, we want to surpass<\/em> expectations and build features that collaborate to produce unexpected value in unexpected ways.<\/p>\n One thing that unites the two streams of completion<\/em> and success<\/em> is value<\/strong> in and of itself. A feature can be considered successfully done<\/em> if it produces a net increase in value to the parties that have a stake in the project.<\/p>\n But value<\/em> can be difficult to measure. It can be defined with the same terminology we use when talking about micro and macro conversions<\/a> in analytics. There is value that can be directly derived<\/em> from the development of a feature. Typically this would have a currency symbol in front of it. Then there is value that is indirectly inferred<\/em>, usually after a passage of time.<\/p>\n To refer to the examples earlier in this article, we could say that the new single-page checkout flow has direct value<\/em>, because we can measure its impact on the revenue generated by the site when compared to the old checkout flow (preferably in an A\/B test!).<\/p>\n Similarly, we can say that creating a new marketing dashboard has indirect value<\/em>, because we hope that by making the data more transparent we can help build better campaigns that will eventually increase the value of our marketing process.<\/p>\n Regardless of how you pin it down, you should always be able to describe good success criteria by using value<\/em> as a focal point.<\/p>\n Here\u2019s one final thought to wrap up this article.<\/p>\n I want to emphasize that Definition of Success<\/strong> is a communication tool rather than a set of strict validation criteria. It simply doesn\u2019t make sense to block development while waiting for some feature to produce enough usable data to determine what its impact was.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n![\"definition](\"https:\/\/www.simoahava.com\/images\/2018\/02\/definition-of-success.jpg#ZgotmplZ\")
<\/p>\n\n
![\"Definition](\"https:\/\/www.simoahava.com\/images\/2015\/11\/definition-of-done.jpg#ZgotmplZ\")
<\/p>\n
\n X<\/span><\/p>\n
\n The Simmer Newsletter<\/strong>\n <\/p>\nDoing the right things vs. doing things right<\/h2>\n
![\"Yeah,](\"https:\/\/www.simoahava.com\/images\/2018\/03\/done-complete.jpg#ZgotmplZ\")
<\/p>\n\n
\n
\n
![\"Example](\"https:\/\/www.simoahava.com\/images\/2018\/02\/success-examples.jpg#ZgotmplZ\")
<\/p>\n\n
Definition of Success<\/h2>\n
\n
\n\n
\n \nFeature<\/th>\n DoS (end user)<\/th>\n DoS (client)<\/th>\n DoS (team)<\/th>\n<\/tr>\n<\/thead>\n \n Single-page checkout<\/td>\n Drop<\/strong> in Checkout Abandonment<\/strong><\/td>\n Increase<\/strong> in Revenue per User<\/strong><\/td>\n Deployed successfully to production<\/td>\n<\/tr>\n \n Migrate from AWS to Google Cloud<\/td>\n ???<\/td>\n Reduced costs<\/strong> for pipeline management<\/td>\n More familiar tools available for pipeline management<\/td>\n<\/tr>\n \n Auto-complete site search<\/td>\n Find more relevant<\/strong> search results faster<\/strong><\/td>\n Increase<\/strong> in Revenue<\/strong> from site search<\/td>\n Improved performance<\/strong> of the search feature<\/td>\n<\/tr>\n \n Marketing dashboard<\/td>\n More relevant<\/strong> campaigns<\/td>\n More visibility to ROI<\/strong><\/td>\n More visibility to ROI<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Success === Value<\/h2>\n
Fuzzy success<\/h2>\n
![\"Validation](\"https:\/\/www.simoahava.com\/images\/2018\/03\/validation-vs-sprint.jpg#ZgotmplZ\")
<\/p>\n