Scaling Scrum

Scrum is a wonderful framework to manage projects. It is usually talked about in terms of a single project team and how to manage product and sprint backlog for the team. But most of the times, a team will not exist in isolation. It is part of a bigger structure, and mostly the product is being developed by multiple teams who are focusing on different aspects.

Hence, it is important to understand how the Scrum framework should scale to accommodate bigger teams and projects. Here are a few techniques to scale scrum

Scrum of Scrums: This is a very basic practice to manage multiple teams following the Scrum framework. Each team sends a representative to a daily Scrum of Scrums exercise, where each representative will talk about the health of his project. This practice helps everyone to have a bigger picture, manage dependencies, call out blockers and everyone is on the same page as to when and how the next release of the product is taking shape.

Scaled Agile Framework or SAFe: SAFe is an extension of the Scrum framework for managing bigger projects. It introduces the concept of following additional teams and members

    System teams – Dev ops people shared across teams and responsible for deployments- CI and CD are created at project and product level.
    Architecture Teams- Architects that own the design for the overall system and each project team. It is important that everyone in this team understand the overall design and dependencies on other components.
    Product Managers – manages a team of product owners for different teams.
    Release Train Engineers- Scrum masters form the team of Release train engineers and manage the overall release process.

ART or Agile Release Train clubs different aspects which are dependent and can be released together.

Disciplined Agile Delivery or DAD: Disciplined Agile Delivery is the response to the teams which claim to not care about some of the core best practices in the name of Agile. If you have been part of Agile teams, you might have heard people saying that we don’t spend time on design and documentation as we are following Agile. This can cause issues especially with large projects where multiple teams are dependent on each other.

To handle the situation, an induction phase is added at the start of the project. During this phase many important things happen to help scale, like designing the solution, POCs are done, high-level architectures are created and shared across teams, release planning and roadmaps are established. After this, the normal Scrum process is followed.

Understanding the Scrum Framework

We are hearing a lot of noise about Agile project development techniques and more often Scrum-based development these days. Here I will try to answer some of the basic project management questions related to Scrum.

What is Agile?

Agile, if we look at the word only, it means fast. With the competition and fast-paced advancements in technology, it is important for software products to release features as fast as possible to the market.

With that aim in mind, the following core principles were finalized.

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

What is a project charter?

A project charter is a short document which gives you an outline of the project. It normally has the following details.

Project Objectives – What is the trigger for this project.
Stakeholders- Who all are affected by and involved in this project
Constraints- Dependencies on outside factors.
Risks- Any Risks foreseen for the project and if they have been mitigated or accepted.
Definition of Done – When can the project be considered as done

What are the basic roles in Scrum?

Product Owner- manages and prioritizes the product backlog. Should have complete knowledge of the product. It helps if he also has technical knowledge.

Scrum Master – The job includes facilitating the scrum, organizing meetings, communicating with stakeholders, product owners, and team to make sure everyone is on the same page. Usually, this is not a standalone job. Normally a Project Manager will assume the role of Scrum master who might ave additional responsibility of managing resources and keeps a check on project overall health. At times a business analyst can also take a role of Scrum Master, who has an understanding of business and can take up some responsibility from product owner if the later is not available full time.

Development Team- A typical scrum development team can have following team members
Developer
Business Analysts
Tester
Technical Writer
Architect
Support or DevOps people

How the work is managed in Scrum?

Work is managed in the form of backlogs. There are a product backlog and a sprint backlog, which contains work items in the form of stories.

Product Backlog: The product backlog has all the work related to project. The product owner works with stakeholder to come up with the product backlog, which is then prioritized and mostly all the work is delivered in the form of multiple releases.

Sprint Backlog: A spring is usually a 2-3 week time period. Product owner normally prioritizes work for each sprint, which is then added in form of Sprint backlog.

What is the Scrum process?

The product owner creates product backlog with the help of stakeholders. This usually has all the work which has to be taken up during the course of the project. Work is done and delivered in the form of Sprints. A sprint is usually a 2-3 week time period, in which a small but complete chunk of work is picked up by the team and completed.

The process starts with spring planning where tasks to be delivered for the current sprint are finalized. Daily scrum meetings are conducted where ever team member provides the status of work done. At the end of Scrum, a review and a retrospective meeting take place to showcase the progress and evaluate process improvements respectively.

What is Sprint Planning?

In the sprint planning meeting, the Product backlog is reviewed by the team. The product owner prioritizes the stories to be taken up for the sprint. Team reviews, discusses and refines the stories. Finally, Team commits to a set of stories, which then is treated as sprint backlog.

One should Avoid long discussion on requirements and design in planning meetings. If things are not clear, the story should be dropped and re-reviewed by the product owner or architects / senior devs separately. Ideally, there should be a pre-planning meeting among senior devs, product owners and scrum master to save the time of the team.

Pick stories which make logical sense in delivering an incremental update.

What is Planning Poker?

During the sprint planning meeting, the complexity of all the stories being considered for the sprint is decided. Based on which the final sprint backlog is decided, considering how many stories can be taken up during the sprint. Knowledge from previous sprints is used to understand how much complexity can be handled during the sprint.

Planning poker is a process to involve all team members in evaluating the complexity of a story. A base story with a base complexity, called story point is used for comparison. Every team member gives complexity or story point (normally in Fibonacci series) to each story. Outliers are discussed and the team comes on a consensus for story points.

It helps everyone understand the story and make sure everyone is one same page. Every voice is understand.

What is Daily Scrum?
A daily standup meeting where each team member talks about the following three items

    What was done yesterday?
    What will be done today?
    Any blockers

Advantage of daily scrum meeting is that everyone knows what is happening through the project, and team members can help each other or call out dependencies on other’s work.

Usually, a Kanban board or Scrumban board is used to track stories and tasks. This helps in making sure that only a fixed set of stories will be taken up at a point. There can be 3 or more states for a story or a task i.e. New->In Progress-> Done.

What is the Sprint Review?

At the end of a sprint, the team showcases the progress to the Product Owner and other stakeholders. The progress is shown in the form of a working product or demos. One should make sure that there is representation from all stakeholder parties so that everyone agrees to the product being built.

What is the Sprint Retrospective meeting?

This meeting is among team members at the end of a sprint where everyone is supposed to discuss on following items for the last sprint.
What went well?
What can be improved?
How can we improve it?
A plan is created for improvements and tracked.

What goes into a sprint?
Sprint Backlog

What is the outcome of a sprint?
A Potentially shippable iteration of the product. That means designed, developed, tested, unit tested, integrated, acceptance tested piece of code.

What is a user story?

A user story is usually the smallest yet complete work item which can be delivered in entirety. The story can be further divided into tasks. A story has the following components – Value statement, Assumptions and Acceptance Criteria.

A value statement is normally stated in the form of
As a [Who]
I want to [What]
In order to [Why]
For example, “As an Admin, I want to access the settings page, in order to update language option.

Assumptions are any additional information about the story implementation as if it is dependent on another story or other stories are dependent on this, how these will affect each other.
For example, “Assuming external service is ready or takes {set of input} params.”

Acceptance Criteria is testing criteria of the story. This also covers any nonfunctional requirements like performance or security needs. Acceptance Criteria removes ambiguity and provides the sort of a checklist for completion of the story.

In addition, a story will also be associated with Definition of Done, which is defined at the product level. This can include –
Reviews by stakeholders
Documentation
Design and compliance reviews

Tripple cost constraint

The age-old knowledge we have that when building a product, you can choose two of the three – Cheap, Good and Fast.

So if you need something to built with good quality and fast speed, it won’t come cheap.

In terms of a software project, these constraints take the form of –

Scope
Budget
Schedule

Quite often, a project manager is caught in the dilemma of controlling these aspects of the project. Striking a good balance among three is often difficult.

Let’s take a look at some of the practices we can use to manage these three aspects.


Scope

In a traditional project management approach, you would use Work breakdown structure (WBS), to define feature components which we need to develop. If a change comes in at a later point of time, that has to go through a Change Control Board (CCB), which will analyze the impact and approve if required.

In a lean project management approach, the scope is controlled in the form of tickets and requests.

In the more popular agile project management approach, we control scope in terms of the product backlog and sprint backlog.


Schedule

To manage the schedule in a traditional waterfall approach, techniques like Program Evaluation and Review Technique (PERT) and Critical Path Method (CPM) are used

In the Lean project management approach, Kanban & Queues are used to manage the work. The work is managed in a list and executed based on priority. Service Agreements sets the priority of work by defining what is critical, major, or minor.

In an Agile project management approach, a Sprint based model is popular to manage scope, where releases and roadmaps are used to set goals for major features to be released together.


Budget

In the traditional project management approach, we use Earned Value Management (EVM) approach which compares current performance Earned Value (EV) to the Planned Value (PV) and evaluate it over Actual Cost (AC) which is the cost so far to complete the work

In the Lean project management approach, we control KPIs or Key performance indicators to showcase the outcome of the work done.

In Agile project management, we measure Return on Investment or ROI, and Burndown charts to measure performance.

Cloud Computing – an Introduction

When I started my career, analyzing and finalizing hardware needs for deployments was a major task and had to be taken up months before actual production deployments. Hardware was costly. Though we had providers which would provide machine virtually, you would need to decide the requirements beforehand as once you procured a machine, you had to pay at least a month’s rent. And if you decide to upgrade or downgrade the server machine, it was a painful manual task.

Just imagine what a nightmare it would have been to scale up during a surge in requests. You had to foresee it, plan for it, arrange hardware for it (monthly rents).

With the cloud, things have changed for the better. You have a pay as you go model, so you actually pay for the usage of hardware only. With autoscaling features inbuilt into the cloud infrastructure, it is easy to increase or decrease compute power without any human intervention. Setting up databases and scaling them is another area which the cloud takes care for us. Most of the cloud service providers support both relational and NoSQL databases in an easy to use manner.

Security, access management, monitoring, encryption, and storage are some of the other services which are provided by cloud services providers of the shelf. Another popular set of services off-late is serverless compute. This means one can write code directly which can be run as functions on the cloud, without worrying about the deployment details completely. Cloud provider is responsible for scaling and maintaining such functions. This is in sync with the microservice approach where each function can behave as an independent microservice.

With one’s mind taken away from hardware details, it is easier for software engineers to focus on building quality products. But it is important that we design our products in a manner which are capable of taking advantage of cloud services. For example, it will be easier for a microservices-based application to autoscale in a cloud than a monolith application. A stateless service is easier to be deployed and scaled on cloud than stateful service. One still needs to take care of the fact which services are exposed on the internet and which should be exposed only to internal service. With the ease of deployment, it might be easier to mess up a running service, so proper automated and manual checks are required to be implemented.

Cloud, though makes things easier, but one needs to be cautious of using its capabilities and designing the system in a manner to make maximum use of services being provided

Developing With Containers

In the last few years, the cloud has completely changed the way we used to think about and design applications. With our eyes taken away from hardware availability, we can think more in terms of scalability and reliability. This has given a rise to microservices based design. Containers further support our idea of microservices based implementations by providing another layer of hardware independence.

What is a container?

Let’s start from the beginning. a decade or so back, when you need to deploy an application, you would go to your hardware department and tell them to get you a new box. So if you are deploying 10 applications, you would end up having 10 different sets of hardware being managed. In all probability, you were using less than half the processing power of a machine your application was deployed on because you would not take a risk of running a production application on a low powered machine.

Then came the era of Virtual machines, which helped us installing multiple virtual machines on the same machine. With the popularity of the cloud, virtual machines got more popular. Now you could easily boot up a virtual machine and deploy your application on that. One pressing downside of a virtual machine is that if you have 4 virtual machines running on a box, that would mean you have an overhead of running 4 operating system which ends up eating a lot of processing power.

This gave rise to the thought process for Containers, which helps us run an application inside its own container rather than an independent physical or virtual machine.

Let’s look at what a container is and how is it different from a VM with help of Docker container technology.

Image source – https://www.docker.com/

What is Docker?

Docker the technology, created by a company namesake, gives us tools to implement the concept of containers. As we can see in the image above, in case of a virtual machine, every virtual box has its independent Operating System image. That would mean that each Virtual machine is somewhat an independent system in itself. On the contrary, Docker installs a Docker engine on top of OS, which can further run multiple containers with a different set of applications.

Here are some useful docker commands

docker run
— Run an image
Docker ps
— Lists down all the running containers
Docker ps -a
— Lists down all the containers
Docker images
— Lists down all the images

More docker commands.
https://docs.docker.com/engine/reference/commandline/docker/

What is Kubernetes?

Running your application using Docker containers is the job half done. In the modern world, you need to take care of problems like how your application will handle the increase or decrease of load? How will we handle a situation when a node is down? Kuberenetes answers these questions helping us orchestrating the nodes and applications. It keeps a check on load and helps us increase or decrease firepower or add/ remove application instances. It takes care of failing nodes and brings in more nodes if required.

Spring Cloud

With more and more applications moving to the cloud, it is important for developers and architects to start thinking in terms of applications which are ready for cloud deployment. Spring cloud in an umbrella which consists of a set of spring projects that can help us create applications which are cloud ready. The advantage of using a spring cloud is that you get a solution to most of the challenges you face when you deploy on the cloud at one place.

Let’s look at some of the core tools provided by spring cloud.

Spring Config: An important challenge distributed applications poses is how to manage your configurations. Each application and service need to manage some configuration properties. Spring config server provides a centralized way to manage configurations. It can read configuration from multiple sources, by default it would expect the configuration to be available at Git, but this can be modified. The most important factor of about using spring configuration server is that you can refresh properties or configuration in services without the need for redeployment or restarting the services.

Discovery service- Eureka: Spring cloud borrows a lot of services from Netflix open sourced projects. Eureka is one such service from Netflix, which provides a discovery mechanism for services deployed, Spring provides seamless integration to Eureka. With a design supporting deployment and autoscaling fo microservices, this becomes important the services can be deployed and removed on the fly without manual interaction. A challenge in this approach is how will the calling services know about the location or URL for the services to be called. A discovery service mechanism provided by Eureka comes to help. Any new service getting added to the system will register itself with Eureka server. A service that needs to access this remote service will communicate with the Eureka server and ask for the service location. Eureka server returns the location of the service, which is then accessed by the client service.

Fault Tolerance with Hytrix: Another important factor one needs to be worried about is how failure is managed in an application. In a distributed system, where multiple services are calling each other, will the failure in a service bring down the whole system? Or is there a way we can contain the failure to a single service. Spring integrates with hystrix, which again is a project by Netflix, that provides us mechanism like a circuit breaker to manage failures in an application. For example, if a remote service is down or not responding, Hystrix helps us configure an alternate service or method which can take over. Also, we can add settings that after N number of failures, client service will stop calling remote service (the circuit is open), and set a timer when next retry happens (if the service has healed itself, operations continue as usual).

Routing with Zuul: With multiple services being part of the system, it becomes difficult for client service to keep a track of all the remote services. A routing gateway in such an environment is like a front gate, behind which all the services are available. A client system need not be aware of details of deployment of each service, and it will just request routing server to get the required information, which in turn will communicate with remote service and returns a response. The advantage of this approach is we can implement security, logging, auditing of incoming requests in a centralized manner and each individual service need not worry about these factors.

Load Balancing with Ribbon: Load balancing is an important feature in order to manage autoscaling of the services. If there are 10 instances of a service, we want to make sure each of the instances manages almost equal load to take advantage of scale. Load balancing can be implemented at the server level for which various cloud service provider already give us a certain option. But there can be cases when you want to implement Load balancing at the client side. For this, Spring integrates with Ribbon project from Netflix and provide us seamless load balancing with the help of annotations like @LoadBalance and @RibbonClient.

Additional Spring Cloud projects: As already mentioned spring cloud is an umbrella which contains a number of projects that help us make our application cloud ready. The number of projects under this umbrella keeps on increasing with an increase in the popularity of cloud-based deployment and distributed applications. For a detailed view of all the projects under spring cloud, one can visit and explore projects under https://spring.io/projects/spring-cloud

Top Free BI Tools

Here are some old notes from my analysis of the comparison of leading free BI tools.

1. Pentaho

Pros:
Can be considered as a complete BI tool
The learning curve is short
Has both paid and community edition.

Cons:
Community edition has some restrictions, so one needs to look carefully if looking for a free solution.

2. Talend

Pros:
Has support for charts and graphs
Has support for advanced features like big data

Cons:
Slower than Pentaho
The learning curve is more than Pentaho

3. Birst

Pros-
Can connect to Apps and Big data

Cons-
Does not provide too many details is documentation
Does not provide a free version
Looks interesting but does not open up too many details. If one is going for a paid solution, one can explore other options like Tibco as well.

4. Dynamic Reports

Pros:
Uses Jasper Report API to develop
One can write Java based code and use libraries to provide graphics

Cons:
Too much coding Required
It is like writing code to generate reports
Reports will be static (based on code).

5. Spago World / Spago BI

Pros: Completely open source solution

Cons: Community support does not look too active. One can get stuck with issues.

6. JasperSoft

Pros: Better quality graphics
Provides mobile support

Cons:
Lesser functionality than competitors like Pentaho.
It is suggested to use more in terms of reporting rather than a complete BI solution.

7. BIRT

Pros: Eclipse-based, hence low learning curve to get started
hosted reporting

Cons:
Looks more of ETL tool rather than a complete BI solution

Building Reactive Systems

In today’s world, we are always striving for building applications which can adapt to constantly changing needs. We want our systems to flexible, resilient, scalable and withstand end user’s high expectations.

Considering these needs, a Reactive Manifesto was put together, with best practices which will help us build robust applications. Following four pillars makes a strong base of reactive application.

1. Responsive
2. Resilient
3. Elastic
4. Message Driven

You can see none of these concepts are new in nature, and you might be already implementing them in the applications you build. Reactive Manifesto brings them under one umbrella and emphasizes their importance. Let’s take a look at these pillars one by one and see what all well-known patterns we can use to implement each of them.

Responsive: An application is responsive if it responds to the user in a timely manner. A very simple example is you clicked on a button or link in a web application, it does not give you a feedback that button was clicked and the action gets completed after few seconds. Such an application is non-responsive as the user is left guessing if he is performing the right action.

Some of the well-known practices and design patterns which help us make sure if the application is responsive
– Asynchronous communication
– Caching
– Fanout and quickest reply pattern
– Fail-fast pattern

Resilience: An application is called resilient if it can handle failure conditions in a graceful manner.

Some of the patterns that help maintain resilience
– Circuit breaker pattern
– Failure handling pattern
– Bounded Queue Pattern
– Bulkhead Pattern

Elasticity: An application is called elastic if it can stand increase or decrease in load without any major impact on overall working and performance.

Some of the practices and patterns to implement elasticity
– Single responsibility
– Statelessness
– Autoscaling
– Self-containment

Message Driven: An application which uses message driven communication makes sure we are implementing various components and services in a loosely coupled manner. This helps us keep our components scalable and makes failure handling easy.

Practices used to implement Message driven communicaiton
– Event driven
– Publisher Subscriber pattern
– Idempotency pattern

I have covered some of these patterns in details in my book on Design Patterns and Best Practices.

Design before you code

Somehow, with the penetration of agile development practices, I have been observing that less and less time is spent on designing the solution. Engineers treat Agile as a license to develop without design.

What is the problem with developing without designing or architecting the system first? I still remember when I was in college, my professor drew a parallel between architecting a building and architecting a software. Would you start building a house without thinking about design? you will not just start placing bricks without considering how many rooms you need? Where all these rooms go? How large should be every room? Where will the kitchen and bathroom be? Where all the wiring and plumbing will go? You will come up with the design, put in on a paper or a software and then calculate the feasibility.

Think of the complications that can happen if you build your house without thinking about design first. One room might be so big that there is very little space for the others. Or you are not left with any space to create a bathroom. These mistakes are costly. And that brings me to the other important factor which is causes ignoring of design in software. The manager thinks that change at a later state is going to be easy. At the end it is code, we can just change it. Well, changing the code is definitely possible, but never easy or cheap. It comes with its own cost.

More than often, where developers need to make changes at a later stage, they would be more inclined to apply quick fixes or hacks rather than making a bigger correct change. Of course, you have a tried and tested code in hand, why would you make too many changes to it, even if you know that is the right thing. And then there is developer ego, I mean, let’s admit it, it is not easy to accept one’s fault, especially if that means you would need to put in extra hours to fix that. More than ego, it is actually denial at times. People would try to stick to the solution they developed initially even though it is realized at a later stage that there could have been a better solution. Reason being, you have already invested too much.

This situation can be avoided if we think about design first. We can make sure our design covers all the possible requirements. It will not be possible to anticipate all the changes that are going to come at a later stage, but we can try to keep our design flexible. The idea is, set the ground rules, understand what all component and services we are required to create? What all data needs to be persisted and how are we going to do that? How security and error handling will be done? We need not get into implementation details, but the high-level design is a good start. We can fill in the details as and when requirements are clearer and we start actual work on the given piece of requirement.

In the end, all I would like to highlight is, if you think you are saving time by not thinking about the design of the application before you start to code, you are going to end up spending more time applying patches and fixes at the end.