Category Archives: Architecture

Building Reactive Systems

In today’s world, we are always striving for building applications which can adapt to constantly changing needs. We want our systems to flexible, resilient, scalable and withstand end user’s high expectations.

Considering these needs, a Reactive Manifesto was put together, with best practices which will help us build robust applications. Following four pillars makes a strong base of reactive application.

1. Responsive
2. Resilient
3. Elastic
4. Message Driven

You can see none of these concepts are new in nature, and you might be already implementing them in the applications you build. Reactive Manifesto brings them under one umbrella and emphasizes their importance. Let’s take a look at these pillars one by one and see what all well-known patterns we can use to implement each of them.

Responsive: An application is responsive if it responds to the user in a timely manner. A very simple example is you clicked on a button or link in a web application, it does not give you a feedback that button was clicked and the action gets completed after few seconds. Such an application is non-responsive as the user is left guessing if he is performing the right action.

Some of the well-known practices and design patterns which help us make sure if the application is responsive
– Asynchronous communication
– Caching
– Fanout and quickest reply pattern
– Fail-fast pattern

Resilience: An application is called resilient if it can handle failure conditions in a graceful manner.

Some of the patterns that help maintain resilience
– Circuit breaker pattern
– Failure handling pattern
– Bounded Queue Pattern
– Bulkhead Pattern

Elasticity: An application is called elastic if it can stand increase or decrease in load without any major impact on overall working and performance.

Some of the practices and patterns to implement elasticity
– Single responsibility
– Statelessness
– Autoscaling
– Self-containment

Message Driven: An application which uses message driven communication makes sure we are implementing various components and services in a loosely coupled manner. This helps us keep our components scalable and makes failure handling easy.

Practices used to implement Message driven communicaiton
– Event driven
– Publisher Subscriber pattern
– Idempotency pattern

I have covered some of these patterns in details in my book on Design Patterns and Best Practices.

Design before you code

Somehow, with the penetration of agile development practices, I have been observing that less and less time is spent on designing the solution. Engineers treat Agile as a license to develop without design.

What is the problem with developing without designing or architecting the system first? I still remember when I was in college, my professor drew a parallel between architecting a building and architecting a software. Would you start building a house without thinking about design? you will not just start placing bricks without considering how many rooms you need? Where all these rooms go? How large should be every room? Where will the kitchen and bathroom be? Where all the wiring and plumbing will go? You will come up with the design, put in on a paper or a software and then calculate the feasibility.

Think of the complications that can happen if you build your house without thinking about design first. One room might be so big that there is very little space for the others. Or you are not left with any space to create a bathroom. These mistakes are costly. And that brings me to the other important factor which is causes ignoring of design in software. The manager thinks that change at a later state is going to be easy. At the end it is code, we can just change it. Well, changing the code is definitely possible, but never easy or cheap. It comes with its own cost.

More than often, where developers need to make changes at a later stage, they would be more inclined to apply quick fixes or hacks rather than making a bigger correct change. Of course, you have a tried and tested code in hand, why would you make too many changes to it, even if you know that is the right thing. And then there is developer ego, I mean, let’s admit it, it is not easy to accept one’s fault, especially if that means you would need to put in extra hours to fix that. More than ego, it is actually denial at times. People would try to stick to the solution they developed initially even though it is realized at a later stage that there could have been a better solution. Reason being, you have already invested too much.

This situation can be avoided if we think about design first. We can make sure our design covers all the possible requirements. It will not be possible to anticipate all the changes that are going to come at a later stage, but we can try to keep our design flexible. The idea is, set the ground rules, understand what all component and services we are required to create? What all data needs to be persisted and how are we going to do that? How security and error handling will be done? We need not get into implementation details, but the high-level design is a good start. We can fill in the details as and when requirements are clearer and we start actual work on the given piece of requirement.

In the end, all I would like to highlight is, if you think you are saving time by not thinking about the design of the application before you start to code, you are going to end up spending more time applying patches and fixes at the end.

Krutchen 4+1 Architectural view model- Agile perspective

I have a strong belief that Solutions Architecture is as much of a science as it is an art. There is no set of fixed rules you can apply to get a final architecture. There are rules, but they can get changed based on kind of project, teams and constraints you are working with.

Coming to 4+1 Architectural view of software architecture, Krutchen shared an interesting way of looking at software architecture in this paper.

Original paper- https://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-architecture.pdf

In crux, the paper suggest that we can look at any software architecture from 4 perspectives or views to get a complete picture.

Logical view: This is end user view of the system. How many entities or classes are there and how they interact, for example, how Employee will be related to Department and Project, what will happen when someone joins or leaves an organization.
UML: Class Diagrams, State Diagrams.

Process View: This talks about how the business works as a process. If you need to open an account in a bank, what process needs to be followed. In addition, we take care of non-functional requirements like scalability, performance etc in this view.
UML: Activity Diagrams

Development View: This is a view for developers, understanding how the system will be implemented (also known as implementation view). How many components will be created and how will they interact with each other.
UML: Component Diagram, Package Diagram

Physical View:
This view explains how this system will be deployed physically. What kind of machines are there and how these are interacting.
UML: Deployment diagram

Scenarios: Scenarios or Use cases are given special attention. Because before getting into any other views, one needs to understand all the use cases we need to handle for the system being developed.

Well the paper explains about these views in details, so here I would like to add my understanding of how to use this model in agile development methodology.

Agile Perspective:
When you are building a software in agile manner, you are taking up one use case at a time, broken down in form of stories. Once you have sorted out what all use cases are you dealing with in current sprint or cycle, you can start by understanding logical view for these cases. Moving on to Process view then Development view and finally Physical view, case by case. So rather than creating the whole picture in one go, we will be creating our architecture as and when we are working on a particular use case.

Technology Agnostic Design

When developing high level design for a solution, it is not a good idea to think about technology choices. We need to keep a gap between design and implementation.

You should not finalize at this point if the solution is going to be build in Java, Python, NodeJS or PHP at this time. For example, you just define that there will be a employee service to provide employee data, but which language will be used to build it, is not part of high level design.

You should take a call what kind of database is well suited, will it be a RDBMS or a document based database, but we should not take a decision which specific vendor’s database we are going to use at this point. For example, we will take a call that we will use RDBMS, but will it be Oracle, mySQL, postgres or some other vendor provided DB, we will decide when we will think about implementation details.

Similarly, all your vendor and technology decisions will not be part of high level design. The details should be filled at a later point, once you have finalized your high level design and made sure you have all the components, services and communications identified.

Why should we not think about technology choices while building high level design? Because it limits our design and solution. Because strengths and weaknesses of a technology becomes strength and weakness of our solution. Because strengths and weaknesses of technologies change over time, hence our architecture should be independent of that.

Once you bring in vendor and technology at a high level design, you commit too much to that. For example say XYZ RDBMS provider is currently the best in market as they provide fastest operations. You design your architecture around that, you use vendor specific data structures and data types. In future, if there is a vendor providing similar services at a cheaper price, we will figure out making a change is very costly as we have to change too much of code. If we would have thought of RDBMS as just a plug and play provide, we could have made this change easily. Infact, keeping our architecture technology agnostic will force us to think beyond a vendor. We will need to think ways of improving our database performance, think of better indexing, sharding, caching, making our architecture robust and independent of technology and vendors.

https://bigmedium.com/ideas/links/managing-technology-agnostic-design-systems.html

https://www.infoq.com/news/2007/09/technology-agnostic-soa

http://bradfrost.com/blog/post/managing-technology-agnostic-design-systems/

Node vs Java Comparison

A good read on Node vs Java https://rclayton.silvrback.com/speaking-intelligently-about-java-vs-node-performance. Explains about the way Java and Node code will work to handle load.

My personal opinion is that programming language is just a tool to deliver the final solution. Only in specialized cases, where we know that a particular language has proven history of solving similar kind of problems, hence has sample code, libraries, knowledge base, forums to help, we should consider that as part of solution. Otherwise non-technical reasons like client preferences, team’e expertise will come into picture.

SOLID Principles for object oriented design

There are many best practices and principles figured out by developers and architects for object oriented design. Robert Martin has intelligently put a subset of these good practices together, and gave them acronym SOLID which helps easy remembrance.

Single responsibility principle: A class should handle only one single responsibility and have only one reason for change. For example a class “Employee” should not change if there a change in project or some reporting details.

Open Closed principle: Code should be open for extension but closed for modification. If you want to add a new type of report in the system, you should not be changing any existing code. More here

Liskov substitution principle: “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program.” So if we have Employee class, which is extended by Manager. We should be able to use Manager instead of Employee and all the Employee methods like calculate Salary, generate annual report etc should work without any issues. Say if there is an object like “ContractWorker” that does not support a few functions of Employee like annual report, one should be careful not to make it subtype of Employee.

Interface Segregation principle: “no client should be forced to depend on methods it does not use”. Coming back to previous example, if “ContractWorker” does not need to support annual report, we should not force it to implement an iEmployee interface. We should break the interfaces say iReport and iEmployee, iEmployee can extent iReport and iContractWorker should implement only iReport. iReport can further be divided into reporting types if required.

Dependency Inversion principle: This one seems to be one of my favorite as I have written about it here, here, here and here. This one indeed is one of the most important design patterns which can be followed to make the code loosely coupled and hence making it more maintainable (golden rule- low coupling + high cohesiveness). In traditional programming, when a high level method calls a low level method, it needs to be aware of the low level method at compile time, whereas using DI we can make high level method depend on an abstraction or interface and details of implementation will be provided at run time, hence giving us freedom to use which implementation to be used. Coming back to my previous example, I can have multiple implementations of Employee Reporting, iReport. Some implementation need and excel report, other might need a PDF reporting, which can be decided at runtime.

Generating ER diagram from database -2

Sometime back I wrote about DBvisualizer to generate schema ER design from database.

Here is another way by using schemaspy.

http://schemaspy.sourceforge.net/

This is a simple java based tool/ jar file. As per example given in link above, all you need to run the jar file providing database access details.

java -jar schemaSpy.jar -t dbType -db dbName [-s schema] -u user [-p password] -o outputDir 

You might want to give database drivers jar file path. For example, for Postgres

java -jar /home/kamal/pathto/schemaSpy_5.0.0.jar -t pgsql -db dbnamehere -s public -u dhusername -p dbpassword -host localhost -port 5432  -o /home/kamal/outputdir -dp /home/kamal/pathto/postgresql-9.3-1104.jdbc4.jar

Enterprise Architecture- Building the core model

One challenge often faced by organizations is that IT is often reactive rather than guiding the operations. You bring in IT once you face a problem and build an IT solution. Once solution is build, struggle starts to integrate with other parts/ applications. A lot of time is than spent on making different pieces work together, which can be avoided if proper EA practices are in place.

Ideally, IT should look at the system and come up with opportunities of improving existing system like automating ordering system and adding new services like moving to mobile platforms.

IT and Business can than prioritize the solutions/ projects based on value addition. Having a big picture in front, it will be easier to take decisions and less time will be spent in making things communicate and work with each other.

Building a core model is important so that newer services can easily gets integrated. For example if centralized data handling and services to share data securely are already in place, getting a new mobile app to market is much easier than in a scenario where we do not have any such centralized solution in place already.

Creating the core IT model is not easy. You need to take a call what to keep as core and what should be customizable. As a rule of thumb, identify what is fixed and what can be changed/ customized in your business. Based on this information, we need to design which part of design is fixed and which is flexible. For example, in a particular business, product information might be centralized but sales can be customized.

In addition, core model need to take care of that fact which processes can be standardized and what data will be centralized and shared. More- http://kamalmeet.com/architecture/enterprise-architecture-manage-your-data-and-processes/

Enterprise Architecture- Manage your data and processes

Any Enterprise architecture needs to take care of two important things, processes and data. A standardized process makes sure certain operations are done in a certain way no matter who is performing them. Data ofcourse is a very important part for any organization, it helps in every aspect of business from fulfilling sales orders, maintaining inventory, making decisions for future etc. So it is important that data is shared across units in effective and secured manner.

Based on the business, need for data sharing and process standardization might vary. We know a standard process adds to predictability (less flexible = agile) but might not work in cases where innovation and flexibility is needed, for example in sales or research. So if we are looking at a business like McDonald’s, we know that each unit needs to follow similar process. Hence process will be part of my core architecture. But in case we are dealing with a Insurance sales business, where each unit might need a different strategy, we will not be be standardizing the process to detail level.

Similarly, decisions needs to taken on centralization of data. For example in a car manufacturing and sales unit, it is important to keep data on inventory, sales, production in sync. Whereas, for an insurance company, it might need flexibility of keeping car insurance and personal insurance data separately. Though you might need to keep products available information at common place. Nonetheless, the data definitions should be strict through out the organization, a completed sales mean same in all aspects.

Based on decisions made as per above analysis, we will be able to create our core architecture effectively. We will know what to add to core and what to keep flexible. The good core design will help business in maintainability and scalability. Adding a new business unit (or a new product or service) and integrating to existing business will depend on readiness of architecture we have finalized.