Category Archives: System Design and documentation

Design before you code

Somehow, with the penetration of agile development practices, I have been observing that less and less time is spent on designing the solution. Engineers treat Agile as a license to develop without design.

What is the problem with developing without designing or architecting the system first? I still remember when I was in college, my professor drew a parallel between architecting a building and architecting a software. Would you start building a house without thinking about design? you will not just start placing bricks without considering how many rooms you need? Where all these rooms go? How large should be every room? Where will the kitchen and bathroom be? Where all the wiring and plumbing will go? You will come up with the design, put in on a paper or a software and then calculate the feasibility.

Think of the complications that can happen if you build your house without thinking about design first. One room might be so big that there is very little space for the others. Or you are not left with any space to create a bathroom. These mistakes are costly. And that brings me to the other important factor which is causes ignoring of design in software. The manager thinks that change at a later state is going to be easy. At the end it is code, we can just change it. Well, changing the code is definitely possible, but never easy or cheap. It comes with its own cost.

More than often, where developers need to make changes at a later stage, they would be more inclined to apply quick fixes or hacks rather than making a bigger correct change. Of course, you have a tried and tested code in hand, why would you make too many changes to it, even if you know that is the right thing. And then there is developer ego, I mean, let’s admit it, it is not easy to accept one’s fault, especially if that means you would need to put in extra hours to fix that. More than ego, it is actually denial at times. People would try to stick to the solution they developed initially even though it is realized at a later stage that there could have been a better solution. Reason being, you have already invested too much.

This situation can be avoided if we think about design first. We can make sure our design covers all the possible requirements. It will not be possible to anticipate all the changes that are going to come at a later stage, but we can try to keep our design flexible. The idea is, set the ground rules, understand what all component and services we are required to create? What all data needs to be persisted and how are we going to do that? How security and error handling will be done? We need not get into implementation details, but the high-level design is a good start. We can fill in the details as and when requirements are clearer and we start actual work on the given piece of requirement.

In the end, all I would like to highlight is, if you think you are saving time by not thinking about the design of the application before you start to code, you are going to end up spending more time applying patches and fixes at the end.

Steps for Designing a system from scratch

You are into a new project, and provided with some requirements. How would you go about designing the new system

We are in this situation a lot of times. So here I am trying to create a step by step guide from taking a requirement doc to finalizing a system architecture.

Stage 0: Before getting started with the design process, we need to make sure about the following.

– Do we have clear understanding of requirements?
– Are we creating something from scratch or enhancing and existing system. In later case we will have design and technology constraints from previous system?
– Have we identified non functional requirements- security, performance, availability etc.
– Have we identified all stakeholders and their role?
– Have we identified key players that will help in creating architecture (architects, Business Analysts, Product owners)
– Have we decided on time/ money to spent on design activities?
– Have we identified reference material? Do we have artifacts for similar design problems, from either inhouse or external sources?
– How are we going to maintain the design artifacts- wiki, git, svn, confluence etc. We will need to maintain versioning?
– Have we identified any guiding principles for the design- we will use open source softwares and tools, or we will be using linux system of deployment etc.
– Are there any constraints- say client wants to use any specific third party tools or technologies, any specific compliance required by law (multilingual support), service availability grantee
– Have we identified all third party systems with which our system will interact and how the interaction will be done?
– Are we creating the system in one go or will it be a phased delivery. Have we identified the value add provided by various components being built and prioritized the delivery?
– In case of phased delivery, we need to identify scope of each phase?
– Have we identified risks involved and mitigated them?
– If we are modifying or enhancing an existing system, we need to understand what areas can be reused, enhanced and built from scratch?
– Better to create a formal document to identify what all design artifacts are required.
– Define KPIs (Key performance indicators) and SLAs (Service Level Agreements)
– Have we defined acceptance criteria for the design?

Stage 1: Now we need to understand the business and what changes do we need.

– Have we understood organization structure?
– Have we identify business goals and objectives for the organization and what changes are required?
– Identify all business requirements, for example customer should be able to return a product is a business requirement.
– Identify and design current business processes (How current business work, does it fulfill all the business requirements or not, if yes, do we need to change or enhance the way it is being done right now, for example current purchase process is manual and we want to provide online options.)
– Identify changes or modification required in business processes

– Design artifacts to be delivered in this stage
Catalogs
— Organizations
— Actors
— Goals
— Roles
— Business Services
— Locations
— Process / Products
Matrices
— Business interaction
— Actor/ Role
Diagrams
— Business Services
— Functional Decomposition
— Product/ Process lifecycle
— Goal/ Service diagram
— Business Use cases
— Process Flow
— Event diagram

Stage 2: Focus on Data used

– What data is being used in the application? how it is originated and used?
– How the data is shared securely in enterprise
– Create common vocabulary and data definitions
– Identify security measures to be taken

– Design artifacts to be delivered in this stage
Catalogs
– Data Entities

Matrices
– Data Entity/ Business function
– Data Entity/ Application matrix

Diagrams
– Conceptual Data Diagram
– Logical Data Diagram
– Physical Data diagram
– Data lifecycle diagram
– Data Security diagram
– Data migration diagram

Stage 3: What all Applications are available? Changes required and new ones to be created

Application- Core parameters
– Platform independence
– Easy to use
– Identify existing applications and newly ones to be created at logical level and than map to physical level

– Design artifacts to be delivered in this stage
Catalogs
– Application portfolio
– Interface catalog

Matrices
– Application/ Organization
– Role/ Application
– Application/ Function
– Application/ Interactions

Diagrams
— Application communication
— Application and user location
— Application use case
— Application details – components/ modules and services
— Application details – Layered architecture if used

Stage 4: Understand the technology working behind the scenes

Control technical diversity: Minimizes cost of expertise.

Catalogs
— Technology portfolio

Matrices
— Application/ Technology

Diagrams
— Deployment diagram
— Environments and locations
— Communication engineering diagram (firewalls)

Stage 5: Lets consider Non Functional Requirements
— Security
— Performance
— Availability
— Disaster recover
— Data backups
— Others (Project specific)

Stage 6: Post Design phase:

– Did we identify reusable artifacts and services which can be used by other projects?
– Have we conducted periodic validation that design and product being build are in sync?
– Does the design change due to any change requests? Has that been reflected in design?
– Have we met all the acceptance criteria that were set initially?

Data Modeling at different levels

When you are designing database for an application, there can be 3 core levels at which you can design your database.

1. Conceptual Level: At this level you are only aware of high level entities and their relationships. For example you know that you have “Employee” Entity who “works for” a “Department” and “has” an “Address”. You are not worried about details.

2. Logical Level: You try to add as much details as possible, without worrying about how it will actually be converted to a physical database structure. So will provide any attributes for “Employee” i.e. Id, FirstName, LastName, AddressId, Salary and define primary and foreign key relations.

3. Physical Level: This is the actual representation of your database design with exact column names, types etc.

database

More info- http://www.1keydata.com/datawarehousing/data-modeling-levels.html

HTTP verbs implementation for REST webservices

A REST webservice would normally support four HTTP verbs GET, POST, PUT and DELETE.

Simply putting, the verbs have following function.

GET: requests some information from server.
POST: sends/ posts some data to application.
PUT: requests that sent data to be put under given URI location.
DELETE: Deletes the given entity.

Most common of these are GET and POST, as they can mostly handle all the requests.

Another interesting debate is when to use PUT and POST as both tend to send some data to server. Ideally, when we know exactly where to add the data, we will use put, and if we want application to take a call, we will use post. For example if I want to add/ edit user id 112, I will use PUT. But if I just say add a user and let application assign the id, I will use POST.

More on PUT vs POST-
http://stackoverflow.com/questions/630453/put-vs-post-in-rest?rq=1

Message Oriented Middleware

In last post I talked about what is middleware, I will focus on message implementation of same today. Message oriented middleware or MOM mostly uses message queues to send and receive data between two systems.

In simple terms, a message is anything that is being sent from one system to another. Mostly MOM uses XML formats, sometimes SOAP based requests or plain texts. An example MOM system will send message to a  message queue or MQ, from where the receiver will pick up the message.

Advantages of Message Oriented Middleware

  1. Persistence: In normal client-server architecture, we will need to make sure both the systems to be available to have a successful communication. Whereas if we are using MQs, one system can still send messages even if the second is down.
  2. Support for Synchronous and Asynchronous communication: by default the communication is asynchronous but we can implement a synchronous system where a request message sender will wait for the response from other party.
  3. Messages can be consumed at will: If one system is busy when messages are received (which do not need immediate response), it can consume the messages when load is less. For example, a lot of systems are designed to consume the messages at non business hours.
  4. Reliability: As messages are persistent, threat of losing information is low even if one or more systems are not available. Additional security mechanism can be implemented in MQ layer.
  5. Decoupling of systems: Both client and server work independently, and often do not have knowledge for other end. System A creates a message and adds to message queue, without concerning who will pick it up as long as it gets the response message (if required). So one system can be written in Java and other can be in Dot Net.
  6. Scalability: As both machines involved in interaction are independent of each other, it is easier to add resources at either end without impacting the whole system.
  7. Group communication: Sender can send message to multiple queues or same queue can have multiple listeners. In addition Publisher- Subscriber approach can help broadcast a message.

Types of Messaging:

Point to Point: This is a simple messaging architecture where a sender will directly send a message to receiver through message queue.

Publisher-Subscriber (Pub-Sub): This type of communication is required when sender wants to send messages to multiple receivers. Topics are defined to which subscriber can subscribe and receive requests based on same. For example, say a core banking system can trigger messages on various events like new account open, a withdrawal is made, interest rate changed etc. For an event, multiple other systems might want that information to take an action, so say for all withdrawal events, systems like fraud detection, mobile messaging system, daily reporting system, account maintenance system subscribe. Whenever, publisher publishes the message to “Withdrawal” topic, all of these systems will receive the message and take appropriate action.

Understanding Open/Closed Principle

Open closed principle is an important rule in software design, which we need to follow when designing a software system. It states “A Software code should be open for extension but closed for modification”.

What does that mean? Let’s take a very simple example from day to day life. We all use TV remotes. Say, we have created a very simple software code for remote operations like on/ off, vol increase/ decrease, channel change etc. Now what should be my design considerations

1. Any one should be able to extend the code to add new functionality, for example, for advanced TV’s one should be able to add functionality for subtitle language selection, operator specific options etc.
2. At the same time I need to make sure that no one should be able to alter existing core functionality like vol increase/ decrease, so that it does not break down the functions which are working fine perfectly.

Point 1 is what we say open for extension and point 2 states closed for modification principle. The open/ closed principle helps maintainability of code perfectly, by making sure we do not break what is working fine with flexibility of adding new features at the same time.

What is Middleware?

If you look around for the term Middleware, you will see it being used widely but unfortunately not explained properly anywhere. What is middleware? If you are working on simple web applications, the possibility is that you have mostly worked on client-server paradigm without worrying about middleware concepts. Middleware genrally comes into picture when we are dealing with multiple(atleast 2) applications. It helps these applications talk to each other, even if these applications are on different platforms, technologies and servers.

How does Middleware do it? Lets take a simple example. We have a college A, which has an inhouse application (colA app), used to track students data, attendence, results etc. Now there is an internet application which is supposed to display results of students online (result.com). Now result.com app needs to communicate to colA app to fetch student result data. We introduce a middleware application which helps these 2 app communicate. Let’s say this is message oriented middleware, so as soon as a new result is available on colA app, it writes it on a queue of middleware app. Now middleware might do some processing on the data if required and add it to another queue, which is listened to by result app.

Why can’t result app directly talk to colA app? It can, but what will happen if complexity increases and we introduce many new applications into the arena. Say we have hundereds of colleges and tens of result apps. And to complicate it further, college apps also need to communicate to each other. Just imagine the comexity that each app is listening to tens of other apps and fetching data from tens of apps. Each app need to worry about security, concurrency, connectivity, handling so much load. Different apps might be using different technologies to make the communication difficult. With introduction of middleware, we are moving all the complexity out of the apps so that they can focus on their core areas and let middleware worry about handling load, setting up clustors, managing security, dealing with different technologies and platforms.

Middleware

Using Javadoc for documentation

What is Javadoc? Most of the times people confuse it with the actual HTML documentation for the Java code. This is somewhat true but not entirely. Javadoc, itself is not the documentation, it is a tool from SUN Microsystem that helps you generate the documents.

How does it work? Javadoc allows developers to add documentation for a code in the java file itself in some specially formatted comments. The tool will read these comments and create a HTML document file.

Why do we use Javadoc? Why can’t we just create documents on our own? Well That is possible. Javadoc is there to make your life simpler as a developer. It helps you keep your code and documentation information in the same file. Just imagine if you are keeping the documentation separately, everytime you make a change in code, you ill need to modify the documentation also. What if someone misses it? If it is in the same file it is easier to maintain and hard to miss.  Most importantly, you are not writing the whole documentation, but just adding some quick comments which will be interpreted by the Javadoc tool. Lot more easier and faster than actually writing the documentation.

How to implement it? Simple, just add some specially formatted comments in your code

Class Level comments

@author : Author of the class
@version : automatically provide accurate version and date updates.
@since :  Describes when this functionality has first existed.

Method Level

@param : method or constructor argument names and description
@return :  What does your method returns
@throws : Does this method throw an exception
@deprecated : 	Describes an outdated method.

General Commets

@see : Adds 'See also' entry
@link : Adds an inline link with a label for users of your API documentation

The final code will look something like

import java.util.ArrayList;

/**
 * This is a Java Class to find anagrams. Any class level description goes here
 *
 * @author kamal
 * @version 1.0
 * @since 21-Sep-2011
 */
public class test{

/**
 * This is the main method method
 *
 * use {@link #findAnagrams(String)}
 * @param str
 * 		Any param description goes here
 */
public static void main(String str[])
{
	try
	{
		 String finalString="TestString";
		 test t =new test();
		 t.findAnagrams(finalString);
	}
	catch(Exception e) {
	   e.printStackTrace();
	}
}

/**
 * This is another method and we will add the description here
 *
 * @param input
 * 		Any param description goes here
 * @return
 * @throws NullPointerException
 */
private ArrayList findAnagrams(String input) throws NullPointerException
{
	ArrayList retList=new ArrayList();
	if(input.equals("")) throw new NullPointerException();
	printAnagrams(retList, "",input);
	return retList;
}

/**
 * Yet another method description goes here
 *
 * @param retList
 * @param prefix
 * @param word
 */
public void printAnagrams( ArrayList retList, String prefix, String word) {

if(word.length() <= 1) {
	retList.add(prefix + word);
} else {
for(int i = 0; i < word.length(); i++) {
String cur = word.substring(i, i + 1);
String before = word.substring(0, i); // letters before cur
String after = word.substring(i + 1); // letters after cur
printAnagrams(retList,prefix + cur, before + after);
}
}
}
}

and your javadoc comments will look like

image link

testdoc

System Design and documentation

Found some old notes of mine on system documentation. What all documentation you will deliver to customer

1.Business Process Model (The broader picture)

2.System Process Model (How does the system work)

3.Business Concept Model (Business Rules, Requirement Documents)

4.Use Case diagrams

5.Use Case Narratives

6.Solution analysis (Analyze various solutions)

7.Application Architecture Overview (existing components, new components, users, systems and subsystems, responsibilities, files queues, databases)

8.Solution Option Comparison (Business Criteria, Technical Criteria)

9.Deployment Diagrams- Network, Systems, subsystems, application servers, database servers versions

10.Database Diagram

11.Component Architecture- Interactions, Interfaces

12.Infrastructure Architecture- JDBC driver, DB2 driver version etc

13.User Interface UI

14.Test Cases

15. Class Diagrams and Sequence Diagrams