EDA, messaging and NServiceBus at NNUG Oslo 25th of May

On the next Norwegian .Net User Group I will give a talk on NServiceBus. Topics which will be covered include an overview of the architecture and capabilities of NServiceBus  and configuration options provided by the framework.

Ole-Marius Moe-Helgesen will give an introduction to Event Driven Architecture and messaging and he will also share experiences from a project at an insurance company which made use of NServiceBus.

Jan Ove Skogheim Olsen will share project experiences with NServiceBus from Call Norwegian.

More information available here: http://www.nnug.no/Avdelinger/Oslo/Moter/Brukergruppemote-25-mai-2010/

Persistence with Command and Query Responsibility Segregation

Command and Query Responsibility Segregation (CQRS) is a pattern where reading of data and commands for updating the domain model are separated into separate services. Architectures for distributed systems built on the CQRS pattern offers high scalability and reliability and has gained in popularity during the last couple of years.

Greg Young visited this month’s javaBin meeting in Oslo for a talk on CQRS based architectures, and in this blog post I will share some of the new insight I got into CQRS.

Event sourcing and eventual consistency are two essential concepts which fit well together with a CQRS based architecture, and previously I considered these two concepts to be mandatory in order to make the architecture scalable and reliable. However, the complexity introduced to a system by using these two concepts may scare many brave developers away from building real production systems which makes the most out of an architecture built on the CQRS pattern.

Most developers feel more comfortable with using well-known architectures built on a relational model stored in a RDBMS supporting ACID capable transactions. The mind shift required when changing to event sourcing and eventual consistency may seem too big and risky.

Typical architecture utilizing the CQRS pattern
Figure 1, typical architecture utilizing the CQRS pattern

In the javaBin talk Greg Young actually advised against using eventual consistency when starting to implement a new system and rather gradually introduce the concept in parts of the system as it evolves and scalability issues appear. This will simplify the initial implementation and make it easier to get started with CQRS.

The simplest alternative: No event sourcing and no eventual consistency

This is the simplest option for handling consistency and concurrency because the domain model and denormalized read model can be updated in a single transaction.

The domain model and the denormalized read model can either be stored in the same database server or in different servers. A distributed transaction is required if updates are made on different threads or to different database servers, which will have an impact on the performance. An ORM is typically used for persistence of the domain model.

The read model can even be implemented as views on top of an existing schema modeled for OLTP.

Slide2
Figure 2, Domain model and read model updated in a single transaction

Solving scaling issues as they arise: Event sourcing and no/partial eventual consistency

Greg Young prefers using event sourcing rather than a relational schema when persistence of the domain model. To quote his thoughts about ORMs: “Using an ORM is like kissing your sister!”

The events can for example be stored in a RDBMS, an object database, a document database or be serialize to flat files. The event store must support transactions.

As the system evolves and scalability issues surface, an event queue (and hence eventual consistency) for updates to the read model can be party introduced.

Slide3
Figure 3, event store and read model is updated in a transaction.
An event queue and eventual consistency is introduced in areas where scaling issues arise.

The most complex and powerful alternative: Event sourcing and eventual consistency

In this alternative the domain events are stored in an event storage, and a queue is used to update the read model.

Two different queues can be used when updating the read model. The most traditional architecture is to publish the event to a separate queue in the same transaction as updates the event store. Tools like NServiceBus are typically used when publishing to the queue.

The second alternative was described in a recent blog post by Greg Young and uses the event storage as a queue. This means that there are no requirements for distributed transactions, as the only write happening when processing a command is to the event store. The read model is updated from the events in the event store and not from a separate queue. This has the advantage that there is only one version of the truth; it’s not possible to publish events which have different content from the ones stored in the event storage.

Slide4
Figure 4, event store and event queue

 

Slide5
Figure 5, using the event store as a queue

Conclusions

There is a wide range of options available for how to design persistence in a CQRS based architecture. The most important thing to consider is that the persistence requirements for the domain model on the command service usually will not conform well to the data retrieval requirements for the read service (think OLTP vs. OLAP).

Other factors which must be taken into consideration when designing the persistence models are the cost requirements, is it a greenfield or a brownfield project, the skills and competency of the developers, SLAs and enterprise architecture guidelines for the organization.

LEAP part 3 – SOA

The third master class: Loosely Coupled Business Systems: SOA on the Microsoft platform

Udi Dahan – The Software Simplist had been hired to present the third LEAP master class in Oslo. He is an well known international expert on enterprise software architecture and design, and is the author of the open source messaging framework nServiceBus.

The entire class was based on discussion and interaction with the audience, and the only Power Point slide used was the one showing the agenda.

He started out with sketching a naive traditional n-tier application (big ball of mud), and based on suggestions from the audience we explored different solutions which might improve the solution. Whatever suggestions we threw at him, he always had a thoroughly considered answer describing pros and cons with the suggested solution. He obviously has a lot of experience with real world enterprise SOA applications.

The goal was to create autonomous services – standalone services with loose coupling to other services. The system should be scalable and reliable and use as few resources as possible.

Topics discussed

Coupling

  • Avoid coupling by slicing independent logic into separate vertical autonomous boundaries / services
    • Example of services: Order, inventory management, billing, shipping
  • How should services communicate with each other?

Where to put the orchestration/workflow logic?

  • Layer on top of the business layer components?
  • In an Enterprise Service Bus on the side of the services?
  • In the GUI?

When to use…

  • Fire and forget
  • Messaging (async / synchronous)
  • Events

Duplicate data in order to remove dependencies and keep services autonomous?

  • How to synchronize the data?
    • Publish / subscribe pattern can be used. E.g. when an address is updated in the customer service, the customer service can publish the updated address.
    • Context of the update is important. Why did the change happen? This is information which is available close to the user and the business process (i.e. NOT at the data base tier), and is usually triggered from the UI.
    • Versioning – published events have to be backwards compatible with subscribers
  • Duplication of data is usually considered a bad practice, but for SOA it may have advantages:
    • Avoid service calls (e.g. retrieve customer address RPC style when needed)
    • Services become more reliable and autonomous. What if the customer service is down? The the shipment service won’t be able to do it’s work since the address can’t be retrieved.
    • The duplicated data can be considered a local cache for the service

Publish/subscribe vs. request/response

  • Doesn’t both solutions create dependencies between services?
    • Request/response creates design-time dependencies
    • Pub/sub makes it possible to create services which are both design-time and run-time autonomous
      • Run-time autonomous: Will continue to work even when other services goes down
      • Design-time autonomous: Has no direct references to other services
  • Pub/sub advantages:
    • Better performance
      • Local cache (duplicated data) in each service
      • No blocking transactions across service boundaries
      • Easier to run processes in parallel – scales better
      • Fire and forget gives faster response in the UI
    • Better resource utilization
      • Messages can be queued up and processed when resources get available
  • Pub/sub disadvantages:
    • Systems might be harder to design correctly (requires untraditional thinking)
    • Systems get harder to understand and debug
  • Request/response advantages:
    • Explicit service calls / dependencies makes the system easier to implement, understand and debug
  • Request/response disadvantages:
    • Synchronous blocking architecture requires more resources
      • Ties up more resources over longer time periods, ie. the system will not scale well.
  • Choose the right architecture for the right place, there is no silver bullet.

Achieving autonomous services by letting each service have it’s own UI

Consider a shipment service which requires a shipment address. The traditional design would be to let the shipment service call the customer service. Another option would be to let the shipment service collect the shipment address through it’s own GUI, and thus keeping the service autonomous.