Event-Driven Architecture: reconcile Notification and Event-Carried State Transfer patterns
Event-driven architectures have tremendous benefits: decoupling application components brings improved resilience, the ability to isolate non-scalable workloads from unpredictable user traffic and better user experience (returning a response before we do the complexe processing).
However, their design is not easy and can lead to numerous debates among developers and architects: should we have very minimalistic events, requiring consumers to fetch additional information? do we need fully-qualified events? ..
In this post, I explain the different types of events and propose a way to simply reconcile the different approaches by relying on AWS EventBridge. A Github repo with fully functional examples awaits you at the end of the article!
Different approaches to event design
A common pattern is the "Notification" event, one that just contains an identifier. Here is an example for an OrderCreated event:
{ "orderId": "1234567" }
This minimal event has the advantage of not requiring any knowledge of the business domain model: there is little risk of violating the interface contract when modifying the producer app. If a consumer wants to know more, they can fetch data with the communicated identifier (and if the source system has a GraphQL API, they can fetch only whatever data is necessary for them).
Another approach is the "Event-carried State Transfer" events (named after the famous "REpresentational State Transfer", aka REST APIs). The event has all the domain data related to the event. Here is an example for the same OrderCreated event:
{
"id": "1234567",
"status": "PAYMENT_ACCEPTED",
"customer": "Bob",
"content": [ ... ]
}
The benefit associated to this approach is that the event can be consumed without any additional information. It also enhances filtering options that the Event Bus will provide: we can for example choose to only consume events representing an order that has the PAYMENT_ACCEPTED status (for example to send an order confirmation email).
A third way consists of publishing a "Delta", i.e. also transmitting the previous state in addition to the current state.
Here is a summary of the advantages and limitations of each approach:
Reconciling the Notification approach and the Event-carried State Transfer approach
We may want to take advantage of the benefits of each approach
- without unnecessarily complicating the architecture of the producer or consumer applications
- without sometimes having control over the applications source code.
This is where a serverless approach mixing EventBridge event bus and Lambda makes sense. In this approach, we will set up
- EventBus rules matching events of type " Notification"
- and enrichment micro-services which will retrieve data from the corresponding business domain and republish the enriched event.
I will start with a simple example, before showing how we can extend this pattern. At the bottom of this article you will find a link to a repository that implements both examples.
The simple version: single enrichment
In this example, a payment management application publishes a PAYMENT -type event bearing only the event id (a notification event).
On the EventBridge side, a rule will explicitly match these events by checking that no additional data is provided
{
"detail-type": ["PAYMENT"],
"detail.payment_data.id": [{ "exists": false }]
}
If this rule matches an event, it will invoke a Lambda which will publish the same event, but enriched with domain data.
We will therefore see two events successively in the event bus (with the same business id):
- the notification event
{
"version": "0",
"id": "a23a7513-b67a-d455-f90c-1f9ddbd14820",
"detail-type": "PAYMENT",
"source": "PaymentSystem",
"account": "112233445566",
"time": "2024-07-04T09:06:47Z",
"region": "eu-west-1",
"resources": [],
"detail": {
"id": "2237082"
}
}
- and the fully-qualified (ECST) one:
{
"version": "0",
"id": "51bbf35e-97d8-8f80-1cc2-debac66460e6",
"detail-type": "PAYMENT",
"source": "PaymentSystem",
"account": "112233445566",
"time": "2024-07-04T09:06:49Z",
"region": "eu-west-1",
"resources": [],
"detail": {
"id": "2237082",
"payment_data": {
"id": "2237082",
"type": "Credit",
"description": "Credit Card - HSBC",
"status": "Confirmed",
"state": "Paid",
"value": 1700,
"currency": "EUR",
"date": "2018-12-15"
}
}
}
(here, the event structure is a little more complex than in the theoretical part, as I display the typical structure of an EventBridge event, which encapsulates the business content with some metadata.)
This fully-qualified event is then sent.
The EventBridge event bus provides:
- Decoupling between Producers and Consumers with scalable and highly available middleware
- Advanced event matching capabilities
- events logging, archiving and replaying
- Retry management for synchronous invocations made by the event bus in case of error.
- Input Transform to format the event as expected by the consumer, without having to deploy this transformation as a Lambda function.
All of these features are demonstrated in the code available at the end of the article.
A more complex enrichment
Let's imagine a more complex case: the payment system publishes a payment event. But this payment is linked to an order, which has its own life cycle, managed in several other applications. And this order is linked to a customer, which also has its own life cycle, managed in a CRM app.
Here we will implement more complex pattern matching logic, but the code of the enrichment functions does not change!
Deploy these examples
In this blog post, I demonstrated how to reconcile the two main event management models, using AWS EventBridge and AWS Lambda
You will find in this Github repo two CloudFormation templates to deploy these fully functional examples.