Microservices in Delphi – Part 1
Contents
Introduction
The last ten months have been very interesting for me when it comes to coding. I have discovered the world of microservices and, from the first moment, it caught my eye.
I started looking at it when I realised that an old application I had to rewrite was too complex a project. I was looking for an approach that would allow me to break down the project to small chunks that I could easily manage and scale up and, also, I could understand and remember when I looked back at the code after some time. On top of that, I wanted to follow the MVVM pattern; another layer of complication.
Very soon I realised that there are not any out-of-the box packages or libraries available to help me write applications with microservices in Delphi. I had watched the videos released by Embarcadero in which you have a REST server serving a specific resource but this seemed to me an effort to capitalise on a buzz word 😕 rather than to implement proper microservices design. What the videos show is only one third of the story.
I ended up writing my own library to help me use microservices in my applications and in this post I introduce it and explain how it works.
Microservices
I am not planning to go into depth here about the theories and approaches around the topic. The best resource I know is the https://microservices.io website. I’ve, also, read a few great books mostly in .NET and Java but this site has everything there is to know about it. I, personally, find the site a bit chaotic but that’s another story.
I quote from the website,
Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of services that are
- Highly maintainable and testable
- Loosely coupled
- Independently deployable
- Organized around business capabilities
- Owned by a small team
In practical terms, when you design an application using microservices, you identify every process you have in the application and you break it down to smaller processes (services) that deal with as few things as possible; you are, literally, encouraged to have the minimum number of tasks you need; even down to one.
For example, say our application saves data about customers. Customers have personal data, web presence, orders, etc. It is very common to have a huge database and store all those elements in different tables. Then, you have the application hitting the database server to deal with the data.
With microservices, the above elements are independent and are served by separate, self-preserved microservices. So,we have microservices for personal data, the web presence, the orders. But we do not stop here; for each one of those elements, we perform actions like adding a new customer or deleting an existing one. These actions translate to separate loosely coupled microservices; and we access them using patterns like in the following examples:
role: customers, cmd: add-personal-data role: customers, cmd: add-email role: customers, cmd: add-linkedin-profile role: customers, cmd: delete-customer role: customers, cmd: delete-all
The deployment of the above set of microservices would lead to five separate services. In the web, this translates to REST-capable services (i.e., servers) and, most likely, to five separate data storage containers.
There are many pros and cons in this architectural style. Personally, I like the modularity, the simplicity and the ability to write focused tests (more on this in a bit) but I found that many times repetition and code duplication is unavoidable along with the weakened (or non-existing) ability to run complex database queries.
Pimlico
Pimlico is a microservices toolbox for Delphi. It is, heavily, inspired by Seneca, an open-source library for microservices for Node.js. You can download Pimlico from this repository where you can inspect the source code (please, do! and let me know how to improve :-)) and you can find the source code for the next demo in the Demo.Customers.Local
project. Although it is early days for the library, it is functional and stable but lacks some functionality that has, mostly, to do with logging and debugging.
We are going to use the aforementioned example with the customer and we are going to deal with the name (personal data) and the email address (web presence). For each of elements, we implement the add and delete actions.
Let’s see how we can use Pimlico.
Local Microservices
In the context of Pimlico, when I refer to local services I mean services embedded (in-process) in the application and not services running in the same environment as the application.
Adding a Customer
We have two elements to add every time we create a new customer: the name and the email. This means we need to separate microservices:
role: customers, cmd: add-name role: customers, cmd: add-email
The building block for local microservices in Pimlico is the TmServiceBase
class. For each one of the above services, we create two separate classes TServiceCustomersAddName
and TServiceCustomersAddEmail
which inherit from TmServiceBase
. You can find the implementation in the demo files in the github repository.
We will come back to the actual implementation in a moment. Once we have the microservice classes, we need to let Pimlico know of their existance. We do this by registering them with the library in the OnCreate
event of the main form:
procedure TFormMain.FormCreate(Sender: TObject); begin pimlico.add('role: customers, cmd: add-name', TServiceCustomersAddName.Create); pimlico.add('role: customers, cmd: add-email', TServiceCustomersAddEmail.Create); end
When the user clicked on the Add
button in the main form, a simple call to pimlico.act
will make sure the correct pattern is executed and it will pass the data. One important point to note here is that in this programming style, communication between microservices is done with excessive exchange of strings; anything from one simple string to JSON and XML representations of objects.
You are free to setup your own rules when it comes to exchanging data between microservices. For this example, when we want to add the name, we will pass it with the Name:
prefix. When the operation is successful, we expect the service to respond with the customer ID (GUID in this case) because we need it when we add the email. A microservice can also respond with and error if something went wrong. It is all up to us to design and manage.
The code in OnClick
event of the Add
button first adds the name, waits for the response and then adds the email. Finally, it informs the user of the outcome of the operation.
procedure TFormMain.btnAddClick(Sender: TObject); var guid: string; begin pimlico.act('role: customers, cmd: add-name', 'Name: '+efName.Text.Trim, atSync, procedure(aStatus: TStatus) begin guid:=aStatus.Response; end); pimlico.act('role: customers, cmd: add-email', 'Email: '+efEmail.Text.Trim+', ID: '+guid, atAsync, procedure(aStatus: TStatus) var msg: string; begin if aStatus.Response.Contains('ERROR') then msg:=aStatus.ErrorMsg else msg:='Customer added with ID: '+ guid + ' and Email: '+aStatus.Response; TThread.Synchronize(nil, procedure begin ShowMessage(msg); end); end); end;
In the above code, note the different modes the two services are called (synchronous, asynchronous) as flagged by the atSync
and atASync
parameters and how we extract the response. The callback mechanism in Pimlico is an anonymous method that receives the TStatus
record.
type TServiceStatus = (ssIdle, ssError, ssRunning, ssStarted, ssStopped); TStatus = record Status: TServiceStatus; ErrorMsg: string; Response: string; end;
Apart from the Status
field, how you pass messages around is on you. ErrorMsg
and Response
sound like reasonable labels but there is unlimited flexibility as to what kind of data they host.
Every time act
is called, Pimlico locates the registered microservice and executes the invoke
procedure. Therefore, we need to implement the invoke
method in the AddName
service: we, first, recover the name from the parameters, create a new customer and return the customer ID to the callback Status
record.
procedure TServiceCustomersAddName.invoke(const aParameters: string); var name: string; customer: TCustomer; guid: TGUID; begin inherited; name:=extractValueFromParams(aParameters, 'Name'); customer:=TCustomer.Create; customer.Name:=name; try CreateGUID(guid); customer.ID:=guid.ToString; customerList.Add(guid.ToString, customer); fStatus.Response:=guid.ToString; except fStatus.Response:='ERROR'; fStatus.ErrorMsg:='Something went wrong. The customer is not saved'; end; end;
For this example, I am using a simple object dictionary to emulate a storage mechanism. Obviously, in a full blown application, the method would access a more sophisticated storage container.
The add-email service implements a very similar approach.
procedure TServiceCustomersAddEmail.invoke(const aParameters: string); var email: string; guidStr: string; begin inherited; guidStr:=extractValueFromParams(aParameters, 'ID'); email:=extractValueFromParams(aParameters, 'Email'); if customerList.ContainsKey(guidStr) then begin try customerList.Items[guidStr].Email:=email; // This is redundant but we demonstrate the use of Response fStatus.Response:=email; except fStatus.Response:='ERROR'; fStatus.ErrorMsg:='Something went wrong. The customer email is not saved'; end; end else begin fStatus.Response:='ERROR'; fStatus.ErrorMsg:='The customer does not exist'; end; end;
As you notice, I use the same storage container in both services; strictly speaking, this is not necessary. In my case, it is very convenient. The consensus in the world of microservices is that each microservice can manage separate storages with the option to have different types in each service. This is both an opportunity and challenge. On one hand, flexibility emerges but the downside is that you miss the ability to run complex (database) inquiries. Another important implication is that you may find challenging to synchronise data. For this reason, relevant approaches have been developed (Saga Pattern).
Updating a Customer
Updating a customer follows the same approach we saw when we added the email. Therefore, I will not repeat the code here. We just need to register two new services.
role: customers, cmd: update-name role: customers, cmd: update-email
Note: You may wonder why I need to break down each element in customer’s profile. I would agree with you that in this particular case it is pointless as I could have one class and do everything in one microservice. This example is very trivial indeed to justify this approach but I am using it to show how you can retrieve and pass information between services.
In general, though, the message in microservices is that you try to construct the business logic of an application by creating a web of services that perform one (or simple) thing. This resonates very well with the SOLID principals in object-oriented programming although it is not the same.
Getting the List of Customers
We register a new service
procedure TFormMain.FormCreate(Sender: TObject); begin ... pimlico.add('role: customers, cmd: get-list', TServiceCustomersGetList.Create); end
Then, we build the new microservice. We need to return a list of TCustomer
and we do it via a JSON representation. Again, JSON is not a requirement as you are free to use whatever suits you.
... implementation ... uses REST.Json; ... procedure TServiceCustomersGetList.invoke(const aParameters: string); var customer: TCustomer; key: string; list: TObjectList; begin inherited; list:=TObjectList.Create; for key in customerList.Keys do begin customer:=TCustomer.Create; customer.ID:=customerList.Items[key].ID; customer.Name:=customerList.Items[key].Name; customer.Email:=customerList.Items[key].Email; list.Add(customer); end; fStatus.Response:=TJSON.ObjectToJsonString(list); list.Free; end;
Back to the main form, we retrieve the JSON string and populate the grid with the customers in the OnClick
event of the List
button.
procedure TFormMain.btnListClick(Sender: TObject); begin gridList.RowCount:=0; pimlico.act('role: customers, cmd: get-list', '', atAsync, procedure(aStatus: TStatus) var list: TdJSON; customer: TdJSON; begin try list:=TdJSON.Parse(aStatus.Response); for customer in list['listHelper'] do TThread.Synchronize(nil, procedure begin gridList.RowCount:=gridList.RowCount + 1; gridList.Cells[0, gridList.RowCount - 1]:= customer['iD'].AsString; gridList.Cells[1, gridList.RowCount - 1]:= customer['name'].asString; gridList.Cells[2, gridList.RowCount - 1]:= customer['email'].AsString; end); except TThread.Synchronize(nil, procedure begin ShowMessage('Error retrieving the list'); end); end; end); end;
Note: I use REST.Json
to create the JSON string but I use another simpler library (delphi-json) to parse the string. REST.Json
has some weird behaviour with fields that don’t start with F
and I couldn’t get my head around it, so I opted in for the other library.
Deleting a Customer
Again, in terms of how we structure our code, deleting a customer is not different to what we have done so far; we register a new service, delete the customer in the invoke
method and call the service when we need it.
Unit Tests
Unit and other types of tests along with code coverage have gained popularity during the last decade and for good reasons I would say. The vibe is so strong that people tend to freak out if you tell them that your code is, mostly, untested.
In the world of microservices, we have a chance here simply because not everything needs to be tested. Breaking down business logic and processes to small and simple (micro)services, may lead to so trivial code that you may realise that there is nothing to test as the code is self-proving. Of course, other types of tests (integration, UI, etc.) may still be relevant.
(Unexpected but Pleasant) Use of Local Services
You may wonder what is the usefulness of having local services; i.e. in-process services. Personally, I found that microservices force me to think more about business logic and then about coding. On top of this, I was pleasantly surprised when I realised that local microervices can help me solve two aspects that I hadn’t found a viable solution for long time:
- Internationalisation (Translations)
- Plugins
Internationalisation (Translations)
For ages, I have been looking for translation solutions in Delphi. There are some commercial libraries but they do not meet my number one criterion: cross-platform implementation. The other thing that makes me uneasy is that all the solutions require that you either include the ability to translate from the very beginning or, if you need it at a later stage, you need to go through all the components in forms and frames in your application and alter the code.
Local microservices opened a different possibility to me. The biggest problem when you want to translate a component is that each one has a very specific property that holds the translatable item; thus, we can’t devise a generic way to do the translation for all the components.
My approach is to create one microservice for each component that needs translation and register it with Pimlico. You can find the full demo in the repository (Demo.i18n
).
pimlico.add('role: i18n, class: TLabel', Ti8nLabel.Create); pimlico.add('role: i18n, class: TButton', Ti8nButton.Create); pimlico.add('role: i18n, class: TRadioButton', Ti8nRadioButton.Create);
Of course, the actual translation of the text is done in a separate microservice which is, again, registered the usual way.
pimlico.add('role: i18n, cmd: translate', Ti18nTranslate.Create);
I don’t include here how the actual translation is done because it is very trivial and naive. You can, of course, see the code in the demo files.
Then, the microservice responsible to translate TLabel
looks like this:
procedure Ti18nLabel.invoke(const aParameters: string); var obj: TObject; component: TLabel; pointerStr: string; fromLang: string; toLang: string; poin: Pointer; begin inherited; pointerStr:=extractValueFromParams(aParameters, 'Pointer'); fromLang:=extractValueFromParams(aParameters, 'from'); toLang:=extractValueFromParams(aParameters, 'to'); poin:=Pointer(pointerStr.ToInteger); obj:=TObject(poin^); if (obj is TLabel) then begin component:=TLabel(poin^); pimlico.act('role: i18n, cmd: Translate', 'Text: ' + component.Text + ', From: ' + fromLang + 'To: ' + toLang, atSync, procedure(aStatus: TStatus) begin component.Text:=aStatus.Response; end); end; end;
A couple of things here worth noticing:
- How does this microservice get access to the component that requests translation? We can only pass strings back and forth in Pimlico, so I supply the address of the component via a
Pointer
converted to a string - As the code shows, we can freely call other microservices within the
invoke
method of a given microservice. We need, however, to make sure the method is called under the correct synchronisation mode according to our needs
The services to translate the TButton
and TRadioButton
are very similar to the above code.
We need to do one more thing; to call the services from the main form to initiate the translation. Now, we can simply iterate through all the components and pass them to the services. This snippet can be found in the OnClick
event of the Translate
button in the main form.
for num := 0 to self.ComponentCount - 1 do begin comp:=Self.Components[num]; pimlico.act('role: i18n, class: *', 'Pointer: '+IntToStr(Integer(@comp))+ ', From: '+fromLang + ', To: '+toLang, atSync); end;
Notice how the services are called; Pimlico understands glob patterns.
Plugins
Plugins is the same story for me as internationalisation. As far as I know, there is no cross-platform library that offers this functionality. Again, to my surprise and satisfaction, microservices gave me a gateway to this.
As an example, suppose we have an application which provides a form for the options of the project (application) as in the figure below (Fig. 1).
Figure 1: Project Options Form
What we want to achieve is to write a plugin that adds a third check box in the project options form (Fig. 2).
Figure 2: Project Options Form with Additional Option
As a good Plugin API designer 😀 , I expose the IPluginProjectOptions
interface that allows any external class (plugin) to manipulate the form. I will not provide here all the implementation details as it will make the already long post even worse. You can find the full source code in Demo.Plugin.Basic
.
Open-source
A plugin for us is another microservice and the new option plugin is implemented in Plugin.ProjectOptions.NewOption
unit under the class TPluginProjectOptionsNewOption
. Then, we register the new class to Pimlico in the initialization
part of the unit.
pimlico.add('role: plugin, space: project-options, cmd: add-new-option', TPluginProjectOptionsNewOption.Create);
I have used the flexibility of the patterns in Pimlico to create a new domain (role: plugin, space: project-options
) that any plugins that want to affect the project form must use. Another good job for the API designer 😆 . Finally, we access the plugin the usual way we have seen in the examples above.
The difference this time is that we do not care to do anything meaningful in the invoke
method of the service. It is enough to only start it.
pimlico.act('action: start', '');
Of course, this really depends on what the plugin does. In this example, our plugin only adds a checkbox and an event when it is clicked but, in more sophisticated situations, I image that lots of work can take place in the invoke
method.
Closed-source
If I haven’t grabbed you attention so far, here’s one more twist: looking at plugins as microservices which can be triggered and accessed via patterns, is a great example of loosely coupled code and, moreover, of code that can be loaded on the fly.
Open-source plugins are great but they need to be available at compilation time. Closed-source code has the advantage of being accessed afterwards.
In Delphi, there are two ways to load external code to binary files: DLLs and Delphi Packages. DLLs lose the battle in my eyes as they are not cross platform. On the other hand, packages are.
In Demo.Plugin.BPL
project, I have created a package that contains two sample “plugins”, i.e. revamped microservices based on IPluginBase
interface.The plugin classes are, basically, the same as in the case of our open-source plugin. If you are playing with the code, make sure you build the bpl
package before you look at how we load plugins at runtime.
If you go back to the Demo.Plugin.Basic
project, you will find the implementation of a Plugin Manager. The manager loads all the available plugins (local and those in the package) in the FormCreate
event.
procedure TFormMain.FormCreate(Sender: TObject); begin PluginManager.LoadPlugins; end;
Then, in the options form OnCreate
event, we start all the relevant plugins.
procedure TFormProjectOptions.FormCreate(Sender: TObject); ... begin ... PlugInManager.startPlugins('role: plugin, space: project-options, cmd: *'); end;
That’s it. Local and external plugins are loaded and executed (Fig. 3)
Finally, Delphi packages become useful outside the IDE: cross-platform and compiler-indepedent runtime plugins. Big time for the packages :P. Now they deserve all our love…
Source Code
The source code of the examples I use in this post can be found in this repository. Please feel free to check and let me know of any comments, omissions, errors, improvements you can think of.
Next on Microservices…
In this post, we looked at the idea of microservices, we briefly explored Pimlico -a toolbox to write microservices in Delphi-, how they can be implemented and some useful applications. However, we dealt with microservices that exist in the host application. In the next part, we’re going to look at how Pimlico can handle microservices scattered in the cloud. Stay tuned.
Excelente el aporte, muchas gracias.
Excellent!
Thanks for share!
Hi, Thanks.
What about part 2?
Thanks a lot!
I never got myself to write it 😉.
I think I should 😄