Introduction

The last ten months have been very interesting for me when it comes to coding. I have discovered the world of microservices and, from the first moment, it caught my eye.

I started looking at it when I realised that an old application I had to rewrite was too complex a project. I was looking for an approach that would allow me to break down the project to small chunks that I could easily manage and scale up and, also, I could understand and remember when I looked back at the code after some time. On top of that, I wanted to follow the MVVM pattern; another layer of complication.

Very soon I realised that there are not any out-of-the box packages or libraries available to help me write applications with microservices in Delphi. I had watched the videos released by Embarcadero in which you have a REST server serving a specific resource but this seemed to me an effort to capitalise on a buzz word 😕 rather than to implement proper microservices design. What the videos show is only one third of the story.

I ended up writing my own library to help me use microservices in my applications and in this post I introduce it and explain how it works.

Microservices

I am not planning to go into depth here about the theories and approaches around the topic. The best resource I know is the https://microservices.io website. I’ve, also, read a few great books mostly in .NET and Java but this site has everything there is to know about it. I, personally, find the site a bit chaotic but that’s another story.

I quote from the website,

Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of services that are

• Highly maintainable and testable
• Loosely coupled
• Independently deployable
• Organized around business capabilities
• Owned by a small team

In practical terms, when you design an application using microservices, you identify every process you have in the application and you break it down to smaller processes (services) that deal with as few things as possible; you are, literally, encouraged to have the minimum number of tasks you need; even down to one.

For example, say our application saves data about customers. Customers have personal data, web presence, orders, etc. It is very common to have a huge database and store all those elements in different tables. Then, you have the application hitting the database server to deal with the data.

With microservices, the above elements are independent and are served by separate, self-preserved microservices. So,we have microservices for personal data, the web presence, the orders. But we do not stop here; for each one of those elements, we perform actions like adding a new customer or deleting an existing one. These actions translate to separate loosely coupled microservices; and we access them using patterns like in the following examples:

role: customers, cmd: add-personal-data
role: customers, cmd: add-email
role: customers, cmd: add-linkedin-profile
role: customers, cmd: delete-customer
role: customers, cmd: delete-all

The deployment of the above set of microservices would lead to five separate services. In the web, this translates to REST-capable services (i.e., servers) and, most likely, to five separate data storage containers.

There are many pros and cons in this architectural style. Personally, I like the modularity, the simplicity and the ability to write focused tests (more on this in a bit) but I found that many times repetition and code duplication is unavoidable along with the weakened (or non-existing) ability to run complex database queries.

Pimlico

Pimlico is a microservices toolbox for Delphi. It is, heavily, inspired by Seneca, an open-source library for microservices for Node.js. You can download Pimlico from this repository where you can inspect the source code (please, do! and let me know how to improve :-)) and you can find the source code for the next demo in the Demo.Customers.Local project. Although it is early days for the library, it is functional and stable but lacks some functionality that has, mostly, to do with logging and debugging.

We are going to use the aforementioned example with the customer and we are going to deal with the name (personal data) and the email address (web presence). For each of elements, we implement the add and delete actions.

Let’s see how we can use Pimlico.

Local Microservices

In the context of Pimlico, when I refer to local services I mean services embedded (in-process) in the application and not services running in the same environment as the application.

Adding a Customer

We have two elements to add every time we create a new customer: the name and the email. This means we need to separate microservices:

role: customers, cmd: add-name
role: customers, cmd: add-email

The building block for local microservices in Pimlico is the TmServiceBase class. For each one of the above services, we create two separate classes TServiceCustomersAddName and TServiceCustomersAddEmail which inherit from TmServiceBase. You can find the implementation in the demo files in the github repository.

We will come back to the actual implementation in a moment. Once we have the microservice classes, we need to let Pimlico know of their existance. We do this by registering them with the library in the OnCreate event of the main form:

procedure TFormMain.FormCreate(Sender: TObject);
begin
  pimlico.add('role: customers, cmd: add-name', TServiceCustomersAddName.Create);
  pimlico.add('role: customers, cmd: add-email', TServiceCustomersAddEmail.Create);
end

When the user clicked on the Add button in the main form, a simple call to pimlico.act will make sure the correct pattern is executed and it will pass the data. One important point to note here is that in this programming style, communication between microservices is done with excessive exchange of strings; anything from one simple string to JSON and XML representations of objects.

You are free to setup your own rules when it comes to exchanging data between microservices. For this example, when we want to add the name, we will pass it with the Name: prefix. When the operation is successful, we expect the service to respond with the customer ID (GUID in this case) because we need it when we add the email. A microservice can also respond with and error if something went wrong. It is all up to us to design and manage.

The code in OnClick event of the Add button first adds the name, waits for the response and then adds the email. Finally, it informs the user of the outcome of the operation.

procedure TFormMain.btnAddClick(Sender: TObject);
var
  guid: string;
begin
  pimlico.act('role: customers, cmd: add-name', 'Name: '+efName.Text.Trim, atSync,
                          procedure(aStatus: TStatus)
                          begin
                            guid:=aStatus.Response;
                          end);
  pimlico.act('role: customers, cmd: add-email', 'Email: '+efEmail.Text.Trim+', ID: '+guid, atAsync,
                          procedure(aStatus: TStatus)
                          var
                            msg: string;
                          begin
                            if aStatus.Response.Contains('ERROR') then
                              msg:=aStatus.ErrorMsg
                            else
                              msg:='Customer added with ID: '+ guid + ' and Email: '+aStatus.Response;
                            TThread.Synchronize(nil, procedure
                                                     begin
                                                       ShowMessage(msg);
                                                     end);
                          end);
end;

In the above code, note the different modes the two services are called (synchronous, asynchronous) as flagged by the atSync and atASync parameters and how we extract the response. The callback mechanism in Pimlico is an anonymous method that receives the TStatus record.

type
  TServiceStatus = (ssIdle, ssError, ssRunning, ssStarted, ssStopped);
 
  TStatus = record
    Status: TServiceStatus;
    ErrorMsg: string;
    Response: string;
  end;

Apart from the Status field, how you pass messages around is on you. ErrorMsg and Response sound like reasonable labels but there is unlimited flexibility as to what kind of data they host.

Every time act is called, Pimlico locates the registered microservice and executes the invoke procedure. Therefore, we need to implement the invoke method in the AddName service: we, first, recover the name from the parameters, create a new customer and return the customer ID to the callback Statusrecord.

procedure TServiceCustomersAddName.invoke(const aParameters: string);
var
  name: string;
  customer: TCustomer;
  guid: TGUID;
begin
  inherited;
  name:=extractValueFromParams(aParameters, 'Name');
  customer:=TCustomer.Create;
  customer.Name:=name;
  try
    CreateGUID(guid);
    customer.ID:=guid.ToString;
    customerList.Add(guid.ToString, customer);
    fStatus.Response:=guid.ToString;
  except
    fStatus.Response:='ERROR';
    fStatus.ErrorMsg:='Something went wrong. The customer is not saved';
  end;
end;

For this example, I am using a simple object dictionary to emulate a storage mechanism. Obviously, in a full blown application, the method would access a more sophisticated storage container.

The add-email service implements a very similar approach.

procedure TServiceCustomersAddEmail.invoke(const aParameters: string);
var
  email: string;
  guidStr: string;
begin
  inherited;
  guidStr:=extractValueFromParams(aParameters, 'ID');
  email:=extractValueFromParams(aParameters, 'Email');
  if customerList.ContainsKey(guidStr) then
  begin
    try
      customerList.Items[guidStr].Email:=email;
      // This is redundant but we demonstrate the use of Response
      fStatus.Response:=email;
    except
      fStatus.Response:='ERROR';
      fStatus.ErrorMsg:='Something went wrong. The customer email is not saved';
    end;
  end
  else
  begin
    fStatus.Response:='ERROR';
    fStatus.ErrorMsg:='The customer does not exist';
  end;
end;

As you notice, I use the same storage container in both services; strictly speaking, this is not necessary. In my case, it is very convenient. The consensus in the world of microservices is that each microservice can manage separate storages with the option to have different types in each service. This is both an opportunity and challenge. On one hand, flexibility emerges but the downside is that you miss the ability to run complex (database) inquiries. Another important implication is that you may find challenging to synchronise data. For this reason, relevant approaches have been developed (Saga Pattern).    

Updating a Customer

Updating a customer follows the same approach we saw when we added the email. Therefore, I will not repeat the code here. We just need to register two new services.

role: customers, cmd: update-name
role: customers, cmd: update-email

Note: You may wonder why I need to break down each element in customer’s profile. I would agree with you that in this particular case it is pointless as I could have one class and do everything in one microservice. This example is very trivial indeed to justify this approach but I am using it to show how you can retrieve and pass information between services.

In general, though, the message in microservices is that you try to construct the business logic of an application by creating a web of services that perform one (or simple) thing. This resonates very well with the SOLID principals in object-oriented programming although it is not the same.

Getting the List of Customers

We register a new service 

procedure TFormMain.FormCreate(Sender: TObject);
begin
  ...
  pimlico.add('role: customers, cmd: get-list', TServiceCustomersGetList.Create);
end

Then, we build the new microservice. We need to return a list of TCustomer and we do it via a JSON representation. Again, JSON is not a requirement as you are free to use whatever suits you.

...
implementation
...
uses 
  REST.Json;
...
procedure TServiceCustomersGetList.invoke(const aParameters: string);
var
  customer: TCustomer;
  key: string;
  list: TObjectList;
begin
  inherited;
  list:=TObjectList.Create;
  for key in customerList.Keys do
  begin
    customer:=TCustomer.Create;
    customer.ID:=customerList.Items[key].ID;
    customer.Name:=customerList.Items[key].Name;
    customer.Email:=customerList.Items[key].Email;
    list.Add(customer);
  end;
  fStatus.Response:=TJSON.ObjectToJsonString(list);
  list.Free;
end;

Back to the main form, we retrieve the JSON string and populate the grid with the customers in the OnClick event of the List button.

procedure TFormMain.btnListClick(Sender: TObject);
begin
  gridList.RowCount:=0;
  pimlico.act('role: customers, cmd: get-list', '', atAsync,
                  procedure(aStatus: TStatus)
                  var
                    list: TdJSON;
                    customer: TdJSON;
                  begin
                    try
                      list:=TdJSON.Parse(aStatus.Response);
                      for customer in list['listHelper'] do
                        TThread.Synchronize(nil, procedure
                                                 begin
                                                   gridList.RowCount:=gridList.RowCount + 1;
                                                   gridList.Cells[0, gridList.RowCount - 1]:=
                                                              customer['iD'].AsString;
                                                   gridList.Cells[1, gridList.RowCount - 1]:=
                                                              customer['name'].asString;
                                                   gridList.Cells[2, gridList.RowCount - 1]:=
                                                              customer['email'].AsString;
                                                 end);
                    except
                        TThread.Synchronize(nil, procedure
                                            begin
                                              ShowMessage('Error retrieving the list');
                                            end);
                    end;
                  end);
end;

Note: I use REST.Json to create the JSON string but I use another simpler library (delphi-json) to parse the string. REST.Json has some weird behaviour with fields that don’t start with F and I couldn’t get my head around it, so I opted in for the other library.

Deleting a Customer

Again, in terms of how we structure our code, deleting a customer is not different to what we have done so far; we register a new service, delete the customer in the invoke method and call the service when we need it.

Unit Tests

Unit and other types of tests along with code coverage have gained popularity during the last decade and for good reasons I would say. The vibe is so strong that people tend to freak out if you tell them that your code is, mostly, untested.

In the world of microservices, we have a chance here simply because not everything needs to be tested. Breaking down business logic and processes to small and simple (micro)services, may lead to so trivial code that you may realise that there is nothing to test as the code is self-proving. Of course, other types of tests (integration, UI, etc.) may still be relevant. 

(Unexpected but Pleasant) Use of Local Services

You may wonder what is the usefulness of having local services; i.e. in-process services. Personally, I found that microservices force me to think more about business logic and then about coding. On top of this, I was pleasantly surprised when I realised that local microervices can help me solve two aspects that I hadn’t found a viable solution for long time:

• Internationalisation (Translations)
• Plugins

Internationalisation (Translations)

For ages, I have been looking for translation solutions in Delphi. There are some commercial libraries but they do not meet my number one criterion: cross-platform implementation. The other thing that makes me uneasy is that all the solutions require that you either include the ability to translate from the very beginning or, if you need it at a later stage, you need to go through all the components in forms and frames in your application and alter the code.

Local microservices opened a different possibility to me. The biggest problem when you want to translate a component is that each one has a very specific property that holds the translatable item; thus, we can’t devise a generic way to do the translation for all the components.

My approach is to create one microservice for each component that needs translation and register it with Pimlico. You can find the full demo in the repository (Demo.i18n).

pimlico.add('role: i18n, class: TLabel', Ti8nLabel.Create);
pimlico.add('role: i18n, class: TButton', Ti8nButton.Create);
pimlico.add('role: i18n, class: TRadioButton', Ti8nRadioButton.Create);

Of course, the actual translation of the text is done in a separate microservice which is, again, registered the usual way.

pimlico.add('role: i18n, cmd: translate', Ti18nTranslate.Create);

I don’t include here how the actual translation is done because it is very trivial and naive. You can, of course, see the code in the demo files.

Then, the microservice responsible to translate TLabel looks like this:

procedure Ti18nLabel.invoke(const aParameters: string);
var
  obj: TObject;
  component: TLabel;
  pointerStr: string;
  fromLang: string;
  toLang: string;
  poin: Pointer;
begin
  inherited;
  pointerStr:=extractValueFromParams(aParameters, 'Pointer');
  fromLang:=extractValueFromParams(aParameters, 'from');
  toLang:=extractValueFromParams(aParameters, 'to');
 
  poin:=Pointer(pointerStr.ToInteger);
 
  obj:=TObject(poin^);
  if (obj is TLabel) then 
  begin
    component:=TLabel(poin^);
    pimlico.act('role: i18n, cmd: Translate',
               'Text: ' + component.Text + ', From: ' + fromLang + 'To: ' + toLang, atSync,
                procedure(aStatus: TStatus)
                begin
                  component.Text:=aStatus.Response;
                end);
  end;
end;

A couple of things here worth noticing:

1. How does this microservice get access to the component that requests translation? We can only pass strings back and forth in Pimlico, so I supply the address of the component via a Pointer converted to a string
2. As the code shows, we can freely call other microservices within the invoke method of a given microservice. We need, however, to make sure the method is called under the correct synchronisation mode according to our needs

The services to translate the TButton and TRadioButton are very similar to the above code.

We need to do one more thing; to call the services from the main form to initiate the translation. Now, we can simply iterate through all the components and pass them to the services. This snippet can be found in the OnClick event of the Translate button in the main form.

  for num := 0 to self.ComponentCount - 1 do
  begin
    comp:=Self.Components[num];
    pimlico.act('role: i18n, class: *',
          'Pointer: '+IntToStr(Integer(@comp))+
          ', From: '+fromLang + ', To: '+toLang, atSync);
  end;

Notice how the services are called; Pimlico understands glob patterns.

Plugins

Plugins is the same story for me as internationalisation. As far as I know, there is no cross-platform library that offers this functionality. Again, to my surprise and satisfaction, microservices gave me a gateway to this.

As an example, suppose we have an application which provides a form for the options of the project (application) as in the figure below (Fig. 1).

Figure 1: Project Options Form

What we want to achieve is to write a plugin that adds a third check box in the project options form (Fig. 2).

Figure 2: Project Options Form with Additional Option

As a good Plugin API designer  😀 , I expose the IPluginProjectOptions interface that allows any external class (plugin) to manipulate the form. I will not provide here all the implementation details as it will make the already long post even worse. You can find the full source code in Demo.Plugin.Basic.

Open-source

A plugin for us is another microservice and the new option plugin is implemented in Plugin.ProjectOptions.NewOption unit under the class TPluginProjectOptionsNewOption. Then, we register the new class to Pimlico in the initialization part of the unit.

  pimlico.add('role: plugin, space: project-options, cmd: add-new-option', TPluginProjectOptionsNewOption.Create);

I have used the flexibility of the patterns in Pimlico to create a new domain (role: plugin, space: project-options) that any plugins that want to affect the project form must use. Another good job for the API designer 😆 . Finally, we access the plugin the usual way we have seen in the examples above.

The difference this time is that we do not care to do anything meaningful in the invoke method of the service. It is enough to only start it.

pimlico.act('action: start', '');

Of course, this really depends on what the plugin does. In this example, our plugin only adds a checkbox and an event when it is clicked but, in more sophisticated situations, I image that lots of work can take place in the invoke method.

Closed-source

If I haven’t grabbed you attention so far, here’s one more twist: looking at plugins as microservices which can be triggered and accessed via patterns, is a great example of loosely coupled code and, moreover, of code that can be loaded on the fly. 

Open-source plugins are great but they need to be available at compilation time. Closed-source code has the advantage of being accessed afterwards.

In Delphi, there are two ways to load external code to binary files: DLLs and Delphi Packages. DLLs lose the battle in my eyes as they are not cross platform. On the other hand, packages are.

In Demo.Plugin.BPL project, I have created a package that contains two sample “plugins”, i.e. revamped microservices based on IPluginBase interface.The plugin classes are, basically, the same as in the case of our open-source plugin. If you are playing with the code, make sure you build the bpl package before you look at how we load plugins at runtime.

If you go back to the Demo.Plugin.Basic project, you will find the implementation of a Plugin Manager. The manager loads all the available plugins (local and those in the package) in the FormCreate event.

procedure TFormMain.FormCreate(Sender: TObject);
begin
  PluginManager.LoadPlugins;
end;

Then, in the options form OnCreate event, we start all the relevant plugins.

procedure TFormProjectOptions.FormCreate(Sender: TObject);
...
begin
  ...
  PlugInManager.startPlugins('role: plugin, space: project-options, cmd: *');
end;

That’s it. Local and external plugins are loaded and executed (Fig. 3)

Figure 3: List of plugins

Finally, Delphi packages become useful outside the IDE: cross-platform and compiler-indepedent runtime plugins. Big time for the packages :P. Now they deserve all our love…

Source Code

The source code of the examples I use in this post can be found in this repository. Please feel free to check and let me know of any comments, omissions, errors, improvements you can think of.

Next on Microservices…

In this post, we looked at the idea of microservices, we briefly explored Pimlico -a toolbox to write microservices in Delphi-, how they can be implemented and some useful applications. However, we dealt with microservices that exist in the host application. In the next part, we’re going to look at how Pimlico can handle microservices scattered in the cloud. Stay tuned.