Introduction to distributed systems

One of the main benefits of this middleware is its ability to build distributed applications [http://en.wikipedia.org/wiki/Distributed_application] in quite an easy and reliable way.

However, until now, we did not take any network consideration into account in our sample applications. Indeed, in the quinoa.py file, the information related to the network was minimal.

 # ----- Physical Distributed System
 # ----- Domain
 domain1 = data.DomainDescription()
 domain1.name = ""                 <------- No specific domain name
 domain1.bin_directory = BINDIR
 domain1.var_directory = "/tmp"
 domain1.type = "NO_NETWORK"

 # ----- Machines
 H1 = data.HostDescription()
 H1.domain = Domain1
 H1.name = "localhost"             <------- localhost i.e. 127.0.0.1

Working on a generic unspecified domain on the local machine was convenient for an introduction to basic concepts, but it is now time to escape from the sandbox and envision what real life looks like.

The current lesson introduces some general concepts about distributed systems and exhibits the capabilities of the middleware in terms of distributed application management. A way to distribute our sample application is examplified in another lesson.

Illustrative example

When dealing with distributed applications, one can consider several machines and several domains. The latter could be physical domains or virtual domains. We provide hereafter an example to illustrate the flexibility provided in this way.

The following description is extracted from the quinoa.py file of another application.

Domains

 # ----- Physical Distributed System
 # ----- Domain
 Domain1 = data.DomainDescription()
 Domain1.name = "intra.cea.fr"
 domain1.bin_directory = BINDIR
 domain1.var_directory = "/tmp"
 Domain1.type = "The linux cluster"

Domain1 is declared as the cluster of Linux machines within the intranet of the CEA. This description includes the path where the objects running on the machines of this given domain may find the required libraries.

The following description declares now the cluster of Windows machines.

 Domain2 = data.DomainDescription()
 Domain2.name = "intra.cea.fr"
 Domain2.bin_directory = "c:\Program Files\middleware\devel"
 Domain2.var_directory = "c:\Program Files\middleware\Tests\trom"
 Domain2.type = "The windows cluster"

You may notice that it uses the same domain name as the previous one: intra.cea.fr. But both descriptions differ on their type definition. This type property (a simple string) is used to define what is considered as an independent virtual domain. Indeed, you can see that each cluster defines its own path for the required libraries, and that this path obeys the dedicated notation scheme of the related operating system.

Following the same principles, we may declare a laptop that shares the same domain name but has its own installation of the middleware on its local drives :

 Domain3 = data.DomainDescription()
 Domain3.name = "intra.cea.fr"
 Domain3.bin_directory = "/Local/cea/workspace/middleware/devel"
 Domain3.var_directory = "/Local/cea/workspace/middleware/Tests/trom1"
 Domain3.type = "a laptop"

Finally, we declare another new domain, simply by defining a different domain name value :

 Domain4 = data.DomainDescription()
 Domain4.name = "cea.net"
 Domain4.bin_directory = "/export/cea/workspace/middleware/devel"
 Domain4.var_directory = "/export/cea/workspace/middleware/Tests/trom1"
 Domain4.type = "my personal machine"

All these domains are used in the description of the machines involved in the considered application.

Machines

 # ----- Machines
 H1 = data.HostDescription()
 H1.domain = Domain1
 H1.name = "caramel"

 H2 = data.HostDescription()
 H2.domain = Domain1
 H2.name = "nougat"

 H3 = data.HostDescription()
 H3.domain = Domain1
 H3.name = "praline"

We declared here three machines running off the Linux cluster. Their names are caramel.intra.cea.fr, nougat.intra.cea.fr and praline.intra.cea.fr. They all use the libraries declared at domain level.

 H4 = data.HostDescription()
 H4.domain = Domain2
 H4.name = "granier"

One Windows machine : granier.intra.cea.fr.

 H5 = data.HostDescription()
 H5.domain = Domain3
 H5.name = "lempa"

One laptop, lempa.intra.cea.fr, which uses its own installation located on its local disk.

 H6 = data.HostDescription()
 H6.domain = Domain4
 H6.name = "hanarpatcha"

 H7 = data.HostDescription()
 H7.domain = Domain4
 H7.name = "n810"

One machine with the logical name hanarpatcha.cea.net and a tablet nokia n810 called n810.cea.net

Ultimately, we end up with 7 machines (hosts) attached to 4 distinct domains.

Minimization of heterogeneity annoyances

Distributed applications may involve a wide variety of heterogeneous systems, including several operating systems and many types of hardware components ranging from small devices to servers. As you may know, there is no universal way to launch a software component on these multiple types of different equipments. We may complain about this but, as this constraint exists, we have to deal with it.

This middleware is provided with a minimal bootstrap, dependent of the equipment and often specific to it. This minimal bootstrap, called daemon (or demon), has to get started on the equipment, manually or with the help of scripts or services (see below). The purpose of this daemon is to level and hide most of the differences existing between the distinct system components. As a benefit, once and while this daemon is running, the system does not have to be considered as heterogeneous any more, and the same components are able to run on any of the available equipment.

As a consequence, the last problem we have to solve to minimize the difficulties induced by heterogeneous systems is now reduced to the launch of this minimal bootstrap. This can be done in different ways:

Launching the bootstrap daemon on UNIX machines

By hand

You will probably never have to use this approach, except on devices that you cannot access through ssh for some security reasons.

In this case, you just have to log on to the target machine and run the following commands:

 cd *<middleware_install_dir>*/src/middleware/binaries/
 python demon_server.py *<port>* *<key>*
  • **** is the repository in which you have installed the middleware (in general *~/middleware_install*
  • **** is a port number on which the *demon* listens to.
  • **** is a string used as identifier and shared between the *quinoa.py* and the *demon*.
  • This key is not defined for security reasons but only to avoid the unwanted use of another user's demon.

By default the value of port in 16000 and the value of key is key and, if you are alone using the middleware, you can just keep to that.

In case several people in your organization use the middleware, it is safer to attribute a pair ****, **** to each user in order to separate the different applications running on the same machines.

Through ssh

When you start an application using a standard quinoa.py file ( with a command such as python quinoa.py --start_objects ...), the system first checks if the required demon is already running. If no demon is detected, then the system tries to start the demon on the remote machine, using the classical unix command ssh. (See http://en.wikipedia.org/wiki/Secure_Shell for more details).

Basically, the command launched from the quinoa.py file is something like this:

ssh hanarpatcha.cea.com -n -l cea /usr/bin/python /usr/local/src/middleware/binaries/demon_server.py 16000 key

If the system does not ask for any password, then your are done! This means that authentication through a public key is already in place. Otherwise, the system will ask you for the corresponding password each time a demon is started through ssh. Even if this solution may be used, it is not really suitable for real-life applications. In this case, you should set up the public key scheme as described hereafter:

First you have to generate a pair of keys (leave the password blank when asked):

ssh-keygen -t rsa

Next, you have to copy the generated public key onto the remote machine(s). This is done by copying the content of the file named ~/.ssh/id_rsa.pub of the local machine (the one from which you execute the quinoa.py file) into the file ~/.ssh/authorized_keys of the remote machine(s). More explanations may be found here on ssh.

Once the public key is propagated, the demons start transparently and automatically each time they are required.

Note : The demons stop automatically when they are idle for more than one day. Then you can just forget about them.

Launching the middleware bootstrap daemon on WINDOWS machines

TBC