• Integration – as a key concept

(1)

2019| Concurrent and Distributed Programming - © Lenuta Alboaie

Integration in Distributed Systems

Lenuța Alboaie [email protected]

“Alexandru Ioan Cuza” University of Iasi Faculty of Computer Science

1

Course Master

I+II

(2)

Summary

• Integration – as a key concept

• Integration and/or Interoperability

• Integration contexts

• Service Oriented Architecture

• Evolution of Integration Technologies

• ESB - Enterprise Service Bus

• Technology mapping: SOA, ESB, BPM

• Integration Through Cloud Eye

• Integration Roadmap

2

(3)

Integration

• Continuous Integration [DMG07]

• coined by Kent Beck, as part of Extreme Programming [Bec04]

• methodology employed in software development intended to increase software quality and also to adapt dynamically to the client requirements

• refers actually to a first step in a Continuous Delivery process, which assures that the work-in-progress code belonging to a project is continually integrated with code belonging to others developers from the same project

3

(4)

Integration

• “connecting computer systems, companies or people” *HW03+

• “Integration services are detailed design and implementation services that link application functionality (custom software or package software) and/or data with each other or with the

established or planned IT infrastructure. Specific activities might include project planning, project management, detailed design or implementation of application programming interfaces, Web

services, or middleware systems.” *GarI]

• Integration in our discussion has consistent tangents with

Enterprise Service Bus (ESB), Integration Platform as a Service (iPaaS) or Data Integration.

4

(5)

Integration and/or Interoperability

• interoperable systems, there is an environment in which sharing the information is performed without the need of any middleware

• Issues: a(n) change/upgrades of a party can cause

interoperability issues or even the end of communication

• interoperability implies general principles of communication rather than a technical specification

• integration area ensures a full functional systems that provides backwards and forwards compatibility with various versions of the product, a reliable data transfer, automated maintenance et.al.

interoperability offers an exchange and integration offers a full functionality

5

(6)

Integration contexts

• Integration in a context (e.g. enterprise, applications or services in cloud or on- premise) suffers changes in accordance with the needs which appear in time:

new services/extra applications, new technologies, new mechanisms of storage provided by a third party.

• Instruments:

• ESB (necessary for Application to Application(A2A) or Service Oriented Architecture Systems (SOA)),

• Business Process Execution Language (BPEL) for process integration

• Integration platform as a service (iPaaS) (used for A2A or SOA Services in cloud)

• Internet of Things (IoT Systems - Channels)

• Application programming interface management (API management) (used for microservices)

6

(7)

Service Oriented Architecture

• Service Oriented Architecture is a design methodology and architecture aimed at maximizing the reuse of multiple services (possibly implemented on different platforms and using multiple programming languages)

– Services share a formal contract – contains details (and additional metadata) such as their capabilities, interfaces, policies, supported protocols

– Services must be loosely coupled

– Abstracting functionality and technology

– Reuse principle promotes the idea of service as a “repeatable value”. When a business logic is built, reusing it decreases the time of creation

– Service Discoverability

– Services can be composed to provide other services – composability – Services are individually useful – they are autonomous.

• =>predictable performance, independence of the service, flexibility, scalability, ability to be easily replaced, fault tolerance and, of course, reuse

– Service stateless

7

(8)

Service Oriented Architecture

• Open Group SOA Integration Maturity Model (OSIMM) proposed seven levels that should be respected by a SOA application in

order to reach “maturity level”

– E.G. a point-to-point integration is considered an integration service with 2 value (Service Integration Maturity Model - SIMM = 2) or a hub-and spoke architecture corresponds to SIMM =3

• Obs. For small-scale systems, redundancy or even refactoring and consolidation may not be so expensive, for large companies these can spell total failure

8

(9)

Service Oriented Architecture

9

Figure: Point to point explosion in number of connections

Figure: Point to point integration

(10)

Evolution of Integration Technologies

• In any Enterprise, business processes need an integration layer that allows communication among various applications.

• Example: in an ERP (Enterprise Resource Planning) system, if components such as CRM (Customer Services), Human Resource, Production (PLM), Sales et.al.

cannot communicate, than internal processes cannot be automated, ultimately leading to inefficiency. Therefore, the integration solutions for an Enterprise are known as Enterprise Application Integration [Cla15].

• Mechanisms for integration have been proposed [HW03]:

– File Transfer

– Shared database

– Remote procedure invocation scenario – Integration through messages

10

(11)

Evolution of Integration Technologies

File Transfer: an application A produces and exports a file that is imported and processed by a B application

In this scenario, there are some issues that should be discussed:

• when it is the ”right” moment to export/import the file, which also implies the transfer frequency.

Whenever you have to deal with a large amount of data, some processing power is needed making such a decision not so obvious.

• there should be a convention in place regarding the file name, its unicity and its format (XML, JSON)

• there should be rules regarding “time-of-life” for a file and when it is the right moment to delete/update

• there may be situations in the B application when multiple components need some parts of the imported file. In this case, some synchronization mechanisms are required.

All these aspects imply that a massive amount of work is required from developers; there are some benefits linked to this method:

• No external integration tools are necessary

• Applications are decoupled: changes inside the applications are allowed, on condition the data is provided in the same format

• The integrators do not need to know details regarding applications’ architecture or implementation details

11

(12)

Evolution of Integration Technologies

Shared database: multiple applications store data in a shared database

The benefits of this approach are:

• Inconsistencies due to faulty synchronisation mechanisms are avoided through database transactions

• Semantic dissonance (data appears to be the same but has different meaning) is avoided through the existence of data schema for SQL-based relational databases;

There are, however, some issues that may affect an Enterprise’s business process:

• Designing a database for multiple applications is difficult

• This architecture is prone to bottlenecks (if many reads/writes operations are needed) and deadlocks (if an application performs locks, the rest will not be able to perform operations on data)

• The system extension (what happens when a new Application is needed?)

• A change in one application determines changes in the database affecting the other applications.

Therefore, in this model unencapsulated data [HW03] are a drawback to maintain or integrate future applications.

12

(13)

Evolution of Integration Technologies

• Remote Procedure Invocation: is based on the encapsulation principle and interfaces in order to assure integration

This approach has advantages such as:

• Due to encapsulation, applications can expose multiple interfaces to the same data. Thus, a 1 to n type relation with the clients is possible with this model

• From the implementation point of view, developers are accustomed with procedure calls and there are many standards for RPC (Remote Procedure Call): CORBA, COM(Component Object Model), .NET Remoting, Java RMI. Also, Web Services [AB06] with associated standards comply with this paradigm.

Usually, these are HTTP, presenting more flexibility from the communication perspective (e.g. easy to get through firewalls).

Some issues brought by this approach are linked to some fundamental factors that should be considered for integration:

• the main weakness of this model is determined by the fact that remote calls are slower and packages can be lost in networks. Furthermore, if there are multiple applications communicating through RPC mechanism, it is possible for one failure to trigger failures in cascade.

• applications are somehow tightly coupled, because they should know details about their interfaces;

any change determines other changes

13

(14)

Evolution of Integration Technologies

• Messaging allows “to transfer packets of data frequently, immediately, reliably, and asynchronously, using customizable formats” *HW03+.

14

Figure: Integration through messages

(15)

Evolution of Integration Technologies

Messaging Systems or Message-oriented middleware(MOM) advantages

• the asynchronous mechanism changes the approach when you plan to design distributed applications (e.g. the communication - the applications may not be up in the same moment, the remote call can take some time et.al.)

• decoupling from File Transfer is achieved by the fact that messages can be transformed, broadcasted to various receivers without the transmitter and recipient knowing it; therefore the integration level is separated by the application development

Issues

• event messages are sent fast while we still have the networks lagging, therefore some updates may not perform quite simultaneously

• testing and debugging in an asynchronous environment is not an easy feat

• some “glue code” in usual cases cannot be avoided in order to fit everything

• from the programmatic point of view, the employed mechanism is not familiar to the programmers any more, so there is an increased level of complexity.

15

(16)

Evolution of Integration Technologies

• All these solutions have been used to ensure integration and interoperability between disparate systems of an Enterprise, therefore we reach the Enterprise Application Integration.

• Topologies that were used in EAI started from Point-to-Point Integration, Integration Broker model and evolved to Enterprise Service Bus solutions.

• In Point-to-Point Integration or Vertical integration [SW93] manner, there are special components (called connectors) that are in front of the application intended to interoperate with others. This connector is implemented for each application pair. This approach is also known as hub-and-spoke topology.

16

- No scalability (each adapter/connector is bundled to a specific data format or transmission mechanism, any attempt to add new applications implies individual implementation of a connector)

- Reusing some parts to create new services is not possible

- This mesh network implies a huge maintenance effort

(17)

Evolution of Integration Technologies

• Integration Broker or Star Integration model comes with a central engine/central hub or broker that provides message flow, message transformation or message routing.

• In comparison with point-to-point, this model performs decently in case of loose coupling (e.g. there may be situations when the sender does not know the receiver’s endpoint) and asynchronous communication is encouraged

• The broker manages any message format, therefore from the application perspective, there is no need for a canonical format.

17

- Having a central point is obviously leading to scaling problems. The system can maintain its state in a shared database and it is possible to scale if the central hub is for example clustered – but is not a flexible architectural alternative

- Heavy load of transactions can lead to bottleneck

- Difficulty of implementation and administration of a central hub, in case you intend to integrate products belonging to different companies (heterogeneity).

(18)

Evolution of Integration Technologies

• Taking all these drawbacks into account, it was necessary for a system to offer more functionality than previously mentioned (e.g. transaction security, error handling mechanisms) or to entirely avoid the issues by a central integration logic, and therefore Enterprise Service Bus or Horizontal Integration [SW93] have become reality.

• Service bus is different from message buses because it supports patterns such as publish/subscribe, request/reaction, request/reply and also SOA concepts like contracts.

Examples of such service buses are: Apache CXF, Apache Camel, Windows Azure service bus, MuleESB, WSO2, Talend ESB, SwarmESB (UAIC)

18

Figure: Messaging patterns [HW03]

(19)

Evolution of Integration Technologies

Example of such patterns:

• Message Channel represents a logical channel among applications where (one writes and the other one reads that piece of information)

• Message Bus pattern is a “connecting middleware” between applications similar to a communication bus in a computer that allows communication between memory, I/O devices and CPU.

• In a Message Broker pattern, the received messages from multiple senders reach a central point that controls the flow and route messages to the correct channel. Using a MessageBroker as central point leads us to the hub-and-spoke architectural style.

• Messaging Endpoint contains code for application and messaging system. Therefore, applications know little about the message path, format or underlying communication details.

• For an Event-Driven Consumer or asynchronous receiver, receiving a message causes/triggers the receiver to perform an action. For example RabbitMQ consumers are using this event-driven pattern.

• Control Bus uses the same messaging mechanism used by applications, but separate channels are used for management control messages.

• Message Sequence allows transmitting large amounts of data in form of chunk data that have a unique identification field.

19

(20)

Evolution of Integration Technologies

20

Figure: Google Cloud Pub/Sub [GooglePS]

(21)

Evolution of Integration Technologies

• In a Publish-Subscriber pattern, one input channel can split in multiple output channels that has just one corresponding subscriber that consumes a message, which will be deleted. There are many flavors of Publish-Subscriber pattern, for example in Google Cloud Pub/Sub, they offer this mechanism but there are also features as Topics and Subscriptions [GooglePS].

• An application (the sender) publisher sends a message to a topic. The receiver creates a subscription to a topic in order to receive messages.

The detailed steps, from an implementation point of view are:

• Sender creates a topic in Google Pub/Sub service and sends messages to that topic; a message has a payload and attributes that characteristic the payload; the messages are stored persistently until an acknowledgment is received

• The Pub/Sub service forwards the message to all subscriptions, individually, irrespective of the mechanism used by the receiver ( Pub/Sub pushing operation, or receiver/subscriber pulling)

• The subscriber sends acknowledgment for the received message and the Pub/Sub will delete the message

For Google Cloud Pub/Sub this asynchronous communication that decouples senders and receivers, can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many. In [GooglePS], some basic uses of this Pub/Sub pattern are depicted: balancing workloads in network clusters (multiple tasks can be distributed among multiple Google Compute Engine instances) or logging to multiple systems (an instance can write logs in a database, these logs can be interrogated latter by other applications).

21

(22)

Evolution of Integration Technologies

• A Message Router consumes a message from one MessageChannel, without changing the content, and sends MessageChannel to other(s)

• Pipe-and-Filters pattern allows receiving messages, performing various operations on it (filtering operations such as: decryption, deduplication et.al.) and these steps are connected through pipes

• Command Messages is a pattern that allows to invoke a procedure in another application.

In SOAP protocol [AB06], a request SOAP message can be considered a Command Message

• Envelope Wrapper pattern performs an encapsulation of application data in an envelope that is compliant with the messaging infrastructure. A good example of this pattern use is SOAP messages

ESB also includes the functionality of a bus service and may well constitute the base for a Service Oriented Architecture (SOA). It provides fundamental services to develop more complex architectures, for complexity hiding, data access streamlining, routing messages, transforming data, executing transactions, orchestrating services or enabling “publish- subscribe” mechanisms, etc. Usually, ESB functionalities are packaged as single products

22

(23)

Service Oriented Architecture

23

Figure: Using ESB in a SOA architecture

(24)

ESB - Enterprise Service Bus

• The notion of “Enterprise Service Bus” was introduced by Gartner Group in 2002 and thoroughly presented in the book ”The Enterprise Service Bus” by David Chappell [CHa09].

• ESB has the same goal like the previously integration models and represents a collection of principles, rules and features that create the environment for an architectural layer providing a bus based integration among various

applications/services

24

(25)

ESB - Enterprise Service Bus

• Each application has an adapter that assures communication among applications using bus functionalities.

• The adapter in an ESB is a “smart” one and includes many features:

transformation, marshalling/unmarshalling, security et.al. From this point of view an ESB is different from a broker model.

• Therefore, in an ESB there is a good level of decoupling, applications being able to communicate without having knowledge about others on the bus or about particular characteristics of one application (e.g. message format

et.al.)

• In an implementation, a service bus can be deployed in multiple ways [RD08]

25

(26)

ESB

• The model from Figure resembles point-to-point or hub-and-spoke model, but in an ESB implementation, the adapters are smart connectors. An adapter uses two queues for inbound/outbound messages and offers many features (e.g.

transformation, security et.al.).

• The older integration models used in EAI, were implemented as a monolithic stack but in an ESB we have coordinated service interaction.

• This model is easy to manage (program, debug) but the bus is bottleneck and we have limited scalability (just scale up solution is possible in this case).

26

Figure: ESB as a coordinator for services integration

(27)

ESB

27

Figure: A P2P network of ESBs

In this case services have their own bus, and these are connected in a peer-to-peer network.

Figure: ESB in a Federated setup

a service bus serves a limited number of services and multiple buses are organized in a “federated setup”. Even a service is tight to one bus, it can communicate with services connected to other buses from the federation.

Obs. In both cases we have scalability but high complexity for configure and debug.

(28)

ESB Functionalities

28

Figure: Functions of an ESB [KMNSST13]

(29)

ESB Functionalities

Operations and Management

The services from this category allow reliable operations and management of an ESB.

• Statistics & Status provides ESB services’ statistics such as: the number of system errors, the minimum and maximum response time or statistics link to the number of processed messages.

• Alerting provides a mechanism to send warning messages through different channels.

• SLA Rules represent rules that may be conceived based on the data provided by the running Statistics & Status components. This enables SLA (Service Level Agreement) monitoring and measuring. Any SLA breach is communicated using the warning

component.

• Message tracking enables de possibility to follow the messages sent by ESB and it should be activated whenever required to reduce the system load.

• Message Redelivery ensures that the messages not immediately processed are re-sent automatically after a previously defined period of time. The number of tries and the interval between them are also configurable.

• Endpoint Failover enables the possibility to specify a service provider to be called automatically in case the main provider is not available.

• Load Balancing is a component enabling load distribution to several equivalent services providers.

29

(30)

ESB Functionalities

Operations and Management

• Message Throttling is a component allowing to set the maximum messages load to be sent to a specific service provider on a given amount of time. This manner, the overloading of service provider and ESB waiting queue is avoided. This component enables the prioritisation of messages, in order to process first the higher priority messages.

• Logging & Reporting enables the messages’ recording, in order to be displayed later. It may also provide audit functions.

• Configuration Management makes it possible to modify a running ESB configuration without loss of integrity. A history log of these changes may be stored in order to reset a service to a previous state, at any given moment.

• Service Registry enables recording and management of an ESB services.

• High Availability ensures that the services provided by the ESB run safely regardless of the state of the server they are running on.

• Error Hospital is the destination of messages unable to be processed after multiple tries.

Here, they may be visualized, corrected if necessary and reprocessed.

• Deployment provides the possibility to install an automated service on ESB.

• Service Usage monitors and registers the services used by the user, in order to be paid. It’s similar with the metered billing used in Cloud Computing.

30

(31)

ESB Functionalities

Mediation

The services from this category correspond to operational components employed for the implementation of messages’ flux in an ESB.

• Message Routing enables the messages to be routed to a certain service depending on their content. It may also split a message so it can be sent to several services, or it may combine various messages in a single one.

• Message Transformation enables the conversion of a message from one format to another (e.g. text in binary, CSV in XML/JSON etc.).

• Service Callout enables the possibility to access other services from an ESB messages flux. A service may be a Web Service, but the ESB may call directly a code locally installed on the ESB, such as a method of a Java class.

• Reliable Messaging supports message transfer using queues or WS-* standards, such as WS-ReliableMessaging.

• Protocol Translation enables the transition from one communication protocol to another, such as TCP/IP to HTTP or HTTP to WS.

• Message Validation ensures that the messages are valid. In case of a message in XML format, this means it verifies that the message is well formed and complies with a XML schema or with a WSDL [AB06].

31

(32)

ESB Functionalities

Mediation

• Message Exchange Pattern (MEP) enables various models: synchronous or asynchronous, allows one direction calls or publish/subscribe mechanism.

• Result Cache enables the possibility to save to a cache the returns of a service call, in order to allow subsequent service calls to receive the answer from the cache, without the need for the service to be called again. This is applicable mainly to static data that rarely change.

• Transaction enables atomic operations in an ESB: a message is completely processed or not at all. There is no possibility for the message to be partially processed.

• Message Resequencing enables the messages that are together but not in the proper order to be reordered. Such a component comprises an internal buffer that processes the messages until the complete sequence is available and may be sent.

• Pass-Through Messaging ensures efficient message transmission to the ESB. This service may be especially useful if the ESB is employed for services virtualization and the messages are sent from the services user to their provider without modification. In this case, the ESB does not process the message, simply transmitting it along.

32

(33)

ESB Functionalities

Security

The services from this category enable security through its various components, both on transport level and on message level.

• Authentication authenticates services consumers whenever they access

ESB services and verifies the ESB authentication for services providers.

• Authorization provides a user or role authorization system in order to get

access to services.

• Security Mediation provides support for interactions communicating

outside security domains, by converting authorization data from one domain to the corresponding specific data from other security domains.

• Encryption/Decryption

provides support to encrypt and decrypt the content of a message.

33

(34)

ESB Functionalities

Adapters/Transport

• This module includes adaptors required to connect ESB provided services through the Hosting Services. An ESB provides a predefined set of adaptors, but enables the developers to create their own adaptors depending on customer’s specific requirements

Service Hosting

This module enables services to be installed and run directly on ESB and it is usually required in case the ESB is based on an application server.

• Service Container provides one or more containers wherein the services may be installed and managed.

• Component Model provides a model of abstract components, such as Java EJB, Java Spring Framework or Microsoft COM+, depending on how the services are developed ([KMNSST13],[IBM-ESB], [IBM09], [Mule], [ESBPr]).

34

(35)

ESB in practical scenarious

• Let there be a certain client and service provider. The latter provides a web service for the client to employ. In such a straightforward case the client may connect directly to the web service through the interface it provides. However, an ESB may be employed to hide the service interface to the client. As such, should there be any modifications of the web service, the ESB may be easily set up to meet the client’s requirements without the need for modifications on the client side. Furthermore, the ESB property to monitor the activity inside it for statistics purposes – such as the number of service calls per period of time, compliance to SLA, etc.- may be used as well.

35

Figure: Decoupling and Monitoring

(36)

ESB in practical scenarious

• Let there be a certain client and service provider. The latter knows how to communicate through SOAP and the client through REST. Usually, the client should create a proxy able to communicate through SOAP and to answer through REST. In this case, if the service provider changes the interface, the client must modify his proxy in his turn, in order to use the new interface. On the other hand, if an ESB is to be used, the client is not required to modify his proxy in case the service provider changes the interface, but rather the ESB should be reconfigured to transform the provider’s result in a known format of the client.

Thus, the client is not to worry about modification on the provider’s side, while the provider must not notify all his customers whenever he implement modifications.

36

Figure: Services interoperability

(37)

ESB in practical scenarious

• There can be cases when similar services can be supplied using different protocols. We can have critical client applications requiring certain services, therefore without an ESB the client is forced to implement two or multiple interrogation modules or the provider is forced to keep up multiple interfaces.

An ESB can behave as a mediator among clients and various service providers, and therefore not only the Client will not be affected, but also the provider interfaces will remain the same. There are situations when you have no direct control/contact with the service provider therefore an ESB can come with the adapter and the provided services can be used. Furthermore, other services compatible with other protocols can be integrated with the ESB, and changes will remain at this level only

37

Figure: General Integration Mechanism

(38)

ESB in practical scenarious

• As any software system, services can suffer changes in time and new versions compatible with the new standards can appear. An ESB can be used to perform the transformation from one version to another version. Due to resource

consumption, keeping up multiple versions of the same service is almost impossible, therefore an ESB can be used for this purpose.

38

Figure: Versioning

(39)

Technology mapping: SOA, ESB, BPM

39

Figure: ESB as a part of SOA and BPM landscape

(40)

Technology mapping: SOA, ESB, BPM

40

Figure: Stack layers for an lightweight ESB

(41)

Technology mapping: SOA, ESB, BPM

• Business Monitoring refers to services as aggregation, analysis and presentation (desirable in real time) information about inside activities that involves the owner, but also customers or business partners. At this level we have a dashboard that provides information about activities, performances, correlation among business processes or possibility to integrate with other monitoring systems.

• At the connectivity level we can use a MessageBroker pattern that will assure the messages flow. The MessageBroker can be embedded in the ESB or can be

external.

• The routing level contains mechanisms which determine what messages must be sent and where. This level can be provided using some protocol translator, a router module with usual functionality, and all configuration can be done using Spring- XML or a DSL (Domain Specific Language) [Fow10] in our case. At this level, some messaging patterns or others newly created over the existing ones can be used.

• At the transformation level there are no specific rules, you can meet “no business at the integration level” (and the Canonical Data Model pattern can be used) or this decision depends on Enterprise and contexts.

41

(42)

Technology mapping: SOA, ESB, BPM

• Orchestration level includes two sublevels: technical orchestration and business orchestration [Fer11].

• In *Rot12+ orchestration is defined as a pattern: “Orchestration—Make business processes agile and adaptable while using services based on the Request/Reply or Request/Reaction interaction patterns”

• Technical orchestration sublevel has direct link with the levels below and its

implementation “derives from the technical design of the flow (complex routing, errors management, etc.).”

• Business orchestration sublevel offers the base for Business Rules Engine and therefore, it is the bottom-up implementation of the business processes.

• When sophisticated business process definitions are necessary, the Business

Orchestration sublevel may contain implementation for stateful processes (which can span over long periods of time) or may contain support for parallel executions and eventually merging mechanisms based on conditions. In this context,

paradigms like SOA (Service Oriented Architecture) emerge and play an essential role in building bridges between business and IT departments.

42

(43)

Technology mapping: SOA, ESB, BPM

• It’s well known that “processes run on services” *Jen12+ and therefore BPM (Business Process Management) and SOA coexist.

• Gartner describes in [GarBPM17] Business Process Management as “a discipline that uses various methods to discover, model, analyze, measure, improve, and optimize business processes. A business process coordinates the behavior of people, systems, information, and things to produce business outcomes in support of a business strategy. Processes can be structured and repeatable or unstructured and variable. Though not required,

technologies are often used with BPM.”

In [Sal11] main concepts linked to BPM are defined and we are presenting them below:

• Process - “a set of order actions that tends to transform an input to a desired output”

• Business - “relative to a domain, company or a scoped scenario, where certain rules and actions needs to be applied to achieve a goal”

• Business goal - “objective to be satisfied inside the business scope that gives to the company a profit/benefit in some way”

• Business process - “Sequence of actions performed by humans and systems to achieve a business goal”

BPM contains some stages that are intended for Business analyst or for Technical staff, which are involved in a “forever” iterative cycle

43

(44)

Technology mapping: SOA, ESB, BPM

44

(45)

Technology mapping: SOA, ESB, BPM

• “BPM is a way of building operational solutions and SOA is a thought model that helps decompose complex problems into well-defined and reusable components.”*Jen12]

• From top to down view, a BPM offers an upper integration level, because it facilitates interaction between systems and human actors.

– Usecase: analysis of an enterprise environment seen from two different perspectives

• From a software architect’s/engineer’s perspective, in order to create an environment that offers services decoupling, component reutilization, no visible details regarding communication, a combination of ESB and SOA is a successful solution. But if there is a need for a new application in the company (or an integration with a legacy application), there is a good chance that a new application will be created for end users.

• From the engineer’s point of view there is no problem, but for end users this means:

multiple logins (there may be multiple users’ accounts or the need to provide the password twice), or performing the same task in different systems. Also, for new end users there are many things to learn and trainings are time consuming.

• BPM offers to users (end/new users, managers) a proper environment with a Unified User Interface to do a specific task and for developers an integration environment, to coordinate systems and human interactions, to develop or add new functionalities in a unified way

45

(46)

Technology mapping: SOA, ESB, BPM

46

Figure: Mapping of ESB and SOA and BPM

(47)

Integration Through Cloud Eye

• SOA and ESB were pertinent solutions for integration, but the high cost of implementation has determined many organisations to use old point-to-point EAI patterns.

• The software systems as a whole tend towards XaaS and a new era for application programming interface (APIs), and all these imply a reposition of the integration process in an enterprise (both at

business and at technological level).

• The integration technologies in cloud, sometimes called SMAC

(Social, Mobile, Analytics, Cloud) even if they benefit from previous technologies, are prone to suffer radical transformations.

• We are not talking anymore just about intra-organisational

integration (where an ESB can play a perfect role), instead we talk also about inter-organizational integration.

47

(48)

Integration Through Cloud Eye

• It is possible to use an ESB for the inter-organizational integration, but there are some drawbacks [Jos14]:

– Installing integration environments involves many supplementary

environments, leading to lost time required for configuration and installation – Imagine that something is changed at one endpoint. In an ESB necessary

changes are required at levels that belong to flows touching the endpoint

• iPass Models

• Gartner provides in [GariPaaS17] a precise definition for iPaaS (Integration Platform as a Service: it “is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations”.

• iPaaS is still at the beginning of its evolution, and Gartner mentioned in 2017

different classes of integration providers, which probably do not cover all features an iPaaS can offer: B2B integration, cloud integration, ESB and SOA infrastructure [MuleiPaaS17].

48

(49)

Integration Through Cloud Eye

• In the world of everything as a service (XaaS) even integration is a Service on cloud and that means there is no capital expenditure but just operational costs for these services’ clients.

• To allows a rapid integration, iPaaS providers preconfigure and build connectors for well known SaaS endpoints. Also some pre-packaged integration templates are provided in order to reduce overall integration effort.

• Models for integration supported by an iPaaS – On-Premise - Cloud Integration

– Cloud - Cloud Integration

– In the case of mobile device management (MDM), integration deployment can happen on cloud in order to allow access to various software (applications, web services, databases). We are mainly in a cloud-cloud integration model.

– In the Internet of Things for integration case we deal with high-volumes of data, and a proper management can be done with proper resources that can be

found in cloud, therefore in most cases, this kind of integration happens at cloud level

49

(50)

Integration Through Cloud Eye

• The integration endpoints are on-premise and therefore the integration deployment (flow/tasks) happens on-premise

50

Figure: On-Premise - Cloud Integration

(51)

Integration Through Cloud Eye

• The integration endpoints are on cloud and therefore the integration deployment happens on cloud

51

Figure: Cloud - Cloud Integration

(52)

Integration Through Cloud Eye

iPaaS comes with advantages, that are characteristics in general for SaaS:

• No capital expenditure for an integration infrastructure

• No installation, versions, upgrade to take care of or manual processes

• No maintenance costs

• Existence of pre configured connectors/adapters and drag and drop tools may ease the integration process

• High availability, load balancing

• Security, reliability for data transfers

• Takes advantage of cloud associated technologies: Dynamic scaling, containers, resource management

The limitations of iPaaS are somehow similar with those specific to cloud systems:

running on someone else's hardware, vendor lock-in, issues regarding security, privacy, governance

52

(53)

Integration Through Cloud Eye

These factors can be considered as essential issues when someone wants to design and develop an iPasS system [GPGTIW17]

• Degree of "cloudiness" - means that aspects specific to cloud such as tenant isolation, elasticity, dynamic scaling, self-service, billing mechanism should be provided

• Enterprise worthiness - aspects such us high availability, disaster recovery, technical support and secure access should be offered

• Functional completeness - represents the range of services provided by the iPaaS: “core integration capabilities (multiprotocol support and bridging; multiple data/message delivery styles; data/message validation, transformation and routing), adapters, data quality, development tools, administration, monitoring and management environment, support for secure communication, governance/API management, and community collaboration/crowdsourcing facilities”

• Openness - an iPaaS system should offer extensibility, support for open standards and integration with on-premise application or on-premise integration platforms

53

(54)

Integration Through Cloud Eye

These factors can be considered as essential issues when someone wants to design and develop an iPaaS system [GPGTIW17]

• Productivity - an iPaaS platform should make available documentation, examples and a repository with instruments that enable rapid integration for various contexts

• Ease of operation - this aspect contains two types of functionalities. The first category refers to the fact that an iPaaS system should provide a HUI interface or at least a command line interface to ease integration operations. The second category refers to instruments that assure backward compatibility in order to accomplish an integration flow.

• Citizen integrator support - as we have seen for ESB, there are users without technical skills but with well developed business skills. An iPaaS platform is supposed to provide support to a suite of “non-coding” tools.

• Versatility - an iPaaS should offer support for emerging integration requirements that many times provoke changes at Business level with effects at technical level

54

(55)

Integration Through Cloud Eye

55

Figure: iPaaS Gartner Magic Quadrant [GPGTIW17]

(56)

Integration Through Cloud Eye

Representatives solution:

• DellBoomi offers the AtomSphere platform that supports many types of integration and some of them are in trend with new evolving technologies: IoT integration, MDM (Mobile Device Management), EDI Services (Electronic Data Interchange), API management, B2B integration. Boomi offers a visual design interface that hides complexity and allows developers to develop web services or various SOA architectures. Also, it provides the possibility to build integration using pre-built connectors and processes and friendly drag-and-drop tools.

• In [Sel16], a cloud-to-cloud integration example is presented, in which two SaaS applications (Oracle EBS and Salesforce CRM) are connected through AtomSphere. This solution was chosen by A10Networks, a well known company in networking and security. Because they experienced fast business growth they chose Oracle E-Business Suite as ERP solution and for CRM they used Salesforce solution. Before integration, the data had been manually introduced twice.

Measurement showed that after integration errors had dropped by 90% and working hours for staff implied in those operations decreased by 40%.

56

(57)

Integration Through Cloud Eye

• IBM has three solutions for iPaaS designed for different users categories:

– IBM App Connect Professional focuses on ad-hoc and citizen integrator – IBM Integration Bus on Cloud is addressed to specialist integrators

– IBM API Connect “provides capabilities to manage and secure the integration flows when exposed as APIs”

• Microsoft provides Azure Logic Apps (from the middle of 2016) as their iPaaS stand-alone solution and also Microsoft Flow addressed to citizen integrators.

• Microsoft Flow is built on top of Azure Logica Apps, which is in turn based on other services offered in Azure:BizTalk Server, SQL Databases, SharePoint, API Management and Service Bus and their intentions are to integrate with IoT Suite, Machine Learning, MIcrosoft Dynamics that will increase the range of

functionalities.

• What should be noted is a very wide range of connectors that cover protocol connectors, Azure Services and Power APP Connectors, SaaS connectors, B2B or EDI connectors, Hybrid connectors [Cru17]

57

(58)

Integration Through Cloud Eye

• MuleSoft offers Anypoint iPaaS solution providing: API management, on-premise and SaaS integration, B2B integration using EDI. Anypoint Exchange is a huge

repository that contains connectors, templates, APIs available to perform

integration in their iPaas that is layering in AWS. Also, their solution is available as Docker image therefore, it can be employed for on-premise or hybrid

environment.

• In [Sel16], a use case of Anypoint iPaaS is presented by Splunk , a company that monitors, collects and analyzes data to offer digital transformation [WBM14]

services. Because of the business growth, a new level of efficient internal operation was needed. Therefore, they used Mule Solution to integrate

Salesforce and Netsuite. The quote-to-cash [Har16] process has experienced a rapidity of time from minutes to real-time operations.

58

(59)

Integration Through Cloud Eye

• Oracle offers two solutions for iPaaS Oracle Integration Cloud Service (ICS) for ad hoc integrators and Oracle SOA Cloud Service for specialist integrators.

• Oracle iPaas solutions are integrated with: “Oracle Self Service Automation for citizen integrators, Oracle Process Cloud Service for improved orchestration, Oracle Real-Time Integration Business Insight for business activity monitoring, Oracle API Platform Cloud Service for API management, Oracle Managed File Transfer Cloud Service for managed file transfer and Oracle IoT Cloud Service for IoT integration”

• We present below some partial results of a comparison performed in [CapiPaaS17], that take into account “application and integration middleware” that are not iPaaS solutions, but can be/are already part of an iPaas.

59

(60)

Integration Through Cloud Eye

iPaaS architecture and services

• iPaaS is composed by two main levels:

– top level designated to technical users and business users.

– the second level can be viewed as background integration platform services for: data formats (XML, JSON) and standards support (Health Level Seven Standards [HL7], Electronic Data Interchange For Administration, Commerce and Transport [EDIFACT] et.al.), data transformation, connectors/adaptors for on-premise packaging application or for SaaS, routing, top operations (orchestration, monitoring, management, integration with various tools, APIs management et.al.).

• As one may notice, from the structural point of view, there are similarities with an ESB system: we have on top instruments for business users and technical specialists and on subsequent levels there are integration mechanisms.

60

(61)

Integration Roadmap

• In the light of Cloud Computing new technologies and challenges, integration has new dimensions.

• The first forms of integration were referring to the process of replicating data to another cloud (cloud-cloud integration) or replicated some data from/to on- premise systems (cloud-on-premise integration).

• “data” has become step by step the central concept around which technologies gravitate

• In [Lin14] David Linthicum oversaw the evolution of data integration technologies

61

Figure: Data Integration evolution [Lin14]

(62)

Integration Roadmap

• In the 2010-2014 period there is an increased evolution on the cloud level

technologies, while on the data level we encounter: “data replication, semantic mediation, data cleansing and mass data migration.”

• Enterprises began to have hybrid or multi-cloud architectures, and that allowed them, in order to offer maximum benefits for business processes, to move data (replicate, storage) from cloud-to-cloud, from enterprise-to-cloud, from

enterprise-to-enterprise in private-clouds et.al.

• In this period, the semantic web technologies were viewed as solutions to

improve customer satisfaction and from a technological perspective to assure a more automate mediation. Ontologies were also used in order to simplify

operation for services reuse.

62

(63)

Integration Roadmap

David Linthicum envisions, and the technological reality partially confirms his point of view, the cloud based architectures becoming essential parts in an Enterprise and as such, there is need for [Lin14]:

• Intelligent Data Service Discovery: integration technology needs to automatically find (discover, identify details about the API) and provide data services from cloud or non-cloud environments. We can understand this as a generalisation of UDDI [AB06] plus smart technologies (such as Semantic Web based) where Enterprises can benefit by all available data.

• Data Virtualization - comes with the concept as data viewed as service itself; it is not a new concept, and does not imply back-end changes; instead it “redefines how the database is made visible and consumed by other systems”*Lin14+

• Data Orchestration - has influences from service orchestration and refers to various mechanisms to combine data and therefore to fulfill the needs for multiple solutions

• Data Identity - refers to the mechanism that allows to control who and when to consume the data

• The rise of shared Enterprise business services and data is viewed as a real path for data reuse

(“repurpose data sets”) that can bring benefits to many new or existing enterprises. You can envision a company who can access databases providing them the ability to correlate and avoid future

potential losses.

• Through credential-based identity and centralized trust, David Linchcum envisions the existence of techniques that “can validate both the data (structure and content), as well as humans and machines that would like to view or manipulate the data”.

63

(64)

Integration Roadmap| PrivateSky Vision

• Our vision regarding the level of centralization takes into account the recent

development from the blockchain technology area. In accordance with our vision from PrivateSky project [PrivateSky16] we believe in Decentralised Trust (using Blockchain based mechanisms or other underlying technologies (Self Suveranity Data & Identities [Alb17])

64

(65)

Summary

• Integration – as a key concept

• Integration and/or Interoperability

• Integration contexts

• Service Oriented Architecture

• Evolution of Integration Technologies

• ESB - Enterprise Service Bus

• Technology mapping: SOA, ESB, BPM

• Integration Through Cloud Eye

• Integration Roadmap

65

(66)

Bibliography

• [AB06] Lenuta Alboaie, Sabin Buraga, Web Services (in Romanian), Polirom Publishing House, ISBN 973-681- 522-6 Iasi, 320 pages (2006)

• [Bec04] Kent Beck, Extreme Programming Explained: Embrace Change, Publisher Addison-Wesley Professional, 224 pages, ISBN-10: 0321278658, 2nd edition (2004)

• [DMG07] Paul Duvall, Steve Matyas,Andrew Glover, Continuous Integration - Improving Software Quality and Reducing Risk, ISBN-13: 978-0321336385 (2007)

• [GarI][Online] Available: https://www.gartner.com/it-glossary/integration/

• [Fow14][Online] Martin Fowler, Microservices - a definition of this new architectural term, https://martinfowler.com/tags/microservices.html (2014)

• [Jos14][Online] Maneesh Joshi, Why Buses Don't Fly in the Cloud: Thoughts on ESBs. Available:

http://insights.wired.com/profiles/blogs/why-buses-don-t-fly-in-the-cloud-thoughts-on-esbs#axzz4z9L9T8Pc (2014)

• [Mule][Online] MuleESB. Available: http://www.mulesoft.org/

• [MuleiPaaS17] [Online] What is iPaaS? Gartner Provides a Reference Model,. Available:

https://www.mulesoft.com/resources/cloudhub/what-is-ipaas-gartner-provides-reference-model (2017)

• [GPGTIW17][Online] Keith Guttridge, Massimo Pezzini, Elizabeth Golluscio, Eric Thoo, Kimihiko Iijima, Mary Wilcox, Magic Quadrant for Enterprise Integration Platform as a Service. Available:

https://www.gartner.com/doc/reprints?id=1-3X0Y452&ct=170403&st=sb (2017)

• [Cru17][Online] Paco de la Cruz, Microsoft Azure Integration Platform as a Service (iPaaS). Available:

https://blog.mexia.com.au/azure-integration-platform-as-a-service-ipaas (2017)

66