Clouds offer significant advantages over traditional cluster computing architectures including flexibility, high-availability, ease of deployments, and on-demand resource allocation - all packed up in an attractive pay-as-you-go economic model for the users. However, cloud users are often forced into vendor lock-in due to the use of incompatible APIs, cloud-specific services, and complex pricing models used by the cloud service providers (CSPs). Cloud management platforms (CMPs), supporting hybrid and multi-cloud deployments, offer an answer by providing a unified abstract interface to multiple cloud platforms. Nonetheless, modelling applications to use multi-clouds, automated resource selection based on the user requirements from various available CSPs, cost optimization, security, and runtime adaptation of deployed applications and services still remain a challenge.

In this talk, I'll give an introduction to Melodic, which is a middleware platform for Cross-Cloud data-intensive applications. The Melodic platform enables data-intensive applications to run within defined security, cost, and performance boundaries seamlessly on geographically distributed and federated cloud infrastructures. Melodic thereby realizes the potential of heterogeneous cloud environments for big data and data-intensive applications by transparently taking advantage of distinct characteristics of available private and public clouds, dynamically optimize resource utilization, consider data locality, conform to the user’s privacy needs and service requirements, and counter vendor lock-in.

Clouds offer significant advantages over traditional cluster computing architectures including flexibility, high-availability, ease of deployments, and on-demand resource allocation - all packed up in an attractive \emph{pay-as-you-go} economic model for the users. However, cloud users are often forced into vendor lock-in due to the use of incompatible APIs, cloud-specific services, and complex pricing models used by the cloud service providers (CSPs). Cloud management platforms (CMPs), supporting hybrid and multi-cloud deployments, offer an answer by providing a unified abstract interface to multiple cloud platforms. Nonetheless, modelling applications to use multi-clouds, automated resource selection based on the user requirements from various available CSPs, cost optimization, security, and runtime adaptation of deployed applications and services still remain a challenge. 

In this tutorial, we provide a practical introduction to the multi-cloud application modelling, configuration, deployment, and adaptation. We survey existing CMPs, compare their features, modelling methods, and, not the least, provide a practical hands-on training for getting your applications ready for the multi-clouds using selected tools. By the end of this tutorial, attendees should be able to understand various tools and technologies available for the multi-clouds, and prepared to spin-off their first multi-cloud ready application.

Data-intensive computing, often simply referred to as big data, is one of the major current trends in ICT. In areas as diverse as social media, business intelligence, information security, Internet-of-Things, and scientific research, a tremendous amount of data is created or collected at a speed surpassing what we can handle using traditional data management techniques. Life sciences are not different. With the vast amount of biological information available, such as Omics data, unprecedented opportunities for modern research and scientific breakthroughs arise, all depending on the efficient and cost-effective data analysis. Cloud computing, characterized by the paradigm of on-demand network access to computational resources and pay-as-you-go economic model, promises great potential of providing required computational resources for data analytics in Bioinformatics. However, challenges such as lack of data privacy and data-aware cloud federation keeps cloud computing from realizing the full potential for data-intensive applications. At the same time, non-standardized cloud interfaces make it complex to migrate big data applications between platforms thus preventing cloud users from achieving optimal cost-performance ratio for their applications by encouraging vendor lock-in.

In this talk, we  provide an introduction to the MELODIC H2020 project and show how it can be of great value in Bioinformatics. The vision of MELODIC is to enable federated cloud computing for data-intensive applications, and provide the user with an easy-to-use unified cloud environment, hiding the complexity of a multi-cloud. The MELODIC platform enables big data applications to transparently take advantage of distinct characteristics of available private and public clouds by dynamically optimizing resource allocations considering data locality and user's performance and privacy needs. From the perspective of the user, the MELODIC framework appears as an infrastructure-agnostic middleware platform supporting development, deployment, and execution of data-intensive applications on distributed and heterogeneous multi-clouds. For the Bioinformatics community, this could mean utilizing the resources available for multiple cloud providers and private infrastructures in a secure, transparent, efficient, cost-effective, and reliable manner for their big data workloads.

Cloud Computing has seen a tremendous popularity in last several years. A scalable and efficient data center network is essential for a performance capable cloud computing infrastructure. This thesis provides practical solutions to enable an efficient, flexible, multi-tenant network architecture suitable for high-performance cloud computing, using InfiniBand (IB) as a demonstration technology. The work is motivated by the needs of the future data centers to provide efficient cloud solutions for increasing uptake of the cloud technology for both big data and traditional High-Performance Computing (HPC) applications.

Research contributions of this thesis lie within three main categories. First, we propose a set of improvements to the fat-tree routing algorithm to make it suitable for HPC workloads in the cloud. Fat-Tree is a popular network topology in HPC systems. Our proposed improvements to the fat-tree routing make it more efficient, provides performance isolation among tenants in multi-tenant systems, and enable routing of both physical end nodes and virtualized end nodes according to the policies set by the provider. Second, we design new network reconfiguration methods to significantly reduce the time it takes to reroute the IB network. Reduced network reconfiguration time means that the interconnection network in a HPC cloud can optimize itself quickly to adapt to changing tenant configurations, faults, running workloads, and current network conditions. Last, we demonstrate a self-adaptive network prototype for IB-based HPC clouds, fully equipped with autonomous monitoring and adaptation, and configurable through a high-level condition-action language for the service providers.

The research conducted in this thesis has potential impacts on both private cloud infrastructures, such as medium sized clusters used for enterprise HPC, and public clouds offering innovative HPC solutions to the customers at scale. The industrial application of the thesis is reflected by the eight patent applications resulted from this work.

Clouds offer significant advantages over traditional cluster computing architectures including ease of deployment, rapid elasticity, and an economically attractive pay-as-you-go business model. However, the effectiveness of cloud computing for HPC systems still remains questionable. When clouds are deployed on lossless interconnection networks, challenges related to load balancing, low-overhead virtualization, and performance isolation hinder full potential utilization of the underlying interconnect. In this work, we attack these challenges and propose a novel holistic framework of a self-adaptive IB subnet for HPC clouds. Our solution consists of a feedback control loop that effectively incorporate optimizations based on the multidimensional objective function using current resource configuration and provider-defined policies. We build our system using a bottom-up approach, starting by prototyping solutions tackling individual research challenges associated, and later combining our novel solutions into a working self-adaptive cloud prototype. All our results are demonstrated using state-of-the art industry software to enable easy integration into running systems.
 

The research on network optimization in InfiniBand (IB) networks has been evolved in several directions, e.g. increasing network utilization, fault-tolerance, congestion control, and energy-aware systems. However, for efficient HPC clouds based on IB, the optimization problem becomes both complex and multi-dimensional, while individually proposed solutions often yield contradictory management decisions. We believe that a holistic closed-loop control system is required to effectively incorporate multidimensional objective function in future IB systems. Based on control theory, a self-adaptive model for the IB subnet system, may help acheiving better network utilization while effectively keeping user level SLAs in HPC clouds.