Big Data Management

In order to face challenges introduced by novel applications requirements, the database community came up with new ways of delivering the system's internal and external functions to applications: as components (by the end of the 90's), as peer-to-peer networks (in the beginning of the 2000's), and as service-based database management systems the last 10 or 15 years. These new ways of delivering data management functions have been done under different architectures: centralized, distributed, parallel and on-cloud. Data management services are deployed on different system/hardware architectures: client-server (on classic and embedded systems), distributed on the grid, on P2P networks, on the W2.0, and recently on the clouds.

From our high-level view of the architecture of DBMS, we can conclude that although they make efficient use of the resources of their underlying environment, once configured they are fixed to that environment as well. Of course, there are settings with DBMS deployed on top of different target architectures but then it is up to the database manager and the application to give an integrated view of such multi-database solution. Designing and maintaining multi-databases requires important effort and know how. Such solutions tend to require also effort when they evolve due to new/changing data storage, data processing and application requirements. In a service-oriented environment where various hardware and software platforms are hidden behind service interfaces, deciding where to store data, is up to the data provider and the data consumer has few or no control on these issues, given services autonomy. Furthermore, for several applications in dynamic environments ACID transactions may not be feasible or required. In addition, DBMS are not easily portable and often impose a large footprint. A related problem is that they are difficult to evolve and maintain. Indeed, adding new functions, supporting evolutions of the data model, extending the query language and then supporting well adapted optimization strategies, can require important coding effort. For example, extending DBMS for dealing with streams, multimedia, geographical data implied important effort some years ago. Having new transactional models has required big implementation effort to materialize theory into efficient transaction managers. Changes should not penalize previous applications and they must efficiently support new uses. For these reasons, several researchers have concluded that they in fact exhibit under performance or are even inappropriate for a variety of new applications.

Consequently, the core functionality of the DBMS must be adapted for new settings and applications, particularly for dynamic and service-oriented environments. Services allow dynamic binding to accomplish complex tasks for a given client. Moreover, services are reusable, and have high autonomy because they are accessed through well defined interfaces. Organizing services on layers is a solution to compose a large numbers of services, and will help in making decisions about their granularity (e.g., micro-services). This allows to reuse optimized functionality that is shared by several applications instead of having to invest effort on the implementation of the same functionality again. Another way of extending the system is by invoking internal services through calls from external Web services or Web servers. Users can thereby have their own tailored services on their personal computers to replace or extend existing SBDBM services according to their needs.

Developers of new applications can benefit from service reuse by integrating one or more services that run on the SBDBMS, from any available layer, in an application. This can provide optimized access to their application-specific data. For example, assume that an application needs access to the storage level of the DBMS in order to obtain statistical information, such as available storage space or data fragmentation. In this case, the developer can add the necessary information services to the storage level. Thus, she provides the information source required for her application. Then, the application has to just invoke these services to retrieve the data. Furthermore, other services from other layers can be used together with this kind of extension services if required. Services can be distributed, and be made redundant by using several computers connected through a network. Therefore, a SBDBMS can be customized to use services from other specific locations to optimize particular tasks. This approach introduces a high degree of adaptability into the database system. Priorities can be assigned to services that demand a considerable amount of resources, thereby enabling Quality of Service agreements (QoS) for special data types like multimedia and streaming data.

These challenges imply the construction of service-based middleware with two open problems:

1. Exploit available resources making a compromise between QoS properties and Service Level Agreements (SLA) considering all the levels of the stack (i.e., from the network (infrastructure) to the application level).
2. Optimally coordinate services considering applications' and users' characteristics for fulfilling their requirements.