Background

Scientists are accustomed to make predictions based on consolidated and accepted theories pertaining to the domain of prediction. However, nowadays big data analytics (BDA) is able to deliver predictions based on executing a sequence of processing while seemingly abstaining from being theoretically informed about the subject matter. Seizing these new opportunities is tempting: some researchers have been trapped by the sheer amount of datasets made available by leading data-driven companies, which are either directed towards the companies' own prosperity or representing rather small subsets (e.g. of users). For example, understanding the differences between the vast majority of users (i.e. humanity) and smaller subsets of people, whose activities are captured in big datasets, is critical to correct analysis of the data. Surely, BDA needs exploration, but at the same time also reflection to guide BDA research to a prospering future.

Big data analytics research applies machine learning, data mining, statistics, and visualization techniques in order to collect, process, analyze, visualize, and interpret results. BDA, as a process, is based on many disciplines that analyze data to elucidate hidden knowledge. Yet, BDA research either employs exploratory data analysis to generate hypotheses, or alternatively pursues predictions relying heavily on advanced machine learning, data mining and statistical algorithms.

Our analysis and argument focuses on predictive research since it lends itself to BDA more than exploratory research. We share the point of view that for BDA to be useful in the long run, it needs epistemological reflection and it needs also to be theory-driven, not only driven by data that is easily available. However, the question addressed here is: How to address the epistemological challenges in the process of BDA? Accordingly, we analyze the sequence of processing in BDA and seek to identify the various needs of theoretically informing BDA throughout all of its steps.

This paper is organized as follows. The next section briefly reviews the role of scientific theory in generating predictions. Then we clarify the term BDA and discuss various challenges that already have been identified. The core of the paper analyzes BDA as a process: data acquisition, preprocessing, analysis, interpretation – and for each step we examine and exemplify required critical decisions and point to the underlying epistemological problems and possible solutions based on adequate theoretical foundation. Before concluding we discuss (a) the theoretical groundwork to be done in order to lead BDA to safe and prosperous grounds as well as (b) how to deal with the shift from theory to process in BDA-based prediction.