Methods and results

Step 3

Constructing the prototype of prediction model

The LR analysis was used to model dichotomous outcome variables and forecast relationships between the dependent variable and a set of independent explanatory variables. The LR model constructed a two-way classification system as a substitute for linear discriminant analysis, and avoided the unreasonable assumption that the binary type covariance matrix must be equal. Based on the all 17 input variables of the BISE and enterprise characteristics derived from the measurement instrument of step 2, this step first adopted LR analysis to model the influence and explanatory power of prediction variables.

Analytical results demonstrated that the value of Cox-Snell \(R^2\) was 0.583 and that of Nagelkerke \(R^2\) was 0.808, suggesting that the model had receivable prediction power. Table 5 also indicates that the four predictive performance measures, accuracy, precision, recall, and F1-measure, of the prototype LR model, were 89.61, 78.13, 96.15 and 86.21 %, respectively, leaving the effectiveness of the prototype unproven. Additionally, the two prediction variables were \(X_{15}\) and \(X_{13}\), with the estimated values of 3.703 and 3.408, and \(p\)-values were 0.003 and 0.006 (<0.05), respectively, which indicated good explanatory power. Meanwhile, under the prototype model, the remaining 15 prediction variables had less explanatory power. To improve the prototype model, this study first extracted the critical input variables and structured the association rules, and then constructed the refined LR model to enhance the predictive power and accuracy. The associated calculation work is presented below.


Identifying the critical prediction indicators and association rules

The DT algorithm was one of the methods used in data mining for knowledge discovery, and systematically analyzed the data to identify rules and relations for use in data classification and prediction. This algorithm comprised a supervised learning method for data mining. The classification tree was adopted, and included parameter setting and standards for calculating divergence. Model accuracy was assessed using the actual DT performance to calculate the proportion correctly classified as judgment. This study administered the CART algorithms, and the splitting criteria, impurity measures and Gini criterion Breiman et al. were as described below.

At node \(t\), the optimal split \(s\) was selected to maximize a splitting criterion \(Δi(s,t)\). When the impurity measure (i(t)) for a node was defined, the splitting criterion corresponded to a decrease in impurity. \(ΔI(s,t) = p(t)Δi(s,t)\) was labeled the improvement.

\( 

i(t) = \sum\limits_{i,j} {C\left( {i\left| j \right.} \right)p} \left( {i\left| t \right.} \right)p\left( {j\left| t \right.} \right)
\)

\( \Delta i\left( {s,t} \right) = i\left( t \right) - p_{L} i\left( {t_{L} } \right) - p_{R} i\left( {t_{R} } \right) \)

\(p(t)\) The probability of a case in node \(t. p(j|t)\) The probability of a case in class \(j\) given that it falls into node \(t. C(i|j)\) The cost of miss-classifying a class\( j\) case as a class I case. \(C(j|j) = 0\)

Analytical results demonstrated that the structure of DT that obtained the most accurate classification, including the minimum number of cases, was two in the total branching nodes and the maximum DT depth was five hierarchies. The performance measures, accuracy rate, precision rate, recall rate, and F1-measure rate, of the structure of DT were 94.81, 92.31, 92.31 and 92.31 %, respectively (see Table 5). Furthermore, Fig. 2 illustrates the tree structure of the DT algorithm. The details of each terminal node included description of rule paths, categories of belonging, numbers entering the node, and analysis of category purity. The tree structure was such that five paths (namely, association rules) existed from the root node to the leaf nodes.

Fig. 2


Results of tree structure of the DT algorithm

  • Rule A: If \(X_{12}\) = {1,2,3} and \(X_{15}\) = {1,2,3}, then low BISE. P = 95.20 %.
  • Rule B: If \(X_{12}\) = {1,2,3} and \(X_{15}\)  = {4,5}, then high BISE. P = 85.70 %.
  • Rule C: If \(X_{12}\) = {4,5} and \(X_{16}\) = {1,2}, then low BISE. P = 80.00 %.
  • Rule D: If \(X_{12}\) = {4,5}, \(X_{16}\) = {3,4,5} and \(X_1\) = {1,2,3}, then high BISE. P = 66.70 %.
  • Rule E: If \(X_{12}\) = {4,5}, \(X_{16}\)  = {3,4,5} and \(X_1\) = {4,5}, then high BISE. P = 100 %.

\(Xi\) = variable of BISE

The extent of promotion of or support for \(Xi\); very low = 1, low = 2, medium = 3, high = 4, very high = 5

\(P\) = purity of predictive association rule

Analyzing all five prediction rules revealed that the first critical determining attribute and indicators of BISE is the variable \(X_{12}\) : promotion of the accumulation of business intelligence is a human competency attribute. Based on rules A and B, the second is variable \(X_{15}\) : support for new service development and customer acquisition. Meanwhile, rule C showed that the third is \(X_{16}\) : support for service continuity and customer retention. The above two critical determining indicators are embedded in support for specific business processes attributes. Finally, the last is \(X_{1}\) : promotion of data availability is a technological attribute, based on analysis of rules D and E.


Developing the refined prediction model

To improve the prototype prediction model constructed based on the first LR analysis in "Constructing the prototype of prediction model" section, this study adopted four critical prediction indicators (variables) of BISE derived from the classification and prediction model by using the DT algorithm to refine the prototype prediction model to enhance improve its predictive power and accuracy.

Analytical results demonstrated that the suggested prediction equation was as follows:

\(p = \dfrac{{e^{f(x)} }}{{1 + e^{f(x)} }} \,\)

\(\ln \left( {\frac{p}{1 - p}} \right) = f(x) = - 3 8. 4 8 2 { } + 2. 8 0 0 * X 1 2 { } + 2. 7 5 9 * X 1 5 { } + 3.862 * X 1 6 { } + 2.477 * X 1 { }\)

The probability (P) in the range 0–1 is used to identify the BISE in the BI implementation, where the value of \(P\)  is close to 1 means high BISE and the value of P is close to 0 means low BISE. The four critical prediction indicators included \(X_{1}\) ,\(X_{12}\) , \(X_{15}\) and \(X_{16}\) , and achieved the good explanatory power (see Table 4). The variable of \(X_{16}\) : support for service continuity and customer retention had the highest predictive power.

Based on the Hosmer–Lemeshow goodness of fit test, the \(X^2\) value was 6.777 and the p-value was 0.561(>0.05), demonstrating that the model and data were suitable and the model had good overall fitness. The value of Cox-Snell \(R^2\)was 0.621 and that of Nagelkerke \(R^2\) was 0.860, which signified the LR model had good predictive and explanatory capability. The omnibus test \(X^2\) was 74.641 and the p-value was 0.000(<0.05), which means the model was capable of predicting the BISE.

Table 4 Results of LR analysis

Variables Estimate Standard
error		 Wald \(X^2\) p-values
Constant −38.482 11.501 11.196 0.001***
\(X_{16}\)  : support for service continuity and customer retention 3.862 1.617 5.704 0.017**
\(X_{12}\) : promotion of the accumulation of business intelligence 2.800 1.158 5.850 0.016**
\(X_{15}\) : support for new service development and customer acquisition 2.759 1.208 5.219 0.022**
\(X_{1}\)  : promotion of data availability 2.477 1.120 4.892 0.027**
Model fit properties Omnibus test \(x^2\) = 74.864, p-value = 0.000***
Hosmer–Lemeshow test \(x^2\) = 6.777, p-value = 0.561
Cox-Snell \(R^2\)=0.622
Nagelkerke\(R^2\)=0.862

      1. *** \(p < 0.01\); ** \(p < 0.05\)

Additionally, Table 5 list the results of the four predictive performance measures - accuracy, precision, recall, and F1-measure of the refined LR model, and the comparison between actual conditions and test results of the two LR models and the structure of DT. For the refined LR model, the total predictive accuracy, precision, recall, and F1-measure were 94.81, 92.31, 92.31 and 92.31 %, respectively. The results demonstrated that the refined LR model exhibited a better predictive performance in terms of accuracy, precision, recall and F1-measure, than the prototype LR model, and the same predictive performance as the structure of DT. These two models had the same four predictive performance measures because of the small sample sizes; when the analyzed samples small, the probability that the two models are equally accurate is high.

Table 5 Results of predictive performance measures of the LR model and the DT structure

Methods Groups Actual condition
Low BISE group High BISE group
LR model (prototype) Test result
 Low BISE group 25(TP) 7(FP)
 High BISE group 1(FN) 44(TN)
Accuracy 89.61 %
Precision 78.13 %
Recall 96.15 %
F1-measure 86.21 %
LR model (refined) Test result
 Low BISE group 24(TP) 2(FP)
 High BISE group 2(FN) 49(TN)
Accuracy 94.81 %
Precision 92.31 %
Recall 92.31 %
F1-measure 92.31 %
DT structure Test result
 Low BISE group 24(TP) 2(FP)
 High BISE group 2(FN) 49(TN)
Accuracy 94.81 %
Precision 92.31 %
Recall 92.31 %
F1-measure 92.31 %
    1. TP true positive; FP false positive; FN false negative; TN true negative


    A live case


    Introduction

    AC Company, a financial services firm, was founded in Taiwan in the 1990s. The company mostly provides loans and sale intermediation in the used equipment market. Recently, the company developed and implemented BI systems based on information technology (IT), substantially improving its business agility and operational innovativeness.


    BI system architecture

    The information system of AC Company includes customer relationship management systems, enterprise resource planning systems (loan management information systems and equipment financing information systems), supply chain management systems, internal auditing systems, administrative information systems, and financial management systems. The company uses data mining techniques, multidimensional database techniques, data warehouse tools, and SQL2008 tools to integrate all of these information systems into its BI systems.

    The BI systems of AC Company are IT-driven and function-oriented systems for analyzing data and presenting actionable information to executives and decision-makers to help them make business and management decisions. BI systems include several data processing tools, and support operation and management applications that enable the company effectively to collect data from internal systems that support loan activities and operations, and external sources that are associated with suppliers, customers, government, and related organizations. BI systems also analyze or virtualize obtained data to create reports for executives and decision-makers that provide the results of analyses on which they can act.


    BI system effectiveness

    The implementation of BI systems has enabled AC Company to transform collected data into useful information and reports that help to improve decision-making and management, and to construct a featured knowledge database that accumulates BI and increases BISE. For example, recently, to cope with the competition in Chinese loan markets, the company adopted a business strategy to develop innovative loan service systems to improve customer service, and to use the aforementioned information systems and BI systems in its subsidiary in mainland China to open up new loan markets.

    Executives of the company examined the effectiveness of implemented BI systems using the four key indicators of BISE, which were extracted by logistic regression analysis as follows.

    • BI systems strongly support (level 4 on the Likert scale) service continuity and customer retention (\(X_{12}\) ).
    • BI systems provide very effectively (level 5 on the Likert scale) promote the accumulation of business intelligence (\(X_{12}\)).
    • BI systems strongly support (level 4 on the Likert scale) new service development and customer acquisition (\(X_{15}\) ).
    • BI systems provide very effectively (level 5 on the Likert scale) promote data availability (\(X_{1}\)).