An extreme learning machine algorithm for semi-supervised classification of unbalanced data streams with concept drift

Abstract

Data streams are important sources of information nowadays, and with the popularization of mobile devices and sensor systems that collect all kinds of data, more and more information is generated at an ever increasing speed. This growth in data supply poses some problems for traditional machine learning algorithms. Tasks such as data classification, regression, or data clustering presents some limitations regarding very large datasets, data streams, or variations in data. The high cost of labeling instances for training classification algorithms makes it difficult to use fully supervised algorithms. Unbalanced datasets tend to cause algorithms to ignore one or more classes. Moreover, concept drifts in data streams require algorithms to be retrained from time to time. In order to tackle such problems mentioned, a semi-supervised and online algorithm based on Extreme Learning Machine (ELM) called SSOE-FP-ELM is proposed and detailed. Experimental results show that the proposed algorithm outperform others in the literature in accuracy, generalization ability and concept drift detection and recovery, showing suitable alternatives for data streams classification.

Appendices

Appendix A: Standard ELM

Different from gradient descent-based learning methods, ELM does not perform an iterative adjustment of weights. The input weights of the network are randomly generated and maintained during the training phase. The network output weights are calculated analytically using the least squares method [24]. Thus, the training time of the ELM algorithm is a few hundred (or thousands) times less than the backpropagation algorithm, according to [24]. Furthermore, ELM is not susceptible to getting stuck in local minima or slow convergence, common in backpropagation.

Fig. 12

Illustration of SLFN trained with ELM

Figure 12 shows a SLFN trained with ELM algorithm. Consider a training set $(x_{i}, t_{i}), x_{i} \in R^d, t_{i} \in R^m, i = 1, \ldots , N$, where N is the number of training samples, d is the number of features and m is the number of classes; an activation function g(x), a sigmoidal function for example; and a number of neurons n in the hidden layer. The relationship between the input data $x_{i}$ and the network output values $t_{i}$ is given by:

$$\begin{aligned} t_{i} = \sum _{j=1}^{n}\beta _{i}g(w_{j}x_{i} + b_{j}), \ i = 1,..., N \end{aligned}$$

(A1)

where $\beta _{i}$ is the output weights, $w_{j}$ and $b_{j}$ are the input weights and bias. In ELM, $w_{j}$ and $b_{j}$ are set randomly before the training phase, and are not adjusted.

Equation (A1) can be written in the matrix form $H\beta = T$, where H is the hidden layer matrix (feature matrix) given by:

$$\begin{aligned} H = \left[ \begin{array}{ccc} g(w_{1}x_{1} + b_{1}) &{} ... &{} g(w_{n}x_{1} + b_{n}) \\ \vdots &{} \ddots &{} \vdots \\ g(w_{1}x_{N} + b_{1}) &{} ... &{} g(w_{n}x_{N} + b_{n}) \end{array} \right] _{N \times n} \end{aligned}$$

(A2)

The linear system $H\beta = T$ is solved by employing the Moore-Penrose generalized inverse (pseudo-inverse) of the H matrix, represented by $H^{\dagger }$. There are different ways to calculate the pseudo-inverse of a matrix, including the orthogonal projection, iterative methods, and decomposition into singular values (SVD) [7]. The solution of ELM and $H^{\dagger }$ matrix computation are given by:

$$\begin{aligned} H^{\dagger } = (H^{T}H)^{-1}H^{T} \end{aligned}$$

(A3)

$$\begin{aligned} \beta = H^{\dagger }T \longrightarrow \beta = (H^{T}H)^{-1}H^{T}T \end{aligned}$$

(A4)

The addition of a small value on the diagonal of a matrix guarantees this matrix is non-singular, enabling the orthogonal projection method to be used in the computation of the pseudo-inverse. As demonstrated by [25], this regularization factor makes the solution obtained more stable and with greater generalization ability. Thus, for a given regularization factor $\alpha$, $H^{\dagger }$ matrix and $\beta$ matrix can be obtained as follows:

$$\begin{aligned} H^{\dagger } = (H^{T}H + \alpha I)^{-1}H^{T} \end{aligned}$$

(A5)

$$\begin{aligned} \beta = (H^{T}H + \alpha I)^{-1}H^{T}T \end{aligned}$$

(A6)

The ELM training pseudocode, as proposed by [24], is described in Algorithm 3.

Algorithm 3

ELM training.

Table 15 Standard ELM computational complexity, based on [2]

Table 16 SSOE-FP-ELM computational complexity for each training partition k, based on [2]

Appendix B: Time complexity of SSOE-FP-ELM

For many classification problems, the total number of training samples N and the number of neurons n are the highest values, generally greater than the number of features d or the number of classes c. Therefore, to compute the ELM time complexity, the order of importance of the parameters from the most relevant to the least relevant is: number of training samples and number of neurons; number of features; number of classes.

Considering the dimensions of the input matrices $X_{[N \times d]}$, $W_{[d \times n]}$ and $T_{[N \times c]}$, and the greater impact of parameters N and n on computational time, Table 15 presents the computational complexity of each operation in Standard ELM, and the final complexity. It is important to note that in classification problems with a large number of training samples, the term $n^{2}N$ gains greater importance in the calculation of the final time complexity [10]. In problems where the number of neurons n is greater than the number of training samples N, the term $n^{3}$ stands out. Both operations are performed only once, during $\beta$ computation.

In the SSOE-FP-ELM algorithm, which performs the training process in an online manner, the total number of training samples N is replaced by the size of each training partition $N_{k}$ in the input matrices dimensions and in the calculations performed. Generally, the size of training partitions is smaller than the number of neurons, making parameter n the most important when calculating computational time of SSOE-FP-ELM.

Table 16 presents the computational complexity of each operation in SSOE-ELM, and the final complexity. In addition to the standard ELM operations, SSOE-FP-ELM computes the intermediate matrices L, J (for semi-supervised learning), U and V (for online update) for each training partition. SSOE-FP-ELM has a longer training time than the SSOE-ELM, due to the supervised factor of the forgetting parameter. To assess the accuracy of the model from one training partition to another and check whether a label concept drift has occurred, it is necessary to calculate the $\beta$ output weights matrix, and this is the operation with the highest computational cost. Thus, at each training partition the SSOE-FP-ELM needs to carry out all operations (calculate the intermediate matrices L, J, U and V, compute the $\beta$ matrix and calculate the forgetting parameter), different from the SSOE-ELM that does not need to calculate $\beta$.

It is important to note that the SSOE-FP-ELM algorithm is based on SSOE-ELM, so the operations of the semi-supervised learning and online update steps are the same for both algorithms. The difference between the two algorithms, in terms of computational time, is the calculation of the semi-supervised forgetting parameter in the SSOE-FP-ELM.