Hybrid Artificial Bees Colony and Particle Swarm on Feature Selection

. This paper investigates feature selection method using two hybrid approaches based on artificial Bee colony ABC with Particle Swarm PSO algorithm (ABC-PSO) and ABC with genetic algorithm (ABC-GA). To achieve balance between exploration and exploitation a novel improvement is integrated in ABC algorithm. In this work, particle swarm PSO contribute in ABC during employed bees, and GA mutation operators are applied in Onlooker phase and Scout phase. It has been found that the proposed method hybrid ABC-GA method is competitive than exiting methods (GA, PSO, ABC ) for finding minimal number of features and classifying WDBC, colon, hepatitis, DLBCL, lung cancer dataset. Experimental results are carried out on UCI data repository and show the effectiveness of mutation operators in term of accuracy and particle swarm for less size of features.


Introduction
Classification problem can be well resolved using machine learning algorithm and efficient feature selection (FS) methods.FS algorithms act as filter of redundant and irrelevant features and are categorized on three types: filter, wrapper and embedded methods.In filters approach, feature relevance are evaluated by applying statistical methods, wrapper approach find the best subset of features based on machine learning algorithms evaluations [1][2][3].Embedded methods incorporate filter inside wrapper.
Existing feature selection FS methods combine various evaluation measures and search strategies for selecting optimal feature subset.Large number of FS algorithms used but, none of them can outperform all others FS methods with all types of data.There is a need to find better algorithm with less computational cost.
The Artificial Bee Colony (ABC) algorithm [5] proposed by Karaboga is one of the most recent swarm intelligence based optimization technique, which simulates the foraging behavior of a bee swarm.Because of ABC is easy to implement, converge fastly, it has been often applied in numerical optimization domain [7][8][9] and feature selection [3,10].However, there are still some drawbacks.ABC is computationally inefficient if it is trapped in the local optima [11].The exploration and the exploitation abilities of the population based algorithms are both necessary characteristics.The exploration acts to search the various unknown regions to find the global optimum, while the exploitation attempts to apply the knowledge of the previous good solutions to find better solutions.The exploration and exploitation abilities should be well balanced to achieve good performance.However, research shows that the solution search equation of ABC algorithm is good at exploration but poor at exploitation [7][8]12].
Many variants of ABC algorithm were developed to enhance the exploitation ability, Zhu and Kwong [7] presented a global best solution guided ABC (GABC) algorithm by incorporating the information of global best into the solution search equation.Wu et al [13] proposed improvement of global swarm optimization (GSO) by hybridizing it with ABC and PSO.They used neighborhood solution generation scheme of ABC and accepted new solution only when it was better than previous one to improve GSO performance.
Yan et al [14] proposed for clustering a hybrid artificial bee colony (HABC) algorithm and embedding the crossover operator of GA to ABC in information exchange phase between bees.Gao et al. [15] develop a hybrid ABC using the modified search equation and orthogonal learning strategies to achieve efficiency.
Applying machine learning algorithms as evaluation function is time consuming especially for high dimensional data.So, it is necessary to improve any lack of FS methods to produce best selected subset.
In this paper, a hybridized swarm intelligence ABC with PSO and meta-heuristic approach like GA for optimal feature selection is presented to improve the balance between exploration and exploitation.To achieve that, particles swarm PSO contribute in ABC during employed bees, in addition, GA mutation operators are applied in Onlooker and Scout phase.The main contributions are:  Reduce the features space to the relevant characteristics achieving the highest accuracy. Improve the exploitation process by mutating bad solutions of onlookers. Replace abandoned solutions of scout by mutating worst source foods and also random good solutions of onlookers. Compare the behavior of ABC-PSO, ABC-GA with PSO, GA and ABC. Validate the efficiency of the exploitation and exploration of the proposed solutions.The manuscript is organized as follows.Section II describes features selection methods: ABC, Particle swarm and Genetic algorithms and related work.Section III gives a detailed description of the proposed FS methods: ABC-PSO and ABC-Genetic Algorithm.Experimental results and discussions are presented in section IV and V. finally conclusion in section VI.

Artificial Bees Colony
Artificial bee colony (ABC) was proposed by Karaboga in 2005 [5] and mimics the foraging behaviors of honey bee colony.In optimization problem, the food sources represent candidate solutions and the nectar amounts correspond to fitness values.The ABC has three types of bees: employed, onlooker, scout.Employed bees exploit their food sources and convey the information concerning quality and position of food sources to onlooker bees via waggle dance.Onlooker bees select food source to be exploited with the help of the employed bees.Scout bees search a new food source.[9,16].ABC algorithm has slower convergence speed in some cases because of the lack of powerful local exploitation capacity [17].

Particle Swarm Optimization
Particle swarm optimization PSO [6] inspired by the social behavior of flocks of birds.Feature selection has intrinsic discrete binary search spaces.The original Binary BPSO was proposed by Kennedy and Eberhart [19] to allow PSO to operate in binary problem spaces.In this version, particles could only fly in a binary search space by taking values of 0 or 1 for their position vectors.
PSO method presents the advantage of high convergence rate and low computational cost.PSO adjusts only a few parameters.It is also efficient.PSO has much better global optimization ability and lower computational time than GA and has been widely used in feature selection [20].Unfortunately, falling into local optimum is the principal drawback of particles swarms.

+ (4)
The position is: Update personal best; Update global best; End; End; End.

Genetic Algorithm
Genetic algorithm (GA) is an evolutionary algorithm developed by John Holland and follow Darwin's principle of survival of the fittest [21].GA is a successful wrapper method where each individual is a chromosome (candidate solution) and the set of chromosomes constitutes a population.However, GA is computationally expensive and tends to prematurely converge into local optima if the diversity mechanism is not well adapted.GA phases start with population initialization and generate new individuals by using successive genetic operators (selection, crossover, mutation) during every iteration, the best chromosomes of them are preserved.

Related Work
Hancer and al [3] propose a binary artificial Bees colony (ABC) for feature selection.Evolutionary based similarity search mechanism is introduced in binary ABC.This method is compared to PSO, GA, discrete ABC on 10 datasets.The results indicate that the performance is higher in term of accuracy and eliminate effectively redundant features.
In this paper [11], a hybrid approach based on the life cycle for the artificial bee colony algorithm is achieved, to ensure appropriate balance between exploration and exploitation.Each individual can reproduce or die dynamically throughout the searching process and population size can dynamically vary.The bees incorporate the information of global best solution into the search equation for exploration.Powell's search enables the bees deeply to exploit around the candidate area.Experiments based on the CEC 2014 benchmarks compared the performance of Hybrid ABC against other bio-mimetic algorithms and proved the strength of this algorithm.
Gao et al. [15] combined ABC with Differential evolution DE.An eliminative rule and the new search strategy are introduced into the iteration of ABC to improve the convergence rate.Then, to stabilize the population diversity, DE simulates evolution and all individuals are taken into account in each generation.
Authors [17] proposed a hybrid algorithm called PS-ABC, which combines the local search phase in PSO with two global search phases in ABC for the global optimum.In the iteration process, the algorithm checked the aging degree of pbest for each individual to decide which type of search phase to adopt.The proposed PS-ABC algorithm is validated on 13 high-dimensional benchmark functions from the IEEE-CEC 2014 competition problems, and compared with ABC, PSO.Results show that the PS-ABC algorithm is efficient, fast converging and robust optimization method for solving high dimensional optimization problems.
In this paper [18], authors described a method to perform a crop classification based on the Gray Level Co-Occurrence Matrix (GLCM) and the artificial bee colony (ABC) algorithm.The proposed methodology selects the set of features from the GLCM that allow classify the crops with a good accuracy using the ABC algorithm in terms of a distance classifier.
A hybrid PSO local search HPSO-LS [20] has been implemented for feature selection.They embedded local search in PSO to select low and salient feature subset.Local search guides the search process of the PSO to select distinct features by considering their correlation information.HPSO-LS was compared with GA, PSO, ACO, simulated annealing SA on 13 benchmark and prove the superiority of this approach.
Djellali and al [22] propose two hybrid FS methods: multivariate filter fast correlation based filter FCBF and PSO (FCBF-PSO) and FCBF with genetic algorithm (FCBF-GA), FCBF-PSO combines in two phases particles swarm PSO and In first stage, FCBF filter is applied and provide a subset of features (relevant and no redundant), in the second stage, PSO is performed with previous selected features and executed on five UCI datasets.The experiments show that both FCBF-PSO and FCBF-GA (genetic algorithm replace PSO) are competitive.

Proposed Hybrid Feature Selection Method
In this study ABC, mutation operators and PSO are combined in ABC phases.This combination is called ABC-PSO mutate in figure (1).The reason is the fact that the ABC shows poor balance between exploitation and exploration.
This scheme attempts to improve the search capability of Bees when their behavior do not provide good source foods, So, mutation is introduced in both Onlookers and Scout Bees to produce better solutions and PSO particles are compared to Employed Bees to take advantage of them.To establish the efficiency of the proposed algorithm ABC-PSO-mutate, we first compare two hybrids algorithms ABC-PSO and ABC-GA.
In ABC-PSO, the PSO particles are incorporated in employed Bees like described in Modified employed Bees code.Scout and onlookers keep the same code as standard ABC.In ABC-GA, we preserve standard employed bees unchanged and insert mutation in onlooker and scouts bees phases.

Fig 1. Hybrid ABC-PSO mutate Algorithm
Proposed Hybrid ABC-PSO-mutate algorithm is built.Firstly, ABC employed Bees is modified by using PSO velocity and comparison between fitness of Employed Bees solutions and Particle solution.The best is saved.
Secondly, GA mutation is used in onlooker phase.After enough iteration, when probability of food source (i) is not improved, this food source solution is mutated and compared its fitness with previous value, in case where the new solution is better, we replace the current food source with new food source.
Scout Bees search new solutions when employed and onlookers are stuck.So, we propose to modify scouts behavior by mutating both 'the worst solution' and selected two 'random good solutions'.The new food source is the best of them.

Hybrid ABC-PSO Algorithm
ABC-PSO is proposed to stimulate competitive behavior between onlooker bees and PSO particles and applied to achieve better balance between exploration and exploitation.First, all initial employed Bees are copied into swarm PSO particles with their fitness.Second, both particles and employed Bees cooperate, particles compute  1) and evaluate it.
J is a feature index 1..D.

Hybrid ABC-GA Algorithm
In ABC-GA, we introduce mutation operators first on scout bees where worst source food are mutated an compared with random mutated good source food.Second, Onlookers bees are also modified with mutation operator.The code is presented below:

Modified Onlookers Bees and Mutation
New source food solution Vi is selected depending on probability Pi.In fact, onlookers will not choose the worst solutions Vi from employed Bees only if in the next iteration the solutions are improved.Vi is selected and improved depending on Pi.
In the modified onlookers code, we mutate source foods(i) = Vi only after few iterations (20% of MaxIterations) and if (prob(i) < rand(0,1) ) is true, then, the new mutated source is compared to source Vi.If the new source is better, we update Vi and its fitness.

Modified Onlookers()
Input Fitness, Foods,data, label; Output Fitness, Foods; For each Onlooker Bees i Calculate the probability values Pi for the solution Xi by Eq. (7).

Scout Bees phase
Scout Bees search new solutions using ImprovedLazyBees function.In original ABC code, equation (3) initialized new solutions, but there is no guarantee that these new solutions provide better results.In this regard, mutate both previous worst ABC solutions and random good ABC solution is required to guide scout bees towards better solutions.
Mutation operators help to diversify solutions whatever the source issue, from bad or good solution.Mutating good solutions can deliver also good solutions and improve the search process, consequently, increase the search ability of hybridized ABC.The solution is to replace the abandoned solutions (counter > limit) with a new one Xi by applying this code:
-Evaluate the three food sources and keep the best as new source of food.

Experimental Results
In this section, we first describe the experimental settings, then, we evaluate the performance of the proposed feature selection method ABC-PSO and ABC-GA on Support Vector Machine (SVM) [23].We also compare the proposed algorithms with FS methods: ABC, GA, PSO.

Databases
To evaluate the performance of the proposed feature selection algorithms, we choose the dataset from UCI [24] machine learning repository.The dataset tested are Wisconsin Breast cancer Dataset WBCD, hepatitis, colon cancer, DLBCL.The datasets properties are summarized in Table 1.

Parameters Setting of FS Algorithms
We describe the parameters setting of GA, PSO, hybrid ABCGA, ABC-PSO; all algorithms are implemented by using MATLAB 7.0.Each dataset is divided into training set (70%) and test set (30%).Classification accuracies of algorithms are measured by the standard 10 fold cross-validation methods on training data.To provide the test classification accuracy, the selected features are assessed on test set [25].The average accuracy of the algorithms are computed with 20 independent runs.The parameters setting of algorithms are described in table 2. Support Vector machine is an efficient machine learning technique that uses kernel to construct linear classification boundaries in higher dimensional space [6], [23].SVMs have good generalization performance in various applications.

Discussions
The experiments on WDBC data in Table 3 show that ABC-GA provides best accuracy (99.41%) and minimal number of features involved (13).ABC-PSO (accuracy= 96.71 and 16 features selected) accomplishes less efficiency in both accuracy and minimal number of features.ABC is better than PSO and Genetic algorithm (accuracy=99.41%15 features).
The best results on hepatitis are achieved with hybrid ABC-GA (accuracy =91.67 % with 9 features) in table 4 but ABC-PSO provide less number of features (accuracy =87.50% with 4 features).

Conclusions
In this paper two hybrid approaches based on artificial bees colony and particles swarm ABC-PSO and ABC with Genetic algorithm ABC-GA are proposed.It has been found that the proposed method ABC-GA outperforms the proposed ABC-PSO method and exiting methods (ABC, GA, PSO) on hepatitis, WDBC and DLBCL.Despite the lowest accuracy of ABC-PSO, this hybrid method tends to produce less number of features.These observations lead us to make assumption that adding PSO in employed phase and mutations in onlookers and scout phase in ABC will maximize accuracy and minimize the feature subset size.
Adapting the behaviors of scout and onlooker bees lead to obtain the best result with hybrid ABC-GA and consolidate the first assumption concerning increasing the numbers of possible solutions in every iteration and consequently improve the exploitation process.We propose ABC-PSO mutate scheme and to establish efficiency of mutations alone we tested ABC-GA, to validate the performance of particle swarm on ABC alone, we introduce particles in employed bees phase.Both mutations and particles swarm are competitive, which means incorporating all these modification in ABC-PSO mutate should provide better results.We have to test all these modification in ABCPSOGA in future work.

Scout Bees phase:
Apply the greedy selection process.
Employed Bees calculate new source food.Each solution is evaluated and compared, then the best is affected to Employed Bees.Initialize the population of solutions Xi ∀i, i = 1, ..., NB.2: Evaluate the population Xi, ∀i, i = 1, ..., NB. 3: Copy the population Xi and its fitness in PSO initialization population.For each Employed Bees i 2. Produce new solutions Vi using Eq. ( class, NBfeatures, CrossRate, MuRate, PSO parameters,

Table 1 .
Dataset Tested in the Experimental Study

Table 2 .
Setting Parameters

Table 4 .
HepatitisColon cancer shows in Table5that the GA algorithm realizes the best SVM accuracy (96.34%) with 1001features.ABC-PSO achieves minimal number of features (599) but less accuracy 79.17.The results on DLBCL data in Table6demonstrate that ABC-GA achieves the highest accuracy (91.45%) but more features involved(1943).ABC-PSO algorithm outperforms all FS methods for minimal number of features (1383) but less accuracy 90.00%.PSO provides (90.43% accuracy but more features 2068).It was observed that the combination of ABC-GA gives the best results for hepatitis and WDBC.ABC-PSO provides low feature subset size than ABC-GA on all dataset except for WDBC data.