Multicriteria Evaluation Method in PLM Environment: A Pilot Study

. During the whole design process, evaluation stages are required and should consider more and more criteria but some of them are not mastered by designers. They also need support tools or methods to ease the product evaluation during the design process and avoid the subjectivity of this task, to limit the choice’s risks and to allow the repeatability of the whole process. This article presents through an experiment the necessity of evaluation tools by comparing assisted and not-assisted evaluation of 9 products.


Introduction
To win the innovation race firms must set up innovation management evaluation during their design process to control the evolution of their products.This race reduces the time allowed for product development.This modification of the design requirements promotes the use of evaluation method during the design process.Each evaluation phase needs its own performance indicators and method to fit with the product's progress.As a performance criterion, the innovative characteristic of the product must be evaluated to validate the economic and functional viability.
However, the performance of the product depends on design and politic goals.More and more criteria of performance are added to suit the consumers' evolution and the policy evolution as the sustainable development.
Sustainable development is "development that meets the needs of the present without compromising the ability of future generations to meet their own needs" [1].It is composed of three aspects: social, as the human being, environment, as the respect of the nature and economy as the industrial development.This approach is currently growing and is described as the next world problem is global warming as presented in the Climate change conference in 2015 [2] .Environmental impact reduction is sup-ported by Policy [3] and is also promoted by consumers as a selection criteria [4].By considering the sustainability impact of their product, firms can increase their profitability and improve the life cycle of their products by optimising the end of life of their product or reduce the consumption of the product manufacturing.
Innovation and sustainability are therefore promising opportunities for designing new products.The requirements relative to its themes must be integrated as soon as possible into the design process that means during the early design stages, to provide the market with innovative and sustainable products.
The motivation of this work is to expose the difficulty of multiple performances evaluation during the design process.The state of the art positioned the design methodologies in PLM approach and described the limits of evaluation tools for multiple performance assessment.An experiment is proposed to highlight the benefit of tools during the design process.

State of the art
Our study is focused on the evaluation stage of innovation and sustainability during the design process.The state of the art first aims to position the Product Lifecycle Management into the design process.In a second time, the integration of innovation and sustainability in the design process are investigated.In the third part of the state of the art, a description of evaluation tool typology is proposed.

Early PLM and design methodologies
Design methodologies are widely used in our community.For engineering design practices in industry, engineers and managers are concerned with the results of design by following specific design process more rigorously to ensure that the final design result meets the objectives of the product design and development projects [5].PLM is an approach to support this complexity, including the design methodology and the complete product life cycle [6], [7].As PLM addresses the entire lifecycle of the product, it has a cross-functional nature and deals closely with the way a company runs [8].With the development of Product Data Management (PDM), PLM and associated workflows, software firms have proposed solutions to the everyday problems of engineering design departments (versioning of documents, naming etc.).PLM aims to cover all the stages of product development, by integrating the processes and people taking part in the project [9].Thus, PLM is an approach in which processes are just as important as data, or even more so.Thus, when dealing with PLM implementation, the design process is naturally the starting backbone to define processes and associated workflows.Hence, if one wants to provide a successful PLM environment, it is mandatory to structure the methodologies that will be supported.Design process is also the backbone of an innovation process and early design stages, starting from the research of concepts to the delivery of a preliminary layout, are the key stages of the innovative design [10] as presented in Figure 1.If we focus on the early design stages, current «PLM methodological backbones» are not evaluated by designers or end-users.This paper presents the benefit these kinds of tools could provide.

Figure 1: Design process and position of the Early Design Stages Innovation and sustainability in the design process
The innovative aspect of a product is created during early design stages [11].Two kinds of innovations are differentiated: incremental innovation which ameliorate a product and breakthrough innovation which corresponds to new products for new needs [12].Saunders identifies characteristics of innovative products based on the study of popular products [13].This study permits to identify innovation criteria for product development.The innovative aspect of a product can be managed by two approaches: -An innovative design process: The whole process is modified and managed to increase the innovative characteristic of the product.-An innovative evaluation of the project: each intermediate representation of the product is evaluated to control the innovative characteristic of the product.Throughout the rest of the article, we proposed to study only the second approach, suitable for every type of design process.As opposed to innovation, sustainability is not specific to early design stages.It can be evaluated during the whole process.The complexity of methods follows the evolution of the product during all design stages.First, method permits to identify the product scenario of use based on the environmental impact criteria [14], then select and improve the ideas [15], [16] also improve the product concept or architecture [17], [18] and then optimise the life cycle of the whole defined product [19].
Innovation and sustainability aspects of the product have to be evaluated and monitored during the design process.The different kinds of evaluation tool are described in the following part.

Design evaluation
Each successive stages of the design process is the alternation of generation and evaluation of intermediate representation of the product [20].It means that firstly designers must answer the specific problems of the product design and then select with the appropriate criteria the best intermediate representations.
According to Blessing [21], criteria " are used to be able to focus the investigation of the existing situation; to assess the contribution of the findings of such investigations to the research goal; to focus the development of support on the most relevant factors; to plan the appropriate evaluation; to focus the of the support on this evaluation; and to assess the evaluation results." Two types are defined: -Successful criteria: they represent the benefit generated by the placing on the market of the product -Measurable success criteria: they represent the possible success of the product, innovative and sustainable indicators are used here for measuring the performance of the product.

Typology of evaluation tools
To quantify these measurable success criteria, a specific evaluation method is needed.Literature distinguishes two types of evaluation: -Expert evaluation based on the knowledge of experts in the domain.This type of method is well used for innovation projects and relies on evaluation grids [22].-Tools evaluation based on the quantification of product criteria.This kind of method needs tangible measurement and is used to determine environmental impact of intermediate representation with survey [15] or, matrix [23].It is based on the measurement of objective criteria.Evaluations are crucial during the design process and even more during the early design stage where the greater part of the engaged cost is defined.Innovation and sustainable require experts.To reduce the resource dedicated to the evaluation stage and to draw on objective indicators, tools are the best compromise.

Evaluation needs during design process
In this section, an experiment made to investigate the impact of evaluation tools for designers is proposed.Twelve M.Sc.students attending courses on sustainable manufacturing have been involved: subjective and objective evaluation sessions are performed to understand whether the use of tools during evaluations is a benefit for designers or not.Secondly, a method for tool evaluation is proposed and tested.

Protocol
First, a set of products to be evaluated has been identified: nine heterogeneous products designed with particular concerns for the purpose of sustainability or innovation, or manufactured with innovative technologies (such as additive manufacturing) have been selected.The whole list of products is provided in Table 1.
The experiment is composed of two main steps and lasted 1h30 as described in Figure 2.

Figure 2: Planning for experiment
In the first stage, each participant has to assess the sustainability and the innovation levels for each product without any support tool.Participants have to access a Google form and provide a subjective answer to the question "how much is the innovative aspect?" through a Likert scale (1-5): 1 is for bad and 5 is for good.
After this evaluation, the same set of products is evaluated by using two appropriate tools for sustainability [15] and innovation [13].These tools were adapted in a Google form.
Then, the results have been collected and compared to identify whether differenced among the two evaluations can be found.

Results
This pilot study needs the use of non-parametric methods due to the sample size.For each product, we analyse the distribution and the difference between the two kinds of evaluation by using the following method: -Kolmogorov-Smirnov method is used for non-parametric sample.It compared each sample with a normal distribution.The results of each product are shown on Table 2.This method provide four criteria: o the mean value of the evaluation o the standard deviation o the coefficient of deviation o the signication that describe if the normal distribution is validated.-Wilcoxon method permits to identify if the evolution between two samples of data are correlated.The results are shown on Table 3.

Table 2 Kolmogorov-Smirnov results
Most of the results on Table 2 show that the hypothesis of a normal distribution is never rejected when the tool is used (signification value higher than 0.05) for both sustainability and innovation.When the tool is not used, the hypothesis of normal distribution is not accepted in three cases only (see the results highlighted in orange).In 14 out of the 15 remaining evaluation pairs, exhibit a reduced sustainability or innovation average score: therefore, if the participants are not guided during the evaluation process, they tend to overestimate products' performances.Further, in 12 out of 15 cases the overall standard deviation is smaller, although a clear pattern for the coefficient of variation (given by the ratio between the standard deviation and the mean value) cannot be identified: so, the overall evaluation range is reduced but, since also the average values are usually lower, the relative dispersion may increase.
For the case study, final products are evaluated, the use of tools permit to standardize and homogenize this step.The results of Table 3 show the results of Wilcoxon test.It compared two linkedsamples and test whether their evolutions are constant.If the signification is under 0.05, the hypothesis is invalidated.For innovation evaluation, the hypothesis is rejected that means that participants change significantly their point of view between the two evaluations.This result is in accordance with the Table 2, the range is clearly different with and without tools.This test shows that innovation is clearly a subjective characteristic and it is difficult to evaluate without tools.

Analysis and Conclusion of the experiment
This experiment shows that it is difficult to evaluate products regarding to different subjects.The Table 2 shows how much participants overestimate the performance of product.Moreover, it permits to identify that tools reduce the deviations.The Table 3 permits to understand how evolve the evaluation between the two steps.For sustainability aspect, the participants overestimate the performances but in average follow the same distribution with or without tools.For innovative aspect, the subjectivity of this characteristic can explain why the participants change drastically their evaluation.
This experiment was made with final products which are easier than during the design process.The less the product is developed the better the benefit of evaluation tool is.
These results show the need of assistance for evaluation stages.The next part of this article aims to validate a method of classification for evaluation method.
One limit of this experiment could be the reduce number of participants.

Conclusion
This article proves through the experiment the necessity of evaluation tool during design by the comparison of two evaluations: an evaluation subjective and an evaluation with tools.This experiment was performed by engineering students and highlighted that tangible constraint as the sustainability is more valuable than subjective characteristic as innovation.Evaluation tools permit for both types of feature to reduce the uncertainty.Assistance makes repeatable and reliable evaluation stages.
The choice of evaluation tool depends of different condition link to the design process or designers' knowledge or product goals, and makes the selection more and more difficult.The second experiment is a validation of an evaluation method of tools.This method is using unlabelled criteria to fit with each disciplinary found in the design process.Engineering students had to make their own evaluation of scientific papers by following the proposed method.