風結ぶ言葉たち

System Evaluation is a comprehensive assessment of system value. And value is usually understood as the recognition of the evaluation subject's satisfaction with the evaluation object based on its utility perspective. It is closely related to the evaluation subject and the environmental conditions in which the evaluation object is located.

Matrix Method#

Matrix method is a common comprehensive evaluation method that uses a matrix to represent the relationship between the alternative solutions and the evaluation values of specific indicators.

• $ A_1 , ⋯ , A_m $ are $ m $ alternative solutions for a certain evaluation object;

• $ X_1 , ⋯ , X_n $ are $ n $ evaluation indicators or evaluation items for evaluating alternative solutions;

• $ W_1 , ⋯ , W_n $ are the weights of $ n $ evaluation indicators;

• $ V_{i1} , ⋯ , V_{mn} $ are the value evaluation quantities of the $ i $th alternative solution $ A_i $ for the $ j $th indicator $ X_j $ ($ 1 \leq i \leq m, 1 \leq j \leq n $).

The following will combine an example to accept two methods of determining weights and value evaluation quantities.

A company produces a popular product and has planned the following three production plans:

$ A_1 $: Design a new production line by ourselves;

$ A_2 $: Import a highly automated production line from abroad;

$ A_3 $: Retrofit a production line based on existing equipment.

After discussions by authoritative departments and experts, five evaluation indicators are determined, namely: expected profit; product yield rate; market share; investment cost; product appearance.

According to the predictions and estimates of professionals, the results of the five evaluation items after implementing these three plans are shown below.

Pairwise Comparison Method#

The basic steps of this method are as follows:

1. Perform pairwise comparison of the evaluation indicators of each alternative solution, and give higher scores to relatively important indicators to obtain the weights $ W_j $ of each evaluation item;

2. According to the evaluation scale given by the evaluation subject, evaluate each alternative solution under different evaluation indicators one by one to obtain the corresponding evaluation values;

3. Calculate the weighted sum to obtain the comprehensive evaluation value.

For the example, we first need to use the pairwise comparison method to determine the weights of each evaluation indicator. As shown in the figure below, the expected profit is more important than the product yield rate, so the former gets a score of 1 and the latter gets a score of 0. Finally, the weights are calculated based on the accumulated scores of each evaluation item.

Then, the evaluation scale is determined by the evaluation subject, as shown in the figure below, to unify the actual results of the plans under different indicators for weighted sum.

Based on the previous two figures, the comprehensive evaluation of each alternative solution is shown in the figure below. It can be seen that $ V_2 \gt V_1 \gt V_3 $, so $ A_2 \gt A_1 \gt A_3 $.

Gurin Method#

When the importance of each evaluation item can be quantitatively estimated, the Gurin method can be used to determine the indicator weights and solution value evaluation quantities, which is the basic method for determining the weights and value evaluation quantities.

The basic steps of this method are as follows:

1. Determine the importance of the evaluation item $ R_j $;

2. Normalize $ R_j $ with $ K_j $ as the unit, and obtain the weights $ W_j $;

3. Calculate the evaluation values of the alternative solutions using the same method as the matrix method;

4. Calculate the weighted sum to obtain the comprehensive evaluation value.

For the example, we first need to determine the importance of the evaluation indicators $ R_j $.

Then, normalize $ R_j $, set $ K_j $ as the result of normalization, and calculate the weights $ W_j $.

After calculating the weights of each evaluation item, according to the same calculation method, evaluate each alternative solution item by item. For example, for the importance of expected profit $ X_1 $, the expected profit of $ A_1 $ is 6.5 million yuan, and the expected profit of $ A_2 $ is 7.3 million yuan, so $ R_{11} = 650/730 = 0.890 $, $ R_{21} = 730/520 = 1.404 $. Then, calculate $ K_{ij} $ and normalize to obtain $ V_{ij} $.

In the calculation of the fourth step, since the smaller the investment cost, the better, the reciprocal of the proportion is taken, that is, $ R_{14} = 180/110 = 1.636 $, $ R_{24} = 50/180 = 0.279 $.

It can be seen that $ V_2 \gt V_1 \gt V_3 $, so $ A_2 \gt A_1 \gt A_3 $.

Simplified method:

Divide the value evaluation quantities of each alternative solution by the value in the last row;

Then divide the values in each row by the sum of the values in each column;

Normalize to quickly obtain the weighted sum $ V_i $ to get the result.

Analytic Hierarchy Process#

The analytic hierarchy process (AHP) can be modeled in four steps:

1. Establish a hierarchical structure model to describe the relationship between elements in the evaluation system;

2. Create pairwise comparison matrices to compare each element in the same layer and construct the matrix;

3. Calculate the relative weights of the elements, analyze the importance of each element in the same layer to the upper-level criterion, calculate the relative weights of each element through the judgment matrix, and perform consistency checks;

4. Calculate the synthesis (overall) weights of each element to the overall objective of the system and rank each candidate solution.

Let's explain using the example.

A university needs to scientifically review a series of research projects and select the evaluation criteria for research projects as shown in the figure below. How to make a scientific selection among the many candidate evaluation projects? Please use the analytic hierarchy process to determine the weights of $ C_1 $ to $ C_6 $ relative to the overall objective $ A $.

First, calculate the weights $ W_i $ and normalize them to obtain the following results.

Then, perform the calculation, and the final result is as follows. By comparing the evaluation values, the priority order is determined as $ C_2 $, $ C_2 $, $ C_3 $.

Fuzzy Comprehensive Evaluation Method#

The fuzzy comprehensive evaluation method is based on fuzzy mathematics and uses the principle of fuzzy relation synthesis to quantify factors that are difficult to quantify and make judgments about the membership level status of multiple factors in evaluating things.

The fuzzy comprehensive evaluation method can be roughly divided into three steps:

1. Determine the factor set $ F $ and the evaluation set $ E $. The factor set $ F $ is a set of evaluation items or indicators, generally $ F = {f_i}, 1 \leq i \leq n $. The evaluation set is a set of evaluation levels, generally $ E = {e_j}, 1 \leq j \leq m $.

2. Statistically determine the membership degree vector of single-factor evaluation and form the membership degree matrix $ R $. The membership degree is the most basic and important concept in fuzzy comprehensive judgment. The membership degree $ R_{ij} $ refers to the possibility size of multiple evaluation subjects making a $ e_j $ evaluation of a certain evaluation object in terms of $ f_i $.

3. Determine the weight vector $ W_F $, which is the weight or coefficient vector of the evaluation items or indicators. In addition, there may be the numerical results (standard satisfaction vector) $ W'_E $ or weights $ W_E $ obtained from the evaluation set.

Let's explain using the example.

A person needs to determine the priority order of models $ A_1 $, $ A_2 $, and $ A_3 $ before purchasing a refrigerator. Five family members use the fuzzy comprehensive judgment method to evaluate. The evaluation items (factors) include price $ f_1 $, quality $ f_2 $, and appearance $ f_3 $. The corresponding weights are obtained through the judgment matrix in Figure 1. The evaluation scale is divided into three levels, for example, the price is divided into low (0.3), medium (0.2), and high (0.1). The evaluation results are shown in Figure 2. Please calculate the priority and rank of these three refrigerators.

First, calculate the weights $ W_i $ and normalize them to obtain the following results.

Then, perform the calculation, and the final result is as follows. By comparing the evaluation values, the priority order is determined as $ A_2 $, $ A_2 $, $ A_3 $.

This article is also updated to xLog by Mix Space.
The original link is https://nishikori.tech/posts/tech/Systems-Evaluation-Method