Journal of Manufacturing Systems 65 (2022) 158–168

Contents lists available at ScienceDirect

Journal of Manufacturing Systems

journal homepage: www.elsevier.com/locate/jmansys

Technical paper

A consistency evaluation method for digital twin models

He Zhang a, b, Qinglin Qi a, Fei Tao a, c, *
a
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100083, China
b
Shenyuan Hornors College, Beihang University, Beijing 100083, China
c
Digital Twin International Research Center, International Research Institute for Multidisciplinary Science, Beihang University, Beijing 100191, China

A R T I C L E I N F O A B S T R A C T

Keywords: Digital twin has shown its great application values in various fields in the past few years, especially
Consistency manufacturing. During the process of digital twin implementation, the digital twin model is one of the most
Digital twin important foundations. Because the effectiveness and accuracy of digital twin models could directly affect the
Model validation
quality of related services. Nowadays, some efforts have been made on digital twin modeling methods, e.g. multi-
Model verification
disciplinary modeling method, multi-dimension modeling method, and multi-scale modeling method. Besides,
Shop-floor
the research on Industrial Internet of Things makes it possible to realize the synchronization between digital twin
models and the corresponding physical objects. However, the attention on how to conduct a comprehensive
consistency evaluation between the models and corresponding physical objects is not enough. Against the
background, this paper proposed a consistency evaluation framework towards digital twin shop-floor (DTS)
models. The consistency evaluation method mainly contains two phases, i.e. before and after model assembly and
model fusion. Before model assembly and model fusion, the geometry models, physics models, behavior models,
and rule models are mainly considered. After model assembly and fusion, the overall performance of assembled
models and fusion models is paid more attention. Furthermore, the analytic hierarchy process(AHP) is used to
form a comprehensive consistency evaluation result for DTS models. At last, a specific application scenario in
complex product AIT (Assembly, Integration and Test) shop-floor is given to explain how to apply the proposed
framework and method.

1. Introduction digital twin model construction for the wind turbine and autoclave [4,
5]. Luo et al. constructed the descriptive models of mechanical subsys­
In the past few years, digital twin has been applied into more and tem, electrical subsystem, and hydraulic subsystem to describe the fea­
more fields, including aerospace, manufacturing, healthcare, city, en­ tures of CNC machine tools [6]. Based on MBSE, Liu et al. believed the
ergy, etc. These applications and explorations have shown the great four dimensions of the shop-floor requirements, structure, behavior, and
superiority of digital twin. The core idea of digital twin is building an parameters were the components of a comprehensive digital twin model
ultra-fidelity virtual replica by making full use of physical data and for shop floor [7]. To express product quality in a fine-grained way, Liu
mechanism. Then digital twin models could be used to monitor, control, et al. constructed a digital twin quality knowledge model in the macro,
and optimize the physical world through the interactions and iterations meso, and micro levels, which could support in-depth analysis of
between virtual and physical worlds [1,2]. It can easily be seen that product quality [8]. The accuracy and effectiveness of digital twin
ultra-fidelity models are one of the most important foundations in the models could affect the quality of services. Then, there are the demands
process of digital twin implementation. Therefore, many researchers on how to conduct the consistency evaluation between digital twin
have conducted the related work on building more comprehensive models and the corresponding physical objects. However, current
models towards various objects to make models closer to the corre­ research pays little attention on this aspect. Many efforts are made to
sponding physical objects. For example, Tao et al. proposed study model update strategies or communication mechanism to keep
multi-dimension models, including geometrical model, physical model, synchronization. For example, Talkhestani et al. adopted an
behavior model, and rule model, to construct digital twin shop-floor Anchor-Point method to realize variance detection between digital twin
(DTS) [3]. Besides, the multi-dimension models were also used in the models and the corresponding physical objects, which was used to

* Corresponding author at: School of Automation Science and Electrical Engineering, Beihang University, Beijing 100083, China.
E-mail address: [email protected] (F. Tao).

https://doi.org/10.1016/j.jmsy.2022.09.006
Received 29 July 2022; Received in revised form 4 September 2022; Accepted 5 September 2022
Available online 14 September 2022
0278-6125/© 2022 The Society of Manufacturing Engineers. Published by Elsevier Ltd. All rights reserved.
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

synchronize digital twin models of the manufacturing automation sys­ way, which could reflect the changes of DTS. Therefore, in some con­
tem [9]. Wei et al. proposed a consistency retention method for CNC ditions, different methods should be adopted for different attributes.
machine tool digital twin models under the consideration of perfor­ Based on the analysis above, the model attributes, model dimensions,
mance attenuation for wear and other damages [10]. Zhang et al. and model layers are the factors considered for consistency evaluation.
studied the model update method based on accurate model of assembly In this paper, the consistency evaluation refers to judge whether
interface to keep the consistency between physical and virtual infor­ digital twin models depict the characteristics of modeling objects
mation [11]. Yu et al. proposed a real-time model updating strategy correctly and accurately. For the models which could be measured,
based on improved Gaussian particle filter (GPF) and Dirichlet process simulated, or calculated, the consistency evaluation is to validate the
mixture model (DPMM) to realize complex system health monitoring accuracy degree between the models and the corresponding physical
[12]. Vijayakumar et al. applied Real Time Location Sensing (RTLS) objects. In some other conditions, the aim of consistency evaluation is to
technology to get information on movements or orientation, thereby verify whether the models could represent the modeling objects
realizing the synchronization [13]. Liu et al. analyzed the influence correctly in logic dimension.
factors of the machining system at first. And an adaptive evaluation Model assembly and model fusion are two important processes in the
network method was proposed to analyze, evaluate, and optimize the process of constructing DTS models [18]. Model assembly could realize
digital twin machining system, thereby providing a reliable description the model integration in space dimension. And the model fusion is used
of the changing laws of physical objects [14]. Moreover, some other to integrate multidisciplinary models. These two processes work
efforts have been made to realize dynamic connections and interactive together to build digital twin models from the simple to the complex.
iterations between the physical objects and virtual models [15–17]. Besides, geometric models, physical models, behavior models, and rule
After model updating, the consistency evaluation for digital twin models models are all covered in the processes. Therefore, in this paper, the
is still needed to check the effects of model update. consistency evaluation is divided into two phase, i.e. before and after
At present, smart manufacturing has become a major trend and core model assembly and model fusion. Considering the features of DTS
content of future manufacturing development in the worldwide. And models and the goal of consistency evaluation, the framework of con­
DTS is regarded as a new shop-floor paradigm to enhance the intelligent sistency evaluation for DTS models is proposed, as shown in Fig. 1.
level of shop-floor [3]. Against the background, this paper takes DTS as Before model assembly and model fusion, the models refer to the orig­
the research object to discuss how to conduct consistency evaluation inal geometry models, physics models, behavior models, and rule
between the models and corresponding physical objects. This papers models of basic elements. As for geometry models, the aim of consis­
firstly proposed a consistency evaluation framework for DTS models, tency evaluation is to validate that models could depict the shape, size
considering the multi-layer and multi-dimension features. Furthermore, and structure of modeling objects accurately. This process usually uses
the corresponding procedures and methods for consistency evaluation some measuring methods and could get quantitative results. The con­
are discussed before and after model assembly and model fusion. Then, sistency evaluation of physics models is to ensure that physical attri­
the comprehensive consistency evaluation results for DTS models could butes could correctly reflect the physical characteristics of modeling
be obtained based on analytic hierarchy process(AHP). Finally, an objects. The physical attributes are usually expressed by using some
application scenario in complex product AIT (Assembly, Integration and parameters. The consistency evaluation for behavior models is to verify
Test) shop-floor is used to explain how to conduct proposed consistency the correctness of behavior responses and temporal logic of the elements
evaluation method specifically. The research work would provide a in DTS. And the related rules embodied into three models above also
guidance for conducting consistency evaluation towards digital twin should be verified to guarantee the correctness and reasonableness of
models. And in the same time, the consistency evaluation results could the rules. Moreover, for each kind of model, the static attributes and
help modelers discover the insufficient and improve the models. The dynamic attributes are both considered.
information also contributes to understanding the models for users. After ensuring the consistency between the physical objects and four
dimension models, model assembly and model fusion can be conducted
2. The framework of consistency evaluation for DTS models to build more complex models. Because more relationships are intro­
duced in this process, the consistency evaluation is still needed after
The consistency evaluation methods are different for different kinds model assembly and model fusion. After model assembly, positional
of models. Therefore, the model scope and type of DTS should be figured relations, connected relations, and coordination relations should be
out at first. The key characteristics of DTS could be depicted by using further evaluated for assembled models. Moreover, the model functions
geometry models, physics models, behavior models, and rule models and performance should be considered for fused models. The goal of
[3]. The geometry model is used to depict the shape, size, and positional consistency evaluation for fused models is to validate the correctness of
relations of the elements in DTS, e.g. human, machines, materials, and coupling relationships among different models. V&V represents valida­
environment. The physics model is mainly used to depict some physical tion and verification, which should be considered for both assembled
attributes of key elements in DTS, e.g. tool wear. The behavior model is models and fused models.
used to reflect the behavior characteristics and responding mechanism Because some consistency evaluation results are qualitative, the
to various changes, including machine behavior, human behavior, related method is proposed to make it quantitative, thereby forming a
technological process, etc. And the rule model contains the constraint comprehensive consistency evaluation result towards digital twin
rules, association rules, and deductive rules of the elements in DTS. models. The comprehensive consistency evaluation results are obtained
After the types of models in DTS are defined, the features of digital based on the results from the geometry model, physics model, behavior
twin models should be further analyzed for making more proper plans model, rule model, assembled model, and fused model. The results can
for consistency evaluation. The models of DTS contains more than one provide a reference with users to understand the performance of digital
layers. Specifically, DTS could be divided into part layer, component twin models comprehensively, which is also helpful for guiding the
layer, machine layer, production line layer, and shop-floor layer [18]. related services based on these models. It should be noticed that not all
And there are close relationships in spatial, time or logic dimension consistency evaluation mentioned above should be conducted because
among the models in the same layer or different layers. Besides, the not all kinds of the models are built in some situations. Moreover, the
model attributes contain not only static attributes but also dynamic at­ demands of consistency degree is also different for different objects,
tributes. The static attribute refers to that attribute values are constant which would influence the consistency evaluation index. Therefore, the
or change slowly. The slow changes here refer to that the changes could consistency evaluation strategy needs to be adjusted according to
be neglected under the condition of not influencing the whole perfor­ practical demands, goals, and scenarios. This section tries to give a
mance. The dynamic attributes are usually updated dynamically in some relatively generic consistency evaluation framework for DTS models.

159
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

Fig. 1. The framework of consistency evaluation towards DTS models.

3. The consistency evaluation method for digital twin models should be monitored continuously by using some online measuring
tools, e.g. coordinate measuring machine. Though the format of geom­
Based on consistency evaluation framework in Section 2, specific etry models exists differences, the consistency evaluation method be­
consistency evaluation procedures and methods are discussed. As shown tween the models and the corresponding objects is almost the same. The
in Fig. 2, the consistency evaluation method for multi-dimension models specific steps of consistency evaluation method towards geometry
is discussed at first, including geometry model consistency evaluation, models based on measuring tools are as follows.
physics model consistency evaluation, behavior model consistency Step 1. Geometry model feature creation. The consistency evaluation
evaluation, and rule model consistency evaluation. Then, the consis­ features for geometry models include geometric elements such as points,
tency evaluation for assembled models and fused models is explored lines, faces, circles, spheres, and cylinders, as well as geometric tolerance
further. such as straightness, flatness, roundness, cylindricity, perpendicularity,
inclination, parallelism, position, symmetry and concentricity. The focus of
3.1. Before model assembly and model fusion measuring features are different for different objects. For example, the
cutting tool pay attention to the cutting edge inclination, tool cutting edge
3.1.1. Consistency evaluation for geometry models angle, tool minor cutting edge angle, etc. Moreover, in this step, which
For geometry models, consistency evaluation is to judge whether feature should be measured one time (i.e. static features) and which feature
geometric errors are within the allowable range. There are certain dif­ should be measured continuously(i.e. dynamic features) should be defined.
ferences in the error limits for different structures, for example, the The features can be expressed as (1)-(3). GF represents the set of geometry
crucial structural parts require smaller errors in comparison to the un­ models’ features. And GFs and GFd represent the sets of static and dynamic
important parts. Therefore, geometry models can be divided into the features respectively. fsi (0 ≤ i ≤ the number of static features) and fdj
models of key components and non-key components. Besides, consid­ (0 ≤ j ≤ the number of dynamic features) denote the specific features which
ering the features of parameter changing, the parameters could be belong to GFs and GFd .
divided into dynamic parameters and static parameters. The static pa­ GF = (GFs , GFd ) (1)
rameters only need be checked one time. And dynamic parameters

160
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

Fig. 2. The consistency evaluation method for multi-dimension models.

GF s = (fs1 , fs2 , fs3 , fs4 , fs5 …) (2) ments are usually adopted.
Step 5. Geometry model consistency analysis. The consistency
GF d = (fd1 , fd2 , fd3 , fd4 , fd5 …) (3) analysis is to make a comparison between the measurement results and
geometry model parameters. There are many analysis methods to
Step 2. Measurement point planning. After GFs and GFd are defined, evaluate the degree of consistency for static or dynamic data [19]. The
the number of measurement points and the distribution of measurement static data can be processed by variance analysis, interval estimation,
points are reasonably planned according to the geometric characteristics etc. For dynamic time-varying data, Theil’s inequality coefficient can be
of consistency evaluation objects and the requirements of dimensional used to assess the consistency. As shown in Eqs. (4-6), x(fd ), y(fd )
tolerance. In some conditions, the density of measurement points should represent measurement results and corresponding geometry parameters
change according to the surface curvature of measured objects, which is respectively. Cd(GM) denotes the consistency degree of geometry
helpful for improving efficiency while maintaining accuracy. models. And the smaller the Cd(GM)’s value is, the better consistency
Step 3. Measurement path planning. After measurement points are between the models and the corresponding physical objects is [20]. In
planned, the planning of the measurement path could be carried out. some conditions, different geometric features have different re­
And the path optimization scheme can adopt some intelligent algo­ quirements of accuracy, the data can be further divided into a few sets to
rithms, e.g. genetic algorithm. The aim of measurement path planning is make evaluation.
to reduce the displacement distance of measuring tools under the con­
dition of avoiding collision, thereby improving measurement efficiency. x(fd ) = (x1 , x2 , x3 , .., xN ) (4)
Step 4. Measurement implementation. Following the measurement
points and paths in Step 2 and Step 3, the measurement results can be y(fd ) = (y1 , y1 , y1 , …, yN ) (5)
obtained based on measurement tools. For GFs , the measurement only
need to be done once time. The dynamic properties are usually related
with time or events. Therefore, the measurement should be done at
certain intervals to grasp the changes for GFd ,. To decrease the
measuring errors, averaging the testing values and repeated measure­

161
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ ⃒ ⃒
∑N ⃒ min(Cs , Ct ) ⃒
1
(xi − yi )2 Rcom = ⃒⃒ ⃒ × 100% (8)
N
i=1 max(C , C )⃒
s t
Cd(GM) = √̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ √̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ (0 ≤ Cd(GM) ≤ 1) (6)
∑N ∑N ⃒ ⃒
( N1 xi 2 + N1 yi 2 ) ⃒ min(Ws , Wt ) ⃒
Rrob = ⃒⃒ ⃒ × 100% (9)
i=1 i=1
max(Ws , Wt )⃒
At present, some other measuring technologies are explored to make
Cd(PM) = wbm1 × Rrel + wbm2 × Rcom + wbm3 × Rrob (10)
the measurement more convenient or accurate, for example, the mea­
surement method based on machine vision [21,22]. This sub-section
wbm1 + wbm2 + wbm3 = 1 (11)
only chooses one of approaches to conduct consistency evaluation for
geometry models.
3.1.3. Consistency evaluation for behavior models
In the behavior dimension, the main purpose is to assess the cor­
3.1.2. Consistency evaluation for physics models
rectness of the behavior sequence and behavior response. Before con­
The aim of physics model consistency evaluation is to judge whether
sistency evaluation, the format or the form of representation of behavior
the physical parameter setting is correct and reasonable. These param­
models should be defined. In this sub-section, the behavior features are
eters usually affect the performance of physics models. Physical pa­
expressed by means of sequence diagram and state diagram based on
rameters are usually related to specific structure, material, environment,
SysML. SysML is a kind of standard modeling languages for systems
etc. It is difficult to use a general method to conduct consistency eval­
engineering, which is on the basis of reusing and extending a subset of
uation because of different physical properties, goals and conditions.
UML2.0. The sequence diagram is used to describe the interactions be­
Against the background, the physical experiment method is adopted to
tween the system and the environment, and the interactions among
fully consider the individual characteristics of physical objects. The
various parts in the system. The state diagram is used to describe the
consistency evaluation for physics models includes the following steps.
state transition of the system and the actions performed in response to
Step 1. Physical parameter choice. In a physical model, there are
various events [24]. The consistency evaluation between behavior
mainly two kinds of parameters to depict physical features. The first
models and the corresponding physical objects can be conducted by
class is that the parameters are fixed. The second class is the parameters
using expert knowledge. Besides, the consistency evaluation for
affected by environment factors or some other operation conditions. The
behavior models is also could be realized based on some software. The
consistency evaluation for physics models mainly consider the second
static properties and dynamic properties of behavior models are verified
class which are usually chosen by experience. It is obvious that physical
as follows.
parameters are different for different objects. Therefore, the physical
For the static properties, the verification can be conducted based on
parameters to be evaluated should be defined at first.
the functions in Rhapsody, a SysML software. The static properties
Step 2. Consistency evaluation scheme design. According to the
include naming rule, property scope, etc. And the model verification for
chosen physical parameters and physical attributes to be evaluated, the
static properties contains two aspects. On the one hand, the evaluation
scheme design can be conducted. For example, for blades, the physics
for some possible syntactic semantic mistakes is to ensure the correct­
models focus on the accuracy of blade strength and blade fatigue, which
ness in grammar. On the other hand, according to individual re­
can be evaluated through load tests and fatigue tests. During this pro­
quirements, the potential errors could be found by using the rules
cess, the test condition should be defined based on the scope of physical
selected from 255 rules for static properties in Rhapsody.
model application. The experiment scheme also should consider the
For the dynamic properties, the verification should be conducted in
enablement, operability, and safety.
the process of model running. To make the sequence diagrams and state
Step 3. Physical experiment implementation. Based on the scheme in
diagram executable, related transformation methods should be adopted.
step 2, the experiment can be carried out under the premise of ensuring
Nowadays, some software have supported model transformation, e.g.
safety. In the same time, the physical model should be simulated in the
Simulink and Modelica. Moreover, some scholars have studied the
same condition by setting related parameters in the software. For
transformation rules, e.g. from the state machine diagram to the time
example, strains and stresses of cutting tools for CNC machine tools
Petri net, and from the activity diagrams to the Petri net [25,26]. Then,
could be obtained by using ANSYS.
after defining the initial conditions, the simulations could be conducted
Step 4. Consistency analysis for physics models. The consistency
based on executable models by using related simulation software. It
analysis can be conducted through comparison between simulation re­
should be noticed that it is time-consuming to go through all conditions
sults from physics models and practical results from specific experi­
if the system is complex. Therefore, it is necessary to select the important
ments. For different kinds of data, e.g. static data, time domain data, and
parts of complex systems to improve efficiency. Finally, the consistency
frequency domain data, the analysis methods are different. The spec­
evaluation results can be obtained through the analysis of simulation
trum analyzing methods, e.g. the classic spectral analysis method and
results and practical situations.
the window spectrum estimation method, are usually adopted for fre­
The behavior models are a kind of logic model. Therefore, the con­
quency domain data. In this sub-section, to reduce contingency, a
sistency evaluation results usually are qualitative. To give an overall
complexity measurement is adopted to evaluate the consistency be­
evaluation for DTS models, a quantitative calculation method for
tween the physical objects and physics models. Then, the consistency
behavior model consistency evaluation is proposed. For state machine
evaluation can be made in the aspects of relevance (Rrel ), complexity
diagram, there are two key elements, i.e. state and transition. Then, the
(Rcom ), and robustness (Rrob ), as shown in (7)-(9) [23]. Cd(PM) repre­
consistency degree could be calculated based on the number of same
sents the consistency evaluation result of physics models. m and n are the
states and transitions. To show the differences among the states and
number of same nodes and the total number of nodes in the system di­
transitions, different weights are assigned to different elements. The
agram which is a kind of description of the information structure. Cs and
weight usually depends on the importance degree of elements. Finally,
Ct represent the complexity measurement of simulation results and
the consistency evaluation results for behavior models can be calculated
physical testing results respectively. Ws and Wt denote the robustness of
as (12− 17). BF(stm) denotes the behavior factors of state machine di­
simulation models and experiment systems’ topology. Finally, the con­
agram(stm), which is composed of states (S) and transitions (T). M(S)
sistency results could be obtained by giving corresponding weight wbm1 ,
and M(T) are the sets of states and transitions in behavior models. P(S)
wbm2 , wbm3 to Rrel , Rcom , Rrob , as shown in Eqs. (10) and (11).
and P(T) are the sets of states and transitions in practical shop-floor. m
Rrel =
m
(7) and n are the number of states and transitions in models. p and q are the
n

162
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

number of states and transitions of the physical object. Cd(BM) repre­ specific experiments are needed for specific objects to assess the model
sents the consistency evaluation result of behavior models. a and b performance. For example, the model fusion for CNC machine tools
denote the collections of serial numbers of the same states and transi­ mainly considers the mechanical model, control model, and electric
tions between behavior models and practical objects. The result of the model. The coupling relationships include the electromechanical
consistency evaluation is the ratio of the consistent states and transitions coupling relation inside the motor, the electromechanical coupling
to the total states and transitions in the physical space. Similarly, the relation between the motor and the serve motor, the electromechanical
consistency between the sequence diagram and the physical object can coupling relation between the motor and mechanical systems, etc. The
be also obtained quantificationally through the comparison the key el­ consistency evaluation for fused models is realized by giving orders,
ements in the diagram with the physical objects. including the feed direction, speed, and position, to both the physical
CNC machine tool and the fused model. Then the results can be calcu­
BF(stm) = (S, T) (12)
lated by comparison the working processes in physical and virtual
spaces. Different from machines, the focuses for shop-floor are the in­
M(S) = (s1 , s2 , s3 , …sm ) (13)
formation flow, material flow, energy flow, process flow, etc. The ma­
M(T) = (t1 , t2 , t3 , …sn ) (14) terial flow is used to reflect material transformation, transportation,
processing, etc. The information flow is used to reflect the information
P(S) = (s1 ′ , s2 ′ , s3 ′ , …sp ′ ) (15) acquisition, transmission, and processing. The energy flow is used to
depict energy regulation, usage, conversion, decomposition, recovery.
P(T) = (t1 ′ , t2 ′ , t3 ′ , …sq ′ ) (16) The process flow includes the operations in the process of executing
tasks. These flows are closely related with each other. To validate the
∑ ∑
wsa + wtb correctness of fused models, the shop-floor should be assigned a specific
Cd(BM) = ∑p
a b
∑ q (17) machining task. Then, the physical shop-floor and the corresponding
i=1 wsi ′ + j=1 wtj ′ models would execute the related operations. Finally, the consistency
degree could be evaluated by analyzing the differences between the
3.1.4. Consistency evaluation for rule models simulation results based on fused models and the practical situation.
The rule model is to describe the laws and rules embodied in the During the process of consistency evaluation for assembled models
physical shop-floor. It also includes the laws or rules in geometry, and fused models, there exist two cases.
physics, and behavior models. Following rule models, the models have
the ability to evaluate, evolve, and reason. The rules could usually be 1) The assembled model is related with the fused model. In this case,
obtained by accumulated experience and data mining. In this paper, the these two consistency evaluation processes can be merged into one
rule model adopts a formal expression method to describe equipment process. For example, for most electromechanical equipment, the
maintenance rules, personnel scheduling rules, parameter optimization external performance of fused models (including control model,
rules, production process control rules, etc. Considering rule models hydraulic model, physics model, etc.) is the machine actions which
involving a wide range, the consistency evaluation for rule models is could be depicted by assembled models. In this condition, the model
difficult to realize in exactly the same way. On the one hand, expert consistency evaluation for assembled models and fused models can
knowledge can be helpful for making the judgement whether the rule be realized by giving a specific machining task.
model depicts the features of physical shop-floor correctly. On the other 2) The assembled model is irrelevant to the fused model. In this con­
hand, some sematic mistakes and rule conflicts can be found by using the dition, the model consistency evaluation for assembled models and
functions offered by some software. Besides, the similar method with fused models should be conducted separately. For example, the fused
physics models, designing some specific experiments, can be used in the model of autoclave mainly considers fluid-thermal-solid interaction,
consistency evaluation for rule models. Specifically, the actual data is which should be validated by composite material curing molding.
input into rule models to motivate the rule models to respond accord­ The assembled model is validated by using some setting actions, e.g.
ingly. And then, the response of the rule model and the actual response opening and closing the door of autoclave.
are compared. If the response errors are within the allowable range, the
rule models are valid. Otherwise, the rule models are incorrect or 4. The comprehensive consistency evaluation for digital twin
inaccurate. models

3.2. After model assembly and model fusion After the consistency evaluation results for geometry models, physics
models, behavior models, rule models, assembled models and fused
Though the geometric sub-models are correct, some mistakes may models are obtained, the comprehensive evaluation results for digital
also exist in the assembled models because of incorrect assembly pro­ twin models are required. There exist some comprehensive evaluation
cess. For assembled models, they pay more attention to whether these methods at present. Considering the applicability of various methods
geometry models could achieve target position and attitude through and the demands of consistency evaluation, AHP is chosen to form
movements and rotations following the defined constraints or spatial comprehensive consistency evaluation results for digital twin models in
relationships. Therefore, different from the consistency evaluation for this section. The flow of consistency evaluation for digital twin models is
geometry models, the consistency evaluation for assembled models is shown in Fig. 3. The specific explanations of each step are as follows.
more suitable to design a specific task. For example, to judge whether Step 1. Construction of the hierarchical structure model. According
the assembled models of a robot arm are consistent with the physical to the features of DTS models, the hierarchical structure model is built as
arms, a specific task can be allocated to this arm. In the virtual space, the shown in Fig. 4. The consistency degree of all models mentioned in
arm would move and rotate according to the target position. In the same Section 3 are the main indexes of comprehensive consistency evaluation.
time, the physical arm would execute the same task. Finally, the con­ In the process of consistency evaluation, not all indexes must be covered.
sistency evaluation can be reached by comparison between the practical We usually choose some meaningful indexes to calculate according to
position and posture, and model position and posture. the specific objects and goals.
The consistency evaluation method for fused models is similar with Step 2. Construction of the judgement matrix. The judgement matrix
assembled model. The consistency evaluation is to judge whether is used to depict the degree of importance among different factors.
various multi-domain models could work together to achieve the desired Table 1 provides a reference on the values of the important degree be­
results according to defined ports and coupling relationships. Therefore, tween different factors. Then, the judgement matrix could be expressed

163
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

1
aij = (19)
aji

aii = 1 (20)
Step 3. Weight calculation. The weight can be calculated according
to (21). λmax denotes the largest eigenvalue of matrix A. w denotes the
eigenvector, which is also the weight vector.
Aw = λmax w (21)
Step 4. Consistency evaluation for the judgement matrix. This step
mainly assesses the degree of logic crash in the judgement matrix. It
should be noticed this consistency is different from the consistency
evaluation for digital twin models in Section 2 and Section 3. CI repre­
sents the consistency index, which can be calculated as (22). The smaller
CI is, the better the consistency of the judgement matrix is. RI denotes
mean random consistency index. Table 2 provides a reference on the
values of RI [16]. Then, consistency ratio (CR) can be obtained ac­
cording to (23). If CR < 0.1, it is believed that the judgement matrix
could meet the demands. Otherwise, the judgement matrix should be
adjusted [27].
λmax − n
CI = (22)
n− 1

CI
CR = (23)
RI
Step 5. The comprehensive consistency evaluation towards digital
twin models. According to the weight in Step 3, the comprehensive
Fig. 3. The flow chart of comprehensive consistency evaluation towards digital consistency evaluation results can be calculated.
twin models.
5. Case study

5.1. Case description

In this section, a complex product AIT shop-floor is used to explain

how to use the proposed method to evaluate the degree of consistency
between digital twin models and the corresponding physical objects. To
show the differences among the objects at different layers in DTS, the
power-assisted pose adjustment spreader and the shop-floor are chosen
as the application objects to explain respectively.

5.2. Consistency evaluation towards machine models

The power-assisted pose adjustment spreader includes a three-

Fig. 4. The hierarchy structure model for DTS models.
translational mechanism which is mainly used to support moving
along the xyz axis, a 3-SPS/S adjusting parallel mechanism which is the
Table 1 key mechanism for adjusting the attitude, and a supporting mechanism.
Scale of relative importance. Because the supporting mechanism belongs to non-key components,
more attention is paid on three-translational mechanism and 3-SPS/S
Value Meaning
adjusting parallel mechanism. Considering the characteristics of the
1 factor i and factor j are equally important power-assisted pose adjustment spreader, the consistency evaluation
3 factor i is moderate important in comparison to factor j
5 factor i is strongly important in comparison to factor j
focuses on the geometry model, behavior model and rule model.
7 factor i is very strongly important in comparison to factor j Moreover, assembled models and fused models are related with each
9 factor i is extremely important in comparison to factor j other. Therefore, the consistency evaluation for assembled models and
2,4,6,8 the importance between the two adjacent cases above fusion models can be conducted together.
According to the functions and features of the power-assisted pose
adjustment spreader, the scheme of consistency evaluation for machine
as (18). where aij denotes the importance of factor i in comparison to
digital twin models is conducted as follows. In geometry dimension, the
factor j. The properties of aij can be expressed as Eqs. (19)-(20) [28].
design parameters of the machine could be obtained, which are accurate
⎡ a11 a12 … a1n ⎤ to the millimeter. The components and parts are not easy to wear.
Therefore, the geometry models of various parts and components are
⎢ a21 a22 … a2n ⎥
deemed to be highly consistent with modeling objects. The consistency
A=⎣ ⎦ (18)
⋮ ⋮ ⋱ ⋮ evaluation result can be considered to be 1. In behavior dimension, the
an1 an2 ⋯ ann logic of task execution is expressed based on activity diagram. For
power-assisted pose adjustment spreader, the position and posture are

164
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

Table 2
The mean random consistency index.
n 1 2 3 4 5 6 7 8 9 10
RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.51

interlocking, which means rotation and translational motion can not be information for the power-assisted pose adjustment spreader.
done in the same time. As shown in Fig. 5, The activity diagram includes
action states, transitions, decisions, etc. The consistency result can be
calculated based on the ratio of the number of the correct elements to the 5.3. Consistency evaluation towards shop-floor models
total number of the elements. The consistency evaluation result of the
behavior model is 1. In the rule dimension, the constraint rules are For the shop-floor, the models of key elements in complex product
mainly considered. Specifically, the angular range of pitch, roll and yaw AIT shop-floor, including AGV, glue spreading machine, optical scan­
is ±5◦ . And the maximum stroke in the x,y,z direction is 3 m, 1.5 m, and ning device, leak detection bench, etc., should be evaluated in the ma­
1.2 m. Based on the practical situations, it can be judged that the rule chine layer. Therefore, the consistency evaluation for DTS models
model is correct. concentrates on the correctness of the material flow, information flow,
To further evaluate the assembled models and fused models of the and process flow. In this section, a timed colored Petri net is used to
power-assisted pose adjustment spreader, the roof installation above the depict the relationships among the complex product, process, and
complex product +Z is used as the test condition, as shown in Fig. 6. The various resources. Apart from place, transition, and arcs (flow relation),
task for the power-assisted pose adjustment spreader is to move the roof the timed colored Petri net could be used to realize the modeling and
from the starting position (2155,491.5,1000) to the end position analysis for large complex distributed manufacturing systems by intro­
(330.91,550.46,200). According to target position, the specific move­ ducing color set and time attributes. The different color sets could
ment route is planned. To better know the machine operating condi­ represent different resources in shop-floor. And the time attributes could
tions, a laser tracker is used to get position information. The target is reflect the execution time in each operation [29,30].
attached to the end of the power-assisted pose adjustment spreader. And The complex product AIT shop-floor model has been built as Fig. 7.
the position coordinates can be obtained through the supporting soft­ s1-s4 denote the components of the complex product to be assembled, i.
ware. 24 sets of position data are collected during task execution. Then, e. the cabin board, the parts on the cabin board, the roof of the complex
based on the model data and real data, the consistency could be calcu­ product, and the main body of the complex product. w1-w6 represent
lated by comparison the position or attitude between the model and the the workers which operate equipment or systems. The transition t1-t15
physical equipment. According to Theil’s inequality coefficient, the present the process of complex product assembly, including cabin board
values of consistency degree could be calculated. The value is 0.0023, installation, roof installation, leak detection, etc. The unit of time on
which illustrates that the model is almost the same with the physical transitions is seconds, which represents the execution time in each
object. process step. r1-r8 are the machines or tools in the process of complex
Based on the analysis results above, the consistency evaluation for product assembly, e.g. gumming machine, the power-assisted pose
digital twin models could be obtained by using AHP. The value is more adjustment spreader, automatic guided vehicle. p1-p15 are the states of
than 99 %, which illustrates that the models are not only correct but the complex product in different stages. Based on CPN Tools, a tool for
accurate. The reason of high consistency is that there is sufficient editing, simulating, and analyzing the colored petri nets, the model of
complex product AIT shop-floor could be simulated. The consistency

Fig. 5. The activity diagram for the power-assisted pose adjustment spreader.

165
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

Fig. 6. The consistency evaluation for the power-assisted pose adjustment spreader model.

evaluation could be realized by comparing the states in the models and requirements, the processes could be conducted, which has already
physical shop-floor at the same time. For example, in 200 s, the robotic embodied into the model in Fig. 7. In other words, the transition could
arm moves the parts. If the physical shop-floor execute the same process, be triggered under the condition that place, token, and arc are correct.
it can be considered the model to be consistent with the modeling object. The rule settings are reasonable, and the consistency evaluation result
The quantization results of consistency evaluation could be calculated for the rule model can be considered to be 1. Finally, based on the AHP,
∑15
based on Cd = 1 − i=1 Tei /Tpi × wi . Tei represents the time errors be­
the comprehensive consistency evaluation result for the complex prod­
tween the models and practical situations in each process step. Tpi rep­ uct AIT shop-floor model is 0.9074.
resents the practical execution time in each process step. And wi is the
weight of each process step. In this case, all the processes are considered 6. Conclusion and outlooks
equally important, i.e. wi = 1/15 (i = 1,2,…15). Then the consistency
degree can be calculated based on the practical execution time. The The consistency evaluation between digital twin models and the
value of consistency degree is 0.8574. The model is not very accurate. corresponding elements in shop-floor is significant for DTS imple­
On the one hand, the operation time of workers is affected by various mentation. Against the background, this paper has proposed a consis­
factors, e.g. the states and proficiency of workers, which leads to that it tency evaluation method towards DTS models. The two main stages,
is difficult to predict the operation time. On the other hand, the execu­ before and after model assembly and model fusion, are considered.
tion time of some processes is uncertain. During the consistency evalu­ Before model assembly and fusion, the consistency evaluation methods
ation, it is found that the fastener installation tool is neglected, which for geometry models, physics models, behavior models and rule models
leads to the deletion of one place and the arcs between the tool and are discussed. After model assembly and fusion, how to judge whether
transition t3. The consistency degree on coupling relationship can be introduced relations are correct and accurate is also explored. Further­
calculated based on (17). The value is 0.9827. The rule model mainly more, the AHP method is used to give a comprehensive evaluation result
refers to that only the workers, tools, or materials are all meet the for DTS models. And a complex product AIT shop-floor and a key

166
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

Fig. 7. The consistency evaluation for the complex product AIT shop-floor model.

machine are chosen as the case to illustrate the proposed method. The [3] Tao F, Zhang M. Digital twin shop-floor: a new shop-floor paradigm towards smart
manufacturing. IEEE Access 2017;5:20418–27. https://doi.org/10.1109/
proposed framework and method could provide a reference for under­
ACCESS.2017.2756069.
standing and using DTS models, which could further influence the [4] Wang Y, Tao F, Zhang M, Wang L, Zuo Y. Digital twin enhanced fault prediction for
quality of the services based on models. In the future, more efforts the autoclave with insufficient data. J Manuf Syst 2021;60:350–9. https://doi.org/
should be made to support using digital twin models better. For 10.1016/j.jmsy.2021.05.015.
[5] Tao F, Zhang M, Liu Y, Nee AYC. Digital twin driven prognostics and health
example, if digital twin models are not consistent with the correspond­ management for complex equipment. CIRP Ann 2018;67:169–72. https://doi.org/
ing physical objects, it should be solved that what strategy would be 10.1016/j.cirp.2018.04.055.
adopted to adjust the models and how to realize model modification by [6] Luo W, Hu T, Zhang C, Wei Y. Digital twin for CNC machine tool: modeling and
using strategy. J Ambient Intell Human Comput 2019;10:1129–40. https://doi.
using real-time data. Besides, it should also be explored that how to org/10.1007/s12652-018-0946-5.
conduct the consistency evaluation for some other kinds of digital twin [7] Liu J, Liu J, Zhuang C, Liu Z, Miao T. Construction method of shop-floor digital
models, twin based on MBSE. J Manuf Syst 2021;60:93–118. https://doi.org/10.1016/j.
jmsy.2021.05.004.
[8] Liu S, Lu Y, Li J, Song D, Sun X, Bao J. Multi-scale evolution mechanism and
knowledge construction of a digital twin mimic model. Robot Comput Integr Manuf
Declaration of Competing Interest
2021;71:102123. https://doi.org/10.1016/j.rcim.2021.102123.
[9] Talkhestani BA, Jazdi N, Schloegl W, Weyrich M. Consistency check to synchronize
The authors declare that they have no known competing financial the digital twin of manufacturing automation based on anchor points. Procedia
interests or personal relationships that could have appeared to influence CIRP 2018;72:159–64. https://doi.org/10.1016/j.procir.2018.03.166.
[10] Wei Y, Hu T, Zhou T, Ye Y, Luo W. Consistency retention method for CNC machine
the work reported in this paper. tool digital twin model. J Manuf Syst 2020;58:313–22. https://doi.org/10.1016/j.
jmsy.2020.06.002.
[11] Zhang C, Sun Q, Sun W, Mu X, Wang Y. A construction method of digital twin
Acknowledgements
model for contact characteristics of assembly interface. Int J Adv Manuf Technol
2021;13(9):2685–99. https://doi.org/10.1007/s00170-021-06751-x.
This work is financially supported by National Key Research and [12] Yu J, Song Y, Tang D, Dai J. A digital twin approach based on nonparametric
Development Program of China under Grant 2020YFB1708400, Na­ bayesian network for complex system health monitoring. J Manuf Syst 2021;58:
293–304. https://doi.org/10.1016/j.jmsy.2020.07.005.
tional Natural Science Foundation of China (NSFC) under Grant [13] Vijayakumar K, Dhanasekaran C, Pugazhenthi R, Sivaganesan S. Digital twin for
52005024, 52120105008. The authors also would like to thank Beijing factory system simulation. Int J Recent Technol Eng 2019;8(1):63–8.
Institute of Satellite Environment for providing information and data in [14] Liu S, Sun Y, Zheng P, Lu Y, Bao J. Establishing a reliable mechanism model of the
digital twin machining system: an adaptive evaluation network approach. J Manuf
case study. Syst 2022;62:390–401. https://doi.org/10.1016/j.jmsy.2021.12.008.
[15] Jiang H, Qin S, Fu J, Zhang J, Ding G. How to model and implement connections
References between physical and virtual models for digital twin application. J Manuf Syst
2021;58:36–51. https://doi.org/10.1016/j.jmsy.2020.05.012.
[16] Jia W, Wang W, Zhang Z. From simple digital twin to complex digital twin Part I: a
[1] Liu M, Fang S, Dong H, Xu C. Review of digital twin about concepts, technologies, novel modeling method for multi-scale and multi-scenario digital twin. Adv Eng
and industrial applications. J Manuf Syst 2021;58:346–61. https://doi.org/ Inform 2022;53:101706. https://doi.org/10.1016/j.aei.2022.101706.
10.1016/j.jmsy.2020.06.017. [17] Moghadam FK, Nejad AR. Online condition monitoring of floating wind turbines
[2] Negri E, Fumagalli L, Macchi M. A review of the roles of digital twin in CPS-based drivetrain by means of digital twin. Mech Syst Signal Process 2022;162:108087.
production systems. Procedia Manuf 2017;11:939–48. https://doi.org/10.1016/j. https://doi.org/10.1016/j.ymssp.2021.108087.
promfg.2017.07.198.

167
H. Zhang et al. Journal of Manufacturing Systems 65 (2022) 158–168

[18] Zhang H, Qi Q, Tao F. A multi-scale modeling method for digital twin shop-floor. time systems with energy constraints. 2008 Int Conf Adv Electron Micro-Electron,
J Manuf Syst 2022;62:417–28. https://doi.org/10.1016/j.jmsy.2021.12.011. Valencia, Spain 2008:1–6. https://doi.org/10.1109/ENICS.2008.19.
[19] Roungas B, Meijer SA, Verbraeck A. A framework for optimizing simulation model [26] Huang E, McGinnis LF, Mitchell SW. Verifying SysML activity diagrams using
validation & verification. Int J Adv Syst Meas 2018;11:137–52. formal transformation to Petri nets. Syst Eng 2019;23:118–35. https://doi.org/
[20] Song J, Wei L, Ming Y. A method for simulation model validation based on Theil’s 10.1002/sys.21524.
inequality coefficient and principal component analysis. Asian Simul Conf Berl, Ger [27] Acharya V, Sharma SK, Gupta SK. Analyzing the factors in industrial automation
2013:126–35. using analytic hierarchy process. Comput Electr Eng 2017;71:877–86. https://doi.
[21] Yu J, Cheng X, Lu L, Wu B. A machine vision method for measurement of org/10.1016/j.compeleceng.2017.08.015.
machining tool wear. Measurement 2021;182:109683. https://doi.org/10.1016/j. [28] Tian G, Zhang H, Zhou M, Li Z. AHP, gray correlation, and TOPSIS combined
measurement.2021.109683. approach to green performance evaluation of design alternatives. IEEE Trans Syst
[22] Hashmi AW, Mali HS, Meena A, Khilji IA, Saffe SNM. Machine vision for the Man Cybern Syst 2018;48:1093–105. https://doi.org/10.1109/
measurement of machining parameters: a review. Mater Today: Proc 2022: TSMC.2016.2640179.
1939–46. https://doi.org/10.1016/j.matpr.2021.11.271. [29] Gehlot V, Nigro C. An introduction to systems modeling and simulation with
[23] Liu SC, Wei W, Yan Q, Lai Y. Credibility analysis of simulation model for torque Colored Petri nets. Proc 2010 Winter Simul Conf Baltim, MD, USA 2010:104–18.
converters based on complexity measurement. Trans Chin Soc Agric Mach 2012;43: https://doi.org/10.1109/WSC.2010.5679170.
19–24. [30] Lv YQ, Lee CKM, Wu Z, Chan HK, Ip WH. Priority-based Distributed manufacturing
[24] Hause M. The SysML modelling language. 15th Eur Syst Eng Conf 2006:1–12. process modeling via hierarchical timed color Petri net. IEEE Trans Ind Inform
[25] Carneiro E, Maciel P, Callou G, Tavares E, Nogueira B. Mapping sysml state 2012;9:1836–46. https://doi.org/10.1109/TII.2012.2215043.
machine diagram to time petri net for analysis and verification of embedded real-

168