Digital Twin Technology of Human–Machine Integration in Cross-Belt Sorting System

Abstract

The Chinese express delivery industry processes nearly 110 billion items in 2022, averaging an annual growth rate of 200%. Among the various types of sorting systems used for handling express items, cross-belt sorting systems stand out as the most crucial. However, despite their high degree of automation, the workload for operators has intensified owing to the surging volume of express items. In the era of Industry 5.0, it is imperative to adopt new technologies that not only enhance worker welfare but also improve the efficiency of cross-belt systems. Striking a balance between efficiency in handling express items and operator well-being is challenging. Digital twin technology offers a promising solution in this respect. A realization method of a human–machine integrated digital twin is proposed in this study, enabling the interaction of biological human bodies, virtual human bodies, virtual equipment, and logistics equipment in a closed loop, thus setting an operating framework. Key technologies in the proposed framework include a collection of heterogeneous data from multiple sources, construction of the relationship between operator fatigue and operation efficiency based on physiological measurements, virtual model construction, and an online optimization module based on real-time simulation. The feasibility of the proposed method was verified in an express distribution center.

1 Introduction

From 2012 to 2021, China’s annual express business volume increased from 5.7 to 108.3 billion pieces, accomplishing an 18-fold increase. Since 2014, the express business volume has consistently ranked first worldwide for eight consecutive years. Express distribution centers serve as logistical strongholds that integrate various functions, including receiving, dispatching, distribution, and sorting. They are widely recognized as the “heart” of express enterprises. With the continuous advancement of technology, distribution centers have been equipped with automated logistics equipment, thereby significantly enhancing their operational efficiency and reducing their dependence on manual labor. By the end of June 2022, the industry boasted over 400 fully automated distribution centers.

Automatic sorting systems constitute the forefront of modern logistics technology and serve as vital components in advanced distribution centers owing to their high sorting efficiency. In contrast to manual sorting, automated sorting systems utilize automated control and information recognition technologies to achieve efficient and precise sorting of express parcels. This significantly reduces the risk of parcel loss, enhances parcel safety, and minimizes labor costs. The courier sector employs various types of automated sorting systems; in particular, cross-belt automated sorting systems stand out as the most popular and mature technology, offering a broader range of applications than other types of sorting systems.

However, the widespread use of cross-belt sorting systems also presents new challenges. Certain key aspects of cross-belt sorting systems still rely heavily on human involvement. Currently, there are no proven intelligent alternatives to human labor for tasks such as unpacking, infeeding, packing, and manually complementing codes throughout the cross-belt sorting process. The work environment in sorting centers, as depicted in Figure 1, is complex, featuring equipment noise, light pollution from industrial cameras, and harsh weather conditions (“colder in winter and hotter in summer”). This environment is not conducive to extensive periods of continuous work from a “human-centered” perspective. Furthermore, several ecommerce sales periods occur throughout the year, such as “618”, “Double 11”, and “Double 12”, resulting in a surge in the volume of express items. This leads to high-intensity work, which further impacts the health of workers.

Figure 1

Working environment of parcel sorting

In 2021, the European Union introduced a new concept for the development of Industry 5.0, highlighting the significance of prioritizing the interests of workers at the core of the operational process [1]. In this context, distribution centers are evolving in a human-centered, intelligent, and environmentally friendly direction. The seamless integration and safe interaction between humans and machines are of utmost importance. Human beings represent the most dynamic and energetic element of the system, and the exploration and examination of the relationship between workers and machines have always been intertwined with the overall advancement of the industry.

The difficulty in human–machine integration within sorting systems lies in the real-time and dynamic nature of human–machine behavior, coupled with the ever-changing fatigue state of operators. Thus, it becomes crucial to adjust the operation plan of the system to safeguard the interests of the operators while simultaneously improving the handling capacity of the system. Digital twin technology, as an emerging field, has garnered significant attention from researchers. Leveraging its features of virtual-real interaction, real-time mapping, and dynamic evaluation, this technology can effectively assess the state of humans and machines in real time and determine corresponding strategy adjustments, thereby enabling seamless human–machine integration.

Therefore, this study combines the unique characteristics of digital twin technology to explore human–machine integration techniques for cross-belt sorting systems. The main objective is to enhance the processing capability of sorting systems while keeping human-centered considerations in mind.

The paper is organized as follows. Section 1 presents a comprehensive literature review of existing research concerning cross-belt sorting systems and digital twin technology. In Section 2, a method for human–machine co-integration through digital twins is proposed to facilitate closed-loop interaction among biological humans, virtual humans, virtual devices, and physical devices. Additionally, an operating framework designed to support this integration is described. Section 3 summarizes the key technical aspects of constructing the operating system of a human–machine sorting system for its digital twin. In Section 4, the construction of a digital twin cross-belt sorting system is described; this system allows the verification of the effectiveness of different sorting strategies.

2 Literature Review

2.1 Cross-Belt Sorting System

The automatic sorting system is a pivotal equipment in modern distribution centers. Currently, several types of automatic sorting systems are available in the market, including swing-arm, tilt-tray, cross-belt, and automated guided vehicle (AGV) sorting systems. Each type of sorting system features distinct operation principles with its own advantages, characteristics, and application scenarios. Swing-arm sorting systems constitute a relatively precise sorting solution widely employed in express distribution centers. Their working principle involves using rotating baffles to block goods on the conveyor line, thereby guiding them into the correct channel. However, as a result of technological advancements, limitations of swing-arm systems such as constrained sorting options, low efficiency, and high breakage rates of express goods have gradually led to a declining presence in the market. By contrast, tilt-tray sorting systems are efficient and highly accurate. They are extensively used in airport baggage sorting. This system employs tray tilting to direct express items to the correct channel. Nonetheless, they are not suitable for sorting soft bags and lightweight pieces owing to the lack of power between the tray and express items. Consequently, their usage in the express industry remains limited. AGV sorting systems rely on assigned AGV cars that transport express goods and shelves to their designated locations for sorting. These systems are typically applied in electronic shopping scenarios, such as Amazon’s Kiva robot, which facilitates goods picking for the operator. They have also been applied in express scenarios, such as logo sorting. Nevertheless, AGV systems demand stringent site requirements and substantial investment, resulting in limited adoption by the industry.

Cross-belt sorting systems comprise a central drive belt conveyor linked to a trolley (also known as carriage) equipped with a small belt conveyor. As the trolley moves to the designated sorting position, the conveyor belt rotates to facilitate the sorting and delivery of the express items. This type of sorting system derives its name from the cross-shaped arrangement formed by the main drive conveyor and belt conveyor on the trolley. It is the most widely used sorting system in the express delivery industry. Compared to that of other types of sorting systems, the popularity of the cross-belt system stems from its broad applicability resulting from low investment requirements, sorting flexibility, comprehensive sorting categories, and technological maturity.

The typical structure of a cross-belt sorting system is illustrated in Figure 2; it primarily consists of cross-belt trolleys, a control system, a feed desk, driving devices, tracks, chutes, an information management system, and information recognition units. The workflow of the sorting system is displayed in Figure 3. When the express mail arrives, the operator unpacks the bags, separates the express items into a single-piece stream, and loads them into the system. After weighing the express items, measuring their volume, detecting multiple items within a single vehicle, and scanning barcode identifiers, the express items are dropped into the lattice. Once the number of express items in the lattice matches the desired quantity for packaging, the operator completes the parcel collection process. Considering the unpacking process, which requires manual involvement, as an example, one person can unpack approximately 300 pieces per hour, assuming that an unpacking task is completed every 12 s, and the working time is approximately 8 h. Standing for long periods of time and bending over frequently have an impact on the health of personnel. The work projects of each operator are shown in Figure 4.

Figure 2

Schematic diagram of the cross-belt sorting system

Figure 3

Workflow of the cross-belt sorting system

Figure 4

The work projects of each operator

Research on sorting systems primarily revolves around enhancing the efficiency of the infeed system and optimizing the sorting strategy. Boysen et al. [2] based on a novel two-step multiple-scenario approach applied under real-time conditions, can be a serviceable tool to significantly improve the sortation throughput. Zhang et al. [3] proposed the multi-object sorting system based on deep learning can sort stacked objects efficiently, accurately and stably in unstructured scenes. Tan et al. [4] investigates the optimization problem of parcel sorting in an e-commerce warehouse where parcels are waiting to be sorted and delivered. The assignment among the parcels, picking stations and AGVs are determined together with the objective of minimizing the finishing time of processing the last parcel, so that a detailed operational plan for warehouse sorting can be worked out.

In summary, cross-belt sorting systems are technically mature. However, they are still inseparable from manual participation in some links, and the work intensity is high.

2.2 Digital Twin

Digital twin is an innovative technique for predicting the dynamic behavior of systems. The concept of digital twin was first introduced in 2003 by Professor Michael Grieves in a product cycle management course at the University of Michigan. Initially, it was mainly applied in the military and aerospace sectors [5]. For example, the U.S. Air Force Research Laboratory and NASA have developed vehicle health control applications based on digital twin technology, and Lockheed Martin has applied digital twin technology to the F-35 fighter jet production process to improve the process and increase production efficiency [6]. Lu et al. [7] summarized the meaning and application scenarios of digital twin-driven intelligent manufacturing within Industry 4.0. Zhang et al. [8] summarized the definition of digital twin provided by several researchers in recent years and proposed that a digital twin is defined as a digital model of physical objects. This digital model can evolve in real time by receiving data from physical objects, ensuring consistency with physical objects throughout their entire life cycle. Guo et al. [9] proposed a digital twin modeling framework based on digital twins and studied issues such as resource modeling and digital definition of process information.

Digital twin technology has been extensively researched and implemented in various fields, including smart factory construction, warehouse management, workshop optimization scheduling, and human health supervision. Tao et al. [10] introduced a five-dimensional model of a digital twin and explored its applications in ships, vehicles, and three-dimensional storage. Zhao et al. [11] proposed a digital twin-driven method for energy-efficient scheduling of multiple overhead cranes in workshops. They evaluated the energy consumption of different configurations of overhead cranes for operations in a virtual system and utilized genetic algorithms to obtain the optimal selection and scheduling scheme for a number of overhead cranes. Research conducted demonstrates that digital twin technology can effectively address dynamic shop floor scheduling. By integrating a reinforcement learning algorithm, it becomes feasible to achieve dynamic scheduling management for shop floor operations [12, 13]. Kosacka-Olejnik et al. [14] conducted a comprehensive review of the application of digital twins in logistics and warehousing systems, presenting current research trends in these areas. Bao et al. [15] introduced a methodology for establishing digital twins encompassing products, processes, and operations, with a focus on achieving the fusion of physical and virtual domains within the context of a manufacturing workshop. These digital twins simulate the conditions and behavior of tangible entities while concurrently enhancing the optimization of the production workflow. Hu et al. [16] introduced the concept of a digital twin diagnosis and treatment system, detailing the methodology for establishing such a system. They also explored its applications in clinical diagnosis and treatment, basic medical research, education and training, and medical device development. Liu et al. [17] utilized a digital twin to drive a flowing workshop, achieving mutual iterative optimization of the static configuration and dynamic execution mechanism of the system. Furthermore, Minerva et al. [18]) established a digital twin model for a specific intelligent logistics center, significantly enhancing the operational efficiency of the logistics center through optimization strategies, such as real-time scheduling and resource allocation. Leng et al. [19] proposed a digital twin joint-optimization method, providing periodic optimal decisions for high-level repositories and significantly improving the efficiency and utilization of automated stereoscopic repositories. The intelligent operation of physical systems through simulation optimization and virtual-real interaction [20], offers a wide range of applications in smart factory construction for smart manufacturing [21, 22], production workshops [23], distributed smart workshops [24], shop-floor production processes [25], and intelligent production management and control [26]. Lu et al. [27] studied a digital twin construction method for smart manufacturing, providing support for the realization of the entire life cycle of the manufacturing industry. Humans are important in every production process. Consequently, they have received attention from researchers. Building upon the principle of human-centric design, Bao et al. [28] introduced a collaborative digital twin technology for human–machine–environment integration. This innovation optimizes the utilization of robots while enhancing both the ergonomic quality and overall job satisfaction of human workers. Wang et al. proposed a framework for human–machine integration at the enterprise level and highlighted that under the concept of “human-centered” development, a human digital twin holds great promise as a research direction [29, 30]. Song et al. [31] proposed a digital twin framework for human–machine–environment integration taking the human skeleton into account. They exemplified the bidirectional mapping of the physical and digital space of the human lumbar spine, verifying the feasibility of a human digital twin framework for human–machine interaction, production monitoring, and health supervision. These studies have explored the application of digital twin technology in various fields. However, there is no research study on the application of digital twins to sorting systems. Based on the existing research, we propose a construction method for cross-belt sorting systems based on a human–machine-integration digital twin.

It should be noted that there is a need for research on digital twin technology for application in cross-belt sorting systems. Significant breakthroughs are required in core theories and technological innovations, particularly in sorting systems where the entire process cannot be fully automated. Integrating a digital twin with human–machine interaction enhances the flexibility and adaptability of such systems, ultimately promoting sustainable development. Since ZJUESA is designed by following the physiological parameters of the human upper limb, with such a device the human operator can control the manipulator more comfortably and intuitively than the system with the joystick or the keyboard input.

3 Framework of Human–Machine Integrated Digital Twin Cross-Belt Sorting System

3.1 Theoretical Configuration of Human–Machine Communion

With the advent of the new industrial paradigm—the concept of Industry 5.0—and the ongoing advancement of digital twin technology, human-centered digital twin systems have garnered attention from researchers across various disciplines. Built upon the principles of the digital twin concept, real-time interaction, and seamless integration of reality and virtuality, such systems can address challenges in virtual scenarios that are difficult in real-world situations, thus exhibiting promising prospects for achieving harmonious human–machine integration and collaborative operation. Notably, the fatigue of operators, being at the core of the sorting process, significantly impacts the processing capacity and error rate of the system.

This study proposes a digital twin framework for sorting systems. It is inspired by a five-dimensional model of the digital twin and is specifically tailored for human–machine relationships (Figure 5). The framework comprises four primary components: human–machine physical space, human–machine virtual space, twin data centers, and services.

• (1) The human–machine physical space serves as the foundation of the framework, constituting an objective collection of entities within the space that primarily encompasses various operators, sorting equipment, sensing equipment, and communication devices. It possesses the capability of sensing access and fusing heterogeneous multi-source data. In the context of the human–machine communion relationship, human beings, occupying a central role in the structure, assume diverse roles such as operators, decision-makers, and service recipients [32]. By gathering physiological, psychological, and work efficiency data of individuals within the physical space, machines can swiftly and accurately discern the physical and mental states of individuals, thereby facilitating a “people-oriented” approach to factory operations. As active participants and executors within the human–machine communion relationship, machines bolster device control and cooperation through data acquisition related to the geometry, physics, behavior, and rules of the machine. Both entities engage in interactions within the physical space.

• (2) The human–machine virtual space constitutes the core of the framework, representing a collection of entity-based high-fidelity models within the digital simulation platform. Through the construction of mechanism and data models corresponding to humans and machines, this virtual space can achieve functionalities such as scene replication, virtual-to-real mapping, dynamic updates, and decision guidance. The process of twinning models necessitates the selection of modeling tools and platforms, the establishment of real-time model updates and interaction interfaces, and the determination of the composition and presentation format of the twinning model. Within the virtual space, both humans and machines assume roles as participants, performers, and service recipients.

• (3) The twin data center serves as the bridge that connects the human–machine digital and virtual spaces, providing a data integration and sharing platform encompassing all elements, processes, and services. It eliminates information silos and enables a closed-loop interaction between the digital and virtual human–machine spaces. Data processing, transmission, and analysis are integral to the entire model lifecycle. The realization of the digital twin necessitates the management of vast amounts of data, with real-time man–machine data centrally stored in the twin data center and processed and analyzed through cloud computing and big data technologies. The monitoring, evaluation, and early warning of the physiological state of the operator and operating conditions of the equipment are achieved by combining real-time data with high-fidelity digital models.

• (4) Service is a pivotal component within the human–machine integrated digital twin framework. The challenge lies in effectively merging human–machine capabilities and providing optimal strategies to achieve target tasks more efficiently and effectively, while reducing human workload and enhancing worker satisfaction. This is accomplished by encapsulating diverse data, models, algorithms, and results, including support services that facilitate the mapping between the aforementioned three components in the form of tool groups, module engines, and the like.

• (5) Application services, presented in the form of application software, cater to customer requirements. Simultaneously, the application service fosters a novel interactive relationship between the biological body and virtual device, enabling operators to engage in operational training within the virtual device interface and access corresponding contingency measures for various scenarios.

Figure 5

Human–machine inclusion digital twin

Through the interplay of the aforementioned four components, a continuous, iterative, and optimized real-time management process is formed between the virtual and real. This fosters the integration and continuous fulfillment of four relationships within the sorting system between the biological, virtual, and physical entities:

$$DTF \, :: = \, \{ PE,VE,DD,SS\} ,$$

(1)

where $DTF$ denotes the digital twin framework, $:: =$ denotes “definition”, $PE$ denotes the physical space of the sorting center, $VE$ denotes the virtual space of the sorting center, $DD$ denotes the twin data center, and $SS$ denotes the support and application services.

3.2 System Reference Architecture

Tao et al. [33] proposed a digital twin maturity model that categorizes the maturity of a digital twin into six levels:

• I. The digital twin model is used to describe and characterize physical entities.

• II. The digital twin model is used to reproduce the real-time state and change the process of physical entities in real time.

• III. The digital twin model is used for indirect control of the operation process of physical entities.

• IV. The digital twin model is used for indirect control of the operation process of physical entities.

• V. The digital twin model is used to optimize physical entities.

• VI. The physical entity and digital twin model autonomously realize twinning through dynamic reconstruction throughout the life cycle.

The operational goal of a human–machine integrated digital twin cross-belt sorting system (MMIDTCBSS) is to achieve optimal control of the sorting process while simultaneously reducing the workload of workers and improving work process satisfaction. This corresponds to the fourth level of the aforementioned digital twin maturity model. At this level, the digital twin model not only provides real-time reflection of the operational status of the physical space and predicts its future development but also leverages the strategies, algorithms, and knowledge accumulated and registered during earlier stages to achieve time-sensitive intelligent decision-making and optimization. This enables intelligent control of the physical space based on real-time interaction mechanisms. The operating framework of the proposed MMIDTCBSS is shown in Figure 6.

Figure 6

The operating framework of the MMIDTCBSS

3.2.1 Physical Layer

The physical layer comprises four main components: physical entities, control and execution module, perception module, and network communication module. The physical entities consist of cross-belt sorters, modular belt units, conveyors, and operators. The perception module encompasses equipment perception, human body perception, process perception, and task perception. The control and execution module includes a warehouse control system (WCS), programmable logic controller (PLC), display terminal, and other related elements. The network communication module employs wireless, radio frequency identification (RFID) devices, and other communication methods. During system operation, data from the physical space system as well as real-time data are collected through various sensors and transmitted to the twin data center using data transmission protocols and wireless communication. Simultaneously, the control and execution module responds to commands received from the twin data center.

3.2.2 Data Layer

The data layer is a fundamental component utilized to receive multi-source heterogeneous data transmitted in the physical space. It encompasses various essential functions, including data cleaning and transformation, data storage and management, data comparison and analysis, and data security and protection. The data cleaning and conversion module plays a crucial role in effectively eliminating invalid, duplicate, and abnormal data after collection, while also converting data from different sources, formats, and types to ensure uniformity and compatibility. Within the data storage and management module, data related to equipment operation, human body state, sorting plan, simulation, instruction, and model are systematically classified and stored in the form of a data warehouse. This organized approach facilitates efficient data retrieval and management. The data comparison and analysis module efficiently identifies abnormal situations during the system operation. It provides decision results and generates the corresponding types of instructions, offering valuable insights for system optimization. The data security and protection module implements access control, data encryption, data backup, and other measures to safeguard all types of data in the system, thereby preventing leakage, damage, tampering, and unauthorized access.

3.2.3 Virtual Layer

The virtual layer serves as an abstract and digital representation of the physical world, encompassing multidimensional human–machine models and model-driven optimization decision modules. The multi-dimensional model is constructed based on geometric, physical, behavioral, and rule models to create an accurate digital image of the cross-belt sorter and operator. The optimization decision module can be continuously optimized and enhanced through the updating and feedback of real-time data, thereby enabling the selection of optimal strategies based on various objectives. This iterative process ensures a constant improvement in the accuracy and reliability of the digital model.

3.2.4 Virtual Layer

The service layer is a comprehensive encapsulation of various algorithms and models that facilitate the operation and implementation of internal functions within the digital twin system. It comprises services oriented towards digital model verification, validation, and accreditation (VV&A), services for task-specific model granularity selection, data-oriented management services, models for operator fatigue state and operational efficiency transformation, and multi-objective combined optimization services for cross-belt sorters.

3.2.5 Application Layer

The application layer consists of a diverse set of user-oriented functions, offering standardized input and output through application software, thus enabling users to easily utilize it according to their specific requirements. This layer encompasses fault monitoring for sorters, health detection for operators, an energy-saving system based on shipment arrival time, and a dispatch system based on operator fatigue state.

4 Key Technology of MMIDTCBSS

4.1 Collection of Heterogeneous Data from Multiple Sources

The real-time data generated by individuals and equipment in the distribution center during the sorting process serve as the cornerstone for process optimization and control. To meet diverse data collection needs, a variety of sensing devices are deployed to enable interconnection and mutual sensing between operators and machines. This ensures real-time and reliable access to multiple sources of information during the sorting process, encompassing physical system data and real-time operational data from diverse and complex sources and structures. Therefore, data acquisition and transmission constitute the key technologies for implementing digital twin sorting systems.

Notably, a modern distribution center features a high degree of automation, robust communication interface and network, and comprehensive logistics management system, all of which facilitate data collection. The data acquisition for system equipment operation includes PLCs, data acquisition and monitoring control systems (SCADA), distributed control systems (DCS), and intelligent video surveillance systems (IVSS). Operator status data are collected through technologies such as wearables and machine vision. By employing wearable devices, the system can capture various internal physiological indicators of the operator, such as heart rate and brain waves. Additionally, machine vision technology records the movements and facial features of operators, identifies hazardous movements, and timely alerts the operators about their health status. Mission data are collected through the user logistics management system (LMS), encompassing metrics such as daily storage, vehicle departure and arrival times, and proportion of express cargo types. As depicted in Figure 7, the acquisition platform first analyzes the different communication protocols of each collection unit, then realizes data partitioning, selection, storage, and indexing based on the distributed management engine, and finally and finally forms a unified data standard format for data transmission, achieving multi-source heterogeneous data sources and data interaction with the system.

Figure 7

Collection of heterogeneous data from multiple sources

4.2 Construction of Relationship Between Operator Fatigue and Operation Efficiency Based on Physiological Measurements

The real-time data generated by individuals and equipment in the distribution center during the sorting process serve as the cornerstone for process optimization and control. To meet diverse data collection needs, a variety of sensing devices are deployed to enable interconnection and mutual sensing between operators and machines. This ensures real-time and reliable access to multiple sources of information during the sorting process, encompassing physical system data and real-time operational data from diverse and complex sources and structures. Therefore, data acquisition and transmission constitute the key technologies for implementing digital twin sorting systems.

Operators play a crucial role in the sorting process at a distribution center, and factors such as extended working hours and working environment lead to variations in operator fatigue. On the one hand, excessive fatigue poses risks to the physical and mental health of the operators and jeopardizes work safety. On the other hand, varying degrees of fatigue can reduce the efficiency and accuracy of operators, thereby impacting the overall efficiency of the sorting system. We propose a fatigue monitoring method based on physiological indicators that calculates the heart rate index, fuses facial features, and combines them with a state efficiency model to perceive the fatigue state of the operator and increase efficiency in real time, as depicted in Figure 8.

• 1) Percentage of eyelid closure over the pupil time (PERCLOS) is a metric that measures the extent to which the eyelid of the human eye droops over a specific time interval. It is considered the most effective indicator of fatigue monitoring based on visual characteristics [34], 2010. This metric posits that the percentage of time the eyes are closed over a sustained period best represents human fatigue. In a non-fatigued state, the PERCLOS value is low, while in a fatigued state, the number of frames registering eye closure significantly increases, leading to a corresponding rise in the PERCLOS value. PERCLOS is calculated as follows:

  $$P[k] = \frac{{\sum\nolimits_{z = k - c + 1}^{k} {Blink[z]} }}{c} \times 100\% ,$$

  (2)

  where $c$ is the total number of video frames per unit time, $Blink[z]$ is the number of frames registering closed eyes, and $\sum\nolimits_{z = k - c + 1}^{k} {Blink[z]}$ is the total number of frames per unit time with closed eyes.

• 2) As fatigue sets in, the heart rate of the operator decreases. Traditionally, characterizing the electrocardiograph (ECG) signal involves calculating the mean value of the R-R waveform in time-domain analysis, which does not reflect the real-time state. However, frequency-domain calculation based on the Fourier transform significantly enhances the speed of analysis [35]. The heart rate, i.e., the frequency of the ECG signal per minute, can be expressed as follows:

  $$f = X \times \frac{{f_{s} }}{F} \times 60,$$

  (3)

  where $X$ is the harmonic amplitude, $F$ is the number of Fourier sampling points, and $f_{s}$ is the Fourier sampling frequency.

Figure 8

Operator condition monitoring based on physiological indicators

4.3 Virtual Model Construction

The virtual layer model is fundamental for the virtual layer, necessitating a unified logical structure for its establishment. Considering diverse physical entities and functional multiplicity, along with the data produced by the entity, it is essential to construct a twin model in the digital domain. The sorting process incorporates crucial elements such as operators, cross-belt sorters, parcel, and venues, as depicted in Figure 9. Thus, the virtual layer element model of the sorting process is expressed as follows:

$$DT_{m} = DT_{pers} \bigcup {DT_{cbs} \bigcup {DT_{par} } } \bigcup {DT_{sit} } ,$$

(4)

where $DT_{m}$ denotes the digital model of the virtual layer, $DT_{pers}$ denotes the digital model of the operator, $DT_{cbs}$ denotes the digital model of the cross-belt sorter, $DT_{par}$ denotes the digital model of the parcel, and $DT_{sit}$ denotes the digital model of the site.

Figure 9

Virtual model composition

4.3.1 Digital Model Construction of Operators

The operator model in the sorting process mainly consists of two parts: task category and physiological state. Operators belonging to different task categories are constructed using a three-dimensional model, and the task-type and fatigue-state data of operators are acquired through the location/state data interface. The digital model of the operator is defined as follows:

$$DT_{pers} = \left\{ {T{\text{as}}kM,SInterface,SService} \right\},$$

(5)

where $T{\text{as}}kM$ represents the operator model for different task classes,$SInterface$ represents the location/status data interface, and $SService$ represents the status evolution service.

4.3.2 Digital Model Construction of Operators

The core module of the cross-belt sorter comprises three main components. (1) Feeding desk module: It is responsible for sending the express parcels onto the main line. Its operation principle relies on the coordinated control of the belt machine, sensor, and light curtain to accurately position the parcels. (2) Conveyor belt module: It transports the express parcels to their designated destinations through the collaboration of linear motors, electric drums, and sensors. (3) Chute module: It gathers bounds for the same destination.

To establish a genuine mapping between the virtual layer and physical world, the following steps are proposed: (1) Construction of a high-fidelity model: A model with consistent geometry, behavior, and rules is built based upon the actual entity. (2) Establishment of a virtual-real communication interface: This ensures real-time data communication between the virtual and physical layers. (3) Definition of related services: Various services are defined to enable the functionalities of the system. The model of the cross-belt sorter is defined as follows:

$$DT_{cbs} = \left\{ {FuntionM,VRInterface,VService} \right\},$$

(6)

where $FuntionM$ represents the functional model containing three-dimensional dimensions; $VRInterface$ represents the communication signal interface of the core module, and $VService$ represents the support service including signal processing and function realization.

4.3.3 Digital Model Construction of Parcel

The digital model of the parcel primarily includes information about its geometric size, material, and destination. These data can be stored in the virtual label of the digital space parcel using the data interface. Within the virtual layer, the parcel model plays a key role in mapping the flow of the parcel lifecycle. The parcel model is defined as follows:

$$DT_{par} = \left\{ {FuntionM,IIInterface,CService} \right\},$$

(7)

where $FuntionM$ is the geometric model of the parcel; $IIInterface$ is the interface of the docking information data, and $CService$ indicates corresponding services to support full lifecycle tracking.

4.3.4 Digital Model Construction of Site

The digital model of the site is reflected in terms of geometric size, spatial capacity information, etc., which is mainly used to reflect the congestion state of the cache area in the site. The model of the site is defined as follows:

$$DT_{sit} = \left\{ {FuntionM,TInterface,YService} \right\},$$

(8)

where $FuntionM$ is the geometric model of the site; $TInterface$ is the data interface of the docking camera, and $YService$ denotes the corresponding services to support the monitoring of the storage yard.

4.4 Online Optimization Module Based on Real-Time Simulation

Unlike in traditional off-line optimization, the optimization module of the digital twin system adopts the simulation principles of discrete event systems. It facilitates simulation execution through event scheduling methods and conducts state cyclic scanning before each simulation execution to ensure real-time simulation. After simulation initialization, operator status, sorter status, and parcel task information are scanned from the key element model and stored into the future event table as initial events. The simulation operation is driven by event scheduling methods to collect statistical data. Subsequently, optimization objectives are determined by comparing the statistical data, and corresponding strategies are selected from the policy pool. The future event table is updated accordingly. The system undergoes continuous updates, and the simulation program is executed in a loop to achieve continuous simulation at a certain frequency, as depicted in Figure 10. The pooling-sorting objective optimization strategy is established as follows.

Figure 10

Online optimization module based on real-time simulation

4.4.1 Transmission-Line Speed Regulation

The sorting efficiency is optimized based on speed regulation of the conveyor belt as follows:

$$V_{r} = V_{c} - \frac{{\left( {E_{c} - E_{r} } \right) \cdot L_{p} }}{\delta },$$

(9)

where V_r denotes optimized sorter mainline operating speed, V_c denotes real-time sorter mainline operating speed, E_r denotes sorting target efficiency per unit time, E_c denotes real-time sorter sorting efficiency per unit time, L_p denotes trolley intercept, and δ is the actual efficiency conversion factor.

4.4.2 Multi-destination Shared Exit

To address the situation in which the egress cannot cover all addresses, multi-address shared egress is carried out according to the principle of small quantity merging.

4.4.3 Egress and Address Adjustment Binding

An adjustment strategy is proposed based on the fatigue state of the parcel personnel, utilizing real-time data and historical efficiency to calculate the current operation efficiency of the operator under the prevailing fatigue state. Given that there may be significant variations in the number of pieces handled per unit time for each address, the grid ports and address information must be re-assigned according to the principle of minimizing the difference between the number of pieces handled per unit time by the operator and the number of pieces handled per unit time for each address. This optimization problem can be effectively solved using a 0-1 integer programming model.

Let us define $n$ as the parcel address in $D_{i} \left( {i = 1,2, \cdots ,n} \right)$. The unit-time total number is $N$; if $i$ addresses $D_{i}$, the proportion is $f_{i}$, the unit-time address is $D_{i}$, the parcel amount is $a_{i} = Nf_{i} \left( {i = 1,2, \cdots ,n} \right)$, and No. $j$ finished in unit time is $b{}_{j}\left( {j = 1,2, \cdots ,n} \right)$. Let us also define the decision variable $X_{ij}$ as an indicator for assigning the $i\text{th}$ address to the $j\text{th}$ operator:

$$X_{ij} = \left\{ \begin{array}{ll} 1,\;{\text{assign}}\;{\text{the}}\;i\text{th} \;{\text{address}}\;{\text{to}}\;{\text{the}}\;j\text{th} \;{\text{operator}}, \\ 0,\;{\text{don}}{'}{\text{t}}\;{\text{assign}}\;{\text{the}}\;i\text{th} \;{\text{address}}\;{\text{to}}\;{\text{the}}\;j\text{th} \;{\text{operator}}. \\ \end{array} \right.$$

The egress and address adjustment binding model can be described as follows:

$$min\sum\limits_{j = 1}^{m} {\left( {\sum\limits_{i = 1}^{n} {x_{ij} a_{i} - } b_{j} } \right)^{2} } ,$$

(10)

_s.t.,

$$\sum\limits_{i = 1}^{n} {f_{i} } = 1,$$

(11)

$$a_{i} = Nf_{i} \left( {i = 1,2, \cdots ,n} \right),$$

(12)

$$\sum\limits_{i = 1}^{n} x \le c_{j} \left( {j = 1, \, 2 \, , \cdots ,m} \right),$$

(13)

$$\sum\limits_{j = 1}^{m} {x_{ij} } = 1.$$

(14)

Eq. (10) yields the objective function of egress and address adjustment binding; it represents the sum of squares of the differences between the number of parts assigned to the operator per unit time and the number of parts that each operator can complete per unit time. Eq. (11) indicates that the sum of the proportions of all parts is equal to 1. Eq. (12) yields the number of parcels per unit time per address. Eq. (13) restricts the upper limit of the number of addresses that can be assigned by each operator. Eq. (14) indicates that each address can only be assigned to one operator.

4.4.4 Drop-Frame Beat Adjustment

Falling tempo adjustment and courier flooding are operator optimization strategies based on early warning fatigue. When an operator at a certain address experiences warning fatigue, the drop strategy is adjusted according to different levels of fatigue, which includes single-loop and double-loop drop strategies. In the single-loop drop, the shipment passes through the corresponding grid a second time to complete the drop, while in the double-loop drop, the shipment passes through the corresponding grid a third time to complete the drop. These strategies reduce the workload intensity of the operator by adjusting the drop tempo, thereby reducing operator fatigue.

Untimely sorted pieces resulting from the fall rate are temporarily stored on the main line of the sorter, and any excess pieces beyond the threshold are released through the abnormal piece gate to prevent blockages on the main line.

4.4.5 Shift Change of Personnel

When the system detects that the real-time data allotted to an operator has reached alarm fatigue, the operator is prompted by both a buzzer and a display to stop any operation immediately.

5 Application Validation

The Beijing distribution center belongs to an express delivery company that achieved a total throughput of 115 million pieces in 2021, with approximately 92 million of them being small parcels. The express business points under the jurisdiction of this distribution center receive express parcels in batches and deliver them to 192 distribution centers distributed across China on a daily basis. At the distribution center, a double-layer double-zone cross-belt sorting system is utilized for sorting and processing small parcels.

To implement the digital twin cross-belt sorting system, a digital twin model of the cross-belt sorting system of the distribution center was established. Plant simulation (Siemens Tecnomatix Plant Simulation software) was employed for digital modeling, and a SCADA system provided dynamic mapping, allowing real-time display of the system’s operational state. This facilitated online optimization of the system, driven by real-time data.

5.1 Data Collection

The sorting plan of the express delivery is collected from the logistics management system of the enterprise, which stores both the number of express arrivals and the proportion of express sent to the 192 distribution centers at every moment. The data in the cross-belt sorting system are collected using the SCADA system; these data include the number of feeding desks open at each time, the running speed of the conveyor belt, and grid information. Information about the change in the operator fatigue state is obtained through the hypothesis.

5.2 Construction of Virtual Scene

The digital model of the system was established based on 24 part feeding desks, 492 conveyors, 200 exit chutes, 48 operators, and other entities present in the field, ensuring consistency between the model and actual entities in terms of geometric size and motion characteristics. Based on the running logic, communication control signal interfaces were established inside the model to facilitate data interaction between the models, as shown in Figure 11.

Figure 11

Construction of virtual scene

5.3 Real-Time Mapping Construction

The SCADA system was employed to integrate and process the data from the parcel sorting system and operators and provide visual display. The simulation program accessed the data through the socket interface at regular intervals, synchronized with the clock, thereby enabling a highly realistic simulation of the data-driven model in real time, as shown in Figure 12.

Figure 12

Real-time mapping construction

5.4 Simulation Optimization

5.4.1 Drop-Frame Beat Adjustment

The total time of the simulation program was set to be 10 h. For each simulation, the current status of the sorting system and operators was scanned. The following assumptions were made:

• 1) Each physical cell port can be shared by a maximum of two destination addresses.

• 2) Each packing operator must be assigned to at least one exit chute.

• 3) The workload of the operator is directly proportional to the fatigue, with fatigue levels increasing one level per hour at full capacity and decreasing one level per hour when operating below 50% capacity.

• 4) The historical data of the distribution center were used as the control group for analyzing the optimization results.

5.4.2 Simulation Operation Strategy

• 1) Transmission Line Speed Regulation

  The empty load rate of the trolley is a crucial indicator for assessing the performance of the sorting system. When the empty load rate approaches zero, the sorting system cannot improve its efficiency. By contrast, when the empty load rate exceeds 30%, the sorting capacity is wasted. The conveyor digital model incorporates multiple running speed stalls. Based on the task objective, the appropriate speed gear that meets the efficiency standard is selected, and the empty load rate of the trolley is calculated during the operation of each speed gear.

• 2) Multi-destination Shared Exit Chute

  The total sorting time encompasses the entire duration needed to sort a batch of express parcels, including the initial sorting time using the cross-belt sorter and secondary sorting time at shelves requiring manual sorting. The digital model of the outlet chute and the manual shelf is employed to simulate these two sorting procedures. When the number of destinations at a certain moment surpasses the number of chutes, chute sharing is initiated. Using combination optimization, the count of secondary sorting operators and the cumulative sorting time are outputted.

• 3) Unloading Beat Adjustment

  The total fatigue refers to the fatigue level of all personnel involved in the operation in the current shift. The digital model of the operator and the transmission line were used to simulate the process, capacity value Q of the transmission line, and fatigue parameter R. When R > 0.5, the express shipment of the operator was discharged in a single cycle; when R > 1, the cargo was discharged in a double cycle. The express delivery of the transfer line was unloaded to the flood chute for manual sorting. The total fatigue per shift was determined.

• 4) Egress and Address Adjustment Binding

  The parcel operator and outlet chute models were utilized to validate the effectiveness of the strategy. Through combination optimization, the current operating efficiency of each operator was analyzed for a maximum of 10 volume differences of pieces in destination addresses. The total difference of the optimal scheme was then outputted as the result.

5.4.3 Simulation Results

As shown in Table 1, the optimized initial operation scheme has evident advantages over the pre-optimized scheme in terms of main line speed and personnel allocation and effectively saves labor and energy consumption. The empty load rate of the trolley is a crucial indicator for assessing the status of the sorting system. As shown in Figure 13(a), the fixed transmission belt speed of the control group leads to a relatively high empty load rate of the trolley at certain times. However, the optimization scheme with real-time speed regulation demonstrates a consistently lower empty load rate of the trolley. This strategy effectively minimizes efficiency wastage in the sorting process and results in energy conservation. In the control group, sequential address merging leads to a higher number of secondary sorting instances and a longer total sorting time. As shown in Figure 13(b), adopting combinatorial optimization to combine addresses results in a reduction in secondary sorting occurrences and a decrease in the total sorting time. As shown in Figure 13(c), the unloading beat adjustment based on operator fatigue changes results in multiple single and double cycle drops in the shift, and the total fatigue is lower than that of the control group. As shown in Figure 13(d), the total difference of schemes in the optimization group is significantly lower than that in the control group; therefore, this strategy effectively balances the number of operations carried out by operators in different states.

Table 1 Optimization group and control group initial parameters

Figure 13

Analysis of simulation result

6 Conclusions

The development of modern industry should implement a human-centric concept. Despite the widespread application of intelligent sorting in express distribution centers, human involvement remains essential in the sorting process. The advancements of digital twin technology constitute an opportunity to address the issue of man–machine mutual induction. This study explores the integration of humans and machines in a digital twin cross-belt sorting system and proposes an operational mechanism. The goal is to address key challenges in the system construction, including the acquisition, transmission, and storage of multi-source heterogeneous data, the development of fatigue and efficiency conversion models, the construction of a virtual model, and the implementation of online optimization. The results demonstrate that the construction method of the cross-belt sorting system based on a digital twin enhances the self-organization, self-decision-making, and self-adaptation capabilities of the distribution center, which is critical for transforming express distribution centers. A digital twin system was constructed using the cross-belt sorting system of a courier company as a prototype, and the effectiveness of online optimization was verified. The research presented in this paper is still in its early stages, and future studies should focus on the following aspects: (1) multi-granularity modeling methods for different levels and details; (2) extensive application of online optimization technology based on transient simulation in express delivery centers.