Study of Robot Manipulator Control via Remote Method

INTRODUCTION: The study introduces a novel approach to the design and management of industrial robots using virtual reality technology, enabling humans to observe a wide range of robot behaviors across various environments. OBJECTIVES: Through a simulation program, the robot's movements can be reviewed, and a program for real-world task execution can be generated. Furthermore, the research delves into the algorithm governing the interaction between the industrial robot and humans. METHODS: The robot utilized in this research project has been meticulously refurbished and enhanced from the previously old version robotic manipulator, which lacked an electrical cabinet derived. RESULTS: Following the mechanical and electrical upgrades, a virtual setup, incorporating a headset and two hand controllers, has been integrated into the robot's control system, enabling control via this device. CONCLUSION: This control algorithm leverages a shared control approach and artificial potential field methods to facilitate obstacle avoidance through repulsive and attractive forces. Ultimately, the study presents experimental results using the real robot model.


Introduction
Robotic systems have revolutionized the landscape of modern technology and industry, encompassing a diverse range of robotic platforms, including mobile robots [1], robot manipulators [2], and reconfigurable robots [3].These machines are at the forefront of automation, enhancing efficiency, precision, and adaptability across various domains.In this era of rapid technological advancement, these robotic systems are not only characterized by their physical presence but also by the advanced techniques and control mechanisms that govern their operation.
One crucial aspect of modern robotics [4] is the utilization of virtual models which allows engineers and researchers to design and simulate robotic systems in a digital environment before physical implementation.This approach offers the advantage of cost-effective prototyping, enabling thorough testing and refinement of robotic functionalities without the need for extensive physical resources.It is instrumental in optimizing the performance and capabilities of robotic systems [5][6][7][8].
Motion control is another fundamental technique in the realm of robotics [9][10][11].Precise control of a robot's movements, whether for navigation [12], manipulation [13], or reconfiguration [14], is essential for achieving desired outcomes.Advanced motion control algorithms and systems enable robots to execute tasks with unparalleled accuracy, speed, and dexterity, making them indispensable in industries such as manufacturing, healthcare, and logistics.
Adaptive control techniques are pivotal in ensuring that robotic systems can seamlessly adapt to dynamic and changing environments [15,16].These methods enable robots to continually learn and adjust their behaviour based on real-time data and sensory input.As a result, robotic T. Nguyen 2 systems become more versatile and capable of handling unforeseen challenges, further extending their applications in unpredictable settings.
In recent years, the world of robotics has witnessed a remarkable transformation, thanks to the seamless integration of Artificial Intelligence (AI) into robotic systems [17][18][19][20].This synergy between AI and robotics has unlocked a plethora of possibilities, revolutionizing how robots' function, interact with their environments, and serve various industries.From manufacturing [21] and healthcare [22] to autonomous vehicles and space exploration, the applications of AI in robotics have not only expanded the capabilities of machines but have also reshaped the way we perceive and utilize them.In this exploration of the diverse applications of AI in robotics [23][24][25], we will delve into the innovative ways in which AI is enhancing the intelligence, adaptability, and efficiency of robots, ultimately shaping the future of automation and human-machine collaboration.
In the age of Industry 4.0, cyber-physical control plays a pivotal role in connecting robotic systems with the broader digital infrastructure [26][27][28].These systems allow for real-time monitoring [29], remote operation [30], and the exchange of data between robots and other interconnected devices [31].This integration facilitates collaborative and intelligent decision-making, ultimately enhancing the efficiency and adaptability of robotic systems across industries.

Prefaces
This section introduces several peripheral devices and communication protocol to connect.Additionally, the working environment is launched to embed our model.

Introduction to Virtual Reality
Virtual Reality (VR) is a groundbreaking technological marvel that has transcended the boundaries of traditional human-computer interaction, ushering us into a realm where the lines between the physical and digital worlds blur into a seamless, immersive experience [32][33][34].This transformative technique has emerged as a revolutionary force in the realms of entertainment, education, healthcare, architecture, and countless other industries.
At its core, VR is a computer-generated simulation that transports users into a three-dimensional, artificial environment, often facilitated by a head-mounted display and specialized input devices [35].The primary objective of VR is to create a sensory-rich environment that engulfs users, allowing them to interact with and explore these synthetic worlds as if they were real.
The magic of VR lies in its ability to engender a profound sense of presence.It tricks our senses, fooling our brains into believing that we are truly present in a virtual environment [36].Visual, auditory, and sometimes even tactile stimuli converge to create an immersive experience that is unlike any other.As a result, VR has become a powerful tool for simulating situations, training individuals, and fostering experiences that were previously unattainable.
In Fig. 1, VR glasses, also known as VR headsets or VR goggles, come in various forms and designs, but they generally share a common structure and components that enable immersive VR experiences.The typical structure of VR glasses consists of • Head-Mounted Display (HMD): The core component of VR glasses is the head-mounted display, which is worn on the user's head.This display is responsible for showing the virtual environment to the user.It typically consists of two screens, one for each eye, to create a stereoscopic 3D effect • Optics: VR glasses have a series of lenses or optical elements that sit between the screens and the user's eyes.These lenses are crucial for adjusting the focus and magnification of the displayed images, ensuring that the virtual world appears clear and immersive • Straps and Adjustments: To keep the VR headset securely in place, it is equipped with adjustable straps that go around the user's head.These straps are often padded for comfort and can be tightened or loosened to fit different head sizes.There may also be additional adjustments to fine-tune the position and angle of the display for optimal viewing • Sensors and Tracking Technology: VR glasses incorporate various sensors, such as accelerometers, gyroscopes, and magnetometers, to track the user's head movements.This tracking data is essential for updating the view in real-time as the user looks around the virtual environment, enhancing the sense of immersion.Some advanced VR systems also use external sensors or cameras for room-scale tracking, allowing users to move around in physical space.

Working platform
Unity is a powerful and versatile game development platform that has emerged as a driving force behind some of the most captivating and interactive digital experiences of our time.It stands at the forefront of the technology industry, enabling developers to create immersive 2D, 3D, augmented reality (AR), and VR applications across multiple platforms.Unity's impact reaches far beyond gaming, extending into fields such as education, architecture, healthcare, and entertainment, where its capabilities have been harnessed to craft innovative solutions and captivating content.Besides, unity is a cross-platform game engine and development environment, renowned for its ease of use and accessibility.It empowers creators, from indie developers to large studios, to bring their ideas to life, blurring the lines between the real and digital worlds.Unity's robust ecosystem provides a comprehensive suite of tools, including a powerful visual editor, scripting capabilities, and a vast library of assets and plugins, all designed to streamline the development process

Camera setup
Setting up a RealSense camera involves a few steps to ensure that it is connected correctly and that users have the necessary software drivers and applications installed.
• Connect the Camera: Plug the RealSense camera into an available USB port on your computer.Ensure that it is securely connected • Check Camera Compatibility: Ensure that the RealSense camera is compatible with the version of the SDK (Software Development Kit) installed in computer Both 2D and depth images captured by Realsense cameras have a resolution of 640x480 pixels, with each pixel in 2D photos corresponding to a depth camera point.When obtaining the pixel coordinates of a point in an RGB image, it is possible to recover the x, y, and z values of that point in camera coordinates.This is because the true coordinate system in Realsense is defined to align with the colour space in the camera coordinate system.

Robot model
Unity3D does support the creation of 3D models within the project.However, designing complex-looking models may be challenging using Unity3D's fewer professional tools, as Unity3D is primarily tailored for game development.
Therefore, it is most common to design a mechanical model in other professional software such as AutoCAD, SolidWorks, Fusion, and similar tools.However, it's not possible to directly import 3D files into Unity3D because it only supports specific formats.Thus, this thesis focuses on converting the file formats into ones compatible with Unity.This conversion process involves using a middleware to alter the format without modifying the original data of the files.In this case, Blender is employed to address this issue as Fig. 2.

Programming procedure
Working within the virtual environment involves a working table (initially 500mm in length and 300mm in width before scaling) and a leader cube.The user guides the cube to desired positions, and the robot arm follows the cube's path.In this scenario, it's necessary to transform the coordinates of the cube into the robot's coordinate system, as the robot is controlled using the inverse kinematic method.As shown in Fig. 3, it is important to note that Unity employs a left-handed, Y-Up coordinate system Because of the different robot model, the DH table is based on the five axes robot model as Fig. 4, even though the general placement of coordinates is the same as the common robotic platform.The components are organized in a hierarchical order so that the rotation angles of each link correspond to the rotational movement of each part.Due to the automatic coordinate placements, each part must be aligned with a virtual coordinate system that coincides with the rotation axis.This ensures that the entire part rotates as required

Camera calibration
It is necessary to acquire obstacle data from OpenCV in addition to the Kinect library.OpenCV (Open-Source Computer Vision Library) is a collection of programming functions primarily designed for real-time computer vision applications.The primary programming language used in this thesis is C#.To transmit the data to Unity, socket communication based on the local host is employed.

Data exchange
The computer-based sampling time refers to the time required to generate new control data for the robot after collecting data from VR and processing it according to the algorithm, without considering the camera factor.This time can be estimated to be less than 3ms.Therefore, the condition for selecting the sampling time is as follows:   >   , where the computer program's time,   , is set to 500ms.However, once the real robot's pose is set, it becomes challenging to predict the precise time it will take for the robot to transition to its next set of movements.This uncertainty arises because the drivers must generate new control signals for the motors only after the previous movements have been completed.In some cases, this process might go smoothly, but there is a possibility of encountering errors that render the drivers unresponsive.Given these uncertainties, the interval for receiving new signals from the drivers is determined in a somewhat randomized manner, typically set at 3 seconds.This selection is made with a specific condition in mind: that the displacement of the end-effector remains relatively short during each sampling period.

The proposed control scheme
The artificial potential field comprises two primary components: repulsion potential energy and attractive potential energy.As a general principle of this algorithm, the robot is steered in a direction that reduces the total potential energy, thereby moving toward the target while concurrently navigating away from obstacles.
Repulsion potential energy involves the exertion of a repulsive force on the robot arm by an obstacle at a defined distance.This implies that the force's direction will likely be opposite to the vector pointing from the robot arm to the obstacle, as depicted in Fig. 5. Consequently, this mechanism enables the robot to effectively evade obstacles.In this thesis, such forces are applied individually to each joint to ensure obstacle avoidance for the entire manipulator.The calculation for the repulsion potential energy of each joint is as follows (1) i: the order of each joint.i=3 for this investigation Kr: repulsion gain di: distance between joint to obstacles.dsafe: safe distance which allow robot work in normal mode

Experiment of Study
In this section, real-world validations of the proposed method were conducted to evaluate its feasibility and effectiveness.The system comprises a robotic platform, an electric cabinet, a motion controller, virtual glass, and a teach pendant.The overall setup is depicted in Fig. 6a, while an operator primarily interacts with the proposed system.During robot manipulation, data is collected from various sensors, as shown in Fig. 6b In our tests, this system demonstrated effective performance in various real-world scenarios.It exhibited flexibility in responding to human movements and vibrations.Given the continuous and intricate nature of human motion, it is imperative to implement data filtering to attain precise estimations.Moreover, this dataset can be leveraged for predicting human actions or supporting human-robot interactions when necessary.Comparing to previous works [37,38], our study gained many benefits owing to the powerful performance of mathematical model, vivid illustration, and easy interface.It is expected that this approach could be implemented some distinguished schemes and computations.

Conclusion
This study introduces a novel approach for controlling a robotic platform within a virtual environment.The objectives of this proposed system encompass three key aspects: (1) establishing a robust hardware connection to facilitate seamless and continuous data transmission, (2) realizing the concept of a cyber-physical system within the realm of robot manipulation, and (3) enhancing humanrobot interaction.Within this design framework, several components must be described and installed to meet specific requirements.The proposed method presents a mechanism for data flow exchange and an overall scheme for manipulation.To illustrate the effectiveness and feasibility of this approach, a real-world example of driving the robotic system was implemented.In the future, deploying modern hardware platforms will be essential to achieve high-performance results.
Additionally, considering enhanced schemes such as machine learning or human-in-the-loop interactions could further improve the system's behaviour.

Figure 2 .
Figure 2. Description of the proposed design.

Figure 3 .
Figure 3. Description of inter-connection for data exchange.

Figure 4 .
Figure 4. Description of the proposed design for robot model.

Figure 5 .
Figure 5. Illustration of the repulsion forces in the proposed model.

Figure 6 .
Figure 6.Result of the proposed approach, (a) practical implementation and (b) rotational angles of robot platform.