Model Protection Scheme Against Distillation Attack in Internet of Vehicles

Aiming at the problems of model security and user data disclosure caused by the deep learning model in the Internet of Vehicles scenario, which can be stolen by malicious roadside units or base stations and other attackers through knowledge distillation and other techniques, this paper proposes a scheme to strengthen prevent against distillation. The scheme exploits the idea of model reinforcement such as model self-learning and attention mechanism to maximize the difference between the pre-trained model and the normal model without sacrificing performance. It also combines local differential privacy technology to reduce the effectiveness of model inversion attacks. Our experimental results on several datasets show that this method is effective for both standard and data-free knowledge distillation, and provides better model protection than passive defense.


Introduction
The Internet of Vehicles (IoV) is a technology that uses artificial intelligence and 5G communications to achieve intelligent traffic management and vehicle control through multidimensional interactions between vehicles and other vehicles, vehicles and people, and vehicles and the road environment. The goal of IoV is to provide a safe, comfortable and efficient driving experience and transport services. In the IoV system, vehicles are equipped with devices that have data collection, processing and storage capabilities. These devices generate a large amount of network data, such as vehicle speed, orientation, road information and traffic conditions. This data supports the development of various technologies and applications, including traffic flow prediction, vehicle trajectory prediction, pedestrian collision detection, high-precision in-vehicle navigation, and in-vehicle entertainment. Deep learning provides a new solution for efficiently fusing and processing this data and information [1].
In the IoV scenario, it takes a lot of effort and resources for companies to train advanced deep learning models for vehicles. These trained models and proprietary training data have high intellectual property rights, making it legally and ethically prohibited to share them publicly. However, during the process of information exchange between vehicles and external nodes, attackers can use knowledge distillation [2] techniques to mimic the input and output behavior of the black box, and thereby steal vehicle deep learning models. In addition, data-free knowledge distillation combined with adversarial network attack generation, membership inference, model inversion, and other reverse engineering methods can enable the recovery of private training data from black box models [3][4][5][6], seriously undermining the privacy of IoV users.
Scholars around the world have conducted a lot of research on this topic and achieved a number of results. For example, Reference [7] combined federated learning and local differential privacy to propose the LDP-FedSGD algorithm to coordinate cloud servers and vehicles to collaboratively train models, which significantly reduces the risk of data leakage while considering practicality.
Weiping Peng et al.

2
Reference [8] proposed a hybrid blockchain architecture consisting of a permissioned blockchain and a local directed acyclic graph to reduce the transmission load and address the privacy concerns of providers. The reliability of the shared data can also be ensured by integrating the learned model into the blockchain and performing a twostep verification. Liu et al. [9] proposed a hybrid proxy authentication scheme by introducing the concept of proxy vehicles and integrating hybrid authentication based on identity and PKI, which ensures the data security of IoV users while improving the effectiveness of roadside units in terms of authentication messages. The above solutions have provided some security protection for different application in IoV; however, all of them only consider the traditional data leakage problem and ignore the possibility of theft of vehicle deep learning models. Recent work relies on watermark-based [10] or passport-based [11] authentication methods to protect models. However, they can only detect model attribution and are ineffective in avoiding model cloning. The above defense methods are all reactive and have not explored knowledge distillationbased model attacks in the IoV environment, which is a problem worth investigating.
To address the aforementioned limitations, this paper proposes a defense scheme, called Strengthen Prevent Distillation (SPD), for the three-layer architecture of the cloud-side-end of the IoV [12]. The scheme constructs a deep learning model of the vehicle as a specially trained network that performs similarly to the corresponding normal model, but renders inversion of the model by an attacker through methods such as knowledge distillation ineffective. Our main contributions are summarized as follows: • First, we have summarized the defense methods for model and data theft in the context of the IoV and analyzed their pros and cons. • Next, we creatively embedded the ideas of attention mechanism and local differential privacy into the method for defending against knowledge distillation attacks. Simulation experiments have verified the effectiveness and rationality of the SPD algorithm. • Finally, we conducted extensive comparative experiments to verify the superiority of our method over other traditional methods, demonstrated its terrific performance in the absence of data distillation, and identified the effectiveness of our method through qualitative analysis.
The remainder of this paper is organized as follows. In section 2, we introduced the basic principles of knowledge distillation and the foundations of model reinforcement and differential privacy, and reviewed some of the previous contributions. In Section 3, we presented the overall architecture of the IoV and provided a detailed discussion of our proposed method. The simulation results were presented in Section 4 to demonstrate the effectiveness of our proposed mechanism. Finally, Section 5 summarised the work done in this paper and outlined future research directions.

Image Classification
Image classification has extensive applications in various fields, such as computer vision, natural language processing, intelligent transportation, and medical image analysis. By selecting appropriate feature extraction methods and classification algorithms, high-precision image classification tasks can be achieved. With the rising number of vehicles on urban roads, Intelligent Transportation Systems (ITS) play a vital role in enhancing traffic flow and efficiency while minimizing accidents. The vast amount of data generated by various digital devices connected to the transportation network facilitates the creation of datasets, which can be analyzed using advanced deep learning techniques. This approach helps in predicting traffic performance, automating traffic signal management, detecting lanes, and recognizing objects in close proximity to vehicles, thereby improving the safety and efficacy of ITS [13]. Wang et al. introduced Particle Swarm Optimization to construct a PSO-guided Self-Tuning Convolution Neural Network (PSTCNN), enabling the model to automatically adjust hyperparameters and allowing deep learning models to more quickly and accurately diagnose COVID-19, effectively alleviating the problem of global healthcare resource scarcity [14]. The effectiveness of artificial intelligence technology in diagnosing COVID-19 and the superiority of Adaptive Jaya algorithm over Jaya algorithm in medical image classification tasks were demonstrated in Reference [15].

Knowledge Distillation
Knowledge distillation is a widely used method for model compression and optimisation in deep learning. It is based on the concept of a "teacher-student model" for training and is highly regarded for its simplicity and effectiveness. Knowledge Distillation facilitates the training of student models by extracting "knowledge" from one or more pretrained teacher models using the soft-label probabilistic output of the teacher models. This soft-label output is a mapping from input vectors to output vectors that captures specific knowledge from instantiated objects, with incorrect classification predictions providing insight into how the teacher model generalizes. The student model can improve its performance by mimicking the probabilistic output of the teacher model, and can incorporate the knowledge that the teacher model has already acquired. The process of knowledge distillation is illustrated in  Teacher networks can transfer their model capabilities to student networks through knowledge distillation.
As shown in Eq. (1), neural networks typically generate class probabilities by using a "softmax" output layer that compares the output i z of each class with other logits, converting the logit z i calculated for each class into a probability i q in a standardized way. In addition, where T represents temperature in knowledge distillation, using a larger value than 1 for it produces a softer class probability distribution that allows better transfer of knowledge to the model to be distilled.   represent the K-L divergence and cross-entropy loss functions, respectively. The introduced "softmax temperature" function () S   produces a soft probability output when a larger temperature S  is selected, decays to a normal softmax function ()  equal to 1, and another hyperparameter  to balance the cost minimization of knowledge distillation.

Prevent Knowledge Distillation
Combining knowledge distillation with methods such as generative adversarial networks can lead to the theft of deep learning models and user privacy. Ma et al. [16] proposed a special deep learning model with slightly worse performance compared to its normal counterpart, both with the ability of classification and regression in deep learning, from which no malicious third-party network can extract useful parameters using knowledge distillation. The algorithm is implemented by maintaining its correct category assignment and disrupting its incorrect category assignment as much as possible to prevent attackers from stealing model information and raw data through distillation. The process of constructing the prevent distillation model is shown in Eq. (3): The first part of Eq. (3) aims to maintain the accuracy of the model by minimizing the cross-entropy loss, while the second part maximizes the K-L divergence between the pre-trained model and a regular network to hide the "useful knowledge" and achieve "prevent-distillation". In this equation, A  represents the temperature of self-sabotage, and  balances the weight of the loss function accounted for by both normal training and adversarial learning.

Model Enhancement
Zhang et al. [17] proposed the concept of self-distillation by closing the gap between the deep and shallow modules of the model without the help of an external model, which improves the overall accuracy of the model. Chen et al. [18] proposed a knowledge review approach to improving the performance of the student model by packaging the knowledge of the shallow modules of the teacher model and imparting it to the student model. Hou et al. [19] proposed a self-distillation-based lane line detection algorithm that utilizes the concept of an intermediate layer attention map, where each layer receives attention-guided training from the last layer to improve the performance of the lane line detection model by passing features from the deeper layers of the model to the shallower layers in advance for learning. Vaswani et al. [20] creatively proposed a simple network architecture based on an attention mechanism that reduces the training time while optimizing the model. Hu et al. [21] automatically obtain the importance of each channel by explicitly modelling the interdependencies between the feature channels, then boost the useful features and suppress the features that are less useful for the current task according to their importance, EAI Endorsed Transactions on e-Learning 10 2022 -11 2022 | Volume 8 | Issue 3 | e4 and finally, improve the overall performance of the network. These methods are validated on public datasets and provide good ideas for the self-optimization of the model.

Differential Privacy
Differential privacy is a cryptographic technique that aims to maximize the accuracy of data queries and reduce the chance of identifying records from statistical database queries and is widely used in deep learning models to protect data privacy [22]. Local differential privacy (LDP) is one of these models, which does not have any trusted third party and needs to add perturbations to its data before sharing it with other data parties. Arachchige et al. [23] proposed the LATENT algorithm, which redesigned the training process and added a randomization layer at this stage before the data leaves the device and reaches the server, significantly improving the utility of differential privacy in the deep learning process. Using the concept of LDP, Wei et al. [24] proposed a user-level differential privacy algorithm that adds artificial noise to the shared model before uploading it to the server and derived a theoretical convergence upper bound for the framework.
The mechanism of differential privacy with parameters ( , ) Where  is the privacy budget, which indicates the distinguishable boundary between two adjacent datasets, it takes a value greater than 0, and a smaller value indicates a higher level of data protection. ( (0,1])  is the privacy leakage probability.
Knowledge distillation-based attack and defense in the IoV environment covers many aspects. This section reviews the ways of knowledge distillation, the basic practices of prevent distillation, and briefly introduces the classical practices of model reinforcement and how to add local differential privacy noise to deep learning models. These contents laid the foundation for the subsequent methods proposed.

System Architecture of IoV
The IoV system architecture is shown in Fig. 2 and consists of 3 parts: vehicle, roadside unit(RSU), and base station(BS). It is assumed that a single base station can cover all the areas shown in Fig. 2 and provide remote communication services to initialize the whole IoV application system and generate system-related parameters. Three roadside units are deployed near each road section, connecting upwards to the base station and downward to the vehicles on the road via wired or wireless channel communication links to provide authentication and real-time data services to the vehicles. In terms of computing and communication capabilities, the base station is more powerful and the roadside unit is weaker [25]. The vehicle is equipped with an intelligent vehicle system that communicates with roadside units and base stations in real-time and can select the appropriate roadside unit for authentication and information interaction according to its area and handle complex and changing road information, to ensure that the vehicle can be safely exercised on the road. If a vehicle is not within the coverage area of any roadside unit, it interacts directly with the base station for information.   The system model proposed in this paper consists of a master chain computation process consisting of the base station and the roadside units within its coverage area, a slave chain computation process consisting of the roadside units and the vehicles within its coverage area, and a local computation process of the vehicle deep learning model. In this paper, we assume that some of the nodes can be exploited by attackers to eavesdrop on the output information of the vehicle's deep learning model, and that there is no slack state in the nodes themselves and no possibility of malicious uploading of incorrect parameters. The relevant parameters are listed in Table 1. The i j wt − . After each round of training, these updated weight parameters are uploaded to nearby roadside units via a wireless network.
During the iterative process from the slave chain, the roadside unit receives the model prediction results from all the vehicles involved in the training. It can aggregate these data to minimize the loss function and improve the accuracy of the vehicle deep learning model. The weighted aggregation approach used in this paper is as shown in Eq.
In the iterative process of the master chain, similar to the learning process of the roadside unit, the base station stores the predictions of the roadside unit locally and simultaneously aggregates all the received parameters globally, where the loss function is defined as Eq. (8).

Strengthen Prevent Distillation Process
The vehicle deep learning model is denoted by S, and the information collected by the vehicle is denoted by X,  parts of the model. In this paper, we use the Attentionbased Fusion (ABF) mechanism to adjust the deep highdimensional features and the shallow low-dimensional features to the same size, and connect the features of different dimensions to generate an attention graph. This attention map is multiplied with the two previous feature maps and finally stitched into the final output. The ABF architecture is shown in Fig. 3. In order to reduce the complexity of model training, the final architecture is progressively improved in this paper. Taking the classic residual network as an example, the specific implementation is that the output of each layer needs to be combined with the output of the later layers to produce a cross-entropy loss function that uses backpropagation to strengthen the model accuracy of the previous layer by layer and to expand the error distribution of the incorrect class until the final output can achieve a prediction accuracy comparable to or better than the initial model. The specific approach is shown in Fig. 4.

Adding Adaptive Differential Privacy Process
In this paper, a Vehicle Adaptive Differential Privacy (VADP) algorithm is proposed to further prevent malicious attribute inference during the information interaction between the master and slave chains of the connected vehicle system architecture. The algorithm is incorporated into the previous model to further enhance its effectiveness in protecting private data and preventing information leakage during the upload process. This is done by

Experimental Setup
The proposed SPD scheme in this paper first performs selfdestructive training according to Eq. (2) to create a distillation-proof model in IoV and performs an strengthen prevent distillation process, and adds adaptive Gaussian noise to optimize itself. To evaluate the effectiveness of the proposed model, we use Eq. (1) to conduct knowledge distillation on a given malicious third-party model and EAI Endorsed Transactions on e-Learning 10 2022 -11 2022 | Volume 8 | Issue 3 | e4 7 evaluate the performance of the model. We draw corresponding conclusions from the comparison.
In order to verify the effectiveness of the proposed mechanism in this paper, We used the CIFAR-10, CIFAR-100, SVHN, and Tiny-imagenet datasets. The CIFAR and Tiny-imagenet dataset is used to validate the general applicability of the SPD approach, while the SVHN dataset is used to evaluate the effectiveness of the SPD approach specifically in IoV. CIFAR-10 and CIFAR-100 are often used as classical datasets to test the effectiveness of image classification models, and they both consist of 60,000 32× 32 colour images, of which 50,000 are used as training sets and 10,000 are used as test sets; the difference between them is that CIFAR-10 is used for 10 classification problems, while CIFAR-100 is used for 100 classification problems, and CIFAR-100 is much more difficult to train than the former. The SVHN dataset is extracted from Google Street View images of door numbers and is suitable for in-vehicle sensors reading image data around vehicles in IoV. SVHN contains over 600,000 digital images, including 73,257 images in the training set and 26,032 images in the test set; an additional 531,131 images are also available for training if the model requires a larger amount of data. Tiny-imagenet is derived from the classic dataset ImageNet. It consists of 200 classes, with each class having 500 training images, 50 validation images, and 50 test images, all of which are 32×32 color images. ResNet-18 and ResNet-50 are used as vehicle deep learning models, and ResNet-18, ShufflenetV2, MobilenetV2 and 5-layer normal CNN are used as attacker models as a way to fully evaluate the scheme.
All experiments were conducted on GPU devices under the pytorch 1.11.0 environment. Each network was trained for 100 epochs on two different datasets using the SGD optimizer to optimize the neural network. The initial learning rate was set to 0.1, and it decreased by a factor of 1/10 at 30, 60, and 90 epochs. Other training hyperparameters include weight_decay=5e-4 ， momentum=0.9, and a batch size of 128.
In this section, the following comparison scheme is designed for simulation and verification of the algorithm proposed in this paper.
• The prediction accuracies of the vehicle deep learning and attacker models are obtained experimentally and used as a baseline. Comparing the SPD model constructed by the method proposed in this paper with the common vehicle model, it can be seen that the present method hardly affects the prediction accuracy of the model. • By comparing the distillation of the model constructed by the method proposed in this paper with the distillation of a standard vehicle model, it can be seen that the present method significantly reduces the utility of knowledge distillation, making it meaningless to obtain a vehicle model by means of knowledge distillation.
• By comparing the accuracy in a data distillation-free environment, it is concluded that the SPD scheme can protect the data privacy of users. • The superiority of the proposed algorithm in this paper is derived by comparing it with the standard resistance distillation algorithm [16] and the adaptive false alarm algorithm [26].

Experimental Results
The experimental results on CIFAR-10, CIFAR-100, SVHN and Tiny-imagenet are shown in Table 2, Table 3, Table 4 and Table 5 respectively, where the normal model is denoted by NM (Normal) and the enhanced resistance to distillation model is denoted by SPD (Strengthen Prevent Distillation). For ease of presentation, we define the vehicle deep learning model as the teacher network and the attacker model as the student network. To further eliminate chance, 10 simulation experiments are run for each of the above algorithms and the results of each iteration are averaged as the final result. First, we observed that all SPD models performed similarly to the corresponding normal models. Second, the attacker model steals the normal vehicle model through knowledge distillation, which can improve the accuracy by up to 9.53%. However, distillation of the model proposed in this paper reduces the accuracy by 1.92% to 66.44%, indicating that distillation-prevent vehicle deep learning models can successfully provide a false sense of generalization for malicious roadside units or base station models. In addition, comparing the data in the table shows that weaker attacker networks (e.g. MobilenetV2) may be more vulnerable to errors than stronger networks (e.g. . The published vehicle deep learning models are experimentally "distillation-prevent", so knowledge distillation-based model steganography is no longer be applicable.   In order to more clearly see the effectiveness of the algorithm proposed in this paper in preventing knowledge distillation, Fig. 5 visualizes the iterative accuracy of the ordinary model, this paper's model, the attacker's model, the distillation ordinary model, and the distillation this paper's model (assuming the dataset is CIFAR-100, ResNet18 is the vehicle model, and MobilenetV2 is the attacker model). The experimental results show that the model proposed in this paper can reach convergence faster under the condition that the accuracy is not inferior to that of the normal model, and the model accuracy of the attacker will be severely reduced in the face of distillation.

Figure 5.
Several iterations of the model. As distillation attacks continue, the performance of the model against a normal model attacker will improve, but its performance will severely degrade against an SPD model. To verify that the model proposed in this paper is still valid in a data distillation-free (DAFL) environment, we used the classical ResNet18 as the underlying network and conducted experiments using the method proposed by Chen et al. [3], and obtained the results shown in Table 6. Comparing the data in the table, it can be seen that the attacker's gain will be greatly reduced compared to distillation ordinary model by DAFL's method to steal the user privacy of distillation resistant vehicle deep learning model.
To verify the superiority of the method proposed in this paper, the accuracy of the models constructed by the strengthen prevent distillation (SPD) scheme, the ordinary prevent distillation (PD) scheme, the adaptive false alarm (AFA) scheme and the normal model (NM) are compared, and the changes in model accuracy caused by the distillation of the models constructed by the above algorithms are compared (still assuming that the dataset is CIFAR-100, ResNet18 is the vehicle model , MobilenetV2 is the attacker model), it can be seen that the SPD scheme has the least impact on the accuracy of the model itself and produces the best model protection in the face of knowledge distillation. The experimental results are shown in Fig. 6.

Qualitative Analysis
The scheme is effective in the IoV environment because it maximizes and reinforces the output of the correct categories and confounds the ranking of the incorrect categories. A visualization of the output probability of the ResNet-18 model on the CIFAR-10 dataset is shown on Fig. 7 to qualitatively analyze the reasons why the scheme is effective. As shown in Fig. 7, it visualizes the logit response of the normal ResNet-18 and its counterpart after processing with the strengthen prevent distillation function, using the output of a truck and a car as examples. It can be seen that the normal model always outputs one peak, but the output response of the strengthen prevent distillation vehicle deep learning model consists of multiple peaks. Multi-peak logic misleads the learning process of knowledge distillation and degrades the performance of the attacker model, giving the attacker model a false sense of generalization, then the malicious roadside unit or base station also learns the wrong knowledge from the vehicle model, leading to a decrease in its own accuracy, which in turn protects the security of the vehicle model as well as the privacy of the user.

Ablation Experiment
As shown in Fig. 8, the proposed method is capable of reducing model performance for malicious attackers, irrespective of the selected value of parameter  , which varies from 0 to 0.01 on CIFAR-100 dataset (assuming that the dataset is CIFAR-100, ResNet18 is the vehicle model , ShufflenetV2 is the attacker model). Additionally, by adjusting the value of w, it is possible to achieve a balance between performance loss and resistance to distillation attacks. Specifically, a higher value of w can result in a more resilient model against distillation attacks, but at the expense of greater accuracy loss. Figure 8. Several iterations of the model. As the balance parameter  increases, the performance of the attacker model will severely degrade, but at the cost of the defender model's performance being negatively impacted as well.

Conclusion
In practice, the owner of the vehicle deep learning model can achieve the effect that the model cannot be stolen by resisting distillation training, self-enhancement training, and adding local differential privacy noise without sacrificing its own performance. The related performance improvements are due to the fact that resistance to distillation training has reconstructed the internal structure of the model, self-enhancement training has expanded the degree of reconstruction, and adding differential privacy noise has further improved the privacy protection efficiency of the scheme. Even if attackers have the same training data, they do not have the ability to use knowledge distillation to clone published models, as their model performance would be severely degraded instead of being boosted as usual where performance degradation is unacceptable in some security-critical environments, such as autonomous driving, so that cloned models or illegal data theft through knowledge distillation can be avoided.

Study Limitations
Extensive experiments conducted on multiple datasets quantitatively show that the vehicle deep learning model with strengthened prevent distillation is effective in either the standard knowledge distillation or data-free knowledge distillation settings. This scheme is more complex and takes longer in the training process, but it is acceptable in the model training phase, and the size of the model itself is not affected.

Future Scope of Research
In the future, other methods will be explored to improve the current resistance to distillation and to speed up the training time of the model so that the proposed concept can be generally applied in practice. At the same time, we will also consider adding a model watermark to protect the ownership of the vehicle model.