POX and RYU Controller Performance Analysis on Software Defined Network

From the last decades different types of network schemes are pitched to enhance the user performance. Software Defined Networks (SDN) is also considered as important factor for different network schemes and its proper administration or management. Due to major deployment in today’s networking era SDN are further sub divided in to commercial and open-source controllers. Commercial and open-source controllers are utilized in different type of businesses. According to our knowledge considerable amount of literature is available on these controllers but did not provide or analyse performance of these controllers on different network parameters. This paper evaluates and compares the performance of two well-known SDN open-source controllers POX and RYU with two performance assessments. The first assessment is the implementation of optimal path by using Dijkstra's algorithm from source to destination. Second assessment is the creation of a custom topology in our desired tool (MiniNet emulator). Then, the performance in terms of QoS parameters such as Jitter, throughput, packet loss, and packet delivery ratio are computed by two end hosts in each network. After the assessments, the performance of POX are optimal as compare to the RYU and best suited to be deployed in any scenario.


Introduction
Tremendous volumes of data have contributed to massive data centres. Large-scale Processing and storage across these data centres have resulted in new market models. More complex computer networks have also increased the complexity of computing and storage. Well-developed conventional network infrastructure is a good candidate for domain networking for network engineers. They have limited access to customize new network policies by converting highlevel dynamic firmware into a new network policy. Low-level configuration commands that require manual intervention. Because of this limitation has become extremely difficult for the internet to evolve from advanced protocols and efficient performance. This challenge could result in network management and control capacities confinement, requiring manual improvements to lead to error-prone tasks. This challenge is referred to as 'Internet ossification' by researchers and network engineers [1]. Thus, the current Internet situation is limited to addressing emerging technological trends in networking [2] [36] [37].
To facilitate new network developments, the concept of programmable networks has been proposed [3]. Software-Defined Networks (SDN) are a relatively new programmable network organization technique. Software-Defined Networking (SDN) facilitates control and data plane isolation [4]. This paradigm shift in networking architecture promises to solve several large-scale networks, particularly data centers. SDN has three essential features: first, it can isolate core networking devices' data and control functions using a well-defined application programming interface (API) (generally referred to as Southbound APIs) [5]. Second, SDN offers a single control plane feature, such that a specific software application can conveniently manage different elements of the data plane (usually referred to as Network Operating System) (NOS) [6]. Thirdly, SDN facilitates hierarchical controls, and this architecture offers a global vision of an entire underlying physical network to network engineers or network operators who can improve globally. A well-defined open API (usually called Northbound APIs between control planes and network running applications) enables innovation and efficient network control. Different types of SDN controllers provide various services with different criteria for quality of service (QoS) and scalability. The controller can also alter the network's configuration and facilities at runtime [7]. SDN controllers are mainly implemented in large-scale environments where performance is a key concern. Therefore, an extensive assessment framework is required to select the well-liked controller in each scenario based on the quality of service (QoS) requirement, custom topology, routing protocols, and workload, impacting the controller performance tremendously [39] [40].
This research paper has analysed two well-known opensource SDN controllers with the help of MiniNet commandline interface tools for simulation. We implemented Dijkstra's shortest route algorithm in the POX and RYU SDN controllers and compared these controllers' performance based on various metrics like delay, throughput, packet delivery ratio, jitter, and bitrate to identify the best controllers.
This research article is organized such that the next section defines different terms and definition related to SDN. In section III, we study some related work done in softwaredefined networking controller assessment.

Software-Defined Network Architecture
The primary idea of the software-defined network is elementary. The SDN architecture has three key components described in the below section.
The first is the management plan, which consists of a collection of network programs that handle the softwaredefined network control logic. SDN-enabled networks use programmability rather than command-line interfaces to offer simplicity and ease in deploying new software and facilities, including routing, policy enforcement, load balancing, or custom service provider applications. The network's automation and orchestration are also possible through the current API [8].
The control plane is the second and most clever and critical aspect of the fundamental SDN architecture. This layer contains a controller that manages the packet transmission through the Southbound interface, forwarding various rules and policies to the infrastructure layer [8].
The infrastructure layer, also known as the data plane, is the third level, and it represents network communication devices such as switches, routers, and load balancers.). The southbound APIs link to the control plane by gathering policies, forwarding rules, and applying them to the relevant equipment [9].

SDN Controller
In a software-defined network, SDN controllers' function as the network's "brain." It is the network's operating system (os). It is an intermediate SDN architecture layer, as seen in Figure.1. It is a strategic control point that uses northbound APIs to handle networking appliances or business applications and uses southbound APIs to transfer control information to the underlying routers or switches. An SDN controller (also referred to as an OpenFlow controller) uses the OpenFlow Protocol (OFP) in the SDN architecture to configure the underlying core network equipment and select the right route for data traffic forwarding. Since the control plane is typically a centrally controllable software program, it can handle network traffic dynamically. Several opensource SDN controller programs (NOS) are currently used to deploy the architecture; these are Floodlight [10], POX [11],

Literature Review
The POX and Floodlight controller's performance was compared based on delay and throughput. Experiments were carried out in MiniNet for various network topologies. This investigation was also narrowed to two controllers, rather than any other java or python developed controllers; just a few network variables were computed [18].
The performance compression of ONOS, Open Daylight, POX, and RYU controllers was done based on bandwidth and end-to-end delay parameters. A fixed four-level tree topology with 16 hosts was employed to test the controllers' performance. The analysis concluded that RYU had the least end-to-end delay of these four controllers, and ONOS had the maximum bandwidth. Depending on the aims or needs of the outcome, the best-qualified controllers were frequently chosen. The POX controller was adopted as the most acceptable set of configuration simplicity as the highest priority. Still, performance is not equivalent to Open Daylight, RYU, and ONOS controllers [19].
They differentiated the performance of well common OpenFlow controllers such as POX, NOX, RYU, FloodLight, and OpenFlow reference controllers based on their packet handling capability by changing the packet size and coming pattern in the IP traffic flows. The distributed internet traffic flow generator tool has been used to compute throughput, jitter, packet loss, and delay. Their experimentation outcomes display that Floodlight has better throughput and less delay when differentiated from other controllers [20].
The performance of five controllers (POX, ONOS, Open Daylight, RYU, libfluid) is evaluated using the linear topology in a MiniNet emulator with different switches. Ping and Iperf commands perform the performance assessment. This paper provides a new contribution to measuring and comparing the delay in and throughput responses of the five controllers while increasing the load on the linear topology and stopping responding to the network load (number of switches). Finally, the findings demonstrate that libfluid provides the best throughput performance, and POX offers the best delay performance [21].
The performance of SDN controllers such as Floodlight, Beacon, Open-MUL, and Open-IRIS was assessed. The assessment used three types of traffic: TCP, ICMP, and UDP using Iperf and Ping commands. A method to improve the network's performance by using the QoS technique was the Floodlight controller [22].
The exploration and comparison analysis of POX, RYU, and Open Daylight on network performance parameters such as packet loss, throughput, and jitter are done in this article. Although, using an open-source simulation tool called MiniNet to create different topologies. The data's assessment clearly shows that Ryu has higher throughput relative to Floodlight in all topologies. In all topologies, except Torus, Ryu performs best in cases of latency and jitter [23].
This paper discusses and analyzes POX characteristics, and Floodlight controllers and contrasts their performance parameters to select the popularly known SDN open-source controllers. The parameters are evaluated in various topologies. When varying the number of measurements and the data rate, it is found that, in terms of the packet transmitting time, Floodlight results are much quicker (31 times faster) in all topologies when compared to POX [24].
In terms of available delay and packet forwarding capacity, the implementation of two well-known SDN controllers, Open Daylight and Floodlight, was compared. The simulation modelling was based on a network flow, and the shortest or lowest path technique was also applied. The load-balancing algorithm's introduction has made it possible to optimize the Software-Defined Networking's QoS activity, reduce response times, and optimally spread the load from the connections. Consequently, the proposed load-balancing algorithm dramatically improves the performance of the Open Daylight-based controller in terms of QoS given [25].

Proposed Approach
This section describes SDN open-source controllers, Dijkstra's shortest route algorithm, and the simulation tool used for experimentation.

3.1Dijkstra'sAlgorithm
The majority of the issues that contemporary networks that prohibit adequate load balancing are connected to the routing algorithm itself. The present routing technique is based on the shortest path algorithm. Each packet seeks a route that can cover the fewest number of hops, and this is the same for all packets, even though alternative routes are higher but considerably faster. To simulate the traffic behavior and evaluate the network performance based on the regular algorithm Dijkstra's shortest routing in SDN [26].
The Dijkstra algorithm is called the shortest single-source route. It calculates the length of the shortest route from the source to each of the vertices remaining in the graph. The shortest route problem for a single source can be defined as follows: Let G= be a weighted graph directed with V having the vertices set. The special vertex in V, where s is the source and can be used for any edge in E, Edge Cost (e) is the length of edge e. It should be non-negative for all weights in the graph.

Experimental Setup
The POX and RYU controllers are run in a virtual environment created by VMware Workstation Pro. Ubuntu 18.04 is installed in Virtual Box to construct the simulation's operating environment. The network simulation employs a MiniNet simulator, which can create the network environment and the accompanying simulation within the scope of the virtual environment. For comparison, we build a Python-based RYU, POX controller. The available throughput and PDR of ICMP queries are measured for a situation in which the shortest path technique is used [27].

MiniNet
MiniNet is the open-source network emulator for the SDN in a virtual environment to simulate an extensive network. The most important reason for using the MiniNet is supporting the Open Flow Protocol, a better environment to simulate software Defined Networking controller and test custom network topologies.

Iperf
The iperf network testing program is widely used for measuring bandwidth and network connections. The program can create TCP and UDP data streams as well as assess network throughput, bandwidth, and network quality for these streams [28]. The iperf utility may assess unidirectional or bi-directional throughput between the two endhosts and perform client and server functions. It enhances the tuning of multiple buffers, protocols, and timing parameters. It calculates the failure of packets, latency jitters, etc., and supports several simultaneous links.

Results and Discussion
In this chapter, we simulate and show the simulation outcomes in graphs. We evaluate the network's performance with a custom network topology to measure the algorithm's nature with the simulation results. The parameters used to evaluate the network's performance were throughput, packet delivery ratio, jitter, and packet loss. The parameters have been graphically displayed concerning the number of times.

Comparison Parameters
Comparison parameters played a vital role in evaluating routing protocol algorithms in different network scenarios.

Performance Evaluation
This section describes the results obtained by RYU and POX controllers when the shortest path algorithms mentioned in the preceding section are used. It is worth noting that both SDN controllers operate on the same customized network topology.

Throughput
The amount of data transferred in a unit of time is measured in bites per second (kbps). The throughput can be calculated mathematically by processing several bits per unit of time. Average throughput is calculated according to the following formula [29]: The iperf utility is a well-known network testing tool for measuring bandwidth and network connections. The program can create TCP and UDP data streams and calculate network performance for these streams. The iperf tool performs both server and client features and can measure uni-directionally or bidirectionally throughput between the two end-hosts. The network throughput is computed in bits per second or data packets per second using the iperf real-time technique between source and destination nodes with and without Dijkstra's algorithm implementation and various topologies in the RYU SDN controller. The iperf program was used to assess the controller throughput performance by creating a TCP node-to-node connection where one node acts as the server and the other as the client. To figure out TCP throughput, iperf has carried out in 10 s on the client side, and data have been obtained every 1 s on the server side.

RYU Controller Throughput
The throughput result between source and destination nodes of the RYU SDN controller in Dijkstra's algorithm and normal flow is measured and tabulated in Table 1 and is displayed graphically in Figure. 5.1. The throughput graph helps in discovering end-to-end performance.

POX Controller Throughput
The throughput test between source and destination nodes of POX SDN controller in Dijkstra's algorithm and normal flow is measured and tabulated in Table 2 and is displayed graphically in Figure. 5. The throughput graph helps in discovering end-to-end performance. Table2: POX Controller Throughput

POX and RYU Controller Throughput
The throughput test between source and destination nodes of the RYU SDN controller is measured and tabulated in Table  3 and is displayed graphically in Figure. 6. The throughput graph helps in discovering end-to-end performance.   Figure 6 shows a comparison of the average network throughput of various controllers. Consequently, when compared to RYU, a controller across customized and conventional network topologies, the POX controller has the greatest throughput value. When evaluating the effects of network overload on various numbers of switches, the POX controller outperforms RYU. In comparison, the RYU controller demonstrates the lowest throughput value.

Packet Delivery Ratio
The packet delivery ratio (PDR) is s major indicator for evaluating the efficiency of a routing mechanism in any network. The Packet Delivery Ratio is an essential characteristic for measuring the performance of a routing system in any network. The protocol's performance is determined by the simulation settings chosen. The packet delivery ratio is determined by dividing the total number of data packets arriving at destinations by the total number of data packets transmitted from sources. When there is a high PDR, performance improves. In this research study packet received ratio is derived from the following formula [30].
(2) Figure 5.4. illustrate the packet delivery ratio (PDR) of the RYU controller in Dijkstra's and normal flow. The X-axis represents the number of times for each experiment, and the y-axis represents Packet Delivery Ratio (%).   Figure 8, illustrates the packet delivery ratio (PDR) of the POX controller in Dijkstra's algorithm and normal flow. The X-axis represents the number of times for each experiment, and the y-axis represents Packet Delivery Ratio (%).   Figure 9, illustrates the comparison packet delivery ratio of RYU and POX controller. The X-axis represents the number of times for each experiment, and the y-axis displays Packet Delivery Ratio (%).   Figure 9 depicts a comparison of these two controllers. Finally, the DRR performance of the RYU controller is better than a POX controller.

Jitter
In the last experiment, the jitter is the variance in the time delay or the packet delay between when a packet is transmitted and when it is received, the measuring of jitter by making UDP connection between server and client of POX and RYU controller for the various number of times in standard and custom MiniNet topology [31].

RYU Jitter
Below is the jitter table for the value collected for different packets of the RYU controller in Dijkstra's algorithm and normal flow. The X-axis displays the number of times in seconds for each experiment and the y-axis displays the average jitter calculated for different packets.

POX Jitter
Below is the table of jitter for the value collected for different packets of the POX controller in Dijkstra's algorithm and normal flow. The X-axis displays the number of times in seconds for each experiment and the y-axis represents the average jitter calculated for different packets.   Figure 12, illustrates the comparison jitter of RYU and POX controller. The X-axis displays the number of average jitter times for each experiment, and the y-axis represents the number of packets.   Figure 5.9 depicts a comparison of these two controllers. Finally, the Jitter performance of the RYU controller is better than a POX controller.

Packet Loss
Data is sent and retrieved in small units known as packets in every network system. Packet loss refers to data packets that do not arrive at their destination after being transmitted through a computer network and the number of packets lost or discarded during their journey through a computer network [32] [33].

RYU Controller Packet Loss
The Bellow

POX Controller Packet Loss
The Bellow table is of packet loss ratio variability of POX SDN controller in Dijkstra's algorithm and normal flow, the measuring of packet loss by making UDP connection between server and client of POX for the different number of times in custom MiniNet topology.   Figure 15, illustrates the comparison packet loss of RYU and POX controller. The X-axis represents the number of times of packet losses for each experiment, and the y-axis represents the number of packets.   Figure 15 depicts a comparison of these two controllers. Finally, the RYU controller outperforms the POX controller in packet loss. Software-Defined Networking is a new concept which exists for about 20 years. However, it has become relevant in the network area in the last few years. This is due to the increasing necessities in network programmability and traffic that have been driven by the development of other areas such as network virtualization, mobile devices, and others. A controller is the principal construction of SDN. In this paper, the performance evaluation of two open-source controllers (POX, and RYU) was compared based on jitter, packet loss, throughput, and packet delivery ratio for custom topology in the MiniNet emulator. The performance evaluation of POX and RYU shows that the POX controller provides a better result in terms of throughput. In the packet delivery ratio, jitter, and packet loss the RYU controller provides better performance.