A Network Measurement System for Wide Area Networks Matei Ciobotaru, Cătălin Meiroşu, Miron Brezuleanu, Mihai Ivanovici January 15, 2003
Contents 1 Introduction 3 2 Network Performance Measurements Systems 4 2.1 2.2 2.3 Network testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.1 IXIA IxCore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.2 Brix 2500 Verifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.3 Spirent Adtech AX/4000 . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.4 Surveyor and RIPE . . . . . . . . . . . . . . . . . . . . . . . . . . .
CONTENTS 5 Traffic Generation 5.1 Traffic profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 20 20 6 Measurements and Results 24 7 Conclusions and Future Work 26 A Installation 28 A.1 Technical requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.2 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.3 Cable connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 A.4 Software . . .
Chapter 1 Introduction In this report we present a system for measuring the performance of wide area computer networks. The system is used to test network devices (switches, routers) and various network topologies that will be part of the ATLAS experiment at CERN. This experiment (at the CERN Large Hadron Collider) will generate a huge amount of data that will have to be processed and filtered in real time.
Chapter 2 Network Performance Measurements Systems Network testers are devices that can perform measurements of various network parameters. Usually they are deployed in key points of a network and they work in a distributed fashion. They perform measurements by injecting traffic into one end of the network and capturing it at the other end. Using special data embedded in the traffic streams, they are able to determine one-way latencies, packet loss, jitter or throughput.
CHAPTER 2. NETWORK PERFORMANCE MEASUREMENTS SYSTEMS 5 Figure 2.1: The IXIA IxCore Network Tester system, BrixWorx. The Brix 2500 Verifier calculates fundamental network performance statistics (such as one-way packet latency, jitter and packet loss) and application responsiveness (such as Web page download and call setup time) by measuring high-level application transactions.
CHAPTER 2. NETWORK PERFORMANCE MEASUREMENTS SYSTEMS 6 accuracy of the measurements is good (around 10us) but depends on the operating system. The traffic generated is for 10/100Mbps links. 2.2 Synchronization for long distance measurements Time synchronization is a critical piece of infrastructure for any distributed system. In the case of network research, we need synchronized clocks in order to accurately measure delays in a distributed network that may span over a wide geographical area.
CHAPTER 2. NETWORK PERFORMANCE MEASUREMENTS SYSTEMS 7 hierarchical configuration where clocks are synchronized to each other and to world-wide time standards (UTC time). There is a hierarchy of NTP time servers, depending on the accuracy of their time references (from stratum 0 – atomic clocks, up to stratum 15). NTP operates in a client/server fashion. The client queries periodically the time server and it dynamically adjusts its clock to match the one from the server.
CHAPTER 2.
CHAPTER 2. NETWORK PERFORMANCE MEASUREMENTS SYSTEMS 9 • Some of them can be plugged directly into a computer (Meinberg PCI card) Instead of a GPS receiver, one can use a CDMA receiver. The main advantage over GPS is that it does not need a roof-mounted antenna because CDMA signals can be received inside buildings. The system uses the CDMA cellular mobile communications network which, for time and frequency applications, acts as a GPS repeater. The accuracy that can be obtained is around 10 microseconds.
CHAPTER 2. NETWORK PERFORMANCE MEASUREMENTS SYSTEMS 10 The computers involved in the system run a Linux with the nanokernel patches. This allows a nanosecond resolution of the system clock. The PPS signal is distributed from the GPS to the serial ports of all computers via standard Cat5 cabling. The PPS distribution contains also a voltage level converter from TTL to RS-232. The precision obtained with this setup is of 10 microseconds.
Chapter 3 System Architecture 3.1 General overview The network tester that we use can generate Gigabit Ethernet traffic and can measure all the important performance parameters of a network. The functionality is similar to other products that can be found on the market, but our main advantage is the high degree of flexibility and accuracy. The system uses mainly off-the-shelf hardware and custom associated software.
CHAPTER 3. SYSTEM ARCHITECTURE 12 The network cards have internal clocks which are synchronized with the global clock cards. These clock cards are interconnected with wires and are always synchronized between them. All the outgoing packets are marked with timestamps (values of the clock counter) and with sequence numbers (for packet loss calculation). The NIC receiving traffic uses the timestamps and its global clock reference to determine the latencies of the incoming packets.
CHAPTER 3. SYSTEM ARCHITECTURE 13 This measurement system is currently used (with up to 32 Gigabit Ethernet ports) to characterize switches and LANs for the ATLAS Data Collection. The parameters that can be measured are throughput, one-way latency and packet loss. 3.2 Testing Wide Area Networks Initially the tester was designed to run on a Local Area Network (LAN) with all the agents placed nearby and being interconnected by switches, but then we had to extend it to a Wide Area Network (Internet).
CHAPTER 3. SYSTEM ARCHITECTURE 3.3 14 Generating IP packets We have modified the firmware on the network cards in order to be able to generate IP packets instead of just simple Ethernet frames. The traffic descriptor table that is loaded into the card before a test contains all the fields in the IP header, including the IP Type of Service. The TCP and UDP were not implemented due to the increased overhead associated with these protocols.
CHAPTER 3. SYSTEM ARCHITECTURE 15 previous system. Each computer hosting traffic generator NICs requires a slave clock card. Typically tens of traffic generators can be accommodated in one system to generate tens of gigabits of traffic. Each packet can be accurately timestamped. In the following section we shall present in more detail this clock synchronization system.
Chapter 4 The GPS-based Clock Synchronization System To overcome the issues related to the synchronization of geographically separated nodes we decided to use the Global Positioning System (GPS) to provide a time reference. The method that is proposed uses custom made clock boards and GPS cards to do the synchronization.
CHAPTER 4. THE GPS-BASED CLOCK SYNCHRONIZATION SYSTEM 4.2 17 Description of the method At each site of the testbed we use one GPS card and several clock cards (one for each computer). The two signals from the GPS (10MHz and PPS) are distributed to all clock cards. The slave clock cards are forced to count at the same rate by using the 10MHz output from the GPS clock card — in this way they are automatically synchronized. The setup inside a site is presented in Figure 4.
CHAPTER 4. THE GPS-BASED CLOCK SYNCHRONIZATION SYSTEM 18 When the clock cards receive this signal, they start waiting for the next pulse on the 1Hz PPS wire. Keep in mind that this pulse comes from the GPS and is perfectly synchronized with the UTC time. When the pulse comes on the 1Hz wire, it means that the reference time was reached and all the cards will reset their counters.
CHAPTER 4. THE GPS-BASED CLOCK SYNCHRONIZATION SYSTEM 19 Local Site #1 Local Site #3 Internet Local Site #2 Figure 4.3: Global synchronization over the Internet 4.3 Testing the method The setup can be tested by telling all the cards in the system to save in the same time the values of their clock counters. On a local site this is done using a wire from the master clock card — when a pulse comes on this wire, the slaves save the time in a buffer memory.
Chapter 5 Traffic Generation The network tester generates traffic according to a traffic description table that is loaded into the network cards before a test is started. By network traffic we understand a stream of packets. The way the packets are related inside the stream depends on the application, but in general we can specify some statistical parameters that characterize the traffic. Our traffic generator produces the packet stream according to a specified traffic pattern description.
CHAPTER 5.
CHAPTER 5.
CHAPTER 5. TRAFFIC GENERATION (a) Packet size 23 (b) VLAN Id Figure 5.2: Sample histograms for the generated traffic The histograms that result for two of the fields (packet size and the vlan id) are shown in Figure 5.2 The traffic generator system can generate Ethernet frames when testing a Layer 2 environment (LAN) or streams of IP packets for the Internet. The following section presents some results that were obtained with the network tester.
Chapter 6 Measurements and Results The tester was put into operation in the CERN network and some measurements were performed at the IP level ([7]). The system was composed of 2 PCs located in different buildings inside CERN. We measured the average latency at different loads. For the clock synchronization we used the GPS global clock system. The network topology between the two ends is shown in Figure 6.1.
CHAPTER 6. MEASUREMENTS AND RESULTS 25 Latency histogram −3 x 10 18 16 14 12 10 8 6 4 2 0 1085 1090 1095 1100 1105 Latency [us] Figure 6.2: Histogram of latencies for the test inside CERN and national and regional networks. A histogram of latencies obtained during this test is shown in Figure 6.3. NORMALIZED LATENCY HISTOGRAM 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 20 30 40 50 LATENCY [ms] 60 70 80 Figure 6.
Chapter 7 Conclusions and Future Work The network tester was extended to an Internet environment. This implies IP traffic generation and global clock synchronization. The system can measure one-way latency, packet loss, can build histograms. It can generate traffic according to the statistical distributions of the fields of the packets and it can reach the Gigabit Ethernet line speed for all packet sizes.
Bibliography [1] Testing and Modeling Ethernet Switches and Networks for use in ATLAS High-Level Triggers Dobinson, R W; Haas, S; Korcyl K; Le Vine, M J; Lokier, J; Martin, B; Meirosu, C; Saka, F; Vella, K; in: IEEE Trans Nucl Sci.: 48 (2001) no. 3 pt. 1 pp.607-12 [2] Testing Ethernet networks for the ATLAS data collection system Barnes, F R M; Beuran, R; Dobinson, R W; Le Vine, M J; Martin, B; Lokier, J; Meirosu, C in: IEEE Trans Nucl. Sci.: 49 (2002) no. 1 pp.
Appendix A Installation The installation consists in placing the GPS and clock boards in the computers, connecting the cables and the GPS antenna and installing the required software. A.
APPENDIX A. INSTALLATION 29 • switch 10 - ON (10MHz clock on pin 4) • switch 4 - ON (1Hz pulse on pin 8) • all other - OFF Also some clock cards have to be installed into the computers. In the computer with the GPS card you have to put a master clock card and in the others - slave clock cards (in fact you can put master cards in all computers). The GPS is working fine if the green LED is ON (on the connector of the board). You can also see if the GPS is working using an oscilloscope.
APPENDIX A. INSTALLATION A.4.1 30 The GPS card driver The GPS kernel module is provided by Meinberg – we are using MBGtools for Linux v0.2.3beta. On the local master computers you need to install the GPS kernel module – mbgclock.o . The solution was tested with Linux kernels 2.4.4 and 2.4.6. It seems that the Meinberg kernel module does not work on Linux 2.4.2. The module has to be compiled first for the local kernel version. You have to go into the directory mbgtools-lx-0.2.3- beta and type make .
APPENDIX A. INSTALLATION 31 clock test/clock test 1 3 which sends the command ”3” (READ TIME) to board number 1 (the first board /dev/hslclock0 ). Then you can run dump hslclock.sh 1 to see if the clock is counting. NOTE: In order for the clock card to work, it has to be connected to the 10MHz signal from the GPS card – otherwise the counter does not change. A.4.
APPENDIX A. INSTALLATION A.4.4 32 Using the hs master This program is used to drive the synchronization mechanism on all computers. After a successful installation you can start the hs master program on all ”master” computers hosting GPS cards. Before trying anything else you should set the clock cards in READ TIME mode: clock test 1 3 . Then you should check with dump hslclock.sh if the clocks are counting. After this you can start the hs master program and synchronize the boards.
Appendix B Implementation details In this section we give the basic information on how the synchronization method is implemented. The synchronization setup at each site (local group of nearby nodes) of the measured network is the same. We have one ”master” node which hosts a GPS card and a master clock card. All the other nodes only host a slave clock card. There are some hardware differences between the master and slave clock cards - see Catalin’s log book for detail.
APPENDIX B. IMPLEMENTATION DETAILS 34 For this project we use to types of clock cards: master cards and slave cards. The only difference between them is that the master cards have the Write Down Output enabled. The connectors of the clock cards are shown in the figure Figure B.1. 24 Port #1 Port #2 Port #3 Port #4 Port #1 − 10 MHz clock from GPS (input) Port #2 − Write down (OUTPUT) − only for master card Port #3 − 1 Hz PPS signal from GPS (input) Port #4 − Write down (input) Figure B.
APPENDIX B. IMPLEMENTATION DETAILS B.1.2 35 Software and Firmware The firmware is the program that is implemented by the FPGA and that controls all the activity of the clock board. The firmware is written in Handel-C/VHDL. After the compilation of the sources of the firmware one gets a hardware description/implementation. This description is fitted for the FPGA using a software package called Altera MAX PLUS II software. The computer hosting the clock board controls it via a Linux kernel module.
APPENDIX B. IMPLEMENTATION DETAILS B.3 36 Manager software The driver software consists of 3 small programs that work together: hs master This is the manager program on a site. At each site there is only one instance of hs master that runs on the master computer. This program uses the services of the GPS card and sends commands to the other computers. hs daemon It is the program that listens to commands from the hs master and talks directly to the clock cards. It runs on each computer from the system.
Appendix C Results obtained during development A lot of testing was performed to verify the method (see [4]). The first tests were intended to check the GPS cards and to see how well they behave. Then some tests and measurements were done to see if the synchronization method is working. C.1 Testing the GPS card The GPS card can provide position and timing information to the host computer.
APPENDIX C. RESULTS OBTAINED DURING DEVELOPMENT 38 Figure C.1: GPS Positional parameters for a period of 5 days the difference between the values samples at two consecutive 1Hz PPS pulses and we saw that the values varied between 39999995 and 40000005. These was explained by the fact that the GPS adjusts its output signals to match the satellites. Figure C.2: Ticks per second C.3 Global synchronization test For this test we installed 2 GPS cards in two computers located in different buildings.
APPENDIX C. RESULTS OBTAINED DURING DEVELOPMENT 39 clocks lose their synchronization [4]) so the preferred method is the one that corrects the clocks at each second. The test involves recording the values of all counters in the system at each second, as triggered by the 1Hz PPS signal from the GPS. The table found in Appendix A.4.4 on page 32 shows some of these values. A second test was to run ping type program (echo – reply) that measures the one-way delay and the round trip time between the 2 sites.
Appendix D GPS Synchronization HOW-TO 1. Install the cards and connect the cables 2. Load the kernel modules: for GPS mbgclock.o and clock card hslclock.o 3. Make sure that the modules for the GPS and the clock card are loaded and fully functional: • Check the file /var/log/messages to see if the modules are loaded properly • For the clock card: use the script dump hslclock.
APPENDIX D. GPS SYNCHRONIZATION HOW-TO 41 9. Configure the file hs nodes.conf at each of the sites. In this file you list ONLY the computers from that site - do not list computers from other sites. For example: pcatb56.cern.ch:40000 10. Start the hs master program on the master computer at each of the sites. 11. Choose the option ”Synchronize with PPS thread” 12. Enter the same time on all the master computers, but take into account the local time.
Appendix E Troubleshooting synchronization problems Problems with the cables • If the clock is not counting - probably the 10MHz signal from the GPS is not plugged into the clock card. • If the GPS does not synchronize, check the antenna connection and that the antenna sees enough open sky (NOTE: in this case the clock card will not count because the 10MHz signal will not be available).
APPENDIX E. TROUBLESHOOTING SYNCHRONIZATION PROBLEMS 43 The GPS cards • Time is not synchronized – check the antenna and the cable. The card should see more than 3 satellites. Please note that when moved to a new location, the card needs around 15 minutes to synchronize. • The 10MHz and PPS signals are not available – check the jumpers on the card to activate these outputs. Other problems • If the kernel modules can’t be loaded – probably they need to be recompiled for the current linux kernel version.
