IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Application Note - Rev 1.
Information in this document is provided in connection with Intel® products. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted by this document.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Contents 1.0 Introduction................................................................................................................................. 7 1.1 1.2 1.3 1.4 2.0 System Overview ......................................................................................................................13 2.1 2.2 2.3 2.4 2.5 2.6 3.0 Purpose of ATM Example Design ......................................................
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3.7.2 3.7.3 4.0 Software Subsystems & Data Structures ............................................................................... 29 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 iv CRC-32 Checker and Generator High Level Algorithm.......................... 29 CRC-32 Computation ............................................................................. 29 Virtual Circuit Lookup Table - atm_vc_table.uc........................
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.9 4.10 4.11 5.0 4.8.2.2 Counter Index ............................................................................47 4.8.2.3 Global Counter Enable and Flags .............................................48 4.8.3 counters.uc.............................................................................................49 4.8.3.1 counter_reset() ..........................................................................49 4.8.3.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figures 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 vi IP over ATM Encapsulation Format ...................................................................... 9 Frame and PDU Length vs. IP Packet Length .................................................... 10 Expected Ethernet Transmit Bandwidth..............................................................
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 1.0 Introduction Intel develops example software to demonstrate the capabilities of the IXP1200 Network Processor Family. This document describes the implementation of example software demonstrating the IXP1200, IXP1240, and IXP1250 in an ATM environment. In particular, this example design uses the IXP12xx to route IP packets between ATM and Ethernet networks.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 1.2.1 Supported / Not Implemented Functions The following identifies the ATM, Ethernet, and StrongARM supported functions, as well as those functions that are not supported. ATM Support Ethernet Support 1xOC-12 port or up to 4xOC-3 ports (full-duplex). Up to 8 100Mbps Ethernet ports (full duplex). Segmentation and Reassembly (SAR). ATM Adaptation Layer 5 (AAL5 with CRC-32). Routing from Ethernet to ATM ports based on IP.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 1.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design PDUs because 8-bytes of LLC/SNAP plus 8 bytes of AAL5 trailer push them over the 48 byte payload capacity of a single ATM cell. • Fully populated 64-byte minimum-sized Ethernet frames carry 46-byte IP packets, and also fit into 2 cell PDUs, as do IP packets up through 80 bytes. Figure 2. Frame and PDU Length vs. IP Packet Length 1.3.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design A 33-byte IP packet overflows into 2 cells, requiring 53 more bytes on the input wire. This effectively slows down the input rate, and the theoretical best-case Ethernet Transmit bandwidth for this input drops to 475Mbps, well within the capacity of the 8 100Mbps Ethernet ports. Indeed, only in the one-cell/PDU case does the Ethernet transmit bandwidth requirement exceed the 800Mbps available.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 4 shows how data stream PDUs can be created in the Workbench for ATM, Ethernet, IP, and other protocol data streams. These data streams can then be assigned to feed different ports. To test how the example design performs IP routing, different destination IP addresses can be chosen in the PDU. Figure 4. Developer’s Workbench - ATM Data Stream Dialog Box Figure 5 shows the IX Bus Device Status window.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design . Figure 5. Developer’s Workbench - IX Bus Device Status Window 1.4.2 Hardware The README.txt file contained in the vxworks subdirectory of the project source code describes how to build and run the project on hardware using VxWorks®. While the project runs in simulation mode by default, some simple changes to the project configuration must be made before it will run on hardware. To run on hardware, Tornado 2.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design The StrongARM core shares access to SRAM and DRAM with the microengines, and thus can manage the VC and IP tables. The StrongARM core runs a Developer’s Workbench debug library to connect to Developer’s Workbench running on a remote host to debug and download microcode. Figure 6. System Programming Model 2.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 2.3 Software Partitioning The following figures show how the microcode functional blocks are partitioned on IXP12xx hardware for the three system configurations. Figure 7.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design In the OC-12 configuration, there are two message queues (MSGQs) in scratchpad RAM, one for PDUs from each Ethernet Receive microengine. The pool of threads in the ATM transmit microengine alternately poll the two MSGQs. In the OC-3 configurations, there is a buffer descriptor queue (BDQ) in SRAM associated with each ATM transmit port.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 9.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design syndrome is updated appropriately. The VC Table Entry also contains an AAL type field. Currently, this example design supports only classical IP over ATM, where the AAL type can be either 0 or 5. A value of 0 indicates that the VC is not open, so any cell received on that VC is immediately discarded. The LLC/SNAP field specifies the protocol type.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 2.4.2 Ethernet to ATM Data Flow Figure 11 outlines the sequence of events that takes place when processing incoming Ethernet packets. Incoming Ethernet packets can either fit within a single MPKT ("m-packet", 64 byte packet "fragment"), or span multiple MPKTs. The SOP (start of packet) and EOP (end of packet) bits indicate the starting and ending MPKTs. As MPKTs are received, they are stored in an DRAM data buffer.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3. Run the IXP1200 Developer’s Workbench debug library, and connects it to a remote system host via the PCI Ethernet NIC to download and debug IXP1240 microcode. Then, atm_init() is invoked to initialize data structures in memory: • • • • • • Buffer Descriptor Free-list. CRC-32 Lookup Table. IP Lookup Table. VC Lookup Table and hash miss free-list. IP directed broadcast address hash table.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design OC-12 Port 3.1.2 OC-3 Ports "Fast-port" speculative receive requests. "Slow-port" status check before receive requests. VC Cache enabled. VC Cache disabled. NUMBER_OF_ATM_PORTS must be 1. NUMBER_OF_ATM_PORTS may be 1, 2, or 4. High Level Algorithm In all configurations, each Receive thread gets its own RFIFO element, as assigned by port_rx_init(). Figure 12.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3.2 ATM Transmit Microengine The ATM Transmit microengine is an AAL5 Unspecified Bit Rate (UBR) Transmitter that uses a single microengine to move cells at wire-rate in either single OC-12 or up to four OC-3 port configurations. No attempt is made to mix, schedule, or otherwise ’shape’ the order of the cells on the wire.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3.3 IP-Router Microengine The IP Router microengine consumes packets from the ATM receive microengine via a message queue, and routes them to the appropriate Ethernet transmit packetq. In the IXP1200 software-CRC configuration, this function is carried out by two threads residing on the ATM Receive microengine rather than on a dedicated IP router microengine. 3.3.1 Structure All threads are identical.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design • For ATM destinations, enqueue to the ATM Transmit microengine, or for software CRC, to the appropriate AAL5 CRC-32 generation queues. The ETHERNET_LOOPBACK build option enables routing packets from Ethernet Receive ports to Ethernet Transmit ports. This is useful for equipment checkout in the lab.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3.5.1 Ethernet Transmit Structure The Ethernet Transmit microengine contains three fill threads and one transmit scheduler thread. The Ethernet transmitter uses the eight even TIFO elements, allowing the ATM transmitter to use the eight odd Transmit FIFO elements. This is the same TFIFO sharing mechanism that is used by the L3fwd8_1f SDK example, except here the peer transmitter is ATM instead of Ethernet. 3.5.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Quadwords 1-5 are transferred by an sdram_crc[r_fifo_rd, 5] instruction. Quadword 6 contains "Data 11" -- the eleventh 32-bit longword of the cell. Data 11 is stored in the VC table entry to be consumed when the next cell in this PDU arrives.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design • Upon reception of the first cell, data11 is saved in the VC cache/table entry. Upon reception of the 2nd cell, data11 is retrieved from the VC cache/table entry, combined with data0 of the second cell, and written in a single burst to DRAM.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design The hardware byte aligner operates on the data before the CRC computation hardware. This can be seen in the transfer to quadword 0 of the TFIFO element with sdram_crc[t_fifo_wr], mask_right with a byte alignment of 2 and a CRC mask value of 4. Quadwords 1-5 are transferred with sdram_crc[t_fifo_wr, 5] with the same alignment.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 3.7.2 CRC-32 Checker and Generator High Level Algorithm Figure 20.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design The OC-12 configuration uses a VC Table Cache in conjunction with the VC table, however the description of the backing VC table in this section applies with or without the presence of a VC Cache. The VC table entry answers the following questions for the ATM Receive thread: • • • • • 4.1.2 Is the VC open? (If no, discard the cell) Which LLC/SNAP patterns are expected at the start of each PDU? (If no match, discard cell.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 21. Hashed VC Table Structure Primary VC Table (SRAM) VC Entry on collision list Primary VC Entry with a collision list Primary VC Entry without a collision list … Hardware Top of Stack Registers VC Entry at end of freelist ...
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 22. VC Table Index bit positions: - Z Y X Port VPI VCI Bit Position Description X VCI_SIGNIFICANT_BITS - 1 Y VCI_SIGNIFICANT_BITS + VPI_SIGNIFICANT_BITS - 1 Z VCI_SIGNIFICANT_BITS + VPI_SIGNIFICANT_BITS + PORT_SIGNIFICANT_BITS - 1 The project defaults to support a 64K-entry VC table - independent of the number of ports.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Entry Description Next Address of the next entry in the chain of entries that hash to the same row. 0 indicates no next entry. (21 bit SRAM address) Key Hash key used to find this entry, also used to confirm arrival at the desired entry. Key = (atm_header & 0xFFFFFFF0) | port# Buffer Offset Indicates which 64-bit DRAM word in the buffer should receive the next payload.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Entry Description 5: ATM Adaptation Layer 5 AAL 0: VC is not open CRC The CRC-32 syndrome associated with the PDU. It is saved in the VC table entry after a cell is moved, and then retrieved and used when the next cell in the PDU is received. Cell Data11 The last four bytes of the previous cell in this PDU.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Register(s) 4.2.3 Description @vc_crc0...@vc_crc3 Local working copy of the CRC syndrome in the VC Table Entry. @data11_0...@data11_3 Holds the last four bytes of the previous cell in the VC table, so the microengine can combine it with the first four bytes of the subsequent cell and perform a single 8-byte DRAM write including them both. @vc_address...
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.3.3 IP Table Management API The route table is managed by the Route Table Manager (RTM), which may be used from both Transactor Scripts and VxWorks. It may be compiled and loaded as a local foreign model, thus allowing its C functions to be called from a Transactor Script. Or, it can be compiled as a VxWorks loadable object.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.3.3.4 enet_route_add() Adds a route with Ethernet destination to the route table. enet_route_add(char *dest, char *netmask, char *gateway, int itf, int gateway_da_hi32, int gateway_da_lo16,int gateway_sa_hi16, int gateway_sa_lo32) Parameter Description char *dest String IP destination, e.g. "1.1.1.1" char *netmask String netmask, e.g., "255.255.0.0" char *gateway String next hop gateway, e.g., "255.255.0.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 25.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Both descriptors and buffers are stored in arrays. The array index is used to associate a unique DRAM Data Buffer with each SRAM Descriptor: Figure 27. SRAM Descriptor to DRAM Buffer Mapping SRAM data buffer [i+2] SRAM data buffer [i+1] descriptor [i+2] descriptor [i+1] data buffer [i] descriptor [i] A9783-01 4.4.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 29. Buffer Descriptor Format for Ethernet Transmit Destination Port 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1 0 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 0 1 RCV_PORT FL_ID START_BYTE END_BYTE 2 ELE_COUNT -1 3 Entry RCV_PORT Receive Port FL_ID Free list ID START_BYTE END_BYTE ELE_COUNT 4.4.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.4.3 System Limit on Packet Buffers Several factors are involved in the number of packet buffers the system can support: • The Ethernet transmitter uses packetqs (packetq.uc), and the implementation of packetqs can address only 16,000 different buffers. • DRAM capacity used = 2KB/buffer * number of buffers. Therefore, for 16,000 buffers, 32MB of DRAM is consumed, which is half the memory capacity of most baseboards.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.5.2 Usage Model The following model is described by an analogy to waiting in line at a bakery:. Step Sequence Operation Bakery Line Analogy 1 sequence_enter() returns a sequence number to a thread and updates the absolute.enter so that the next time sequence_enter() is invoked, the following sequence number will be returned Enter bakery and take a ticket.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.6.1 MSGQ_HANDLE Parameters The following parameters make up MSGQ_HANDLE and are common to all macros in msgq.uc: Parameter Description io_index GPR storing the current index into the queue. An absolute register is used to share the index between threads. However, if the threads don’t share access to the queue, a relative GPR can be used.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design msgq_send(io_message, MSGQ_HANDLE, RAM_OPTION) Parameter 4.6.5 Description io_message The message to be sent. Valid messages must have bit 31 clear, and must not be 0. 0 is returned on success, the message is untouched on failure. MSGQ_HANDLE Parameters described in “MSGQ_HANDLE Parameters”. RAM_OPTION ctx_swap, sig_done, no_option -- depending on the behavior desired for the write at the end of msgq_send().
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design ... ; process the message, threads may get out of order. move(message, $xfer) sequence_wait(MY_SEQUENCE) ; wait until it is my turn to send msgq_send(message, $xfer, MY_MSGQ, ctx_swap) .if (message != 0) counter_inc(OUTPUT_MSGQ_IS_FULL) ; record failure buf_push(message, ...) ; if message is descriptor, return it... .endif sequence_exit(MY_SEQUENCE) ; allow next thread through sequence_wait() 4.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design For the synchronous empty->non-empty queue notification feature to be used, only one microengine can be assigned to dequeue from each queue. Further, it is optimal when threads on that microengine dequeue from a single queue rather than from multiple queues.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design • On hardware, counters.c is compiled into the atm_utils.o VxWorks-loadable module to provide counters at the VxWorks console. 4.8.1 Global Parameters Parameter 4.8.2 Description COUNTERS_BASE Base address of the scratchpad counter array (mandatory) COUNTER_LOCATIONS Size of the counter array (optional).
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.8.2.3 Global Counter Enable and Flags Global Counter Enable and Flags COUNTERS_ENABLE_MASK is the global counter enable and is set via a #define statement in system_config.h: #define Statement Description COUNTERS_ENABLE_MASK 0xFFFFFFFF Enable all counters (default). COUNTERS_ENABLE_MASK 0 Disable all counters. To enable a counter for a command: 1. Ensure that the COUNTERS_ENABLE_MASK is set to enable. 2.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Counter 4.8.3 Group Description COUNT_QUEUE_FAIL (1 << 8) enqueue/dequeue error events COUNT_CRC32 (1 << 9) normal CRC-32 activity COUNT_CRC32_FAIL (1 << 10) CRC-32 error counters.uc 4.8.3.1 counter_reset() Resets the specified counter to zero. counter_reset(in_counter_base, in_counter_offset, IN_ENABLE_FLAGS) Parameter Description in_counter_base Base counter number. in_counter_offset Counter offset.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Parameter IN_TOTAL_DISCARDS Description Address of global discard counter. Highest valid port number -- from a per-port counters point of view. IN_MAX_PORT_NUMBER IN_ENABLE_FLAGS If the sum of IN_PORT_BASE and in_port_index exceeds IN_MAX_PORT_NUMBER, then the port number is truncated to IN_MAX_PORT_NUMBER.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 4.8.4 counters.c 4.8.4.1 counters_init() Initializes all counters. 4.8.4.2 counters_print() Prints the names and values of all counters. Example In this example of output from counters_print(), the system ran the dual-OC-3 software-CRC configuration overnight with an ATM loop-back cable. All counters were enabled.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 237:[42]: 238:[43]: 239:[44]: 240:[45]: 241:[46]: 242:[47]: 243:[48]: 244:[49]: 245:[50]: 246:[51]: 247:[52]: 248:[53]: 249:[54]: 250:[55]: 251:[56]: 252:[57]: 253:[58]: 254:[59]: 192: 128:[port 138:[port 144:[port 4.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design // sram[read, $foo], ordered, ctx_swap 4.10 Mutex Vectors Mutex vectors are an extension to critical sections that allows multiple critical sections to be contained within a single absolute register. (critsect.uc implements critical sections, critsect macros are documented in the IXP1200 Macro Library Reference Manual.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Parameter out_abs_reg Description Absolute register containing the semaphores. bit number of the semaphore. in_bit_number 0 bits: critical section available. 1 bits: critical section occupied. mutex_vector_exit clears specified bit. 4.11 Inter-Thread Signalling Inter-thread signals are used in four ways: • Initialization, as detailed in the “Microengine Initialization” section.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design // Define DEBUG to enable all the counters and run-time checking. // Disable for maximum performance. // #define DEBUG // Define COUNTERS_ENABLE_MASK to all 1’s to enable every system // Otherwise its default is set in system_config.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 7.0 Simulation Support (Scripts, etc.) Simulation support for this example design is provided by using a combination of the Foreign Model DLLs (libraries linked to the Transactor simulator), with interpreted Transactor scripts (.ind files). The IP Route Table Manager and associated RFC1812 utilities are implemented in the rtm_dll.dll foreign model.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design 10.0 Document Conventions In illustrations of 32-bit registers, or data structures in memory; smaller addresses appear toward the top of the figure, - as they would appear in a memory dump on the screen. Bit positions are numbered from the right to the left. Figure 37.
IXP1200 Network Processor Family ATM OC-3/12/Ethernet IP Router Example Design Figure 39. Definitions (Continued) Term 12.0 Definition PDU Protocol Data Unit Rosetta Intel IXB8055 IX Bus to Utopia Bridge RTM Route Table Manager Slow Port A port that does not have dedicated status lines, and must poll for status Transactor IXP1240 Software Simulator UBR Unspecified Bit Rate VC Virtual Circuit Related Documents Title RFC1577 Description Classical IP over ATM.
