world-class technical training Are your company’s technical training needs being addressed in the most effective manner? MindShare has over 25 years experience in conducting technical training on cutting-edge technologies. We understand the challenges companies have when searching for quality, effective training which reduces the students’ time away from work and provides cost-effective alternatives. MindShare offers many fl exible solutions to meet those needs. Our courses are taught by highly-skilled, enthusiastic, knowledgeable and experienced instructors. We bring life to knowledge through a wide variety of learning methods and delivery options. training that fi ts your needs MindShare recognizes and addresses your company’s technical training issues with: • Scalable cost training • Customizable training options • Reducing time away from work • Just-in-time training • Overview and advanced topic courses • Training delivered effectively globally • Training in a classroom, at your cubicle or home offi ce • Concurrently delivered multiple-site training MindShare training courses expand your technical skillset 2 PCI Express 2.0 ® 2 Serial Attached SCSI (SAS) 2 Intel Core 2 Processor Architecture 2 DDR2/DDR3 DRAM Technology 2 AMD Opteron Processor Architecture 2 PC BIOS Firmware 2 Intel 64 and IA-32 Software Architecture 2 High-Speed Design 2 Intel PC and Chipset Architecture 2 Windows Internals and Drivers 2 PC Virtualization 2 Linux Fundamentals 2 USB 2.0 ... and many more. 2 Wireless USB All courses can be customized to meet your 2 Serial ATA (SATA) group’s needs. Detailed course outlines can be found at www.mindshare.com bringing life to knowledge. real-world tech training put into practice worldwide *PCI Express ® is a registered trademark of the PCISIG MindShare Learning Options MindShare MindShare MindShare MindShare Classroom Virtual Classroom eLearning Press Intro eLearning In-House Training Virtual In-House Training Books Modules Comprehensive Public Training Virtual Public Training eBooks eLearning Modules Classroom Training Virtual Classroom Training eLearning Module Training MindShare Press Invite MindShare to train The majority of our courses MindShare is also an eLearning Purchase our books and you in-house, or sign-up to live over the web in an inter- company. Our growing list of eBooks or publish your attend one of our many public active environment with WebEx interactive eLearning modules own content through us. classes held throughout the and a phone bridge. We deliver include: MindShare has authored year and around the world. training cost-effectively across • Intro to Virtualization over 25 books and the list No more boring classes, the multiple sites and time zones. Technology is growing. Let us help ‘MindShare Experience‘ is Imagine being trained in your • Intro to IO Virtualization make your book project sure to keep you engaged. cubicle or home offi ce and • Intro to PCI Express 2.0 a successful one. avoiding the hassle of travel. Updates Contact us to attend one of • PCI Express 2.0 our public virtual classes. • USB 2.0 • AMD Opteron Processor Architecture • Virtualization Technology ...and more Engage MindShare Have knowledge that you want to bring to life? MindShare will work with you to “Bring Your Knowledge to Life.” Engage us to transform your knowledge and design courses that can be delivered in classroom or virtual class- room settings, create online eLearning modules, or publish a book that you author. We are proud to be the preferred training provider at an extensive list of clients that include: ADAPTEC • AMD • AGILENT TECHNOLOGIES • APPLE • BROADCOM • CADENCE • CRAY • CISCO • DELL • FREESCALE GENERAL DYNAMICS • HP • IBM • KODAK • LSI LOGIC • MOTOROLA • MICROSOFT • NASA • NATIONAL SEMICONDUCTOR NETAPP • NOKIA • NVIDIA • PLX TECHNOLOGY • QLOGIC • SIEMENS • SUN MICROSYSTEMS SYNOPSYS • TI • UNISYS 4285 SLASH PINE DRIVE COLORADO SPRINGS, CO 80908 USA www.mindshare.com M 1.602.617.1123 O 1.800.633.1440 PCI Express System Architecture MINDSHARE, INC. Ravi Budruk Don Anderson Tom Shanley Technical Edit by Joe Winkles ADDISON-WESLEY DEVELOPER’S PRESS Boston • San Francisco • New York • Toronto Montreal • London • Munich • Paris • Madrid • Sydney Cape Town • Tokyo • Singapore • Mexico City Many of the designations used by manufacturers and sellers to distinguish their prod- ucts are claimed as trademarks. Where those designators appear in this book, and Add- ison-Wesley was aware of the trademark claim, the designations have been printed in initial capital letters or all capital letters. The authors and publishers have taken care in preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connec- tion with or arising out of the use of the information or programs contained herein. Library of Congress Cataloging-in-Publication Data ISBN: 0-321-15630-7 Copyright ©2003 by MindShare, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopy- ing, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America. Published simultaneously in Canada. Sponsoring Editor: Project Manager: Cover Design: Set in 10 point Palatino by MindShare, Inc. 1 2 3 4 5 6 7 8 9-MA-999897 9th Printing, April, 2008 Addison-Wesley books available for bulk purchases by corporations, institutions, and other organizations. For more information please contact the Corporate, Government, and Special Sales Department at (800) 238-9682. Find A-W Developer’s Press on the World-Wide Web at: http://www.awl.com/devpress/ Contents About This Book The MindShare Architecture Series ....................................................................................... 1 Cautionary Note ......................................................................................................................... 2 Intended Audience .................................................................................................................... 2 Prerequisite Knowledge ........................................................................................................... 3 Topics and Organization .......................................................................................................... 3 Documentation Conventions................................................................................................... 4 PCI Express™ ....................................................................................................................... 4 Hexadecimal Notation ........................................................................................................ 4 Binary Notation .................................................................................................................... 4 Decimal Notation ................................................................................................................. 4 Bits Versus Bytes Notation ................................................................................................. 5 Bit Fields ................................................................................................................................ 5 Active Signal States.............................................................................................................. 5 Visit Our Web Site ..................................................................................................................... 5 We Want Your Feedback........................................................................................................... 6 Part One: The Big Picture Chapter 1: Architectural Perspective Introduction To PCI Express.................................................................................................... 9 The Role of the Original PCI Solution............................................................................. 10 Don’t Throw Away What is Good! Keep It ............................................................ 10 Make Improvements for the Future......................................................................... 10 Looking into the Future ............................................................................................. 11 Predecessor Buses Compared ................................................................................................ 11 Author’s Disclaimer........................................................................................................... 12 Bus Performances and Number of Slots Compared ..................................................... 12 PCI Express Aggregate Throughput ............................................................................... 13 Performance Per Pin Compared ...................................................................................... 14 I/O Bus Architecture Perspective ......................................................................................... 16 33 MHz PCI Bus Based System........................................................................................ 16 Electrical Load Limit of a 33 MHz PCI Bus............................................................. 17 PCI Transaction Model - Programmed IO .............................................................. 19 PCI Transaction Model - Peer-to-Peer ..................................................................... 22 PCI Bus Arbitration .................................................................................................... 22 PCI Delayed Transaction Protocol ........................................................................... 23 PCI Retry Protocol: .............................................................................................. 23 PCI Disconnect Protocol: .................................................................................... 24 PCI Interrupt Handling.............................................................................................. 25 vii Contents PCI Error Handling .................................................................................................... 26 PCI Address Space Map ............................................................................................ 27 PCI Configuration Cycle Generation ....................................................................... 29 PCI Function Configuration Register Space ........................................................... 30 PCI Programming Model .......................................................................................... 31 Limitations of a 33 MHz PCI System ....................................................................... 31 Latest Generation of Intel PCI Chipsets .................................................................. 32 66 MHz PCI Bus Based System........................................................................................ 33 Limitations of 66 MHz PCI bus ................................................................................ 34 Limitations of PCI Architecture................................................................................ 34 66 MHz and 133 MHz PCI-X 1.0 Bus Based Platforms................................................. 35 PCI-X Features............................................................................................................. 36 PCI-X Requester/Completer Split Transaction Model .................................. 37 DDR and QDR PCI-X 2.0 Bus Based Platforms ............................................................. 39 The PCI Express Way .............................................................................................................. 41 The Link - A Point-to-Point Interconnect ................................................................ 41 Differential Signaling ................................................................................................. 41 Switches Used to Interconnect Multiple Devices................................................... 42 Packet Based Protocol ................................................................................................ 42 Bandwidth and Clocking........................................................................................... 43 Address Space ............................................................................................................. 43 PCI Express Transactions .......................................................................................... 43 PCI Express Transaction Model................................................................................ 43 Error Handling and Robustness of Data Transfer ................................................. 44 Quality of Service (QoS), Traffic Classes (TCs) and Virtual Channels (VCs) .... 44 Flow Control................................................................................................................ 45 MSI Style Interrupt Handling Similar to PCI-X ..................................................... 45 Power Management.................................................................................................... 45 Hot Plug Support........................................................................................................ 46 PCI Compatible Software Model.............................................................................. 46 Mechanical Form Factors........................................................................................... 47 PCI-like Peripheral Card and Connector ......................................................... 47 Mini PCI Express Form Factor........................................................................... 47 Mechanical Form Factors Pending Release............................................................. 47 NEWCARD Form Factor .................................................................................... 47 Server IO Module (SIOM) Form Factor............................................................ 47 PCI Express Topology ....................................................................................................... 48 Enumerating the System............................................................................................ 50 PCI Express System Block Diagram................................................................................ 51 Low Cost PCI Express Chipset ................................................................................. 51 High-End Server System............................................................................................ 53 PCI Express Specifications ..................................................................................................... 54 viii Contents Chapter 2: Architecture Overview Introduction to PCI Express Transactions........................................................................... 55 PCI Express Transaction Protocol ................................................................................... 57 Non-Posted Read Transactions................................................................................. 58 Non-Posted Read Transaction for Locked Requests ............................................. 59 Non-Posted Write Transactions................................................................................ 61 Posted Memory Write Transactions......................................................................... 62 Posted Message Transactions.................................................................................... 63 Some Examples of Transactions....................................................................................... 64 Memory Read Originated by CPU, Targeting an Endpoint................................. 64 Memory Read Originated by Endpoint, Targeting System Memory.................. 66 IO Write Initiated by CPU, Targeting an Endpoint ............................................... 67 Memory Write Transaction Originated by CPU and Targeting an Endpoint .............................................................................................. 68 PCI Express Device Layers ..................................................................................................... 69 Overview............................................................................................................................. 69 Transmit Portion of Device Layers........................................................................... 71 Receive Portion of Device Layers ............................................................................. 71 Device Layers and their Associated Packets.................................................................. 71 Transaction Layer Packets (TLPs) ............................................................................ 71 TLP Packet Assembly.......................................................................................... 72 TLP Packet Disassembly..................................................................................... 73 Data Link Layer Packets (DLLPs) ............................................................................ 74 DLLP Assembly ................................................................................................... 75 DLLP Disassembly .............................................................................................. 76 Physical Layer Packets (PLPs) .................................................................................. 77 Function of Each PCI Express Device Layer .................................................................. 78 Device Core / Software Layer .................................................................................. 78 Transmit Side. ...................................................................................................... 78 Receive Side.......................................................................................................... 78 Transaction Layer ....................................................................................................... 79 Transmit Side. ...................................................................................................... 80 Receiver Side ........................................................................................................ 81 Flow Control......................................................................................................... 81 Quality of Service (QoS) ..................................................................................... 82 Traffic Classes (TCs) and Virtual Channels (VCs).......................................... 84 Port Arbitration and VC Arbitration ................................................................ 85 Transaction Ordering.......................................................................................... 87 Power Management ............................................................................................ 87 Configuration Registers ...................................................................................... 87 Data Link Layer........................................................................................................... 87 ix Contents Transmit Side ....................................................................................................... 88 Receive Side.......................................................................................................... 89 Data Link Layer Contribution to TLPs and DLLPs ........................................ 89 Non-Posted Transaction Showing ACK-NAK Protocol ................................ 90 Posted Transaction Showing ACK-NAK Protocol ......................................... 92 Other Functions of the Data Link Layer........................................................... 92 Physical Layer ............................................................................................................. 93 Transmit Side ....................................................................................................... 93 Receive Side.......................................................................................................... 93 Link Training and Initialization ........................................................................ 94 Link Power Management ................................................................................... 95 Reset....................................................................................................................... 95 Electrical Physical Layer..................................................................................... 96 Example of a Non-Posted Memory Read Transaction ...................................................... 96 Memory Read Request Phase.................................................................................... 97 Completion with Data Phase .................................................................................... 99 Hot Plug ................................................................................................................................... 101 PCI Express Performance and Data Transfer Efficiency ................................................ 101 Part Two: Transaction Protocol Chapter 3: Address Spaces & Transaction Routing Introduction............................................................................................................................. 106 Receivers Check For Three Types of Link Traffic ....................................................... 107 Multi-port Devices Assume the Routing Burden........................................................ 107 Endpoints Have Limited Routing Responsibilities..................................................... 107 System Routing Strategy Is Programmed .................................................................... 108 Two Types of Local Link Traffic......................................................................................... 108 Ordered Sets ..................................................................................................................... 108 Data Link Layer Packets (DLLPs).................................................................................. 111 Transaction Layer Packet Routing Basics.......................................................................... 113 TLPs Used to Access Four Address Spaces.................................................................. 113 Split Transaction Protocol Is Used................................................................................. 114 Split Transactions: Better Performance, More Overhead.................................... 114 Write Posting: Sometimes a Completion Isn’t Needed....................................... 115 Three Methods of TLP Routing...................................................................................... 117 PCI Express Routing Is Compatible with PCI ............................................................. 117 PCI Express Adds Implicit Routing for Messages ............................................... 118 Why Were Messages Added to PCI Express Protocol? ............................... 118 How Implicit Routing Helps with Messages................................................. 118 Header Fields Define Packet Format and Routing ..................................................... 119 x Contents Using TLP Header Information: Overview.................................................................. 120 General ....................................................................................................................... 120 Header Type/Format Field Encodings ................................................................. 120 Applying Routing Mechanisms .......................................................................................... 121 Address Routing .............................................................................................................. 122 Memory and IO Address Maps .............................................................................. 122 Key TLP Header Fields in Address Routing ........................................................ 123 TLPs with 3DW, 32-Bit Address...................................................................... 123 TLPs With 4DW, 64-Bit Address ..................................................................... 124 An Endpoint Checks an Address-Routed TLP..................................................... 125 A Switch Receives an Address Routed TLP: Two Checks.................................. 125 General ................................................................................................................ 125 Other Notes About Switch Address-Routing................................................ 127 ID Routing......................................................................................................................... 127 ID Bus Number, Device Number, Function Number Limits ............................. 127 Key TLP Header Fields in ID Routing................................................................... 128 3DW TLP, ID Routing ....................................................................................... 128 4DW TLP, ID Routing ....................................................................................... 129 An Endpoint Checks an ID-Routed TLP ............................................................... 130 A Switch Receives an ID-Routed TLP: Two Checks............................................ 130 Other Notes About Switch ID Routing.................................................................. 130 Implicit Routing ............................................................................................................... 131 Only Messages May Use Implicit Routing............................................................ 132 Messages May Also Use Address or ID Routing ................................................. 132 Routing Sub-Field in Header Indicates Routing Method ................................... 132 Key TLP Header Fields in Implicit Routing ......................................................... 132 Message Type Field Summary................................................................................ 133 An Endpoint Checks a TLP Routed Implicitly ..................................................... 134 A Switch Receives a TLP Routed Implicitly ......................................................... 134 Plug-And-Play Configuration of Routing Options ......................................................... 135 Routing Configuration Is PCI-Compatible .................................................................. 135 Two Configuration Space Header Formats: Type 0, Type 1 .............................. 135 Routing Registers Are Located in Configuration Header .................................. 135 Base Address Registers (BARs): Type 0, 1 Headers .................................................... 136 General ....................................................................................................................... 136 BAR Setup Example One: 1MB, Prefetchable Memory Request........................ 138 BAR Setup Example Two: 64-Bit, 64MB Memory Request................................. 140 BAR Setup Example Three: 256-Byte IO Request ................................................ 142 Base/Limit Registers, Type 1 Header Only ................................................................. 144 General ....................................................................................................................... 144 Prefetchable Memory Base/Limit Registers ......................................................... 144 Non-Prefetchable Memory Base/Limit Registers................................................ 146 xi