IN DEPTH Gavin M. Roy M A N N I N G RabbitMQ in Depth RabbitMQ in Depth GAVIN ROY MANNING SHELTER ISLAND For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: [email protected] ©2018 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Karen Miller 20 Baldwin Road Technical editor: James Titcumb PO Box 761 Technical development editor: Phillip Warner Shelter Island, NY 11964 Copyeditor: Andy Carroll Proofreader: Alyson Brener Technical proofreader: Karsten Strøbæk Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617291005 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – EBM – 22 21 20 19 18 17 brief contents PART 1 RABBITMQ AND APPLICATION ARCHITECTURE ...............1 1 ■ Foundational RabbitMQ 3 2 ■ How to speak Rabbit: the AMQ Protocol 18 3 ■ An in-depth tour of message properties 38 4 ■ Performance trade-offs in publishing 58 5 ■ Don’t get messages; consume them 79 6 ■ Message patterns via exchange routing 101 PART 2 MANAGING RABBITMQ IN THE DATA CENTER OR THE CLOUD..........................................................133 7 ■ Scaling RabbitMQ with clusters 135 8 ■ Cross-cluster message distribution 148 PART 3 INTEGRATIONS AND CUSTOMIZATION.........................175 9 ■ Using alternative protocols 177 10 ■ Database integrations 205 v contents preface xiii acknowledgments xiv about this book xv PART 1 RABBITMQ AND APPLICATION ARCHITECTURE......1 1 Foundational RabbitMQ 3 1.1 RabbitMQ’s features and benefits 4 RabbitMQ and Erlang 5 ■ RabbitMQ and AMQP 6 1.2 Who’s using RabbitMQ, and how? 7 1.3 The advantages of loosely coupled architectures 8 Decoupling your application 10 ■ Decoupling database writes 11 ■ Seamlessly adding new functionality 12 Replication of data and events 12 ■ Multi-master federation of data and events 13 ■ The Advanced Message Queuing model 14 1.4 Summary 16 vii viii CONTENTS 2 How to speak Rabbit: the AMQ Protocol 18 2.1 AMQP as an RPC transport 19 Kicking off the conversation 20 ■ Tuning in to the right channel 20 2.2 AMQP’s RPC frame structure 21 AMQP frame components 21 ■ Types of frames 22 Marshaling messages into frames 23 ■ The anatomy of a method frame 24 ■ The content header frame 26 The body frame 26 2.3 Putting the protocol to use 27 Declaring an exchange 27 ■ Declaring a queue 28 Binding a queue to an exchange 29 ■ Publishing a message to RabbitMQ 29 ■ Consuming messages from RabbitMQ 30 2.4 Writing a message publisher in Python 32 2.5 Getting messages from RabbitMQ 36 2.6 Summary 37 3 An in-depth tour of message properties 38 3.1 Using properties properly 39 3.2 Creating an explicit message contract with content-type 41 3.3 Reducing message size with gzip and content-encoding 43 3.4 Referencing messages with message-id and correlation-id 45 Message-id 45 ■ Correlation-id 45 3.5 Born-on dating: the timestamp property 46 3.6 Automatically expiring messages 47 3.7 Balancing speed with safety using delivery-mode 48 3.8 Validating message origin with app-id and user-id 49 app-id 50 ■ user-id 51 3.9 Getting specific with the message type property 51 3.10 Using reply-to for dynamic workflows 52 3.11 Custom properties using the headers property 53 3.12 The priority property 54 3.13 A property you can’t use: cluster-id/reserved 54 3.14 Summary 55 CONTENTS ix 4 Performance trade-offs in publishing 58 4.1 Balancing delivery speed with guaranteed delivery 59 What to expect with no guarantees 60 ■ RabbitMQ won’t accept non-routable messages with mandatory set 62 ■ Publisher Confirms as a lightweight alternative to transactions 64 Using alternate exchanges for unroutable messages 66 Batch processing with transactions 68 ■ Surviving node failures with HA queues 70 ■ HA queues with transactions 72 Persisting messages to disk via delivery-mode 2 72 4.2 When RabbitMQ pushes back 75 Checking the connection status with rabbitpy 77 ■ Using the management API for connection status 77 4.3 Summary 78 5 Don’t get messages; consume them 79 5.1 Basic.Get vs. Basic.Consume 80 Basic.Get 80 ■ Basic.Consume 82 5.2 Performance-tuning consumers 84 Using no-ack mode for faster throughput 85 ■ Controlling consumer prefetching via quality of service settings 86 Using transactions with consumers 89 5.3 Rejecting messages 90 Basic.Reject 90 ■ Basic.Nack 91 ■ Dead letter exchanges 92 5.4 Controlling queues 94 Temporary queues 94 ■ Permanent queues 97 ■ Arbitrary queue settings 99 5.5 Summary 99 6 Message patterns via exchange routing 101 6.1 Simple message routing using the direct exchange 102 Creating the application architecture 103 ■ Creating the RPC worker 107 ■ Writing a simple RPC publisher 110 6.2 Broadcasting messages via the fanout exchange 115 Modifying the facial detection consumer 116 ■ Creating a simple image-hashing consumer 117 6.3 Selectively routing messages with the topic exchange 119 6.4 Selective routing with the headers exchange 122