Más contenido relacionado La actualidad más candente (20) Similar a Advanced messaging with Apache ActiveMQ (20) Advanced messaging with Apache ActiveMQ1. Advanced Messaging with
Apache ActiveMQ
Bosanac Dejan
May 2011
A Progress Software
1 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
2. About me
Bosanac Dejan
Senior Software Engineer at FUSESource - http://
fusesource.com
Apache ActiveMQ committer and PMC member
Co-author of ActiveMQ in Action
A Progress Software
2 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
3. What we are going to cover?
What is ActiveMQ
The Basics
Flow control
Scaling
High Availability
Future
Conclusion
A Progress Software
3 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
4. What is ActiveMQ?
A Progress Software
4 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
5. Apache ActiveMQ
Apache ActiveMQ
• Leading Open Source messaging platform
• Supported Java Standards:
o JMS 1.1, J2EE 1.4, JCA 1.5 and XA
Reliable, high performance messaging
• Out-performs many legacy proprietary message queues
• Configurable for many different deployments
Multi-Protocol/Multi-Language Support
A Progress Software
5 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
6. Background
ActiveMQ started in 2005 at CodeHaus
Moved to Apache Software Foundation in 2006
1,117,537 lines of code
24 committers
Now the most widely used open source messaging
system on the planet
A Progress Software
6 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
7. The Basics
A Progress Software
7 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
8. Messaging is
Loosely coupled exchange of messages between
applications
Location transparency
Can be persistent or non-persistent
Can be transactional
A Progress Software
8 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
9. Topics
A Progress Software
9 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
10. Topics
One message goes to 0-to-many consumers
based on the current subscribers
Think like mailing lists or discussion forums
Ideal for publishing business events
Distributed observer pattern
Allows one part of your system to notify anyone
else who may be interested in an event
A Progress Software
10 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
11. Queues
A Progress Software
11 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
12. Queues
Messages are load balanced across many consumers
Each message goes to exactly one consumer
Consumers compete for messages
Its easy to browse and monitor queues
Ideal for grid style applications
A Progress Software
12 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
13. Challanges
Create a general messaging platform
Support variety of use-cases
• Large number of clients
• Large number of destinations
• Slow consumers
Provide enterprise feaures
• Security
• High availability
• Management
• etc
A Progress Software
13 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
14. Flow Control
A Progress Software
14 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
15. Flow Control - Why?
Dealing with deep queues
Dealing with slow consumers
We want to prevent broker from being flooded with
messages
We want to prevent broker running out of memory and
other resources
A Progress Software
15 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
16. Flow Control - How?
Message Cursors
Producer Flow Control
A Progress Software
16 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
17. Flow Control - Cursors
A Progress Software
17 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
18. Flow Control - Store-based Cursor
A Progress Software
18 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
19. Flow Control - Non-persistent cursor
A Progress Software
19 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
20. Flow Control - Limits
Per destination
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="10mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
System settings
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="256 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
A Progress Software
20 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
21. Flow Control - Producer Flow Control
Throttling producer speed to the speed of consumers
A Progress Software
21 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
22. Flow Control - VM Cursor
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
A Progress Software
22 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
23. Scaling
A Progress Software
23 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
24. Scaling - Types
Vertical scaling
Horizontal scaling
Traffic partitioning
A Progress Software
24 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
25. Vertical Scaling
Increase load capacity using a single broker can handle.
Problems:
• Thread count
• Memory usage
• CPU usage
A Progress Software
25 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
26. Vertical Scaling - Threads
What are threads used for
• For Connections - Thread per Connection (blocking transport)
• For Dispatching - Thread per Destination
A Progress Software
26 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
27. Vertical Scaling - Number of Connections
Use non-blocking transport
<transportConnectors>
<transportConnector name="nio" uri="nio://0.0.0.0:61616"/>
</<transportConnectors>
Enables handling large number of clients
A Progress Software
27 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
28. Vertical Scaling - Number of Destinations
Don’t use dedicated task runner
ACTIVEMQ_OPTS="-Dorg.apache.activemq.UseDedicatedTaskRunner=false"
Use optimized dispatch for queues
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" optimizedDispatch="true">
...
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
A Progress Software
28 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
29. Vertical Scaling - Memory
Give broker enough memory
ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
Configure big enough memory usage
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="1024 mb" />
</memoryUsage>
...
</systemUsage>
</systemUsage>
A Progress Software
29 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
30. Vertical Scaling - CPU
Disable tight encoding
It uses more CPU to create smaller packets
uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)";
A Progress Software
30 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
31. Vertical Scaling - Conclusion
There is a limit to the scalability a single machine can
give
A Progress Software
31 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
32. Horizontal Scaling
Increase load capacity using networked brokers
Concepts:
Network of Broker
A Progress Software
32 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
33. Horizontal Scaling - Network of Brokers
A Progress Software
33 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
34. Horizontal Scaling - NOB Usage
Configuration
<networkConnector name="broker1-broker2"
uri="static:(tcp://broker2:61617)"
dynamicOnly="true"
prefetchSize="1000"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="true"
suppressDuplicateTopicSubscriptions="true"
networkTTL="3">
</networkConnector>
Connecting
failover://(tcp://broker1:61616,tcp://broker2:61616)?randomize=true
A Progress Software
34 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
35. Horizontal Scaling - Conclusion
More latency in processing messages
Beware of complex topologies
A Progress Software
35 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
36. Hybrid Scaling
Partition traffic to more non-connected brokers
A Progress Software
36 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
37. Hybrid Scaling - Conclusion
Pros
• You can use all the tuning techniques used in Vertical scaling
• Have better Horizontal scaleability than using Network Of
Brokers (Less broker cross talk)
Cons
• Added complexity required on the end user Application
A Progress Software
37 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
38. High Availability
A Progress Software
38 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
39. High Availability
Pure Master/Slave
JDBC Master/Slave
Shared File System Master/Slave
A Progress Software
39 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
40. Pure Master-Slave
A Progress Software
40 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
41. Pure Master-Slave
Shared nothing
Fully replicated
• All messages
• All acknowledgements
• All transactions
Slave does not start any transports or network
connections
A Progress Software
41 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
42. JDBC Master-Slave
A Progress Software
42 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
43. JDBC Master-Slave
Extreme reliability – but not as fast
Recommended if already using an enterprise database
No restriction on number of slaves
Simple configuration
A Progress Software
43 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
44. Shared Storage Master-Slave
A Progress Software
44 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
45. Shared Storage Master-Slave
Recommended if you have a SAN
No restriction on number of slaves
Simple configuration
N.B. – ensure file locking works – and times out – NFSv4
good!
A Progress Software
45 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
46. Future
A Progress Software
46 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.
FuseSource Company
A Progress Software
Company
47. Future - ActiveMQ Apollo
http://activemq.apache.org/apollo
ActiveMQ 5.x reached scalability and performance
limits with the current architecture
New broker core
A Progress Software
47 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
48. Future - ActiveMQ Apollo
Reactor Based Thread Model
Scala 2.8 Implementation
Protocol Agnostic
REST Based Management
A Progress Software
48 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
49. Future - ActiveMQ Apollo Performance
http://hiramchirino.net/blog
10 producers/10 consumers
single topic
using Stomp
20 byte payload
A Progress Software
49 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
50. Conclusions
Dynamic community
Leading in terms of messaging innovation
Built for Enterprise
Scalable, Good Performance, Reliable
A Progress Software
50 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
51. Questions?
ActiveMQ Web sites:
• http://activemq.apache.org/
• http://fusesource.com/products/enterprise-activemq/
Blog:
• http://www.nighttale.net/
Twitter:
• http://twitter.com/dejanb
• http://twitter.com/fusenews
A Progress Software
51 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company