Learn the advanced administration and management features for production deployments and clustering

1. UltraESB
Administration, Management
Production Deployments and Clustering

2. Production Deployment
● Prepare OS
● Pick Linux - Ubuntu, Debian, RHEL/CentOS
● Kernel 2.6 or later
● Better performance on real HW than VMs
● If a VM dedicate CPU cores and memory
● Ensure Gigabit networking, network card offloading etc is enabled
where applicable
● Download the minimal distribution and the management
distribution

3. Tune OS – e.g. for Debian / Ubuntu
● /etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
● /etc/security/limits.conf
* soft nofile 4096
* hard nofile 65535
● /etc/pam.d/common-session
session required pam_limits.so

4. Create a RAM disk
● /etc/fstab
tmpfs /tmp/ram tmpfs defaults,size=2048M
● conf/ultra-root.xml
● Uncomment and configure “RAMDiskFileCache”
● Mounting RAM disk to /tmp/ram is better since if its not
mounted, a clear error message will appear since directory
will not exist

5. Execution / Startup / Shutdown
● Create a new non-privileged Linux user
● e.g: sudo adduser ultraesb –disabled-login
● Configure an /etc/init.d LSB script and register
● See sample at
http://docs.adroitlogic.org/display/esb/Production+Deployment+Best+Practices
● Configure conf/wrapper.conf to configure service attributes
● Recommended Heap 2G min/max
● Do not go crazy with GC tuning, leave defaults unless there is a
reason

6. Verify adequate logging
● Wrapper service has a log file (wrapper.conf)
● UltraESB has a log file (conf/log4j.properties)
● Point the log files to a location with adequate disk space,
and allow adequate log files with rotation
● Select log levels:
● A new customer may want to leave HTTP connection
debugging on for first few months
● conf/ultra-root.xml – HTTP/S transports

7. Configure JMX
● Enable remote JMX access
● conf/ultra-root.xml & ultraesb.sh / wrapper.conf
● Ensure that ports 1099 and 9994 are open via firewall
● You may change these ports
● Configure JMX security
● conf/management/jmxremote.password
● conf/management/jmxremote.access
● Enable detailed MXBeans
● conf/ultra-root.xml

8. Tuning for Performance
● Do not allocate more than 2GB of Heap (-Xms / -Xmx)
● File cache tuning
● Allocate adequate RAM disk for expected load
● Allow for overflow to local disk
● Work Manager tuning
● Two thread pools
– Primary and Secondary
● Can define more than one WorkManager

9. Tuning XML and Security
Performance
● Use VTD XML where possible
● http://vtd-xml.sourceforge.net/
● Tunable XML processing attributes
● DOM Parser/Serializer caching
● XPath/XSLT instance caching
● XML Secure processing – prevents attacks
● Cipher, XML Cipher and Key Generator Caching

10. Performing Load / Stress testing
● ESB Performance -
http://esbperformance.org/display/comparison/ES
B+Performance
● ToolBox for simple load tests
● Be aware that the UI may not be responsive
● Note:
!! Ensure a RAM disk based file cache exists!!
Maybe better to turn on/off metrics (~5-7% overhead)

11. Production Support
● Is optional, but recommended
● If reporting an issue, include as much information as possible
upfront so that it can be resolved as quickly as possible
● e.g.
– Release / patch version, Log file ** Zipped **, configuration file etc
● Try to get a debug log or for HTTP a header/wire log if suspecting
connection issues (conf/log4j.conf)
● If a P1/P2 issue, notify by phone call as well for instant
acknowledgement

12. Managing Configurations
●
Production, Staging, QA and Dev have different backend service hosts, databases,
passwords etc
●
Use Spring and property files for environment specifics
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>file:conf/ultra.properties</value>
</property>
</bean>
●
Any place in config files can now be substituted with ${key}
●
To specify a default if a property is not specified use ${key:default-value}
●
System properties or environment variables can override – optionally configurable

13. Securing Passwords
● Use Jasypt or simpler Jboss style passwords
● http://docs.adroitlogic.org/display/esb/Securi
ng+Configurations

14. Setting up a Cluster

15. Prerequisites
● Apache ZooKeeper
● For cluster coordination
● Shared file system
● For configuration sharing between nodes

16. External Monitoring
● Java Management Extensions - JMX is the
standard mechanism for Java applications to
expose management aspects
● External systems store metrics read
continuously about the ESB for historical
analysis and reporting
● e.g. Zabbix, Nagios, Hyperic, AppDynamics

17. Using Zabbix
● Free and open source
● Used by Jesta Digital
● http://www.zabbix.com/img/zabconf2012/presentations/Marek_Neumann_Jesta_JEE_Monitoring__FINAL_.pdf
● Semi automated monitoring support built-in for
Zabbix

18. Built-in Monitoring & Alerting
● Summary stats
● Details for a fixed moving window
● Configured by Metrics engine
● conf/ultra-metrics.xml
● Optionally extended by DataPointListeners
● Sync and Async