Transaction processing systems are generally considered easier to scale than data warehouses. Relational databases were designed for this type of workload, and there are no esoteric hardware requirements. Mostly, it is just matter of normalizing to the right degree and getting the indexes right. The major challenge in these systems is their extreme concurrency, which means that small temporary slowdowns can escalate to major issues very quickly.
In this presentation, Gwen Shapira will explain how application developers and DBAs can work together to built a scalable and stable OLTP system - using application queues, connection pools and strategic use of caches in different layers of the system.
We are a managed service AND a solution provider of elite database and System Administration skills in Oracle, MySQL and SQL Server
When we have hundreds of processes going as fast as possible, pile-ups will happen.
Key take away: A developer may or may not know anything about how the connection pool is configured. Prepare to dig.
Something causes response times to go up. Maybe we’ll figure out the cause, or maybe it was too short and fleeting to be recorded. The slowdown causes the application server to open more and more connections. Adding load to the shared pool. In addition, these connections may well all be running the same query, with the application configured to retry again and again until something works. This causes many latches, and depending on memory configuration, perhaps swapping too. This causes more load of the CPU, which causes response times to climb, which means the app will open more connections…
Keeping in mind that OLTP workloads are typically CPU-bound, the number of concurrent users the system can support is limited by the number of cores on the database server. A database with 12 cores can typically only run 12 concurrent CPU-bound sessions.
Most developers are well aware that if the connection pool is too small, the database will sit idle while users are either waiting for connections or are being turned away. Since the scalability limitation of small connection pools are known, developers tend to avoid them by creating large connection pools, and increasing their size at the first hint of performance problems. However a too large connection pool is a much greater risk to the application scalability. Here is what the scalability of an OLTP system typically looks like. Amdahl’s law say that the scalability of the system is constrained by its serial component as the users are waiting for shared resources such as IO and CPU (This is the contention delay), but according to the Universal Scalability Law there is a second delay called “coherency delay” – which is the cost of maintaining data consistency in the system, this models waits on latches and mutexes. After a certain point, adding more users to the system will decrease throughput
Even when throughput doesn’t increase, at the point where throughput stops growing linearly, data starts to queue and response times suffer proportionally. If you check the wait events for a system that is past the point of saturation, you will see very high CPU utilization, high “log file sync” event as a result of the CPU contention and high waits for concurrency events such as “buffer busy waits” and “library cache latch”.
The oversized connection pools have to be re-established during failover events or database restarts. The larger the connection pool is, the longer the application will take to recover from failover event, as a result decreasing the availability of the application.
Load testing for the number of connections the DB can easily sustain is the key. When determining the maximum number of connections in the pool, you should run the test with a fixed number of users (Typically the highest you expect to support) and keep increasing the number of connections in the connection pool until response times become unacceptable, the wait events go from “CPU” to concurrency events and the database CPU utilization goes above 70%. Typically all three of these symptoms will start occurring at approximately the same time.
While we believe that most of the connections will be idle most of the time, we can’t be certain that this will be the case. When the database suffers a performance issue, the application or users are bound to retry running same slow queries over and over, eventually using the entire pool and overloading the database.If your connection pool can grow at these times, it means that it will open new connections, a resource intensive operation as we previously noted, to a database that is already abnormally busy.
When running a load test is impractical, you will need to estimate the number of connections based on available data. The factors to consider are:How many cores are available on the database server?How many concurrent users or threads does the application need to support?When an application thread takes a connection from the pool, how much of the time is spent holding the connection without actually running database queries? The more time the application spends “just holding” the connection, the larger the pool will need to be to support the application workload.How much of the database workload is IO-bound? You can check IOWAIT on the database server to determine this. The more IO-bound your workload is, the more concurrent users you can run without running into concurrency contention (You will see a lot of IO contention though). “Number of cores”x4 is a good connection pool starting point. Less if the connections are heavily utilized by the application and there is little IO activity and more if the opposite is true.
The remaining problem is what to do if the number of application servers is large and it is inefficient to divide the connection pool limit among the application servers. Well-architected systems usually have a separate data layer that can be deployed on separate set of servers. This data layer should be the only component of the application allowed to open connections to the databases. In this architecture, the connections are divided between small number of data-layer servers.This design has three great advantages: First, the data layer usually grows much slower than the application and rarely requires new servers to be added, which means that pools rarely require resizing. Second, application requests can be balanced between the data servers based on the remaining pool capacity and third, if there is a need to add application-side caching to the system (such as Memcached), only the data layer needs modification.
There are many options for handling excessing user requests. The only thing that is not an option – throw everything at the database and let the DB queue the excessive load
If service time suddenly increases, investigateArrival rate is monitored for capacity planningQueue size can be calculated. Increase in queue length will immediately impact SLA, and should be investigated and resolved. Note that bursts of longer queue size are expected, so investigate if queue size does not decrease within reasonable time.High utilization rates will impact queue size. Measuring utilization will assist in capacity planning and determining when more or less resources are needed.
Marc Fielding created a high-performance queue system with a database table and two jobs. Some prefer to implement their queues with a file, split and xargs.But if your requirements include multiple subscribers, which may or may not need to retrieve the same message multiple times, which may or may not want to ack messages, and that may or may not want to filter specific message types, then I recommend using an existing solution.
Memcached is probably the cheapest way to use more low-latency RAM in your system. It is simple and scales very well.
It looks like Memcached doesn’t scale well. But a closer look reveals that it is the client that did not scale – I could not drive more than 60K requests per second. Not bad for a small cache at 18 cents per hour.
There is no downside to Memcached that is too big.However, Memcached servers can and will fail, sending more traffic to the database. Make sure that Memcached nodes are sized so failure of a single node will not cripple the application.
You use SimCache to see that with cache size of 10G you will have hit ratio of 95% in Memcached. Memcached has latency of 1ms in your system. With 5% of the queries hitting the database, you expect the database CPU utilization to be around 20%, almost 100% of the DB Time on the CPU, and almost no wait time on the queue between the business and the data layers (you tested this separately when sizing your connection pool). In this case the database latency will be 5ms, so we expect the average latency for the data layer to be 0.95*1+0.05*5=1.2ms.
An increase in the number of cache misses will definitely mean that the database load is increasing at same time, and can indicate that more memory is necessary. Make sure that the number of gets is higher than the number of sets. If you are setting more than getting, the cache is a waste of space. If the number of gets per item is very low, the cache may be oversized. There is no downside to an oversized cache, but you may want to use the memory for another purpose.An increase in the number of evictions can also indicate that more memory is needed. Evicted time shows the time between the last get of the item to its eviction. If this period is short, this is a good indication that memory shortage makes the cache less effective