【高并发优化实践】10倍请求压力来袭，你的系统会被击垮吗？【石杉的架构笔记】

欢迎关注微信公众号：石杉的架构笔记（id：shishan100）

个人新课**《C2C 电商系统微服务架构120天实战训练营》在公众号儒猿技术窝**上线了，感兴趣的同窗，能够点击下方连接了解详情：

《C2C 电商系统微服务架构120天实战训练营》

“ 上篇文章一次 JVM FullGC的背后，竟隐藏着惊心动魄的线上生产事故！，给你们讲了一个线上系统由于JVM FullGC异常宕机的case。
这篇文章，咱们继续给你们聊聊另一个线上系统在生产环境遇到的问题。

1、背景介绍

背景状况是这样：线上一个系统，在某次高峰期间MQ中间件故障的状况下，触发了降级机制，结果降级机制触发以后运行了一小会儿，忽然系统就彻底卡死，没法响应任何请求。

给你们简单介绍一下这个系统的总体架构，这个系统简单来讲就是有一个很是核心的行为，就是往MQ里写入数据，可是这个往MQ里写入的数据是很是核心及关键的，绝对不允许有丢失。

因此最初就设计了一个降级机制，若是一旦MQ中间件故障，那么这个系统立马就会把核心数据写入本地磁盘文件。

额外提一句，若是有同窗不太清楚MQ中间件的概念，建议看一下以前发的一篇文章「Java进阶面试系列之一」大家系统架构中为什么要引入消息中间件？，先对MQ中间件这个东西作一个基本的了解。

可是若是说在高峰期并发量比较高的状况下，接收到一条数据立马同步写本地磁盘文件，这个性能绝对是极其差的，会致使系统自身的吞吐量瞬间大幅度降低，这个降级机制是绝对没法在生产环境运行的，由于本身就会被高并发请求压垮。

所以当时设计的时候，对降级机制进行了一番精心的设计。

咱们的核心思路是一旦MQ中间件故障，触发降级机制以后，系统接收到一条请求不是立马写本地磁盘，而是采用内存双缓冲 + 批量刷磁盘的机制。

简单来讲，系统接收到一条消息就会立马写内存缓冲，而后开启一个后台线程把内存缓冲的数据刷新到磁盘上去。

整个过程，你们看看下面的图，就知道了。

这个内存缓冲实际在设计的时候，分为了两个区域。

一个是current区域，用来供系统写入数据，另一个是ready区域，用来供后台线程刷新数据到磁盘里去。

每一块内存区域设置的缓冲大小是512kb，系统接收到请求就写current缓冲区，可是current缓冲区总共就512kb的内存空间，所以必定会写满。

一样，你们结合下面的图，一块儿来看看。

current缓冲区写满以后，就会交换current缓冲区和ready缓冲区。交换事后，ready缓冲区承载了以前写满的512kb的数据。

而后current缓冲区此时是空的，能够继续接着系统继续将新来的数据写入交换后的新的current缓冲区。

整个过程以下图所示：

此时，后台线程就能够将ready缓冲区中的数据经过Java NIO的API，直接高性能append方式的写入到本地磁盘文件里。

固然，这里后台线程会有一整套完善的机制，好比说一个磁盘文件有固定大小，若是达到了必定大小，自动开启一个新的磁盘文件来写入数据。

2、埋下隐患

好！经过上面一套机制，即便是高峰期，也能顺利的抗住高并发的请求，一切看起来都很美好！

可是，当时这个降级机制在开发时，咱们采起的思路，为后面埋下了隐患！

当时采起的思路是：若是current缓冲区写满了以后，全部的线程所有陷入一个while循环无限等待。

等到何时呢？一直须要等到ready缓冲区的数据被刷到磁盘文件以后，清空掉ready缓冲区，而后跟current缓冲区进行交换。

这样current缓冲区要再次变为空的缓冲区，才可让工做线程继续写入数据。

可是你们有没有考虑过一个异常的状况有可能会发生？

就是后台线程刷新ready缓冲区的数据到磁盘文件，实际上也是须要一点时间的。

万一在他刷新数据到磁盘文件的过程当中，current缓冲区忽然也被写满了呢？

此时就会致使系统的全部工做线程没法写入current缓冲区，线程所有卡死。

给你们上一张图，看看这个问题！

这个就是系统的降级机制的双缓冲机制最根本的问题了，在开发好这套降级机制以后，采用正常的请求压力测试过，发现两块缓冲区在设置为512kb的状况下，运做良好，没有什么问题。

3、高峰请求，问题爆发

可是问题就出在高峰期上了。某一次高峰期，系统请求压力达到了平时的10倍以上。

固然正常流程下，高峰期的时候，写请求其实也是直接所有写到MQ中间件集群去的，因此哪怕你高峰期流量增长10倍也无所谓，MQ集群是能够自然抗高并发的。

可是当时不幸的是，在高峰期的时候，MQ中间件集群忽然临时故障，这也是一年遇不到几回的。

这就致使这个系统忽然触发了降级机制，而后就开始写入数据到内存双缓冲里面去。

要知道，此时是高峰期啊，请求量是平时正常的10倍！所以10倍的请求压力瞬间致使了一个问题的发生。

这个问题就是瞬时涌入的高并发请求一下将current缓冲区写满，而后两个缓冲区交换，后台线程开始刷新ready缓冲区的数据到磁盘文件里去。

结果由于高峰期请求涌入过快，致使ready缓冲区的数据还没来得及刷新到磁盘文件，此时current缓冲区又忽然写满了。。。

这就尴尬了，线上系统瞬间开始出现异常。。。

典型的表现就是，全部机器上部署的实例所有线程都卡死，处于wait的状态。

4、定位问题，对症下药

因而，这套系统开始在高峰期没法响应任何请求。后来通过线上故障紧急排查、定位和抢修，才解决了这个问题。

其实说来解决方法也很简单，咱们经过jvm dump出来快照进行分析，查看系统的线程具体是卡在哪一个环节，而后发现大量线程卡死在等待current缓冲区的地方。

这就很明显知道缘由了，解决方法就是对线上系统扩容双段缓冲的大小，从512kb扩容到一个缓冲区10mb。

这样在线上高峰期的状况下，也能够稳稳的让降级机制的双缓冲机制流畅的运行，不会说瞬间高峰涌入的请求打满两块缓冲区。

由于缓冲区越大，就可让ready缓冲区被flush到磁盘文件的过程当中，current缓冲区没那么快被打满。

可是这个线上故障反馈出来的一个教训，就是对系统设计和开发的任何较为复杂的机制，都必需要参照线上高峰期的最大流量来压力测试。只有这样，才能确保任何在系统上线的复杂机制能够经得起线上高峰期的流量的考验。

End

若有收获，请帮忙转发，您的鼓励是做者最大的动力，谢谢！

一大波微服务、分布式、高并发、高可用的原创系列文章正在路上

欢迎扫描下方二维码，持续关注：

石杉的架构笔记（id:shishan100）

十余年BAT架构经验倾囊相授

**> **推荐阅读：** > > 一、[拜托！面试请不要再问我Spring Cloud底层原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be13b83f265da6116393fc7) > > 二、[【双11狂欢的背后】微服务注册中心如何承载大型系统的千万级访问？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be3f8dcf265da613a5382ca) > > 三、[【性能优化之道】每秒上万并发下的Spring Cloud参数优化实战](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be83e166fb9a049a7115580) > > 四、[微服务架构如何保障双11狂欢下的99.99%高可用](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be99a68e51d4511a8090440) > > 五、[兄弟，用大白话告诉你小白都能听懂的Hadoop架构原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5beaf02ce51d457e90196069) > > 六、[大规模集群下Hadoop NameNode如何承载每秒上千次的高并发访问](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bec278c5188253e64332c76) > > 七、【[性能优化的秘密】Hadoop如何将TB级大文件的上传性能优化上百倍](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bed82a9e51d450f9461cfc7) > > [八、](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4)[拜托，面试请不要再问我TCC分布式事务的实现原理坑爹呀！](https://juejin.cn/post/6844903716089233416) [](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4) > > 九、[【坑爹呀！】最终一致性分布式事务如何保障实际生产中99.99%高可用？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4) > > 十、[拜托，面试请不要再问我Redis分布式锁的实现原理！](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf3f15851882526a643e207) > > **十一、****[【眼前一亮！】看Hadoop底层算法如何优雅的将大规模集群性能提高10倍以上？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf5396f51882509a768067e)** > > **十二、****[亿级流量系统架构之如何支撑百亿级数据的存储与计算](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Fjuejin.im%25252525252Fpost%25252525252F5bfab59fe51d4551584c7bcf)** > > 1三、[亿级流量系统架构之如何设计高容错分布式计算系统](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Fjuejin.im%252525252Fpost%252525252F5bfbeeb9f265da61407e9679) > > 1四、[亿级流量系统架构之如何设计承载百亿流量的高性能架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Fjuejin.im%2525252Fpost%2525252F5bfd2df1e51d4574b133dd3a) > > 1五、[亿级流量系统架构之如何设计每秒十万查询的高并发架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Fjuejin.im%25252Fpost%25252F5bfe771251882509a7681b3a) > > 1六、[亿级流量系统架构之如何设计全链路99.99%高可用架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Fjuejin.im%252Fpost%252F5bffab686fb9a04a102f0022) > > 1七、[七张图完全讲清楚ZooKeeper分布式锁的实现原理](https://link.juejin.im/?target=https%3A%2F%2Fjuejin.im%2Fpost%2F5c01532ef265da61362232ed) > > 1八、[大白话聊聊Java并发面试问题之volatile究竟是什么？](https://juejin.cn/post/6844903730303746061) > > 1九、[大白话聊聊Java并发面试问题之Java 8如何优化CAS性能？](https://juejin.cn/post/6844903731234865160) > > 20、[大白话聊聊Java并发面试问题之谈谈你对AQS的理解？](https://juejin.cn/post/6844903732061159437) > > 2一、[大白话聊聊Java并发面试问题之公平锁与非公平锁是啥？](https://juejin.cn/post/6844903732883226637) > > 2二、[大白话聊聊Java并发面试问题之微服务注册中心的读写锁优化](https://juejin.cn/post/6844903734267510798) > > 2三、[互联网公司的面试官是如何360°无死角考察候选人的？（上篇）](https://juejin.cn/post/6844903734930046989) > > 2四、[互联网公司面试官是如何360°无死角考察候选人的？（下篇）](https://juejin.cn/post/6844903735655661581) > > 2五、[Java进阶面试系列之一：哥们，大家的系统架构中为何要引入消息中间件？](https://juejin.cn/post/6844903736444207117) > > 2六、[【Java进阶面试系列之二】：哥们，那你说说系统架构引入消息中间件有什么缺点？](https://juejin.cn/post/6844903737123667975) > > 2七、[【行走的Offer收割机】记一位朋友斩获BAT技术专家Offer的面试经历](https://juejin.cn/post/6844903741213130765) > > 2八、[【Java进阶面试系列之三】哥们，消息中间件在大家项目里是如何落地的？](https://juejin.cn/post/6844903742114906125) > > 2九、[【Java进阶面试系列之四】扎心！线上服务宕机时，如何保证数据100%不丢失？](https://juejin.cn/post/6844903742928601095) > > 30、[一次JVM FullGC的背后，竟隐藏着惊心动魄的线上生产事故！](https://juejin.cn/post/6844903743712935944)**