app开发定制【springcloud gateway诡异间歇性500异常解决】500 Server Error，ConcurrentModificationException: null

app开发定制最近在搭建一个新的网关时，app开发定制发布页面访问或者访问，会间歇性,500异常,app开发定制折腾了很长的时间，搜了国内国外各种网站，都没有具体的解决方案。

    经过2，3天的 关注和分析定位，最终解决，mark一下

具体异常信息

  1. 022-02-19 21:50:20.279 ERROR 35366 --- [ctor-http-nio-3] a.w.r.e.AbstractErrorWebExceptionHandler : [84a6cc5f-36]  500 Server Error for HTTP POST "/template/apilog/add"
  2.  
  3. java.util.ConcurrentModificationException: null
  4. 	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) ~[na:1.8.0_202]
  5. 	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
  6. Error has been observed at the following site(s):
  7. 	|_ checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
  8. 	|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
  9. 	|_ checkpoint ⇢ com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]
  10. 	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
  11. 	|_ checkpoint ⇢ HTTP POST "/template/apilog/add" [ExceptionHandlingWebHandler]
  12. Stack trace:
  13. 		at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) ~[na:1.8.0_202]
  14. 		at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[na:1.8.0_202]
  15. 		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:254) [reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  16. 		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225) [reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  17. 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  18. 		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:228) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  19. 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  20. 		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  21. 		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  22. 		at reactor.core.publisher.Flux.subscribe(Flux.java:8325) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  23. 		at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  24. 		at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:80) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  25. 		at reactor.core.publisher.Mono.subscribe(Mono.java:4213) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  26. 		at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:192) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
  27. 		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  28. 		at reactor.core.publisher.FluxBuffer$BufferExactSubscriber.onComplete(FluxBuffer.java:179) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  29. 		at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  30. 		at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  31. 		at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
  32. 		at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:445) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  33. 		at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:251) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  34. 		at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:382) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  35. 		at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  36. 		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  37. 		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:645) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  38. 		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
  39. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  40. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  41. 		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  42. 		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  43. 		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
  44. 		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
  45. 		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  46. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  47. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  48. 		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  49. 		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  50. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  51. 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  52. 		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  53. 		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  54. 		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  55. 		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  56. 		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  57. 		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
  58. 		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
  59. 		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
  60. 		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
  61. 		at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_202]
  62.  
  63. 2022-02-19 21:50:20.768  INFO 35366 --- [ctor-http-nio-2] c.y.b.a.filter.WhiteListUrlFilter        : req method:POST,url:/template/apilog/add

  注意异常信息中

com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]

    因为出现异常的情况和概率完全不可控，非常随机。一开始怀疑是网关的的问题，网关参数配置问题，hystrix 或者 sentinel的问题。各种排查，均没有解决。

    经过国外一个issues 的点拨，扩展开来，怀疑到 spring-boot-starter-actuator 端点监控上来，于是通过 开放端点用于SpringBoot Admin的监控

management.endpoints.web.gateway=false 关闭监控，再次测试仍然会出现。

   为了定位，在网关中把 spring-boot-starter-actuator 全部删除和排除 ，再次重启测试，访问很稳定，没有间歇性出现500的异常。

  分析： 可能是由于 spring-boot-starter-actuator 在获取 网关上的路由配置时，会将路由信息保存在 java.util.ArrayList中，当更新时，需要清空当前路由。然而当页面访问 网关时，actuate的MetricsWebFilter  需要读取路由信息，发现是空的，所以报了 java.util.ConcurrentModificationException: null 异常，当次请求失败。

  解决办法：在网关中把 spring-boot-starter-actuator 全部删除和排除，从根本上避免出现这样的问题。          此为比较粗暴的解决方案，欢迎同行提供更好的解决办法，可与我联系：tangwwk@126.com 

                                                       2022年2月20日  23：25分

   

有用的链接

网站建设定制开发 软件系统开发定制 定制软件开发 软件开发定制 定制app开发 app开发定制 app开发定制公司 电商商城定制开发 定制小程序开发 定制开发小程序 客户管理系统开发定制 定制网站 定制开发 crm开发定制 开发公司 小程序开发定制 定制软件 收款定制开发 企业网站定制开发 定制化开发 android系统定制开发 定制小程序开发费用 定制设计 专注app软件定制开发 软件开发定制定制 知名网站建设定制 软件定制开发供应商 应用系统定制开发 软件系统定制开发 企业管理系统定制开发 系统定制开发