记一次 .NET 某风控管理系统内存泄漏分析

一、背景

1. 讲故事

上个月中旬，星球里的一位朋友在微信找我，说他的程序跑着跑着内存会不断的缓慢增长并无法释放，寻求如何解决 ?

得，看样子星球还得好好弄!!!不管怎么说，先上 windbg 说话。

二、Windbg 分析

1. 经验推理

从朋友的截图看，有大量的 8216 字节的 byte[]，这表示什么呢?追随本系列的朋友应该知道，有一篇某三甲医院的内存暴涨的dump中，也同样有此 size= (8216-24=8192) 的 byte[] 数组，他的问题是 Oracle 中读取某大字段时sdk里的 OraBuf 出了问题，换句话说，这肯定又是底层或者第三方库中的池对象搞出来的东西，接下来从托管堆看起。

2. 查看托管堆

0:000>!dumpheap-stat
Statistics:
00007ffe107248f048370715478624System.Threading.PreAllocatedOverlapped
00007ffe1079c16048374415479808System.Threading.ThreadPoolBoundHandle
00007ffe1079cff848370123217648System.Threading._IOCompletionCallback
00007ffe106e7a9048370423217792Microsoft.Win32.SafeHandles.SafeFileHandle
00007ffe1079b08848370330956992System.IO.FileSystemWatcher+AsyncReadState
00007ffe1079ceb048370734826904System.Threading.OverlappedData
00007ffe1079ccb048370734826904System.Threading.ThreadPoolBoundHandleOverlapped
0000016c646510802456521473128080Free
00007ffe105abf304881723977571092System.Byte[]

扫完托管堆，卧槽，byte[] 没吸引到我，反而被 System.IO.FileSystemWatcher+AsyncReadState 吸引到了，毕竟被 System.IO.FileSystemWatcher 折腾多次了，它已经深深打入了我的脑海。。。毕竟让程序卡死，让句柄爆高的都是它。。。这一回八成又是它惹的祸，看样子还是有很多程序员栽在这里哈。

为做到严谨，我还是从最大的 System.Byte[] 入手，按size对它进行分组再按totalsize降序，丑陋的脚本我就不发了，直接上脚本的输出结果。

!dumpheap-mt00007ffe105abf30
size=8216,count=483703,totalsize=3790M
size=8232,count=302,totalsize=2M
size=65560,count=6,totalsize=0M
size=131096,count=2,totalsize=0M
size=4120,count=11,totalsize=0M
size=56,count=301,totalsize=0M
size=88,count=186,totalsize=0M
size=848,count=16,totalsize=0M
size=152,count=85,totalsize=0M
size=46,count=242,totalsize=0M
size=279,count=38,totalsize=0M
!dumpheap-mt00007ffe105abf30-min0n8216-max0n8216-short
0000016c664277f0
0000016c66432a48
0000016c6648ef88
0000016c6649daa8
0000016c6649fb00
0000016c664a8b90
...

从输出结果看，size=8216 的 byte[] 有 48w 个，然后脚本也列出了一些 8216 大小的 address 地址，接下来用 !gcroot 看下这些地址的引用。

0:000>!gcroot0000016c664277f0
HandleTable:
0000016C65FC28C0(asyncpinnedhandle)
->0000016C6628DEB0System.Threading.OverlappedData
->0000016C664277F0System.Byte[]
Found1uniqueroots(run'!gcroot-all'toseeallroots).
0:000>!gcroot0000016c667c80d0
HandleTable:
0000016C65FB7920(asyncpinnedhandle)
->0000016C663260F8System.Threading.OverlappedData
->0000016C667C80D0System.Byte[]

从输出中可以看到这些 byte[] 都是 async pinned，也就是当异步IO回来的时候需要给 byte[] 填充的存储空间，接下来我们看看如何通过 OverlappedData 找到源码中定义为 8192 大小的 byte[] 地方。

如果你了解 FileSystemWatcher ，反向查找链大概是这样的 OverlappedData -> ThreadPoolBoundHandleOverlapped -> System.IO.FileSystemWatcher+AsyncReadState -> Buffer[]，这中间涉及到 ThreadPool 和 SafeHandle 的绑定。

0:000>!do0000016C663260F8
Name:System.Threading.OverlappedData
MethodTable:00007ffe1079ceb0
EEClass:00007ffe107ac8d0
Size:72(0x48)bytes
File:C:\ProgramFiles\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
MTFieldOffsetTypeVTAttrValueName
00007ffe106e3c0840009ce8System.IAsyncResult0instance0000000000000000_asyncResult
00007ffe104a0c6840009cf10System.Object0instance0000016c66326140_callback
00007ffe1079cb6040009d018...eading.Overlapped0instance0000016c663260b0_overlapped
00007ffe104a0c6840009d120System.Object0instance0000016c667c80d0_userObject
00007ffe104af50840009d228PTR0instance00000171728f66e0_pNativeOverlapped
00007ffe104aee6040009d330System.IntPtr1instance0000000000000000_eventHandle
00007ffe104ab25840009d438System.Int321instance0_offsetLow
00007ffe104ab25840009d53cSystem.Int321instance0_offsetHigh
0:000>!do0000016c663260b0
Name:System.Threading.ThreadPoolBoundHandleOverlapped
MethodTable:00007ffe1079ccb0
EEClass:00007ffe107ac858
Size:72(0x48)bytes
File:C:\ProgramFiles\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
MTFieldOffsetTypeVTAttrValueName
00007ffe1079ceb040009d68...ng.OverlappedData0instance0000016c663260f8_overlappedData
00007ffe1079b81840009c010...ompletionCallback0instance0000016f661ab8a0_userCallback
00007ffe104a0c6840009c118System.Object0instance0000016c667ca0e8_userState
00007ffe107248f040009c220...locatedOverlapped0instance0000016c66326090_preAllocated
00007ffe104af50840009c330PTR0instance00000171728f66e0_nativeOverlapped
00007ffe1079c16040009c428...adPoolBoundHandle0instance0000000000000000_boundHandle
00007ffe104a723840009c538System.Boolean1instance0_completed
00007ffe1079b81840009bf738...ompletionCallback0static0000016f661ab990s_completionCallback
0:000>!do0000016c667ca0e8
Name:System.IO.FileSystemWatcher+AsyncReadState
MethodTable:00007ffe1079b088
EEClass:00007ffe107a9dc0
Size:64(0x40)bytes
File:C:\ProgramFiles\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.IO.FileSystem.Watcher.dll
Fields:
MTFieldOffsetTypeVTAttrValueName
00007ffe104ab258400002b30System.Int321instance1k__BackingField
00007ffe105abf30400002c8System.Byte[]0instance0000016c667c80d0k__BackingField
00007ffe106e7a90400002d10...es.SafeFileHandle0instance0000016c66326028k__BackingField
00007ffe1079c160400002e18...adPoolBoundHandle0instance0000016c66326058k__BackingField
00007ffe107248f0400002f20...locatedOverlapped0instance0000016c66326090k__BackingField
00007ffe1079b8c8400003028...eSystem.Watcher]]0instance0000016c66326078k__BackingField

上面的 k__BackingField 就是当初丢给 OverlappedData 作为异步IO 读写的缓冲，然后看下 System.IO.FileSystemWatcher+AsyncReadState 的源码。

记一次 .NET 某风控管理系统内存泄漏分析

有了这些原理之后，接下来就可以问朋友是否有对 appsettings 设置了 reloadonchange=true 的情况，朋友找了下代码，写法大概如下：

publicobjectGetxxxFlag()
{
stringvalue=AppConfig.GetConfig("appsettings.json").GetValue("xxxx","0");
returnnew
{
state=200,
data=value
};
}
publicclassAppConfig
{
publicstaticAppConfigGetConfig(stringsettingfile="appsettings.json")
{
returnnewAppConfig(settingfile);
}
}
publicclassAppConfig
{
privateAppConfig(stringsettingfile)
{
_config=newConfigurationBuilder().AddJsonFile(settingfile,optional:true,reloadOnChange:true).Build();
_settingfile=settingfile;
}
}