ENOMEM 错误码的含义如下:
Either there is not enough memory for the operation, or the process is out of address space. 据此,有两种出错的情况:

内存耗尽

进程异常退出,看不到FATAL 信息,即便设置为ulimit -c unlimited , 也没有core;

1.确认是OOM

2.如何分析是哪个进程导致的

如果进程支持
用top 找到耗用内存多的进程;

3. 如何定位是哪个线程导致的

(?)

4.如何定位是哪个函数导致的

地址空间耗尽

跟进链接 http://www.redhat.com/magazine/001nov04/features/vm/

The max_map_count file allows for the restriction of the number of VMAs (Virtual Memory Areas) that a particular process can own. A Virtual Memory Area is a contiguous area of virtual address space. These areas are created during the life of the process when the program attempts to memory map a file, links to a shared memory segment, or allocates heap space. Tuning this value limits the amount of these VMAs that a process can own. Limiting the amount of VMAs a process can own can lead to problematic application behavior because the system will return out of memory errors when a process reaches its VMA limit but can free up lowmem for other kernel uses. If your system is running low on memory in the NORMAL zone, then lowering this value will help free up memory for kernel use.

参考上面的说明,max_map_count这个参数就是允许在VMAs(虚拟内存区域)拥有最大数量,VMA是一个连续的虚拟地址空间,当进程创建一个内存映像文件时VMA的地址空间就会增加,当达到max_map_count了就是返回out of memory errors。这个数据通过下面的命令可以查看:cat /proc/sys/vm/max_map_count

解决方法:扩大max_map_count
参考命令如下:
sudo sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144