转：手工释放linux内存——/proc/sys/vm/drop_cache

[popexizhi:

文中提到使用swap的使用率和si/so来衡量 内存是不是真的吃紧，

感觉很认同，linux的机制一直让自己不太明内存使用到底如何判断，这个倒是很可以参考，

下文搜索：标准认同位置

] 

https://blog.csdn.net/wyzxg/article/details/7279986/

author:skate

time:2012/02/22

 

 手工释放linux内存——/proc/sys/vm/drop_cache

 

转载一篇文章

 

linux的内存查看：

[root@localhost 0.1.0]# free -m
                   total       used       free     shared    buffers     cached
Mem:          4032        694       3337          0          0         25

 

需要说明的是，mem的used=free+buffers+cached，有些情况是cached占用很多资源，算起来数值就是不对，其实不影响实际使用，下面转载部分有说明如何清除cached的占用（实际上可以不清除，不会影响实际使用）

 

当在Linux下频繁存取文件后，物理内存会很快被用光，当程序结束后，内存不会被正常释放，而是一直作为caching。这个问题，貌似有不少人在问，不过都没有看到有什么很好解决的办法。那么我来谈谈这个问题。

 

一、通常情况

先来说说free命令：
引用[root@server ~]# free -m
total used free shared buffers cached
Mem: 249 163 86 0 10 94
-/+ buffers/cache: 58 191
Swap: 511 0 511

其中：
引用total 内存总数
used 已经使用的内存数
free 空闲的内存数
shared 多个进程共享的内存总额
buffers Buffer Cache和cached Page Cache 磁盘缓存的大小
-buffers/cache 的内存数:used - buffers - cached
+buffers/cache 的内存数:free + buffers + cached

可用的memory=free memory+buffers+cached。

有了这个基础后，可以得知，我现在used为163MB，free为86MB，buffer和cached分别为10MB，94MB。

那么我们来看看,如果我执行复制文件,内存会发生什么变化.
引用[root@server ~]# cp -r /etc ~/test/
[root@server ~]# free -m
total used free shared buffers cached
Mem: 249 244 4 0 8 174
-/+ buffers/cache: 62 187
Swap: 511 0 511

在我命令执行结束后，used为244MB，free为4MB，buffers为8MB，cached为174MB，天呐，都被cached吃掉了。别紧张，这是为了提高文件读取效率的做法。

为了提高磁盘存取效率，Linux做了一些精心的设计，除了对dentry进行缓存（用于VFS，加速文件路径名到inode的转换），还采取了两种主要Cache方式：Buffer Cache和Page Cache。前者针对磁盘块的读写，后者针对文件inode的读写。这些Cache有效缩短了 I/O系统调用（比如read，write，getdents）的时间。
那么有人说过段时间，linux会自动释放掉所用的内存。等待一段时间后，我们使用free再来试试，看看是否有释放？

[root@server test]# free -m
total used free shared buffers cached
Mem: 249 244 5 0 8 174
-/+ buffers/cache: 61 188
Swap: 511 0 511

似乎没有任何变化。（实际情况下，内存的管理还与Swap有关）
那么我能否手动释放掉这些内存呢？回答是可以的！

二、手动释放缓存
/proc是一个虚拟文件系统，我们可以通过对它的读写操作做为与kernel实体间进行通信的一种手段。也就是说可以通过修改/proc中的文件，来对当前kernel的行为做出调整。那么我们可以通过调整/proc/sys/vm/drop_caches来释放内存。操作如下：

[root@server test]# cat /proc/sys/vm/drop_caches
0
首先，/proc/sys/vm/drop_caches的值，默认为0。

[root@server test]# sync

手动执行sync命令（描述：sync 命令运行 sync 子例程。如果必须停止系统，则运行sync 命令以确保文件系统的完整性。sync 命令将所有未写的系统缓冲区写到磁盘中，包含已修改的 i-node、已延迟的块 I/O 和读写映射文件）

[root@server test]# echo 3 > /proc/sys/vm/drop_caches
[root@server test]# cat /proc/sys/vm/drop_caches
3

将/proc/sys/vm/drop_caches值设为3

[root@server test]# free -m
total used free shared buffers cached
Mem: 249 66 182 0 0 11
-/+ buffers/cache: 55 194
Swap: 511 0 511

再来运行free命令，会发现现在的used为66MB，free为182MB，buffers为0MB，cached为11MB。那么有效的释放了buffer和cache。
有关/proc/sys/vm/drop_caches的用法在下面进行了说明

引用/proc/sys/vm/drop_caches (since Linux 2.6.16)
Writing to this file causes the kernel to drop clean caches,dentries and inodes from memory, causing that memory to become free.

To free pagecache, use

 echo 1 > /proc/sys/vm/drop_caches;

 to free dentries and inodes, use

echo 2 > /proc/sys/vm/drop_caches;

to free pagecache, dentries and inodes, use

 echo 3 >/proc/sys/vm/drop_caches.

Because this is a non-destructive operation and dirty objects are not freeable, the user should run sync first.

三、我的意见
上述文章就长期以来很多用户对Linux内存管理方面的疑问，给出了一个比较“直观”的回复，我更觉得有点像是核心开发小组的妥协。
对于是否需要使用这个值，或向用户提及这个值，我是有保留意见的：

1、从man可以看到，这值从2.6.16以后的核心版本才提供，也就是老版的操作系统，如红旗DC 5.0、RHEL 4.x之前的版本都没有；
2、若对于系统内存是否够用的观察，我还是原意去看swap的使用率和si/so两个值的大小；[popexizhi: 标准认同位置 , 这个是磁盘被当成内存使用的情况，si/so 就是这个swap的参数，vmstat 可以试试多看看，当然碰上不用swap的就看看别的吧:)]

用户常见的疑问是，为什么free这么小，是否关闭应用后内存没有释放？
但实际上，我们都知道这是因为Linux对内存的管理与Windows不同，free小并不是说内存不够用了，应该看的是free的第二行最后一个值："-/+ buffers/cache: 58 191" 这才是系统可用的内存大小。

实际项目中告诉我们，如果因为是应用有像内存泄露、溢出的问题，从swap的使用情况是可以比较快速可以判断的，但free上面反而比较难查看。相反，如果在这个时候，我们告诉用户，修改系统的一个值，“可以”释放内存，free就大了。用户会怎么想？不会觉得操作系统“有问题”吗？所以说，我觉得既然核心是可以快速清空buffer或cache，也不难做到（这从上面的操作中可以明显看到），但核心并没有这样做（默认值是0），我们就不应该随便去改变它。一般情况下，应用在系统上稳定运行了，free值也会保持在一个稳定值的，虽然看上去可能比较小。当发生内存不足、应用获取不到可用内存、OOM错误等问题时，还是更应该去分析应用方面的原因，如用户量太大导致内存不足、发生应用内存溢出等情况，否则，清空buffer，强制腾出free的大小，可能只是把问题给暂时屏蔽了。

我觉得，排除内存不足的情况外，除非是在软件开发阶段，需要临时清掉buffer，以判断应用的内存使用情况；或应用已经不再提供支持，即使应用对内存的时候确实有问题，而且无法避免的情况下，才考虑定时清空buffer。（可惜，这样的应用通常都是运行在老的操作系统版本上，上面的操作也解决不了）。

 

测试

[root@testserver ~]# uname -a
Linux testserver 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@testserver ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2013       1661        352          0        223       1206
-/+ buffers/cache:        231       1782
Swap:         2047          0       2047

[root@testserver ~]# sync
[root@testserver ~]# sync
[root@testserver ~]# cat /proc/sys/vm/drop_caches
0
[root@testserver ~]# echo 3 > /proc/sys/vm/drop_caches
[root@testserver ~]# cat /proc/sys/vm/drop_caches    
3
[root@testserver ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2013        100       1913          0          0         14
-/+ buffers/cache:         85       1927
Swap:         2047          0       2047
[root@testserver ~]#

 

VirtualAllocEx -windows- flAllocationType/flProtect说明

https://msdn.microsoft.com/zh-cn/library/windows/desktop/aa366890(v=vs.85).aspx

LPVOID WINAPI VirtualAllocEx(
  _In_     HANDLE hProcess,
  _In_opt_ LPVOID lpAddress,
  _In_     SIZE_T dwSize,
  _In_     DWORD  flAllocationType,
  _In_     DWORD  flProtect
);

flAllocationType 
  

The type of memory allocation. This parameter must contain one of the following values.

Value	Meaning
MEM_COMMIT 0x00001000	Allocates memory charges (from the overall size of memory and the paging files on disk) for the specified reserved memory pages. The function also guarantees that when the caller later initially accesses the memory, the contents will be zero. Actual physical pages are not allocated unless/until the virtual addresses are actually accessed. [popexizhi：这里的虚拟内存申请要求保证在同一个虚拟内存页上完成，并且帮忙初始化0，但是如果虚拟内存使用完了，可以被分配成物理内存。这个逻辑好奇怪，windows是先使用虚拟内存吗？go一下，不过这个函数名称就是VirtualAllocEx所以首选使用虚拟内存吧？！ go了一下 https://blog.csdn.net/luckyxiaoqiang/article/details/7200767 真的是windows首选使用虚拟内存，在使用完的情况下才考虑物理内存，好吧和linux正好是反着的，这个背后的哲学是什么？io不慢吗？不猜测，[next]找一些windows内存管理机制的相关内容看一下。 ] To reserve and commit pages in one step, call VirtualAllocEx with MEM_COMMIT | MEM_RESERVE. Attempting to commit a specific address range by specifying MEM_COMMIT without MEM_RESERVE and a non-NULLlpAddress fails unless the entire range has already been reserved. The resulting error code is ERROR_INVALID_ADDRESS. [popexizhi: 当前例子中追加shellcode内存就是使用全部地址+扩展地址的申请吧，这里特殊强调没有 MEM_RESERVE的MEM_COMMIT使用 在非全部地址时会失败的。这个是要保证虚拟地址的完整块上的连续性啊。 ] An attempt to commit a page that is already committed does not cause the function to fail. This means that you can commit pages without first determining the current commitment state of each page. If lpAddress specifies an address within an enclave, flAllocationType must be MEM_COMMIT.
MEM_RESERVE 0x00002000	Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk. You commit reserved pages by calling VirtualAllocEx again with MEM_COMMIT. To reserve and commit pages in one step, call VirtualAllocEx with MEM_COMMIT | MEM_RESERVE. Other memory allocation functions, such as malloc and LocalAlloc, cannot use reserved memory until it has been released.
MEM_RESET 0x00080000	Indicates that data in the memory range specified by lpAddress and dwSize is no longer of interest. The pages should not be read from or written to the paging file. However, the memory block will be used again later, so it should not be decommitted. This value cannot be used with any other value. Using this value does not guarantee that the range operated on with MEM_RESET will contain zeros. If you want the range to contain zeros, decommit the memory and then recommit it. When you use MEM_RESET, the VirtualAllocEx function ignores the value of fProtect. However, you must still set fProtect to a valid protection value, such as PAGE_NOACCESS. VirtualAllocEx returns an error if you use MEM_RESET and the range of memory is mapped to a file. A shared view is only acceptable if it is mapped to a paging file.
MEM_RESET_UNDO 0x1000000	MEM_RESET_UNDO should only be called on an address range to which MEM_RESET was successfully applied earlier. It indicates that the data in the specified memory range specified by lpAddress and dwSize is of interest to the caller and attempts to reverse the effects of MEM_RESET. If the function succeeds, that means all data in the specified address range is intact. If the function fails, at least some of the data in the address range has been replaced with zeroes. This value cannot be used with any other value. If MEM_RESET_UNDO is called on an address range which was not MEM_RESET earlier, the behavior is undefined. When you specify MEM_RESET, the VirtualAllocEx function ignores the value of flProtect. However, you must still set flProtect to a valid protection value, such as PAGE_NOACCESS. Windows Server 2008 R2, Windows 7, Windows Server 2008, Windows Vista, Windows Server 2003 and Windows XP:  The MEM_RESET_UNDO flag is not supported until Windows 8 and Windows Server 2012.

flProtect [in]

https://docs.microsoft.com/zh-cn/windows/desktop/Memory/memory-protection-constants

Constant/value	Description
PAGE_EXECUTE 0x10	Enables execute access to the committed region of pages. An attempt to write to the committed region results in an access violation. This flag is not supported by the CreateFileMapping function.
PAGE_EXECUTE_READ 0x20	Enables execute or read-only access to the committed region of pages. An attempt to write to the committed region results in an access violation. Windows Server 2003 and Windows XP: This attribute is not supported by the CreateFileMapping function until Windows XP with SP2 and Windows Server 2003 with SP1.
PAGE_EXECUTE_READWRITE 0x40	Enables execute, read-only, or read/write access to the committed region of pages. Windows Server 2003 and Windows XP: This attribute is not supported by the CreateFileMapping function until Windows XP with SP2 and Windows Server 2003 with SP1. [popexizhi: 只读或读写都可以，但是要求系统的支持版本]
PAGE_EXECUTE_WRITECOPY 0x80	Enables execute, read-only, or copy-on-write access to a mapped view of a file mapping object. An attempt to write to a committed copy-on-write page results in a private copy of the page being made for the process. The private page is marked as PAGE_EXECUTE_READWRITE, and the change is written to the new page. This flag is not supported by the VirtualAlloc or VirtualAllocEx functions. Windows Vista, Windows Server 2003 and Windows XP: This attribute is not supported by the CreateFileMapping function until Windows Vista with SP1 and Windows Server 2008.
PAGE_NOACCESS 0x01	Disables all access to the committed region of pages. An attempt to read from, write to, or execute the committed region results in an access violation. This flag is not supported by the CreateFileMapping function.
PAGE_READONLY 0x02	Enables read-only access to the committed region of pages. An attempt to write to the committed region results in an access violation. If Data Execution Prevention is enabled, an attempt to execute code in the committed region results in an access violation.
PAGE_READWRITE 0x04	Enables read-only or read/write access to the committed region of pages. If Data Execution Prevention is enabled, attempting to execute code in the committed region results in an access violation.
PAGE_WRITECOPY 0x08	Enables read-only or copy-on-write access to a mapped view of a file mapping object. An attempt to write to a committed copy-on-write page results in a private copy of the page being made for the process. The private page is marked as PAGE_READWRITE, and the change is written to the new page. If Data Execution Prevention is enabled, attempting to execute code in the committed region results in an access violation. This flag is not supported by the VirtualAlloc or VirtualAllocEx functions.
PAGE_TARGETS_INVALID 0x40000000	Sets all locations in the pages as invalid targets for CFG. Used along with any execute page protection like PAGE_EXECUTE, PAGE_EXECUTE_READ, PAGE_EXECUTE_READWRITE and PAGE_EXECUTE_WRITECOPY. Any indirect call to locations in those pages will fail CFG checks and the process will be terminated. The default behavior for executable pages allocated is to be marked valid call targets for CFG. This flag is not supported by the VirtualProtect or CreateFileMapping functions.
PAGE_TARGETS_NO_UPDATE 0x40000000	Pages in the region will not have their CFG information updated while the protection changes for VirtualProtect. For example, if the pages in the region was allocated using PAGE_TARGETS_INVALID, then the invalid information will be maintained while the page protection changes. This flag is only valid when the protection changes to an executable type like PAGE_EXECUTE, PAGE_EXECUTE_READ, PAGE_EXECUTE_READWRITE and PAGE_EXECUTE_WRITECOPY. The default behavior for VirtualProtect protection change to executable is to mark all locations as valid call targets for CFG.

oom killer 时 total-vm 远远大于anon-rss+file-rss

https://stackoverflow.com/questions/18845857/what-does-anon-rss-and-total-vm-mean/22326766

As I understand, the size of the virtual memory that a process uses is listed as "total-vm". Part of it is really mapped into the RAM itself (allocated and used). This is "RSS".

Part of the RSS is allocated in real memory blocks (other than mapped into a file or device). This is anonymous memory ("anon-rss") and there is also RSS memory blocks that are mapped into devices and files ("file-rss").

So, if you open a huge file in vim, the file-rss would be high, on the other side, if you malloc() a lot of memory and really use it, your anon-rss would be high also. 【popexizhi: 这个是分配了并且在用，基本是一起high】

On the other side, if you allocate a lot of space (with malloc()), but nevers use it, the total-vm would be higher, but no real memory would be used (due to the memory overcommit), so, the rss values would be low. [popexizhi: 这里解释看来是进程分配了过多内存，但是自己没有用啊！！！ 这个是站着xxx不xxx的典型吗？！kill它就是理所当然了:) . ]

ps:

https://blog.csdn.net/yangguosb/article/details/78247285

total-vm、anon-rss和file-rss含义：

1. total-vm：进程总共使用的虚拟内存；
2. anon-rss：虚拟内存实际占用的物理内存；
3. file-rss：虚拟内存实际占用的磁盘空间

转:释放linux的swap内存

https://blog.csdn.net/chenghuikai/article/details/77476830

第一步：先执行sync命令
#sync
1
sync命令用于强制被改变的内容立刻写入磁盘，更新超块信息，以防止释放，sync命令则可用来强制将内存缓冲区中的数据立即写入磁盘中。
第二步：(如果仅仅是清理swap的话，这一步可以不执行)

#echo 3 > /proc/sys/vm/drop_caches
1
此指令输入立即生效，意在释放所有缓存。
关于drop_caches：
drop_caches的详细文档如下：
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
To free pagecache:
* echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
* echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
* echo 3 > /proc/sys/vm/drop_caches
As this is a non-destructive operation, and dirty objects are notfreeable, the user should run “sync” first in order to make sure allcached objects are freed.
This tunable was added in 2.6.16.
**echo 1:释放页面缓存
echo 2:释放目录文件和inodes
echo 3:释放所有缓存(页面缓存，目录文件和inodes)**

smen 内存统计

https://yq.aliyun.com/articles/54405?commentId=9029

在linux中有一个工具叫smem，其实就是通过smaps来统计的。 
PSS是Pss的相加 
USS则是Private的相加

yum install -y smem smemstat

smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.

https://blog.csdn.net/adaptiver/article/details/7084364

Terms

• VSS- Virtual Set Size 虚拟耗用内存（包含共享库占用的内存）
• RSS- Resident Set Size 实际使用物理内存（包含共享库占用的内存）
• PSS- Proportional Set Size 实际使用的物理内存（比例分配共享库占用的内存）
• USS- Unique Set Size 进程独自占用的物理内存（不包含共享库占用的内存）

一般来说内存占用大小有如下规律：VSS >= RSS >= PSS >= USS

smem
  PID User     Command                         Swap      USS      PSS      RSS 
23716 digoal   postgres: postgres postgres        0     4924     5387     7040 

对应的RSS, PSS, USS分别等于以下相加.  
# cat /proc/23716/smaps | grep Rss
# cat /proc/23716/smaps | grep Pss
# cat /proc/23716/smaps | grep Private_

读书笔记--内存检查--特征归纳

特征归纳

• 简化列表
  • 系统范围的物理和虚拟内存使用率
    • 某个时刻free（第一行和第二行）
    • 按时间显示：vmstat 【memory】
  • 饱和度
    • 换页
      • sar -B [pgpgin/s 页换入；pgpgout/s 页换出]
      • linux ： vmstat 【si/so】
    • 交换
      • 只对solaris有效 ---vmstat [so/si]
    • OOM终结者
      • 只对linux有效 ---
        
        • /var/log/message 中grep "Out of memory"
          dmesg 中grep “Out of memory”
  • 内核和文件系统缓存使用情况
    • 文件系统使用free或者vmstat的buff和cached查看
    • 内核使用 free中的used；
  • 每个进程的物理和虚拟内存使用情况
    • ps aux中的RSS/VSZ (%MEM)
  • 是否存在内存资源控制
    • 【？】这个是什么不太明白
•