Troubleshoot using the proc filesystem on Linux


Предыдущая | Следующая

The proc filesystem is an important feature of Linux that you can't ignore. proc is a pseudo or virtual filesystem that provides an interface to kernel data structures. In other words, proc isn't an actual filesystem in the real-world sense; rather, it resides only in memory and not on a disk. It is automatically mounted by the system.

Most of its contents are regular files and directories, so you can use most regular Linux tools to navigate the proc filesystem. The examples in this article should run the same on any Linux distribution. My system uses:


 
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
$
 
$ uname  -r
3.10.0-1127.el7.x86_64
$

Basic proc usage

To find out where the /proc filesystem resides, run the mount command and search for proc. Here, you can see that proc is mounted on the /proc directory:


 
$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9506)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
$

Move into the /proc directory and explore it. You will see a lot of directories with numbers; the numbers are just process IDs (PIDs) for the processes that are executing. The files with names are general files that are applicable system-wide and not related to a specific process:


 
$ cd /proc/
$ pwd
/proc
$ ls
1      157    22827  33    3413  3618  384   4709  512   5571  66    741   798   88         consoles     keys          scsi
10     15868  23     3340  3414  362   385   4768  513   5581  6723  742   799   880        cpuinfo      key-users     self
101    15900  23323  3341  3415  3622  39    4769  514   56    68    7466  8     891        crypto       kmsg          slabinfo
1033   16     23370  3342  346   363   392   4770  515   5637  69    747   80    9          devices      kpagecount    softirqs
104    17     23380  3343  3467  364   394   48    516   5652  7     75    800   906        diskstats    kpageflags    stat
10416  17580  23383  3344  347   365   4     4804  517   57    70    76    8367  928        dma          loadavg       swaps
105    18     23469  3346  349   37    40    4805  518   58    7071  77    839   96         driver       locks         sys
106    19     23491  3365  35    374   4094  4807  519   59    71    78    840   98         execdomains  mdstat        sysrq-trigger
107    2      23524  3366  351   375   4096  482   52    6     7199  783   842   9838       fb           meminfo       sysvipc
11     20767  23527  3392  352   376   41    483   53    601   72    784   8446  99         filesystems  misc          timer_list
11412  21     24     3397  3523  377   4180  49    5347  61    73    785   85    993        fs           modules       timer_stats
12     21014  26     3398  358   378   42    494   5348  62    735   786   86    994        interrupts   mounts        tty
120    21035  27     34    359   379   428   495   54    624   736   79    869   9970       iomem        mtrr          uptime
1263   21059  28     3408  36    38    43    508   5421  625   737   793   87    acpi       ioports      net           version
1265   21083  29     3409  360   380   44    509   5463  63    738   794   870   buddyinfo  ipmi         pagetypeinfo  vmallocinfo
1272   22     30     3410  3602  381   45    51    5464  636   739   795   874   bus        irq          partitions    vmstat
13     22055  31     3411  3603  382   46    510   5500  64    74    796   878   cgroups    kallsyms     sched_debug   zoneinfo
14     22074  32     3412  361   383   47    511   5519  65    740   797   879   cmdline    kcore        schedstat
$

Begin by exploring some general system-wide files. For example, /proc/cpuinfo shows information about the CPU on the system—specifically, its make, model, number of cores, speed, CPU flags, etc.:

$ cat /proc/cpuinfo

Similarly, /proc/meminfo provides information about the primary memory or RAM on the system. It also provides additional statistics, like how much memory is used, free, and so on:

$ cat /proc/meminfo

To view all the kernel modules that are loaded, look at the /proc/modules file:

$ cat /proc/modules

To see what all filesystems are supported by your system, look at /proc/filesystems:

$ cat /proc/filesystems

Next, look at the per-process files; for example, systemd, which has a PID of 1:


 
$ ps aux | grep -i systemd | head -1
root         1  0.0  0.0 195976  9088 ?        Ss   Mar03   0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
$

Move to the directory named 1 within /proc and see what is there. There are many files present, and their names are somewhat self-explanatory:


 
$ cd /proc/1
$ pwd
/proc/1
$

$ ls
attr        cmdline          environ  io         mem         ns             pagemap      sched      stack    task
autogroup   comm             exe      limits     mountinfo   numa_maps      patch_state  schedstat  stat     timers
auxv        coredump_filter  fd       loginuid   mounts      oom_adj        personality  sessionid  statm    uid_map
cgroup      cpuset           fdinfo   map_files  mountstats  oom_score      projid_map   setgroups  status   wchan
clear_refs  cwd              gid_map  maps       net         oom_score_adj  root         smaps      syscall
$

To find out if these files are present for each of the processes, pick the current shell ID. You can get that information by running echo $$$$ is a special variable that saves the current shell's PID. Go to /proc and cd into the directory with the same number as the PID. The files are almost identical to the ones for PID 1 or systemd:


 
$ echo $$
21059
$
$ cd /proc/21059
$ pwd
/proc/21059
$
$ ls
attr        cmdline          environ  io         mem         ns             pagemap      sched      stack    task
autogroup   comm             exe      limits     mountinfo   numa_maps      patch_state  schedstat  stat     timers
auxv        coredump_filter  fd       loginuid   mounts      oom_adj        personality  sessionid  statm    uid_map
cgroup      cpuset           fdinfo   map_files  mountstats  oom_score      projid_map   setgroups  status   wchan
clear_refs  cwd              gid_map  maps       net         oom_score_adj  root         smaps      syscall
$

To see what these files contain, run a long-running process. Open a terminal and run the tty command to see which terminal you are using. Then type the cat command and hit Enter. This command will sit and wait for your input:


 
$ tty
/dev/pts/2
$
$ cat

Open another terminal and find the PID of cat using the pgrep command:


 
$ pgrep cat
24335
$

Now, explore the very first per-process file, cmdline. This file shows how the command line is executed, if any parameters are used, etc.:


 
$ cat /proc/24335/cmdline
cat$

Next, look at the directory called cwd, which is kind of a symbolic link (symlink) to the directory where the cat command was executed; in this case, that is /root:


 
$ ls -l  /proc/24335/cwd
lrwxrwxrwx. 1 root root 0 Mar  4 03:35 /proc/24335/cwd -> /root
$

If a binary is executed and a process is spawned, it opens up a few files by default. In general, it opens a standard in (stdin), standard out (stdout), and a standard err (stderr). If you list the fd directory under /proc/ followed by the PID, you can see it returns three symbolic links, all pointing to the terminal (pts2) that was used to execute the command:


 
$ ls -l  /proc/24335/fd/
total 0
lrwx------. 1 root root 64 Mar  4 03:35 0 -> /dev/pts/2
lrwx------. 1 root root 64 Mar  4 03:35 1 -> /dev/pts/2
lrwx------. 1 root root 64 Mar  4 03:35 2 -> /dev/pts/2
$

Another important file is exe, which is a symlink pointing to the absolute path of the binary that was executed. In this case, it is the path of the cat command:


 
$ ls -l /proc/24335/exe
lrwxrwxrwx. 1 root root 0 Mar  4 03:35 /proc/24335/exe -> /usr/bin/cat
$

Similarly, if you cat the environ per-process file, you can view all the environment variables that are defined for the cat process:

$ cat /proc/24335/environ

Important files in proc

 

More Linux resources
  • Linux commands cheat sheet
  • Advanced Linux commands cheat sheet
  • Free online course: RHEL Technical Overview
  • Linux networking cheat sheet
  • SELinux cheat sheet
  • Linux common commands cheat sheet
  • What are Linux containers?
  • Our latest Linux articles

The examples above provide a lot of information about running processes and your system in general that can aid in debugging and troubleshooting issues.

 

Important per-process files

The following are important per-process files to look for in the /proc/ filesystem:

Important system-wide files

Here is a list of important general files (i.e., not specific to any process):

Further steps

This overview of the proc filesystem only touches on the possibilities it offers. For more information, check proc's man page by entering the man proc command or accessing the online version of the proc man page.