Demonstrations of opensnoop, the Linux eBPF/bcc version.

opensnoop traces the open() syscall system-wide, and prints various details.

Example output:

# ./opensnoop

PID COMM FD ERR PATH

17326 <...> 7 0 /sys/kernel/debug/tracing/trace_pipe

1576 snmpd 9 0 /proc/net/dev

1576 snmpd 11 0 /proc/net/if_inet6

1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms

1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms

1576 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding

1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms

1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms

1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms

1576 snmpd 11 0 /proc/sys/net/ipv6/conf/lo/forwarding

1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms

1576 snmpd 9 0 /proc/diskstats

1576 snmpd 9 0 /proc/stat

1576 snmpd 9 0 /proc/vmstat

1956 supervise 9 0 supervise/status.new

1956 supervise 9 0 supervise/status.new

17358 run 3 0 /etc/ld.so.cache

17358 run 3 0 /lib/x86_64-linux-gnu/libtinfo.so.5

17358 run 3 0 /lib/x86_64-linux-gnu/libdl.so.2

17358 run 3 0 /lib/x86_64-linux-gnu/libc.so.6

17358 run -1 6 /dev/tty

17358 run 3 0 /proc/meminfo

17358 run 3 0 /etc/nsswitch.conf

17358 run 3 0 /etc/ld.so.cache

17358 run 3 0 /lib/x86_64-linux-gnu/libnss_compat.so.2

17358 run 3 0 /lib/x86_64-linux-gnu/libnsl.so.1

17358 run 3 0 /etc/ld.so.cache

17358 run 3 0 /lib/x86_64-linux-gnu/libnss_nis.so.2

17358 run 3 0 /lib/x86_64-linux-gnu/libnss_files.so.2

17358 run 3 0 /etc/passwd

17358 run 3 0 ./run

^C

While tracing, the snmpd process opened various /proc files (reading metrics),

and a "run" process read various libraries and config files (looks like it

was starting up: a new process).

opensnoop can be useful for discovering configuration and log files, if used

during application startup.

The -p option can be used to filter on a PID, which is filtered in-kernel. Here

I've used it with -T to print timestamps:

./opensnoop -Tp 1956

TIME(s) PID COMM FD ERR PATH

0.000000000 1956 supervise 9 0 supervise/status.new

0.000289999 1956 supervise 9 0 supervise/status.new

1.023068000 1956 supervise 9 0 supervise/status.new

1.023381997 1956 supervise 9 0 supervise/status.new

2.046030000 1956 supervise 9 0 supervise/status.new

2.046363000 1956 supervise 9 0 supervise/status.new

3.068203997 1956 supervise 9 0 supervise/status.new

3.068544999 1956 supervise 9 0 supervise/status.new

This shows the supervise process is opening the status.new file twice every

second.

The -U option include UID on output:

# ./opensnoop -U

UID PID COMM FD ERR PATH

0 27063 vminfo 5 0 /var/run/utmp

103 628 dbus-daemon -1 2 /usr/local/share/dbus-1/system-services

103 628 dbus-daemon 18 0 /usr/share/dbus-1/system-services

103 628 dbus-daemon -1 2 /lib/dbus-1/system-services

The -u option filtering UID:

# ./opensnoop -Uu 1000

UID PID COMM FD ERR PATH

1000 30240 ls 3 0 /etc/ld.so.cache

1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libselinux.so.1

1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libc.so.6

1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libpcre.so.3

1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libdl.so.2

1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libpthread.so.0

The -x option only prints failed opens:

# ./opensnoop -x

PID COMM FD ERR PATH

18372 run -1 6 /dev/tty

18373 run -1 6 /dev/tty

18373 multilog -1 13 lock

18372 multilog -1 13 lock

18384 df -1 2 /usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo

18384 df -1 2 /usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo

18384 df -1 2 /usr/share/locale/en_US/LC_MESSAGES/coreutils.mo

18384 df -1 2 /usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo

18384 df -1 2 /usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo

18384 df -1 2 /usr/share/locale/en/LC_MESSAGES/coreutils.mo

18385 run -1 6 /dev/tty

18386 run -1 6 /dev/tty

This caught a df command failing to open a coreutils.mo file, and trying from

different directories.

The ERR column is the system error number. Error number 2 is ENOENT: no such

file or directory.

A maximum tracing duration can be set with the -d option. For example, to trace

for 2 seconds:

# ./opensnoop -d 2

PID COMM FD ERR PATH

2191 indicator-multi 11 0 /sys/block

2191 indicator-multi 11 0 /sys/block

2191 indicator-multi 11 0 /sys/block

2191 indicator-multi 11 0 /sys/block

2191 indicator-multi 11 0 /sys/block

The -n option can be used to filter on process name using partial matches:

# ./opensnoop -n ed

PID COMM FD ERR PATH

2679 sed 3 0 /etc/ld.so.cache

2679 sed 3 0 /lib/x86_64-linux-gnu/libselinux.so.1

2679 sed 3 0 /lib/x86_64-linux-gnu/libc.so.6

2679 sed 3 0 /lib/x86_64-linux-gnu/libpcre.so.3

2679 sed 3 0 /lib/x86_64-linux-gnu/libdl.so.2

2679 sed 3 0 /lib/x86_64-linux-gnu/libpthread.so.0

2679 sed 3 0 /proc/filesystems

2679 sed 3 0 /usr/lib/locale/locale-archive

2679 sed -1 2

2679 sed 3 0 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache

2679 sed 3 0 /dev/null

2680 sed 3 0 /etc/ld.so.cache

2680 sed 3 0 /lib/x86_64-linux-gnu/libselinux.so.1

2680 sed 3 0 /lib/x86_64-linux-gnu/libc.so.6

2680 sed 3 0 /lib/x86_64-linux-gnu/libpcre.so.3

2680 sed 3 0 /lib/x86_64-linux-gnu/libdl.so.2

2680 sed 3 0 /lib/x86_64-linux-gnu/libpthread.so.0

2680 sed 3 0 /proc/filesystems

2680 sed 3 0 /usr/lib/locale/locale-archive

2680 sed -1 2

^C

This caught the 'sed' command because it partially matches 'ed' that's passed

to the '-n' option.

The -e option prints out extra columns; for example, the following output

contains the flags passed to open(2), in octal:

# ./opensnoop -e

PID COMM FD ERR FLAGS PATH

28512 sshd 10 0 00101101 /proc/self/oom_score_adj

28512 sshd 3 0 02100000 /etc/ld.so.cache

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libwrap.so.0

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libaudit.so.1

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libpam.so.0

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libselinux.so.1

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libsystemd.so.0

28512 sshd 3 0 02100000 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2

28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libutil.so.1

The -f option filters based on flags to the open(2) call, for example:

# ./opensnoop -e -f O_WRONLY -f O_RDWR

PID COMM FD ERR FLAGS PATH

28084 clear_console 3 0 00100002 /dev/tty

28084 clear_console -1 13 00100002 /dev/tty0

28084 clear_console -1 13 00100001 /dev/tty0

28084 clear_console -1 13 00100002 /dev/console

28084 clear_console -1 13 00100001 /dev/console

28051 sshd 8 0 02100002 /var/run/utmp

28051 sshd 7 0 00100001 /var/log/wtmp

The --cgroupmap option filters based on a cgroup set. It is meant to be used

with an externally created map.

# ./opensnoop --cgroupmap /sys/fs/bpf/test01

For more details, see docs/special_filtering.md

USAGE message:

# ./opensnoop -h

usage: opensnoop.py [-h] [-T] [-U] [-x] [-p PID] [-t TID]

[--cgroupmap CGROUPMAP] [--mntnsmap MNTNSMAP] [-u UID]

[-d DURATION] [-n NAME] [-e] [-f FLAG_FILTER]

Trace open() syscalls

optional arguments:

-h, --help show this help message and exit

-T, --timestamp include timestamp on output

-U, --print-uid include UID on output

-x, --failed only show failed opens

-p PID, --pid PID trace this PID only

-t TID, --tid TID trace this TID only

--cgroupmap CGROUPMAP

trace cgroups in this BPF map only

--mntnsmap MNTNSMAP trace mount namespaces in this BPF map on

-u UID, --uid UID trace this UID only

-d DURATION, --duration DURATION

total duration of trace in seconds

-n NAME, --name NAME only print process names containing this name

-e, --extended_fields

show extended fields

-f FLAG_FILTER, --flag_filter FLAG_FILTER

filter on flags argument (e.g., O_WRONLY)

examples:

./opensnoop # trace all open() syscalls

./opensnoop -T # include timestamps

./opensnoop -U # include UID

./opensnoop -x # only show failed opens

./opensnoop -p 181 # only trace PID 181

./opensnoop -t 123 # only trace TID 123

./opensnoop -u 1000 # only trace UID 1000

./opensnoop -d 10 # trace for 10 seconds only

./opensnoop -n main # only print process names containing "main"

./opensnoop -e # show extended fields

./opensnoop -f O_WRONLY -f O_RDWR # only print calls for writing

./opensnoop --cgroupmap mappath # only trace cgroups in this BPF map

./opensnoop --mntnsmap mappath # only trace mount namespaces in the map