In this exercise, you’ll run the CCA Key Broker demo inside a Realm and use kernel tracing to confirm that bounce buffers (SWIOTLB) are used for VirtIO network I/O. You will confirm bounce buffer usage using the Key Broker demo from the CCA Essentials Learning Path .
Pull the Docker container image with the pre-built KBS, and run the container:
docker pull armswdev/cca-learning-path:cca-key-broker-v2
docker run --rm -it armswdev/cca-learning-path:cca-key-broker-v2
List network interfaces:
ip -c a
The output should look like:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
Start the KBS on eth0 using the address from your output:
./keybroker-server -v --addr 172.17.0.2
The output should look like:
INFO starting 16 workers
INFO Actix runtime found; starting in Actix runtime
INFO starting service: "actix-web-service-172.17.0.2:8088", workers: 16, listening on: 172.17.0.2:8088
With the Key Broker Server running in one terminal, open up a new terminal in
which you will run the Key Broker Client (KBC). The intent is to confirm that the data transmitted over the network (through virtio_net) are indeed using bounce buffers.
Pull the Docker container image with the pre-built KBC, and then run the container:
docker pull armswdev/cca-learning-path:cca-simulation-v2
docker run --rm -it armswdev/cca-learning-path:cca-simulation-v2
Launch the Arm CCA prebuilt binaries on the FVP:
./run-cca-fvp.sh
The script connects to the FVP UARTs. When the host Linux kernel finishes booting, log in as root:
[ 4.169458] Run /sbin/init as init process
[ 4.273748] EXT4-fs (vda): re-mounted 64d1bcff-5d03-412c-83c6-48ec4253590e r/w. Quota mode: none.
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: [ 5.254843] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: broadcasting select for 172.20.51.1, server 172.20.51.254
udhcpc: lease of 172.20.51.1 obtained from 172.20.51.254, lease time 86400
deleting routers
adding dns 172.20.51.254
OK
Welcome to the CCA host
host login: root
(host) #
Change directory to /cca and launch a Realm guest with lkvm:
cd /cca
./lkvm run --realm --disable-sve --irqchip=gicv3-its --firmware KVMTOOL_EFI.fd -c 1 -m 512 --no-pvtime --disk guest-disk.img --restricted_mem --virtio-transport pci --pmu --network mode=user
You should see the realm boot. Note that lkvm is invoked with --network mode=user, which makes the guest see the network through a VirtIO device.
After boot, which might take some time, you will be prompted to log in at the guest Linux prompt. Use root again as the username:
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: broadcasting select for 192.168.33.15, server 192.168.33.1
udhcpc: lease of 192.168.33.15 obtained from 192.168.33.1, lease time 14400
deleting routers
adding dns 172.20.51.254
OK
Welcome to the CCA realm
realm login: root
(realm) #
Confirm that kernel tracing is available:
ls /sys/kernel/debug/tracing/events/
__output__9p i2c_slave qcom_glink
__output__alarmtimer icmp qcom_smp2p
__output__asoc initcall qdisc
__output__block interconnect ras
__output__bpf_test_run io_uring raw_syscalls
__output__bpf_trace iomap rcu
__output__bridge iommu regmap
__output__capability ipi regulator
__output__cgroup irq rpcgss
__output__chipidea jbd2 rpm
__output__clk kmem rpmh
__output__cma ksm rseq
__output__compaction kvm rtc
__output__cpuhp kyber sched
__output__cros_ec libata scmi
__output__csd lock scsi
__output__dev lockd signal
__output__devfreq maple_tree skb
__output__devlink mdio smbus
__output__dma memcg sock
__output__dma_fence migrate spi
__output__dpaa2_eth mmap spmi
__output__dpaa_eth mmap_lock sunrpc
__output__dwc3 mmc swiotlb
__output__e1000e_trace module task
__output__enable mtu3 tcp
__output__error_report musb tegra_apb_dma
__output__ext4 napi thermal
__output__fib neigh thermal_power_allocator
__output__filelock net thp
__output__filemap netfs timer
__output__fsl_edma netlink timer_migration
__output__ftrace nfs timestamp
__output__gadget nfs4 tlb
__output__gpio notifier udp
__output__gpu_mem oom ufs
__output__handshake optee vmalloc
__output__header_event page_isolation vmscan
__output__header_page page_pool watchdog
__output__hns3 pagemap workqueue
__output__huge_memory percpu writeback
__output__hugetlbfs power xdp
__output__hw_pressure printk xhci-hcd
__output__hwmon pwm
__output__i2c qcom_aoss
As shown above, you should see a list of the available trace points.
Now, enable the kernel tracing infrastructure together with bounce buffer tracing, read the trace in the background (filtering on keybroker-app-) and run the Key Broker Client application in the realm, using the endpoint address that the Key Broker Server is listening on (from the other terminal):
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 1 > /sys/kernel/debug/tracing/events/swiotlb/enable
grep keybroker-app- /sys/kernel/debug/tracing/trace_pipe &
keybroker-app -v --endpoint http://172.17.0.2:8088 skywalker
In the keybroker-app command above, skywalker is the key name that is requested from the KBS.
The output should look like:
INFO Requesting key named 'skywalker' from the keybroker server with URL http://172.17.0.2:8088/keys/v1/key/skywalker
INFO Challenge (64 bytes) = [5c, ec, 1e, f5, 93, 54, 4a, 8a, ee, 2e, 46, a0, 50, 0d, 41, dd, d4, 60, b0, 58, 5b, 51, 71, 76, d1, 66, d3, b7, 38, e8, af, ae, 0a, 07, 4e, c5, 60, dc, 4a, c0, b8, 73, 98, d9, bd, af, 41, 96, 99, 6d, 74, cc, 19, 70, 24, c4, c9, 5c, 21, 61, 1a, cb, 76, 75]
INFO Submitting evidence to URL http://172.17.0.2:8088/keys/v1/evidence/1928844131
INFO Attestation success :-) ! The key returned from the keybroker is 'May the force be with you.'
keybroker-app-143 [000] b..2. 1772.607321: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
keybroker-app-143 [000] b..2. 1772.644478: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
Note that the interleaving of the trace messages and KBC messages might differ from one run to another. The swiotlb_bounced events confirm that bounce buffers are being used in the Realm.