TinyComputers.io (Posts about ebpf)https://tinycomputers.io/categories/ebpf.atom2024-03-28T02:59:52ZA.C. JokelaNikolaBuilding a Kernel and Disk Image for the Radxa CM3https://tinycomputers.io/posts/building-a-kernel-and-disk-image-for-the-radxa-cm3.html2023-01-31T12:42:15-06:002023-01-31T12:42:15-06:00A.C. Jokela<p>With my eventual goal of testing out network and router capabilities of four compute modules that are pin compatible with the Raspberry Pi CM4, I have been doing setup work. My last few postings (<a href="https://tinycomputers.io/posts/polycube-network-on-arm-hardware-WIP.html">here</a>, <a href="https://tinycomputers.io/posts/polycube-on-arm-based-sbc-follow-up-1.html">here</a> and <a href="https://tinycomputers.io/posts/polycube-complete-installation-on-raspberry-pi-cm4.html">here</a>) on getting Polycube, a drop-in replacement for <code>iptables</code> and a number of other utilities that uses <a href="https://ebpf.io/">eBPF</a> instead of the usual <code>netfilter</code>-based mechanisms. The objective is to test out <code>netfilter</code>and <code>ebpf</code> routing on the four modules (giving me a collection of eight test sets).</p>
<p>I have Polycube compiled and appearing to function on the <a href="https://tinycomputers.io/posts/polycube-complete-installation-on-raspberry-pi-cm4.html">Raspberry Pi CM4</a>, the Pine64 SOQuartz module, and the code compiled and runnable on the Radxa CM3. There is one problem with running Polycube on the CM3: the SYSCALL for eBPF was not compiled into the kernel. Even though the code successfully compiled to an executable binary, the necessary kernel hooks are not present. The solution: compile a new kernel and create a new disk image.</p>
<p>If you are a person who is interested in tiny computers of various flavors, you will have noticed that there are an abundance of different distributions out on the internet. An example, for the Pine64 Quartz64 model A, there are at least three different variant distributions - <a href="https://github.com/Plebian-Linux/quartz64-images">Plebian Linux</a>, <a href="https://dietpi.com/#download">DietPI</a>, and <a href="https://users.armbian.com/balbes150/quartz64/">balbes150's Armbian</a>. They all have one thing in common, they all use Debian packages and are in one sense or another, a derivative of Debian and the Debian ecosystem. If you have used Ubuntu, you have used a distribution that leverages Debian architecture and infrastructure.</p>
<p>The available distributions for Radxa CM3 also use Debian ecosystem components; everything from being able to utilize other arm64 packages, to using the build infrastructure for bundling up things into a handy disk image that can be burned/written to media. </p>
<p>Many single board computer distributions are what is called a "board support package", or BSP for short. A BSP includes low level boot programs (a first stage bootloader, prebuilt binaries and Arm Trustzone Firmware) a boot program (a second stage bootloader , like u-boot or Tianocore EFI), an operating system and the compatible drivers for that are specific to the board. The BSP is a unique bundling of software that is specific to a given board or family of boards. Often times, the Linux kernel that is included with a given BSP has been modified and new drivers have been added. The kernel is essentially a fork and no longer tracks the "main branch" of Linux kernel development and any upstream changes in the main branch maybe difficult or impossible to incorporate. The kernel is, therefore, a snapshot in time that all too often fades into obscurity because of lack of attention from the developers or a broader community (if a community exists).</p>
<div style="width: 100%; text-align: center;">
<img src="https://tinycomputers.io/images/radxa-cm3-rpi-cm4-io-board.png.webp" loading="lazy" style="zoom: 75%">
</div>
<p>Despite not having the following and community backing like that of Raspberry Pi, Radxa does have well maintained series of BSP distributions. Many do have their kernels pegged to a specific version within the Linux kernel repository, but much of the userland software is not usually tied to specific features found in specific versions -- unless the software is something like Polycube.</p>
<p>Radxa does a great job of providing build frameworks for both configuring and compiling a new kernel, as well as downloading packages and building a disk image. Let's get started.</p>
<hr>
<p>The following information is based on <a href="https://wiki.radxa.com/Rock3/dev/Debian">this</a>.</p>
<ol>
<li>
<p>As a pregame note, I made a virtual machine using <a href="https://www.virtualbox.org/">VirtualBox</a>. Specifically, Debian 11 for build the new kernel in order to prevent any unnecessary contaminations of packages, dependencies or the like on my laptop. The building of the distribution image uses Docker and will not pose any issues.</p>
</li>
<li>
<p>Clone the Github repository <a href="https://github.com/radxa/rockchip-bsp">rockchip-bsp</a> and specifically the <code>stable-4.19-rock3</code> branch. The pull in any submodules.
</p><pre><code class="command">git clone -b stable-4.19-rock3 https://github.com/radxa/rockchip-bsp.git</code>
<code class="command">cd rockchip-bsp</code>
<code class="command">git submodule init</code>
<code class="command">git submodule update</code></pre>
<p>The <code>stable-4.19-rock3</code> branch has support for the following boards:</p>
<ul>
<li>ROCK 3A</li>
<li>ROCK 3B</li>
<li>Radxa CM3 IO</li>
<li>Radxa E23</li>
<li>Radxa E25</li>
<li>Radxa CM3 RASPCM4IO</li>
</ul>
<p>Cloning the repository and checking out the <code>stable-4.19-rock3</code> branch produces the following directories:</p>
<ul>
<li>build: <em>Some script files and configuration files for building u-boot, kernel and rootfs.</em></li>
<li>kernel: <em>kernel source code, current version is </em><em>4.19.193</em><em>.</em>.</li>
<li>rkbin: <em>Prebuilt Rockchip binaries, include first stage loader and <a href="https://github.com/ARM-software/arm-trusted-firmware">Arm TrustZone Firmware</a>.</em></li>
<li>rootfs: <em>Bootstrap a Debian based rootfs, support architecture <code>armhf</code> and <code>arm64</code>, </em><em>supports Debian Jessie, Stretch and Buster</em><em>.</em></li>
<li>u-boot: <em>u-boot as the second stage bootloader</em></li>
</ul>
<p>There are a few things to note. First, our kernel is version <a href="https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.193">4.19.193</a>. Polycube <a href="https://polycube-network.readthedocs.io/en/latest/installation.html#updating-linux-kernel">requires at minimum v4.15</a>. With v4.19, we are covered. Second, this repository/project contains scripts to bootstrap and build a disk image. We will not be using this functionality. The supported Debian distributions are too old. We have been using at least Debian <em>bullseye</em> for all of our Polycube testing.</p>
</li>
<li>
<p>Install a <a href="https://www.linaro.org/">Linaro</a> toolchain. This is used for compiling code on an x86/amd64 and producing arm64 binaries.<br>
<br>
</p><pre><code class="command">wget https://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz</code>
<code class="command">sudo tar xvf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz -C /usr/local/</code></pre>
<blockquote>
<p><em>Linaro has driven open source software development on Arm since 2010, providing the tools, Linux kernel quality and security needed for a solid foundation to innovate on. Linaro works with member companies and the open source community to maintain the Arm software ecosystem and enable new markets on Arm architecture.</em></p>
</blockquote>
</li>
<li>
<p>In your user's <code>.bashrc</code> file, append the following line:
</p><pre>export PATH="/usr/local/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin:$PATH"</pre>
Then <code>source</code> <code>.bashrc</code> to update your <code>PATH</code> variable.
<pre><code class="command">source ~/.bashrc</code></pre>
<p>Verify that the Linaro GCC toolchain is visable from your <code>PATH</code>
</p><pre><code class="command">which aarch64-linux-gnu-gcc</code>
/usr/local/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc</pre>
</li>
<li>
<p>Install a few packages:
<br>
</p><pre><code class="command">sudo apt-get install gcc-aarch64-linux-gnu \</code>
device-tree-compiler libncurses5 libncurses5-dev \
build-essential libssl-dev mtools bc python dosfstools</pre>
</li>
<li>
<p>Build u-boot for Radxa CM3 and specifically for use with a Raspberry Pi CM4 carrier/io board.
<br>
</p><pre><code class="command">./build/mk-uboot.sh rk3566-radxa-cm3-raspcm4io</code></pre>
<p>There should be files in <code>out/u-boot</code>
</p><pre><code class="command">ls -l out/u-boot</code>
total 2132
-rw-rw-r-- 1 alex alex 299008 Feb 1 22:43 idbloader.img
-rw-rw-r-- 1 alex alex 453056 Feb 1 22:43 rk356x_spl_loader_ddr1056_v1.10.111.bin
-rw-rw-r-- 1 alex alex 1426944 Feb 1 22:43 u-boot.itb</pre>
</li>
<li>
<p>Configure a new kernel.
If you have ever cloned the Linux source code repository or unarchived a tar-file of the source and then configured kernel and then compiled it, the following step will be familiar. The build process has been remarkably similar for better part of twenty-five years. I had not configured and compiled a kernel from source in a very long time; the kernel configuration process was remarkably familiar.
</p><pre><code class="command">cd kernel</code>
<code class="command">export ARCH=arm64</code>
<code class="command">export CROSS_COMPILE=aarch64-linux-gnu-</code>
<code class="command">make rockchip_linux_defconfig</code></pre>
<p>There will be a file named <code>.config</code>, you can either edit this by hand (if you have an idea of what you are doing and need to do) or you can use a handy menu-driven interface. Either way, for my specific needs of enabling eBPF, I simply opened <code>.config</code> in an editor, and searched for references to <code>BPF</code>. </p>
<p>If you want to try the menu-driven method, execute the following:
</p><pre><code class="command">make menuconfig</code></pre>
<p>Save your new configuration (run this whether you editted by hand or used <code>menuconfig</code>)
</p><pre><code class="command">make savedefconfig</code>
<code class="command">cp defconfig arch/arm64/configs/rockchip_linux_defconfig</code>
<code class="command">cd ..</code></pre>
<p></p><div style="width: 100%; text-align: center;"><img src="https://tinycomputers.io/images/compile-safedefconfig.gif" loading="lazy"><br></div>
</li>
<li>
<p>Build a kernel
</p><pre><code class="command">./build/mk-kernel.sh rk3566-radxa-cm3-raspcm4io</code></pre>
This will kick off the compilation of the kernel; obviously, depending upon your build machine, it might take a while.
<p>You will likely be presented with some configuration questions:</p>
<p></p><div style="width: 100%; text-align: center;"><img src="https://tinycomputers.io/images/building-radxa-cm3-kernel-first-time.png.webp" loading="lazy"></div>
<p>Give that I am not entirely versed in things-kernel, I answered <code>y</code> to all of the questions. Leave a comment below if you have some insight into the questions that are presented during the build process.</p>
</li>
<li>
<p>Pack up your new kernel and associated headers into Debian package files (e.g. <code>.deb</code>). The parameters for <code>pack-kernel.sh</code> are: 1) the name of the kernel configuration file (from step #7); 2) <code>ebpf</code> is a release value, it should be something useful.
</p><pre><code class="command">./build/pack-kernel.sh -d rockchip_linux_defconfig -r ebpf</code></pre>
This will compile the kernel, again, but this appears to be necessary because this steps does not configure the appropriate chip and board as in the previous step.
<p></p><div style="width: 100%; text-align: center;"><img src="https://tinycomputers.io/images/building-radxa-cm3-kernel-package.png.webp" loading="lazy"></div>
<p></p><pre><code class="command">ls out/packages/</code>
linux-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.changes
linux-headers-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.deb
linux-image-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.deb
linux-image-4.19.193-ebpf-rockchip-g67a0c0ce87a0-dbg_4.19.193-ebpf-rockchip_arm64.deb
linux-libc-dev_4.19.193-ebpf-rockchip_arm64.deb</pre>
<p><strong>These Debian packages will be needed when we build a Debian <code>bullseye</code> distribution.</strong></p>
</li>
<li>
<p>You will also need to copy <code>rk3566-radxa-cm3-rpi-cm4-io.dtb</code> from <code>out/kernel</code> directory; this device table is needed when writing a new disk image to the CM3.</p>
<p>If you do want to assemble an older distribution (Debian buster or stretch), you can follow steps for <em>Make rootfs image</em> found <a href="https://wiki.radxa.com/Rock3/dev/Debian">here</a>.
I have a pre-built <em>Debian buster with Desktop</em> disk image available <a href="https://cdn.tinycomputers.io/radxa-rock3/debian-buster-linux-4.19.193-2a-eBPF-rockchip-rk3566-radxa-cm3-rpicm4io.img.xz">here</a></p>
</li>
<li>
<p>Change directories to place outside of the <code>rockchip-bsp</code> directory, and now, clone the Radxa <code>rbuild</code> tool
</p><pre><code class="command">git clone https://github.com/radxa-repo/rbuild.git</code></pre>
<p>You will need <code>docker</code> and associated software packages. Installing these tools should be straightforward and there are dozens if not hundreds of howtos widely available to assist you. If you do not have <code>docker</code> command line tools installed and you looking for a quick guide, follow <a href="https://docs.docker.com/engine/install/ubuntu/">these instructions</a> before proceding.</p>
</li>
<li>
<p>Make a directory for your kernel packages; copy kernel packages
</p><pre><code class="command">cd rbuild</code></pre>
Make a directory for the kernel packages; I will be using <code>docker</code> outside of the virtual machine that I used to build the kernel packages. You are free to use the VM for building the <code>bullseye</code> disk image, I ran into issues and decided to use my laptop to directly use <code>docker</code>. I used <code>scp</code> to copy the kernel packages from the VM into a directory named <code>kernel</code> that is in the <code>rbuild</code> directory containing the cloned repo.
</li>
<li>
<p>Run <code>rbuild</code> to magically assemble a disk image for you; this will take a while, best to grab some coffee, or lunch, or just go home for the day. There is also a strong chance of having network timeouts while downloading necessary files. I ended up having at least five times where a package download failed and killed the whole build process. On a my Dell XPS Developer Edition laptop, in a VirtualBox VM, the process took over eight hours. It should be noted that even if there is a timeout, by specifying the <code>-r</code> parameter to <code>rbuild</code>, this is caching the necessary Debian packages.
</p><pre><code class="command">./rbuild -r -k kernel/linux-image-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.deb radxa-cm3-rpi-cm4-io cli</code></pre>
<p></p><div style="width: 100%; text-align: center;"><img src="https://tinycomputers.io/images/building-radxa-bullseye-distro.png.webp" loading="lazy"></div>
<p></p><pre><code class="command">ls -l</code>
total 1434692
-rw-rw-r-- 1 alex alex 3322 Feb 1 22:17 action.yaml
drwxrwxr-x 6 alex alex 4096 Feb 2 09:38 common
drwxrwxr-x 2 alex alex 4096 Feb 1 22:17 configs
drwxrwxr-x 2 alex alex 4096 Feb 1 22:38 kernel
-rw-r--r-- 1 alex alex 6442450944 Feb 2 11:48 radxa-cm3-rpi-cm4-io_debian_bullseye_cli.img
-rw-rw-r-- 1 alex alex 175 Feb 2 11:48 radxa-cm3-rpi-cm4-io_debian_bullseye_cli.img.sha512
-rwxrwxr-x 1 alex alex 18869 Feb 1 22:17 rbuild
-rw-rw-r-- 1 alex alex 1542 Feb 1 22:17 README.md</pre>
</li>
<li>
<p>And there we have it. <code>radxa-cm3-rpi-cm4-io_debian_bullseye_cli.img</code> is your new disk image, complete with a custom compiled kernel with eBPF enabled. We can compress the disk image with <code>xz</code> to make it more manageable.
</p><pre><code class="command">xz -z -v radxa-cm3-rpi-cm4-io_debian_bullseye_cli.img
radxa-cm3-rpi-cm4-io_debian_bullseye_cli.img (1/1)
3.0 % 5,938.2 KiB / 185.7 MiB = 0.031 10 MiB/s 0:18 9 min 50 s</code></pre>
</li>
<li>
<p>You can download the kernel and disk image that was built during the writing of this post:
<a href="https://cdn.tinycomputers.io/radxa-rock3/debian-buster-linux-4.19.193-2a-eBPF-rockchip-rk3566-radxa-cm3-rpicm4io.img.xz">https://cdn.tinycomputers.io/radxa-rock3/debian-buster-linux-4.19.193-2a-eBPF-rockchip-rk3566-radxa-cm3-rpicm4io.img.xz</a></p>
<p>The Device Table file built during the writing of this post:
<a href="https://cdn.tinycomputers.io/radxa-rock3/linux-image-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.dtb">https://cdn.tinycomputers.io/radxa-rock3/linux-image-4.19.193-ebpf-rockchip-g67a0c0ce87a0_4.19.193-ebpf-rockchip_arm64.dtb</a></p>
</li>
<li>
<p>Instructions on writing the disk image to eMMC on the Radxa CM3, you can follow the instructions on my previous post, <a href="https://tinycomputers.io/posts/posts/raspberry-pi-cm4-and-pin-compatible-modules.html">Raspberry Pi CM4 and Pin Compatible Modules</a></p>
</li>
</ol>
<p>More Information on Radxa's build scripts, <a href="https://radxa-doc.github.io/quick-start/start.html">rbuild documentation</a> and its <a href="https://github.com/radxa-repo/rbuild">github repo</a></p>
<script type="text/javascript">
amzn_assoc_placement = "adunit0";
amzn_assoc_tracking_id = "tinycompute05-20";
amzn_assoc_ad_mode = "manual";
amzn_assoc_ad_type = "smart";
amzn_assoc_marketplace = "amazon";
amzn_assoc_region = "US";
amzn_assoc_title = "Tiny Computers Picks";
amzn_assoc_linkid = "57e3849d20157bd2e23384360758ad44";
amzn_assoc_rows = "4";
amzn_assoc_design = "text_links";
amzn_assoc_asins = "B0BG5CFFZZ,B0BGJ2HHRZ,B0BLN7TYH6,B0B7QMX8RJ";
</script>
<script src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US"></script>Polycube - Complete Installation on Raspberry Pi CM4https://tinycomputers.io/posts/polycube-complete-installation-on-raspberry-pi-cm4.html2023-01-27T10:51:05-06:002023-01-27T10:51:05-06:00A.C. Jokela<ol>
<li>
<p>Adding backports and stretch package locations to /etc/apt/source.list
<br>
</p><pre>deb https://deb.debian.org/debian bullseye-backports main contrib non-free
deb https://deb.debian.org/debian/ stretch-backports main contrib non-free
deb https://deb.debian.org/debian/ stretch main contrib non-free</pre>
</li>
<li>
<p>Update local cache
<br>
</p><pre><code class="command">sudo apt update</code></pre>
</li>
<li>
<p>Install packages
<br>
</p><pre><code class="command">sudo apt-get -y install git build-essential cmake bison flex \</code>
libelf-dev libllvm9 llvm-9-dev libclang-9-dev libpcap-dev \
libnl-route-3-dev libnl-genl-3-dev uuid-dev pkg-config \
autoconf libtool m4 automake libssl-dev kmod jq bash-completion \
gnupg2 golang-go-1.19 tmux bc libfl-dev libpcre2-dev libpcre3-dev</pre>
</li>
<li>
<p>Add <code>go</code> to your <code>$PATH</code> in <code>.bashrc</code>; this needs to be done for <code>root</code> user, as well.<br>
<br>
</p><pre>export PATH=/usr/lib/go-1.19/bin:$PATH</pre>
</li>
<li>
<p>Verify <code>go</code> is in <code>$PATH</code>
<br>
</p><pre><code class="command">go version</code>
<code>go version go1.19.4 linux/arm64</code></pre>
</li>
<li>
<p>Install <code>pistache</code> - needed for the RESTful control daemon, <code>polycubed</code>
<br>
</p><pre><code class="command">git clone https://github.com/oktal/pistache.git</code>
<code class="command">cd pistache</code>
# known working version of pistache
<code class="command">git checkout 117db02eda9d63935193ad98be813987f6c32b33</code>
<code class="command">git submodule update --init</code>
<code class="command">mkdir -p build && cd build</code>
<code class="command">cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DPISTACHE_USE_SSL=ON ..</code>
<code class="command">make -j $(getconf _NPROCESSORS_ONLN)</code>
<code class="command">sudo make install</code></pre>
</li>
<li>
<p>Install <code>libtins</code>
<br>
</p><pre><code class="command">cd
<code class="command">git clone --branch v3.5 https://github.com/mfontanini/libtins.git</code>
<code class="command">cd libtins</code>
<code class="command">mkdir -p build && cd build</code>
<code class="command">cmake -DLIBTINS_ENABLE_CXX11=1 \</code>
-DLIBTINS_BUILD_EXAMPLES=OFF -DLIBTINS_BUILD_TESTS=OFF \
-DLIBTINS_ENABLE_DOT11=OFF -DLIBTINS_ENABLE_PCAP=OFF \
-DLIBTINS_ENABLE_WPA2=OFF -DLIBTINS_ENABLE_WPA2_CALLBACKS=OFF ..
<code class="command">make -j $(getconf _NPROCESSORS_ONLN)</code>
<code class="command">sudo make install</code>
<code class="command">sudo ldconfig</code></code></pre>
</li>
<li>
<p>Install <code>libyang</code>
<br>
</p><pre><code class="command">cd</code>
<code class="command">git clone https://github.com/CESNET/libyang.git</code>
<code class="command">cd libyang</code>
<code class="command">git checkout libyang1</code>
<code class="command">mkdir build; cd build</code>
<code class="command">cmake ..</code>
<code class="command">make</code>
<code class="command">sudo make install</code></pre>
</li>
<li>
<p>Clone <code>polycube</code> repository that contains the necessary changes to <code>config.cpp</code>
<br>
</p><pre><code class="command">cd</code>
<code class="command">git clone https://github.com/ajokela/polycube</code>
<code class="command">cd polycube</code>
<code class="command">git submodule update --init --recursive</code></pre>
</li>
<li>
<p>Build <code>prometheus-cpp</code>
</p><pre><code class="command">cd src/libs/prometheus-cpp</code>
<code class="command">mkdir build; cd build</code>
<code class="command">cmake .. -DBUILD_SHARED_LIBS=ON</code>
<code class="command">make -j4</code>
<code class="command">sudo make install</code></pre>
</li>
<li>Configure <code>polycube</code>
<pre><code class="command">cd; cd polycube</code>
<code class="command">mkdir build; cd build</code>
<code class="command">cmake .. -DBUILD_SHARED_LIBS=ON \</code>
-DENABLE_PCN_IPTABLES=ON \
-DENABLE_SERVICE_BRIDGE=ON \
-DENABLE_SERVICE_DDOSMITIGATOR=OFF \
-DENABLE_SERVICE_FIREWALL=ON \
-DENABLE_SERVICE_HELLOWORLD=OFF \
-DENABLE_SERVICE_IPTABLES=ON \
-DENABLE_SERVICE_K8SFILTER=OFF \
-DENABLE_SERVICE_K8SWITCH=OFF \
-DENABLE_SERVICE_LBDSR=OFF \
-DENABLE_SERVICE_LBRP=OFF \
-DENABLE_SERVICE_NAT=ON \
-DENABLE_SERVICE_PBFORWARDER=OFF \
-DENABLE_SERVICE_ROUTER=ON \
-DENABLE_SERVICE_SIMPLEBRIDGE=ON \
-DENABLE_SERVICE_SIMPLEFORWARDER=ON \
-DENABLE_SERVICE_TRANSPARENTHELLOWORLD=OFF \
-DENABLE_SERVICE_SYNFLOOD=OFF \
-DENABLE_SERVICE_PACKETCAPTURE=OFF \
-DENABLE_SERVICE_K8SDISPATCHER=OFF
</pre></li>
<li>
<p>Build <code>polycube</code> (this will take a while; you might want to use <code>tmux</code>)
</p><pre><code class="command">tmux</code>
<code class="command">make -j4</code></pre>
To detach from the <code>tmux</code> terminal, press <code>CTL+b, d</code><br><br>
<p>To reattached, execute:
</p><pre><code class="command">tmux attach -t 0</code></pre><br><br>
<p>Grab a coffee and go stare at your phone for a while.</p>
</li>
<li>
<p>If all goes well, you should see the following:
</p><pre>[100%] Building CXX object src/polycubed/src/CMakeFiles/polycubed.dir/load_services.cpp.o
[100%] Building CXX object src/polycubed/src/CMakeFiles/polycubed.dir/version.cpp.o
[100%] Linking CXX executable polycubed
[100%] Built target polycubed</pre>
</li>
<li>Try to execute <code>polycubed</code>; we should get some sort of error(s)
<pre><code class="command">sudo src/polycubed/src/polycubed</code>
[2023-01-27 13:57:22.022] [polycubed] [info] loading configuration from /etc/polycube/polycubed.conf
[2023-01-27 13:57:22.023] [polycubed] [warning] default configfile (/etc/polycube/polycubed.conf) not found, creating a new with default parameters
terminate called after throwing an instance of 'spdlog::spdlog_ex'
what(): Failed opening file /var/log/polycube/polycubed.log for writing: No such file or directory
Aborted
</pre></li>
<li>This is progress and we can handle this by making a directory.
<pre><code class="command">sudo mkdir /var/log/polycube</code></pre></li>
<li>Run <code>polycubed</code> again, and you should run into kernel header files not being foundation
<pre><code class="command">sudo src/polycubed/src/polycubed </code>
[2023-01-27 14:01:21.048] [polycubed] [info] loading configuration from /etc/polycube/polycubed.conf
[2023-01-27 14:01:21.051] [polycubed] [info] configuration parameters:
[2023-01-27 14:01:21.051] [polycubed] [info] loglevel: info
[2023-01-27 14:01:21.051] [polycubed] [info] daemon: false
[2023-01-27 14:01:21.052] [polycubed] [info] pidfile: /var/run/polycube.pid
[2023-01-27 14:01:21.052] [polycubed] [info] port: 9000
[2023-01-27 14:01:21.052] [polycubed] [info] addr: localhost
[2023-01-27 14:01:21.052] [polycubed] [info] logfile: /var/log/polycube/polycubed.log
[2023-01-27 14:01:21.052] [polycubed] [info] cubes-dump-file: /etc/polycube/cubes.yaml
[2023-01-27 14:01:21.052] [polycubed] [info] cubes-dump-clean-init: false
[2023-01-27 14:01:21.052] [polycubed] [info] cubes-dump-enable: false
[2023-01-27 14:01:21.052] [polycubed] [info] polycubed starting...
[2023-01-27 14:01:21.052] [polycubed] [info] version v0.9.0+ [git: (branch/commit): master/75da2773]
<b>modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.61-v8+
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) or installing the kernel development package for your running kernel version.</b>
chdir(/lib/modules/5.15.61-v8+/build): No such file or directory
[2023-01-27 14:01:21.092] [polycubed] [error] error creating patch panel: Unable to initialize BPF program
[2023-01-27 14:01:21.093] [polycubed] [critical] Error starting polycube: Error creating patch panel</pre></li>
<li>We need to get the Raspberry Pi Linux kernel source.
<pre><code class="command">cd /usr/src</code>
<code class="command">sudo git clone --depth=1 https://github.com/raspberrypi/linux</code></pre>
We need to see what kernel version the Raspberry Pi
<pre><code class="command">uname -a</code>
Linux polycube-network 5.15.61-v8+ #1579 SMP PREEMPT Fri Aug 26 11:16:44 BST 2022 aarch64 GNU/Linux</pre>
We are using 5.15.61-v8+. We will need to checkout the correct branch of the kernel. First move <code>linux</code> to <code>linux-upstream-5.15.61-v8+</code>
<pre><code class="command">sudo mv linux linux-upstream-5.15.89-v8+</code>
<code class="command">cd linux-upstream-5.15.89-v8+</code></pre>
Now, checkout the correct branch. It takes a format like <code>rpi-5.15.y</code> which corresponds to version <code>5.15.89</code>
<pre><code class="command">sudo git checkout rpi-5.15.y</code></pre></li>
<li>Make a symlink from within <code>/lib/modules</code> to our source directory
<pre><code class="command">cd /lib/modules/5.15.61-v8+</code>
<code class="command">sudo ln -s /usr/src/linux-upstream-5.15.89-v8+ build</code></pre></li>
<li>
<p>Build a new kernel to auto-generate the necessary header files.
</p><pre><code class="command">cd /usr/src/linux-upstream-5.15.89-v8+</code>
<code class="command">sudo make ARCH=arm64 bcm2711_defconfig</code>
<code class="command">sudo make -j4</code></pre>
Grab another cup of coffee and stare at your phone for a while; this will take some time to complete. Doing this <em>in situ</em> will be slower than cross-compiling on a faster laptop or desktop, but the point of these instructions is not to productionize the process, it is to show how to make <code>polycube</code> compile and run on Arm-based systems, specifically, Raspberry Pi 4b or CM4 systems.
<p>It might be unnecessary to completely build recompile a kernel; maybe experiment a bit with it.
19. Installing and running. Make sure <code>go</code> is available in your <code>PATH</code> for <code>root</code> user; it is needed to compile <code>polycubectl</code>.
</p><pre><code class="command">sudo su -</code>
<code class="command">cd ~pi/polycube/build
<code class="command">make install</code></code></pre>
<code>make install</code> should finished a message of <code>Installation completed successfully</code>. Now we can run <code>polycubed</code> and it will find all the associated shared libraries for the functionality we will be investigating in the next post.
<pre><code class="command">sudo polycubed</code></pre>
You should get output that looks like this:
<pre>[2023-01-27 17:13:46.791] [polycubed] [info] loading configuration from /etc/polycube/polycubed.conf
[2023-01-27 17:13:46.793] [polycubed] [info] configuration parameters:
[2023-01-27 17:13:46.793] [polycubed] [info] loglevel: info
[2023-01-27 17:13:46.793] [polycubed] [info] daemon: false
[2023-01-27 17:13:46.793] [polycubed] [info] pidfile: /var/run/polycube.pid
[2023-01-27 17:13:46.793] [polycubed] [info] port: 9000
[2023-01-27 17:13:46.793] [polycubed] [info] addr: localhost
[2023-01-27 17:13:46.793] [polycubed] [info] logfile: /var/log/polycube/polycubed.log
[2023-01-27 17:13:46.794] [polycubed] [info] cubes-dump-file: /etc/polycube/cubes.yaml
[2023-01-27 17:13:46.794] [polycubed] [info] cubes-dump-clean-init: false
[2023-01-27 17:13:46.794] [polycubed] [info] cubes-dump-enable: false
[2023-01-27 17:13:46.794] [polycubed] [info] polycubed starting...
[2023-01-27 17:13:46.794] [polycubed] [info] version v0.9.0+ [git: (branch/commit): master/75da2773]
prog tag mismatch 3e70ec38a5f6710 1
WARNING: cannot get prog tag, ignore saving source with program tag
prog tag mismatch 1e2ac42799daebd8 1
WARNING: cannot get prog tag, ignore saving source with program tag
[2023-01-27 17:14:03.905] [polycubed] [info] rest server listening on '127.0.0.1:9000'
[2023-01-27 17:14:03.906] [polycubed] [info] rest server starting ...
[2023-01-27 17:14:04.010] [polycubed] [info] service bridge loaded using libpcn-bridge.so
[2023-01-27 17:14:04.050] [polycubed] [info] service firewall loaded using libpcn-firewall.so
[2023-01-27 17:14:04.149] [polycubed] [info] service nat loaded using libpcn-nat.so
[2023-01-27 17:14:04.277] [polycubed] [info] service router loaded using libpcn-router.so
[2023-01-27 17:14:04.340] [polycubed] [info] service simplebridge loaded using libpcn-simplebridge.so
[2023-01-27 17:14:04.370] [polycubed] [info] service simpleforwarder loaded using libpcn-simpleforwarder.so
[2023-01-27 17:14:04.413] [polycubed] [info] service iptables loaded using libpcn-iptables.so
[2023-01-27 17:14:04.553] [polycubed] [info] service dynmon loaded using libpcn-dynmon.so
[2023-01-27 17:14:04.554] [polycubed] [info] loading metrics from yang files
</pre>
</li>
</ol>
<div style="width: 100%; text-align: center; vertical-align: top;">
<img src="https://tinycomputers.io/images/surrealism-painting-raspberry-pi-ethernet-router.png.webp" loading="lazy" style="zoom: 45%;">
</div>
<hr>Polycube on Arm-based SBC: Follow-up #2 (WIP)https://tinycomputers.io/posts/polycube-on-arm-based-sbc-follow-up-2-wip.html2023-01-26T11:46:08-06:002023-01-26T11:46:08-06:00A.C. Jokela<p>After emailing three of the committers to the original Polycube project, and receiving short replies from each of that basically said, <em>polycube was never tested on an arm-based system will likely not work without significant efforts</em> as well as, <em>I believe the [polycube] project is no longer active</em>, I wanted to follow through and test the former statement and really see how much effort would it take to get a compiled binary of <code>polycubed</code> running on an Arm-based system.</p>
<p>With my <a href="https://tinycomputers.io/posts/polycube-on-arm-based-sbc-follow-up-1.html">previous Work In Progress</a>, I appeared to be able to successfully build and compile an executable, but when run, the program did nothing but consume 100% of one core of the Raspberry Pi's processes.</p>
<p>What does this mean? A hung process, consuming 100% of one core; that feels to me like it is getting stuck in a loop without having an exit/break condition met. I started by doing what any ham-handed developer would do: I started at <code>main()</code> in <code>polycubed.cpp</code> and started to put <code>std::cerr << "Code gets to this spot #1" << std:endl;</code> into the code.</p>
<p>I narrowed this initial issue of the process hang to the following:</p>
<div class="code"><pre class="code literal-block"><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">config</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">exit</span><span class="p">(</span><span class="n">EXIT_SUCCESS</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cerr</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"Configs loaded..."</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">exception</span><span class="w"> </span><span class="o">&</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// The problem of the error in loading the config file may be due to</span>
<span class="w"> </span><span class="c1">// polycubed executed as normal user</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">getuid</span><span class="p">())</span>
<span class="w"> </span><span class="n">logger</span><span class="o">-></span><span class="n">critical</span><span class="p">(</span><span class="s">"polycubed should be executed with root privileges"</span><span class="p">);</span>
<span class="w"> </span><span class="n">logger</span><span class="o">-></span><span class="n">critical</span><span class="p">(</span><span class="s">"Error loading config: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">what</span><span class="p">());</span>
<span class="w"> </span><span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Both of the <code>cerr</code> statements that I added were never getting called. This narrowed down the issue to <code>config.load(argc, argv)</code>.</p>
<p>Looking at <a href="https://github.com/polycube-network/polycube/blob/v0.9.0/src/polycubed/src/config.cpp"><code>config.cpp</code></a> and specifically at the method, <code>load(int argc, char *argv[])</code>, you will find the following:</p>
<div class="code"><pre class="code literal-block"><span class="kt">bool</span><span class="w"> </span><span class="nf">Config::load</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">logger</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">spdlog</span><span class="o">::</span><span class="n">get</span><span class="p">(</span><span class="s">"polycubed"</span><span class="p">);</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">option_index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">ch</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// do a first pass looking for "configfile", "-h", "-v"</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">((</span><span class="n">ch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">getopt_long</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">,</span><span class="w"> </span><span class="s">"l:p:a:dhv"</span><span class="p">,</span><span class="w"> </span><span class="n">options</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">option_index</span><span class="p">))</span><span class="w"> </span><span class="o">!=</span>
<span class="w"> </span><span class="mi">-1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">ch</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'v'</span><span class="p">:</span>
<span class="w"> </span><span class="n">show_version</span><span class="p">();</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'h'</span><span class="p">:</span>
<span class="w"> </span><span class="n">show_usage</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">4</span><span class="p">:</span>
<span class="w"> </span><span class="n">configfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">optarg</span><span class="p">;</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">load_from_file</span><span class="p">(</span><span class="n">configfile</span><span class="p">);</span>
<span class="w"> </span><span class="n">load_from_cli</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">);</span>
<span class="w"> </span><span class="n">check</span><span class="p">();</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">cubes_dump_clean_init</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">ofstream</span><span class="w"> </span><span class="n">output</span><span class="p">(</span><span class="n">cubes_dump_file</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">output</span><span class="p">.</span><span class="n">is_open</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"{}"</span><span class="p">;</span>
<span class="w"> </span><span class="n">output</span><span class="p">.</span><span class="n">close</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">true</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Through some amateur debugging statements, I determined that <code>while (( ch = getopt_long..) != -1)</code> was never ceasing. The <code>while</code> loop never exited. Why would this statement work flawlessly on Intel amd64-based systems and not on Arm64 systems? I am still stumped as why it would matter. However, implementing the <code>while</code> look as the following got me slightly further in the start-up process:</p>
<div class="code"><pre class="code literal-block"><span class="w"> </span><span class="k">while</span><span class="p">(</span><span class="nb">true</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">ch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">getopt_long</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">,</span><span class="w"> </span><span class="s">"l:p:a:dhv"</span><span class="p">,</span><span class="w"> </span><span class="n">options</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">option_index</span><span class="p">);</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">ch</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'v'</span><span class="p">:</span>
<span class="w"> </span><span class="n">show_version</span><span class="p">();</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'h'</span><span class="p">:</span>
<span class="w"> </span><span class="n">show_usage</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">4</span><span class="p">:</span>
<span class="w"> </span><span class="n">configfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">optarg</span><span class="p">;</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="mi">-1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">ch</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
</pre></div>
<p>Maybe someone with more systems experience and C++ knowledge might have an idea as to why these two blocks of code behave differently when run on different architectures.</p>
<p>Anyway, being able to get a little farther into the start-up process was a sign I should keep looking into the issue. Using my <a href="https://www.merriam-webster.com/dictionary/bush-league">Bush-league</a> skills of debugging (e.g. liberal use of <code>std::cerr</code>), I determined that things were getting bound up on:</p>
<div class="code"><pre class="code literal-block"><span class="n">load_from_cli</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">);</span>
</pre></div>
<p>A look at that method reveals another, similar, <code>while</code> statement:</p>
<div class="code"><pre class="code literal-block"><span class="kt">void</span><span class="w"> </span><span class="nf">Config::load_from_cli</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">option_index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">ch</span><span class="p">;</span>
<span class="w"> </span><span class="n">optind</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">((</span><span class="n">ch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">getopt_long</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">,</span><span class="w"> </span><span class="s">"l:p:a:dhv"</span><span class="p">,</span><span class="w"> </span><span class="n">options</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">option_index</span><span class="p">))</span><span class="w"> </span><span class="o">!=</span>
<span class="w"> </span><span class="mi">-1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">ch</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'l'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setLogLevel</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'p'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setServerPort</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'d'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setDaemon</span><span class="p">(</span><span class="n">optarg</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">(</span><span class="n">optarg</span><span class="p">)</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">"true"</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'a'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setServerIP</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'c'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCertPath</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'k'</span><span class="p">:</span>
<span class="w"> </span><span class="n">setKeyPath</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="sc">'?'</span><span class="p">:</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">runtime_error</span><span class="p">(</span><span class="s">"Missing argument, see stderr"</span><span class="p">);</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">1</span><span class="p">:</span>
<span class="w"> </span><span class="n">setLogFile</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">2</span><span class="p">:</span>
<span class="w"> </span><span class="n">setPidFile</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">5</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCACertPath</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">6</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCertWhitelistPath</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">7</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCertBlacklistPath</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">8</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCubesDumpFile</span><span class="p">(</span><span class="n">optarg</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">9</span><span class="p">:</span>
<span class="w"> </span><span class="n">setCubesDumpCleanInit</span><span class="p">();</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">10</span><span class="p">:</span>
<span class="w"> </span><span class="c1">//setCubesNoDump();</span>
<span class="w"> </span><span class="n">setCubesDumpEnabled</span><span class="p">();</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Again, I determined that <code>while (( ch = getopt_long..) != -1)</code> was never breaking from the <code>while</code> loop. Changing it to:</p>
<div class="code"><pre class="code literal-block"><span class="w"> </span><span class="k">while</span><span class="p">(</span><span class="nb">true</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">ch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">getopt_long</span><span class="p">(</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="n">argv</span><span class="p">,</span><span class="w"> </span><span class="s">"l:p:a:dhv"</span><span class="p">,</span><span class="w"> </span><span class="n">options</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">option_index</span><span class="p">);</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="mi">-1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">ch</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
</pre></div>
<p>This did the trick, as it had done with the previous <code>while</code> loop. I was able to execute <code>polycubed</code> but ran into a new error:</p>
<div class="code"><pre class="code literal-block"><span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>configuration<span class="w"> </span>parameters:
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>loglevel:<span class="w"> </span>info
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>daemon:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>pidfile:<span class="w"> </span>/var/run/polycube.pid
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>port:<span class="w"> </span><span class="m">9000</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>addr:<span class="w"> </span>localhost
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>logfile:<span class="w"> </span>/var/log/polycube/polycubed.log
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.131<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-file:<span class="w"> </span>/etc/polycube/cubes.yaml
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.132<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-clean-init:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.132<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-enable:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.132<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>polycubed<span class="w"> </span>starting...
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.132<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>version<span class="w"> </span>v0.9.0
modprobe:<span class="w"> </span>FATAL:<span class="w"> </span>Module<span class="w"> </span>kheaders<span class="w"> </span>not<span class="w"> </span>found<span class="w"> </span><span class="k">in</span><span class="w"> </span>directory<span class="w"> </span>/lib/modules/5.15.84-v8+
Unable<span class="w"> </span>to<span class="w"> </span>find<span class="w"> </span>kernel<span class="w"> </span>headers.<span class="w"> </span>Try<span class="w"> </span>rebuilding<span class="w"> </span>kernel<span class="w"> </span>with<span class="w"> </span><span class="nv">CONFIG_IKHEADERS</span><span class="o">=</span>m<span class="w"> </span><span class="o">(</span>module<span class="o">)</span>
chdir<span class="o">(</span>/lib/modules/5.15.84-v8+/build<span class="o">)</span>:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.180<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>error<span class="o">]</span><span class="w"> </span>error<span class="w"> </span>creating<span class="w"> </span>patch<span class="w"> </span>panel:<span class="w"> </span>Unable<span class="w"> </span>to<span class="w"> </span>initialize<span class="w"> </span>BPF<span class="w"> </span>program
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:25:19.188<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>critical<span class="o">]</span><span class="w"> </span>Error<span class="w"> </span>starting<span class="w"> </span>polycube:<span class="w"> </span>Error<span class="w"> </span>creating<span class="w"> </span>patch<span class="w"> </span>panel
</pre></div>
<p>Next, I grabbed the linux kernel source from Raspberry Pi's github and setup a symlink for <code>polycubed</code> to find kernel headers:</p>
<pre><code class="command">git clone --depth=1 https://github.com/raspberrypi/linux.git</code>
<code class="command">mv linux linux-upstream-5.15.89-v8+</code>
<code class="command">sudo ln -s /usr/src/linux-upstream-5.15.89-v8+ /lib/modules/5.15.89-v8+/build</code>
<code class="command">sudo ~/polycube/build/src/polycubed/src/polycubed</code></pre>
<p>This results in:</p>
<div class="code"><pre class="code literal-block"><span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.035<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>configuration<span class="w"> </span>parameters:
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.035<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>loglevel:<span class="w"> </span>trace
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.035<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>daemon:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>pidfile:<span class="w"> </span>/var/run/polycube.pid
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>port:<span class="w"> </span><span class="m">9000</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>addr:<span class="w"> </span>localhost
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>logfile:<span class="w"> </span>/var/log/polycube/polycubed.log
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-file:<span class="w"> </span>/etc/polycube/cubes.yaml
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-clean-init:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-enable:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>polycubed<span class="w"> </span>starting...
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:19.036<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>version<span class="w"> </span>v0.9.0
bpf:<span class="w"> </span>Failed<span class="w"> </span>to<span class="w"> </span>load<span class="w"> </span>program:<span class="w"> </span>Invalid<span class="w"> </span>argument
jump<span class="w"> </span>out<span class="w"> </span>of<span class="w"> </span>range<span class="w"> </span>from<span class="w"> </span>insn<span class="w"> </span><span class="m">9</span><span class="w"> </span>to<span class="w"> </span><span class="m">37</span>
processed<span class="w"> </span><span class="m">0</span><span class="w"> </span>insns<span class="w"> </span><span class="o">(</span>limit<span class="w"> </span><span class="m">1000000</span><span class="o">)</span><span class="w"> </span>max_states_per_insn<span class="w"> </span><span class="m">0</span><span class="w"> </span>total_states<span class="w"> </span><span class="m">0</span><span class="w"> </span>peak_states<span class="w"> </span><span class="m">0</span><span class="w"> </span>mark_read<span class="w"> </span><span class="m">0</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:46.751<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>error<span class="o">]</span><span class="w"> </span>cannot<span class="w"> </span>load<span class="w"> </span>ctrl_rx:<span class="w"> </span>Failed<span class="w"> </span>to<span class="w"> </span>load<span class="w"> </span>controller_module_rx:<span class="w"> </span>-1
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">15</span>:40:46.800<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>critical<span class="o">]</span><span class="w"> </span>Error<span class="w"> </span>starting<span class="w"> </span>polycube:<span class="w"> </span>cannot<span class="w"> </span>load<span class="w"> </span>controller_module_rx
</pre></div>
<p>It is entirely possible that I am including the wrong version of <a href="https://github.com/polycube-network/bcc">bcc</a>;</p>
<blockquote>
<h5>BPF Compiler Collection (BCC)</h5>
<p>BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.</p>
</blockquote>
<hr>
<p>I decided to step back, and grab a clean copy of <code>polycubed</code> from <a href="https://github.com/polycube-network/polycube">github</a>.</p>
<div class="code"><pre class="code literal-block">pi@raspberrypi:~/polycube<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>submodule<span class="w"> </span>update<span class="w"> </span>--init<span class="w"> </span>--recursive
pi@raspberrypi:~/polycube/build<span class="w"> </span>$<span class="w"> </span>cmake<span class="w"> </span>..<span class="w"> </span>-DENABLE_PCN_IPTABLES<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_BRIDGE<span class="o">=</span>ON<span class="w"> </span><span class="se">\ </span><span class="w"> </span>
<span class="w"> </span>-DENABLE_SERVICE_DDOSMITIGATOR<span class="o">=</span>OFF<span class="w"> </span><span class="se">\ </span><span class="w"> </span>
<span class="w"> </span>-DENABLE_SERVICE_FIREWALL<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_HELLOWORLD<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_IPTABLES<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_K8SFILTER<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_K8SWITCH<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_LBDSR<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_LBRP<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_NAT<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_PBFORWARDER<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_ROUTER<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_SIMPLEBRIDGE<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_SIMPLEFORWARDER<span class="o">=</span>ON<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_TRANSPARENTHELLOWORLD<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_SYNFLOOD<span class="o">=</span>OFF<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-DENABLE_SERVICE_PACKETCAPTURE<span class="o">=</span>OFF<span class="w"> </span>-DENABLE_SERVICE_K8SDISPATCHER<span class="o">=</span>OFF
--<span class="w"> </span>The<span class="w"> </span>C<span class="w"> </span>compiler<span class="w"> </span>identification<span class="w"> </span>is<span class="w"> </span>GNU<span class="w"> </span><span class="m">10</span>.2.1
--<span class="w"> </span>The<span class="w"> </span>CXX<span class="w"> </span>compiler<span class="w"> </span>identification<span class="w"> </span>is<span class="w"> </span>GNU<span class="w"> </span><span class="m">10</span>.2.1
--<span class="w"> </span>Detecting<span class="w"> </span>C<span class="w"> </span>compiler<span class="w"> </span>ABI<span class="w"> </span>info
--<span class="w"> </span>Detecting<span class="w"> </span>C<span class="w"> </span>compiler<span class="w"> </span>ABI<span class="w"> </span>info<span class="w"> </span>-<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Check<span class="w"> </span><span class="k">for</span><span class="w"> </span>working<span class="w"> </span>C<span class="w"> </span>compiler:<span class="w"> </span>/usr/bin/cc<span class="w"> </span>-<span class="w"> </span>skipped
--<span class="w"> </span>Detecting<span class="w"> </span>C<span class="w"> </span>compile<span class="w"> </span>features
--<span class="w"> </span>Detecting<span class="w"> </span>C<span class="w"> </span>compile<span class="w"> </span>features<span class="w"> </span>-<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Detecting<span class="w"> </span>CXX<span class="w"> </span>compiler<span class="w"> </span>ABI<span class="w"> </span>info
--<span class="w"> </span>Detecting<span class="w"> </span>CXX<span class="w"> </span>compiler<span class="w"> </span>ABI<span class="w"> </span>info<span class="w"> </span>-<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Check<span class="w"> </span><span class="k">for</span><span class="w"> </span>working<span class="w"> </span>CXX<span class="w"> </span>compiler:<span class="w"> </span>/usr/bin/c++<span class="w"> </span>-<span class="w"> </span>skipped
--<span class="w"> </span>Detecting<span class="w"> </span>CXX<span class="w"> </span>compile<span class="w"> </span>features
--<span class="w"> </span>Detecting<span class="w"> </span>CXX<span class="w"> </span>compile<span class="w"> </span>features<span class="w"> </span>-<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Version<span class="w"> </span>is<span class="w"> </span>v0.9.0+<span class="w"> </span><span class="o">[</span>git:<span class="w"> </span><span class="o">(</span>branch/commit<span class="o">)</span>:<span class="w"> </span>master/a143e3c0-dirty<span class="o">]</span>
--<span class="w"> </span>Latest<span class="w"> </span>recognized<span class="w"> </span>Git<span class="w"> </span>tag<span class="w"> </span>is<span class="w"> </span>v0.9.0
--<span class="w"> </span>Git<span class="w"> </span>HEAD<span class="w"> </span>is<span class="w"> </span>a143e3c0325400dad7b9ff3406848f5a953ed3d1
--<span class="w"> </span>Revision<span class="w"> </span>is<span class="w"> </span><span class="m">0</span>.9.0-a143e3c0
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>HAVE_NO_PIE_FLAG
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>HAVE_NO_PIE_FLAG<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>HAVE_REALLOCARRAY_SUPPORT
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>HAVE_REALLOCARRAY_SUPPORT<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>Found<span class="w"> </span>LLVM:<span class="w"> </span>/usr/lib/llvm-9/include<span class="w"> </span><span class="m">9</span>.0.1<span class="w"> </span><span class="o">(</span>Use<span class="w"> </span>LLVM_ROOT<span class="w"> </span>envronment<span class="w"> </span>variable<span class="w"> </span><span class="k">for</span><span class="w"> </span>another<span class="w"> </span>version<span class="w"> </span>of<span class="w"> </span>LLVM<span class="o">)</span>
--<span class="w"> </span>Found<span class="w"> </span>BISON:<span class="w"> </span>/usr/bin/bison<span class="w"> </span><span class="o">(</span>found<span class="w"> </span>version<span class="w"> </span><span class="s2">"3.7.5"</span><span class="o">)</span>
--<span class="w"> </span>Found<span class="w"> </span>FLEX:<span class="w"> </span>/usr/bin/flex<span class="w"> </span><span class="o">(</span>found<span class="w"> </span>version<span class="w"> </span><span class="s2">"2.6.4"</span><span class="o">)</span>
--<span class="w"> </span>Found<span class="w"> </span>LibElf:<span class="w"> </span>/usr/lib/aarch64-linux-gnu/libelf.so<span class="w"> </span>
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>ELF_GETSHDRSTRNDX
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>ELF_GETSHDRSTRNDX<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>Could<span class="w"> </span>NOT<span class="w"> </span>find<span class="w"> </span>LibDebuginfod<span class="w"> </span><span class="o">(</span>missing:<span class="w"> </span>LIBDEBUGINFOD_LIBRARIES<span class="w"> </span>LIBDEBUGINFOD_INCLUDE_DIRS<span class="o">)</span>
--<span class="w"> </span>Using<span class="w"> </span>static-libstdc++
--<span class="w"> </span>Could<span class="w"> </span>NOT<span class="w"> </span>find<span class="w"> </span>LuaJIT<span class="w"> </span><span class="o">(</span>missing:<span class="w"> </span>LUAJIT_LIBRARIES<span class="w"> </span>LUAJIT_INCLUDE_DIR<span class="o">)</span>
--<span class="w"> </span>jsoncons<span class="w"> </span>v0.142.0
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread.h
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread.h<span class="w"> </span>-<span class="w"> </span>found
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>CMAKE_HAVE_LIBC_PTHREAD
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>CMAKE_HAVE_LIBC_PTHREAD<span class="w"> </span>-<span class="w"> </span>Failed
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread_create<span class="w"> </span><span class="k">in</span><span class="w"> </span>pthreads
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread_create<span class="w"> </span><span class="k">in</span><span class="w"> </span>pthreads<span class="w"> </span>-<span class="w"> </span>not<span class="w"> </span>found
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread_create<span class="w"> </span><span class="k">in</span><span class="w"> </span>pthread
--<span class="w"> </span>Looking<span class="w"> </span><span class="k">for</span><span class="w"> </span>pthread_create<span class="w"> </span><span class="k">in</span><span class="w"> </span>pthread<span class="w"> </span>-<span class="w"> </span>found
--<span class="w"> </span>Found<span class="w"> </span>Threads:<span class="w"> </span>TRUE<span class="w"> </span>
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_HIDDEN_VISIBILITY
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_HIDDEN_VISIBILITY<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_HIDDEN_INLINE_VISIBILITY<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_DEPRECATED_ATTR
--<span class="w"> </span>Performing<span class="w"> </span>Test<span class="w"> </span>COMPILER_HAS_DEPRECATED_ATTR<span class="w"> </span>-<span class="w"> </span>Success
--<span class="w"> </span>The<span class="w"> </span>following<span class="w"> </span>OPTIONAL<span class="w"> </span>packages<span class="w"> </span>have<span class="w"> </span>been<span class="w"> </span>found:
<span class="w"> </span>*<span class="w"> </span>BISON
<span class="w"> </span>*<span class="w"> </span>FLEX
<span class="w"> </span>*<span class="w"> </span>Threads
--<span class="w"> </span>The<span class="w"> </span>following<span class="w"> </span>REQUIRED<span class="w"> </span>packages<span class="w"> </span>have<span class="w"> </span>been<span class="w"> </span>found:
<span class="w"> </span>*<span class="w"> </span>LibYANG
<span class="w"> </span>*<span class="w"> </span>LLVM
<span class="w"> </span>*<span class="w"> </span>LibElf
--<span class="w"> </span>The<span class="w"> </span>following<span class="w"> </span>OPTIONAL<span class="w"> </span>packages<span class="w"> </span>have<span class="w"> </span>not<span class="w"> </span>been<span class="w"> </span>found:
<span class="w"> </span>*<span class="w"> </span>LibDebuginfod
<span class="w"> </span>*<span class="w"> </span>LuaJIT
--<span class="w"> </span>Found<span class="w"> </span>PkgConfig:<span class="w"> </span>/usr/bin/pkg-config<span class="w"> </span><span class="o">(</span>found<span class="w"> </span>version<span class="w"> </span><span class="s2">"0.29.2"</span><span class="o">)</span>
--<span class="w"> </span>Found<span class="w"> </span>OpenSSL:<span class="w"> </span>/usr/lib/aarch64-linux-gnu/libcrypto.so<span class="w"> </span><span class="o">(</span>found<span class="w"> </span>version<span class="w"> </span><span class="s2">"1.1.1n"</span><span class="o">)</span><span class="w"> </span>
--<span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'libnl-3.0'</span>
--<span class="w"> </span>Found<span class="w"> </span>libnl-3.0,<span class="w"> </span>version<span class="w"> </span><span class="m">3</span>.4.0
--<span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'libnl-genl-3.0'</span>
--<span class="w"> </span>Found<span class="w"> </span>libnl-genl-3.0,<span class="w"> </span>version<span class="w"> </span><span class="m">3</span>.4.0
--<span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'libnl-route-3.0'</span>
--<span class="w"> </span>Found<span class="w"> </span>libnl-route-3.0,<span class="w"> </span>version<span class="w"> </span><span class="m">3</span>.4.0
--<span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'libtins'</span>
--<span class="w"> </span>Found<span class="w"> </span>libtins,<span class="w"> </span>version<span class="w"> </span><span class="m">3</span>.5
--<span class="w"> </span>Found<span class="w"> </span>nlohmann_json:<span class="w"> </span>/home/pi/polycube/cmake/nlohmann_json/Findnlohmann_json.cmake<span class="w"> </span><span class="o">(</span>Required<span class="w"> </span>is<span class="w"> </span>at<span class="w"> </span>least<span class="w"> </span>version<span class="w"> </span><span class="s2">"3.5.0"</span><span class="o">)</span>
--<span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'systemd'</span>
--<span class="w"> </span>Found<span class="w"> </span>systemd,<span class="w"> </span>version<span class="w"> </span><span class="m">247</span>
--<span class="w"> </span>systemd<span class="w"> </span>services<span class="w"> </span>install<span class="w"> </span>dir:<span class="w"> </span>/lib/systemd/system
--<span class="w"> </span>Configuring<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Generating<span class="w"> </span><span class="k">done</span>
--<span class="w"> </span>Build<span class="w"> </span>files<span class="w"> </span>have<span class="w"> </span>been<span class="w"> </span>written<span class="w"> </span>to:<span class="w"> </span>/home/pi/polycube/build
</pre></div>
<div class="code"><pre class="code literal-block"><span class="nb">cd</span><span class="w"> </span>../src/libs/prometheus-cpp
mkdir<span class="w"> </span>build<span class="p">;</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>build
cmake<span class="w"> </span>..<span class="w"> </span>-DBUILD_SHARED_LIBS<span class="o">=</span>ON
make
sudo<span class="w"> </span>make<span class="w"> </span>install
</pre></div>
<p>I made changes to <code>config.cpp</code> to deal with our issue with <code>getopt_long</code> and the <code>while</code> loop. The changes are in my <a href="https://github.com/ajokela/polycube">polycube clone</a>.</p>
<p>I also did not have to add any of the <code>#include</code> lines that I had added during my first attempt on a SOQuartz module.</p>
<div class="code"><pre class="code literal-block">sudo<span class="w"> </span>src/polycubed/src/polycubed
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.453<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>loading<span class="w"> </span>configuration<span class="w"> </span>from<span class="w"> </span>/etc/polycube/polycubed.conf
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>configuration<span class="w"> </span>parameters:
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>loglevel:<span class="w"> </span>info
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>daemon:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>pidfile:<span class="w"> </span>/var/run/polycube.pid
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>port:<span class="w"> </span><span class="m">9000</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>addr:<span class="w"> </span>localhost
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>logfile:<span class="w"> </span>/var/log/polycube/polycubed.log
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-file:<span class="w"> </span>/etc/polycube/cubes.yaml
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.456<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-clean-init:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.457<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>cubes-dump-enable:<span class="w"> </span><span class="nb">false</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.457<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>polycubed<span class="w"> </span>starting...
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:06.457<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>version<span class="w"> </span>v0.9.0+<span class="w"> </span><span class="o">[</span>git:<span class="w"> </span><span class="o">(</span>branch/commit<span class="o">)</span>:<span class="w"> </span>master/a143e3c0-dirty<span class="o">]</span>
prog<span class="w"> </span>tag<span class="w"> </span>mismatch<span class="w"> </span>3e70ec38a5f6710<span class="w"> </span><span class="m">1</span>
WARNING:<span class="w"> </span>cannot<span class="w"> </span>get<span class="w"> </span>prog<span class="w"> </span>tag,<span class="w"> </span>ignore<span class="w"> </span>saving<span class="w"> </span><span class="nb">source</span><span class="w"> </span>with<span class="w"> </span>program<span class="w"> </span>tag
prog<span class="w"> </span>tag<span class="w"> </span>mismatch<span class="w"> </span>1e2ac42799daebd8<span class="w"> </span><span class="m">1</span>
WARNING:<span class="w"> </span>cannot<span class="w"> </span>get<span class="w"> </span>prog<span class="w"> </span>tag,<span class="w"> </span>ignore<span class="w"> </span>saving<span class="w"> </span><span class="nb">source</span><span class="w"> </span>with<span class="w"> </span>program<span class="w"> </span>tag
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:23.636<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>rest<span class="w"> </span>server<span class="w"> </span>listening<span class="w"> </span>on<span class="w"> </span><span class="s1">'127.0.0.1:9000'</span>
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:23.637<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>rest<span class="w"> </span>server<span class="w"> </span>starting<span class="w"> </span>...
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:23.740<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>bridge<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-bridge.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:23.779<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>firewall<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-firewall.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:23.882<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>nat<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-nat.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.012<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>pbforwarder<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-pbforwarder.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.145<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>router<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-router.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.210<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>simplebridge<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-simplebridge.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.239<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>simpleforwarder<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-simpleforwarder.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.282<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>iptables<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-iptables.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.412<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>service<span class="w"> </span>dynmon<span class="w"> </span>loaded<span class="w"> </span>using<span class="w"> </span>libpcn-dynmon.so
<span class="o">[</span><span class="m">2023</span>-01-26<span class="w"> </span><span class="m">20</span>:58:24.412<span class="o">]</span><span class="w"> </span><span class="o">[</span>polycubed<span class="o">]</span><span class="w"> </span><span class="o">[</span>info<span class="o">]</span><span class="w"> </span>loading<span class="w"> </span>metrics<span class="w"> </span>from<span class="w"> </span>yang<span class="w"> </span>files
</pre></div>
<p>The daemon successfully runs. I do, however, need to capture the work I did in getting the <code>linux kernel source headers</code> in place for the daemon to find to compile the eBPF code into byte code.</p>
<ol>
<li>Clone the Linux repository from Raspberry Pi, <a href="https://github.com/raspberrypi/linux">https://github.com/raspberrypi/linux</a>, into <code>/usr/src</code> on the Raspberry Pi</li>
<li>In <code>/lib/modules/5.15.84-v8+/</code> make a symlink named <code>build</code> and point it to <code>/usr/src/linux-upstream-5.15.89-v8+</code></li>
</ol>
<p>That will be it for the Work In Progress posts on <code>polycube</code>; I could attempt to recreate the steps taken, but I feel my notes across three posts should be enough. It is also isn't like <code>polycube</code> deployments are in hot demand. There is a strong likely hood that I am the first and only person who has run it on Arm-based hardware. The next post on <code>polycube</code> will be actually using it and in particular, the drop in replacement for <code>iptables</code>; that is what I am most interested in.</p>
<div style="width: 100%; text-align: center;">
<img src="https://tinycomputers.io/images/server-cords-and-wires.png.webp" style="zoom: 65%;" loading="lazy">
</div>Polycube on Arm-based SBC: Follow-up #1 (WIP)https://tinycomputers.io/posts/polycube-on-arm-based-sbc-follow-up-1.html2023-01-21T16:09:58-06:002023-01-21T16:09:58-06:00A.C. Jokela<p>My <a href="https://tinycomputers.io/posts/polycube-network-on-arm-hardware-WIP.html">first attempt</a> at getting <a href="https://polycube.network/">Polycube</a> compiling and running on an Arm-based single board computer had the compile part be successful, the running part was not. For that test, I used a Pine64 <a href="https://wiki.pine64.org/wiki/SOQuartz">SOQuartz</a> system on a module, which is pin compatible with a Raspberry Pi CM4. Using <a href="https://github.com/Plebian-Linux/quartz64-images">Plebian Linux</a>, there were a number of hoops to jump through; not limited to having to <code>#include</code> extra header files, compile certain pre-requisite libraries as well as including Debian packages from a few different versions of Debian.</p>
<p>But, compiling eventually succeeded, producing executables. Enter a Raspberry Pi CM4 running 64 bit Raspberry Pi OS. </p>
<h4>The Process</h4>
<p>The build process is more straight forward than with Plebian Linux. There is, however, still the need to use packages from several different versions of Debian. This is because Polycube depends upon older packages. These dependencies on old software and fairly brittle build pipeline really does hamper the use and adoption of Polycube.</p>
<p>After having what appears to be all the pre-requisite dependencies, the build ultimately fails on not finding a <code>struct</code> that has not been declared.</p>
<p><img alt="" src="https://tinycomputers.io/images/building-polycube-raspberry-cm4.png.webp"></p>
<p>I ended up having to get a particular version of <code>libyang</code>; specifically <a href="https://github.com/CESNET/libyang/releases/tag/v1.0.225">v1.0.255</a>. That got me past that error.</p>
<p>Following the rest of <a href="https://tinycomputers.io/posts/polycube-network-on-arm-hardware-WIP.html">my instructions</a> is more or less what I did to successfully compile <code>polycubed</code>. This is where things end just the same as my attempts with a SOQuartz module: running <code>polycubed</code> just hangs and does nothing more than consuming 100% of one core.</p>Polycube on Arm-based SBCs: Replacement for IPTables - WIPhttps://tinycomputers.io/posts/polycube-network-on-arm-hardware-WIP.html2023-01-15T13:57:27-06:002023-01-15T13:57:27-06:00A.C. Jokela<h4>The Basics - Work in Progress (WIP)</h4>
<p><strong>SPOILER ALERT</strong>: I have not been able to successfully execute the <code>polycubed</code> daemon. This effort is still a WIP.</p>
<p>Originally called <a href="https://ebpf.io/what-is-ebpf">extended Berkeley Packet Filtering</a>, it has since been simply referred to as <em>eBPF</em>. Vanilla BPF has been around for decades and has been <a href="https://www.kernel.org/doc/html/latest/networking/filter.html">used as a packet filter</a>, but eBPF is a more recent creation. And it has distinct advantages over existing Linux networking methods when used at industrial or commercial scales.</p>
<blockquote>
<p><em>Historically, the operating system has always been an ideal place to implement observability, security, and networking functionality due to the kernel’s privileged ability to oversee and control the entire system. At the same time, an operating system kernel is hard to evolve due to its central role and high requirement towards stability and security. The rate of innovation at the operating system level has thus traditionally been lower compared to functionality implemented outside of the operating system.</em></p>
<p><em>eBPF changes this formula fundamentally. By allowing to run sandboxed programs within the operating system, application developers can run eBPF programs to add additional capabilities to the operating system at runtime. The operating system then guarantees safety and execution efficiency as if natively compiled with the aid of a Just-In-Time (JIT) compiler and verification engine. This has led to a wave of eBPF-based projects covering a wide array of use cases, including next-generation networking, observability, and security functionality.</em></p>
</blockquote>
<p>Why eBPF? Because it has been gaining use within the world of containerization of work loads. Within the Linux world, this is usually Docker (I wrote a post on running <a href="https://tinycomputers.io/posts/pine64-quartz64-a-running-docker.html">Docker on a Pine64 Quartz Model A</a>). Observability is huge within the world of commercial-scale deployments of containers. Containers allow for millions or tens of millions of virtual workloads to run on thousands and millions of physical systems. You need insight into what your systems are doing and how systems are performing. eBPF, for tiny computers, is overkill. The level complexity it introduces and requires far outweighs the benefits of at such a small scale. But, using it within the confines of tiny computers could be a good introduction to this very powerful technology.</p>
<p>We will be attempting to use <a href="https://github.com/polycube-network/polycube">Polycube</a> for our eBPF needs.</p>
<blockquote>
<p><code>Polycube</code> is an <strong>open source</strong> software framework that provides <strong>fast</strong> and <strong>lightweight</strong> <strong>network functions</strong> such as bridges, routers, firewalls, and others.</p>
<p>Polycube services, called <code>cubes</code>, can be composed to build arbitrary <strong>service chains</strong> and provide custom network connectivity to <strong>namespaces</strong>, <strong>containers</strong>, <strong>virtual machines</strong>, and <strong>physical hosts</strong>.</p>
<p>For more information, jump to the project <a href="https://polycube-network.readthedocs.io/en/latest/">Documentation</a>.</p>
</blockquote>
<p>There are two ways in which to run Polycube: 1) via Docker; 2) "baremetal". Docker would be simplest if we were using stable builds of Debian or Ubuntu, but we are not.</p>
<p>For the purposes of this article, I am using a Pine64 <a href="https://wiki.pine64.org/wiki/SOQuartz">SOQuartz module</a>. I have tried both <a href="https://github.com/Plebian-Linux/quartz64-images">Plebian Linux</a> as well as <a href="https://dietpi.com/">DietPi</a>.</p>
<p>I will walk through "baremetal" because there are not Docker images available for arm64 architecture.</p>
<h3>Installing Polycube - <em>baremetal</em></h3>
<h4>1. Install <code>polycube</code> and dependencies:</h4>
<pre># add stretch packages to apt
# /etc/apt/sources.list
deb https://deb.debian.org/debian/ stretch-backports main contrib non-free
deb https://deb.debian.org/debian/ stretch main contrib non-free
deb https://deb.debian.org/debian/ stretch-backports-sloppy main contrib non-free
deb https://deb.debian.org/debian/ bookworm main contrib non-free</pre>
<h5>Update packages cache</h5>
<pre><code class="command language-bash">sudo apt update</code></pre>
<h5>Install Polycube dependencies</h5>
<pre><code class="command">sudo apt-get install golang-go git build-essential cmake bison flex libelf-dev libpcap-dev \
libnl-route-3-dev libnl-genl-3-dev uuid-dev pkg-config autoconf libtool m4 \
automake libssl-dev kmod jq bash-completion gnupg2 libpcre3-dev clang-5.0 \
clang-format-5.0 clang-tidy-5.0 libclang-5.0-dev libfl-dev \
iperf luajit arping netperf</code></pre>
<h5>Verify <code>golang</code> is at <code>v1.16</code> or newer</h5>
<pre><code class="command">go version
go version go1.19.3 linux/arm64</code></pre>
<h5>Install pistache - needed for the RESTful control daemon</h5>
<pre><code class="command">git clone https://github.com/oktal/pistache.git</code>
<code class="command">cd pistache</code>
# known working version of pistache
<code class="command">git checkout 117db02eda9d63935193ad98be813987f6c32b33</code>
<code class="command">git submodule update --init</code>
<code class="command">mkdir -p build && cd build</code>
<code class="command">cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DPISTACHE_USE_SSL=ON ..</code>
<code class="command">make -j $(getconf _NPROCESSORS_ONLN)</code>
<code class="command">sudo make install</code></pre>
<h5>Install libtins</h5>
<pre>
<code class="command">cd</code>
<code class="command">git clone --branch v3.5 https://github.com/mfontanini/libtins.git</code>
<code class="command">cd libtins</code>
<code class="command">mkdir -p build && cd build</code>
<code class="command">cmake -DLIBTINS_ENABLE_CXX11=1 \
-DLIBTINS_BUILD_EXAMPLES=OFF -DLIBTINS_BUILD_TESTS=OFF \
-DLIBTINS_ENABLE_DOT11=OFF -DLIBTINS_ENABLE_PCAP=OFF \
-DLIBTINS_ENABLE_WPA2=OFF -DLIBTINS_ENABLE_WPA2_CALLBACKS=OFF ..</code>
<code class="command">make -j $(getconf _NPROCESSORS_ONLN)</code>
<code class="command">sudo make install</code>
<code class="command">sudo ldconfig</code>
</pre>
<h5>Install <code>libyang</code> version 1</h5>
<pre>
<code class="command">cd</code>
<code class="command">git clone https://github.com/CESNET/libyang.git</code>
<code class="command">cd libyang</code>
<code class="command">git checkout libyang1</code>
<code class="command">mkdir build; cd build</code>
<code class="command">cmake ..</code>
<code class="command">make</code>
<code class="command">sudo make install</code>
</pre>
<h4>2. Clone <code>polycube</code> repository;</h4>
<p>**Note: ** you will need at least 3GB of disk space to be on the safe side for intermediate build artifacts</p>
<pre>
<code class="command">cd</code>
<code class="command">git clone https://github.com/polycube-network/polycube.git</code>
<code class="command">cd polycube</code>
<code class="command">git submodule update --init --recursive</code>
</pre>
<h5>Configure build for <code>prometheus-cpp</code></h5>
<pre>
<code class="command">cd polycube/src/libs/prometheus-cpp</code>
<code class="command">mkdir build && build</code>
<code class="command">cmake .. -DBUILD_SHARED_LIBS=ON</code>
</pre>
<h5>We need to make one modification to one source file</h5>
<pre>
# edit the ../core/src/histogram.cc
vi ../core/src/histogram.cc
</pre>
<p>Below <code>#include <iterator></code>, add the following:</p>
<pre>
#include <limits>
</pre>
<p>Save <code>../core/src/histogram.cc</code></p>
<p>Build <code>prometheus-cpp</code></p>
<pre>
<code class="command">make</code>
<code class="command">sudo make install</code>
</pre>
<h5>We need to make one modification to one source file</h5>
<pre>
<code class="command">cd</code>
<code class="command">cd polycube</code>
<code class="command">vi src/polycubed/src/server/Types/lexical_cast.cpp</code>
</pre>
<p>Add the following directly below <code>#include <string></code></p>
<pre>
#include <limits>
</pre>
<p>Save <code>src/polycubed/src/server/Types/lexical_cast.cpp</code></p>
<h5>We need to make another modification to a source file</h5>
<pre>
<code class="command">cd>
<code class="command">cd polycube</code>
<code class="command">vi src/polycubed/src/server/Resources/Body/ListKey.cpp</code>
</code></pre>
<p>Add the following directly below <code>#include <vector></code></p>
<pre>
#include <stdexcept>
</pre>
<p>Save <code>src/polycubed/src/server/Resources/Body/ListKey.cpp</code></p>
<h5>We need to make another modification to a source file</h5>
<pre>
<code class="command">cd</code>
<code class="command">cd polycube</code>
<code class="command">vi src/services/pcn-dynmon/src/MapExtractor.h</code>
</pre>
<p>Add the following directly below <code>#pragma once</code></p>
<pre>
#include <cstdlib>
</pre>
<p>Save<code>src/services/pcn-dynmon/src/MapExtractor.h</code></p>
<h5>Configure and build <code>polycube</code>; we will also be building the drop in equivalent of <code>iptables</code>, <code>polycube iptables</code></h5>
<pre>
<code class="command">cd</code>
<code class="command">cd polycube</code>
<code class="command">mkdir -p build && cd build</code>
<code class="command">cmake .. -DENABLE_PCN_IPTABLES=ON</code>
<code class="command">make -j 2</code>
<code class="command">go env -w GO111MODULE=off</code>
<code class="command">sudo make install</code>
</pre>
<p><img alt="" src="https://tinycomputers.io/images/compiling-polycube-dietpi.png.webp"></p>
<p>Attempt to start <code>polycubed</code></p>
<pre>
<code class="command">sudo polycubed</code>
</pre>
<p><strong>And that is where it gets hung up.</strong> The daemon never gets to the point of listening on the default port, tcp:9000. There are also no log messages (even turning up the loglevel to <code>debug</code>)</p>