{"id":4753,"date":"2023-02-19T10:41:38","date_gmt":"2023-02-19T16:41:38","guid":{"rendered":"https:\/\/dev.iachieved.it\/iachievedit\/?p=4753"},"modified":"2023-02-19T10:42:16","modified_gmt":"2023-02-19T16:42:16","slug":"compiling-lldb-on-the-risc-v-hifive-unmatched","status":"publish","type":"post","link":"https:\/\/dev.iachieved.it\/iachievedit\/compiling-lldb-on-the-risc-v-hifive-unmatched\/","title":{"rendered":"Compiling LLDB on the RISC-V HiFive Unmatched"},"content":{"rendered":"

Different processors and instruction set architectures<\/a> have always fascinated me. As a child of the late 70s and 80s I wasn’t quite old enough to appreciate the difference between the 6502<\/a> and 6809<\/a> processors, but by the early 90s and the introduction of the PowerPC, particularly in the PowerMac 6100<\/a> it was clear that ISAs can come and go. What I didn’t appreciate at the time, and didn’t until the early 2020s, is the licensing model around ISAs. Plenty has been written on RISC-V and the opportunity for it to revolutionize the market, so we won’t repeat it here. All I can say is that I’ve had fun over the past couple of years supporting RISC-V through purchasing and tinkering with systems built using RISC-V processors.<\/p>\n

Recently I had the good fortune of purchasing a brand new HiFive Unmatched<\/a> and really wanted to stress it as a “daily driver” for building software natively on and for a RISC-V system. Why not use a cross-compiler? Indeed, my M1 MacBook Pro kicks the Unmatched’s ass, and even running benchmarks<\/a> in a Docker container on the M1 is faster:<\/p>\n

On an M1:<\/p>\n

\nroot@2972ef523083:~\/primes_benchmark# .\/primes_benchmark\nStarting run\n3713160 primes found in 7268 ms\n236 bytes of code in countPrimes()\n<\/pre>\n
7.3 seconds.<\/p>\n
On the HiFive Unmatched:<\/p>\n
\n\nroot@stormtrooper # .\/primes_benchmark\nStarting run\n3713160 primes found in 18981 ms\n236 bytes of code in countPrimes()\n<\/pre>\n
19 seconds.<\/p>\n
QEMU RISC-V emulation vs. what was considered a top-of-the-line RISC-V processor, the SiFive Freedom U740 SoC.  Indeed, it still might be.<\/p>\n
\"\"<\/a>
My HiFive Unmatched, affectionately named Stormtrooper<\/figcaption><\/figure>\n
But, this is all missing the point isn’t it?  To propel RISC-V forward as an alternative we must move beyond using emulation and relying on cross-compilers, and push ourselves to “go native.”  So that’s what we’ll do by compiling the LLDB debugger<\/a> natively on the HiFive Unmatched.  Why LLDB?  Well, for whatever reason it wasn’t installed with apt-get install llvm<\/code> or apt-get install clang<\/code> on my machine, so I decided to take the plunge and compile it myself.<\/p>\n
I strongly prefer to use “chroot jails” for building software.  For starters, they are useful to ensure that dependencies are strongly tracked, and that “cruft” that accumulates over time doesn’t pollute your build environment.
\nSince our HiFive Unmatched is running Ubuntu 22.04 “Jammy”, it made sense to use debootstrap<\/a> to create our jail.  First, let’s install debootstrap<\/code> with apt-get install debootstrap<\/code> and then look how our filesystem is structured:<\/p>\n
\nroot@stormtrooper # mkdir -p \/opt\/loadbuild\/jails\nroot@stormtrooper # cd \/opt\/loadbuild\/jails\nroot@stormtrooper # debootstrap jammy build-lldb\n<\/pre>\n
debootstrap<\/code> will create us a nearly<\/i> fully functional environment for cross-compiling LLDB.  Once it’s complete there are a few bind mounts we will need to set up.<\/p>\n
\nroot@stormtrooper # cd \/opt\/loadbuild\/jails\n\n# Do the mounts\nroot@stormtrooper # mount --bind \/dev build-lldb\/dev\nroot@stormtrooper # mount --bind \/proc build-lldb\/proc\nroot@stormtrooper # mount --bind \/sys build-lldb\/sys\nroot@stormtrooper # mount -t devpts none build-lldb\/dev\/pts\/\n\n# chroot\nroot@stormtrooper # chroot build-lldb\n<\/pre>\n
From here, we are in the jail.<\/p>\n
\nroot@build-lldb # mkdir -p \/opt\/build\nroot@build-lldb # mkdir -p \/opt\/src\n<\/pre>\n
Now we’re ready to review the instructions<\/a> for building LLDB.  They’re pretty straightforward, and consist of the installing prerequisites, configuring your building tree, and typing make<\/code>.  Well, in this case, ninja<\/code>.  Your dependencies are the typical cast of characters for a Debian-based distribution:  build-essential<\/code> for the compiler, libraries for text interfaces (libedit-dev<\/code>, libncurses5-dev<\/code>).  LLDB comes with a scripting facility, and its choice for interfacing languages is SWIG<\/a>.<\/p>\n
In Ubuntu 22.04 the swig<\/code> package is in the “Universe” repository, so let’s add it:<\/p>\n
\nroot@build-lldb # sudo apt-get install -y software-properties-common\nroot@build-lldb # sudo add-apt-repository -y universe\n<\/pre>\n
Now we can install all of our dependencies for building LLDB.<\/p>\n
\nroot@build-lldb # sudo apt-get install -y build-essential \\\n  swig \\\n  python3-dev \\\n  libedit-dev \\\n  libncurses5-dev \\\n  ninja-build \\\n  cmake \\\n  screen \\\n  git\n<\/pre>\n
Editor’s note:  I’m aware of multistrap<\/code><\/a> but haven’t had time to research it.<\/p>\n
\nroot@build-lldb # cd \/opt\/src\/\nroot@build-lldb # git clone https:\/\/github.com\/llvm\/llvm-project.git\nroot@build-lldb # cd \/opt\/build\nroot@build-lldb # cmake -G Ninja -DLLVM_ENABLE_PROJECTS=\"clang;lldb\" -DLLVM_TARGETS_TO_BUILD=RISCV -DCMAKE_BUILD_TYPE=Release \/opt\/src\/llvm-project\/llvm\n<\/pre>\n
A couple of comments on the build options.  First, we’re only going to build lldb<\/code> and we’re only going to support the RISCV architecture (that is, our lldb<\/code> won’t be able to read any other architectures).  Building lldb<\/code> requires clang<\/code> to be “enabled”, so LLVM_ENABLE_PROJECTS<\/code> must be set to clang;lldb<\/code>.  Finally, to save some disk space, we’ll build the Release<\/code> version.<\/p>\n
Now, before we build, create a screen<\/code> session.  This is going to allow our us to get back to our build if ssh<\/code> hangs up on us.<\/p>\n
\nroot@build-lldb # screen -S build-lldb\nroot@build-lldb # time ninja lldb lldb-server\n\n[1\/3237] Building CXX object lib\/Support\/CMakeFiles\/LLVMSupport.dir\/ARMAttributeParser.cpp.o\n<\/pre>\n
And we’re off!<\/p>\n
This will take some time.  On my HiFive Unmatched it took 10 hours.<\/p>\n
\"\"<\/a><\/p>\n
\n[3237\/3237] Linking CXX executable bin\/lldb\n\nreal    573m15.878s\nuser    2133m44.806s\nsys     114m43.002s\n<\/pre>\n
Whew!  A little over nine and a half hours to build lldb<\/code>.  Let’s hope it works!<\/p>\n
Testing<\/h2>\n
Let’s look at some basics of LLDB with a toy C application:<\/p>\n
\n#include \n#include \n\nint add(int a, int b) {\n  return a+b;\n}\n\nint* add_arrays(int* a1, int* a2, int len) {\n  int* c = malloc(sizeof(int)*len);\n\n  for (int i = 0; i < len; i++) {\n    c[i] = a1[i] + a2[i];\n  }\n\n  return c;\n}\n\nint main(void) {\n  int a = 42;\n  int b = 97;\n\n  int a1[] = {1, 2, 3, 4, 5};\n  int a2[] = {6, 7, 8, 9, 0};\n\n  add(a, b);\n\n  int* a3 = add_arrays(a1, a2, 5);\n\n  for (int i = 0; i < 5; i++) {\n    printf(\"%d + %d = %d\\n\", a1[i], a2[i], a3[i]);\n  }\n\n  return 0;\n\n}\n<\/pre>\n
To get this in our debugger, compile with gcc -g -o toy toy.c<\/code>, or if you have clang<\/code> installed the command-line arguments are the same.<\/p>\n
Okay, here we go.  \/opt\/loadbuild\/jails\/build-lldb\/opt\/build\/bin\/lldb toy<\/code>.  You can see I have not installed lldb<\/code> anywhere and its still in its \"jail\" directory, but we're no longer in the jail (where it sits at \/opt\/build\/bin\/lldb<\/code>).<\/p>\n
\n~ \/opt\/loadbuild\/jails\/build-lldb\/opt\/build\/bin\/lldb toy\n(lldb) target create \"toy\"\nCurrent executable set to '\/home\/joe\/toy' (riscv64).\n<\/pre>\n
We'll make this brief, as there are a lot<\/i> of LLDB tutorials online.  First, let's set a breakpoint on the \"most interesting\" function, add_arrays<\/code>.  Hitting the TAB key after typing add<\/code> shows the useful completion feature.<\/p>\n
\n(lldb) b add\nAvailable completions:\n    add\n    add_arrays\n(lldb) b add_arrays\nBreakpoint 1: where = toy`add_arrays + 20 at toy.c:9:31, address = 0x00000000000006ee\n<\/pre>\n
Let's run with r<\/code> (and yes I'm aware r<\/code> is a GDB command that LLDB has mapped to):<\/p>\n
\n(lldb) r\nProcess 35357 launched: '\/home\/joe\/toy' (riscv64)\nProcess 35357 stopped\n* thread #1, name = 'toy', stop reason = breakpoint 1.1\n    frame #0: 0x0000002aaaaaa6ee toy`add_arrays(a1=0x0000003ffffff2f0, a2=0x0000003ffffff2d8, len=5) at toy.c:9:31\n   6    }\n   7\n   8    int* add_arrays(int* a1, int* a2, int len) {\n-> 9      int* c = malloc(sizeof(int)*len);\n   10\n   11     for (int i = 0; i < len; i++) {\n   12       c[i] = a1[i] + a2[i];\n<\/pre>\n
Let's dump the RISC-V registers!<\/p>\n
\n(lldb) register read\nGeneral Purpose Registers:\n        pc = 0x0000002aaaaaa6ee  toy`add_arrays + 20 at toy.c:9:31\n        ra = 0x0000002aaaaaa7ca  toy`main + 116 at toy.c:27:13\n        sp = 0x0000003ffffff270\n        gp = 0x0000002aaaaac800\n        tp = 0x0000003ff7fcd3e0\n        t0 = 0x0000003ff7fdd4c8\n        t1 = 0x0000003ff7febefc\n        t2 = 0x0000003ff7fbf420\n        fp = 0x0000003ffffff2b0\n        s1 = 0x0000000000000001\n        a0 = 0x0000003ffffff2f0\n        a1 = 0x0000003ffffff2d8\n        a2 = 0x0000000000000005\n        a3 = 0x0000000000000000\n        a4 = 0x0000003ffffff340\n        a5 = 0x0000002aaaaaa756  toy`main at toy.c:18\n        a6 = 0x0000003ff7fbfdb0\n        a7 = 0x2f56405b0043434a\n        s2 = 0x0000000000000000\n        s3 = 0x0000002aaaaabe18  toy`__do_global_dtors_aux_fini_array_entry\n        s4 = 0x0000002aaaaaa756  toy`main at toy.c:18\n        s5 = 0x0000003ffffff4b8\n        s6 = 0x0000002aaaaabe18  toy`__do_global_dtors_aux_fini_array_entry\n        s7 = 0x0000003ff7ffdd18\n        s8 = 0x0000003ff7ffe050\n        s9 = 0x0000003fc562ab40\n       s10 = 0x0000002ae41ff460\n       s11 = 0x0000003fc562ab30\n        t3 = 0x0000003ff7e9d9a0\n        t4 = 0x000000000000eefc\n        t5 = 0x0000000000000003\n        t6 = 0xffffffffffffffff\n      zero = 0x0000000000000000\n<\/pre>\n
I find it interesting that even the RISC-V zero<\/code> register is printed out, but then again, it is<\/i> a register.<\/p>\n
A few other quick commands.  Since we're in a frame (function) we can print the local variables.<\/p>\n
\n(lldb) frame variable\n(int *) a1 = 0x0000003ffffff2f0\n(int *) a2 = 0x0000003ffffff2d8\n(int) len = 5\n(int *) c = 0x0000000000000001\n<\/pre>\n
The arrays are passed as two integer pointers, and looking back at the registers we can see the RISC-V calling convention in action (a0, a1, and a2 have the function arguments).  Let's read the memory that a1 points to:<\/p>\n
\n(lldb) memory read 0x0000003ffffff2f0\n0x3ffffff2f0: 01 00 00 00 02 00 00 00 03 00 00 00 04 00 00 00  ................\n0x3ffffff300: 05 00 00 00 61 00 00 00 2a 00 00 00 00 00 00 00  ....a...*.......\n<\/pre>\n
Nifty.  We can even get naughty and write values into memory.  Let's write the value 42 at the second index of the array.<\/p>\n
\n(lldb) memory write 0x0000003ffffff2f4 42\n(lldb) memory read 0x0000003ffffff2f0\n0x3ffffff2f0: 01 00 00 00 42 00 00 00 03 00 00 00 04 00 00 00  ....B...........\n0x3ffffff300: 05 00 00 00 61 00 00 00 2a 00 00 00 00 00 00 00  ....a...*.......\n<\/pre>\n
Hitting c<\/code> for continue<\/code> we can see our handiwork:<\/p>\n
\n(lldb) c\nProcess 35357 resuming\n1 + 6 = 7\n66 + 7 = 73\n3 + 8 = 11\n4 + 9 = 13\n5 + 0 = 5\nProcess 35357 exited with status = 0 (0x00000000)\n<\/pre>\n
Well, it looks like the memory write<\/code> command took its argument in base 16, but it worked all the same.<\/p>\n
Closing<\/h2>\n
As I said at the start, there's something about different instruction sets and CPU architectures that I find fascinating.  History has shown that there is still a lot of room for innovation and competition, and RISC-V is proving that out each and every day.  Is it ready to be your daily driver?  Probably not, but that isn't a function of the ISA but of the market and its evolution relative to other platforms.  It will continue to make progress as companies like SiFive and StarFive bring processors and boards to the market.  I have my sights set on the HiFive Pro P550<\/a> \"Horse Creek\" board, and hopefully we can cut some time off compiling LLDB!<\/p>\n","protected":false},"excerpt":{"rendered":"
Different processors and instruction set architectures have always fascinated me. As a child of the late 70s and 80s I wasn’t quite old enough to appreciate the difference between the 6502 and 6809 processors, but by the early 90s and the introduction of the PowerPC, particularly in the PowerMac 6100 it was clear that ISAs […]<\/p>\n","protected":false},"author":1,"featured_media":4785,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,101,116],"tags":[],"class_list":["post-4753","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hacking","category-risc-v","category-software-development"],"_links":{"self":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/4753"}],"collection":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/comments?post=4753"}],"version-history":[{"count":57,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/4753\/revisions"}],"predecessor-version":[{"id":4813,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/4753\/revisions\/4813"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media\/4785"}],"wp:attachment":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media?parent=4753"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/categories?post=4753"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/tags?post=4753"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}