Taming the Beast: A Leap forward for 3DS Homebrew Development

Today, we're releasing a central building block used throughout Mikage: A package management solution for 3DS tools and homebrew development! conan-3ds takes care of setting up toolchains and development libraries, with support for side-by-side installation, semi-reproducible builds, and safe updates. While this is a tool for fellow homebrew and emulator developers, we figured it's a good opportunity to explain a bit more about its background and the problems it solves for Mikage development.

Background

During development of our 3DS emulator, we naturally work with a lot of 3DS homebrew for testing, so we rely heavily on various tools such as the devkitARM compiler toolchain, the ctrtool info tool, or the 3DS Homebrew Menu. To test hardware features in isolation, example projects such as the devkitPro ones come into play.

On top of the existing 3DS ecosystem of apps and tools, we also wrote a lot of our own utilities: From our automated hardware test suite, a title manager (similar to FBI), a sophisticated GPU debugging tool, and more. This is where the homebrew libraries libctru and citro3d come in, but also widely used C++ libraries like fmt or Catch2.

All of this is to say: There are a lot of components related to 3DS development when creating large, complex software such as Mikage. This opens up some interesting challenges:

• How do you organize all the tools, libraries, and applications on disk? How difficult is it to add new things into the mix?
• How do you ensure applications are built with the correct version of each library? How do you keep track of the required version lists in the first place?
• What do you do if one application requires libctru from 2024 but another won't built unless using the one from 2018?
• If you update a library with a breaking change, how do you avoid having to make changes to all projects that use it at once?
• What about external collaborators? How do you ensure their build setup is consistent with yours?
• How do you avoid all of this management overhead to blow up to a full-time job?

Enter package managers

Fellow software developers will already know the answer: A package manager!

A package manager or package-management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer in a consistent manner

We aren't the first to have this idea: devkitPro offer a similar setup for the 3DS based on pacman. Sadly it's not suited to address the problems listed in the beginning however, so we needed to come up with an alternative. After looking into existing options, we found a promising base for our work: Conan, a package manager for C and C++ developers.

While Pacman operates like a traditional system package manager in the Linux world, Conan is more like Python's pip. It allows applications to declare a list of dependencies with specific version ranges; it supports side-by-side installation of multiple versions of the same library; and it has fantastic integration into different build systems like CMake and Autotools. A perfect match!

What we added on top is a number of different 3DS-specific packages, and Conan extensions to facilitate setup of the compiler toolchain. The result: conan-3ds, a package repository that adds 3DS superpowers to Conan. While we want to focus this post on our motivation for writing this, here's a sneak preview:

This innocent-looking command conan install-3ds 3ds_examples will install the 3DS example projects, including all of the heavy-lifting that you didn't want to think about in the first place: It downloads the devkitARM toolchain and it builds the core dependencies libctru, citro2d, and citro3d. Even ImageMagick is compiled behind the scenes since some examples rely on it. You can also specify specific versions, such as 3ds_examples/20170714 --toolchain devkitarm49 to build older releases that wouldn't build with the latest library versions. Pretty cool, huh?

If you're intrigued, head over to our source repository for a full usage guide. For this post, let's continue with the problems this solves for us!

Within arm's reach: More than 1000 libraries

One main advantage of Conan is the huge number of C and C++ libraries that are already packaged for it: At the time of writing, 1737 packages are available! Need to add string formatting, unit testing, or file decompression to your project? No problem. Even the rather heavyweight Boost libraries work!

This is a massive upgrade compared to the set of previously available packages! Not only does Conan offer more than 30 times as many packages, but the various libraries are more up to date too: The extreme case here is SDL, previously limited to the 13-year-old version 1.2. conan-3ds ships the latest version 2.30, released in May of this year!

Of course, not all of the 1737 packages will be compatible with the 3DS due to platform specifics, but in our experience libraries without strong platform dependencies will just work. Additionally, some packages will work fine if specific platform-specific components are disabled. If you find anything that can easily be fixed, be sure to let us know so we can ship the right package settings by default!

The right match for a lone library version

How hard could it be to build old homebrew apps from 2018? In an ideal world, the libctru and citro3d libraries used by most 3DS homebrew would only ever add new features when releasing new versions. Old homebrew (written against old versions of these base libraries) would continue to compile fine even with newer versions then. In theory, a 3DS application that uses an old citro3d version for 3D graphics should be able to move to a newer core libctru library or a more recent devkitARM toolchain without a problem.

In practice, sadly the 3DS library ecosystem hasn't been so reliable. devkitARM, libctru, and citro3d are heavily intertwined with each other: If you update one of the three, you generally have to update all of them. Conversely if you're trying to build an older homebrew app, you will generally need to match up older versions as well - and with so many years passed, it's not easy to figure out which citro3d versions can be used with the ancient libctru 1.6.0!

conan-3ds addresses this by writing down the exact version ranges needed for libraries and their dependencies. Here's how citro3d declares its libctru dependencies:

def requirements(self):
    if self.version >= '1.7.0':
        self.requires("libctru/[>=2.1.0]")
    elif self.version >= '1.6.1':
        self.requires("libctru/[>=2.0.0 <2.1.0]")
    elif self.version >= '1.3.0':
        self.requires("libctru/[>=1.5.1 <2.0.0]")

Manually finding out the compatible ranges was rather painful, but now that it's done others won't have to repeat the effort. Thanks to this list, we can have confidence that when building old software, we'll always combine the right library versions.

Russian update roulette

It's one thing to take care of old, unmaintained apps, but other issues come up with actively developed software. Imagine you're working on a 3DS app and a new libctru version gets released with a cool new feature: You update the dependency, make some tweaks to your code, and it all works. That is, until you realize there was an API change that actually breaks all the other projects you've been working on. You either have to fix all your projects, or move back to the old libctru version.

Updating dependencies has been an all-or-nothing deal that 3DS homebrew developers are all too familiar with. While pacman made updating libraries easier, it made this problem worse since there is no option for downgrades (once you updated, you were stuck!). Due to the sheer number of projects we touched during Mikage development, this often led to days of unplanned follow-up work. Updates became a big, unpredictable, irreversible risk.

With Conan, we win a powerful feature to combat the problem: Side-by-side installation of multiple versions of the same library. When using multiple projects in parallel, each can use their own copy of libctru and other libraries as needed, allowing you to selectively opt into newer package versions individually. If something doesn't work out, you can just move back to the old version without affecting the other projects.

The best part? Since each homebrew app declares which library versions are required, all of this happens automatically. There's no need to juggle library versions yourself. This is how conan-3ds can build any of the 3DS homebrew examples from 2015 to 2024 against matching old versions of the base libraries without a sweat.

Frozen in time: Semi-reproducible builds

Ideally, app authors would eventually update their dependencies to the latest versions, but in practice that doesn't happen for various reasons. For Mikage, this posed a big problem when we tried debugging old homebrew applications: Building from source seemed impossible because the involved libctru versions were so old. But updating would've effectively required a rewrite of the app, at which point it wouldn't have been useful for debugging the original issue.

Once again, side-by-side installation comes to the rescue: All we need to figure out now is what libctru version an app was written against, and conan-3ds will ensure to combine it with the correct versions of other libraries. We also have version checks on devkitARM itself in place, since particularly ancient libctru versions won't build on more recent compiler versions.

As a stress test (and because we actually needed to test a very specific thing!), we went as far and checked out an ancient version of 3ds_examples from August 2015 along with one of the earliest libctru releases ever. The result? It works!

We refer to this as semi-reproducible builds. This plays on the practice of reproducible builds, which has much stronger bit-by-bit requirements that we don't currently aim for - but it's such a big step forward compared to the status quo that we think it's a suitable analogy.

The road ahead

conan-3ds has been in use in Mikage for longer than you might think. Over the years, we went through several iterations to add features and to make usage easier. We're finally at a point where we think the system is robust enough for others to experiment with. While we've had some help testing, this initial release is still anticipated to have rough edges. Contributions to make it even better than today are hence absolutely welcome!

Did you add more versions for recently released libraries? Did you get a new package from Conan Center working by tweaking a few settings? Did you try out conan-3ds in a homebrew app of yours? Are you interested in using Conan on the Nintendo Switch? We want to hear about your experience! And while we can't turn this side-project into a full-time job, by working together we can smoothen things out for everyone.

As far as Mikage is concerned, the next thing we're pushing to release is our graphics test suite, which makes heavy use of the Conan infrastructure and hence had to wait for this initial code drop. Stay tuned!

Credits

We'd like to thank a number of individuals without whom this undertaking would not have possible (listed in no particular order):

• leseratte, for kindly hosting an archive of prebuilt compiler toolchains and for giving us permission us to have conan-3ds fetch builds directly from their server
• The Conan team, for building a powerful package manager
• The Conan Center maintainers, for providing so many packages out-of-the-box
• The various people who kindly tested the project during development

Furthermore, thanks to our Sponsor-level supporters on Patreon, including:

• Francisco Garcia
• Mr. Madness
• Pretendo Network
• Thidguy