compiling – What’s the purpose of using Guix within Gitian? Doesn’t that reintroduce dependencies and security concerns?

There are a number of benefits that come with using Guix as a release
build environment (or just as a build system in general).

Using Gitian, you have to pick a Linux distro to build on. This
defines your toolchains, available packages, the glibc your building
against etc.

This also means you’re subject to choices of the upstream package
maintainers. They control how/when package versions are updated, what
patches are applied to them, how they are compiled etc. This has
downsides.

As an example, our security-check tests rely on a number of -no-*
options being present in mingw-w64 ld. These options were patched into
the ld in the Ubuntu Bionic binutils-mingw-w64 package, however, the
binutils maintainer chose to stop doing that in Ubuntu Focal, which
meant our security-check tests would not have been able to run (they
have since decided to re-patch them in Hirsute). See #18629 for more
details.

In Guix, because we’re in full control of our toolchains, we don’t
have to worry about issues like this, and can just directly apply the
patches we want to use, and this is exactly what we’ve done in #22381,
where we started patching the -no-* linker options into our mingw-w64
toolchain. Compiler / linker options or defaults that we may rely on,
can no-longer be randomly changing out from underneath us.

To that, some might say “well just don’t change the Ubuntu base image,
and things won’t change”. To that, I’d say, I don’t want the project
to be in a position where we are “stuck”, and can’t update our base
image to use new tools (i.e compilers), because by doing so we’ll just
break something else, like our security tests.

It would seem smarter to just remove the potential for those kinds of
problem entirely, by constructing and using exactly the release build
environment we want.

“Well why not just patch / change things in gitian” – I wont say much
on this, except for, the gitian environment is not at all setup to do
the same kind of patching that we can fairly trivially achieve in
Guix. Trying to patch and compile, for example, the mingw-w64
toolchain in a gitian descriptor, would also just turn into a terrible
mess.

The same can be said about trying to gitian build anything using a
glibc different than what is already available on that version of
Ubuntu.

This has backward-compatibility implications, as the version of glibc
you’re building against, essentially determines your runtime glibc
compatiblity. However this can be extended by using the sorts of
“workarounds” that we currently have in our glibc-back-compat code.

Now, similar to the situation where you might want to use a newer
Ubuntu release (for any number of reasons), doing so will mean that
you have to build against a newer glibc (you just get whatever version
comes with that release of Ubuntu).

This means that as you use newer versions of glibc, the number of
“workarounds” you need to maintain backwards compatibility pile up,
get continually more complicated, and even start to leak out of
Bitcoin Core code, and into our dependency system. See all the PRs
linked in this comment:
https://github.com/bitcoin/bitcoin/pull/22418#issuecomment-876379846.

Changes like that leaking into depends are bad, because now normals
builders, using depends, are now being subject to the side-effects of
patches that are only really necessary in our release build
environment. Maybe you’d argue that in that case we should only apply
the depends patches when building releases, however then you’ve got a
situation where release builds are even more divergent from
“normal”/developer builds, and this means either maintaining an even
more complex CI / testing routines, or they just end up less tested
(guess which one is more likely to happen).

If that all just sounds like a complicated mess, it’s basically
because it is. However the solution, using Guix, is actually pretty
straight forward.

When we are in complete control of our release environment, we can
pick exactly the version of glibc we want to use (even at a per-HOST
level), something that we definitely couldn’t achieve, in any sort of
straight forward fashion, if at all, using gitian.

This is something we’ve done recently, in #22365. We are now building
using glibc 2.27 for the RISC-V HOST, and using glibc 2.24 for all
others, while maintaining runtime compatibility with glibc 2.17.
You’ll note this this is achieved without needing to use any of the
work arounds from our glibc-back-compat code (#22405), or any of the
other PRs / changes mentioned in the comment above.

These are just two very practical benefits that using Guix provides
(there are more), which ultimately all boil down to us being in much
greater control of our release build environment. Something I am very
happy about, and I think makes a lot of sense for a project like
Bitcoin Core.

The fact that there are also many people in the Guix space actively
working on bootstrapability is just another big bonus, and even if
that doesn’t work exactly like some may want it too right now, it is
rapidly being improved, month on month.

Guix is also a much more likely pathway to fully bootstrapable Bitcoin
Core builds that what gitian could ever provide.