A long time ago, before the pandemic, I bought a Librem 5 phone from Purism. I also moved home since then, and sadly my phone was sleeping peacefully in a box in the garage since I moved.
When I was in Hamburg last month, I saw how great Mobian and Phosh was coming along, and this inspired me to go dig up the Librem 5 which was about 2 Debian releases behind, and upgrade it to the latest and greatest version.
I followed the instruction on the Debian wiki, and after some stumbles, managed to flash it with the latest Mobian image:
It’s still a big bulky phone, but Phosh has really come a long way and the phone feels so much more responsive and usable now. It’s also the first time I tried out the new GNOME Console app (I hope they consider taking some features from JuiceSSH so that it’s easier to run apps like mc and irssi on the phone).
Next up I want to try out some progressive web apps and also check what the latest state of emulating Android apps is. I’ve also been meaning to follow some GTK tutorials, and trying out some ideas on a mobile device motivates me a bit more to do that.
It’s really impressive the large amount of work people put into making Debian and a mobile GNOME experience work so well on a phone! Good job to everyone who contributes to this eco-system!
Release management is the process of planning, scheduling, testing and deploying new versions of software. To make this process simpler for snap developers, we have released a new feature called progressive releases. Continue reading to understand what they are, why they are important and how you can use them in the Snap Store.
Progressive releases is a technique used to reduce risk while introducing new software releases. When a new version is released, initially, it is only distributed to a small subset of clients. This enables the developer to check how effective the update was, and ensure that no machines were negatively affected. Now knowing that the release was safe, the update can be distributed to the rest of the clients. This can be triggered in a progressive way (hence the name of the feature).
The developer chooses the percentage of snaps that the update will be distributed to. As the update continues to be successful and the confidence of the developer increases, this percentage can increase until all machines have been updated. This technique of release management will be a main feature of publishing snaps to ensure that the update is safe and effective.
Why are progressive releases important?
Release management is an important practice in software development. Imagine you have thousands of clients that are ready to receive a bugfix. When you prepare the update, it is crucial to know how existing devices will behave with the update. This can often only be found out by testing on devices that are already in the field. In this case, testing on a small number of clients can give you direct insight into the stability and efficacy of the update. Once the developer is certain that the update took place without issue, the release can be distributed to the remaining machines. This can either be done as a complete rollout or in staggered stages.
Another reason why release management is important is linked to the timing of updates. There are times where devices are in operation and should not be updated immediately. For example, you are in the middle of writing an email and suddenly your browser closes for an important update. In those situations, you risk losing your work. In more serious situations, such as updating a server during use, this could result in major interruption in service and loss of revenue. In severe cases, it could even cause damage to the device or bricking. For these reasons, scheduling updates is especially important.
There are many situations where progressive releases are especially important, such as in IoT-related use cases for embedded devices. These machines may be critical in factory lines, healthcare equipment, robotic arms or cars. In all of these cases, updates should only take place when scheduled and safe to do so. Poor release management could lead to serious consequences.
Screenshot of progressive releases for a snap in the Snap Store
Using progressive releases ensures that the released software works correctly on a subset of devices before full deployment. This reduces the risk of unstable updates to improve the procedure of software publishing.
Try them out for your next release
Progressive releases are ready and available to use for your snaps in the Snap Store. The documentation explains the full software release process, including creating, monitoring and finalising your progressive release. Read the full documentation here.
Canonical announced the general availability of Ubuntu’s real-time kernel earlier this year. Since then, our community raised several questions regarding the workings of the kernel and tuning guidelines. We aim to provide answers in this and an upcoming follow-up post.
Depending on your background knowledge, you may wish to start with the basics of preemption and a real-time system. In that case, this introductory webinar or our blog series on what is real-time Linux, is for you.
The present blog post highlights two primary test suites for real-time Ubuntu, followed by an explanation of the components and processes involved, from the scheduler role and its different policies, to blocking spinning locks.
If you are already familiar with a real-time Linux kernel and do not wish to refresh your memory, jump to the tuning guide. There, we will go through the three primary metrics to monitor when tuning a real-time kernel, some key configs set at compile time, and a tuning example.
How to enable the real-time kernel
The real-time kernel is available via Ubuntu Pro, Canonical’s comprehensive enterprise security and compliance subscription (free for personal and small-scale commercial use in up to 5 machines) . With an Ubuntu Pro subscription, launching the kernel is as easy as:
pro attach <token> pro enable realtime-kernel
Canonical also provides specific flavours of the real-time kernel. Optimised real-time Ubuntu, currently in beta, is also available on 12th Gen Intel® Core™ processors. Coupled with Intel® Time Coordinated Computing (Intel® TCC) and Time Sensitive Networking (IEEE TSN), the real-time Ubuntu kernel delivers industrial-grade performance to time-bound workloads.
Real-time Ubuntu on Intel SoCs is available via Ubuntu Pro:
pro attach <token> pro enable realtime-kernel --access-only apt install ubuntu-intel-iot-realtime
Furthermore, since last September, real-time kernel Ubuntu supports Intel’s latest FlexRAN Reference Software, with optimisations to the network stack. As a user of real-time Ubuntu, you can positively impact the open-source community by reporting any bugs you may encounter.
Do you want to run real-time Ubuntu in production?
Real-time Ubuntu 22.04 LTS gets its real-time patches from the upstream project, maintained by the Linux Foundation, whose development closely resembles the standard procedure for the mainline kernel. Kernel developers send patches to add new functionalities or fix bugs to the upstream community via the mailing list. If approved, the maintainers apply them to the real-time patch set in the Git repo. While most of the real-time logic is in the mainline, a relevant portion, especially for locking, is still in a patch set form. Real-time Ubuntu relies on the 5.15-rt patch set, just as Ubuntu 22.04 relies on the 5.15 upstream kernel. The 5.15-rt patches, maintained by Canonical’s Joseph Salisbury, will reach end-of-life in 2026-10.
Canonical’s real-time Ubuntu relies on two primary test suites, rt-tests and stress-ng. rt-tests, maintained upstream in a Git repository, includes oslat and Cyclictest, the primary test suite upstream used to establish a baseline and determine if there is regression. Canonical routinely runs stress-ng every SRU cycle to check for regression and changes in kernel stability as well.
The real-time Ubuntu kernel relies on extensive testing, often in combination. For example, stress-ng to put a load on the system and Cyclictest to measure its latency. Furthermore, Canonical also tests the real-time kernel via partner-provided programs, like Intel’s Jitter Measurement Tool (provided as a package and not upstream).
The role of the scheduler in a real-time kernel
The scheduler is a key component of a real-time system. In the Linux kernel, a few scheduling classes, like Early Deadline First, Real-Time, and the Completely Fair Scheduler, are available, with different scheduling policies within each class, as per the table below:
Scheduling Class
Scheduling Policy
EDF
SCHED_DEADLINE
RT
SCHED_RR
SCHED_FIFO
CFS
SCHED_OTHER
SCHED_BATCH
SCHED_IDLE
IDLE
–
The runqueue contains per-processor scheduling information and is the basic data structure in the scheduler. It lists the runnable processes for a given processor and is defined as a struct in kernel/sched.c.
The scheduler can be run via a call to the schedule() function. The Linux kernel will then sequentially check the EDF, RT and CFS runqueue for waiting tasks. Alternatively, the system will do an idle loop if no tasks ready to run are present.
Early Deadline First in the kernel scheduler
EDF’s scheduling policy, SCHED_DEADLINE, is deadline-based. Hence, after calling the schedule() function, the scheduler will run whichever task in the runqueue is closest to the deadline. Whereas in the POSIX (and RT class) approach, the highest-priority task gets the CPU, the runqueue’s process nearest its deadline is the next one for execution in EDF.
The unintended consequence and potential issue with the SCHED_DEADLINE policy is that in case a task was to miss its deadline, it would keep on running, causing a domino effect with the follow-on tasks also missing their deadlines. When using SCHED_DEADLINE, one must pay close attention to the system and application requirements.
The SCHED_DEADLINE policy of the EDF class specifies three vital parameters for a system, the runtime, deadline and period. The runtime denotes how long a thread will run on a processor. The deadline is the specific period of time during which a task has to complete its operation, usually measured in μs. The period states how often it will run.
Furthermore, two parameters, “scheduling deadline” and “remaining runtime”, initially set to 0, describe the task’s state. After a task wakes up, the scheduler computes a “scheduling deadline” consistent with the guarantee. In other words, when a SCHED_DEADLINE task becomes ready for execution, the scheduler checks if:
remaining runtime / (scheduling deadline – current time) > runtime / period
Real-time in the kernel scheduler
Real-time Ubuntu relies on the RT class, a POSIX fixed-priority scheduler, which provides the FIFO and RR scheduling policies, first-in-first-out and round-robin, respectively. In particular, real-time Ubuntu uses the SCHED_RR policy. SCHED_RR and SCHED_FIFO are both priority-based: the higher priority task will run on the processor by preempting the lower priority ones.
The difference between the FIFO and RR schedulers is evident when two tasks share the same priority. In the FIFO scheduler, the task that arrived first will receive the processor, running until it goes to sleep. On the other hand, in the RR scheduler, the tasks with the same priority will share the processor in a round-robin fashion.
The danger with the round-robin scheduling policy is that the CPU may spend too much time in context switching because the scheduler assigns an equal amount of runtime to each task. One can remediate such downsides by properly tuning a real-time kernel and focusing on how long tasks will run and the type of work they will do.
Completely Fair Scheduler and Idle
Finally, the generic kernel uses the CFS by default, whereas IDLE comes in handy when the system is not performing any action.
Assigning scheduling policies in code
Let’s now get our hands dirty and dive into the code directly. Assigning a task to a specific policy type is relatively straightforward. If using POSIX threads, one can set the policy when calling the pthread_attr_setschedpolicy function, as per the example below with SCHED_FIFO:
int main(int argc, char* argv[]) { struct sched_param param; pthread_attr_t attr; pthread_t thread; int ret;
… … …
ret = pthread_attr_setschedpolicy(&attr, SCHED_FIFO); if (ret) { printf("pthread setschedpolicy failed\n"); goto out; }
… … …
An alternative code snippet with comments:
if (pthread_attr_setschedpolicy(&attr, SCHED_FIFO) != 0) { printf("pthread_attr_setschedpolicy: %s\n", strerror(errno)); exit(1); } /* Always assign a SCHED_FIFO priority below 50-99. * Kernel threads run in that range. * It's never a good idea to use fifo:99 for a realtime application; the * migration thread uses fifo:99 and all the interrupt threads run at * fifo:50. Refer to mail on rt mailing list */ struct sched_param param; memset(¶m, 0, sizeof(param)); param.sched_priority = MIN(5, sched_get_priority_max(SCHED_FIFO));
Another way is to set the policy, whether SCHED_DEADLINE, SCHED_RR or SCHED_FIFO in a sched_attr structure, as for instance in:
In the above, the task will repeat every two seconds and can be up to 11 μs late. The thread function would then be:
sched_setattr(0, &attr, 0); for (;;) { printf("sensor\n"); fflush(0); sched_yield(); };
Another practical piece of code assigns priority to a thread. This can be done by directly passing a priority number:
param.sched_priority = 49; ret = pthread_attr_setschedparam(&attr, ¶m); if (ret) { printf("pthread setschedparam failed\n"); goto out;
Assigning a sensible priority number is particularly important when working with the priority-based round-robin and FIFO policies.
An application should never run at priority 90 or higher, as that is where critical kernel threads run. Similarly, watchdogs and migration run at priority 99. Running a task at priority 99 will likely result in the overall system locking up. Hence, one should strive to set a priority below the range of 50-99 when writing a program.
Locks in a real-time kernel
There are two primary types of locks: blocking locks and spinning locks.
Blocking locks
The primary characteristic of blocking locks is that the tasks holding them can be put to sleep. Among examples of blocking locks there are counting semaphores, (per-CPU) Reader/Writer semaphores, mutexes and WW-mutexes and RT-mutexes. Of those, RT-Mutex is the only blocking lock that will not lead to priority inversion, covered in the following section.
These lock types are then converted to sleeping locks, e.g. local_lock (often used to protect CPU-specific critical data), spinlock_t and rwlock_t, when enabling preemption in a real-time Linux kernel. Further details on locking primitives and their rules are available in the Linux kernel’s documentation.
Spinning locks
Let’s now consider spinning locks. To understand their advantages, it is worth remembering classical spin locks can’t sleep and implicitly disable preemption and interrupts. In turn, this can cause unbounded latencies, which are undesirable for real-time applications, as there is no guaranteed upper boundary of execution time. Furthermore, the lock function may have to disable soft or hardware interrupts depending on the context.
In a real-time kernel, classical spin locks convert to sleepable spinlocks and are renamed raw_spinlocks. Hence, a developer may have to recode their applications and drivers to use raw spinlocks in a kernel with PREMPT_RT, depending on whether or not a spin lock is allowed to sleep.
Among spinning locks, reader/writer locks are also available. In particular, rwlock_t is a multiple reader and single writer lock mechanism. Non-PREEMPT_RT kernels implement rwlock_t as a spinning lock, with the suffix rules of spinlock_t applying accordingly.
Processes and threads
Among the reasons why PREEMPT_RT is not in mainline yet is that much of the locking within the kernel has to be updated to prevent priority inversion from occurring in a real-time environment. The present and the following section will introduce unbounded priority inversion and the need for priority inheritance.
Unbounded priority inversion
Let’s begin with priority inversion by looking at the diagram sketched below. Three tasks, L, M, and H, with varying priority levels, low, medium and high, are present in the kernel and about to contest for CPU access.
The low-priority task L runs until it takes a lock; in the diagram below, the blue bar turns red. After acquiring it, task L holds the lock and begins modifying some critical sections within the kernel. Once the higher-priority task H appears, it preempts task L and starts to run. At this point, task H would like to acquire the same lock task L is holding. As it can’t do so, the higher-priority task H goes to sleep and waits for the lower-priority task L to release the lock. Task L thus continues running while Task H is sleeping. In such a setup, priority inversion can occur if a medium-priority task M comes along and preempts task L. Once task M starts running, the high-priority task H will potentially wait for an unbounded amount of time, preventing it from doing work in a critical kernel section. Improving the flexibility to preempt tasks executing within the kernel would thus help guarantee an upper time boundary.
In this specific example, task M finishes running and releases the CPU – where the horizontal bar turns from green to red in the drawing – allowing task L to start running again while still holding the lock. Only once task L releases it, task H will wake up and acquire the lock, starting its work within the critical section.
Priority inversion occurred on the Mars Rover, and it is a critical challenge for developers and engineers working with real-time systems. With unbounded priority inversion, the need for priority inheritance becomes clear.
Priority Inheritance
A real-time Linux kernel resolves the unbounded latencies of priority inversion via priority inheritance.
The diagram below helps illustrate the mechanism. As before, the low-priority task L starts running and acquires the lock. Similarly to the previous scenario, task H wakes up and starts running, but it is soon blocked while attempting to get the lock.
The high-priority task H wants to take the same lock held by the low-priority task L. Differently than in the priority inversion’s case, and instead of H going to sleep and waiting, priority inheritance occurs, with L acquiring H’s priority. The low-priority task L can now run with the same priority as task H, enabling it to finish its work in the critical section and then release the lock. The inheritance mechanism centres around boosting the lower task’s priority, giving it one higher than the upcoming medium priority task M, which would cause unbounded latencies.
Once task L finishes its critical section work, task H acquires the lock, where the red bar turns orange. Whenever task H completes, it will, in turn, release the lock. Only now can the medium-priority task M come along and start running. If needed, the higher-priority task H could further preempt task M to finish its processing. Priority inheritance in a real-time kernel solves the issue of task M starting to run between tasks H and L, which would give rise to unbounded latencies and priority inversion.
This blog covered the technical foundations of a real-time Linux kernel. It is now time to learn how to improve the determinism of network latency with Linux. A system can be tuned to partition up CPU cores to perform specific tasks – for instance, a developer can assign the housekeeping core to handle all IRQ, including network interrupts using IRQ affinity, isolcpus, cpusets and tasksets. The follow-up post will introduce some relevant tuning configs when using a real-time kernel.
If you are interested in running real-time Ubuntu and are working on a commercial project, you can contact us directly. Canonical partners with silicon vendors, board manufacturers and ODMs to shorten enterprises’ time-to-market. Reach out to our team for custom board enablement, commercial distribution, long-term support or security maintenance.
Hoje começamos o episódio com os Deuses a rirem do Miguel e a ungirem o Diogo de boa ventura e prosperidade. O Tiago encontra-se alheiro, alhures. Houve tempo para dizer mal da Samsung, desabafar sobre aparelhos espertos, soldadura, fumos de resina, gatos terroristas e ainda novidades do Castor Biónico, arroz com salpicão, CUPS em Snap, abacate em tudo e tudo em Snap na Letónia.
Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal.
E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8.
Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem.
Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.
O Diogo está finalmente de regresso da sua holiday trip, com renovado vigor para um rebranding do franchising! Temos histórias inenarráveis, emaranhados de cabos e PDF e actividades possivelmente ilegais. O Fairphone 4 com Ubuntu foi posto à prova…será que passa o teste? De caminho ficámos a saber que há caixas amarelas para montar, aprendemos a arte do salpicão à Italiana e não dissemos mal da Vodafone.
Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal.
E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8.
Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem.
Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.
Linux Matters I recently started presenting Linux Matters podcast with my friends Martin Wimpress and Mark Johnson.
In episode 4 (that link will only work once the episode is released) I briefly talked about some simple bots I setup on the Ubuntu Social Mastodon instance (which, incidentally I talked about in episode 1).
This blog post accompanies episode 4. Linux Matters is part of the Late Night Linux (LNL) family. If you support us on the LNL Patreon, you’ll often get the episode delivered early, as well as advert free.
This year I attended Debian Reunion Hamburg (aka MiniDebConf Germany) for the second time. My goal for this MiniDebConf was just to talk to people and make the most of the time I have there. No other specific plans or goals. Despite this simple goal, it was a very productive and successful event for me.
Tuesday 23rd:
Arrived much later than planned after about 18h of travel, went to bed early.
Wednesday 24th:
Was in a discussion about individual package maintainership.
Was in a discussion about the nature of Technical Committee.
Co-signed a copy of The Debian System book along with the other DDs
Submitted a BoF request for people who are present to bring issues to the attention of the DPL (and to others who are around).
Noticed I still had a blog entry draft about this event last year, and posted it just to get it done.
Had a stand-up meeting, was nice to see what everyone was working on.
Had some event budgeting discussions with Holger.
Worked a bit on a talk I haven’t submitted yet called “Current events” (it’s slightly punny, get it?) – it’s still very raw but I’m passively working on it just in case we need a backup talk over the weekend.
Had a discussion over lunch with someone who runs their HPC on Debian and learned about Octopus and Pac.
Was in a discussion about amd64 build times on our buildds and referred them to DSA. I also e-mailed DSA to ask them if there’s anything we can do to improve build times (since it affects both productivity and motivation).
Did some premium cola tasting with andrewsh
Had a discussion with Ilu about installers (and luks2 issues in Calamares), accessibility and organisational stuff.
Thursday 25th:
Spent quite a chunk of the morning in a usrmerge BoF. I’m very impressed by the amount of reading and research the people in the BoF did and gathering all the facts/data, it seems that there is now a way forward that will fix usrmerge in Debian in a way that could work for everyone, an extensive summary/proposal will be posted to debian-devel as soon as possible.
Mind was in zombie mode. So I did something easy and upgraded the host running this blog and a few other hosts to bookworm to see what would break.
Cheese and wine party, which resulted in a mao party that ran waaaay too late.
Friday 26th:
Daytrip
Was very saddened to learn that the Metallica concert in Hamburg that day was not a daytrip option.
Had a BoF where we had an open discussion about things on our collective minds (Debian Therapy Session).
Had a session on upcoming legislature in the EU (like CRA).
Some web statistics with MrFai.
Talked to Marc Haber about a DebConf bid for Heidelberg for DebConf 25.
Closing session.
Monday 29th:
Started the morning with Helmut and Jochen convincing me switch from cowbuilder to sbuild (I’m tentatively sold, the huge new plus is that you don’t need schroot anymore, which trashed two of my systems in the past and effectively made sbuild a no-go for me until now).
Dealt with more laptop hardware failures, removing a stick of RAM seems to have solved that for now!
Das is nicht gut.
Dealt with some delegation issues for release team and publicity team.
Attended my last stand-up meeting.
Wrapped things up, blogged about the event. Probably forgot to list dozens of things in this blog entry. It is fine.
Tuesday 30th:
Didn’t attend the last day, basically a travel day for me.
Thank you to Holger for organising this event yet again!
You might remember that in my last post about the Ubuntu debuginfod
service I talked about wanting to extend it and make it index and
serve source code from packages. I’m excited to announce that this is
now a reality since the Ubuntu Lunar (23.04) release.
The feature should work for a lot of packages from the archive, but
not all of them. Keep reading to better understand why.
The problem
While debugging a package in Ubuntu, one of the first steps you need
to take is to install its source code. There are some problems with
this:
apt-get source required dpkg-dev to be installed, which ends up
pulling in a lot of other dependencies.
GDB needs to be taught how to find the source code for the package
being debugged. This can usually be done by using the dir
command, but finding the proper path to be is usually not trivial,
and you find yourself having to use more “complex” commands like
set substitute-path, for example.
You have to make sure that the version of the source package is the
same as the version of the binary package(s) you want to debug.
If you want to debug the libraries that the package links against,
you will face the same problems described above for each library.
So yeah, not a trivial/pleasant task after all.
The solution…
Debuginfod can index source code as well as debug symbols. It is
smart enough to keep a relationship between the source package and the
corresponding binary’s Build-ID, which is what GDB will use when
making a request for a specific source file. This means that, just
like what happens for debug symbol files, the user does not need to
keep track of the source package version.
While indexing source code, debuginfod will also maintain a record of
the relative pathname of each source file. No more fiddling with
paths inside the debugger to get things working properly.
Last, but not least, if there’s a need for a library source file and
if it’s indexed by debuginfod, then it will get downloaded
automatically as well.
… but not a perfect one
In order to make debuginfod happy when indexing source files, I had to
patch dpkg and make it always use -fdebug-prefix-map when
compiling stuff. This GCC option is used to remap pathnames inside
the DWARF, which is needed because in Debian/Ubuntu we build our
packages inside chroots and the build directories end up containing a
bunch of random cruft (like /build/ayusd-ASDSEA/something/here). So
we need to make sure the path prefix (the /build/ayusd-ASDSEA part)
is uniform across all packages, and that’s where -fdebug-prefix-map
helps.
This means that the package must honour dpkg-buildflags during its
build process, otherwise the magic flag won’t be passed and your DWARF
will end up with bogus paths. This should not be a big problem,
because most of our packages do honour dpkg-buildflags, and those
who don’t should be fixed anyway.
… especially if you’re using LTO
Ubuntu enables LTO by default, and unfortunately we are affected by an
annoying (and complex) bug that results in those bogus pathnames not
being properly remapped. The bug doesn’t affect all packages, but if
you see GDB having trouble finding a source file whose full path
starts without /usr/src/..., that is a good indication that you’re
being affected by this bug. Hopefully we should see some progress in
the following weeks.
Your feedback is important to us
If you have any comments, or if you found something strange that looks
like a bug in the service, please reach out. You can either send an
email to my public inbox (see below) or file a bug against the
ubuntu-debuginfod project on Launchpad.
I got a new SSD and did a fresh Ubuntu 23.04 install. What I usually do, is connecting the old disk via USB and copy data over from the old disk to the new SSD.
But my old disk used encrypted ZFS. It took me some time to figure out how to mount that that so here’s what I did.
The old disk gets detected as /dev/sda . There are 2 pools, rpool and bpool in my case. rpool is the one that contains my home directory and also the root directory. Let’s import that pool:
# zpool import -f rpool
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 1.81T 1.29T 536G - - 21% 71% 1.00x ONLINE -
After the pool import, there is now the LUKS encrypted keystore available under /dev/zvol/rpool/keystore . That keystore does contain the ZFS key for encrypting/decrypting. so let’s luksOpen that one:
# cryptsetup open /dev/zvol/rpool/keystore rpool-keystore
Enter passphrase for /dev/zvol/rpool/keystore:
MY_SUPER_SECRET_PASSPHRASE
And now the newly created mapper device for the opened crypt device:
# mount /dev/mapper/rpool-keystore /mnt/
# ls /mnt/
lost+found system.key
So system.key is there. Let’s load it so ZFS can use it and clean up:
With zfs list the different datasets can be listed. To mount the /home/$USERNAME database, find the right one, change the mountpoint and mount it (/mnt/home-myuser must be created before):
# zfs list|grep home
rpool/USERDATA/myuser_xq8e3k 1.22T 478G 986G /home/myuser
# zfs set mountpoint=/mnt/home-myuser rpool/USERDATA/myuser_xq8e3k
# zfs mount rpool/USERDATA/myuser_xq8e3k
ls /mnt/home-myuser # this should show the files from your home now
That’s it. The last steps can be repeated to mount any other ZFS dataset (eg. the one for /)
I decided to leave my job (Principal Software Engineer) after 4 years. I have no idea what I want to do next, so I’ve been having loads of chats to try and work that out.
I like working in mission focused organizations, working to fix problems across the stack, from interpersonal down to the operating system. I enjoy “going where I’m rare”, places that don’t always get the most attention. At my last job, I most enjoyed working to drive engineering standards for all products across the company, mentoring engineers across all teams and seniority levels, and serving as an advisor for senior leadership as we grew the engineering team from 3 to 150 people.
If you have a role that you think I’d like to hear about, I’d love to hear about it at jobs{}pault.ag (where the {} is an @ sign).
This is a long retrospective, organized into 3 parts, about my 2023 hike of the Portuguese Camino de Santiago.
Part 1 is about my preparation, packing and gear
Part 2 covers a retrospective of insights gained along the way, and advice for anyone else considering the Camino
And Part 3 is a roll-up of the daily social media posts and accompanying photographs from my two weeks on the trail
Comments are disabled on Blogger due to relentless spam, but you reach me on LinkedIn and Twitter with questions. You're also welcome to follow me on Google Maps, where I contribute frequently and reviewed many of the establishments we visited along this trip. Enjoy!
Part 1: Preparation, Packing, and Gear
I love a good, long walk -- I've had some brilliant hikes across Scotland, New Zealand, Peru, Switzerland, and of course throughout Texas and the United States National Parks. I know my limits -- I'm fit enough to easily walk 10-15 miles a day, with a pack, for multiple days in a row. But I also know that I need to be very comfortable in my shoes, and with my pack, and with the right gear. So I spent about 3 weeks walking around my hometown of Austin, Texas, with my pack, "training". I walked about 5-7 miles in my (pretty hilly) neighborhood, almost every day, and did one 10 miler around Lady Bird Lake trail in downtown Austin. I wouldn't call this "training" per se, but I do think it was pretty valuable acclimation to the mileage and additional weight of a backpack.
All in all, I felt very well prepared throughout the actual hike on the Camino.
More importantly, I was generally pleased with the light weight and balance of my pack. With hindsight, are there a few things I would pack differently? You bet. Here's a spreadsheet I used when packing:
See below for a thorough retrospective on my packing and gear.
Disclosure: As an Amazon Associate I earn from qualifying purchases.
Backpack
I decided to upgrade my vintage, 20+ year old Gregory Reality backpack, to a brand new, ultralight Osprey Exos 38L, and that was a great investment! I kept my fully loaded pack to about 18 lbs with an empty water bladder, or 20 lbs with a fully loaded 2L water bladder. That weight was perfect for me. I hardly noticed it at all. I mean, I couldn't "run", but at no point in the 160 miles did I ever even think about the backpack's weight or comfort. It was just part of me. I also brought the over-pack rain shell, which was useful and necessary a couple of times. For what it's worth, my hiking partner used an Osprey Stratos 36L and he loved his, too.
Water
My Osprey backpack supported a water bladder, and I carried a 2L Camelbak bladder. It was awesome. I swear by it. It was really nice to sip water along the way, any time, without stopping.
Shoes
In my early Camino prep, I also originally figured I'd take my beloved heavy duty leather Hanwag hiking boots. However, doing a little bit of research and reading, it seems that all of the distance through hikers these days have moved to using trail running shoes, of which Hoka Speedgoat and Altra Lone Peak seem to be by far the most popular. I tried both, and really liked the Altras better. (I got a pretty good deal by buying last year's model). I treated them with a couple of applications of Scotch Guard for a little bit of extra water resistance.
REI has both (and some others), and you should probably check them out, if you haven't already. There were plenty of people on the Camino wearing both Hoka and Altra, as well as some Soloman's, a few Brooks, and others. I was quite happy with the Altras, all things said. I ended up with a few blisters on the 18 mile days, but I think that was more a matter of my socks.
Socks
Speaking of socks, I only brought my favorite Bombas ankle socks, which are my all-day, everyday socks at home. (I'm a huge fan of Bombas, and generally buy direct from Bombas.com, as they donate socks for every direct purchase). But in retrospect, I probably should have brought purpose made hiking socks instead.
Alternate Pair of Shoes
Without a doubt, you'll also need a second pair of shoes for the afternoon / evening, after you're done hiking. I brought my Olukai flip flops (which I can easily walk/hike 6+ miles in, no question). I liked flip flops because they were light, and could get wet (I used them in the occasional communal / prison-style showers). My hiking partner used lightweight deck shoes as his 2nd pair and he was happy with that as well. The key point is just that you'll absolutely need something to switch into, after a long day of hiking and you really need something that doesn't require socks.
Clothes
I packed 5 sets of socks, underwear, and shirts (3 long sleeves, 2 shortsleeves), and 3 sets of pants (two long hiking pants, one pair of shorts). We did laundry every 4-5 days. Basically I'd wash 4 sets of clothes, while wearing my last set. Most accommodation has some form of laundry facilities. We chose to use the ones that had both washers and dryers (to get everything done in 90 minutes or so), but most people just hang dry their clothes overnight. Plenty of people travel with just 2 or 3 sets of clothes too, and they do laundry more frequently.
Shirts
The weather was really, really quite perfect for our hike in March/April. Just a bit of chill in the air in the morning (mid 40s-50s Fahrenheit), and warm (high 70s Fahrenheit) most afternoons. As such, I was able to wear long sleeves shirts every day (avoiding some sunscreen on the arms). My go-to shirt for hiking are these Fjallraven Abisko wool shirts, and the Fjallraven sun hoodie version. Light, breathable, extraordinarily comfortable. I also brought one Nike Dri-fit long sleeve zip top, for some variety.
Most mornings (in March and April for us) were chilly (low 40s F), and the afternoons sunny and warm (high 70s F). I'd start most mornings with 1, 2, or all 3 layers, and then stop about every hour to shed a layer. I liked the long sleeves and long pants (albeit very thin and breathable), even the hottest part of the day, for sun protection. I wore sunscreen on my face and neck every day.
I also brought an pair of Frog Tog rain shell pants, which, thankfully I never actually used, as we had great weather. I probably could have done without, but on the whole, I'm glad I brought as the weight and space required were negligible.
Hat / Cap / Sunglasses
I also brought a basic baseball cap and Oakley Frogskins Lite sunglasses. My hiking partner swore by his full brim hiking hat, but I didn't like how the back of my full brim hat brushed the top of my backpack. So, instead I used a Fjallraven knit cap/beanie, which was nice a few mornings, but certainly not necessary at all. I typically shed it within the first hour and could have done without it easily.
Tech and Electronics
Charing electronics (phone, watch) was far easier than I expected. I carried a rechargeable power brick that I never really used and, in hindsight, I didn't need, at all.
My Google Pixel 6a phone and Google Pixel watch batteries (both proactively set to power save mode every day), had more than enough battery every day for each day's hike, even with full GPS tracking enabled, lots of pictures, a fair amount of Google maps for restaurant reviews, Google lens for translations, and Wikipedia for learnings.
I also brought a small, rechargeable Black Diamond headlamp that I totally didn't need either. We only started hiking two days before dawn, and even then there were plenty of street lights. And a phone flashlight would have been plenty enough light. Next time, I wouldn't bother bringing a headlamp.
I did bring a set of Airpods, planning to listen to music or audiobooks, but surprisingly, I never actually used them for even one minute of music or books! Rather, I enjoyed the conversation with my hiking partner, and the peace and quiet and sounds of the trail. That said, they were still handy for calling home and talking to the family, though, so no regrets on bringing them.
Credit cards were good about 80% of the time (mostly NFC tap-to-pay, conveniently), though I did pull cash (about 100-150 euros, three times), with my M1 Finance debit card.
Additional Hiking / Camping Gear
The only important thing I neglected to bring, and actually purchased along the way was a hiking pole. Of course I have great trekking poles at home, but I was unable to find out definitively if I could carry them onto the plane (we definitely did NOT check our backpacks). It sounds like the TSA and airline rules against "blunt force weapons" are sometimes (but very inconsistently) applied to hiking poles and walking sticks? So I did not bring my own, but should have.
As it turns out, I'm sure I totally could have, if I had collapsed it all the way down, and stuffed it entirely inside of the pack, rather than cinching it to the outside where it's visible and accessible. While it's nice to walk with two poles, one is enough, when packing light. After about day three, I was missing a walking stick, and I picked up a wooden one (as did my hiking partner), but it was awkward and hurt my wrist. So I bought one for 15 euros at the first place I saw one for sale. It was nice enough -- collapsible and spring loaded, but certainly not the highest end equipment. It does say "Camino de Santiago" on it, so it's a nice souvenir, and it really helped with the up and down hills on the hike. I had no trouble whatsoever bringing it home on the plane in my carry on backpack, so looking back, I'm sure I could have brought it on very easily.
Regarding Accommodations on the Camino
There's no camping really (maybe just one or two spots), along the Camino, and lodging is readily available, so there's no need for a tent. If you're flexible (quality and cost, in both directions, up and down), you'll never have a problem finding a place to sleep. At the lowest end, there are plenty of first-come, first-serve, free (or nearly free, with a nominal "donation" of 10 euros or so), public "Albergues". These are basically hostels, with 30 or more bunk beds in a room, and communal bathrooms and showers. It's meager accommodations, usually in an old monastery or similar historic building. This is the very "traditional" Camino experience.
Similarly, there are also "private" Albergues, which are very similar bunk beds and bath setups, but they do usually take reservations, and are a little newer (maybe cleaner?), and they are also quite affordable (12-20 euros per bed). We poked our head into 3 or 4 public Albergues, and they were all serviceable, but we chose to stay in a couple of private Albergues, mainly because we could make our reservations a few hours or a day or two ahead of time, and have the peace of mind that we were "booked". Some private Albergues also have "private rooms", which are a really good deal, if you're traveling in a small group of 2-4.
I was traveling with one other friend, and 9 of the 12 nights, we managed to book our own private room (usually two twin beds, and an en suite shower and toilet) for about 40 euros for the two of us -- which we always opted for, over say a 12 euro apiece bunk bed. There are a few private apartments and hotels available too -- these are probably in the 50 euro - 80 euro per night range -- still very affordable. For the region we were in, Booking.com (which has a pretty decent app), was absolutely the way to go. It was easy to find availability, prices, features, addresses, ratings, and communicate with the hosts (with translation capabilities built into the messaging app).
The only thing we pre-booked before we left home, was our first night at the Hilton in Porto, and our last nights, at the Marriott in Santiago. These were obviously much, much higher end accommodations, and we paid much more for those (nearly 200 euro a night). The fancy digs were nice, but totally unnecessary.
Oh, and maybe two nights along the way, we stayed at "farmhouses". These were my favorite accommodations by far. One was at a vineyard, with the traditional home cooked, communal dinner, which we shared with 8 other pilgrims. Truly unforgettable experiences. If I were to do the Camino again, I would absolutely seek these out, though you do have to book these in advance (we just got lucky), as they are extremely limited and very popular.
Related to accommodations, I also brought my own lightweight Marmot 55F summer sleeping bag, and a very tiny Summit Aeros Down Inflatable Pillow. This is a tough one. Strictly speaking, these were not entirely necessary. The sleeping bag was a lot of extra weight (1.5 lbs), and every single place we stayed provided sheets, pillows, and pillow cases. I chose to use my own sleeping bag in the 3 bunk room setups, but I totally could have used the provided sheets. All that said, having my own sleeping bag and pillow was an important piece of mind, in that I knew that I could sleep literally anywhere, as long as I had them. On a subsequent Camino, though, I don't think I'd bring a sleeping bag at all, and instead would just plan on taking accommodations that provide sheets, or bring a very light sleep sack.
Cooking / Meals
I considered bringing my ultralight MSR camping stove (and buying fuel), but ultimately did not. That was a good decision, as that would have been completely unnecessary. There are plenty of cafes and restaurants along the way. Most lunch and dinner spots have a "pilgrim menu" which is a basic package for 10 euros of a drink, appetizer, entree, and dessert -- so super cheap to eat. Of course you can order anything you want on the full menu any time. The wine across Galicia and Portugal is amazing! The red wines are mostly Tempranillo (or similar) and the whites are Albarino (or similar) from the Douro valley, and delicious. You'll rarely pay more than 3 euros per glass Full bottles of red or white wine is readily available for under 10 euros -- and their top shelf super reserva is rarely more than 20 euro a bottle, and it's usually unbelievable stuff.
I did bring a Snow Peak cup and Snow Peak spork. The cup was useful -- I used it every day, and I like to have a glass of water by my nightstand at night. I never used the spork -- it's not needed. I did however bring a tiny corkscrew, which came in handy a couple of times.
Coffee
Probably the one thing I considered bringing, but did not, but REALLY should have, was my Aeropress coffee maker. For no good reason, I took it out of my pack at the very last minute, and I very much regretted it, almost every single day.
Excellent coffee (espressos, cortados, cafe con leche, etc.) is always available at every cafe along the way, and it's really cheap (1 euro typically). But at the hostels and hotels, the coffee is universally awful. There's always a kettle that can boil hot water, and there's usually tea. But almost everywhere has that dissolvable Nescafe instant coffee. I found it just undrinkable.
We found a drip coffee maker in just 1 of 12 of our lodgings, and a pod coffee maker (like a Keurig) in just 1 of 12 as well (and that was a private apartment / airbnb type thing). So most mornings meant stumbling out of bed, packing your backpack, maybe choking down some Nescafe, and then bolting down the trail sans caffeine until the first cafe (which was sometimes packed with pilgrims doing the same thing). The Aeropress would have taken very little space, very little weight, and coffee grounds were easily available at markets and kettles for hot water. Seems like such a trivial thing, and maybe I'm more coffee-focused than most, but this was probably my only significant packing regret.
Towels / Laundry
I did bring a quick-dry camping towel, which I'm glad I had for those 2-3 communal showers, but almost everywhere else provided towels. Strictly speaking, this probably isn't 100% necessary, but it was nice to have.
I had a little drawstring collapsible backpack, which I used as my laundry bag. That was nice, and amounted to a negligible amount of extra weight and space.
First Aid
I also brought a very basic, pocket first-aid kit. I used a couple of band aids, and popped a couple of Ibuprofen at the end of the longest days. I also had a sewing kit, which I didn't use, but I did give the sewing needle away to a fellow pilgrim than needed to pop and drain a terribly infected toenail (gross, you can keep the needle, pal). I popped and drained my own blisters just with my fingernails, which was fine (and provided a ton of relief!), and just wiped those down with alcohol wipes. Oh, I also took some Dramamine for the bus ride back to Porto. Thankfully, I didn't need any of the rest, but I had a couple of anti-diarrheal tablets, some antihistamine, bite and itch cream. My hiking partner also brought some melatonin which came in handy on the plane and for that first night of weird jet lag sleep.
My Camino was truly an amazing experience! If you're even remotely considering it, you'd almost certainly enjoy doing it.
History and Basics
Just some basic Camino history, before I dig in...
The Camino de Santiago (or, The Way of St. James) has been a religious pilgrimage for over a 1000 years, traveled by literally millions of people.
There are dozens of different popular starting points, though you can start your Camino anywhere you choose.
The only requirement to getting your "Compostela" (a parchment certificate that officially recognizes your completion), is that you walk the last 100km to the cathedral of Santiago.
The medieval Camino has its roots around the 900s, and then further popularized by a "guidebook" published in 1140, called the Codex Calixtinus.
Also, St. James (the apostle whose remains are believed to be held at the cathedral of Santiago), is the patron saint of Spain, one of the major sponsors of the crusades, and his popularity rose tremendously in medieval times.
The trail was mostly forgotten in the 20th century (just a few hundred pilgrims a year), until a guidebook was published in 1957, and since then, there has been a huge resurgence of interest.
Of course the 2010 film, The Way with Martin Sheen and Emilio Estevez (which is excellent, by the way), also helped bring the Camino to American audiences and interest has surged. Now, several hundred thousand pilgrims make the trip every year.
Of the dozens of routes, the most popular is the "French Way", which is the Camino depicted in the movie The Way. It's 422 miles, and takes about 5 weeks.
Perhaps the second most popular routes is the Portuguese Way, roughly 160 miles and takes about 2 weeks, which is the path we took.
Our Way
We started in Porto, Portugal, and took the "Central Route", which is inland among the hills and vineyards, but there's also the "Coastal Route", which of course hugs the coast (though the weather can be pretty rough, or so we heard).
Our Camino Portuguese Central was about 160 miles, and we did it in 12 days of walking, averaging about 13 miles (a half marathon) a day, with our longest days around 18 miles and our shortest about 9 miles.
There was some elevation to climb and descend every day, but it was all very reasonable. I found it about the same difficulty as hiking around the Texas hill country near Austin. Hilly, but certainly not mountainous. Nothing like Colorado or Switzerland. But also, it's not all just flat walking.
Most days, we walked for about 3-5 hours, typically starting between 7am and 9am, and finishing around Noon-2pm.
There were other pilgrims who left much earlier than us every day (probably either much slower, or going much farther, or both), and some who left later, or walked later.
When we really pushed, with no stops, our fastest miles were about 17 minutes per mile (as measured by my Pixel watch and my hiking partner's Garmin), though a much more comfortable pace, taking a few pictures, conversing, and relaxing a bit, was closer to ~20 minutes a mile.
Perhaps the most surprising, and maybe most disappointing part about the Camino, is how much of it is on asphalt, concrete, pavement, or cobblestones, and how very little of it is on dirt or rock trails.
There are probably some official numbers somewhere, but I'd estimate that less than 10-15% of our whole Camino was spent on trails, and 85-90% was spent walking on paved streets or cobble stones.
Also, I'd say that at least 65-70% was "urban", walking among buildings, towns, villages, and only about 30-35% in fields or the woods or within vineyards or farms.
That might be fine by you, but I think I was expecting something a little more remote. My favorite trails all time are a little more remote treks through Scotland, New Zealand, Switzerland, along the Appalations or the Pacific Northwest -- the Camino is most certainly NOT that.
But what the Camino is, is a walk through history. Many of those cobblestones I'm complaining about were laid by the Romans (and probably their slaves) literally 2000 years ago.
Most of the Portuguese Camino follows Roman Road XIX (yes, the Romans basically had a numbered interstate system throughout Europe).
There are stone mile markers (literally, "milestones"), carved in Latin, dating to the 1st century A.D.
We crossed probably 20 stone arch bridges built by the Romans over 1500 years ago, and another 20 stone bridges built (or re-built) in medieval times 1000 years ago.
There's just so, so, so much history. Castles, keeps, cathedrals, chapels, aqueducts, olive trees, grape groves -- many over 1000 years old. Let that sink in, and the hard asphalt and pavement do melt away.
Other Pilgrims on the Trail
Our first few days, the crowds were very light. We just 1 other pilgrim on the first day, and less than a dozen per day, for the next 3-4 days.
But as of April 1st, things picked up tremendously.
Part of it, as we just got closer to Santiago, the trail got busier (back to the point that you must complete the last 100 km on foot to receive the Compostela).
The other consideration was that we kind of accidentally started our hike that would put us into Santiago for Easter weekend. While that was an accident on our part, thousands of pilgrims traveling for more pious reasons were very deliberately covering the Camino over the course of Holy Week, and planning to land in Santiago specifically for Easter.
We only met perhaps 5 or 6 other Americans on the trail (a group of 60-something retirees from Rhode Island, and a small group of retired ladies from California).
From most-to-fewer, we met many Portuguese, Spaniards, Brazilians, Germans, and a smattering of French, Brits, Canadians, Taiwanese, Australians, Danes, Czechs, Swiss, and others I'm sure I'm missing.
Interestingly, almost no one ever asks your name -- just where you're from. And from then on, you're mostly referred to by your country (or in our case, states, Texas and Colorado). It's kind of a nice convention.
You end up seeing many of the same people every couple of days, roughly traveling in your cohort from stage to stage. That's pretty fun. Easy to engage, and get into a conversation with anyone, but also easy to keep your privacy and distance.
The vast majority of people we met on the Camino, have done it before, many of them multiple times (like 7 or 8 times or so), and often different routes each time.
Logistics Advice for the End of the Trail
We struggled to find much information about how we were supposed to "complete" our Camino. Where do we go first? The Cathedral? The Pilgrims' Office? Our hotel? Where is the 0-km marker? Thus, we made a few mistakes, and so we'll try to help you out here...
Here's what NOT to do....which is exactly what we did....
We waited about 45 minutes, to enter the cathedral and visit the tomb of St. James and get our passport stamped at the cathedral, which seemed like the sensible thing to do, for pilgrims on the Camino.
HOWEVER, the Cathedral does NOT allow backpacks inside. Thus, we waited 45 minutes to be refused entry because of our backpacks.
So each of us took turns going inside (by ourselves, without a pack), while the other waited with the backpacks.
Unbelievably, there is NO passport stamp at the Cathedral itself! Shocking, but true. We asked all over.
The famous front doors are ONLY open during jubilee years (of which 2023 is not). So instead you enter through the side doors.
If you want to see the front doors (which are roped off), you'll need to buy a ticket and a tour (which we did a day later).
The tomb of St. James is under the altar, and there's another 10-15 minute queue inside of the Cathedral to see that.
Here's what TO DO...
Rather, here's a much better plan... Go straight to the Pilgrim's Office, get your Compostela, then check-in to your hotel or accommodations, drop your pack, clean up, and visit the Cathedral afterwards.
The traditional end of the Camino is an old, worn stone tile, with the scallop shell, right in the geometric center of the plaza in front of the cathedral.
After this, go straight to the Pilgrim's Office, where you'll scan a QR code, and complete a short form to register for your Compostela.
It'll ask you a few things -- your name, your nationality, the starting point of your Camino, and your reasons for embarking on this journey.
Once you've completed this form (it takes 60 seconds or less), you'll get a number on your phone (basically like pulling a paper ticket at the DMV).
In the event that there is a longer wait, there's a beautiful garden within the Pilgrim's complex, which would make for a lovely place to relax while waiting for your number.
Then, there's a line that forms inside the building, for the numbers being called.
We arrived at the Pilgrim's office at 2pm on Good Friday -- one of the busiest days of the year -- and we waited in line for less than 5 minutes.
When your number is called, you'll move into a very busy room that looks just like the DMV, with perhaps a dozen or more booths, each staffed by a helpful person who checks your Camino passport stamps briefly, gives you your "final" stamp, and then prints your compostela.
Optionally, you can also add a second "certificate of distance", and purchase a tube for safekeeping your certificates for a couple of euros.
Also, the website will say that there's a "Pilgrim's Mass" every day. Which is true, except for when it isn't. There was no Pilgrim's Mass on Good Friday or Holy Saturday (so we didn't get to attend one).
Would I do it again?
Big question...would I do it again? Complicated answer.
I can say unequivocally that I enjoyed every minute of it -- the distance, the landscape, the clean air, the weather, the people, the food, the pain, the gain, the history, the bridges, the cobblestones, the churches, the cathedral, the vineyards, the orchards, the farm animals, the rivers, the streams, the waterfalls -- all of it.
But, at the same time, I just feel that there's so much more to see in the world. While I was raised Catholic and appreciate the history and solemnity, as mostly an apostate, the religious parts of the pilgrimage are perhaps a little lost on me. The cathedral is an unbelievable feat of medieval architecture and the history is just mind blowing, but I don't think I get quite as much out of it as someone doing it for the religious act of pilgrimage.
Moreover, personally, I'm a little more drawn toward the beauties of nature, maybe dotted with some architectural and historical storylines. With limited opportunities to travel and hike for weeks at a time away from family and work, I think I'd probably tackle some other trails on my list first, before making another Camino. But, I'd never say never...
Part 3: Itinerary, Narrative, and Pictures
March 25, 2023, Day -1: Travel
Setting out toward Portugal to walk the Camino. Big thanks to the ladies for taking care of everything back home!
March 26, 2023, Day 0: Arrival in Porto
Let's do this! Made it to Porto, a little later than expected due to strikes in France (merci). Picked up our passport and toured the Porto Se (Cathedral), c. 1100AD. Dinner and beers and dessert port wines around town, and we're back at our hotel, ready to set out on our Camino tomorrow.
March 27, 2023, Day 1 of 13: Porto to Vilarinho, 18.4 miles
Perfect weather on our first day, sunny and cool in the morning, slightly warm in the afternoon. This will probably be one of our longest days, as it just took a while to get out of the city and suburbs of Porto. The walk was nice, but very urban. Mostly hard, cobblestone streets, lots of lots of vehicle traffic, so not ideal trail conditions. We stopped at the 10 mile mark for beers and sandwiches, then kept moving. We peeked in one aubergue (public hostel for pilgrims) and one other private one, before finding a good match here in Vilarinho. It's not very crowded at all on the trail yet. We've only met 4 other pilgrims on the trail so far -- a young lady from Taiwan, an older couple from France (on their 15th Camino) and a guy from Brazil. We seem to be very early in the season.
March 28, 2023, Day 2 of 13: Vilarinho to Barcelos, 18.2 miles
Another big day, 18.2 miles, some rolling hills, from Vilarinho to Barcelos. It finally feels like we are out of the city and into the countryside. About half of today was about 2/3 pavement, 1/3 trails. We met about 10 other pellegrino's today -- a Brazilian, a Portuguese, 2 Czech, 2 Swiss, 2 Danes, and 2 Canadians. We crossed three different stone bridges built about 1000 years ago (possibly incorporating structures from the Roman era 2000 years ago). Barcelos is our destination for the night, a classic medieval village and home for weary pilgrims for a millennium. Like them, we kicked off our shoes and enjoyed cold beers, delicious wine, and good food. Bom cominho!
March 29, 2023, Day 3 of 13: Barcelos to Sao Simao, 13.5 miles
13.5 miles, from Barcelos to tiny vineyard outside of Sao Simao. Best day on the Camino so far, by far! Much shorter day, we walked a little over 13 miles, mostly on dirt trails through farms and groves and vineyards. We saw a few more pellegrino's today, perhaps a dozen, including a few of the same from the day before. We've covered a few different miles with two different mother/daughter pairs, one Canadian and the other Danish. All of us are more or less in the same cohort, on roughly the same schedule. Our first choice of lodging was totally booked and so we walked an extra couple of miles to our second choice and grabbed the last room. As it turned out, it was an awesome result, as we spent this evening at a hostel within a vineyard, drinking all the wine we could from this vineyard, and eating a communal meal with the other pilgrims here. We shared a table with our Canadian friends, as well as some Brazilians, Germans, and Portuguese. Wonderful day, even better night.
March 30, 2023, Day 4 of 13: Sao Simao to Ponte de Lima, 8.5 miles
About 8.5 easy miles to Ponte de Lima. Absolutely wonderful day. Seems like we're finally done with walking along high traffic motor highways and onto trails through fields, farms, vineyards, and orchards. We crossed several stone bridges built by the Romans, reinforced and then widened in the middle ages. These bridges are 1000 to 2000 years old. Spectacular. Nice, easy day, cool, fair weather, good breeze, cloudy and not too much sun. The cork oak trees are beautiful, and some of these olive trees have been growing since Roman times.
March 31, 2023, Day 5 of 13: Ponte de Lima to Rubiaes, 12.3 miles
Ponte de Lima to Rubiaes. Best day of walking so far! A little over 12 miles but the most elevation gain we should see over the entire hike, all in all really not that intense though. We were on beautiful trails and dirt roads for the vast majority of the day. The pellegrino traffic has picked up considerably, now seeing a few dozen pilgrims per day, plus mountain bikers. We've very quickly gone from walking up to our hotels and just getting a room at the end of a long day, to having to have a reservation a day or two in advance. We met our first Americans all week -- a trio of retirees from Rhode Island. There were some amazing waterfalls and mountain streams, and lush, moss and fern covered walls, in addition of course to the vineyards, olive trees, orchards, and cork oaks. The ground was a little moist, but we generally missed the rain again today. A few more Roman era (1st century AD) bridges and roads and water fountains, and a church built in 1295 that is a national monument in Portugal.
April 1, 2023, Day 6 of 13: Rubiaes to Tui, 13 miles
Really, really nice day, about 13 miles, all the way across the border from Portugal into Spain. We got our first taste of slightly damp weather. The forecast was 'rain' but it was more like a very light fog or light drizzle. The last town at the border, Valenca, was a beautiful medieval town with a proper wall and castle and moat. We walked the ramparts and stumbled on a Renaissance festival and/or passion of the Christ where I had the best crepe ever, cooked over an open flame. We're staying in a hostel outside of town though we walked a few miles back into town to have drinks and dinner. I played goalie for a couple of kids in a Saturday night futbol match and kept a clean sheet (like 9 saves or so, if I do say so myself)...
April 2, 2023, Day 7 of 13: Tui to O Porrino, 10.5 miles
A little over 10 miles from Tui to O Porinño. Really hard to believe, but the journey is well over half way over. We gained a full day by walking extra our first 2 days, so we'll probably spend an extra day at the end in Santiago de Compostela. Today's walk from Tui was very, very, VERY different. There were over a hundred at least pellegrino's setting out from Tui for Santiago. The Camino was packed with pilgrims. Very different. It seems that the pellegrino traffic picks up considerably in the last 100km or so. This was also our first day out walking before sunrise, which made for some great pictures. In any case, we had a nice short walk today, along an ancient, 2000 year old Roman road, across a couple of Roman era bridges, into the town of O Porinño, where we had the best meals of our journey. Because our distance was shorter (and the time change moving into Spain) we arrived much earlier, and enjoyed the best meals of our trip so far. We did some laundry, had a nap and a siesta, and then a great evening too. Hard to believe we are wrapping this up so soon.
April 3, 2023, Day 8 of 13: O Porrino to Redondela, 9.9 miles
Just about 10 miles today, from O Porriño to Redondela, and then another 4 or so miles exploring the town and area. The trail started out somewhat crowded but thinned out a bit mid morning. We climbed about a 1000 foot hill and back down towards the end of today's walk, which afforded some beautiful views of the valley and the water. Our hostel for the night is right next door to a small cheese and meat and meticulously curated bottle shop, where we enjoyed charcuterie plates twice today, plus a delicious Palo Cortado sherry, a tawny port, and a couple of interesting beers. Very enjoyable day!
April 4, 2023, Day 9 of 13: Redondela to Pontevedra, 14.2 miles
From Redondela to Pontevedra, it was a solid 14 mile walk, with a few more hills than we were expecting. Still, very fair weather and a really beautiful day. We opt for a couple of slightly longer, slightly more scenic "complementario" routes, hoping to avoid some of the crowds. Speaking of...it's crowded now. The Camino is a highway of pilgrims trying to make it into Santiago for Easter. Of course we walked along the ancient Roman XIX road (yes, the Romans numbered their major roads across Europe like Interstate highways in the US), and several Roman era bridges, including a beautiful one in Arcade. We also had a magical experience at a bakery in Arcade recommended by our butcher friend a town or two back. In Pontevedra, we visited an extremely well curated art museum with several Velasquez, Goya, and other Spanish masters. But the antiquities really shine, with a couple of gold and silver hordes from the area, dated to almost 3000 years ago. Lots of history and natural beauty today.
April 5, Day 10 of 13: Pontevedra to Caldas de Reis, 14.2 miles
Roughly 14 miles, from Pontevedra to Caldas de Reis, mostly flat with a couple of minor hills, beautiful, fair weather, cold in the morning, warm in the afternoon. The Camino for traffic is in full force from here on in. Hundreds upon hundreds upon hundreds of pellegrino's now cover the trails. Young, old, native, foreign, walking with kids, with dogs, with backpacks and without, and a few annoying groups who think that you want to listen to their Bluetooth speaker for 4 hours. Everyone is on the trail now! Most of today was spent on nice trails or small alleys and away from highways. Caldas de Reis is an interesting little town, surrounding a network of hot springs which have been in continuous use since the time of Ptolemy and the Romans. We dunked our feet a public hot spring that has served pilgrims for literally a millennia!
April 6, Day 11 of 13: Caldas de Reis to Padron, 12.2 miles
Second to last day hiking, we covered a little over 12 miles from Caldas de Reis to Padròn today, mostly flat terrain, and very fair, favorable weather. Crowds are heavy -- according to the Camino website, about 1500 pellegrino's are arriving per day. We enjoyed some nice time in the forest, along trails and streams, the usual Roman roads and bridges of antiquity, and plenty of sunshine. The food and drink of Galicia are most delightful, from the sweet pastries for breakfast, breads and meats and cheeses for lunch, and the many varieties of fermented beverages. And the locals are just so incredibly friendly. Every proprietor and shop owner and bar tender treats you like you are the single most important person in the world. In fact, we had drinks this evening at Pepe's place in Padron, where every patron gets a giant bear hug and a kiss on the cheek from Pepe, after you sign his book. This is timeless and precious.
April 7, 2023, Day 12 of 13: Padron to Santiago de Compostela, 17.5 miles
A long, hard uphill final day of walking, 17+ miles from Padròn to Santiago de Compostela, our destination. We left our albergue well before dawn, knowing that we had a tough day ahead of us, and a lot of pilgrims doing the same walk, expecting to arrive in Santiago on Good Friday in time for Easter. Our earliest start of the whole trip, it was cold and dark when we started, but also quiet and beautiful and less crowded. We saw both a full moon set and a sun rise, and zipped through trails and small towns. We stopped once for a brief breakfast and picked up sandwiches to go, which came in handy 10 miles later. Once the sun came up, it got hot quickly, and the crowds picked up too. The last few miles into the city were less scenic and more urban, but ticked away pretty quickly. Upon arriving, we weren't sure if the protocol, so we went straight to the cathedral where we waited in a line for a half hour and were turned away because backpacks aren't allowed in the church. So we each took turns, watching one another's packs outside, and doing a quick tour of the church. Then we went to the pilgrim's office a few blocks away and learned that we didn't even have to visit the cathedral to claim our certificate, the Compostela, but we did anyway, so..bonus points? We paid a few euros to get our Latin inscribed parchment (pretty cool actually), and then checked into the hotel, and cleaned up. Then we headed back to the cathedral to actually attend the "pilgrim's mass", which happens every day, and they read the names of all the pilgrim's that arrived that day. Except when we got there, we were told that "every day" doesn't include "today" because today's Good Friday, and there's no mass. And every day doesn't include tomorrow either. Or Sunday. So come back on Monday, two days after you leave. At this point, the religious part is lost on me, but it was a mind clearing, spiritual journey none the less. Anyway, amazing walk! Spectacular weather. Great company. Delicious food. Good people. I'm tired, but refreshed. Challenging, but rewarding! Exhausted, but energized. Appreciative of this experience, but also ready for home with my girls. I'll have a bit more to say later in retrospect, but for now, "bom comiño!"
April 8, 2023, Day 13 of 13: Day of Rest in Santiago de Compostela
This is the first day in over two weeks that we haven't had to pack our backpacks and check out of our lodgings and head to the next town. Rather, we enjoyed a nice day touring the beautiful town of Santiago de Compostela, starting with the museum at the cathedral, which included 4 levels of history around the construction of the structure, art, and artifacts. After that, I think I had the most unique tour I've ever experienced, period... We took a guided tour (albeit entirely in Spanish), whereby we spent almost 90 climbing and crawling around THE ROOF OF THE CATHEDRAL. Yes, you heard me... There's a tour where you basically climb out of a window and look at the cathedral from every possible angle, on the damn roof! This was simple indescribable and incredible. Maybe a little unnerving and uncomfortable at times, but a heck of a lot of fun and a one of a kind experience, never to be missed. The roof was renovated in 2021 and this experience is only recently available. Tomorrow morning is an early bus ride back to Porto, then I fly back through Amsterdam to my girls, and I'm very much looking forward to being home. I'll post again a bit more about the logistics of the trip for those considering something like this, and maybe something a little more introspective about what I found along the way. The way.
April 9, 2023, Day 14 of 13: Back in Porto for a day
Just a short bonus post...we spent Easter Sunday commuting by bus from Santiago back to Porto. It was kind of amazing to watch the 12 little towns we spent our night tick by, about every 15 minutes by the highway. 3 different times the highway touched the Camino and pellegrino's were obvious with their backpacks and very determined walks. Porto was a buzz with Easter festivals and activities. Corey and I had a nice meal and a couple of fantastic ports, which put a nice finish on a great trip. We even made our way back to the Porto Cathedral, to the start of our journey and the mile marker that says "248 km to Santiago." To the next pilgrims, bom comiño...
That's all, folks! I'll leave you with this one...bom caminho, good way, buen camino!
Some time ago, before the world locked down, I pondered that KDE wasn’t very good at getting our apps to our users. We didn’t even have a website that listed our apps with download links. If you were an open source app developer using our tech (Qt and KDE Frameworks) would you come into KDE to build your app or just start a project on Github and do it yourself? KDE has community which means some people to help look over your work and maybe contribute and translate and some promo and branding mindshare and there’s teams of people in the distros who specialise in packaging our stuff. But successful projects like Krita and Digikam and indeed my own Plasma release scripts still have to do a lot on top of what KDE communally gives them.
So I launched apps.kde.org and made the All About the Apps goal which was selected in the hope of getting KDE to support taking our apps to the users more slickly. I didn’t manage to make much progress with the goal which I will readily take the blame for. After some fighting I managed to get our announcements linking to the app stores directly but I didn’t manage to get much else slicker.
What my dream still is would be for apps to have a button that…
Bumps the version number in the source
Makes the tar and uploads it to a secret place
Tells the Linux distros to package it
Packaging for Windows/Mac/Android/Snap/Flatpak/Appimage would be in the Git repo and our CI would now build them and upload to the relevant test sites
OpenQA style tests would be in the git repo and our CI would now test these packages
Another button would make the source and packages public in Microsoft Store/Appimagehub/SnapStore/Flathub/download.kde.org and somehow tells the Linux distros and send the announce to the Discuss group and start off a blog post for you
I just released KDE ISO Image Writer (another project I didn’t make much progress with for too many years) and had a chance to see how it all felt
There’s no nice buttons and while we have a tool to make the tar and I have a tool to add the release to the AppStream file, there’s nothing to bump version numbers in cmake or add releases to AppStream or make templates for pre-announcements and announcements.
How’s the packaging and app store situation?
Windows and Microsoft Store
I had to go out and buy a laptop for this, there’s virtual machines available for free which should work but I didn’t trust them with the hardware needed here and they’re time limited so I’m a bit wary of setting up Craft then having to do it again when the time runs out. Craft does a lot of the hard work building for Windows and binary-factory and elite Craft dev hvonreth is often around to give help.
Getting access to the Microsoft Store takes a sysadmin request and working out what to ask for then working out what to upload. I uploaded the wrong thing (a .appx file) when it should have been a .appxupload file and that seemed to break the MS Store from accepting it at all. After lots of twiddling and deleting and generally turning it off and on again I got it uploaded and a day later it was rejected with the claim that it crashed. While the app had installed and run fine for me locally using this .appxupload thing to install it locally did indeed cause it to crash. We diagnosed that to the elevated privileges needed and after some Googling it turns out the Microsoft Store doesn’t seem to support this at all. So my dream of having it available to install there has not worked out, but you can get the installer from download.kde.org and use that.
There’s still only 9 KDE apps on the MS Store at a quick “KDE” search which seems far too few.
AppImage
These have been around for decades and KDE has always had fans of this format (it used to be called Klik at one point e.g. test KOffice). SUSE devs were a big fan at one point. In recent years its gained auto-update, daemons to manage the system integration, build tools, support from top apps like Krita and Digikam and a centralised place to get it in AppimageHub (not to be confused with the other AppimageHub). And yet mass adoption seems as far off as ever.
There’s two ways I found to build it, with appimage-builder which was easy enough to pick up and make a packaging file which uses packages from Ubuntu and neon.
Or you can reuse Craft (used earlier for Windows) to build on Linux for the AppImage. This also allows binary-factory integration but I don’t seem to have got this working yet. It might also be worth exploring openSUSE’s OSB which might allow for other platforms.
I tried to upload it to AppimageHub but that broke the website which needed some back channel chats to fix. Once uploaded it appears shortly, no further bureaucracy needed (which is a bit scary). It doesn’t appear on the KDE Store which seems to be about themes and addons rather than apps. And I put it on download.kde.org.
It’s hard to know how popular AppImage is within KDE, neither of the AppImageHubs seem easy to search and many apps publish their own in various ways. There’s about a dozen (non-Maui) KDE apps with appimages on download.kde.org plus a dozen Maui apps which are developed within KDE and used by the Nitrux distro. I hear complains that AppImage doesn’t support Wayland which will limit them.
Flatpak and Flathub
This format has lots of good feels and mindshare because it integrates well with the existing open source communities.
The flatpak-manifest.json file can be added directly to the repo (which I’m very jealous of, when I suggested it for Snaps it was rejected and caused me to grump off the whole Goal) and that can be added to binary-factory but also to invent.kde.org CI. There’s an active team around to help out. That gets uploaded to a KDE testing repo where you can install and test.
But to get it out to the users there’s a separate process for Flathub the main host for Flatpak packages. That takes a another week or two of bureaucracy to get published (bureaucracy when publishing software for people to install is necessary and important). There’s also a stats website which suggests it has 300 installs.
This works the smoothest if I say so myself. Add the packaging to the snapcraft repo and it builds on the invent.kde.org CI which actually just sends it off to the launchpad builders and it builds for ARM and AMD64. Then you get one of the KDE Snapcraft team (Scarlett, me, Harald) to register it and voila it uploads to candidate channel for testing. It needs manually moved into the stable release channel which can either be done by our team or we can share admin rights. The bureaucracy comes when you need to ask for permissions such ISO Image Writer needing access to disks, that took a week to be accepted. The packages are build using KDE neon for Qt and KDE Frameworks etc and we’ve had troubles before when KDE neon moves onto new versions of Qt but the content Snap has stayed on older ones, but we’re working out when to save a spare snapshot of it. The build tool Snapcraft also has a kde-neon extension which just adds in common parts used by KDE snaps but sometimes that gets out of date too so we’ve had to work out ways around it.
The Snapcraft KDE page has about 140 apps. From the admin page I can see ISO Image Writer has 920 installs around the world (not bad for two days old). The store doesn’t seem great at picking up the AppStream meta data so screenshot and icons are often out of date which I’ve brought up with the devs a bunch of times. It’s centralised around a single Canonical owned store which open source/free software fans can find a bad smell but it is what users want.
Others
I’ve not looked at f-droid, Google Play, Chocolately, or Apple’s App Store. With the probable exception of Apple’s store we should embrace all of these.
I couldn’t find any tools to add release data (the files to download) to AppStream file which is what ends up on apps.kde.org, that feels like a low-hanging-fruit fix. Building pre-release tars which aren’t available publicly seems tricky to do, we have that for KDE neon but none of the app stores have it. Similarly tools to make templates for release announcements can’t be hard, I do that for Plasma already.
So lots of work still to do to make KDE have slick processes for getting our software out there to the users, it’s social and technical challenges and cultural shifts take a long time. Loads of people have put in lots of work to get us where we have today but still lots to do. If you’re up for a challenge and want to help I hope this blog shows the challenges and potential for fixing them rather than sounding too negative. Let’s keep KDE being All About the Apps!
Signed by me Jonathan Esk-Riddell <jr@jriddell.org> E0A3EB202F8E57528E13E72FD7574483BB57B18D https://download.kde.org/stable/kommit/kommit-1.0.2.tar.xz.sig
I’ve never been a fan “errexit” in shell. You’ve probably seen this as set -e, or set -o errexit or sh -e.
People write lists of shell commands in a file and want the script to exit on the first one that fails rather than barreling on and causing damage. That seems sane.
I’ve always strived to write “shell programs” rather than “shell scripts”. The difference being that the program will clean up after itself and give sane error messages. It won’t just exit when mkdir fails and leave the user to understand some message like:
mkdir: cannot create directory ‘/tmp/work/bcd’: No such file or directory
I’ve always felt that errexit makes “good error handling” in shell more difficult.
If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status. If a compound command or shell function sets -e while executing in a context where -e is ignored, that setting will not have any effect until the compound command or the command containing the function call completes.
Here’s an example of how painful it can be, and I took way too long today tracking down what was wrong.
A Programmer starts off with a simple script make-lvs and uses set -e.
This looks fine for a “shell script”. The -x argument even makes shell write to standard error the commands it is running. At this point everyone is happy.
Later the programmer looks at the script and realizes that he/she needs more flags to lvm lvcreate. So now the script looks like:
I’m happy that the programmer here used long format flags as they are much more self documenting so readers don’t have to (as quickly) open up the lvm man page. It is easier to make sense of that than it is to read ‘lvm lvcreate --ignoremonitoring -y -ay -ky -L1G -n mylv’.
After doing so, they realize that they can make this look a lot nicer, and reduce the copy/paste code with a simple function wrapper.
The improvements are great. The complexity of the lvm lvcreate is abstracted away nicely. They’ve even dropped the vile set -x in favor of more human friendly messages.
Output of a failing lvm command looks like this:
$ make-lvs; echo "exited with $?"
Creating lv mylv0 on vg myvg of size 1G
out of space
exited with 1
The next improvement is where sanity goes completely out the window. [I realize you were questioning my sanity long ago due to my pursuit of shell scripting perfection].
The programmer tries to add reasonable ‘FATAL’ messages that you might find in log messages of other other programming languages.
#!/bin/bash -e
lvcreate() {
echo "Creating lv $2 on vg $1 of size $3"
lvm lvcreate "--size=$3" --ignoremonitoring --yes --activate=y \
--setactivationskip=n --name="$2" "$1"
echo "Created $1/$2"
}
fail() { echo "FATAL:" "$@" 1>&2; exit 1; }
lvcreate myvg mylv0 1G || fail "Failed to create mylv0"
if ! lvcreate myvg mylv1 2G; then
fail "Failed to create lylv1"
fi
echo "Success"
Can you guess what is going to happen here?
If the lvm command fails (perhaps the vg is out of space) then the output of this script will look like:
$ make-lvs; echo exited with $?
Creating lv mylv1 on vg myvg of size 1G
error: out of space
Created myvg/mylv1
Creating lv mylv2 on vg myvg of size 1G
error: out of space
Created myvg/mylv2
Success
exited with 0
The attempt to handle the failure of the lvcreate function with || on lines 10 and with if ! on line 12 made it a “compound command”. A compound command disables the error handling and shell exit that would have come from -e when the lvm command on line 4 failed.
Above I’ve demonstrated with bash, but this is actually posix behavior, and you can just as well test the function with sh as well.
Madness.
If you’re interested in further reading, you can see this topic on the BashFAQ. I agree with GreyCat and geirha: “don’t use set -e. Add your own error checking instead.”
If you’re still here, the following is the version of the script that I’d like to see. Of course there are other improvements that can be made, but I’m happy with it.
#!/bin/bash
info() { echo "$@" 1>&2; }
stderr() { echo "$@" 1>&2; }
fail() { stderr "FATAL:" "$@"; exit 1; }
lvcreate() {
local vg="$1" lv="$2" size="$3"
info "Creating $vg/$lv size $size"
lvm lvcreate \
--ignoremonitoring --yes --activate=y --setactivationskip=n \
--size="$size" --name="$lv" "$vg" || {
stderr "failed ($?) to create $vg/$lv size $size"
return 1
}
info "Created $vg/$lv"
}
# demonstrate both 'command ||' and 'if ! command; then' styles.
lvcreate myvg mylv0 1G || fail "Failed to create mylv0"
if ! lvcreate myvg mylv1 1G; then
fail "Failed to create mylv1"
fi
info "Success"
Rust is a hugely popular compiled programming language and accelerating it was an important goal for Firebuild for some time.
Firebuild‘s v0.8.0 release finally added Rust support in addition to numerous other improvements including support for Doxygen, Intel’s Fortran compiler and restored javac and javadoc acceleration.
Firebuild’s Rust + Cargo support
Firebuild treats programs as black boxes intercepting C standard library calls and system calls. It shortcuts the program invocations that predictably generate the same outputs because the program itself is known to be deterministic and all inputs are known in advance. Rust’s compiler, rustc is deterministic in itself and simple rustc invocations were already accelerated, but parallel builds driven by Cargo needed a few enhancements in Firebuild.
Cargo’s jobserver
Cargo uses the Rust variant of the GNU Make’s jobserver to control the parallelism in a build. The jobserver creates a file descriptor from which descendant processes can read tokens and are allowed to run one extra thread or parallel process per token received. After the extra threads or processes are finished the tokens must be returned by writing to the other file descriptor the jobserver created. The jobserver’s file descriptors are shared with the descendant processes via environment variables:
Since getting tokens from the jobserver involves reading them as nondeterministic bytes from an inherited file descriptor this is clearly an operation that would depend on input not known in advance. Firebuild needs to make an exception and ignore jobserver usage related reads and writes since they are not meant to change the build results. However, there are programs not caring at all about jobservers and their file descriptors. They happily close the inherited file descriptors and open new ones with the same id, to use them for entirely different purposes. One such program is the widely used ./configure script, thus the case is far from being theoretical.
To stay on the safe side firebuild ignores jobserver fd usage only in programs which are known to use the jobserver properly. The list of the programs is now configurable in /etc/firebuild.conf and since rustc is on the list by default parallel Rust builds are accelerated out of the box!
Writable dependency dir
The other issue that prevented highly accelerated Rust builds was rustc‘s -L dependency=<dir> parameter. This directory is populated in a not fully deterministic order in parallel builds. Firebuild on the other hand hashes directory listings of open()-ed directories treating them as inputs assuming that the directory content will influence the intercepted programs’ outputs. As rustc programs started in parallel scanned the dependency directory in different states depending on what other Rust compilations finished already Firebuild had to store the full directory content as an input for each rustc cache entry resulting low hit rate when rustc was started again with otherwise identical inputs.
The solution here is ignoring rustc scanning the dependency directory, because the dependencies actually used are still treated as input and are checked when shortcutting rustc. With that implemented in firebuild, too, librsvg’s build that uses Rust and Cargo can be accelerated by more than 90%, even on a system having 12 cores/24 threads!:
Firebuild accelerating librsvg’s Rust + Cargo build from 38s to 2.8s on a Ryzen 5900X (12C/24T) system
On the way to accelerate anything
Firebuild’s latest release incorporated more than 100 changes just from the last two months. They unlocked acceleration of Rust builds with Cargo, fixed Firebuild to work with the latest Java update that slightly changed its behavior, started accelerating Intel’s Fortran compiler in addition to accelerating gfortran that was already supported and included many smaller changes improving the acceleration of other compilers and tools. If your favorite toolchain is not mentioned, there is still a good chance that it is already supported. Give Firebuild a try and tell us about your experience!
All builds took place on the same Ryzen 5900X system with 12 cores / 24 threads in LXC containers limited to using 1-12 virtual CPUs. A warm-up build took place before the vanilla (without any instrumentation) build to download and compile the dependency crates to measure only the project’s build time. A git clean command cleared all the build artifacts from the project directory before each build and ./autogen.sh was run to measure only clean rebuilds (without autotools). See test configuration in the Firebuild performance test repository for more details and easy reproduction.
Firebuild had lower overhead than sccache (2.83% vs. 6.10% on 1 CPU and 7.71% vs. 22.05% on 12 CPUs) and made the accelerated build finish much faster (2.26% vs. 19.41% of vanilla build’s time on 1 CPU and 7.5% vs. 27.4% of vanilla build’s time on 12 CPUs).
The Xubuntu team is happy to announce the immediate release of Xubuntu 23.04.
Xubuntu 23.04, codenamed Lunar Lobster, is a regular release and will be supported for 9 months, until January 2024.
Xubuntu 23.04, featuring the latest updates from Xfce 4.18 and GNOME 44.
Xubuntu 23.04 features the latest Xfce 4.18. Xfce 4.18 delivers a stable desktop environment with a number of performance improvements and new features to enjoy. In particular, the Thunar file manager benefits from a new image preview feature, undo and redo functionality, file highlights, and recursive search. Check out the Xfce 4.18 tour for more details!
Xubuntu 23.04 also welcomes Xubuntu Minimal as an official subproject. Xubuntu Minimal is a slimmed down version of Xubuntu that only includes the bare essentials: the desktop, a few Xfce components, and the Xubuntu look and feel. Longtime Xubuntu fans may better know this as Xubuntu Core. After nearly eight years of being a supported, but community-built project, we’re happy to finally publish downloads along with the main Xubuntu version. Many thanks to the community for keeping the dream alive all these years!
The final release images are available as torrents and direct downloads from xubuntu.org/download/.
As the main server might be busy in the first few days after the release, we recommend using the torrents if possible.
We’d like to thank everybody who contributed to this release of Xubuntu!
Highlights and Known Issues
Highlights
Xfce 4.18, released in December 2022, is included in Xubuntu 23.04.
Xubuntu Minimal is included as an officially supported subproject.
Pipewire (and wireplumber) are now included in Xubuntu.
Known Issues
The shutdown prompt may not be displayed at the end of the installation. Instead you might just see a Xubuntu logo, a black screen with an underscore in the upper left hand corner, or just a black screen. Press Enter and the system will reboot into the installed environment. (LP: #1944519)
The screensaver unlock dialog crashes after unlocking. The session can still be locked and unlocked after this crash. We’re working on a fix and hope to publish it in the next few weeks. (LP: #2012795)
Xorg crashes and the user is logged out after logging in or switching users on some virtual machines, including GNOME Boxes. (LP: #1861609)
For more obscure known issues, information on affecting bugs, bug fixes, and a list of new package versions, please refer to the Xubuntu Release Notes.
The main Ubuntu Release Notes cover many of the other packages we carry and more generic issues.
Support
For support with the release, navigate to Help & Support for a complete list of methods to get help.
We are pleased to announce the release of the next version of our distro, the 23.04 release for both the desktop and Raspberry Pi4. This is a standard release supported for 9 months packed full of all sorts of new capabilities. If you want a well tested and longer term support then our 22.04.2 LTS version is supported for 3 years. New with this release is a full backport of all the budgie...
Ubuntu Budgie 23.04 (Lunar Lobster) is a normal release with 9 months of support, from April 2023 to January 2024. Ubuntu LTS releases are focused on long term support. If stability is more important than having the latest and greatest version of the kernel, desktop environment, and applications then Ubuntu Budgie 22.04 LTS is perfect for you. In these release notes, we are going to cover the...
Up front I shall say: that was OK. Not amazing, but not bad either. It could have been cringy, or worthy, and it was not. It struck a reasonable balance between being overly puffed up with a sense of epic self-importance (which it avoided) and being campy and ridiculous and mocking all of us who are invested in the game (which it also avoided). So, a tentative thumbs-up, I suppose. That’s the headline review.
But there is more to be considered in the movie. I do rather like that for those of us who play D&D, pretty much everything in the film was recognisable as an actual rules-compliant thing, without making a big deal about it. I’m sure there are rules lawyers quibbling about the detail (“blah blah wildshape into an owlbear”, “blah blah if she can cast time stop why does she need some crap adventurers to help”) but that’s all fine. It’s a film, not a rulebook.
I liked how Honour Among Thieves is recognisably using canon from an existing D&D land, Faerûn, but without making it important; someone who doesn’t know this stuff will happily pass over the names of Szass Tam or Neverwinter or Elminster or Mordenkainen as irrelevant world-building, but that’s in there for those of us who know those names. It’s the good sort of fanservice; the sort that doesn’t ruin things if you’re not a fan.
(Side notes: Simon is an Aumar? And more importantly, he’s Simon the Sorcerer? Is that a sly reference to theSimon the Sorcerer? Nice, if so. Also, I’m sure there are one billion little references that I didn’t catch but might on a second or third or tenth viewing, and also sure that there are one billion web pages categorising them all in exhaustive detail. I liked the different forms of Bigby’s Hand. But what happened to the random in the gelatinous cube?)
And Chris Pine is nowhere near as funny as he thinks he is. Admittedly, he’s playing a character, and obviously Edgin the character’s vibe is that Edgin is not as funny as he thinks he is, but even given that, it felt like half the jokes were delivered badly and flatly. Marvel films get the comedy right; this film seemed a bit mocking of the concept, and didn’t work for me at all.
I was a bit disappointed in the story in Honour Among Thieves, though. The characters are shallow, as is the tale; there’s barely any emotional investment in any of it. We’re supposed, presumably, to identify with Simon’s struggles to attune to the helmet and root for him, or with the unexpectedness of Holga’s death and Edgin and Kira’s emotions, but… I didn’t. None of it was developed enough; none of it made me feel for the characters and empathise with them. (OK, small tear at Holga’s death scene. But I’m easily emotionally manipulated by films. No problem with that.) Similarly, I was a bit annoyed at how flat and undeveloped the characters were at first; the paladin Xenk delivering the line about Edgin re-becoming a Harper with zero gravitas, and the return of the money to the people being nowhere near as epically presented as it could have been.
But then I started thinking, and I realised… this is a D&D campaign!
That’s not a complaint at all. The film is very much like an actual D&D game! When playing, we do all strive for epic moves and fail to deliver them with the gravitas that a film would, because we’re not pro actors. NPCs do give up the info you want after unrealistically brief persuasion, because we want to get through that quick and we rolled an 18. The plans are half-baked but with brilliant ideas (the portal painting was great). That’s D&D! For real!
You know how when someone else is describing a fun #dnd game and the story doesn’t resonate all that strongly with you? This is partially because the person telling you is generally not an expert storyteller, but mostly because you weren’t there. You didn’t experience it happening, so you missed the good bits. The jokes, the small epic moments, the drama, the bombast.
That’s what D&D: Honour Among Thieves is. It’s someone telling you about their D&D campaign.
It’s possible to rise above this, if you want to and you’re really good. Dragonlance is someone telling you about their D&D campaign, for example. Critical Role can pull off the epic and the tragic and the hilarious in ways that fall flat when others try (because they’re all very good actors with infinite charisma). But I think it’s OK to not necessarily try for that. Our games are fun, even when not as dramatic or funny as films. Honour Among Thieves is the same.
I don’t know if there’s a market for more. I don’t know how many people want to hear a secondhand story about someone else’s D&D campaign that cost $150m. This is why I only gave it a tentative thumbs-up. But… I believe that the film-makers’ attempt to make Honour Among Thieves be like actual D&D is deliberate, and I admire that.
This game of ours is epic and silly and amateurish and glorious all at once, and I’m happy with that. And with a film that reflects it.
Many of us are exiting another drab, grey winter season. As Spring ramps up and the colors of the world awaken all around us, maybe now is the time to make your Xubuntu desktop just a bit more colorful. Since Xubuntu uses GTK, you can quickly spice up your writing with emoji by using the Ctrl + . keyboard shortcut.
Xubuntu supports emoji out of the box, albeit, of a less colorful variety.
Oh, that&aposs disappointing. We have emoji support, but it&aposs all monochrome. While expressive, without color they lack life and don&apost always convey the proper meaning. The good news is that there is a quick and easy fix.
Install Color Emoji Support
The monochrome emoji come from the fonts-symbola package that&aposs included with Xubuntu. If you install the fonts-noto-color-emoji package, the emoji picker will automatically upgrade to the full-color set. In Xubuntu, you can use the following button or command to install this package.
Afterwards, restart any running applications and the emoji picker will be refreshed, no restart required.
Color emoji are much more interesting to look at and can be easier to understand.
Affected Applications
GTK 3 and 4 graphical apps support the on-demand emoji picker (Ctrl + .) and embedded color emoji.
GTK 3 and 4 terminal apps display color emoji.
KDE apps, from what I can tell, do not take advantage of the color emoji without some additional configuration. If anybody knows what that is, I&aposd love to find out.
Firefox and Thunderbird will display the color emoji in most instances. Chromium browsers tend to have better support.
Bonus: Use emoji in any app
Once you start using emoji, you might be disappointed at the number of apps you use that are not native GTK, and therefore do not support the keyboard shortcut. Don&apost despair, there is another solution. Emote, available in the Snap Store and GNOME Software. Once you install and launch Emote, a new keyboard shortcut is enabled. Type Ctrl + Alt + E to show the app. Click your emoji and it will be inserted into your app.
Emote adds a handy interface for adding emoji in any app.
The Ubuntu OpenStack team at Canonical is pleased to announce the general availability of OpenStack 2023.1 Antelope on Ubuntu 22.04 LTS (Jammy Jellyfish).
The Ubuntu Cloud Archive for OpenStack 2023.1 Antelope can be enabled on Ubuntu 22.04 by running the following command: sudo add-apt-repository cloud-archive:antelope
The Ubuntu Cloud Archive for 2023.1 Antelope includes updates for:
From a pricing perspective, the platform scales with your audience size. Importantly, Ghost does not take a cut of your membership subscription revenue. For this reason, the pricing incentives mean your Ghost site is more cost-effective when you have 100% paid members and 0% free members. I'm sure that ratio is a pipe dream for most publications, but that's how the business model works. Pricing is currently $9/mo for any site with 500 or fewer subscribers, including web hosting.
For context, the new blog that I started in 2023 using Ghost is www.pastadollar.com. The site was originally an email newsletter from my generation of 14 cousins investing together. If you're looking for finance tips and travel hacks, Subscribe here! An email-list-to-ghost-site is a common path for Ghost users. I'm just starting, but major publications like The Atlantic, The Lever, and The Browser use Ghost.
Positives
Ghost has only a handful of themes, but on the whole, they are super clean and approachable. Picking one is as much about your content as it is about which one you list best. All of the themes are responsive and have all the standard customization options like colors and logos. Notably, there is no simple way to use custom fonts. Instead, you can choose between a serif font and a sans-serif font. There is a workaround, but the need for a long list of fonts is a relatively notable omission from a CMS or theme these days.
The Ghost editor has a learning curve, but on the whole, it is excellent. I'm used to WordPress's editors, which are total crap by comparison. There are several Ghost editor keyboard shortcuts worth looking up, such as Shift+Enter to jump to the following line but to remain in the same editor block. Hitting Enter will jump to a new block, skipping more lines. The editor auto-saves your work as you go, just as you would expect any web-based tool to do these days. The saving experience is much like google docs - automatic and out of your way. However, sometimes when I haven't changed anything, a modal pops up asking me if I'm certain I want to navigate away without saving—maybe just a bug, but a bit annoying.
Many tasks that take multiple steps in a WordPress site are automatic in Ghost. For example, when you add an image to a post or page, Ghost will optimize it. There is no more need to use a plugin or a site like tinypng.com to optimize the image in advance because the platform doesn't do it for you.
Ghost was designed with plenty of best practices. For example, the default post URL is the post's title, without any year or dates included. Including dates is possible on other platforms but can quickly cause 404s and redirection issues if/when you update a post. The Ghost developers took the best practice, implemented it as default, and left it to you to simply create the content. It's great.
Negatives
Plugins
Ghost has a great library of plugins, which they accurately call 'integrations.' The catch is that most of these are integrations with other major companies' SaaS products, meaning many of them are paid solutions. The company Ghost seems to rely on the most to expand its functionality is Zapier. Half of the help and support articles I've read on Ghost seem to using Zapier to solve a problem. I don't think it's a stretch to say that Ghost relies heavily on Zapier, a freemium SaaS product. So basically, once you set it up and grow your site, it will eat into your bottom line. That isn't necessarily bad, provided it's worth it. But it seems like the solutions to expand your Ghost site are mostly paid solutions rather than something custom via code injection, which Ghost also supports. To be fair, I need to explore code injection solutions more than I have as of this writing.
Quality of Life Issues
URLs are a big part of any site, so I'm amazed this is an issue: When creating a hyperlink, there isn't proper data validation on the link. If you omit the 'https://www' from a new hyperlink, the Ghost editor saves it, but the URL will not actually work. Saving a link like "pastadollar.com" will show as a broken link and not work when you publish the post. This behavior is incredibly annoying because when you're in your site admin, many URLs around the dashboard omit the https://. To ensure your internal URLs will work, I've solved this by always copying them from the live site.
Bullets and Numbered lists aren't an easy option I've found when using the editor. I finally found them hidden in an editor block called markdown editor card. The markdown block supports some rich text features that literally anyone who uses a computer is accustomed to. It isn't exactly user-friendly to hide them in a block, but I'm sure there was a good reason for doing so. Another simple workaround I used when I started was to simply copy and paste a bulleted or numbered list from a proper text editor. I'm guessing many writers create whole drafts outside the Ghost editor anyway.
Alternatives
Alternativeto.net lists more than 250+ alternatives to Ghost, but that long list includes broader solutions like WordPress. Like probably anyone who has developed a website in their lives, I've used WordPress (and continue to use it to this day). WordPress, by default, needs a lot of help from its vast plugin library to set it up, similar to how Ghost works out of the box. I don't consider WordPress a proper alternative if your goal is primarily an email newsletter and membership site. Ghost is absolutely worth the cost over WordPress in that regard - it's not even close.
Beehiiv
In my opinion, the most similar and direct competitor to Ghost is a platform called Beehiiv. Beehiiv has less favorable pricing, and the themes and designs are uglier, but it boasts a stronger feature set out of the box. For example, two big ones are subscriber referral rewards for sharing with friends and more customizable email templates. To be clear, you can probably add these features to a Ghost site; it just takes some work. Both Ghost and Beehiiv are designed to scale quickly with your audience size.
Conclusion
Ghost support has been both highly responsive and extremely helpful. For example, I reached out about getting custom domain emails, such as support@mydomain.com, and their guidance was perfect. Their < 24-hour replies led me to many paid and free services and cautioned me on a few 'gotchas' others ran into. These were not canned responses, and I feel like I can go back to them for anything, and we will be able to find a solution.
Overall, I love Ghost. The admin interface is spotless and fast. I'm coming primarily from WordPress, and the speed difference is a game-changer. I would work from many tabs simultaneously in WordPress because switching between different interface pages took too many seconds to load. The Ghost admin pages usually take less than a second to load.
Comments
Add a comment via Gitlab. And yes, Ghost has much nicer integrated commentng built-in.
Russell published an interesting post about his first experience with Firebuild accelerating refpolicy‘s and the Linux kernel‘s build. It turned out a few small tweaks could accelerate the builds even more, crossing the 10 second barrier with Linux’s build.
Build performance with 18 cores
The Linux kernel’s build time is a widely used benchmark for compilers, making it a prime candidate to test a build accelerator as well. In the first run on Russell’s 18 core test system the observed user+sys CPU time was cut by 44% with an actual increase in wall clock time which was quite unusual. Firebuild performed much better than that in prior tests. To replicate the results I’ve set up a clean Debian Bookworm VM on my machine:
Compiling Linux 6.1.10 in this clean Debian VM showed build times closer to what I expected to see, ~72% less wall clock time and ~97% less user+sys CPU time:
$ make defconfig && time make bzImage -j18
real 1m31.157s
user 20m54.256s
sys 2m25.986s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real 2m3.948s
user 21m28.845s
sys 4m16.526s
# second run
real 0m25.783s
user 0m56.618s
sys 0m21.622s
There are multiple differences between Russell’s and my test system including having different CPUs (E5-2696v3 vs. virtualized Ryzen 5900X) and different file systems (BTRFS RAID-1 vs ext4), but I don’t think those could explain the observed mismatch in performance. The difference may be worth further analysis, but let’s get back to squeezing out more performance from Firebuild.
Firebuild was developed on Ubuntu. I was wondering if Firebuild was faster there, but I got only slightly better build times in an identical VM running Ubuntu 22.10 (Kinetic Kudu):
$ make defconfig && time make bzImage -j18
real 1m31.130s
user 20m52.930s
sys 2m12.294s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real 2m3.274s
user 21m18.810s
sys 3m45.351s
# second run
real 0m25.532s
user 0m53.087s
sys 0m18.578s
The KVM virtualization certainly introduces an overhead, thus builds must be faster in LXC containers. Indeed, all builds are faster by a few percents:
$ lxc launch ubuntu:kinetic kinetic-container
...
$ make defconfig && time make bzImage -j18
real 1m27.462s
user 20m25.190s
sys 2m13.014s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real 1m53.253s
user 21m42.730s
sys 3m41.067s
# second run
real 0m24.702s
user 0m49.120s
sys 0m16.840s
# Cache size: 1.85 GB
Apparently this ~72% reduction in wall clock time is what one should expect by simply prefixing the build command with firebuild on a similar configuration, but we should not stop here. Firebuild does not accelerate quicker commands by default to save cache space. This howto suggests letting firebuild accelerate all commands including even "sh” by passing "-o 'processes.skip_cache = []'” to firebuild.
Accelerating all commands in this build’s case increases cache size by only 9%, and increases the wall clock time saving to 91%, not only making the build more than 10X faster, but finishing it in less than 8 seconds, which may be a new world record!:
$ make defconfig && time firebuild -o 'processes.skip_cache = []' make bzImage -j18
# first run:
real 1m54.937s
user 21m35.988s
sys 3m42.335s
# second run
real 0m7.861s
user 0m15.332s
sys 0m7.707s
# Cache size: 2.02 GB
There are even faster CPUs on the market than this 5900X. If you happen to have access to one please leave a comment if you could go below 5 seconds!
Scaling to higher core counts and comparison with ccache
Russell raised the very valid point about Firebuild’s single threaded supervisor being a bottleneck on high core systems and comparison to ccache also came up in comments. Since ccache does not have a central supervisor it could scale better with more cores, but let’s see if ccache could go below 10 seconds with the build times…
firebuild -o ‘processes.skip_cache = []’ and ccache scaling to 24 cores
Well, no. The best time time for ccache is 18.81s, with -j24. Both firebuild and ccache keep gaining from extra cores up to 8 cores, but beyond that the wall clock time improvements diminish. The more interesting difference is that firebuild‘s user and sys time is basically constant from -j1 to -j24 similarly to ccache‘s user time, but ccache‘s sys time increases linearly or exponentially with the number of used cores. I suspect this is due to the many parallel ccache processes performing file operations to check if cache entries could be reused, while in firebuild’s case the supervisor performs most of that work – not requiring in-kernel synchronization across multiple cores.
It is true, that the single threaded firebuild supervisor is a bottleneck, but the supervisor also implements a central filesystem cache, thus checking if a command’s cache entry can be reused can be implemented with much fewer system calls and much less user space hashing making the architecture more efficient overall than ccache‘s.
The beauty of Firebuild is not being faster than ccache, but being faster than ccache with basically no hard-coded information about how C compilers work. It can accelerate any other compiler or program that generates deterministic output from its input, just by observing what they did in their prior runs. It is like having ccache for every compiler including in-house developed ones, and also for random slow scripts.
Qemu can allow you to attach to a serial console of your guest.
This is done with either -nographic or -serial mon:stdio.
Its great for getting text output, but after you’ve logged in,
you may find issues editing long command lines or using an editor
such as vi.
To fix this, you simply need to tell linux what the terminal
size you’re using is.
First, figure out what your terminal size is. You have to do this
before connecting to qemu, or some other way such as another tmux
window or terminal.
As a scuba diver who often explores new places, I can say that I have found
myself in some dangerous situations, but I always made it back to the surface
without facing any negative consequences. Does this mean that I never made any
mistakes? Absolutely not: mistakes were made, and lessons were learned.
We can all agree that learning from mistakes is good, but sometimes, when
mistakes happen and consequences don’t manifest themselves immediately, we run
the risk of not noticing them, not learning from them, repeating them, and over
time developing a false sense of confidence, which can drive us to believe that
our repeated mistakes are actually good practices.
Why do we ignore mistakes? Because sometimes outcomes are positive even if we
make mistakes. “I made it out of water even this time, this means that my dive
was executed perfectly.” This is a common way of reasoning, but in reality,
things are much more complex than that. There is a difference between correct
execution and successful outcome, and the two should not be confused. In
fact, everyone should know from experience that goals can be achieved even if
the execution was sloppy and full of mistakes. Catastrophic consequences may
happen if we fail to see that.
An example of the consequences of ignoring mistakes is given by the two space
shuttle disasters: the Challenger disaster of
1986, and the
Columbia disaster of
2003. Both
these instances were caused by NASA leadership ignoring the concerns from the
engineering teams. Problems that occurred in previous shuttle launches should
have been a wake-up call for NASA leadership. Instead, all the previous
successful launches and re-entries despite the problems were seen as
accomplishments, and nourished the leadership’s overconfidence. “We have made
it this time too, this means that all those concerns that engineers raised were
excessive.”
The tendency of diverting from proper procedures, dismissing valid concerns,
and ignoring problems, has a name: it’s called normalization of
deviance. The driving
force of normalization of deviance is overconfidence and the false belief that
positive outcomes are inherently caused by correct executions.
Overconfidence and normalization of deviance can spread like a virus in an
organization. It is important to be vigilant for signs of overconfidence in
individuals, before it infects other people. I once had to deal with a manager
who was a self-declared micromanager (and proud to be) but lacked technical
foundations and knowledge of the product. He would consistently and quickly
dismiss anything that he did not understand, and focus on short-term goals of
questionable usefulness. Whenever his team would accomplish a goal, he would
send a pumped-up announcement, often containing inaccuracies, and carefully
skipping over the shortcomings of the solutions implemented. Given the apparent
success of this management style, other managers started to follow his example.
Soon after (in less than a year), the entire organization became a toxic
environment where raising even the minimal concern was seen as an attack on the
“great new vision”.
I see many parallels between this manager story and what is happening with
‘Twitter 2.0’ right now (although, I must say, in my case engineers did not get
fired on the spot for speaking the truth). And with that manager, just like
with ‘Twitter 2.0’, whenever problems occurred, those problems would either be
ignored or blamed on the preexisting components built before the manager
joined, never on the new, careless developments.
The truth however was that problems that occurred had been preannounced weeks,
or months before, but concerns around them had been promptly dismissed due to
being too challenging to address, and because “everything works right now, so
that’s not a concern”.
The idea that everything must be correct because everything works, goals are
achieved, and outcomes are successful, is a dangerous idea that can potentially
have catastrophic consequences. It’s important to be critical and analytical,
regardless of the outcome. This does not mean that success shouldn’t be
celebrated, but that mistakes should be captured so that lessons can be learned
from them, even if the final outcome was successful. Not learning from
mistakes does not allow us to advance, and on the contrary can only lead us to
repeat them. And if we keep repeating the same mistakes, sooner or later, those
will have some negative consequences.
A common practice in the aviation industry is to write reports on incidents,
close calls, and near misses, whenever they occur, even if the flight was
concluded successfully and no injuries or damages occurred. These reports are
collected in databases like the Aviation Safety Reporting
System (which can be freely consulted online), so
that flight safety experts and regulators can identify common failure scenarios
and eventually introduce mechanisms to improve safety in the aviation industry.
A key element of these reports is that they are not meant to put the blame on
certain people, but rather focus on what chain of events led to a certain
mistake. “Human mistake” is generally not a valid root cause: if a human was
able to make a mistake, it means that a mechanism is missing that can either
prevent the mistake or detect it before it causes any negative consequences.
Some companies in other industries have similar processes for writing reports
or retrospectives when a mistake happens (regardless of the outcome), with the
goal of finding proper root causes and preventing future mistakes. Amazon with
its Correction of
Error
practice is a famous example.
I think introducing these practices in an organization can help to establish a
healthy culture where finding mistakes and raising concerns is encouraged,
rather than being oppressed. However these practices, by themselves, may not be
enough to ensure that such a culture can be maintained over time, because
people can always disagree on what is considered a ‘mistake’. Empathy is
probably the key to a truly healthy culture that allows people to learn and
advance.
There are also cases where we are aware of problems, and we see them as such,
but we deliberately choose not to do anything about them. This is where
resilience comes into play.
Resilience is generally a good quality to have. Resilience can give us the
strength to go through long-term hardships, and can have positive effects on
our tenancy and determination. But even resilience, when taken to the extreme,
can be dangerous. Resilience can lead us to ignore problems, and not react to
them. Resilience can make us tolerate a negative situation, without finding a
proper strategy to cope with it.
Poor planning forces you to consistently work extra hours? Resist and keep
going, until you burn out. The relationship with your partner doesn’t satisfy
you? Resist and think that things will get better, while the relationship
slowly deteriorates. Feel pain in your knee every time you run for more than 30
minutes? Resist and don’t go to see a doctor, the pain will go away, sooner or
later… until you cannot run anymore.
When we let resilience become an excuse to avoid solving problems, we can end
up in situations from which it’s difficult to recover.
It’s important to make a distinction between what is under our control and what
is not. We can fix problems that are under our control, but in situations where
we cannot directly change the course of things, finding an alternative strategy
is the only way. Resisting and hoping that things will get better often does
not give the expected outcome–on the contrary, it can be detrimental.
In the end, I think that the ‘practice’ of ignoring mistakes (because of the
overconfidence built from successful outcomes) or ignoring problems (because of
resilience taken to the extreme) are hidden time bombs, silently ticking,
waiting for the right conditions before exploding. We need to be aware that
just because things seem to work today, it doesn’t mean that we’re making the
right decisions, and this can have consequences in the future. Being critical,
analytical, empathetic, and honest is important to avoid these behaviors and
the dangers that come with them.
About four months ago, a friend of mine got me in touch with his employer at the time. I was in a bit of a bind, using rather old and slow hardware (a 3rd gen mobile i5) for my development work, and was hoping to get a newer, faster system to help make things less painful. My friend and his boss managed to get me hooked up with a bug bounty - in exchange for fixing one of several high-priority bugs in the KDE Desktop Environment, I would receive a new Kubuntu Focus system, either a KFocus XE laptop (which is what I ultimately chose), or a KFocus NX desktop. I jumped at the chance, and managed to play a significant role in resolving two of the bugs. I also got a much better computer.
A Kubuntu Focus XE laptop displaying my personal system’s screen.
That was about four months ago. And if you’ve ever gotten a glimpse of my workflow, you know just how much damage… er… testing I can do in four months. I do not treat my systems all that kindly. After having gone through the trials of over seven terabytes of SSD writes, thousands of software compiles, probably over a hundred VM installs, and three distrohops, I finally feel that I can give this system a proper review. For those of you who have been looking into the Kubuntu Focus systems, this should help you get an idea of what you can expect from them, and for those who are on the hunt for a good Linux laptop, this should hopefully help you choose a good one.
First impressions
The Kubuntu Focus XE is essentially a nicely modded Clevo laptop with Kubuntu 22.04 LTS and a suite of utilities and extra features. For those of you who don’t already know this, Kubuntu is an official flavor of the Ubuntu Linux operating system, using the KDE desktop rather than the GNOME desktop. I personally have always been a huge fan of KDE (in fact it was the desktop that came with the first “real” distro I ever used), which made the idea of contributing to KDE to get a laptop even more appealing. The Kubuntu installation that comes with the Kubuntu Focus has a ton of value-adding customizations, including hardware optimization, curated apps and libraries, a very efficient setup wizard, built-in file versioning and recovery, a special Quick Help Widget, and several custom-made utilities for changing special hardware and software settings that are difficult to change otherwise.
The actual laptop is quite thin and light compared to many older systems, and features quite a few I/O ports of various types, including a Gigabit Ethernet port (which is somewhat rare among modern laptops). The keyboard features backlighting, which is always a nice touch if you do a lot of typing, and the cooling system uses dual fans, which looks cool and does a good job of keeping the system cool while under heavy load. Other systems I’ve used would get quite toasty if you really hammered them, while this one gets warm in just one spot right where the CPU itself is located.
For specs, the system I received came with an i5-1135G7 processor, 32 GB of 3200 MHz RAM, and a 1 TB Samsung 970 EVO PLUS solid state drive. 32 GB of RAM is a godsend when you’re doing really intensive work (like running four or more VMs at the same time, something I’ve been known to do in the past). The SSD’s size and speed has also been quite welcome, especially since a lot of what I do involves copious amounts of disk writes.
All in all, things looked pretty impressive initially. The final verdict, however, would depend on how it performed in real-world use.
Kompilation
Having done some KDE development work, it seemed only fitting to use the new machine for KDE development. But not normal KDE development. I wasn’t going to just hack on some source code and do a few builds to test my changes.
One of the projects I maintain is KForge, which is a KDE development virtual appliance that has all of the needed build and runtime dependencies for almost all KDE software preinstalled. This is quite useful as it has been historically difficult to set up a fully functional KDE development environment due to missing build dependencies. You usually don’t know you’re missing a dependency until a build fails, and sometimes the logs left by the build failures are significantly less than helpful.
However, getting KForge to work just right was tricky for the same reason it’s tricky to get a decent KDE build environment going. So, if I was going to make sure I was pulling in all of the dependencies for all of the KDE software, I was going to have to compile almost everything in KDE.
By “almost everything”, I mean 300+ packages.
I ended up deciding to exclude some KDE applications from the build (namely Krita and Kdenlive, both of which I felt would just add too much bulk to the build and wouldn’t add all that much value), and there was one component I ended up excluding since it was defunct and wasn’t maintained anymore. But the remaining 300+ pieces of software needed built. The idea was that, if I got all the dependencies right, and if all of the KDE development branches were in buildable states, I would be able to successfully build everything, and that would tell me that I had succeeded at finding all of the dependencies.
Needless to say, compiling over 300 applications is virtually always going to be a large task even if the apps are somewhat small. Not only were some of these apps not small, many of them were juggernauts (especially KWin, the window manager used by KDE).
While the system’s fans do ramp up significantly and can be quite loud during compilation, the system itself actually remains responsive during such a job. Responsive enough for me to just keep using it while the compilation happens in the background. This was somewhat amazing, since my old system was known for randomly freezing when it got hit with a lot of disk writes, and so to be able to hammer the system like this in the background and still have it be usable during that time was massively helpful. Not to mention the dramatically reduced compile times compared to my older system.
Multitasking
Part of me isn’t sure why I do this, but I have a tendency to try and work on two or three projects at once, sometimes more. As a result, I generally have a slew of apps open all over the place that I switch between rapidly. As I write this part of the review, I’m review-writing, chatting on both IRC and Matrix, and trying to reproduce a bug in Lubuntu, all at the same time. As a result, a well-streamlined DE is very helpful.
KDE is, for me, very good at making work easy. But the Kubuntu Focus takes things a step further and makes it easy to access virtual desktops, something that Kubuntu usually doesn’t have enabled by default. Virtual desktops let me break up my work into different “screens”. The KFocus has four virtual desktops enabled out of the box, essentially giving me four screens at the same time, all on one screen. Not only that, but I can switch between virtual desktops with Ctrl+F1, Ctrl+F2, etc, allowing me to switch desktops without even having to click on the virtual desktop I want.
Additionally, the lots of RAM my system came with made multitasking comfortable from a performance perspective - lag was very rare and the times it happened, it was very small, and only happened during things like a massive compile job. (And even that rarely caused any lag.)
On the hardware side of things, multitasking is also helped by a good keyboard and touchpad - hard-to-press keys and a jumpy or insensitive touchpad can seriously hamper productivity. The keyboard on the KFocus XE is really nice and allows me to type at 100 WPM without problems (and that’s with a cat on my lap and distractions nearby - I could very likely go faster in an ideal situation). The mousepad was initially tricky since it’s so big and I would keep accidentally brushing it with my hand - this ended up becoming a non-issue with time as I got used to holding my hands away from it (which is probably better for avoiding carpal tunnel too). Oh, and the mousepad is also quite precise, responsive, and works well even when I’m doing things at high speeds. The only other problem I’ve noticed is that the touchpad will occasionally stop responding for no apparent reason. This happens infrequently, but when it happens, I’m able to simply switch to a TTY by pressing Ctrl+Alt+F3, then return to my desktop with Ctrl+Alt+F1 - this makes the touchpad work again.
Graphics
I am not a gamer, and the KFocus XE is not designed for heavy gaming. I have no clue how well this thing performs if you’re intending to run intensive 3D games on it. That being said, graphics are important even in the general productivity world to some degree.
The Kubuntu Focus XE boasts a fairly good Intel iRIS XE iGPU, and while Intel’s integrated graphics are not known for being the pixel-crunching behemoths that NVIDIA and AMD churn out, they’re generally great for light and moderate uses. That was exactly my use case.
Everything I threw at the machine graphics-wise worked swimmingly. Full HD video playback is free of stutters and screen tearing. Windows have no stuttering when dragging them (even with the window transparency and wobbly windows that the OS has enabled by default, which look really cool and smooth). Desktop animations are nice and smooth everywhere.
For 3D performance, I tried using thirdroom.io (which is basically an open-source Metaverse-like thing) in Chrome. My fans got a bit louder, but other than that it seemed to perform OK, though it was somewhat laggy. All of the graphics rendered relatively smoothly. The graphics performance for the Intel iGPU is listed on the KFocus website as being similar to the NVIDIA GeForce MX250 - benchmarks show the Intel iGPU as being as good or better than the MX250 in many tests and only slightly lacking in others.
The other important part of the graphics experience is the screen. Here I have one gripe… and before I describe it, I should say, I’m pretty sure this is my fault - my friend has not had this problem, and I’ve not seen any other reports of it. The one problem is screen burn-in. If stuff around the edges of the screen stays in the same spot too long, and then I turn the screen off and leave it that way for a while, I can see shadows of the old screen contents later, sometimes quite well. The reason I think this is my fault is because it didn’t happen at first, and then I left the laptop upside-down trying to improve its cooling airflow during a large compile job. I left it in this state overnight… and when I opened the laptop in the morning, I could see burn-in. Facepalm. I confirmed with my friend that KFocus XE is not intended to be left upside-down like that, and that I may have damaged the screen in so doing. Which would explain why the burn-in happens most near the edges of the screen and much less in the middle. Thankfully, the burn-in mostly goes away on its own if I leave the screen on long enough, and I can use the LCD Scrub screensaver from XScreenSaver to help it out (which may not even be necessary). Other than that, the screen seems to have survived my accidental mistreatment without problems.
Apart from my most likely self-inflicted burn-in issue, the screen is amazing. Colors are bright and crisp, and details are easily visible even when they’re small thanks to the Full HD resolution. The closest thing I’ve had to a monitor this good was a ViewSonic Full HD monitor, and both of the ones I have are dead. Every other Full HD monitor I have either has horrible color or something wrong with the backlight - the KFocus XE screen has neither problem. Coming from a 1366x768 screen, this high-res screen has been a gigantic relief and makes multitasking a lot easier. I can’t think of any way the screen could be better - if it was any higher resolution, I’d need display scaling due to its size, and that would essentially negate the best effect of having a higher res monitor (more screen real estate).
As a whole, the graphics and screen of the KFocus XE have surpassed my expectations and are perfect for everything I do.
Mobility
A good laptop generally has to be relatively lightweight, have good battery life, and have good WiFi range in order to be useful as a mobile system. In my experience, the Kubuntu Focus XE has all three.
For weight, the XE is, as discussed earlier, quite thin and light. Unlike my older system, which was a leg-squisher and could end up causing actual pain if you left it on your lap too long, the XE can remain on my lap for extended periods of time without problems. It’s not as light as my tablet (an HP Chromebook x2), but it’s fairly close, which is surprising considering that my tablet is a fanless machine with a rather pathetic processor. Whereas the KFocus XE has dual fans and a powerful processor.
Battery life is also quite good. I generally prefer battery lifetime over “how long can I use the battery before it drains dry”, so I don’t have my XE disconnected from its charger for super long, and I use the FlexiCharger feature, which I have set to make the laptop not charge at all until it get to below 40% and stop charging once it hits about 80%. Most of the time, I only use 25% of the battery life at a time, if that. That lasts me for what I would guess is an hour of intensive use, and much longer when in a power-saving mode. According to the Kubuntu Focus team, the battery should last from four hours under heavy use, to as much as ten hours under light use. That fits with my experience, and that also means that the battery in this should last longer than my ChromeOS tablet (which only gives about six to seven hours).
As for WiFi range, I currently live in a relatively small house where each room is about two rooms away from the room furthest away from it. One would think a house of this size would make it tricky to thoroughly review the WiFi range, but the smallish house is offset by my absolutely pathetic router (alright so it’s a cellular mobile hotspot about the size of half a deck of cards). I have the hotspot on one side of my house, yet I still get connectivity and decent speeds on the other side of the house most of the time. Considering the very poor quality of the WiFi signal provided by the hotspot, this is really good, and it works perfectly for what I do.
Networking and Connectivity
The WiFi chip in the KFocus XE is the Intel AX201, which is a WiFi 6 device that is theoretically capable of 2.4 Gbps speeds. My hotspot doesn’t provide anything even close to that, but while I may not get the full speed of the chip, I definitely get the full speed of my hotspot, as I see no difference in speeds whether I use a WiFi connection or a direct USB connection to the hotspot.
The Ethernet included in the KFocus XE is a fairly standard Gigabit Ethernet port with a Realtek controller. It works flawlessly for me, and gives me full gigabit speeds when transferring data from one computer to another via an Ethernet cable. I just set one of the computers to share an IPv4 connection, then put one end of the cable in the laptop, the other end of the cable in the other computer, and then use netcat to beam whatever I need transferred. I can even share an Internet connection to another computer doing this. Ethernet is increasingly rare in modern laptops, yet still very handy in many instances, so this is a very nice feature to have.
As for the other ports on the system, there’s a pair of USB3 ports, a pair of USB-C ports (one of which supports Thunderbolt 4), a headphone jack, an SD card slot, and an HDMI port. This is more than sufficient for my uses, and for those who need more ports, the Thunderbolt 4 port should let you attach a docking station to extend the system.
Another connectivity feature I use is Bluetooth. In my experience, Bluetooth has been quite finicky on other laptops I’ve used, so I didn’t have very high expectations here. However, surprisingly, things mostly worked as I would have expected them to. My Bluetooth earbuds connected without problems, and while they are a little fiddly when getting them reconnected, they reconnect pretty easily and work well once connected. (I disconnect and reconnect them manually in the Bluetooth menu, then they work right.)
Aesthetics
The KFocus XE not only performs beautifully, it also looks beautiful IMO. There are plenty of well-made design choices both within the OS and in the hardware itself. If you’ve ever used Kubuntu before, you’re already familiar with KDE’s aesthetics. The customized Kubuntu installation that comes with the KFocus XE keeps the same basic look-and-feel, but also throws in a few more nice-looking (and even functional) design elements.
The most noticeable of these additions is the Quick Help Widget on the desktop (visible in the screenshot near the start of the review). The widget looks sharp and complements the rest of the design rather well. It has several pages inside loaded with numerous keyboard shortcuts, terminal commands, and even a Vim cheat sheet.
There’s also a couple of window effects enabled by default that I mentioned earlier - transparency and wobbly windows. In essence, when you grab a window in order to move it across the screen, the window becomes transparent, allowing you to see underneath it while moving it. The window will also “flow” as you move it, making the movement look smoother. The wobbly windows effect also causes windows to visibly “snap” into place if you use window snapping, which is kind of the visual equivalent of tactile feedback - you can “feel” the window snap into place properly.
On the hardware side of things, there are several nice touches that add to the system’s aesthetic value, most noticeably the hexagonal hinges on the screen, the Kubuntu logo on the Meta key (where you usually see a Windows logo on most laptops), and the mitered angles on the edges of the recessed keyboard. The backlighting on the keyboard is also a nice touch (both functionally and aesthetically).
Setup
I saved this part for near the end since, in all likelihood, you’re only going to go through the setup procedure once per reinstall, and you’re going to reinstall very rarely, if ever. Most of what you do is going to be done once everything has been set up. But the setup procedure on the Kubuntu Focus is a major feature of the OS it ships with, so I felt it was worth reviewing.
This part was somewhat shocking for me. Most of the time, once I install a new OS on my system, it takes me a while to get fully set up. A lot of the apps I use aren’t installed by default, getting logged into everything takes time, etc. The worst part is that I have a very bad tendency to forget things, so I won’t end up getting some part of my workflow set up at all until I hit something where I need it to be set up. The Kubuntu Focus has an entire initial setup wizard that makes a massive amount of the initial setup work way easier. And there are a ton of preinstalled apps, many of which are ones I actually use. The wizard has several neat features built into it, including helping me remember to log into my email. (That part was particularly helpful since I’m pretty sure I’ve had at least one time when I only remembered to log into Thunderbird because I went to check my email.)
After a few minutes of downloads and logins, most of everything I needed was set up. I did end up having to install a few apps myself, as well as my Ubuntu development utilities. And I think I found one small bug in the wizard when I rejected VirtualBox’s Personal Use and Evaluation License for their expansion pack because I didn’t want to use Oracle’s proprietary VBox addons and comply with the restrictions that came with them. Doing so still let VirtualBox install (which was correct), but also left it marked as autoremovable (not correct). No big deal, I ended up installing a newer version of VirtualBox from the main VBox website anyway. It’s a setup wizard, it’s not going to magically read your mind and set up everything perfectly the first time. But it got pretty close, and that was close enough to save me a ton of setup time.
One unique part of the setup process is the “Curated Apps” feature. Somehow (and I don’t fully understand how), the Kubuntu Focus people have a website where you can click on the icon of an app you want to install (from a list of apps they recommend), and it will automatically do the installation for you. You have to type your password (thankfully!), but aside from that, it’s a one-touch installation procedure. Through a website. (I’m guessing there’s a client-side thingy built into the OS that makes this work.) It can even launch an app if its already installed or if you’ve just installed it. I installed several apps through the Curated Apps page, including VSCode, which is usually a bit tricky to set up. The installation process for each app was simpler even than using the Google Play store.
The whole setup procedure was drastically different from the usual “Bailing out, you’re on your own, good luck” post-install experience most other operating systems leave you with. It was an extremely good intro to a fantastic system, and it helped me a lot.
Distrohopping
The built-in Kubuntu operating system works quite well and should be all most people need. But some of us either need to use a different distro, or want to explore other distros on our physical hardware. So far I haven’t tried distros other than official Ubuntu flavors, since I really like the Ubuntu ecosystem. But within the Ubuntu ecosystem at least, the KFocus XE works like a champ.
I tried two other Ubuntu flavors (Lubuntu 23.04 Alpha and Ubuntu Unity 22.10), and both of them worked very well out of the box. I assume that other distros like Fedora, Arch, or openSUSE would very likely work well, since the hardware in the KFocus is quite Linux-friendly.
There are a couple of downsides of using a different Ubuntu flavor on the KFocus XE. For one, you lose a bunch of the tools that the Kubuntu Focus Suite has built in, like the fan profile settings. You may also not end up being able to set a power profile like KDE allows you to do. You can install the Kubuntu Focus Tools on other distros, however this will probably pull in a lot of KDE dependencies since the Kubuntu Focus Tools, are, unsurprisingly, Kubuntu-focused. Personally, I just lived without them when I was trying out different distros.
The other downside is the loss of system component curation. One hidden feature that you usually won’t see on the KFocus is the fact that the Kubuntu Focus team tests updates to several important components, such as the curated apps and the kernel, before releasing those updates to end-user systems. So if, for instance, a kernel update breaks some hardware, you won’t get that update, and your hardware will continue to work right. When you’re using other distros, the Kubuntu Focus team no longer can control those updates, and so if an update breaks something, you’ll probably end up seeing that breakage.
Aside from those two problems, the user experience when using different Ubuntu flavors was flawless for me. All hardware worked out of the box without issues (except for Bluetooth was much finickier on Ubuntu Unity than it was on Kubuntu, which may have been a consequence of lacking the curated system components). The system performed perfectly well, and I was able to use it just like I’d use any other computer I installed Linux on.
Pretty much anything with a Linux 5.17 or newer kernel should work, if I’m understanding correctly. (The 5.15 kernel is probably not ideal.) Ubuntu 22.04.2 or higher should work out of the box. There aren’t any required third-party drivers, so as long as your distro of choice ships with a reasonable set of in-kernel drivers, everything should just work out of the box, even if you’re using a non-Ubuntu distro.
Final thoughts
I mean, this is the best laptop I’ve ever used in my life. It’s fast, it’s powerful, everything works smoothly, it looks nice, and it makes it easy to get work done. That’s what a good laptop should do. If you’re into Linux and want a laptop that just works, I’d highly recommend the Kubuntu Focus XE. It’s amazing.
Thanks for taking a look, and I hope this has helped you in some way. Thanks to Michael and the Kubuntu Focus team for making these awesome systems, and for giving me the opportunity to work for one!
In this article I want to explore a common problem of modern cryptographic
ciphers: malleability. I will explain that problem with some hands-on examples,
and then look in detail at how that problem is solved through the use of
authenticated encryption. I will describe in particular two algorithms that
provide authenticated encryption: ChaCha20-Poly1305 and AES-GCM, and briefly
mention some of their variants.
The problem
If we want to encrypt some data, a very common approach is to use a symmetric
cipher. When we use a symmetric cipher, we hold a secret key, which is
generally a sequence of bits chosen at random of some fixed length (nowadays
ranging from 128 to 256 bits). The symmetric cipher takes two inputs: the
secret key, and the message that we want to encrypt, and produces a single
output: a ciphertext. Decryption is the inverse process: it takes the secret
key and the ciphertext as the input and yields back the original message as an
output. With symmetric ciphers, we use the same secret key both to encrypt and
decrypt messages, and this is why they are called symmetric (this is in
contrast with public key cryptography, or asymmetric cryptography, where
encryption and decryption are performed using two different keys: a public key
and a private key).
Generally speaking, symmetric ciphers can be divided into two big families:
stream ciphers, which can encrypt data bit-by-bit;
block ciphers, which can encrypt data block-by-block, where a block has a
fixed size (usually 128 bits).
As we will discover soon, both these two families exhibit the same fundamental
problem, although they slightly differ in the way this problem manifests
itself. To understand this problem, let’s take a close look at how these two
families of algorithms work and how we can manipulate the ciphertexts they
produce.
Stream ciphers
A good way to think of a stream cipher is as a deterministic random number
generator that yields a sequence of random bits. The secret key can be
thought of as the seed for the random number generator. Every time we
initialize the random number generator with the same secret key, we will get
exactly the same sequence of random bits out of it.
The bits coming out of the random number generator can then be XOR-ed together
with the data that we want to encrypt: ciphertext = random
sequence XOR message, like in
the following example:
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
⊕
message: I would really like an ice cream right now
=
ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
The XOR operator acts as a toggle that can either flip bits or keep them
unchanged. Let me explain with an example:
a XOR 0 = a
a XOR 1 = NOT a
If we XOR “something” with a 0 bit, we get “something” out; if we XOR
“something” with a 1 bit, we get the opposite of “something”. And if we use the
same toggle twice, we return to the initial state:
a XOR b XOR b = a
This works for any a and any b and it’s due to the fact that b XOR b is
always equal to 0. In more technical terms, each input is its own self-inverse
under the XOR operator.
The self-inverse property gives us a way to decrypt the message that we
encrypted above: all we have to do is to replay the random sequence and XOR it
together with the ciphertext:
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
⊕
ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
=
message: I would really like an ice cream right now
This works because ciphertext = random sequence XOR message, therefore random sequence XOR ciphertext = random
sequence XOR random sequence XOR
message. The two random sequence are the same, so they
cancel each other (self-inverse), leaving only message:
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
⊕
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
⊕
message: I would really like an ice cream right now
=
message: I would really like an ice cream right now
Only the owner of the secret key will be able to generate the random sequence,
therefore only the owner of the secret key should, in theory, be able to
recover the message using this method.
Playing with stream ciphers
The self-inverse property not only allows us to recover the message from the
random sequence and the ciphertext, but it also allows us to recover the random
sequence if can correctly guess the message:
message: I would really like an ice cream right now
⊕
ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
=
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
This “feature” opens the door to at least two serious problems. If we are able
to correctly guess the message or a portion of it, then we can:
decrypt other ciphertexts produced by the same secret key (or at least
portions of them, depending on what portions of the random sequence we were
able to recover);
modify ciphertexts.
And we can do all of this without any knowledge of the secret key.
The first problem implies that key reuse is forbidden with stream ciphers.
Every time we want to encrypt something with a stream cipher, we need a new
key. This problem is easily solved by the use of a nonce (also known as
initialization vector, IV, or starting variable, SV): a random value
that is generated before every encryption, and that is combined in some way
with the secret key to produce a new value to initialize the random number
generator. If the nonce is unique per encryption, then we can be sufficiently
confident that the random sequence generated will also be unique. The nonce
value does not necessarily need to be kept secret, and needs to be known at
decryption time. Nonces are usually generated at random at encryption time and
stored alongside the ciphertext.
The second problem is a bit more subtle: if we have a ciphertext and we can
correctly guess the original message that produced it, we can modify it using
the XOR operator to “cancel” the original message and “insert” a new message,
like in this example:
ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
⊕
message: I would really like an ice cream right now
⊕
altered message: I would really like to go to bed right now
=
tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1
This message, when correctly decrypted with the secret key, will return the
tampered ciphertext without detection!
Note that I do not need to know the full message to carry out this technique,
in fact, the following example (where unknown parts have been replaced by
hyphens) produces the same result as the above one:
ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
⊕
message: --------------------an ice cream----------
⊕
altered message: --------------------to go to bed----------
=
tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1
This problem is known as
malleability,
and it’s a serious issue in the real world because most of the messages that we
exchange are in practice relatively easy to guess.
Suppose for example that I have control over a WiFi network, and I can inspect
and alter the internet traffic that passes through it. Suppose that I know that
a person connected to my WiFi network is visiting an e-commerce website and
that they’re interested in a particular item. The traffic that your browser
exchanges with the e-commerce website may be encrypted, and therefore I won’t
be able to decrypt its contents, but I might be able to guess certain parts of
it, like the HTTP headers sent by the website, or some parts of the HTML that
are common to all pages on that website, or even the name and the price of the
item you want to buy. If I can guess that information (which is public
information, not a secret, and it’s generally easy to guess), then I might be
able to alter some parts of the web page, showing you false information, and
altering the price that you see in an attempt to trick you into buying that
item.
An example of malleability using ChaCha20 with OpenSSL
Here’s a practical example of how we can take the output of a stream cipher,
and alter it as we wish without knowledge of the secret key. I’m going to use
the OpenSSL command line interface to encrypt a message with a stream cipher:
ChaCha20. This is a modern, fast,
stream cipher with a good reputation:
opensslenc-chacha20\-K0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\-iv0123456789abcdef0123456789abcdef\-in<(echo'I would really like an ice cream right now')\-outciphertext.bin
The -K option specifies the key in hexadecimal format (256 bits, or 32 bytes,
or 64 hex characters), the -iv is the nonce, also known as initialization
vector (128 bits, or 16 bytes, or 32 hex characters).
This trivial Python script can tamper with the ciphertext:
withopen('ciphertext.bin','rb')asfile:ciphertext=file.read()guessed_message=b'--------------------an ice cream----------\n'replacement_message=b'--------------------to go to bed----------\n'tampered_ciphertext=bytes(x^y^zfor(x,y,z)inzip(ciphertext,guessed_message,replacement_message))withopen('tampered-ciphertext.bin','wb')asfile:file.write(tampered_ciphertext)
This script is using partial knowledge of the message. It knows (thanks to an
educated guess) that the original message contained the words “an ice cream” at
a specific offset, and uses that knowledge to replace those words with new ones
(“to go to bed”) which add up to the same length. Note that this technique
cannot be used to remove or add parts from the message, only to modify them
without changing their length.
Now if we run this script and we decrypt the tampered-ciphertext.bin file
with the same key and nonce as before, we get “to go to bed” instead of “an ice
cream”, without any error indicating that tampering occurred:
We have seen that stream ciphers alone have a serious problem (malleability)
that allows anyone to modify arbitrary portions of ciphertexts without
detection. Let’s take a look at the alternative: block ciphers. Will they have
the same problem?
While a stream cipher can encrypt variable amounts of data, a block cipher can
only take as input a block of data of a fixed size, and produce as output
another block of data. A good block cipher produces an output that is
indistinguishable from random.
The block size is generally small, usually 128 bits (16 bytes), so if we want
to encrypt larger amounts of data, we have to split the data into multiple
blocks, and encrypt each block individually. If the data is too short to fit in
a block, the data will also need to be padded.
message: The cat is on the table.........
|______________| |______________|
block #1block #2(padded)
ciphertext: c2TNPW3r09hZ6f1PVc32VX41XSy579Y9
This approach however has a problem: if we encrypt multiple blocks with the
same secret key, then portions of messages that are the repeated will produce
the same output. This gives the ability to analyze a ciphertext and find
patterns in it without knowledge of the secret key. This problem is famously
evident when encrypting pictures:
Example of applying a block cipher to an uncompressed image. The original colors are lost, but the overall layout of the image is still understandable. That's because multiple blocks of the image (containing the RGB values of each pixel), for example from the white background, are repeated multiple times, yielding the same exact encrypted blocks. The inspiration for making this image came from Wikipedia.
How this image was generated
Before jumping into how I encrypted the image, let me spend a few words on how
I did NOT encrypt the image: I did not use a modern image format. Modern image
formats are very sophisticated, they’re not a simple sequence of RGB values.
Instead, they have some control structures mixed in the image, they implement
compression to reduce the image size, etc. This complexity means that if I
simply take an image in any format and encrypt it, the result won’t be
visualizable by an image viewer: the image viewer would just throw an error
because it would find invalid data structures.
Note that this does not imply that encrypting modern image formats is more
secure: people can still analyze patterns in them, but it simply means that a
modern image format, once encrypted, would not produce the sensational
visualization that I showed above.
In order to produce this visualization I had to find an uncompressed image
format without too much metadata in it. Thankfully the Wikipedia article on
image file formats provided
a list, which included the Netpbm
family of formats (something I never heard of before). Among the formats in
this family, I chose PPM, because it’s the one that supports colors.
The PPM file format is very simple: it has 3 lines of metadata, followed by the
RGB values for each pixel. No compression. Definitely the right format for this
kind of experiment!
So here’s what I did: first of all I downloaded an image (the Ubuntu “Circle of
Friends” logo, obtained from Wikipedia) and converted it to PPM with
ImageMagick:
The reason why I separated the header from RGB values is that I won’t encrypt
the header. If I did, then the image won’t be visualizable by an image viewer,
just like if I used a modern image format. In a real-world scenario, a person
would be able to easily guess the header if it was encrypted.
I encrypted the RGB values with
AES-256, a
modern, strong block cipher with a good reputation:
This results in a randomized image that can be viewed without problems on an
image viewer. It’s interesting to see that, if you change the encryption key,
you will get a different randomized image!
The problem we have just seen is known as lack of diffusion. This is kinda
analogous to the first problem we identified with stream ciphers: at the root
of both problems there is key reuse. We solved this problem for stream ciphers
by combining a key with a random nonce. We could use the same strategy here,
would be an expensive approach, as initializing a block cipher with a new key
is a relatively expensive operation. It’s much cheaper to initialize the block
cipher once, and reuse it for every block encryption. We need a way to “link”
blocks to each other, so that if two linked blocks contain the same plaintext,
their encryption will give different results.
There are various strategies do that. These strategies are known as mode of
operation of
block ciphers. Let’s take a look at two of them: Cipher Block Chaining
(CBC) and Counter Mode (CTR).
Cipher Block Chaining (CBC)
This mode of operation, as the name suggests, ‘chains’ each block to the next
one. The way it works is by using the XOR operator in the following way:
First of all, a random nonce is generated. The purpose of the nonce is the
same as before (with stream ciphers): ensuring that using the same secret
key to perform multiple encryptions yields different results each time, so
that secret information or patterns are not revealed.
The nonce does not need to be kept secret and is normally stored alongside
the ciphertext so that it can be easily used during decryption. It is
however important that the nonce is unique.
The first block of message m[0] is XOR-ed with the nonce, and then
encrypted the block cipher, producing the first block of ciphertext c[0] =
block_encrypt(m[0] XOR nonce)
The second block of message m[1] is XOR-ed with c[0], and then
encrypted with the block cipher: c[1] = block_encrypt(m2 XOR c[0])
…
The last block of message m[n] is XOR-ed together with c[n-1], and then
encrypted with the block cipher: c[n] = block_encrypt(m[n] XOR c[n-1])
The XOR operator is back! With stream ciphers, the XOR operator was allowing us
to tamper with ciphertexts. Can we do the same thing here? Yes of course! The
approach is slightly different though: instead of acting directly on the block
that we want to change, we will act on the block that precedes it.
For example, if we want to change the sentence “I came home in the afternoon
and the cat was on the table” so that it reads ‘dog’ instead of ‘cat’, we would
need to change the block right before the one that contains the word ‘cat’. If
we want to change the very first block, for example to change the word ‘came’
to ‘left’, we would need to change the nonce instead.
nonce+ciphertext: yzURZRbP6X1w3ZRLXRDnPbEkx3JUP2Fv C2ZWt19EdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx
⊕
message (shifted 1 block to the left): I came home in t he afternoon and the cat was on the table.......
⊕
altered message (shifted 1 block to the left): I left home in t he afternoon and the dog was on the table.......
=
tampered nonce+ciphertext: yzZVQCbP6X1w3ZRLXRDnPbEkx3JUP2Fv C2ZWt67VdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx
If we do the above, and then decrypt the tampered ciphertext, we will get
something like this:
I left home in t���������������the dog was on the table
How to get this result using AES-CBC with OpenSSL
Here’s a step-by-step guide on how to tamper with a ciphertext encrypted with
AES-256 in CBC mode.
First, generate a valid ciphertext:
opensslenc-aes-256-cbc\-K0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\-iv0123456789abcdef0123456789abcdef\-in<(echo'I came home in the afternoon and the cat was on the table')\-outciphertext.bin
Like in the stream cipher example, -K is the key in hexadecimal format (256
bits), while -iv is the nonce (128 bits).
We can perform the tampering with this Python script:
Note that OpenSSL does not store the nonce along with the ciphertext, but
instead expects it to be passed as a command line argument. We need to modify
it separately, so here’s another Python script just for the nonce:
It’s interesting to see that we were successful in changing the word ‘cat’ to
‘dog’ but in doing so we had to sacrifice a block, which, when decrypted,
resulted in random bytes.
In a real world scenario, seeing some random bytes could raise some suspicion,
and maybe generate some errors in applications, however that’s not always the
case (how many times have we seen garbled text on our monitors, and we never
worried that somebody was tampering with our communications). Also, when
dealing with formats like HTML, one could conceal tampering attempts using
comment blocks, or using JavaScript. One example of what I’m describing is the
EFAIL vulnerability: discovered in 2017, it affected some
popular email clients including Gmail, it targeted the use of AES in CBC mode
(as well as another mode very similar to it: Cipher Feedback, CFB), and allowed
the injection of malicious content in HTML emails.
We can conclude that block ciphers in CBC mode, just like stream ciphers, are
also malleable.
Counter Mode (CTR)
Are other modes of operation all malleable like CBC, or will they be different?
Let’s take a look at another, very common, mode of operation: Counter Mode
(CTR), so that we can get a better sense of how the problem of malleability can
affect the world of block ciphers.
The mechanism behind Counter Mode is very simple:
A random nonce is generated. The purpose of the nonce is the usual one:
make sure that repeated encryptions using the same key produce different
results.
A counter (usually an integer) is initialized to 1 (or any other starting
value of your choice).
The nonce is concatenated with the counter, and encrypted using the block
cipher: r[0] = block_encrypt(nonce || counter).
Because the block cipher can only accept as input a block of a fixed size,
it follows that the length of the nonce plus the length of the counter must
be equal to the block size. For example, for a 128-bit block cipher, a
common choice is to have a 96-bit nonce and a 32-bit counter.
The counter is incremented: counter = counter + 1 (the increment does not
necessarily need to be by 1, but that’s a common choice). The nonce and the
new counter are concatenated again, and encrypted using the block cipher:
r[1] = block_encrypt(nonce || counter).
The counter is incremented again (counter = counter + 1), and a new block
is encrypted, just like before: r[2] = block_encrypt(nonce || counter).
…
This mechanism produces a sequence of blocks r[0], r[1], r[2], … which
are indistinguishable from random. This sequence of random blocks can be XOR-ed
with the message to produce the ciphertext.
It’s important that the values for the counter never repeat. If, for example,
we’re using a 32-bit counter, the counter will “reset” (go back to the starting
value) after 232 iterations, and will start repeating the same
sequence of random blocks as it did at the beginning. This introduces the
problem of lack of diffusion that we have seen before, just at a larger scale.
If we’re using a 32-bit counter with a 128-bit block cipher, we cannot encrypt
more than 128·232 bits = 64 GiB of data at once. This is a very
important detail: exceeding these limits may allow the decryption of portions
of ciphertext without knowledge of the secret key.
What Counter Mode is doing is effectively turning a block cipher into a
stream cipher. As such, a block cipher in Counter Mode has the exact same
malleability problems of stream ciphers that we have seen before.
An example of malleability using AES-CTR with OpenSSL
This example is going to be very similar (almost identical) to the example with
ChaCha20 that I showed in the stream cipher section, just that this time I’m
going to use AES-256 in CTR mode.
Let’s produce a valid ciphertext:
opensslenc-aes-256-ctr\-K0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\-iv0123456789abcdef0123456700000001\-in<(echo'Can you give me a ride to the party?')\-in<(echo'Do not give me a ride to the party!')\-outciphertext.bin
Tamper it with Python:
withopen('ciphertext.bin','rb')asfile:ciphertext=file.read()guessed_message=b'--------------------an ice cream----------\n'replacement_message=b'--------------------to go to bed----------\n'tampered_ciphertext=bytes(x^y^zfor(x,y,z)inzip(ciphertext,guessed_message,replacement_message))withopen('tampered-ciphertext.bin','wb')asfile:file.write(tampered_ciphertext)
We have seen that stream ciphers and block ciphers (in their mode of operation)
both exhibit the same problem (in different flavors): malleability. I’ve shown
some examples of how this problem can be exploited with modern ciphers like
ChaCha20 and AES. These ciphers, alone, cannot guarantee the integrity or
authenticity of encrypted data.
In this context, integrity is
the assurance that data is not corrupted or modified in any way.
Authenticity can be
thought of as a stronger version of integrity, and it’s the assurance that a
given ciphertext was produced only with knowledge of the secret key.
Does this mean that modern ciphers, like ChaCha20 and AES, should be considered
insecure and avoided? Absolutely not! The correct answer is that those ciphers
cannot be used alone. You should think of them as basic building blocks, and
you need some additional building blocks in order to construct a complete and
secure cryptosystem. One of these additional building block, that we are going
to explore in this article, is an algorithm that provides integrity and
authentication: welcome Authenticated Encryption (AE).
When using authenticated encryption, an adversary may be able to modify a
ciphertext using the techniques described above, but such modification would be
detected by the authentication algorithm, and decryption will fail with an
error. The decrypted message at that point should be discarded, preventing the
use of tampered data.
There are many different methods to implement authenticated encryption. The
most common approach is to use an authentication algorithm to authenticate the
ciphertext produced by a cipher. Here I will describe two very popular
authentication algorithms:
Poly1305, which is often used in conjunction with the stream cipher
ChaCha20 to form ChaCha20-Poly1305;
and Galois/Counter Mode (GCM), which is often used with the block cipher
AES to form AES-GCM.
These authentication algorithms work by computing a hash of the ciphertext,
which is then stored alongside the ciphertext. This hash is not a regular hash,
but it’s a keyed hash. A regular hash is a function that takes as input
some data and returns a fixed-size bit string:
$$\operatorname{hash}: data \rightarrow bits$$
A keyed hash instead takes two inputs: a secret key and some data, and produces
a fixed-size bit string:
The output of the keyed hash is more often called Message Authentication Code
(MAC), or authentication tag, or even just tag.
During decryption, the same authentication algorithm is run again on the
ciphertext, and a new tag is produced. If the new tag matches the original tag
(that was stored alongside the ciphertext), then decryption succeeds. Else, if
the tags don’t match, it means that the ciphertext was modified (or the tag was
modified), and decryption fails. This gives us a way to detect tampering and
gives us the opportunity to reject ciphertexts that were not produced by the
secret key.
The secret key passed to the keyed hash function is not necessarily the same
secret key used for the encryption. In fact, both ChaCha20-Poly1305 and AES-GCM
operate on a subkey derived from the key used for encryption.
Poly1305
Poly1305 is a keyed hash function proposed by Daniel J.
Bernstein in 2004. It works
by using polynomials evaluated modulo the prime 2130 - 5, hence
the name.
The key to Poly1305 is a 256-bit string, and it’s split into two halves:
the first half (128 bits) is called $r$;
the second half (128 bits) is called $s$.
We’ll see later how this key is generated when Poly1305 is used to implement
authenticated encryption. For now, let’s assume that the key is a random
(unpredictable) bit string provided as an input.
The first half $r$ is also clamped by setting some of its bits to 0. This is
a performance-related optimization that some Poly1305 implementations can take
advantage of when doing multiplication using 64-bit registers. Clamping is
performed by applying the following hexadecimal bitmask:
0ffffffc0ffffffc0ffffffc0fffffff
The message to authenticate is split into chunks of 128 bits each: $m_1$,
$m_2$, $m_3$, … $m_n$. If the length of the message is not a multiple of 128
bits, then the last block may be shorter. The authentication tag is then
calculated as follows:
Interpret $r$ and $s$ as two 128-bit little-endian integers.
Initialize the Poly1305 state $a_0$ to the integer 0. As we shall see
later, this state will need to hold at most 131 bits.
For each block $m_i$:
Interpret the block $m_i$ as a little-endian integer.
Compute $\overline{m}_i$ by appending a 1-bit to the end of the block
$m_i$. If $m_i$ is 128 bits long, then this is equivalent to computing
$\overline{m}_i = 2^{128} + m_i$. In general, if the length of the
block $m_i$ in bits is $\operatorname{len}(m_i)$, then this is
equivalent to $\overline{m}_i = 2^{\operatorname{len}(m_i)} + m_i$.
This step ensures that the resulting block $\overline{m}_i$ is always
non-zero, even if the original block $\overline{m}_i$ is zero. This is
important for the security of the algorithm, as explained later.
Compute the new state $a_i = (a_{i-1} + \overline{m}_i) \cdot r
\pmod{2^{130} - 5}$. Note that, because the operation is modulo
$2^{130} - 5$, the result will always fit in 130 bits.
Once each block has been processed, compute the final state
$a_{n+1} = a_n + s$. Note that the state $a_n$ is at most 130 bits long,
and $s$ is at most 128 bits long, hence the result will be at most 131 bits
long.
Truncate the final state $a_{n+1}$ to 128 bits by removing the most
significant bits.
Return the truncated final state $a_{n+1}$ as a little-endian byte string.
What this method is doing is computing the following polynomial in $r$ and $s$:
$r$ and $s$ are secrets, and they come from the Poly1305 key. Note that if we
didn’t add $s$ at the end, then the resulting polynomial would be a polynomial
in $r$, and one could use polynomial root-finding methods to figure out $r$
from the authentication tag, without knowledge of the key. Therefore it’s
important that $s$ is non-zero.
In Python, this is what a Poly1305 implementation could look like (disclaimer:
this is for learning purposes, and not necessarily secure or optimized for
performance):
defiter_blocks(message:bytes):""" Splits a message in blocks of 16 bytes (128 bits) each, except for the last block, which may be shorter. """start=0whilestart<len(message):yieldmessage[start:start+16]start+=16defpoly1305(key:bytes,message:bytes):assertlen(key)==32# 256 bits# Prime for the evaluation of the polynomialp=(1<<130)-5# Split the key into two parts r and sr=int.from_bytes(key[:16],'little')# 128 bitss=int.from_bytes(key[16:],'little')# 128 bits# Clamp rr=r&0x0ffffffc0ffffffc0ffffffc0fffffff# Initialize the statea=0# Update the state with every blockforblockiniter_blocks(message):# Append a 1-bit to the end of each blockblock=block+b'\1'# Convert the block to an integerc=int.from_bytes(block,'little')# Update the statea=((a+c)*r)%p# Add s to the state and truncate it to 128 bits, removing the most# significant bits and keeping only the least significant 128 bitsa=(a+s)&((1<<128)-1)# Convert the state from an integer to a 16-byte string (128 bits)returna.to_bytes(16,'little')
And here is an example of how that code could be used:
key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')
msg = b'I had a very nice day today at the beach'
print(poly1305(key, msg).hex())
This returns b0c4cb74b3089e9a982e3baa90c1bb5f, which is the same result that
we would get using OpenSSL:
opensslmac\-macopthexkey:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\-in<(echo-n'I had a very nice day today at the beach')\poly1305
A few things to note:
The same key cannot be reused to construct two distinct tags. In fact,
suppose that we use the same hash key to compute tag1 = Poly1305(key,
msg1) and tag2 = Poly1305(key, msg2). Then, because $s$ is the same for
both, we could subtract the two tags (tag1 - tag2) to remove the $s$ part
and obtain a polynomial in $r$. From there, we could use algebraic methods
to figure out $r$. Once we have $r$, we can use either one of the tags and
compute $s$, therefore recovering the full secret key.
Similarly, if the keys were generated using a predictable algorithm (for
example, incrementally: key[i+1] = key[i] + 1), it would still be
possible to use a similar approach to figure out the secret key.
For this reason, Poly1305 keys must be unique and unpredictable.
Generating Poly1305 keys randomly or pseudo-randomly is an acceptable
approach. Authentication functions like Poly1305 are called one-time
authenticators because they can be used only one time with the same key.
If we didn’t add the 1-bits at the end of each block (in other words, if we
used the $m_i$ blocks instead of $\overline{m}_i$), then encrypting a
message full of zero bits would be the equivalent of encrypting an empty
message. Adding the 1-bits is a way to ensure that the length of the
message always has an effect on the output.
Use of Poly1305 with ChaCha20 (ChaCha20-Poly1305)
Let’s see how we can combine ChaCha20 and Poly1305 to construct an authenticated cipher. To recap:
ChaCha20 is a stream cipher;
Poly1305 is a one-time authenticator;
ChaCha20, like most ciphers, requires the use of a unique nonce to allow key
reuse.
Putting the two together gives birth to ChaCha20-Poly1305. Here I’m going to
describe how to implement it as standardized in RFC
8439.
The inputs to the ChaCha20-Poly1305 encryption function are:
a 256-bit secret key;
a 96-bit nonce;
a variable-length plaintext message.
The outputs from the ChaCha20-Poly1305 encryption function are:
a variable-length ciphertext (same length as the input plaintext);
a 128-bit authentication tag.
The ChaCha20-Poly1305 decryption function will accept the same secret key,
nonce, ciphertext, and authentication tag as the input, and produce either the
plaintext or an error as the output. The error is returned in case the
authentication fails.
Data flow during a ChaCha20-Poly1305 encryption. This shows the inputs in blue, the outputs in green, and the intermediate objects in red.
ChaCha20-Poly1305 works in the following way:
The ChaCha20 stream cipher is initialized with the 256-bit secret key and
the 96-bit nonce.
The stream cipher is used to encrypt a 256-bit string of all zeros. The
result is the Poly1305 subkey.
If you recall how a stream cipher works, you should know that encrypting
using a stream cipher is equivalent to performing the XOR of a random bit
stream with the plaintext. Here the plaintext is all zeros, so the process
of generating the Poly1305 subkey is equivalent to grabbing the first 256
bits from the ChaCha20 bit stream.
We previously saw that the Poly1305 subkey must be unpredictable and unique
in order for Poly1305 to be secure. The use of ChaCha20 with a unique nonce
ensures that: because ChaCha20 is a stream cipher, its output will be
random and unpredictable. Therefore, with this construction, the subkey
will be unpredictable even if the nonce is predictable.
The stream cipher is used to encrypt another 256-bit string. The result is
discarded. This is equivalent to advancing the stream cipher state by 256
bits.
This step may seem weird, and in fact is not needed for security purposes,
but it’s a mere implementation detail. This step is here because ChaCha20
has an internal state of 512 bits. In the previous step we obtained the
first 256 bits of the state, and this next step is to discard the rest of
the state to start with a fresh state. There is no particular reason for
requiring a fresh state. The reason why RFC 8439 does that is because…
spoiler alert: ChaCha20 is a block cipher under the hood. Its block size is
512 bits. If you read the RFC, you’ll see that it asks to call the ChaCha20
block encryption function once, grab the first 256 bits, and discard the
rest. Here I’m using ChaCha20 as a stream cipher, so I have to include this
extra step to discard the bits.
The plaintext is encrypted using the stream cipher.
Note that this is done without resetting the state of the cipher. We are
continuing to use the same stream cipher instance that was used to generate
the Poly1305 subkey.
The ciphertext is padded with zeros to make its length a multiple
of 16 bytes (128 bits) and is authenticated using Poly1305, via the
subkey generated in step 2.
This step may be done in parallel to the previous one, that is: every time
we generate a chunk of ciphertext, we feed it to the Poly1305
authentication function.
Why pad the ciphertext before passing it to Poly1305? After all, ChaCha20
is a stream cipher, and Poly1305 can accept arbitrary-sized messages.
Again, this is an detail of RFC 8439 and padding does not serve any
specific purpose.
The length of the ciphertext (in bytes) is fed into the Poly1305
authenticator. This length is represented as a 64-bit little-endian
integer padded with 64 zero bits.
The reason why the length is represented as 64 bits and padded (instead of
representing it as 128 bits) will be clearer later: what I have given you
so far is a simplified view of ChaCha20-Poly1305 and authenticated
encryption in general. I will give you the full picture when talking about
associated data
later on, and at that point this step will be slightly modified.
The ciphertext from ChaCha20 and the authentication tag from Poly1305 are
returned.
The decryption algorithm works in a very similar way: ChaCha20 is initialized
in the same way, the subkey is generated in the same way, the Poly1305
authentication tag is calculated from the ciphertext in the same way. The only
difference is that ChaCha20 is used to decrypt the ciphertext (instead of
encrypting the plaintext) and that the input authentication tag is compared to
the calculated authentication tag before returning.
Here is a Python implementation of ChaCha20-Poly1305, based on the
implementations of ChaCha20 and Poly1305 from
pycryptodome (usual disclaimer: this
code is for educational purposes, and is not necessarily secure or optimized
for performance):
fromCrypto.CipherimportChaCha20fromCrypto.HashimportPoly1305defchacha20poly1305_encrypt(key,nonce,message):assertlen(key)==32# 256 bitsassertlen(nonce)==12# 96 bits# Initialize the ChaCha20 cipher with the key and noncecipher=ChaCha20.new(key=key,nonce=nonce)# Derive the Poly1305 subkey using the ChaCha20 ciphersubkey=cipher.encrypt(b'\0'*32)# 256 bitssubkey_r=subkey[:16]subkey_s=subkey[16:]# Initialize the Poly1305 authenticator with the subkeyauthenticator=Poly1305.Poly1305_MAC(r=subkey_r,s=subkey_s,data=None)# Discard the rest of the internal ChaCha20 statecipher.encrypt(b'\0'*32)# 256 bits# Encrypt the messageciphertext=cipher.encrypt(message)# Authenticate the ciphertextauthenticator.update(ciphertext)# Pad the ciphertext with zeros (to make it a multiple of 16 bytes)iflen(ciphertext)%16!=0:authenticator.update(b'\0'*(16-len(ciphertext)%16))# Authenticate the length of the associated data (0 for simplicity)authenticator.update((0).to_bytes(8,'little'))# 64 bits# Authenticate the length of the ciphertextauthenticator.update(len(ciphertext).to_bytes(8,'little'))# 64 bits# Generate the authentication tagtag=authenticator.digest()return(ciphertext,tag)defchacha20poly1305_decrypt(key,nonce,ciphertext,tag):assertlen(key)==32# 256 bitsassertlen(nonce)==12# 96 bitsassertlen(tag)==16# 128 bits# Initialize the ChaCha20 cipher and the Poly1305 authenticator, in the# same exact way as it was done during encryptioncipher=ChaCha20.new(key=key,nonce=nonce)subkey=cipher.encrypt(b'\0'*32)subkey_r=subkey[:16]subkey_s=subkey[16:]authenticator=Poly1305.Poly1305_MAC(r=subkey_r,s=subkey_s,data=None)cipher.encrypt(b'\0'*32)# Generate the authentication tag, like during encryptionauthenticator.update(ciphertext)iflen(ciphertext)%16:authenticator.update(b'\0'*(16-len(ciphertext)%16))authenticator.update((0).to_bytes(8,'little'))authenticator.update(len(ciphertext).to_bytes(8,'little'))expected_tag=authenticator.digest()# Compare the input tag with the generated tag. If they're different, the# plaintext must not be returned to the calleriftag!=expected_tag:raiseValueError('authentication failed')# The two tags match; decrypt the plaintext and return it to the caller# Note that, because ChaCha20 is a symmetric cipher, there is no difference# between the encrypt and decrypt method: here we are reusing the same# exact code used during decryptionmessage=cipher.encrypt(ciphertext)returnmessage
And here is how it can be used:
key=bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')nonce=bytes.fromhex('0123456789abcdef01234567')message=b'I wanted to go to the beach, but now I changed my mind'ciphertext,tag=chacha20poly1305_encrypt(key,nonce,message)decrypted_message=chacha20poly1305_decrypt(key,nonce,ciphertext,tag)assertmessage==decrypted_messageprint(f'ciphertext: {ciphertext.hex()}')print(f' tag: {tag.hex()}')print(f' plaintext: {decrypted_message}')
Running it produces the following output:
ciphertext: 5d9b09cc5d90ca9ddff2d3470cfd6b563c5158e952bfae6acf1ebf9a3b968a488a41969567ef5ccfe05dcf9e548567028ff374a754af
tag: dac3c05d261920e278ceb22e2800aa95
plaintext: b'I wanted to go to the beach, but now I changed my mind'
This is the same output we would obtain by using the ChaCha20-Poly1305
implementation from pycryptodome directly:
fromCrypto.CipherimportChaCha20_Poly1305key=bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')nonce=bytes.fromhex('0123456789abcdef01234567')message=b'I wanted to go to the beach, but now I changed my mind'cipher=ChaCha20_Poly1305.new(key=key,nonce=nonce)ciphertext,tag=cipher.encrypt_and_digest(message)print(f'ciphertext: {ciphertext.hex()}')print(f' tag: {tag.hex()}')
As already stated, it is extremely important that the nonce passed to
ChaCha20-Poly1305 is unique. It may be predictable, but it must be unique. If
the same nonce is reused twice or more, we can:
Decrypt arbitrary messages without using the secret key, if we can guess at
least one message from its ciphertext.
This can be done using the techniques described at the beginning of this
article: by recovering the random bit string from the XOR of the ciphertext
with the guessed message.
Recover the Poly1305 subkey, and, at that point, tamper with ciphertexts
and forge new, valid authentication tags.
This can be done by using algebraic methods on the polynomial of the
authentication tag.
There is also a variant of ChaCha20-Poly1305, called XChaCha20-Poly1305, that
features an extended 192-bit nonce (the X stands for ‘extended’). This is
described in an RFC
draft but so
far it hasn’t been accepted as a standard yet. I won’t cover XChaCha20 in
detail here, because it’s slightly more complex and does not add much to the
topic of this article, but XChaCha20-Poly1305 has better security properties
than ChaCha20-Poly1305, so you should prefer it in your applications if you can
use it. The reason why XChaCha20-Poly1305 has better properties than
ChaCha20-Poly1305 is that, having a longer nonce, the probability of generating
two random nonces with the same value are much lower.
Galois/Counter Mode (GCM)
Let’s now take a look at Galois/Counter Mode (GCM). This is commonly used with
the Advanced Encryption Standard (AES), to construct the authenticated cipher
AES-GCM. One main difference between Poly1305 and GCM is that Poly1305 can work
with any stream or block cipher, while GCM is designed to work with block
ciphers with a block size of 128 bits.
GCM was proposed by David McGrew and John Viega in 2004 and is standardized in
NIST Special Publication
800-38D
as well as RFC 5288. It takes its
name from Galois fields, also known as finite
fields, which in turn get their
name from the French mathematician Évariste
Galois, who introduced the
concept of finite fields as we know them today.
As we did before with Poly1305, we are going to first see how the keyed hash
function used by GCM works, and then we will see how to use it to construct an
authenticated cipher like AES-GCM on top of it. Before we can do that though,
we need to understand what are finite fields, and what specific type of finite
fields are used in GCM.
Finite Fields (Galois Fields)
What is a field? A field is a mathematical structure that contains a bunch of
elements, and those elements can interact with each other using addition and
multiplication. For both these operations there’s an identity element and an
inverse element. Addition and multiplication in a field must obey the usual
properties that we’re used to: commutativity, associativity, and
distributivity.
A well-known example of a field is the field of fractions. Here is why
fractions form a field:
the elements of the field are the fractions;
addition is well-defined: if we add two fractions, we get a fraction out
(example: $5/3 + 3/2 = 19/6$);
multiplication is also well-defined: if we multiply two fractions, we get
a fraction out (example: $1/2 \cdot 8/3 = 4/3$);
the additive identity element is 0: if we add 0 to any fraction, we get
the same fraction back;
the additive inverse element is the negated fraction (example: $5/1$ is
the additive inverse of $-5/1$ because $5/1 + (-5/1) = 0$);
the multiplicative identity element is 1: multiplying any fraction by 1
yields the same fraction back;
the multiplicative inverse element is what we get if we swap the
numerator with the denominator (example: $3/2$ is the multiplicative inverse
of $2/3$ because $3/2 \cdot 2/3 = 1$)—except for 0, which does not have a
multiplicative inverse.
and so on…
On top of addition, multiplication, and inverse elements, we can define derived
operations like subtraction and division. Subtracting $a$ from $b$ is
equivalent to adding $a$ to the additive inverse of $b$: $a - b = a + (-b)$.
Similarly, division can be defined in terms of multiplication with
multiplicative inverses ($a / b = a b^{-1}$).
Fields are a generalization of structures where addition, multiplication,
subtraction, and division behave according to the rules that we’re used to.
Field elements do not necessarily need to be numbers.
An example of something that is not a field is the integers. That’s because
integers don’t have multiplicative inverses (for example, there’s no integer
that multiplied by 5 makes the result equal to 1). However, there is a way to
turn the integers into a field: if we take the integers and a prime number p,
then we can construct the field of integers modulo p.
When we work in the integers modulo a prime p, whenever we see p appear in
any of our expressions, we can replace it with 0. In other words, in such a
field, p and 0 are two different ways to write the same element–they are two
different representations of the same element.
Here is an example: in the field of integers modulo 7, the expression 5 + 3
equals 1 because
5 + 3 evaluates to 8;
8, by definition, is 7 + 1;
if 7 and 0 are the same element, then 7 + 1 is equal to 0 + 1
0 + 1 evaluates to 1
What we have just seen is that 8 is just a different representation of 1, just
like 7 is a different representation of 0. Different symbols, same object.
Just like, in programming languages, we can have multiple variables point to
the same memory location: here the numbers are like variables, and what they
point to is what really matters.
In the field of integers modulo 7, the additive inverse for 5 is 2, because 5
+ 2 = 7 = 0. If we manipulate the equation, we get that 5 = −2. In other words,
5 and −2 are two different representations of the same element, and for the
same reason 2 and −5 are also two different representations of the same
element. A similar story holds for multiplication: the multiplicative inverse
for 5 is 3 because: 5 · 3 = 15 = 7 + 7 + 1 = 1, so we can write 5 =
3−1 as well as 3 = 5−1.
What we have just seen is an example of a finite field. It’s different from
a general field because it contains a finite number of elements (unlike
fractions, which do not have a limit). In the case of the integers modulo 7,
the number of elements is 7, and the list of elements is: {0, 1, 2, 3, 4, 5,
6}, or {−3, −2, −1, 0, 1, 2, 3}, or {0, 1, 2, 3, 2−1,
3−1, 6}, depending on what representation we like the most.
A few words about terminology, notation, and equivalences of finite fields
There can be many ways to construct a finite field (or even a general field). I
have given an example using numbers, but a field does not necessarily need to
be formed from numbers. We can also use vectors, matrices, polynomials, and
anything you would like. As long as addition, multiplication, identity
elements, and inverse elements are well-defined, you can get a field. Using
programming terms, you can think of a field as an interface or a trait that can
have arbitrary implementations.
An important result in algebra is that finite fields with the same number of
elements are unique up to isomorphism. This means that if two finite fields the
same number of elements, then there is an equivalence relation between the two.
The number of elements of a field is therefore enough to define a field. It’s
not enough to tell us what the elements of the field look like, or how they can
be represented, but it’s enough to know how it behaves. To denote a field with
$n$ elements, there are two major notations: $GF(n)$ and $\mathbb{F}_{n}$.
Another important result in algebra is that $n$ may be either a prime number,
or a power of a prime. For example, we can have finite fields with 2 elements,
or with 9 (= 32) elements, but we cannot have a field with 6 (= 2·3)
elements. For this reason, you will often find finite fields denoted as
$GF(p^k)$ or $\mathbb{F}_{p^k}$, where $p$ is a prime and $k$ is an integer
greater than 0. The prime $p$ is called characteristic of the field, while $n
= p^k$ is called order of the field.
Some common fields also have their own notation: in particular, the field of
integers modulo a prime $p$ is denoted as $Z/pZ$. This notation encodes the
“building instructions” to construct the field, in fact:
$A/B$ is a quotient. This is a way to define an equivalence relation
between elements, and its meaning is: within $A/B$, all the elements of $B$
are equivalent to 0. In the case of $Z/pZ$, all the multiples of $p$ are
equivalent to 0, which is indeed what happens with the integers modulo $p$.
The way I described this equivalence relation earlier is by saying that
multiples of $p$ are different representations for 0.
Note that the integers modulo a power of a prime ($Z/p^kZ$, with $k$ greater
than 1) do not form a field. The problem is that elements in $Z/p^kZ$ sometimes
do not have a multiplicative inverse. For example, in $Z/4Z$, the number 2 does
not have a multiplicative inverse (there is no element that multiplied by 2
gives 1). A field $GF(p^k)$ with $k$ greater than 1 needs to be constructed in
a different way. One such way is to use polynomials, as described in the next
section.
Polynomial fields
Let’s now move our attention from integers to polynomials, like this one:
$$x^7 + 5x^3 - 9x^2 + 2x + 1$$
Polynomials are a sum of coefficients multiplied by a variable (usually denoted
by the letter x) raised to an integral power.
Let’s restrict our view to polynomials that have integer coefficients, like the
one shown above. Something that is not a polynomial with integer coefficients
is $1/2 x^2 + x$, because it has a fraction in it.
Integers and polynomials with integer coefficients are somewhat similar to each
other. They kinda behave the same in many aspects. One important property of
integers is the unique factorization
theorem: if
we have an integer, there’s a way to write it as a multiplication of some
primary factors. For example, the integer 350 can be factored as the
multiplication of 2, 5, 5, and 7.
$$350 = 7 \cdot 5 \cdot 5 \cdot 2$$
This factorization is unique: we can change the order of the factors, but
it’s not possible to obtain a different set of factors (there’s no way to make
the number 3 appear in the factorization of 350, or to make the number 7
disappear).
Polynomials with integer coefficients also have a unique factorization. In the
case of integers, We call the unique factors “prime numbers”; in the case of
polynomials we have “irreducible polynomials”. And just like we can have a
field of integers modulo a prime, we can have a field of polynomials modulo
an irreducible polynomial.
Let’s take a look at how arithmetic in polynomial fields work. Let’s take, for
example, the field of polynomials with integer coefficients modulo $x^3 + x +
1$, and try to compute the result of $(x^2 + 1)(x^2 + 2)$. If we expand the
expression, we get:
$$(x^2 + 1)(x^2 + 2) = x^4 + 3x^2 + 2$$
This expression can be reduced. Reducing a polynomial expression is the
equivalent of what we were doing with integers modulo a prime, when we were
saying that 8 = 7 + 1 = 1 (mod 7). That “conversion” from 8 to 1 is the
equivalent of the reduction that we’re talking about here.
To reduce $x^4 + 3x^2 + 2$, first note that $x^4 = x \cdot x^3$. Also note that
$x^3 = x^3 + x + 1 - x - 1$. Here we have just added and removed the term $x +
1$: the result hasn’t changed, but now the irreducible polynomial $x^3 + x + 1$
appears in the expression, and so we can substitute it with 0. Putting
everything together, we get:
$$\begin{align*}
(x^2 + 1)(x^2 + 2) & = x^4 + 3x^2 + 2 \\
& = x \cdot x^3 + 3x^2 + 2 \\
& = x \cdot (x^3 + x + 1 - x - 1) + 3x^2 + 2 \\
& = x \cdot (0 - x - 1) + 3x^2 + 2 \\
& = -x^2 - x + 3x^2 + 2 \\
& = 2x^2 - x + 2
\end{align*}$$
It’s interesting to note that, if the polynomial field is formed by an
irreducible polynomial with degree $n$, then all the polynomials in that field
will all have degree less than $n$. That’s because if any $x^n$ (or higher)
appears in a polynomial expression, then we can use the substitution trick I
just showed to reduce its degree.
Binary fields
Let’s now look at polynomials where coefficients are from the field of
integers modulo 2, meaning that they can be either 0 or 1. This is an example
of such a polynomial:
$$x^7 + x^4 + x^2 + 1$$
or, in a more explicit form, where we can clearly see all the coefficients:
These are called binary polynomials. It’s interesting to note that if we
ignore the variables and the powers, and keep only the coefficients, then what
we get is a bit string:
$$(1 0 0 1 0 1 0 1)$$
This suggests that there’s an interesting duality between binary polynomials
and bit strings. This means, in particular, that binary polynomials can be
represented in a very compact and natural way on computers.
The duality between binary polynomials and bit strings also suggests that
perhaps we can use bitwise operations to perform arithmetic on binary
polynomials. And this turns out to be true, in fact:
binary polynomial addition can be computed using the XOR on the two
corresponding bit strings;
binary polynomial multiplication can be computed using XOR, AND and
bit-shifting.
Computers are pretty fast at performing these bitwise operations, and this
makes binary polynomials quite attractive for use in computer algorithms and
cryptography.
Arithmetic with binary polynomials
The arithmetic of such polynomials is quite interesting: in fact, because $1 +
1 = 0$ (modulo 2), then also $x^k + x^k = 0$, in fact:
It’s easy to see that addition modulo 2 is equivalent to the XOR binary
operator. And addition of two binary polynomials is equivalent to the bitwise
XOR of their corresponding bit strings:
Multiplication of binary polynomials can also be implemented as a bitwise
operation on bit strings. First, note that multiplying a polynomial by a
monomial is equivalent to bit-shifting:
Putting everything together, we have multiplications by monomials (equivalent
to bit-shifts) and sums (equivalent to bitwise XOR). This suggests that
multiplication can be implemented on top of bitwise XOR and bit-shifting.
Here is some Python code to implement binary polynomial multiplication, where
each polynomial is represented compactly as an int:
defmultiply(a,b):""" Compute a*b, where a and b are two integers representing binary polynomials. a and b are expected to have their most significant bit set to the monomial with the highest power. For example, the polynomial x^8 is represented as the integer 0b10000. """asserta>=0assertb>=0result=0whileb:result^=a*(b&1)a<<=1b>>=1returnresult
Other than XOR and bit-shifting, this code also uses AND to “query” whether a
certain monomial is present or not.
Now that we have introduced binary polynomials, we can of course form binary
polynomials modulo a binary irreducible polynomial. These form a finite
field, which is more concisely called: binary field.
Note that in a binary field where the modulo is an irreducible polynomial of
degree $n$, all polynomials in the field can be represented as $n$-bit strings,
and all $n$-bit strings have a corresponding binary polynomial in the field.
Arithmetic in binary fields
If we have three integers $a$, $b$, and $p$, we can compute $(a + b) \bmod{p}$
or $a \cdot b \bmod{p}$ by performing the binary operation (addition or
multiplication) and then taking the remainder of the division by $p$. This is a
method that returns the results of addition or multiplication using a
representation with the lowest number of digits possible.
What if instead of having 3 integers we have three binary polynomials $A$, $B$,
and $P$ and we want to compute $(A + B) \bmod{P}$ or $A \cdot B \bmod{P}$? It
turns out that these operations can be implemented with code that is even
easier than the integer counterpart: no division needs to be involved!
Let’s start with addition: we have already seen that addition with binary
polynomials can be implemented with a simple XOR operation. This means that if
the degree of $A$ and $B$ is lower than the degree of $P$, then the result of
$A + B$ is also going to have degree less than $P$, hence no reduction is
needed. We can use the result as-is, without any transformation: adding two
binary field elements can be implemented with a single XOR operation.
With multiplication the story is different: the product $A \cdot B$ may have
degree equal to or higher than $P$. For example, if $A = B = x$ and $P = x^2 +
1$, the product $A \cdot B$ is equal to $x^2$, which has the same degree as
$P$. We need to find a way to efficiently reduce the higher-degree terms of
this product. To see one way to do that, note that we can write $P$ like this:
$$P = x^n + Q$$
where $n$ is the degree of $P$ (the maximum power of $P$) and $Q$ is another
binary polynomial, with degree strictly lower than $n$. Rearranging the
equation, we get:
$$x^n = P + Q$$
Note that subtraction and addition are the same operations in a binary field.
Because $P$ equals 0, we can write:
$$x^n = Q$$
This equivalence gives us a way to eliminate higher-level terms that appear
during multiplication: whenever we see an $x^n$ appearing in the result, we can
remove that term and add $Q$ instead. One way to do that, using binary strings,
is to discard the highest bit (the one corresponding to $x^n$) and XOR with
the binary string corresponding to $Q$.
Another way to do it is to just add $P$ (XOR by the binary string corresponding
to $P$). This is equivalent to adding 0, results in the more compact
representation that we’re interested in.
We could use similar tricks to eliminate terms like $x^{n+1}$, but these tricks
are not necessary if we eliminate $x^n$ terms as soon as they appear in an
iterative way.
Here is some Python code for multiplication in binary fields that uses the “add
$P$” trick just described:
defmultiply(a,b,p):""" Compute a*b modulo p, where a, b and c are three integers representing binary polynomials. a, b and p are expected to have their most significant bit set to the highest power monomial. For example, the polynomial x^8 is represented as 0b10000. """bit_length=p.bit_length()-1asserta>=0anda<(1<<bit_length)assertb>=0andb<(1<<bit_length)result=0foriinrange(bit_length):result^=a*(b&1)a<<=1a^=p*((a>>bit_length)&1)b>>=1returnresult
This code is essentially the same as the binary polynomial multiplication code we had before, except for this line in the for loop:
a^=p*((a>>bit_length)&1)
This line is what “adds $P$” whenever adding the shifted $A$ would result in a
$x^n$ term to appear.
Again, we achieved implementing multiplication using only XOR, AND and
bit-shifting.
Note that the binary polynomial $P$ here does not necessarily need to be an
irreducible polynomial for this algorithm to work. However, the resulting
algebraic structure won’t be a field unless $P$ is irreducible. A similar story
holds for integers: we can have integers modulo a non-prime number, but that’s
not a field.
The GHASH keyed hash function
GCM uses a binary field. The irreducible binary polynomial that defines the
binary field used by GCM is:
$$P = x^{128} + x^7 + x^2 + x + 1$$
We will call this field the GCM field. Note that this polynomial has degree
128, hence the GCM field elements can be represented as 128-bit strings, and
each 128-bit string has a corresponding element in the GCM field.
The keyed hash function used by GCM is called GHASH and takes as input a
128-bit key. We will call this key $H$. This key is interpreted as an element
of the GCM field.
The message to authenticate is split into blocks of 128 bits each: $M_1$,
$M_2$, $M_3$, … $M_n$. If the length of the message is not a multiple of 128
bits, then the last block is padded with zeros. Each block of message is also
interpreted as an element of the GCM field.
Here is how the authentication tag is computed from $H$ and the padded message
blocks $M_1$, …, $M_n$:
The initial state (a GCM field element) is initialized to 0: $A_0 = 0$.
For every block of message $M_i$, the next state $A_i$ is computed as $A_i =
(A_{i-1} + M_i) \cdot H \bmod{P}$.
The final state $A_n$ is returned as a 128-bit string.
What this function is doing is computing the following polynomial in $H$:
This construction is somewhat similar to the one from Poly1305, although there
are important differences:
In Poly1305, the elements of the tag polynomial are integers modulo a prime,
in GHASH they are elements of a binary field.
GHASH does not perform any step to encode the length of the message, hence
the tag for an empty message will be the same as the tag for a sequence of
zero blocks. We will see later that GCM fixes this problem by appending the
length of the message to the end of the input passed to GHASH.
Most importantly, the final $Tag$ polynomial is a polynomial in one unknown,
and as such $H$ may be easily recoverable using algebraic methods. For this
reason, GHASH is not suitable as a secure one-time authenticator. We will
see that GCM fixes this problem by encrypting the output of GHASH.
Use of GCM with AES (AES-GCM)
GCM is the combination of a block cipher, Counter Mode (CTR), and the GHASH
function that we have just seen. The block cipher is often AES. When we combine
AES with GCM, the what we get is AES-GCM, which is described below. However the
block cipher does not necessarily need to be AES: what is important is that the
block size of the cipher is 128 bits, and that’s because GHASH only works on
128-bit blocks.
The inputs to the AES-GCM encryption function are:
a secret key (the length of the key depends on the variant of AES used: if
AES-128, this will be 128 bits);
a 96-bit nonce;
a variable-length plaintext message.
The outputs of the AES-GCM encryption function are:
a variable-length ciphertext (same length as the input plaintext);
a 128-bit authentication tag.
The AES-GCM decryption function will accept the same secret key, nonce,
ciphertext, and authentication tag as the input, and produce either the
plaintext or an error as the output. The error is returned in case the
authentication fails.
Data flow during an AES-GCM encryption. This shows the inputs in blue, the outputs in green, and the intermediate objects in red.
AES-GCM works in the following way:
The GHASH subkey $H$ is generated by encrypting a zero-block: $H =
\operatorname{Encrypt}(key, \underbrace{000\dots0}_\text{128 bits})$.
The block cipher AES is initialized in Counter Mode (AES-CTR) with
the key, the nonce, and a 32-bit, big-endian counter starting at 2.
The plaintext is encrypted using the instance of AES-CTR just
created.
The GHASH function is run with the following inputs:
the subkey $H$, computed in step 1;
the ciphertext padded with zeros to make its length a multiple of 16
bytes (128 bits), concatenated to the length (in bits) of the ciphertext
represented as a 128-bit big-endian integer.
The result is a 128-bit block $S = \operatorname{GHASH}(H, ciphertext || padding || length)$.
The AES-CTR counter is set to 1.
The block $S$ is then encrypted using AES-CTR. The result of the
encryption is the authentication tag.
Note that, because $S$ matches the block size of the cipher, this
encryption won’t cause the counter value 2 to be reused.
The ciphertext and authentication tag are returned.
Here is how AES-GCM and GHASH can be implemented in Python, using the AES
implementation from pycryptodome
(usual disclaimer: this code is for educational purposes, and it’s not
necessarily secure or optimized for performance):
fromCrypto.CipherimportAESdefmultiply(a:int,b:int)->int:""" Compute a*b in the GCM field, where a and b are two integers representing elements of the GCM field. a and b are expected to have their least significant bit set to the highest power monomial. For example, the polynomial x^125 is represented as 0b100. """bit_length=128q=0xe1000000000000000000000000000000asserta>=0anda<(1<<bit_length)assertb>=0andb<(1<<bit_length)result=0foriinrange(bit_length):result^=a*((b>>127)&1)a=(a>>1)^(q*(a&1))b<<=1returnresultdefpad_block(data:bytes)->bytes:""" Pad data with zero bytes so that the resulting length is a multiple of 16 bytes (128 bits). """iflen(data)%16!=0:data+=b'\0'*(16-len(data)%16)returndatadefiter_blocks_padded(data:bytes):""" Split the given data into blocks of 16 bytes (128 bits) each, padding the last block with zeros if necessary. """start=0whilestart<len(data):yieldpad_block(data[start:start+16])start+=16defghash(subkey:bytes,message:bytes)->bytes:subkey=int.from_bytes(subkey,'big')assertsubkey<(1<<128)state=0forblockiniter_blocks_padded(message):block=int.from_bytes(block,'big')state=multiply(state^block,subkey)returnstate.to_bytes(16,'big')defaes_gcm_encrypt(key:bytes,nonce:bytes,message:bytes):assertlen(key)in(16,24,32)assertlen(nonce)==12# Initialize a raw AES instance and encrypt a 16-byte block of all zeros to# derive the GHASH subkey Hcipher=AES.new(mode=AES.MODE_ECB,key=key)h=cipher.encrypt(b'\0'*16)# Encrypt the message with AES in CTR mode, with the counter composed by# the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32 bits)# integer, starting from 2cipher=AES.new(mode=AES.MODE_CTR,key=key,nonce=nonce,initial_value=2)ciphertext=cipher.encrypt(message)# Compute the GHASH of the ciphertext plus the ciphertext length in bitss=ghash(h,pad_block(ciphertext)+(len(ciphertext)*8).to_bytes(16,'big'))# Encrypt the GHASH value using AES in CTR mode, with the counter composed# by the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32# bits) integer set at 1. The GHASH value fits in one block, so the counter# won't be increased during this round of encryptioncipher=AES.new(mode=AES.MODE_CTR,key=key,nonce=nonce,initial_value=1)tag=cipher.encrypt(s)return(ciphertext,tag)defaes_gcm_decrypt(key:bytes,nonce:bytes,ciphertext:bytes,tag:bytes):assertlen(key)in(16,24,32)assertlen(nonce)==12assertlen(tag)==16# Compute the GHASH subkey, the GHASH value, and the authentication tag, in# the same exact way as it was done during encryptioncipher=AES.new(mode=AES.MODE_ECB,key=key)h=cipher.encrypt(b'\0'*16)s=ghash(h,pad_block(ciphertext)+(len(ciphertext)*8).to_bytes(16,'big'))cipher=AES.new(mode=AES.MODE_CTR,key=key,nonce=nonce,initial_value=1)expected_tag=cipher.encrypt(s)# Compare the input tag with the generated tag. If they're different, the# plaintext must not be returned to the calleriftag!=expected_tag:raiseValueError('authentication failed')# The two tags match; decrypt the plaintext and return it to the caller.# Note that, because AES-CTR is a symmetric cipher, there is no difference# between the encrypt and decrypt method: here we are reusing the same# exact code used during decryptioncipher=AES.new(mode=AES.MODE_CTR,key=key,nonce=nonce,initial_value=2)message=cipher.encrypt(ciphertext)returnmessage
And here is how the code can be used:
key=bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')nonce=bytes.fromhex('0123456789abcdef01234567')message=b'I went to the zoo yesterday but not today'ciphertext,tag=aes_gcm_encrypt(key,nonce,message)print(f'ciphertext: {ciphertext.hex()}')print(f' tag: {tag.hex()}')decrypted_message=aes_gcm_decrypt(key,nonce,ciphertext,tag)print(f' plaintext: {decrypted_message}')assertmessage==decrypted_message
This snippet produces the following output:
ciphertext: e0c32db2962f9b729c69028d9a1fdfb2c93839fc1188f314c58ee97fd6a242404953bb208df609a33c
tag: 9fa6fe2f77a0c98282868924ace0e4ec
plaintext: b'I went to the zoo yesterday but not today'
This is the same output we would obtain by using the AES-GCM implementation
from pycryptodome directly:
Nonce reuse is catastrophic for AES-GCM in two ways:
Because the ciphertext produced by AES-GCM is just a variant of AES-CTR,
nonce reuse with GCM can have the same consequences as nonce reuse with
AES-CTR, or any other stream cipher: if someone is able to guess the
plaintext, they can recover the random stream, and use that to decrypt other
messages (or portions of them).
If the same nonce is used twice or more, the GHASH subkey $H$ will always be
the same. Even if the output of GHASH is encrypted in step 7, we can use the
XOR of two authentication tags to “cancel” the encryption and obtain a
polynomial in $H$. From there, we can use algebraic methods to recover $H$.
This gives us the ability to forge new, valid authentication tags.
It’s worth mentioning that there’s a variant of AES-GCM, called AES-GCM-SIV,
(Synthetic Initialization Vector) specified in RFC
8452. This differs from AES-GCM in
that it uses a little-endian version of GHASH called POLYVAL (which is faster
on modern CPUs), and in that it allows nonce reuse without the two catastrophic
consequences that I mentioned above.
(Nonce reuse with AES-GCM-SIV however still presents a problem, just not as
serious as the two ones above: specifically, it breaks ciphertext
indistinguishability.)
Authenticated Encryption with Associated Data (AEAD)
The way I have described authenticated encryption, and in particular the
constructions ChaCha20-Poly1305 and AES-GCM, is accurate, but incomplete. What
I have told you is that when you use an authenticated encryption cipher, the
ciphertext is checked for integrity and authenticity. But we can use the same
technique to authenticate anything, not just ciphertexts: we can, for example,
authenticate some plaintext data, or authenticate a piece of plaintext data and
a piece of ciphertext altogether.
When we use a method to authenticate a plaintext message only, what we get is a
Message Authentication Code (MAC). We don’t use the word “encryption” in this
context, because the confidentiality of the message is not ensured (only its
authenticity).
When we use a method to authenticate both a ciphertext and a plaintext message,
what we get is Authenticated Encryption with Associated Data (AEAD). In
this construction, there are two messages involved: one to be encrypted
(resulting in a ciphertext), and one to be kept in plaintext. The plaintext
message is called “associated data” (AD) or “additional authenticated data”
(AAD). Both the ciphertext and the associated data are authenticated at
encryption time, so their integrity and authenticity will be enforced.
The inputs to the encryption function of an AEAD cipher are, generally
speaking:
a key;
a nonce;
the additional data;
the message to encrypt.
The outputs of the encryption are:
the ciphertext;
the authentication tag.
Note that there’s only one authentication tag that covers both the additional
data and the ciphertext.
The inputs to the decryption function are:
the key used for encryption;
the nonce used for encryption;
the additional data used for encryption;
the ciphertext.
And the output of the decryption is either an error or the decrypted
message.
It’s important to note that the associated data must be both at encryption time
and decryption time. Changing a single bit of it will make the entire
decryption operation fail.
Both ChaCha20-Poly1305 and AES-GCM (and their variants, XChaCha20-Poly1305 and
AES-GCM-SIV) are AEAD ciphers. Here’s how they implement AEAD:
When the Poly1305 or GHASH authenticator is first initialized, they are fed
the additional data, padded with zeros to make its size a multiple of 16
bytes (128 bits).
Then the padded ciphertext is fed into the authenticator.
The length of the additional data and the length of the ciphertext are
represented as two 64-bit integers, concatenated, and fed into the
authenticator.
Updated data flow during a ChaCha20-Poly1305 encryption which shows where the Associated Data (AE) is placed.
Updated data flow during an AES-GCM encryption which shows where the Associated Data (AE) is placed.
If the additional data is empty, then what you get are exactly the
constructions that I described earlier in this article.
Authenticated Encryption with Associated Data is useful in situations where you
want to encode some metadata along with your encrypted data. For example: an
identifier for the resource that is encrypted, or the type of data encrypted
(text, image, video, …), or some information that indicates what key and
algorithm was used to encrypt the resource, or maybe the expiration of the
data. The associated data is in plaintext so systems that do not have access to
the secret key can gather some properties about the encrypted resource. It must
however be understood that the associated data cannot be trusted until it’s
verified using the secret key. Systems that analyze the associated data must be
designed in such a way that, if the associated data is tampered, nothing bad
will happen, and such tampering attempt will be detected sooner or later.
A word of caution
Something very important to understand is that when using authenticated
encryption ciphers like ChaCha20-Poly1305 or AES-GCM, decryption can in
theory succeed even if the verification of the authentication tag fails.
For example, we can decrypt a ciphertext encrypted with ChaCha20-Poly1305 by
using ChaCha20 and ignoring the authentication tag. Similarly, we can decrypt a
ciphertext encrypted with AES-GCM by using AES-CTR and, again, ignoring the
authentication tag. This possibility opens the doors to all the nasty scenarios
that we have seen at the beginning of this article, removing all the benefits
of authenticated encryption.
Perhaps the most important thing to remember when using authenticated
encryption is: never use decrypted data until you have verified its
authenticity.
Why am I emphasizing this? Because some AE or AEAD implementations do return
plaintext bytes before verifying their authenticity.
The code samples that I have provided do the following: they first calculate
the authentication tag, compare it to the input tag, and only if the comparison
succeeds they perform the decryption. This is a simple approach, but it may be
expensive when encrypting large amounts of data (for example: several
gigabytes). The reason why this approach is expensive is that, if the
ciphertext is too large, it may not fit all in memory, and the ciphertext would
have to be read from the storage device twice: once for calculating the tag,
and once for decrypting the ciphertext. Also, chances are that by the time the
application has computed the tag, the underlying ciphertext may have changed
without detection.
What some authenticated encryption implementations do when dealing with large
amounts of data is that they calculate the tag and perform the decryption in
parallel. They read the ciphertext chunk-by-chunk, and pass each chunk to both
the authenticator and the decryption function, returning a chunk of decrypted
bytes to the caller at each iteration. Only at the end, when the full
ciphertext has been read, the authenticity is checked, and the application may
return an error only at that point. With such implementations, it is imperative
that the exit status of the application is checked before using any of the
decrypted bytes.
An implementation that works like that (returning decrypted bytes before
authentication is complete) is GPG. Here is an example of the output that GPG
produces when decrypting a tampered message:
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
This is a very long message.
gpg: WARNING: encrypted message has been manipulated!
The decrypted message (“This is a very long message”) got printed, together
with a warning, and the exit status is 2, indicating that an error occurred. It
is important in this case that the decrypted message is not used in any way.
Other implementations avoid this problem by simply not encrypting large amounts
of data. If given a large file to encrypt, the file is first split into
multiple chunks of a few KiB, then each chunk is encrypted independently, with
its own nonce and authentication tag. Because each chunk is small,
authentication and decryption can happen in memory, one before the other. If a
chunk was tampered, decryption would stop, returning truncated output, but
never tampered output. It’s still important to check the exit status of such an
implementation, but the consequences are less catastrophic than before. The
drawback of this approach is that the total size of the ciphertext increases,
because each chunk requires a nonce, an authentication tag, and some
information about the position of the chunk (to prevent the chunks from being
reordered). Storing the nonces or the positions can be avoided by using an
algorithm to generate them on the fly, but storing the tag cannot be avoided.
The method of splitting that I have just described (of splitting long messages
into chunks that are individually encrypted and authenticated) is used for
example in TLS, as
well as the command line tool AGE.
Summary and final considerations
At the beginning of this article we have seen some risks of using bare
encryption ciphers: one of them in particular was malleability, that is: the
property that ciphertexts may be modified without detection.
This problem was addressed by using Authenticated Encryption (AE) or
Authenticated Encryption with Associated Data (AEAD), which are methods to
provide integrity and authenticity in addition to confidentiality when
encrypting data.
We have seen the details of the two most popular authenticated encryption
ciphers and briefly mentioned some of their variants. Their features are
summarized here:
Cipher
Cipher Type
Key Size
Nonce Size
Nonce Reuse
Tag Size
ChaCha20-Poly1305
Stream, AEAD
256 bits
96 bits
Catastrophic
128 bits
XChaCha20-Poly1305
Stream, AEAD
256 bits
192 bits
Catastrophic
128 bits
AES-GCM
Stream, AEAD
128, 192, 256 bits
96 bits
Catastrophic
128 bits
AES-GCM-SIV
Stream, AEAD
128 or 256 bits
96 bits
Reduced risk
128 bits
Authenticated encryption is used in most of our modern protocols, including
TLS, S/MIME, PGP/GPG, and many more. Failure to implement authenticated
encryption correctly has lead to some serious issues in the past.
Whenever you’re using encryption, ask yourself: how is integrity and
authentication verified? And remember: it’s essential to verify the
authenticity of data before using it.
I hope you enjoyed this article! As usual, if you have any suggestions or
spotted some mistakes, let me know in the comments or by contacting me!
From an early age, I’ve been very interested in computers. They were fascinating and profoundly powerful devices, capable of doing things that were (and in some ways still are) beyond my understanding. Like many of us, I started on Windows, and like many of us who started out computing when we were kids, I did a lot of damage to my first computer. I still haven’t grown out of breaking computers, I’ve just gotten a lot better at planning how I’m going to break them and what to do in order to fix them after the fact.
Fast forward several years. I’ve heard of Linux, read books about how to do some things like web development on Linux, and am thinking about trying Linux. I wanted to do music recording at the time too (something I’m still planning on doing). I didn’t have a working Internet connection, so I got in the car with my friend and went to a place that had Internet in order to download some Linux distros to try out.
Thanks for reading Arraybolt's Archives! Subscribe for free to receive new posts and support my work.
The first distro I ever used was HP QuickWeb (Splashtop OS), which came with my Windows 7 laptop. I knew nothing about Linux at the time, I just found this QuickWeb thing included with my computer and decided to try it out in order to keep things isolated from my Windows installation. (In retrospect, this probably didn’t work at all, but I didn’t know that…) Basically it was nothing more than a Linux kernel, a simple and glitchy desktop, and a severely outdated copy of Firefox that I didn’t know better than to use. From this insecure and buggy mess, my journey into Linux began.
After fighting with QuickWeb’s browser for a bit, I learned about a distro called KXStudio, which was supposed to be geared towards music production. (KXStudio has since switched from being a distro to being a third-party repo that you use on top of Kubuntu, which severely confused me, since I didn’t know the difference between a repo and a distro at the time.) After a lot of searching, I managed to find KXStudio 14.04, from back when KXStudio was still a distro. I downloaded it, along with a few other distros that I ultimately didn’t end up getting to work, and also grabbed VirtualBox so that I could run the distros in a VM.
Let’s just say that combining a buggy distro with a total newbie resulted in chaos.
I went through quite a few “stages” learning how Linux worked, everything from “Where on earth is my C: drive?” to “Great, I just broke my audio, now what?”. It was pretty interesting. I also had to deal with the glitchiness and complexity of VirtualBox, as well as the fact that the distro I was using was not exactly newbie-friendly. The whole thing was a total mess. A very fun total mess. Before long, I had decided to install Linux on some physical hardware that I had available (which led to even more chaos, especially when UEFI came into the picture), and after days of headaches, many reinstalls, and tons of breakage, I was finally getting good enough at Linux usage to start using it as my daily driver on three different computers (a desktop and two laptops).
My third distro was ChaletOS 16.04. ChaletOS was essentially a modded Xubuntu variant made to look and feel a lot like Windows. It also has the Wine Windows Compatibility Layer preinstalled, making it compatible with a lot of Windows software out of the box. I didn’t use Wine very much - I was more interested in the beautiful UI, mostly smooth operation, and powerful features it provided. I was starting to get into more advanced Linux usage, and was for the first time using a sorta-halfway supported OS (all the ones I had used up until then were already EOL or very nearly EOL, if I remember correctly). I also had a working Internet connection, which greatly improved my ability to download software and research things. At this point I was almost entirely “sold” on Linux, and was only still using Windows on one laptop. That one laptop was important because it had a development environment on it that ran on Windows 7 inside a Hyper-V virtual machine inside Windows 8.0 Pro. I was reluctant to switch to Linux there because I knew Hyper-V wouldn’t run on Linux and I didn’t want to mess up the Windows 7 VM’s activation status by moving from Hyper-V to QEMU.
While using ChaletOS on my other laptop (not the one with the important development environment), I tried a bunch of different distros in VMs, including really weird ones like BasicLinux (don’t ever use it, it’s a nightmare) and Calculate Linux (Gentoo derivative, that was a total fail because hilariously I couldn’t figure out what the package manager’s name was). And then, in the midst of my distro-hopping adventures, I stumbled across two distros that almost immediately became my favorites. It was around the time of April 2020. Ubuntu 20.04 and its flavors has just been released recently. And it was during that time I discovered Kubuntu and Lubuntu.
I can’t remember which one I tried first, but I remember quite well some of my initial experiments with them. After playing with both Kubuntu and Lubuntu in VMs, I decided to install Lubuntu onto a flash drive and boot my non-development laptop from it. The extremely smooth experience Lubuntu 20.04 provided, the beautiful and simple UI, everything was almost perfect. Kubuntu was really nice, but it was a bit heavier and the laptop I was using for testing only had 4 GB RAM and no internal hard drive (it always booted from a USB stick, or from an SD card using a hack), so a lightweight OS was important to me. I used that OS a lot, and even booted my important development laptop from my Lubuntu USB stick to see what would happen (it worked quite well).
While I ended up loving Lubuntu, I was somewhat partial to Kubuntu since it was the basis of KXStudio, and the extreme power of the KDE desktop was a serious plus that Lubuntu’s LXQt desktop didn’t have. I had ended up having a slight mess with my Windows 7 VM on the development laptop which resulted in me having to re-activate it (long story short, don’t expect to restore a Hyper-V VM from a backup of only the .vhd files without the rest of the VM config!), and as a result, I was no longer so scared of having to move my VM to Linux using QEMU. I took the plunge and wiped Windows 8 with Kubuntu 20.04 on my main development laptop, then got my Windows 7 VM set up inside QEMU (where it actually worked better than it did on Hyper-V). Everything was working exactly as I had hoped (except for one small hiccup that resulted in an OS reinstall due to a combination of inexperience and paranoia, but that was overcome without too much trouble in the long run).
After a long bit of life lived in 20.04-land (and a brief stint with Lubuntu and Ubuntu Studio 21.10), Ubuntu 22.04 was released. I had finally mostly gotten a grip on how Ubuntu and Linux really worked, and was interested in trying to help with Ubuntu’s development. I had looked into helping in the past, but hadn’t actually gone ahead and tried to help. I joined the #lubuntu-devel channel with the nick “Guest3” and asked about the possibility of writing software documentation for Lubuntu. I surprisingly got replied to almost immediately by two members of the Lubuntu team, switched my IRC nick from Guest3 to arraybolt3, and before long was helping with testing, documentation writing, and debugging. (It all just kinda happened - I was there and stuff was happening and I ended up getting to try to help and it all worked really well.) I ended up helping with the development of Lubuntu and a couple of the other Ubuntu flavors during the Kinetic Kudu development cycle and the 22.04.1 release, and six hectic months of excitement later, Ubuntu 22.10 and its flavors were released.
That first step into the Ubuntu development world was about 10 months ago. We are now nearing the release of Ubuntu 23.04, and have since released 20.04.5 and 22.04.2. The joy of getting to contribute to an OS I’ve been using for years has been amazing, and I look forward to continuing to contribute in the future.
That was about four years of computing packed into a short article. Hope you found this enjoyable and helpful, and see you later!
Thanks for reading Arraybolt's Archives! Subscribe for free to receive new posts and support my work.
Xubuntu Minimal provides a lighter-weight alternative to the full-size Xubuntu desktop. The core desktop experience allows you to build your very own version of Xubuntu with the tricky configuration already done for you. Scroll through the screenshots below to see Xubuntu Minimal in action. When you&aposre done, download Xubuntu Minimal from cdimage.ubuntu.com to test it for yourself!
The Xubuntu Minimal desktop looks identical to the standard Xubuntu desktop.The application menu is very empty, featuring only the Accessories, Settings, and System categories.While browsing all applications (continued in the next two screenshots), you&aposll find only the essentials.The Mail Reader and Web Browser apps won&apost do much until you&aposve added your own.So, fire up the Xfce Terminal and apt install or snap install your favorites. NetworkManager is still included, so you can configure your network settings.The notification plugin is here, no loss of functionality.The PulseAudio plugin is emptier than normal, since there are no audio apps installed. Note: Xubuntu Minimal ships with PipeWire but is still controlled by the PulseAudio plugin.The clock is here to remind you of the time you saved while installing a much smaller Ubuntu flavor.The Settings Manager has a few less settings than normal, but is still fully functional.Here&aposs the bottom half of your available settings.No visual tour would be complete without a neofetch screenshot. You&aposre welcome.
I’m happy to announce that Netplan version 0.106 is now available on GitHub and is soon to be deployed into an Ubuntu/Debian/Fedora installation near you! Six months and 65 commits after the previous version, this release is brought to you by 4 free software contributors from around the globe.
Highlights
Highlights of this release include the new netplan status command, which queries your system for IP addresses, routes, DNS information, etc… in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine readable YAML/JSON format).
Furthermore, we implemented a clean libnetplan API which can be used by external tools to parse Netplan configuration, migrated away from non-inclusive language (PR#303) and improved the overall Netplan documentation. Another change that should be noted, is that the match.macaddress stanza now only matches on PermanentMACAddress=on the systemd-networkd backend, as has been the case on the NetworkManager backend ever since (see PR#278 for background information on this slight change in behavior).
Left shifting values seems simple, but the following code contains a bug:
The literal value 1 is actually a signed int, so the promotion to a uint64_t type will sign extend the value before assigning to x, causing x to be 0xffffffff80000000.
The fix is simple, just make the literal value 1 an unsigned int using 1U instead of 1.
This is the story of the currently progressing changes to secure boot
on Ubuntu and the history of how we got to where we are.
taking a step back: how does secure boot on Ubuntu work?
Booting on Ubuntu involves three components after the firmware:
shim
grub
linux
Each of these is a PE binary signed with a key. The shim is signed by Microsoft’s
3rd party key and embeds a self-signed Canonical CA certificate, and optionally a
vendor dbx (a list of revoked certificates or binaries). grub and linux (and fwupd)
are then signed by a certificate issued by that CA
In Ubuntu’s case, the CA certificate is sharded: Multiple people each have a part
of the key and they need to meet to be able to combine it and sign things, such as
new code signing certificates.
BootHole
When BootHole happened in 2020, travel was suspended and we hence could not rotate
to a new signing certificate. So when it came to updating our shim for the CVEs, we
had to revoke all previously signed kernels, grubs, shims, fwupds by their hashes.
This generated a very large vendor dbx which caused lots of issues as shim exported
them to a UEFI variable, and not everyone had enough space for such large variables.
Sigh.
We decided we want to rotate our signing key next time.
This was also when upstream added SBAT metadata to shim and grub. This gives
a simple versioning scheme for security updates and easy revocation using a
simple EFI variable that shim writes to and reads from.
Spring 2022 CVEs
We still were not ready for travel in 2021, but during BootHole we developed the
SBAT mechanism, so one could revoke a grub or shim by setting a single EFI variable.
We actually missed rotating the shim this cycle as a new vulnerability was reported
immediately after it, and we decided to hold on to it.
2022 key rotation and the fall CVEs
This caused some problems when the 2nd CVE round came, as we did not have a shim
with the latest SBAT level, and neither did a lot of others, so we ended up deciding
upstream to not bump the shim SBAT requirements just yet. Sigh.
Anyway, in October we were meeting again for the first time at a Canonical sprint,
and the shardholders got together and created three new signing keys: 2022v1, 2022v2,
and 2022v3. It took us until January before they were installed into the signing service
and PPAs setup to sign with them.
We also submitted a shim 15.7 with the old keys revoked which came back at around
the same time.
Now we were in a hurry. The 22.04.2 point release was scheduled for around middle
of February, and we had nothing signed with the new keys yet, but our new shim
which we need for the point release (so the point release media remains bootable
after the next round of CVEs), required new keys.
So how do we ensure that users have kernels, grubs, and fwupd signed with the
new key before we install the new shim?
upgrade ordering
grub and fwupd are simple cases: For grub, we depend on the new version. We decided
to backport grub 2.06 to all releases (which moved focal and bionic up from 2.04), and
kept the versioning of the -signed packages the same across all releases, so we were
able to simply bump the Depends for grub to specify the new minimum version. For fwupd-efi,
we added Breaks.
(Actually, we also had a backport of the CVEs for 2.04 based grub, and we did publish that
for 20.04 signed with the old keys before backporting 2.06 to it.)
Kernels are a different story: There are about 60 kernels out there. My initial idea was
that we could just add Breaks for all of them. So our meta package linux-image-generic which
depends on linux-image-$(uname -r)-generic, we’d simply add Breaks: linux-image-generic (« 5.19.0-31)
and then adjust those breaks for each series. This would have been super annoying, but
ultimately I figured this would be the safest option. This however caused concern, because
it could be that apt decides to remove the kernel metapackage.
I explored checking the kernels at runtime and aborting if we don’t have a trusted
kernel in preinst. This ensures that if you try to upgrade shim without having a kernel,
it would fail to install. But this ultimately has a couple of issues:
It aborts the entire transaction at that point, so users will be unable to run
apt upgrade until they have a recent kernel.
We cannot even guarantee that a kernel would be unpacked first. So even if you got
a new kernel, apt/dpkg might attempt to unpack it first and then the preinst would fail
because no kernel is present yet.
Ultimately we believed the danger to be too large given that no kernels had yet been released
to users. If we had kernels pushed out for 1-2 months already, this would have been a viable
choice.
So in the end, I ended up modifying the shim packaging to install both the latest shim and
the previous one, and an update-alternatives alternative to select between the two:
In it’s post-installation maintainer script, shim-signed checks whether all kernels with a
version greater or equal to the running one are not revoked, and if so, it will setup the
latest alternative with priority 100 and the previous with a priority of 50.
If one or more of those kernels was signed with a revoked key, it will swap the priorities
around, so that the previous version is preferred.
Now this is fairly static, and we do want you to switch to the latest shim eventually, so
I also added hooks to the kernel install to trigger the shim-signed postinst script when
a new kernel is being installed. It will then update the alternatives based on the current
set of kernels, and if it now points to the latest shim, reinstall shim and grub to the
ESP.
Ultimately this means that once you install your 2nd non-revoked kernel, or you install
a non-revoked kernel and then reconfigure shim or the kernel, you will get the latest
shim. When you install your first non-revoked kernel, your currently booted kernel is
still revoked, so it’s not upgraded immediately. This has a benefit in that you will
most likely have two kernels you can boot without disabling secure boot.
regressions
Of course, the first version I uploaded had still some remaining hardcoded “shimx64”
in the scripts and so failed to install on arm64 where “shimaa64” is used. And if that
were not enough, I also forgot to include support for gzip compressed kernels there.
Sigh, I need better testing infrastructure to be able to easily run arm64 tests as
well (I only tested the actual booting there, not the scripts).
shim-signed migrated to the release pocket in lunar fairly quickly, but this caused
images to stop working, because the new shim was installed into images, but no
kernel was available yet, so we had to demote it to proposed and block migration.
Despite all the work done for end users, we need to be careful to roll this out for
image building.
another grub update for OOM issues.
We had two grubs to release: First there was the security update for the recent set
of CVEs, then there also was an OOM issue for large initrds which was blocking critical
OEM work.
We fixed the OOM issue by cherry-picking all 2.12 memory management patches, as well
as the red hat patches to the loader we take from there. This ended up a fairly large
patch set and I was hesitant to tie the security update to that, so I ended up pushing
the security update everywhere first, and then pushed the OOM fixes this week.
With the OOM patches, you should be able to boot initrds of between 400M and 1GB, it
also depends on the memory layout of your machine and your screen resolution and background
images. So OEM team had success testing 400MB irl, and I tested up to I think it was 1.2GB
in qemu, I ran out of FAT space then and stopped going higher :D
other features in this round
Intel TDX support in grub and shim
Kernels are allocated as CODE now not DATA as per the upstream mm changes, might fix boot on X13s
am I using this yet?
The new signing keys are used in:
shim-signed 1.54 on 22.10+, 1.51.3 on 22.04, 1.40.9 on 20.04, 1.37~18.04.13 on 18.04
grub2-signed 1.187.2~ or newer (binary packages grub-efi-amd64-signed or grub-efi-arm64-signed),
1.192 on 23.04.
fwupd-signed 1.51~ or newer
various linux updates. Check apt changelog linux-image-unsigned-$(uname -r) to see if
Revoke & rotate to new signing key (LP: #2002812) is mentioned in there to see if it
signed with the new key.
If you were able to install shim-signed, your grub and fwupd-efi will have the correct
version as that is ensured by packaging. However your shim may still point to the old one.
To check which shim will be used by grub-install, you can check the status of the shimx64.efi.signed
or (on arm64) shimaa64.efi.signed alternative. The best link needs to point to the file ending in
latest:
$ update-alternatives --display shimx64.efi.signed
shimx64.efi.signed - auto mode
link best version is /usr/lib/shim/shimx64.efi.signed.latest
link currently points to /usr/lib/shim/shimx64.efi.signed.latest
link shimx64.efi.signed is /usr/lib/shim/shimx64.efi.signed
/usr/lib/shim/shimx64.efi.signed.latest - priority 100
/usr/lib/shim/shimx64.efi.signed.previous - priority 50
If it does not, but you have installed a new kernel compatible with the new shim, you can
switch immediately to the new shim after rebooting into the kernel by running dpkg-reconfigure shim-signed. You’ll see in the output if the shim was updated, or you can check the output
of update-alternatives as you did above after the reconfiguration has finished.
For the out of memory issues in grub, you need grub2-signed 1.187.3~ (same binaries
as above).
how do I test this (while it’s in proposed)?
upgrade your kernel to proposed and reboot into that
upgrade your grub-efi-amd64-signed, shim-signed, fwupd-signed to proposed.
If you already upgraded your shim before your kernel, don’t worry:
upgrade your kernel and reboot
run dpkg-reconfigure shim-signed
And you’ll be all good to go.
deep dive: uploading signed boot assets to Ubuntu
For each signed boot asset, we build one version in the latest stable release and the
development release. We then binary copy the built binaries from the latest stable release
to older stable releases. This process ensures two things: We know the next stable release
is able to build the assets and we also minimize the number of signed assets.
OK, I lied. For shim, we actually do not build in the development release but copy the
binaries upward from the latest stable, as each shim needs to go through external signing.
The entire workflow looks something like this:
Upload the unsigned package to one of the following “build” PPAs:
Copy the unsigned package back across all stable releases in the PPA
Upload the signed package for stable releases to the same PPA with ~<release>.1 appended to the version
Submit a request to canonical-signing-jobs to sign the uploads.
The signing job helper copies the binary -unsigned packages to the primary-2022v1 PPA where they are
signed, creating a signing tarball, then it copies the source package for the -signed package to the
same PPA which then downloads the signing tarball during build and places the signed assets into
the -signed deb.
This step is not strictly necessary, but it enables tools like sru-review to work, as they cannot access the packages from the normal private “proposed” PPA.
Binary copy from proposed-public to the proposed queue(s) in the primary archive
Lots of steps!
WIP
As of writing, only the grub updates have been released, other updates are still being
verified in proposed. An update for fwupd in bionic will be issued at a later point, removing
the EFI bits from the fwupd 1.2 packaging and using the separate fwupd-efi project instead
like later release series.
In 1701, Asano Naganori, a feudal lord in Japan, was summoned to the shogun’s court in Edo, the town now called Tokyo. He was a provincial chieftain, and knew little about court etiquette, and the etiquette master of the court, Kira Kozuke-no-Suke, took offence. It’s not exactly clear why; it’s suggested that Asano didn’t bribe Kira sufficiently or at all, or that Kira felt that Asano should have shown more deference. Whatever the reasoning, Kira ridiculed Asano in the shogun’s presence, and Asano defended his honour by attacking Kira with a dagger.
Baring steel in the shogun’s castle was a grievous offence, and the shogun commanded Asano to atone through suicide. Asano obeyed, faithful to his overlord. The shogun further commanded that Asano’s retainers, over 300 samurai, were to be dispossessed and made leaderless, and forbade those retainers from taking revenge on Kira so as to prevent an escalating cycle of bloodshed. The leader of those samurai offered to divide Asano’s wealth between all of them, but this was a test. Those who took him up on the offer were paid and told to leave. Forty-seven refused this offer, knowing it to be honourless, and those remaining 47 reported to the shogun that they disavowed any loyalty to their dead lord. The shogun made them rōnin, masterless samurai, and required that they disperse. Before they did, they swore a secret oath among themselves that one day they would return and avenge their master. Then each went their separate ways. These 47 rōnin immersed themselves into the population, seemingly forgoing any desire for revenge, and acting without honour to indicate that they no longer followed their code. The shogun sent spies to monitor the actions of the rōnin, to ensure that their unworthy behaviour was not a trick, but their dishonour continued for a month, two, three. For a year and a half each acted dissolutely, appallingly; drunkards and criminals all, as their swords went to rust and their reputations the same.
A year and a half later, the forty-seven rōnin gathered together again. They subdued or killed and wounded Kira’s guards, they found a secret passage hidden behind a scroll, and in the hidden courtyard they found Kira and demanded that he die by suicide to satisfy their lord’s honour. When the etiquette master refused, the rōnin cut off Kira’s head and laid it on Asano’s grave. Then they came to the shogun, surrounded by a public in awe of their actions, and confessed. The shogun considered having them executed as criminals but instead required that they too die by suicide, and the rōnin obeyed. They were buried, all except one who was not present and who lived on, in front of the tomb of their master. The tombs are a place to be visited even today, and the story of the 47 rōnin is a famous one both inside and outside Japan.
You might think: why have I been told this story? Well, there were 47 of them. 47 is a good number. It’s the atomic number of silver, which is interesting stuff; the most electrically conductive metal. (During World War II, the Manhattan Project couldn’t get enough copper for the miles of wiring they needed because it was going elsewhere for the war effort, so they took all the silver out of Fort Knox and melted it down to make wire instead.) It’s strictly non-palindromic, which means that it’s not only not a palindrome, it remains not a palindrome in any base smaller than itself. And it’s how old I am today.
Yes! It’s my birthday! Hooray!
I have had a good birthday this year. The family and I had delightful Greek dinner at Mythos in the Arcadian, and then yesterday a bunch of us went to the pub and had an absolute whale of an afternoon and evening, during which I became heartily intoxicated and wore a bag on my head like Lord Farrow, among other things. And I got a picture of the Solvay Conference from Bruce.
This year is shaping up well; I have some interesting projects coming up, including one will-be-public thing that I’ve been working on and which I’ll be revealing more about in due course, a much-delayed family thing is very near its end (finally!), and in general it’s just gotta be better than the ongoing car crash that the last few years have been. Fingers crossed; ask me again in twelve months, anyway. I’ve been writing these little posts for 21 years now (last year has more links) and there have been ups and downs, but this year I feel quite hopeful about the future for the first time in a while. This is good news. Happy birthday, me.
This morning I attempted to start work on my desktop PC and couldn’t. The screen is black, it doesn’t want to wake up the displays. I used the old REISUB trick to restart, and it boots, but there’s no output on the display. I did some investigation and this post is mainly to capture my notes and so others can see the problem and perhaps debug and fix it.
The setup is an Intel Skull Canyon NUC connected to an external GPU enclosure which contains an NVIDIA GeForce RTX 2060.
At KDE we have multiple levels of quality assurance ranging from various degrees of a humans testing features to fully automated testing. Indeed automated testing is incredibly important for the continued quality of our software. A big corner stone of our testing strategy are so called unit tests, they test a specific piece of our software for its behavior in isolation. But for many aspects of our software we need a much higher level view, testing pieces of Plasma’s application launcher in isolation is all good and well but that won’t tell us if the entire UI can be easily navigated using the keyboard. For this type of test we require a different testing approach altogether. A couple months ago I’ve set set out to create a testing framework for this use case and I’m glad to say that it has matured enough to be used for writing tests. I’d like to walk you through the technical building blocks and a simple example.
Let us start of by looking at the architecture at large. So… there’s Selenium which is an incredibly popular, albeit web-oriented, testing framework. Its main advantages for us are its popularity and that it sports a server-client split. This means we can leverage the existing client tooling available for Selenium without having to write anything ourselves, we only need to grow a server. The server component, called a WebDriver, implements the actual interaction with UI elements and is generic enough to also apply to desktop applications. Indeed so thought others as well: there already exists Appium – it extends Selenium with more app-specific features and behaviors. Something for us to build upon. The clients meanwhile are completely separate and talk to the WebDriver over a well defined JSON REST protocol, meaning we can reuse the existing clients without having to write anything ourselves. They are available in a multitude of programming languages, and who knows maybe we’ll eventually get one for writing Selenium tests in QML
That of course doesn’t explain how GUI testing can work with this on Linux. Enter: AT-SPI. AT-SPI is an accessibility API and pretty much the standard accessibility system for use on Linux. Obviously its primary use is assistive technologies, like the screen reader Orca, but to do its job it essentially offers a toolkit-independent way of introspecting and interacting with GUI applications. This then gives us a way to implement a WebDriver without caring about the toolkit or app specifics. As long as the app supports AT-SPI, which all Qt apps do implicitly, we can test it.
Since all the client tooling is independent of the server all we needed to get GUI testing going was a WebDriver that talks to AT-SPI.
That is what I set out to write and I’m happy to announce that we now have an AT-SPI based WebDriver, and the first tests are popping into existence already. There is also lovely documentation to hold onto.
So, without further ado. Let us write a simple test. Since the documentation already writes one in Python I’ll use Ruby this time around so we have some examples of different languages. A simple candidate is KInfoCenter. We can test its search functionality with a couple of lines of code.
First we need to install selenium-webdriver-at-spi, clone it, cmake build it, and cmake install it. You’ll also need to install the relevant client libraries. For ruby that’s simply running gem install appium_lib.
Then we can start with writing our test. We will need some boilerplate setup logic. This is more or less the same for every test. For more details on the driver setup you may also check the wiki page.
The driver will take care of starting the correct application and make sure that it is actually running correctly. Next we’ll write the actual test. Let’s test the search. The first order of business is using a tool called Accerciser to inspect the AT-SPI presentation of the application. For more information on how to use this tool please refer to the wiki. Using Accerciser I’ve located the search field and learned that it is called ‘Search’. So, let’s locate it and activate it, search for the CPU module:
To run the complete test we can use the run wrapper: selenium-webdriver-at-spi-run ./kinfocentertest.rb (mind that it needs to be +x). If all has gone well we should get a successful test.
Finished in 1.345276s, 0.7433 runs/s, 1.4867 assertions/s.
1 runs, 2 assertions, 0 failures, 0 errors, 0 skips
I, [2022-12-14T13:13:53.508516 #154338] INFO -- : tests done
I, [2022-12-14T13:13:53.508583 #154338] INFO -- : run.rb exiting true
This should get you started with writing a test for your application! I’ll gladly help and review your forthcoming tests. For more detailed documentation check out the writing-tests wiki page as well as the appium command reference.
Of course the work is not done. selenium-webdriver-at-spi is very much still a work in progress and I’d be glad for others to help add features as they become needed. The gitlab project is the place for that.
The complete code of the example above:
#!/usr/bin/env ruby
# frozen_string_literal: true
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2022 Harald Sitter <sitter@kde.org>
require 'appium_lib'
require 'minitest/autorun'
class TestKInfoCenter < Minitest::Test
attr_reader :driver
def setup
@appium_driver = Appium::Driver.new(
{
'caps' => { app: 'org.kde.kinfocenter.desktop' },
'appium_lib' => {
server_url: 'http://127.0.0.1:4723',
wait_timeout: 10,
wait_interval: 0.5
}
}, true
)
@driver = @appium_driver.start_driver
end
def teardown
driver.quit
end
def test_search
search = driver.find_element(:name, 'Search')
search.click
search.send_keys('cpu')
cpu = driver.find_element(:class_name, '[list item | CPU]')
assert(cpu.displayed?)
cpu.click
cpu_tab = driver.find_element(:class_name, '[page tab | CPU]')
assert(cpu_tab.displayed?)
end
end
It’s a Plasma widget that visualizes what’s going on on your system, music-wise that is. I’ve started this project years ago but only recently found the motivation to get it to a somewhat acceptable state. It’s pretty amazing to have bars flying across the screen to Daft Punk’s `Touch`.
Except where otherwise noted, this issue of the Ubuntu Weekly Newsletter is licensed under a 
Kommit 1.0.2
Xubuntu 23.04, featuring the latest updates from Xfce 4.18 and GNOME 44.
Xubuntu supports emoji out of the box, albeit, of a less colorful variety.
Color emoji are much more interesting to look at and can be easier to understand.
Emote adds a handy interface for adding emoji in any app.
Example of applying a block cipher to an uncompressed image. The original colors are lost, but the overall layout of the image is still understandable. That's because multiple blocks of the image (containing the RGB values of each pixel), for example from the white background, are repeated multiple times, yielding the same exact encrypted blocks. The inspiration for making this image came from 
Data flow during a ChaCha20-Poly1305 encryption. This shows the inputs in 
Data flow during an AES-GCM encryption. This shows the inputs in 
Updated data flow during a ChaCha20-Poly1305 encryption which shows where the Associated Data (AE) is placed.
Updated data flow during an AES-GCM encryption which shows where the Associated Data (AE) is placed.
The Xubuntu Minimal desktop looks identical to the standard Xubuntu desktop.
The application menu is very empty, featuring only the Accessories, Settings, and System categories.
While browsing all applications (continued in the next two screenshots), you&aposll find only the essentials.
The Mail Reader and Web Browser apps won&apost do much until you&aposve added your own.
So, fire up the Xfce Terminal and 
NetworkManager is still included, so you can configure your network settings.
The notification plugin is here, no loss of functionality.
The PulseAudio plugin is emptier than normal, since there are no audio apps installed. Note: Xubuntu Minimal ships with PipeWire but is still controlled by the PulseAudio plugin.
The clock is here to remind you of the time you saved while installing a much smaller Ubuntu flavor.
The Settings Manager has a few less settings than normal, but is still fully functional.
Here&aposs the bottom half of your available settings.
No visual tour would be complete without a neofetch screenshot. You&aposre welcome.
