March 31, 2023

Lubuntu 23.04 Beta Released!

Lubuntu Blog

Thanks to all the hard work from our contributors, Lubuntu 23.04 Beta has been released. With the codename Lunar Lobster, Lubuntu 23.04 will be the 24th release of Lubuntu, the tenth release of Lubuntu with LXQt as the default desktop environment. Support lifespan With 23.04 being an interim release, it will follow the standard non-LTS […]

on March 31, 2023 01:42 AM

March 30, 2023

Time to prepare for Ubuntu 18.04 LTS end of standard support on 31 May 2023 – Options for AWS users.

Ubuntu Blog

As mentioned in our recent blog post, Ubuntu 18.04 LTS ‘Bionic Beaver‘ will reach the end of the standard five-year maintenance window on 31 May 2023. This means that Ubuntu 18.04 LTS servers will stop receiving updates unless you are an Ubuntu Pro subscriber.

With an Ubuntu Pro subscription, your Ubuntu 18.04 LTS deployment will continue to be fully supported until 2028.

This blog sets out what you need to know if you are still running your EC2, ECS, EKS workloads on Ubuntu 18.04 LTS.

Note: If you are running Ubuntu 18.04 LTS on any other AWS service other than EC2, EKS and ECS, please contact AWS support for upgrading options.

Upgrading Ubuntu to newer versions

You might consider upgrading to a new LTS release. If so, you can either upgrade to Ubuntu 20.04 LTS or freshly install Ubuntu 20.04 or 22.04 LTS and upgrade your application accordingly.

For more information see Ubuntu Server upgrade guide ›

Ubuntu Pro

If you need more time to plan your upgrade, or you want to extend the economic life of a project that does not need upgrading, you can use Ubuntu Pro.

Ubuntu Pro is Canonical’s security and compliance subscription. It’s the same Ubuntu we all know and love but enhanced with extra services. It provides access to `esm-infra`, which extends the LTS release coverage from 5 to 10 years from the release date, allowing for continued security fixes for high and critical common vulnerabilities and exposures (CVEs) for x86-64 and arm64 architectures.

Ubuntu Pro also includes access to `esm-apps`, which covers security patching for 3rd party open source applications, present in the Ubuntu Universe repository, with over 23,000 packages.

This access enables organisations with workloads running on Ubuntu LTS releases to maintain compliance standards by providing a secure environment before upgrading.

Learn more at: ubuntu.com/pro

How to get Ubuntu Pro on AWS

There are two ways to get Ubuntu Pro on AWS:

Ubuntu Pro on AWS EC2 and Marketplace for fresh instances

If your workload is ephemeral or easy to redeploy, for example, if it is regularly rebuilt in a CI/CD, you can launch fresh instances of Ubuntu Pro 18.04 LTS from different sources on AWS:

Using the AWS Marketplace to find the Ubuntu Pro 18.04 LTS listing
Using AWS CLI tool to get the Ubuntu Pro 18.04 AMI:

aws ec2 describe-images --filters 
"Name=name,Values=ubuntu-pro-server/images/hvm-ssd/ubuntu-bionic-18.04-amd64*"
--query 'sort_by(Images, &CreationDate)[-1].{Name: Name, ImageId: ImageId, CreationDate: CreationDate, Owner:OwnerId}'

Ubuntu Pro activation tokens for an in-place upgrade to Pro

When a re-deploy is not viable, you can still upgrade to Pro through an activation token from Canonical.

To get a Pro Token, you can contact us directly at ubuntu.com/pro or via our marketplace listing in AWS Marketplace: Ubuntu Pro Upgrade – Annual Subscription

Through this mechanism you will need to purchase the tokens at an annual rate for the number of EC2 instances. You may make the purchase directly with Canonical or through a Private Offer on the AWS Marketplace. Once purchased, you will get a Pro token that you can attach to your instances, either manually or in an automated way via this command:

`sudo pro attach <MY_TOKEN>`

Note: Make sure you have updated the following components:

ubuntu-advantage-tools: `sudo apt install ubuntu-advantage-tools`
Updated snap for livepatch

More information at:

on March 30, 2023 01:43 PM

March 29, 2023

OpenStack 2023.1 Antelope for Ubuntu 22.04 LTS

Corey Bryant

The Ubuntu OpenStack team at Canonical is pleased to announce the general availability of OpenStack 2023.1 Antelope on Ubuntu 22.04 LTS (Jammy Jellyfish).

Details of the Antelope release

Ubuntu 22.04 LTS

The Ubuntu Cloud Archive for OpenStack 2023.1 Antelope can be enabled on Ubuntu 22.04 by running the following command:
sudo add-apt-repository cloud-archive:antelope

The Ubuntu Cloud Archive for 2023.1 Antelope includes updates for:

aodh, barbican, ceilometer, cinder, designate, designate-dashboard, dpdk (22.11.1), glance, gnocchi, heat, heat-dashboard, horizon, ironic, ironic-ui, keystone, magnum, magnum-ui, manila, manila-ui, masakari, mistral, murano, murano-dashboard, networking-arista, networking-bagpipe, networking-baremetal, networking-bgpvpn, networking-hyperv, networking-l2gw, networking-mlnx, networking-odl, networking-sfc, neutron, neutron-dynamic-routing, neutron-fwaas, neutron-taas, neutron-vpnaas, nova, octavia, octavia-dashboard, openstack-trove, openvswitch (3.1.0), ovn (23.03.0), ovn-octavia-provider, placement, sahara, sahara-dashboard, senlin, swift, trove-dashboard, vitrage, watcher, watcher-dashboard, zaqar, and zaqar-ui.

For a full list of packages and versions, please refer to the Antelope version report.

Reporting bugs

If you have any issues please report bugs using the ‘ubuntu-bug’ tool to
ensure that bugs get logged in the right place in Launchpad:

sudo ubuntu-bug nova-conductor

Thank you to everyone who contributed to OpenStack 2023.1 Antelope!

Corey
(on behalf of the Ubuntu OpenStack Engineering team)

on March 29, 2023 07:30 PM

3 reasons why OTA updates are important [Part II]

Ubuntu Blog

In the first part of this blog series, we covered what the term ‘OTA’ means and some key advantages and considerations of using OTA technology in IoT. Read more about what OTA means in the blog post here.

Devices are everywhere but they all need maintenance. Maintaining household gadgets such as laptops, printers and smart watches has become easier over the years as update technology evolves. Today, users do not have to even think about updates – they can seamlessly take place in the background or overnight. These updates take place over-the-air (OTA) and have revolutionised the way that we keep hardware, software or firmware maintained. Now devices can easily stay updated, even when distributed across a city, country or continent.

OTA solutions are important for a number of reasons. Delivering consistent, reliable and robust software updates to any platform is a huge challenge. Delivering such updates to low-powered, inaccessible, and often remotely administered embedded IoT devices, anywhere in the world, is intricately harder. That is why being able to roll-out OTA updates at scale should be an important consideration for device manufacturers and software developers who are looking to choose an operating system to develop on. Here’s how OTA updates make their lives easier.

1. Security

One of the key reasons why OTA is so important is the agility to keep devices secure. The ability for devices to receive updates without being physically plugged means that the manufacturer of the device or software that is being used can easily release updates at any frequency. If a new security vulnerability is discovered, within a few hours the fix can be developed and the patch sent out to all affected devices. This technology enables devices to be kept secure much more easily. Having frequent patching encourages more secure software.

2. Product development

Using OTA technology doesn’t just promote security updates to be sent to devices. It also encourages new functionality to be rolled out. When developers create a new feature, for example, the software can be updated rapidly over-the-air. There’s no need to wait for more features to be ready and do a larger rollout. With OTA updates, it’s up to the hardware or software manufacturer to decide how rapidly they want to push new functionality. Sometimes there is an unexpected bug in the software too. With OTA updates, a fix can easily be sent out to resolve the issue. This ability to roll out features on a more frequent cadence encourages more rapid product development as new functionality can evolve more quickly.

3. Cost-effective maintenance

If devices need to be plugged in each time they require an update, either for improved security or functionality, it would mean that the devices may not receive updates in a timely manner. They may be left in a vulnerable state, either in users’ hands or left in the field. With the power of OTA updates, devices can be distributed around the world, in places where it may not be feasible for a technician to reach them regularly. Choosing OTA technologies improves product experience and opens up the way devices can be used, as you are not limited by the physical location of the hardware. Minimising the need for technicians to have physical access to hardware saves a huge amount of time and money. Automating updates using management platforms and over-the-air technology further saves time and money, and can hugely simplify maintenance at scale.

OTA updates in the field

Canonical offers sophisticated OTA technology for IoT devices and computers around the world. A number of IoT customers use this technology in the field to power their devices with Ubuntu Core. One such customer is Screenly, a world-leading San Francisco-based company transforming screens into powerful digital signs. Their software powers over 10,000 screens across 75 countries.

To keep deployed devices up-to-date with the latest security fixes and features, Screenly originally relied on home-grown infrastructure to deliver over-the-air (OTA) patches to devices in the field. However, this framework was expensive and time consuming to maintain. As the fleet scaled up, devices were occasionally being left nonfunctional in partially updated states. This forced the company’s development team to spend significant time writing recovery code.

Rather than continuing to spend time and resources on its in-house OTA infrastructure, Screenly decided to look for an existing solution that would enable it to leverage the established expertise of a trusted third-party partner. After a comprehensive market evaluation, Ubuntu Core stood out as an ideal fit. Screenly takes care of everything through the IoT App Store, ensuring a smooth, hands free experience for its users. In fact, “Automatic Updates’’ now features prominently on Screenly’s homepage as one of its key value propositions. Read more about Screenly’s story.

OTA updates for the future

OTA updates in IoT are extremely important for the operation, maintenance and distribution of devices. The capability for new devices grows as OTA technology continues to improve, unlocking more sophisticated rollback mechanisms and scaling. Harnessing the power of OTA can unlock new features for a range of devices globally.

The global Snap Store utilises OTA technology and can be used by anyone for free. To harness this technology for larger-scale IoT projects, learn more about the dedicated Snap Store.

Explore the following links to learn more:

If you are looking for ways to advance your next IoT project, please get in touch or chat with us in the forum.

Photo by rawkkim on Unsplash

on March 29, 2023 02:16 PM

March 27, 2023

Ubuntu Weekly Newsletter Issue 780

The Fridge

Welcome to the Ubuntu Weekly Newsletter, Issue 780 for the week of March 19 – 25, 2023. The full version of this issue is available here.

In this issue we cover:

The Ubuntu Weekly Newsletter is brought to you by:

Krytarik Raido
Bashing-om
Chris Guiver
Wild Man
And many others

If you have a story idea for the Weekly Newsletter, join the Ubuntu News Team mailing list and submit it. Ideas can also be added to the wiki!

Except where otherwise noted, this issue of the Ubuntu Weekly Newsletter is licensed under a Creative Commons Attribution ShareAlike 3.0 License

on March 27, 2023 10:00 PM

March 26, 2023

Email list to Ghost site - PastaDollar.com

Bryan Quigley

My brother, Nick, just launched a new website/newsletter using Ghost. Ghost is pretty unique in this space, it is:

Open Source
Backed by a Non-Profit
A web publishing platform
A newsletter publishing platform
A CMS for static sites
A member subscription / Patreon replacement

Here is Nick's experience with Ghost so far...

Pricing and Publishers

From a pricing perspective, the platform scales with your audience size. Importantly, Ghost does not take a cut of your membership subscription revenue. For this reason, the pricing incentives mean your Ghost site is more cost-effective when you have 100% paid members and 0% free members. I'm sure that ratio is a pipe dream for most publications, but that's how the business model works. Pricing is currently $9/mo for any site with 500 or fewer subscribers, including web hosting. For context, the new blog that I started in 2023 using Ghost is www.pastadollar.com. The site was originally an email newsletter from my generation of 14 cousins investing together. If you're looking for finance tips and travel hacks, Subscribe here! An email-list-to-ghost-site is a common path for Ghost users. I'm just starting, but major publications like The Atlantic, The Lever, and The Browser use Ghost.

Positives

Ghost has only a handful of themes, but on the whole, they are super clean and approachable. Picking one is as much about your content as it is about which one you list best. All of the themes are responsive and have all the standard customization options like colors and logos. Notably, there is no simple way to use custom fonts. Instead, you can choose between a serif font and a sans-serif font. There is a workaround, but the need for a long list of fonts is a relatively notable omission from a CMS or theme these days. The Ghost editor has a learning curve, but on the whole, it is excellent. I'm used to WordPress's editors, which are total crap by comparison. There are several Ghost editor keyboard shortcuts worth looking up, such as Shift+Enter to jump to the following line but to remain in the same editor block. Hitting Enter will jump to a new block, skipping more lines. The editor auto-saves your work as you go, just as you would expect any web-based tool to do these days. The saving experience is much like google docs - automatic and out of your way. However, sometimes when I haven't changed anything, a modal pops up asking me if I'm certain I want to navigate away without saving—maybe just a bug, but a bit annoying. Many tasks that take multiple steps in a WordPress site are automatic in Ghost. For example, when you add an image to a post or page, Ghost will optimize it. There is no more need to use a plugin or a site like tinypng.com to optimize the image in advance because the platform doesn't do it for you. Ghost was designed with plenty of best practices. For example, the default post URL is the post's title, without any year or dates included. Including dates is possible on other platforms but can quickly cause 404s and redirection issues if/when you update a post. The Ghost developers took the best practice, implemented it as default, and left it to you to simply create the content. It's great.

Negatives

Plugins

Ghost has a great library of plugins, which they accurately call 'integrations.' The catch is that most of these are integrations with other major companies' SaaS products, meaning many of them are paid solutions. The company Ghost seems to rely on the most to expand its functionality is Zapier. Half of the help and support articles I've read on Ghost seem to using Zapier to solve a problem. I don't think it's a stretch to say that Ghost relies heavily on Zapier, a freemium SaaS product. So basically, once you set it up and grow your site, it will eat into your bottom line. That isn't necessarily bad, provided it's worth it. But it seems like the solutions to expand your Ghost site are mostly paid solutions rather than something custom via code injection, which Ghost also supports. To be fair, I need to explore code injection solutions more than I have as of this writing.

Quality of Life Issues

URLs are a big part of any site, so I'm amazed this is an issue: When creating a hyperlink, there isn't proper data validation on the link. If you omit the 'https://www' from a new hyperlink, the Ghost editor saves it, but the URL will not actually work. Saving a link like "pastadollar.com" will show as a broken link and not work when you publish the post. This behavior is incredibly annoying because when you're in your site admin, many URLs around the dashboard omit the https://. To ensure your internal URLs will work, I've solved this by always copying them from the live site. Bullets and Numbered lists aren't an easy option I've found when using the editor. I finally found them hidden in an editor block called markdown editor card. The markdown block supports some rich text features that literally anyone who uses a computer is accustomed to. It isn't exactly user-friendly to hide them in a block, but I'm sure there was a good reason for doing so. Another simple workaround I used when I started was to simply copy and paste a bulleted or numbered list from a proper text editor. I'm guessing many writers create whole drafts outside the Ghost editor anyway.

is is saved though

Alternatives

Alternativeto.net lists more than 250+ alternatives to Ghost, but that long list includes broader solutions like WordPress. Like probably anyone who has developed a website in their lives, I've used WordPress (and continue to use it to this day). WordPress, by default, needs a lot of help from its vast plugin library to set it up, similar to how Ghost works out of the box. I don't consider WordPress a proper alternative if your goal is primarily an email newsletter and membership site. Ghost is absolutely worth the cost over WordPress in that regard - it's not even close.

Beehiiv

In my opinion, the most similar and direct competitor to Ghost is a platform called Beehiiv. Beehiiv has less favorable pricing, and the themes and designs are uglier, but it boasts a stronger feature set out of the box. For example, two big ones are subscriber referral rewards for sharing with friends and more customizable email templates. To be clear, you can probably add these features to a Ghost site; it just takes some work. Both Ghost and Beehiiv are designed to scale quickly with your audience size.

Conclusion

Ghost support has been both highly responsive and extremely helpful. For example, I reached out about getting custom domain emails, such as support@mydomain.com, and their guidance was perfect. Their < 24-hour replies led me to many paid and free services and cautioned me on a few 'gotchas' others ran into. These were not canned responses, and I feel like I can go back to them for anything, and we will be able to find a solution. Overall, I love Ghost. The admin interface is spotless and fast. I'm coming primarily from WordPress, and the speed difference is a game-changer. I would work from many tabs simultaneously in WordPress because switching between different interface pages took too many seconds to load. The Ghost admin pages usually take less than a second to load.

Comments

Add a comment via Gitlab. And yes, Ghost has much nicer integrated commentng built-in.

on March 26, 2023 11:50 PM

March 23, 2023

Ubuntu 20.04.6 LTS Released

The Fridge

The Ubuntu team is pleased to announce the release of Ubuntu 20.04.6 LTS (Long-Term Support) for its Desktop and Server products.

Unlike previous point releases, 20.04.6 is a refresh of the amd64 installer media after recent key revocations, re-enabling their usage on Secure Boot enabled systems.

Many other security updates for additional high-impact bug fixes are also included, with a focus on maintaining stability and compatibility with Ubuntu 20.04 LTS.

Maintenance updates will be provided for 5 years for Ubuntu Desktop, Ubuntu Server, Ubuntu Cloud, and Ubuntu Base.

To get Ubuntu 20.04.6

In order to download Ubuntu 20.04.6, visit:

https://ubuntu.com/download/alternative-downloads

Users of Ubuntu 18.04 LTS will be offered an automatic upgrade to 20.04.6 via Update Manager. For further information about upgrading, see:

https://help.ubuntu.com/community/FocalUpgrades

As always, upgrades to the latest version of Ubuntu are entirely free of charge.

We recommend that all users read the 20.04.6 release notes, which document caveats and workarounds for known issues, as well as more in-depth notes on the release itself. They are available at:

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes

If you have a question, or if you think you may have found a bug but aren’t sure, you can try asking in any of the following places:

#ubuntu on irc.libera.chat
https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
https://ubuntuforums.org
https://askubuntu.com

Help Shape Ubuntu

If you would like to help shape Ubuntu, take a look at the list of ways you can participate at:

https://discourse.ubuntu.com/contribute

About Ubuntu

Ubuntu is a full-featured Linux distribution for desktops, laptops, clouds and servers, with a fast and easy installation and regular releases. A tightly-integrated selection of excellent applications is included, and an incredible variety of add-on software is just a few clicks away.

Professional services including support are available from Canonical and hundreds of other companies around the world. For more information about support, visit:

https://ubuntu.com/support

More Information

You can learn more about Ubuntu and about this release on our website listed below:

https://ubuntu.com/

To sign up for future Ubuntu announcements, please subscribe to Ubuntu’s very low volume announcement list at:

http://lists.ubuntu.com/mailman/listinfo/ubuntu-announce

Originally posted to the ubuntu-announce mailing list on Thu Mar 23 14:21:41 UTC 2023 by Graham Inggs, on behalf of the Ubuntu Release Team

on March 23, 2023 11:57 PM

E239 Pombo Trovejante

Podcast Ubuntu Portugal

No episódio desta semana o Miguel partilhou uma colecção de sucessos, resultado da suas recentes experiências com Docker, o Carrondo voltou às impressões 3D enquanto o Constantino navegava no mar de perguntas e respostas que é o AskUbuntu. O Bionic vai deixar-nos, o Lobster virá melhor que nunca e o Thuderbird está a pensar em meter-se no Botox.

Já sabem: oiçam, subscrevam e partilhem!

Apoios

Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal. E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8. Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem. Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.

Atribuição e licenças

Este episódio foi produzido por Diogo Constantino, Miguel e Tiago Carrondo e editado pelo Senhor Podcast. O website é produzido por Tiago Carrondo e o código aberto está licenciado nos termos da Licença MIT. A música do genérico é: “Won’t see it comin’ (Feat Aequality & N’sorte d’autruche)”, por Alpha Hydrae e está licenciada nos termos da CC0 1.0 Universal License. Este episódio e a imagem utilizada estão licenciados nos termos da licença: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0), cujo texto integral pode ser lido aqui. Estamos abertos a licenciar para permitir outros tipos de utilização, contactem-nos para validação e autorização.

on March 23, 2023 12:00 AM

March 21, 2023

Building the Linux kernel in under 10 seconds with Firebuild

Balint Reczey

Russell published an interesting post about his first experience with Firebuild accelerating refpolicy‘s and the Linux kernel‘s build. It turned out a few small tweaks could accelerate the builds even more, crossing the 10 second barrier with Linux’s build.

Build performance with 18 cores

The Linux kernel’s build time is a widely used benchmark for compilers, making it a prime candidate to test a build accelerator as well. In the first run on Russell’s 18 core test system the observed user+sys CPU time was cut by 44% with an actual increase in wall clock time which was quite unusual. Firebuild performed much better than that in prior tests. To replicate the results I’ve set up a clean Debian Bookworm VM on my machine:

lxc launch images:debian/bookworm –vm -c limits.cpu=18 -c limits.memory=16GB bookworm-vm

Compiling Linux 6.1.10 in this clean Debian VM showed build times closer to what I expected to see, ~72% less wall clock time and ~97% less user+sys CPU time:

$ make defconfig && time make bzImage -j18
real	1m31.157s
user	20m54.256s
sys	2m25.986s

$ make defconfig && time firebuild make bzImage -j18
# first run:
real	2m3.948s
user	21m28.845s
sys	4m16.526s
# second run
real	0m25.783s
user	0m56.618s
sys	0m21.622s

There are multiple differences between Russell’s and my test system including having different CPUs (E5-2696v3 vs. virtualized Ryzen 5900X) and different file systems (BTRFS RAID-1 vs ext4), but I don’t think those could explain the observed mismatch in performance. The difference may be worth further analysis, but let’s get back to squeezing out more performance from Firebuild.

Firebuild was developed on Ubuntu. I was wondering if Firebuild was faster there, but I got only slightly better build times in an identical VM running Ubuntu 22.10 (Kinetic Kudu):

$ make defconfig && time make bzImage -j18
real	1m31.130s
user	20m52.930s
sys	2m12.294s

$ make defconfig && time firebuild make bzImage -j18
# first run:
real	2m3.274s
user	21m18.810s
sys	3m45.351s
# second run
real	0m25.532s
user	0m53.087s
sys	0m18.578s

The KVM virtualization certainly introduces an overhead, thus builds must be faster in LXC containers. Indeed, all builds are faster by a few percents:

$ lxc launch ubuntu:kinetic kinetic-container
...
$ make defconfig && time make bzImage -j18
real	1m27.462s
user	20m25.190s
sys	2m13.014s

$ make defconfig && time firebuild make bzImage -j18
# first run:
real	1m53.253s
user	21m42.730s
sys	3m41.067s
# second run
real	0m24.702s
user	0m49.120s
sys	0m16.840s
# Cache size:    1.85 GB

Apparently this ~72% reduction in wall clock time is what one should expect by simply prefixing the build command with firebuild on a similar configuration, but we should not stop here. Firebuild does not accelerate quicker commands by default to save cache space. This howto suggests letting firebuild accelerate all commands including even "sh” by passing "-o 'processes.skip_cache = []'” to firebuild.

Accelerating all commands in this build’s case increases cache size by only 9%, and increases the wall clock time saving to 91%, not only making the build more than 10X faster, but finishing it in less than 8 seconds, which may be a new world record!:

$ make defconfig && time firebuild -o 'processes.skip_cache = []' make bzImage -j18
# first run:
real	1m54.937s
user	21m35.988s
sys	3m42.335s
# second run
real	0m7.861s
user	0m15.332s
sys	0m7.707s
# Cache size:    2.02 GB

There are even faster CPUs on the market than this 5900X. If you happen to have access to one please leave a comment if you could go below 5 seconds!

Scaling to higher core counts and comparison with ccache

Russell raised the very valid point about Firebuild’s single threaded supervisor being a bottleneck on high core systems and comparison to ccache also came up in comments. Since ccache does not have a central supervisor it could scale better with more cores, but let’s see if ccache could go below 10 seconds with the build times…

Well, no. The best time time for ccache is 18.81s, with -j24. Both firebuild and ccache keep gaining from extra cores up to 8 cores, but beyond that the wall clock time improvements diminish. The more interesting difference is that firebuild‘s user and sys time is basically constant from -j1 to -j24 similarly to ccache‘s user time, but ccache‘s sys time increases linearly or exponentially with the number of used cores. I suspect this is due to the many parallel ccache processes performing file operations to check if cache entries could be reused, while in firebuild’s case the supervisor performs most of that work – not requiring in-kernel synchronization across multiple cores.

It is true, that the single threaded firebuild supervisor is a bottleneck, but the supervisor also implements a central filesystem cache, thus checking if a command’s cache entry can be reused can be implemented with much fewer system calls and much less user space hashing making the architecture more efficient overall than ccache‘s.

The beauty of Firebuild is not being faster than ccache, but being faster than ccache with basically no hard-coded information about how C compilers work. It can accelerate any other compiler or program that generates deterministic output from its input, just by observing what they did in their prior runs. It is like having ccache for every compiler including in-house developed ones, and also for random slow scripts.

on March 21, 2023 08:54 AM

Fixing the qemu serial console terminal size.

Scott Moser

Fixing the qemu serial console terminal size.

Qemu can allow you to attach to a serial console of your guest. This is done with either -nographic or -serial mon:stdio. Its great for getting text output, but after you’ve logged in, you may find issues editing long command lines or using an editor such as vi.

To fix this, you simply need to tell linux what the terminal size you’re using is.

First, figure out what your terminal size is. You have to do this before connecting to qemu, or some other way such as another tmux window or terminal.

With bash via $LINES and $COLUMNS:

 $ echo rows $LINES columns $COLUMNS
 rows 75 columns 120

With stty, look for ‘rows’ and ‘columns’ below:

 $ stty -a
 speed 38400 baud; rows 75; columns 120; line = 0;
 intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
 eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
 werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
 -parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
 -ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
 -iuclc -ixany -imaxbel iutf8
 opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
 isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
 echoctl echoke -flusho -extproc

 $ echo $TERM
 xterm

Then tell linux inside qemu about it via stty:

$ stty rows 75 columns 120

# now show what it is
$ stty -a |head -n 1
speed 115200 baud; rows 75; columns 120; line = 0;

Last, you probably want to update your TERM variable :

$ export TERM=xterm

Thats it.

on March 21, 2023 12:00 AM

March 18, 2023

On ignoring mistakes, resilience, and the hidden dangers therein

Andrea Corbellini

As a scuba diver who often explores new places, I can say that I have found myself in some dangerous situations, but I always made it back to the surface without facing any negative consequences. Does this mean that I never made any mistakes? Absolutely not: mistakes were made, and lessons were learned.

We can all agree that learning from mistakes is good, but sometimes, when mistakes happen and consequences don’t manifest themselves immediately, we run the risk of not noticing them, not learning from them, repeating them, and over time developing a false sense of confidence, which can drive us to believe that our repeated mistakes are actually good practices.

Why do we ignore mistakes? Because sometimes outcomes are positive even if we make mistakes. “I made it out of water even this time, this means that my dive was executed perfectly.” This is a common way of reasoning, but in reality, things are much more complex than that. There is a difference between correct execution and successful outcome, and the two should not be confused. In fact, everyone should know from experience that goals can be achieved even if the execution was sloppy and full of mistakes. Catastrophic consequences may happen if we fail to see that.

An example of the consequences of ignoring mistakes is given by the two space shuttle disasters: the Challenger disaster of 1986, and the Columbia disaster of 2003. Both these instances were caused by NASA leadership ignoring the concerns from the engineering teams. Problems that occurred in previous shuttle launches should have been a wake-up call for NASA leadership. Instead, all the previous successful launches and re-entries despite the problems were seen as accomplishments, and nourished the leadership’s overconfidence. “We have made it this time too, this means that all those concerns that engineers raised were excessive.”

The tendency of diverting from proper procedures, dismissing valid concerns, and ignoring problems, has a name: it’s called normalization of deviance. The driving force of normalization of deviance is overconfidence and the false belief that positive outcomes are inherently caused by correct executions.

Overconfidence and normalization of deviance can spread like a virus in an organization. It is important to be vigilant for signs of overconfidence in individuals, before it infects other people. I once had to deal with a manager who was a self-declared micromanager (and proud to be) but lacked technical foundations and knowledge of the product. He would consistently and quickly dismiss anything that he did not understand, and focus on short-term goals of questionable usefulness. Whenever his team would accomplish a goal, he would send a pumped-up announcement, often containing inaccuracies, and carefully skipping over the shortcomings of the solutions implemented. Given the apparent success of this management style, other managers started to follow his example. Soon after (in less than a year), the entire organization became a toxic environment where raising even the minimal concern was seen as an attack on the “great new vision”.

I see many parallels between this manager story and what is happening with ‘Twitter 2.0’ right now (although, I must say, in my case engineers did not get fired on the spot for speaking the truth). And with that manager, just like with ‘Twitter 2.0’, whenever problems occurred, those problems would either be ignored or blamed on the preexisting components built before the manager joined, never on the new, careless developments.

The truth however was that problems that occurred had been preannounced weeks, or months before, but concerns around them had been promptly dismissed due to being too challenging to address, and because “everything works right now, so that’s not a concern”.

The idea that everything must be correct because everything works, goals are achieved, and outcomes are successful, is a dangerous idea that can potentially have catastrophic consequences. It’s important to be critical and analytical, regardless of the outcome. This does not mean that success shouldn’t be celebrated, but that mistakes should be captured so that lessons can be learned from them, even if the final outcome was successful. Not learning from mistakes does not allow us to advance, and on the contrary can only lead us to repeat them. And if we keep repeating the same mistakes, sooner or later, those will have some negative consequences.

A common practice in the aviation industry is to write reports on incidents, close calls, and near misses, whenever they occur, even if the flight was concluded successfully and no injuries or damages occurred. These reports are collected in databases like the Aviation Safety Reporting System (which can be freely consulted online), so that flight safety experts and regulators can identify common failure scenarios and eventually introduce mechanisms to improve safety in the aviation industry. A key element of these reports is that they are not meant to put the blame on certain people, but rather focus on what chain of events led to a certain mistake. “Human mistake” is generally not a valid root cause: if a human was able to make a mistake, it means that a mechanism is missing that can either prevent the mistake or detect it before it causes any negative consequences.

Some companies in other industries have similar processes for writing reports or retrospectives when a mistake happens (regardless of the outcome), with the goal of finding proper root causes and preventing future mistakes. Amazon with its Correction of Error practice is a famous example.

I think introducing these practices in an organization can help to establish a healthy culture where finding mistakes and raising concerns is encouraged, rather than being oppressed. However these practices, by themselves, may not be enough to ensure that such a culture can be maintained over time, because people can always disagree on what is considered a ‘mistake’. Empathy is probably the key to a truly healthy culture that allows people to learn and advance.

There are also cases where we are aware of problems, and we see them as such, but we deliberately choose not to do anything about them. This is where resilience comes into play.

Resilience is generally a good quality to have. Resilience can give us the strength to go through long-term hardships, and can have positive effects on our tenancy and determination. But even resilience, when taken to the extreme, can be dangerous. Resilience can lead us to ignore problems, and not react to them. Resilience can make us tolerate a negative situation, without finding a proper strategy to cope with it.

Poor planning forces you to consistently work extra hours? Resist and keep going, until you burn out. The relationship with your partner doesn’t satisfy you? Resist and think that things will get better, while the relationship slowly deteriorates. Feel pain in your knee every time you run for more than 30 minutes? Resist and don’t go to see a doctor, the pain will go away, sooner or later… until you cannot run anymore.

When we let resilience become an excuse to avoid solving problems, we can end up in situations from which it’s difficult to recover.

It’s important to make a distinction between what is under our control and what is not. We can fix problems that are under our control, but in situations where we cannot directly change the course of things, finding an alternative strategy is the only way. Resisting and hoping that things will get better often does not give the expected outcome–on the contrary, it can be detrimental.

In the end, I think that the ‘practice’ of ignoring mistakes (because of the overconfidence built from successful outcomes) or ignoring problems (because of resilience taken to the extreme) are hidden time bombs, silently ticking, waiting for the right conditions before exploding. We need to be aware that just because things seem to work today, it doesn’t mean that we’re making the right decisions, and this can have consequences in the future. Being critical, analytical, empathetic, and honest is important to avoid these behaviors and the dangers that come with them.

on March 18, 2023 08:20 AM

March 17, 2023

An In-depth Review of the Kubuntu Focus XE Gen 1

Aaron Rainbolt

About four months ago, a friend of mine got me in touch with his employer at the time. I was in a bit of a bind, using rather old and slow hardware (a 3rd gen mobile i5) for my development work, and was hoping to get a newer, faster system to help make things less painful. My friend and his boss managed to get me hooked up with a bug bounty - in exchange for fixing one of several high-priority bugs in the KDE Desktop Environment, I would receive a new Kubuntu Focus system, either a KFocus XE laptop (which is what I ultimately chose), or a KFocus NX desktop. I jumped at the chance, and managed to play a significant role in resolving two of the bugs. I also got a much better computer.

That was about four months ago. And if you’ve ever gotten a glimpse of my workflow, you know just how much damage… er… testing I can do in four months. I do not treat my systems all that kindly. After having gone through the trials of over seven terabytes of SSD writes, thousands of software compiles, probably over a hundred VM installs, and three distrohops, I finally feel that I can give this system a proper review. For those of you who have been looking into the Kubuntu Focus systems, this should help you get an idea of what you can expect from them, and for those who are on the hunt for a good Linux laptop, this should hopefully help you choose a good one.

First impressions

The Kubuntu Focus XE is essentially a nicely modded Clevo laptop with Kubuntu 22.04 LTS and a suite of utilities and extra features. For those of you who don’t already know this, Kubuntu is an official flavor of the Ubuntu Linux operating system, using the KDE desktop rather than the GNOME desktop. I personally have always been a huge fan of KDE (in fact it was the desktop that came with the first “real” distro I ever used), which made the idea of contributing to KDE to get a laptop even more appealing. The Kubuntu installation that comes with the Kubuntu Focus has a ton of value-adding customizations, including hardware optimization, curated apps and libraries, a very efficient setup wizard, built-in file versioning and recovery, a special Quick Help Widget, and several custom-made utilities for changing special hardware and software settings that are difficult to change otherwise.

The actual laptop is quite thin and light compared to many older systems, and features quite a few I/O ports of various types, including a Gigabit Ethernet port (which is somewhat rare among modern laptops). The keyboard features backlighting, which is always a nice touch if you do a lot of typing, and the cooling system uses dual fans, which looks cool and does a good job of keeping the system cool while under heavy load. Other systems I’ve used would get quite toasty if you really hammered them, while this one gets warm in just one spot right where the CPU itself is located.

For specs, the system I received came with an i5-1135G7 processor, 32 GB of 3200 MHz RAM, and a 1 TB Samsung 970 EVO PLUS solid state drive. 32 GB of RAM is a godsend when you’re doing really intensive work (like running four or more VMs at the same time, something I’ve been known to do in the past). The SSD’s size and speed has also been quite welcome, especially since a lot of what I do involves copious amounts of disk writes.

All in all, things looked pretty impressive initially. The final verdict, however, would depend on how it performed in real-world use.

Kompilation

Having done some KDE development work, it seemed only fitting to use the new machine for KDE development. But not normal KDE development. I wasn’t going to just hack on some source code and do a few builds to test my changes.

One of the projects I maintain is KForge, which is a KDE development virtual appliance that has all of the needed build and runtime dependencies for almost all KDE software preinstalled. This is quite useful as it has been historically difficult to set up a fully functional KDE development environment due to missing build dependencies. You usually don’t know you’re missing a dependency until a build fails, and sometimes the logs left by the build failures are significantly less than helpful.

However, getting KForge to work just right was tricky for the same reason it’s tricky to get a decent KDE build environment going. So, if I was going to make sure I was pulling in all of the dependencies for all of the KDE software, I was going to have to compile almost everything in KDE.

By “almost everything”, I mean 300+ packages.

I ended up deciding to exclude some KDE applications from the build (namely Krita and Kdenlive, both of which I felt would just add too much bulk to the build and wouldn’t add all that much value), and there was one component I ended up excluding since it was defunct and wasn’t maintained anymore. But the remaining 300+ pieces of software needed built. The idea was that, if I got all the dependencies right, and if all of the KDE development branches were in buildable states, I would be able to successfully build everything, and that would tell me that I had succeeded at finding all of the dependencies.

Needless to say, compiling over 300 applications is virtually always going to be a large task even if the apps are somewhat small. Not only were some of these apps not small, many of them were juggernauts (especially KWin, the window manager used by KDE).

While the system’s fans do ramp up significantly and can be quite loud during compilation, the system itself actually remains responsive during such a job. Responsive enough for me to just keep using it while the compilation happens in the background. This was somewhat amazing, since my old system was known for randomly freezing when it got hit with a lot of disk writes, and so to be able to hammer the system like this in the background and still have it be usable during that time was massively helpful. Not to mention the dramatically reduced compile times compared to my older system.

Multitasking

Part of me isn’t sure why I do this, but I have a tendency to try and work on two or three projects at once, sometimes more. As a result, I generally have a slew of apps open all over the place that I switch between rapidly. As I write this part of the review, I’m review-writing, chatting on both IRC and Matrix, and trying to reproduce a bug in Lubuntu, all at the same time. As a result, a well-streamlined DE is very helpful.

KDE is, for me, very good at making work easy. But the Kubuntu Focus takes things a step further and makes it easy to access virtual desktops, something that Kubuntu usually doesn’t have enabled by default. Virtual desktops let me break up my work into different “screens”. The KFocus has four virtual desktops enabled out of the box, essentially giving me four screens at the same time, all on one screen. Not only that, but I can switch between virtual desktops with Ctrl+F1, Ctrl+F2, etc, allowing me to switch desktops without even having to click on the virtual desktop I want.

Additionally, the lots of RAM my system came with made multitasking comfortable from a performance perspective - lag was very rare and the times it happened, it was very small, and only happened during things like a massive compile job. (And even that rarely caused any lag.)

On the hardware side of things, multitasking is also helped by a good keyboard and touchpad - hard-to-press keys and a jumpy or insensitive touchpad can seriously hamper productivity. The keyboard on the KFocus XE is really nice and allows me to type at 100 WPM without problems (and that’s with a cat on my lap and distractions nearby - I could very likely go faster in an ideal situation). The mousepad was initially tricky since it’s so big and I would keep accidentally brushing it with my hand - this ended up becoming a non-issue with time as I got used to holding my hands away from it (which is probably better for avoiding carpal tunnel too). Oh, and the mousepad is also quite precise, responsive, and works well even when I’m doing things at high speeds. The only other problem I’ve noticed is that the touchpad will occasionally stop responding for no apparent reason. This happens infrequently, but when it happens, I’m able to simply switch to a TTY by pressing Ctrl+Alt+F3, then return to my desktop with Ctrl+Alt+F1 - this makes the touchpad work again.

Graphics

I am not a gamer, and the KFocus XE is not designed for heavy gaming. I have no clue how well this thing performs if you’re intending to run intensive 3D games on it. That being said, graphics are important even in the general productivity world to some degree.

The Kubuntu Focus XE boasts a fairly good Intel iRIS XE iGPU, and while Intel’s integrated graphics are not known for being the pixel-crunching behemoths that NVIDIA and AMD churn out, they’re generally great for light and moderate uses. That was exactly my use case.

Everything I threw at the machine graphics-wise worked swimmingly. Full HD video playback is free of stutters and screen tearing. Windows have no stuttering when dragging them (even with the window transparency and wobbly windows that the OS has enabled by default, which look really cool and smooth). Desktop animations are nice and smooth everywhere.

For 3D performance, I tried using thirdroom.io (which is basically an open-source Metaverse-like thing) in Chrome. My fans got a bit louder, but other than that it seemed to perform OK, though it was somewhat laggy. All of the graphics rendered relatively smoothly. The graphics performance for the Intel iGPU is listed on the KFocus website as being similar to the NVIDIA GeForce MX250 - benchmarks show the Intel iGPU as being as good or better than the MX250 in many tests and only slightly lacking in others.

The other important part of the graphics experience is the screen. Here I have one gripe… and before I describe it, I should say, I’m pretty sure this is my fault - my friend has not had this problem, and I’ve not seen any other reports of it. The one problem is screen burn-in. If stuff around the edges of the screen stays in the same spot too long, and then I turn the screen off and leave it that way for a while, I can see shadows of the old screen contents later, sometimes quite well. The reason I think this is my fault is because it didn’t happen at first, and then I left the laptop upside-down trying to improve its cooling airflow during a large compile job. I left it in this state overnight… and when I opened the laptop in the morning, I could see burn-in. Facepalm. I confirmed with my friend that KFocus XE is not intended to be left upside-down like that, and that I may have damaged the screen in so doing. Which would explain why the burn-in happens most near the edges of the screen and much less in the middle. Thankfully, the burn-in mostly goes away on its own if I leave the screen on long enough, and I can use the LCD Scrub screensaver from XScreenSaver to help it out (which may not even be necessary). Other than that, the screen seems to have survived my accidental mistreatment without problems.

Apart from my most likely self-inflicted burn-in issue, the screen is amazing. Colors are bright and crisp, and details are easily visible even when they’re small thanks to the Full HD resolution. The closest thing I’ve had to a monitor this good was a ViewSonic Full HD monitor, and both of the ones I have are dead. Every other Full HD monitor I have either has horrible color or something wrong with the backlight - the KFocus XE screen has neither problem. Coming from a 1366x768 screen, this high-res screen has been a gigantic relief and makes multitasking a lot easier. I can’t think of any way the screen could be better - if it was any higher resolution, I’d need display scaling due to its size, and that would essentially negate the best effect of having a higher res monitor (more screen real estate).

As a whole, the graphics and screen of the KFocus XE have surpassed my expectations and are perfect for everything I do.

Mobility

A good laptop generally has to be relatively lightweight, have good battery life, and have good WiFi range in order to be useful as a mobile system. In my experience, the Kubuntu Focus XE has all three.

For weight, the XE is, as discussed earlier, quite thin and light. Unlike my older system, which was a leg-squisher and could end up causing actual pain if you left it on your lap too long, the XE can remain on my lap for extended periods of time without problems. It’s not as light as my tablet (an HP Chromebook x2), but it’s fairly close, which is surprising considering that my tablet is a fanless machine with a rather pathetic processor. Whereas the KFocus XE has dual fans and a powerful processor.

Battery life is also quite good. I generally prefer battery lifetime over “how long can I use the battery before it drains dry”, so I don’t have my XE disconnected from its charger for super long, and I use the FlexiCharger feature, which I have set to make the laptop not charge at all until it get to below 40% and stop charging once it hits about 80%. Most of the time, I only use 25% of the battery life at a time, if that. That lasts me for what I would guess is an hour of intensive use, and much longer when in a power-saving mode. According to the Kubuntu Focus team, the battery should last from four hours under heavy use, to as much as ten hours under light use. That fits with my experience, and that also means that the battery in this should last longer than my ChromeOS tablet (which only gives about six to seven hours).

As for WiFi range, I currently live in a relatively small house where each room is about two rooms away from the room furthest away from it. One would think a house of this size would make it tricky to thoroughly review the WiFi range, but the smallish house is offset by my absolutely pathetic router (alright so it’s a cellular mobile hotspot about the size of half a deck of cards). I have the hotspot on one side of my house, yet I still get connectivity and decent speeds on the other side of the house most of the time. Considering the very poor quality of the WiFi signal provided by the hotspot, this is really good, and it works perfectly for what I do.

Networking and Connectivity

The WiFi chip in the KFocus XE is the Intel AX201, which is a WiFi 6 device that is theoretically capable of 2.4 Gbps speeds. My hotspot doesn’t provide anything even close to that, but while I may not get the full speed of the chip, I definitely get the full speed of my hotspot, as I see no difference in speeds whether I use a WiFi connection or a direct USB connection to the hotspot.

The Ethernet included in the KFocus XE is a fairly standard Gigabit Ethernet port with a Realtek controller. It works flawlessly for me, and gives me full gigabit speeds when transferring data from one computer to another via an Ethernet cable. I just set one of the computers to share an IPv4 connection, then put one end of the cable in the laptop, the other end of the cable in the other computer, and then use netcat to beam whatever I need transferred. I can even share an Internet connection to another computer doing this. Ethernet is increasingly rare in modern laptops, yet still very handy in many instances, so this is a very nice feature to have.

As for the other ports on the system, there’s a pair of USB3 ports, a pair of USB-C ports (one of which supports Thunderbolt 4), a headphone jack, an SD card slot, and an HDMI port. This is more than sufficient for my uses, and for those who need more ports, the Thunderbolt 4 port should let you attach a docking station to extend the system.

Another connectivity feature I use is Bluetooth. In my experience, Bluetooth has been quite finicky on other laptops I’ve used, so I didn’t have very high expectations here. However, surprisingly, things mostly worked as I would have expected them to. My Bluetooth earbuds connected without problems, and while they are a little fiddly when getting them reconnected, they reconnect pretty easily and work well once connected. (I disconnect and reconnect them manually in the Bluetooth menu, then they work right.)

Aesthetics

The KFocus XE not only performs beautifully, it also looks beautiful IMO. There are plenty of well-made design choices both within the OS and in the hardware itself. If you’ve ever used Kubuntu before, you’re already familiar with KDE’s aesthetics. The customized Kubuntu installation that comes with the KFocus XE keeps the same basic look-and-feel, but also throws in a few more nice-looking (and even functional) design elements.

The most noticeable of these additions is the Quick Help Widget on the desktop (visible in the screenshot near the start of the review). The widget looks sharp and complements the rest of the design rather well. It has several pages inside loaded with numerous keyboard shortcuts, terminal commands, and even a Vim cheat sheet.

There’s also a couple of window effects enabled by default that I mentioned earlier - transparency and wobbly windows. In essence, when you grab a window in order to move it across the screen, the window becomes transparent, allowing you to see underneath it while moving it. The window will also “flow” as you move it, making the movement look smoother. The wobbly windows effect also causes windows to visibly “snap” into place if you use window snapping, which is kind of the visual equivalent of tactile feedback - you can “feel” the window snap into place properly.

On the hardware side of things, there are several nice touches that add to the system’s aesthetic value, most noticeably the hexagonal hinges on the screen, the Kubuntu logo on the Meta key (where you usually see a Windows logo on most laptops), and the mitered angles on the edges of the recessed keyboard. The backlighting on the keyboard is also a nice touch (both functionally and aesthetically).

Setup

I saved this part for near the end since, in all likelihood, you’re only going to go through the setup procedure once per reinstall, and you’re going to reinstall very rarely, if ever. Most of what you do is going to be done once everything has been set up. But the setup procedure on the Kubuntu Focus is a major feature of the OS it ships with, so I felt it was worth reviewing.

This part was somewhat shocking for me. Most of the time, once I install a new OS on my system, it takes me a while to get fully set up. A lot of the apps I use aren’t installed by default, getting logged into everything takes time, etc. The worst part is that I have a very bad tendency to forget things, so I won’t end up getting some part of my workflow set up at all until I hit something where I need it to be set up. The Kubuntu Focus has an entire initial setup wizard that makes a massive amount of the initial setup work way easier. And there are a ton of preinstalled apps, many of which are ones I actually use. The wizard has several neat features built into it, including helping me remember to log into my email. (That part was particularly helpful since I’m pretty sure I’ve had at least one time when I only remembered to log into Thunderbird because I went to check my email.)

After a few minutes of downloads and logins, most of everything I needed was set up. I did end up having to install a few apps myself, as well as my Ubuntu development utilities. And I think I found one small bug in the wizard when I rejected VirtualBox’s Personal Use and Evaluation License for their expansion pack because I didn’t want to use Oracle’s proprietary VBox addons and comply with the restrictions that came with them. Doing so still let VirtualBox install (which was correct), but also left it marked as autoremovable (not correct). No big deal, I ended up installing a newer version of VirtualBox from the main VBox website anyway. It’s a setup wizard, it’s not going to magically read your mind and set up everything perfectly the first time. But it got pretty close, and that was close enough to save me a ton of setup time.

One unique part of the setup process is the “Curated Apps” feature. Somehow (and I don’t fully understand how), the Kubuntu Focus people have a website where you can click on the icon of an app you want to install (from a list of apps they recommend), and it will automatically do the installation for you. You have to type your password (thankfully!), but aside from that, it’s a one-touch installation procedure. Through a website. (I’m guessing there’s a client-side thingy built into the OS that makes this work.) It can even launch an app if its already installed or if you’ve just installed it. I installed several apps through the Curated Apps page, including VSCode, which is usually a bit tricky to set up. The installation process for each app was simpler even than using the Google Play store.

The whole setup procedure was drastically different from the usual “Bailing out, you’re on your own, good luck” post-install experience most other operating systems leave you with. It was an extremely good intro to a fantastic system, and it helped me a lot.

Distrohopping

The built-in Kubuntu operating system works quite well and should be all most people need. But some of us either need to use a different distro, or want to explore other distros on our physical hardware. So far I haven’t tried distros other than official Ubuntu flavors, since I really like the Ubuntu ecosystem. But within the Ubuntu ecosystem at least, the KFocus XE works like a champ.

I tried two other Ubuntu flavors (Lubuntu 23.04 Alpha and Ubuntu Unity 22.10), and both of them worked very well out of the box. I assume that other distros like Fedora, Arch, or openSUSE would very likely work well, since the hardware in the KFocus is quite Linux-friendly.

There are a couple of downsides of using a different Ubuntu flavor on the KFocus XE. For one, you lose a bunch of the tools that the Kubuntu Focus Suite has built in, like the fan profile settings. You may also not end up being able to set a power profile like KDE allows you to do. You can install the Kubuntu Focus Tools on other distros, however this will probably pull in a lot of KDE dependencies since the Kubuntu Focus Tools, are, unsurprisingly, Kubuntu-focused. Personally, I just lived without them when I was trying out different distros.

The other downside is the loss of system component curation. One hidden feature that you usually won’t see on the KFocus is the fact that the Kubuntu Focus team tests updates to several important components, such as the curated apps and the kernel, before releasing those updates to end-user systems. So if, for instance, a kernel update breaks some hardware, you won’t get that update, and your hardware will continue to work right. When you’re using other distros, the Kubuntu Focus team no longer can control those updates, and so if an update breaks something, you’ll probably end up seeing that breakage.

Aside from those two problems, the user experience when using different Ubuntu flavors was flawless for me. All hardware worked out of the box without issues (except for Bluetooth was much finickier on Ubuntu Unity than it was on Kubuntu, which may have been a consequence of lacking the curated system components). The system performed perfectly well, and I was able to use it just like I’d use any other computer I installed Linux on.

Pretty much anything with a Linux 5.17 or newer kernel should work, if I’m understanding correctly. (The 5.15 kernel is probably not ideal.) Ubuntu 22.04.2 or higher should work out of the box. There aren’t any required third-party drivers, so as long as your distro of choice ships with a reasonable set of in-kernel drivers, everything should just work out of the box, even if you’re using a non-Ubuntu distro.

Final thoughts

I mean, this is the best laptop I’ve ever used in my life. It’s fast, it’s powerful, everything works smoothly, it looks nice, and it makes it easy to get work done. That’s what a good laptop should do. If you’re into Linux and want a laptop that just works, I’d highly recommend the Kubuntu Focus XE. It’s amazing.

Thanks for taking a look, and I hope this has helped you in some way. Thanks to Michael and the Kubuntu Focus team for making these awesome systems, and for giving me the opportunity to work for one!

on March 17, 2023 03:06 AM

March 16, 2023

A Big Thank You Community! KDE Snaps resumed and More.

Scarlett Gately Moore

After my last post a few things have happened. First, I want to thank all of you for your support, monetary and moral. The open source community is amazing and I love being a part of it. We are surviving thanks to all of you. Despite my failed interview, a new door has opened up and I am happy to announce that Canonical is funding me to work part time for a period of 3 months on KDE Snaps! While not the full time role I was hoping for, I’ll take it, and who knows, maybe they will see just how awesome I am!

I started back to work on these last Tuesday. So far I have made good progress and even have a working Core22 snap!

Work done on upstream snapcraft ( tool used to make snaps ):

New content pack version, fixed an issue with snapcraft remote-build choking on part with /sdk, fixed regex as our naming scheme has changed:

https://github.com/snapcore/snapcraft/pull/4069

Ran into a new bug with snapcraft expand-extensions and so I must now enter all of this information manually into snapcraft.yaml until fixed, bug report here:

https://bugs.launchpad.net/snapcraft/+bug/2011719

And without further ado, our first core22 snap is Dragonplayer Version 22.12.3 available in the snap store. Many more coming soon!

With a new month upon us, I must ask for help again, I will not see any money for this work until next month. Please consider a dontation. Thank you from the bottom of our hearts.

https://gofund.me/c9cc02ed

on March 16, 2023 10:07 PM

E238 Chapéus Há Muitos

Podcast Ubuntu Portugal

Neste episódio em que temos os três anfitriões de volta: o Miguel compra artigos de roupa, o Tiago está infiltrado a preparar a revolução, mas teve que lidar com problemas de WiFi, o Diogo descobriu mais Software Livre de game dev e game design, e tivemos ainda mais um cantinho do Firefox com extensões fixes. Já sabem: oiçam, subscrevam e partilhem!

Apoios

Atribuição e licenças

on March 16, 2023 12:00 AM

March 10, 2023

No heroes needed

Riccardo Padovani

Being a hero is nice, isn’t it? You work hard, single-handedly save the day, and your teammates are eternally grateful to you. However, such behavior is, in fact, highly problematic.

In Google, we have an internal document, quite often quoted, that says to let things fail if the workload to keep them alive is too high. From Google, many good practices have been exported and adopted in the IT world, and I believe this could be very useful to many teams.

The story of how “heroism is bad” is not new. The internal Google document has actually been a presentation at SRE TechCon 2015. And, other companies have a similar take. But let’s start from the principle.

What is the significance of heroism?

When we talk about a hero for the team, we mean a single person taking on a systemic problem to compensate for it. No matter the cost, how many hours are needed, or the consequences. Of course, thanks to the hero, the crisis due to the systemic issue is momentarily avoided.

And, heroes get immediate reward! They have completed a gigantic task, they get praised from teammates, and they feel crucial: after all, without them, everything would have failed.

Why being a hero is bad

Of course, we are talking about behavior in the working environment, not of heroic acts to actually save lives. Heroic behavior, although quite encouraged in many realities, at the end of the day, brings more damage than benefits. It is bad for the individual: takes a toll on them, brings to burnout, creates unsatisfiable expectations (you did once, why don’t you do it again?), and puts a lot of pressure on them.

It is also bad for the team: it creates incorrect expectations from the team members, and it leads to inaction because the heroes will pick up anything left behind. Not only that, but it damages the morals because it makes it feel like everything has been solved by the hero, and everybody else has no purpose.

And it is bad for systems and processes: they don’t improve because teams don’t realize they are broken, given that at the end of the day, the heroes make everything work. No systemic fixes are carried on, ‘cause such problems are hidden due to the heroes efforts.

We work in teams for a reason: the idea that one person has the solution to everything is ridiculous.

What to do instead?

On YouTube, there is an astonishing TED talk by Lorna Davis, “A guide to collaborative leadership”. If you have 14 minutes of spare time today, you should really use them to watch it.

There is a key concept in the video: radical interdependence.

What’s radical interdependence?

Radical interdependence is just a fancy way to say that we need each other. In a complex, ever evolving, interconnected world, a single individual cannot change the world on its own.

And a team can delivery things that could be impossible to deliver by an individual alone.

However, quoting from the video:

Interdependence is harder than being a hero. It requires us to be open, transparent, and vulnerable. And that’s not what traditional leaders have been trained to do.

Let things fail

It’s hard, but nowadays, every time I feel the urge to step in to save the day, I stop a minute, and ask myself: “Can’t I sit down with the team and work this out?”. And every so often, this requires things to fail: but that’s okay.

It is imperative that a good process for postmortems is in place, and that such process is blameless.

When things fail, we sit down all together, and we find out what went wrong, and what we can do better the next time. This allows improvements to the system, to the process, and at the end things are remarkably better than what they would have been if they hadn’t failed.

It’s a complex, difficult approach, but it provides significant improvements. Have you ever tried it, or plan to do so? Can you think of an occasion where it would have been useful? Let me know in the comments!

If you have any question, other than here in the comments you can always reach me by email at hello@rpadovani.com.

Ciao,
R.

on March 10, 2023 12:00 AM

March 09, 2023

Authenticated encryption: why you need it and how it works

Andrea Corbellini

In this article I want to explore a common problem of modern cryptographic ciphers: malleability. I will explain that problem with some hands-on examples, and then look in detail at how that problem is solved through the use of authenticated encryption. I will describe in particular two algorithms that provide authenticated encryption: ChaCha20-Poly1305 and AES-GCM, and briefly mention some of their variants.

The problem

If we want to encrypt some data, a very common approach is to use a symmetric cipher. When we use a symmetric cipher, we hold a secret key, which is generally a sequence of bits chosen at random of some fixed length (nowadays ranging from 128 to 256 bits). The symmetric cipher takes two inputs: the secret key, and the message that we want to encrypt, and produces a single output: a ciphertext. Decryption is the inverse process: it takes the secret key and the ciphertext as the input and yields back the original message as an output. With symmetric ciphers, we use the same secret key both to encrypt and decrypt messages, and this is why they are called symmetric (this is in contrast with public key cryptography, or asymmetric cryptography, where encryption and decryption are performed using two different keys: a public key and a private key).

Generally speaking, symmetric ciphers can be divided into two big families:

stream ciphers, which can encrypt data bit-by-bit;
block ciphers, which can encrypt data block-by-block, where a block has a fixed size (usually 128 bits).

As we will discover soon, both these two families exhibit the same fundamental problem, although they slightly differ in the way this problem manifests itself. To understand this problem, let’s take a close look at how these two families of algorithms work and how we can manipulate the ciphertexts they produce.

Stream ciphers

A good way to think of a stream cipher is as a deterministic random number generator that yields a sequence of random bits. The secret key can be thought of as the seed for the random number generator. Every time we initialize the random number generator with the same secret key, we will get exactly the same sequence of random bits out of it.

The bits coming out of the random number generator can then be XOR-ed together with the data that we want to encrypt: ciphertext = random sequence XOR message, like in the following example:

random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
                                     ⊕
        message: I would really like an ice cream right now
                                     =
     ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1

The XOR operator acts as a toggle that can either flip bits or keep them unchanged. Let me explain with an example:

a XOR 0 = a
a XOR 1 = NOT a

If we XOR “something” with a 0 bit, we get “something” out; if we XOR “something” with a 1 bit, we get the opposite of “something”. And if we use the same toggle twice, we return to the initial state:

a XOR b XOR b = a

This works for any a and any b and it’s due to the fact that b XOR b is always equal to 0. In more technical terms, each input is its own self-inverse under the XOR operator.

The self-inverse property gives us a way to decrypt the message that we encrypted above: all we have to do is to replay the random sequence and XOR it together with the ciphertext:

random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
                                     ⊕
     ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
                                     =
        message: I would really like an ice cream right now

This works because ciphertext = random sequence XOR message, therefore random sequence XOR ciphertext = random sequence XOR random sequence XOR message. The two random sequence are the same, so they cancel each other (self-inverse), leaving only message:

random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
                                     ⊕
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F
                                     ⊕
        message: I would really like an ice cream right now
                                     =
        message: I would really like an ice cream right now

Only the owner of the secret key will be able to generate the random sequence, therefore only the owner of the secret key should, in theory, be able to recover the message using this method.

Playing with stream ciphers

The self-inverse property not only allows us to recover the message from the random sequence and the ciphertext, but it also allows us to recover the random sequence if can correctly guess the message:

        message: I would really like an ice cream right now
                                     ⊕
     ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
                                     =
random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F

This “feature” opens the door to at least two serious problems. If we are able to correctly guess the message or a portion of it, then we can:

decrypt other ciphertexts produced by the same secret key (or at least portions of them, depending on what portions of the random sequence we were able to recover);
modify ciphertexts.

And we can do all of this without any knowledge of the secret key.

The first problem implies that key reuse is forbidden with stream ciphers. Every time we want to encrypt something with a stream cipher, we need a new key. This problem is easily solved by the use of a nonce (also known as initialization vector, IV, or starting variable, SV): a random value that is generated before every encryption, and that is combined in some way with the secret key to produce a new value to initialize the random number generator. If the nonce is unique per encryption, then we can be sufficiently confident that the random sequence generated will also be unique. The nonce value does not necessarily need to be kept secret, and needs to be known at decryption time. Nonces are usually generated at random at encryption time and stored alongside the ciphertext.

The second problem is a bit more subtle: if we have a ciphertext and we can correctly guess the original message that produced it, we can modify it using the XOR operator to “cancel” the original message and “insert” a new message, like in this example:

         ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
                                         ⊕
            message: I would really like an ice cream right now
                                         ⊕
    altered message: I would really like to go to bed right now
                                         =
tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1

This message, when correctly decrypted with the secret key, will return the tampered ciphertext without detection!

Note that I do not need to know the full message to carry out this technique, in fact, the following example (where unknown parts have been replaced by hyphens) produces the same result as the above one:

         ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1
                                         ⊕
            message: --------------------an ice cream----------
                                         ⊕
    altered message: --------------------to go to bed----------
                                         =
tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1

This problem is known as malleability, and it’s a serious issue in the real world because most of the messages that we exchange are in practice relatively easy to guess.

Suppose for example that I have control over a WiFi network, and I can inspect and alter the internet traffic that passes through it. Suppose that I know that a person connected to my WiFi network is visiting an e-commerce website and that they’re interested in a particular item. The traffic that your browser exchanges with the e-commerce website may be encrypted, and therefore I won’t be able to decrypt its contents, but I might be able to guess certain parts of it, like the HTTP headers sent by the website, or some parts of the HTML that are common to all pages on that website, or even the name and the price of the item you want to buy. If I can guess that information (which is public information, not a secret, and it’s generally easy to guess), then I might be able to alter some parts of the web page, showing you false information, and altering the price that you see in an attempt to trick you into buying that item.

An example of malleability using ChaCha20 with OpenSSL

Here’s a practical example of how we can take the output of a stream cipher, and alter it as we wish without knowledge of the secret key. I’m going to use the OpenSSL command line interface to encrypt a message with a stream cipher: ChaCha20. This is a modern, fast, stream cipher with a good reputation:

openssl enc -chacha20 \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 0123456789abcdef0123456789abcdef \
    -in <(echo 'I would really like an ice cream right now') \
    -out ciphertext.bin

The -K option specifies the key in hexadecimal format (256 bits, or 32 bytes, or 64 hex characters), the -iv is the nonce, also known as initialization vector (128 bits, or 16 bytes, or 32 hex characters).

This trivial Python script can tamper with the ciphertext:

with open('ciphertext.bin', 'rb') as file:
    ciphertext = file.read()

guessed_message     = b'--------------------an ice cream----------\n'
replacement_message = b'--------------------to go to bed----------\n'

tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in
                            zip(ciphertext, guessed_message, replacement_message))

with open('tampered-ciphertext.bin', 'wb') as file:
    file.write(tampered_ciphertext)

This script is using partial knowledge of the message. It knows (thanks to an educated guess) that the original message contained the words “an ice cream” at a specific offset, and uses that knowledge to replace those words with new ones (“to go to bed”) which add up to the same length. Note that this technique cannot be used to remove or add parts from the message, only to modify them without changing their length.

Now if we run this script and we decrypt the tampered-ciphertext.bin file with the same key and nonce as before, we get “to go to bed” instead of “an ice cream”, without any error indicating that tampering occurred:

openssl enc -d -chacha20 \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 0123456789abcdef0123456789abcdef \
    -in tampered-ciphertext.bin

Block ciphers

We have seen that stream ciphers alone have a serious problem (malleability) that allows anyone to modify arbitrary portions of ciphertexts without detection. Let’s take a look at the alternative: block ciphers. Will they have the same problem?

While a stream cipher can encrypt variable amounts of data, a block cipher can only take as input a block of data of a fixed size, and produce as output another block of data. A good block cipher produces an output that is indistinguishable from random.

The block size is generally small, usually 128 bits (16 bytes), so if we want to encrypt larger amounts of data, we have to split the data into multiple blocks, and encrypt each block individually. If the data is too short to fit in a block, the data will also need to be padded.

   message: The cat is on th e table.........
            |______________| |______________|
                block #1         block #2
                                 (padded)

ciphertext: c2TNPW3r09hZ6f1P Vc32VX41XSy579Y9

This approach however has a problem: if we encrypt multiple blocks with the same secret key, then portions of messages that are the repeated will produce the same output. This gives the ability to analyze a ciphertext and find patterns in it without knowledge of the secret key. This problem is famously evident when encrypting pictures:

Example of applying a block cipher to an uncompressed image. The original colors are lost, but the overall layout of the image is still understandable. That's because multiple blocks of the image (containing the RGB values of each pixel), for example from the white background, are repeated multiple times, yielding the same exact encrypted blocks. The inspiration for making this image came from Wikipedia.

How this image was generated

Before jumping into how I encrypted the image, let me spend a few words on how I did NOT encrypt the image: I did not use a modern image format. Modern image formats are very sophisticated, they’re not a simple sequence of RGB values. Instead, they have some control structures mixed in the image, they implement compression to reduce the image size, etc. This complexity means that if I simply take an image in any format and encrypt it, the result won’t be visualizable by an image viewer: the image viewer would just throw an error because it would find invalid data structures.

Note that this does not imply that encrypting modern image formats is more secure: people can still analyze patterns in them, but it simply means that a modern image format, once encrypted, would not produce the sensational visualization that I showed above.

In order to produce this visualization I had to find an uncompressed image format without too much metadata in it. Thankfully the Wikipedia article on image file formats provided a list, which included the Netpbm family of formats (something I never heard of before). Among the formats in this family, I chose PPM, because it’s the one that supports colors.

The PPM file format is very simple: it has 3 lines of metadata, followed by the RGB values for each pixel. No compression. Definitely the right format for this kind of experiment!

So here’s what I did: first of all I downloaded an image (the Ubuntu “Circle of Friends” logo, obtained from Wikipedia) and converted it to PPM with ImageMagick:

convert UbuntuCoF.png img.ppm

I separated the header from the RGB values:

head -n3 img.ppm > ppm-header
tail -n+4 img.ppm > ppm-image

The reason why I separated the header from RGB values is that I won’t encrypt the header. If I did, then the image won’t be visualizable by an image viewer, just like if I used a modern image format. In a real-world scenario, a person would be able to easily guess the header if it was encrypted.

I encrypted the RGB values with AES-256, a modern, strong block cipher with a good reputation:

openssl enc -aes-256-ecb \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -in ppm-image \
    -out ppm-image-encrypted

Then I joined the header and the encrypted RGB values in a PPM file:

cat ppm-header ppm-image-encrypted > img-encrypted.ppm

This results in a randomized image that can be viewed without problems on an image viewer. It’s interesting to see that, if you change the encryption key, you will get a different randomized image!

The problem we have just seen is known as lack of diffusion. This is kinda analogous to the first problem we identified with stream ciphers: at the root of both problems there is key reuse. We solved this problem for stream ciphers by combining a key with a random nonce. We could use the same strategy here, would be an expensive approach, as initializing a block cipher with a new key is a relatively expensive operation. It’s much cheaper to initialize the block cipher once, and reuse it for every block encryption. We need a way to “link” blocks to each other, so that if two linked blocks contain the same plaintext, their encryption will give different results.

There are various strategies do that. These strategies are known as mode of operation of block ciphers. Let’s take a look at two of them: Cipher Block Chaining (CBC) and Counter Mode (CTR).

Cipher Block Chaining (CBC)

This mode of operation, as the name suggests, ‘chains’ each block to the next one. The way it works is by using the XOR operator in the following way:

First of all, a random nonce is generated. The purpose of the nonce is the same as before (with stream ciphers): ensuring that using the same secret key to perform multiple encryptions yields different results each time, so that secret information or patterns are not revealed.

The nonce does not need to be kept secret and is normally stored alongside the ciphertext so that it can be easily used during decryption. It is however important that the nonce is unique.
The first block of message m[0] is XOR-ed with the nonce, and then encrypted the block cipher, producing the first block of ciphertext c[0] = block_encrypt(m[0] XOR nonce)
The second block of message m[1] is XOR-ed with c[0], and then encrypted with the block cipher: c[1] = block_encrypt(m2 XOR c[0])
…
The last block of message m[n] is XOR-ed together with c[n-1], and then encrypted with the block cipher: c[n] = block_encrypt(m[n] XOR c[n-1])

Visualization of the Cipher Block Chaining (CBC) mode of operation

The XOR operator is back! With stream ciphers, the XOR operator was allowing us to tamper with ciphertexts. Can we do the same thing here? Yes of course! The approach is slightly different though: instead of acting directly on the block that we want to change, we will act on the block that precedes it.

For example, if we want to change the sentence “I came home in the afternoon and the cat was on the table” so that it reads ‘dog’ instead of ‘cat’, we would need to change the block right before the one that contains the word ‘cat’. If we want to change the very first block, for example to change the word ‘came’ to ‘left’, we would need to change the nonce instead.

                             nonce+ciphertext: yzURZRbP6X1w3ZRL XRDnPbEkx3JUP2Fv C2ZWt19EdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx
                                                                ⊕
        message (shifted 1 block to the left): I came home in t he afternoon and  the cat was on  the table.......
                                                                ⊕
altered message (shifted 1 block to the left): I left home in t he afternoon and  the dog was on  the table.......
                                                                =
                    tampered nonce+ciphertext: yzZVQCbP6X1w3ZRL XRDnPbEkx3JUP2Fv C2ZWt67VdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx

If we do the above, and then decrypt the tampered ciphertext, we will get something like this:

I left home in t���������������the dog was on the table

How to get this result using AES-CBC with OpenSSL

Here’s a step-by-step guide on how to tamper with a ciphertext encrypted with AES-256 in CBC mode.

First, generate a valid ciphertext:

openssl enc -aes-256-cbc \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 0123456789abcdef0123456789abcdef \
    -in <(echo 'I came home in the afternoon and the cat was on the table') \
    -out ciphertext.bin

Like in the stream cipher example, -K is the key in hexadecimal format (256 bits), while -iv is the nonce (128 bits).

We can perform the tampering with this Python script:

with open('ciphertext.bin', 'rb') as file:
    ciphertext = file.read()

guessed_message     = b'---------------------cat----------------------------------------'
replacement_message = b'---------------------dog----------------------------------------'

tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in
                            zip(ciphertext, guessed_message, replacement_message))

with open('tampered-ciphertext.bin', 'wb') as file:
    file.write(tampered_ciphertext)

Note that OpenSSL does not store the nonce along with the ciphertext, but instead expects it to be passed as a command line argument. We need to modify it separately, so here’s another Python script just for the nonce:

nonce = bytes.fromhex('0123456789abcdef0123456789abcdef')

guessed_message     = b'--came----------'
replacement_message = b'--left----------'

tampered_nonce = bytes(x ^ y ^ z for (x, y, z) in
                       zip(nonce, guessed_message, replacement_message))

print(tampered_nonce.hex())

If we run that script, we get: 01234a6382bacdef0123456789abcdef.

Now to decrypt the tampered ciphertext with the tampered nonce:

openssl enc -d -aes-256-cbc \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 01234a6382bacdef0123456789abcdef \
    -in tampered-ciphertext.bin

It’s interesting to see that we were successful in changing the word ‘cat’ to ‘dog’ but in doing so we had to sacrifice a block, which, when decrypted, resulted in random bytes.

In a real world scenario, seeing some random bytes could raise some suspicion, and maybe generate some errors in applications, however that’s not always the case (how many times have we seen garbled text on our monitors, and we never worried that somebody was tampering with our communications). Also, when dealing with formats like HTML, one could conceal tampering attempts using comment blocks, or using JavaScript. One example of what I’m describing is the EFAIL vulnerability: discovered in 2017, it affected some popular email clients including Gmail, it targeted the use of AES in CBC mode (as well as another mode very similar to it: Cipher Feedback, CFB), and allowed the injection of malicious content in HTML emails.

We can conclude that block ciphers in CBC mode, just like stream ciphers, are also malleable.

Counter Mode (CTR)

Are other modes of operation all malleable like CBC, or will they be different? Let’s take a look at another, very common, mode of operation: Counter Mode (CTR), so that we can get a better sense of how the problem of malleability can affect the world of block ciphers.

The mechanism behind Counter Mode is very simple:

A random nonce is generated. The purpose of the nonce is the usual one: make sure that repeated encryptions using the same key produce different results.
A counter (usually an integer) is initialized to 1 (or any other starting value of your choice).
The nonce is concatenated with the counter, and encrypted using the block cipher: r[0] = block_encrypt(nonce || counter).

Because the block cipher can only accept as input a block of a fixed size, it follows that the length of the nonce plus the length of the counter must be equal to the block size. For example, for a 128-bit block cipher, a common choice is to have a 96-bit nonce and a 32-bit counter.
The counter is incremented: counter = counter + 1 (the increment does not necessarily need to be by 1, but that’s a common choice). The nonce and the new counter are concatenated again, and encrypted using the block cipher: r[1] = block_encrypt(nonce || counter).
The counter is incremented again (counter = counter + 1), and a new block is encrypted, just like before: r[2] = block_encrypt(nonce || counter).
…

This mechanism produces a sequence of blocks r[0], r[1], r[2], … which are indistinguishable from random. This sequence of random blocks can be XOR-ed with the message to produce the ciphertext.

It’s important that the values for the counter never repeat. If, for example, we’re using a 32-bit counter, the counter will “reset” (go back to the starting value) after 2³² iterations, and will start repeating the same sequence of random blocks as it did at the beginning. This introduces the problem of lack of diffusion that we have seen before, just at a larger scale. If we’re using a 32-bit counter with a 128-bit block cipher, we cannot encrypt more than 128·2³² bits = 64 GiB of data at once. This is a very important detail: exceeding these limits may allow the decryption of portions of ciphertext without knowledge of the secret key.

Visualization of the Counter Mode (CTR) mode of operation

What Counter Mode is doing is effectively turning a block cipher into a stream cipher. As such, a block cipher in Counter Mode has the exact same malleability problems of stream ciphers that we have seen before.

An example of malleability using AES-CTR with OpenSSL

This example is going to be very similar (almost identical) to the example with ChaCha20 that I showed in the stream cipher section, just that this time I’m going to use AES-256 in CTR mode.

Let’s produce a valid ciphertext:

openssl enc -aes-256-ctr \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 0123456789abcdef0123456700000001 \
    -in <(echo 'Can you give me a ride to the party?') \
    -in <(echo 'Do not  give me a ride to the party!') \
    -out ciphertext.bin

Tamper it with Python:

with open('ciphertext.bin', 'rb') as file:
    ciphertext = file.read()

guessed_message     = b'--------------------an ice cream----------\n'
replacement_message = b'--------------------to go to bed----------\n'

tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in
                            zip(ciphertext, guessed_message, replacement_message))

with open('tampered-ciphertext.bin', 'wb') as file:
    file.write(tampered_ciphertext)

And now we can decrypt it:

openssl enc -d -aes-256-ctr \
    -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -iv 0123456789abcdef0123456700000001 \
    -in tampered-ciphertext.bin

The solution: Authenticated Encryption (AE)

We have seen that stream ciphers and block ciphers (in their mode of operation) both exhibit the same problem (in different flavors): malleability. I’ve shown some examples of how this problem can be exploited with modern ciphers like ChaCha20 and AES. These ciphers, alone, cannot guarantee the integrity or authenticity of encrypted data.

In this context, integrity is the assurance that data is not corrupted or modified in any way. Authenticity can be thought of as a stronger version of integrity, and it’s the assurance that a given ciphertext was produced only with knowledge of the secret key.

Does this mean that modern ciphers, like ChaCha20 and AES, should be considered insecure and avoided? Absolutely not! The correct answer is that those ciphers cannot be used alone. You should think of them as basic building blocks, and you need some additional building blocks in order to construct a complete and secure cryptosystem. One of these additional building block, that we are going to explore in this article, is an algorithm that provides integrity and authentication: welcome Authenticated Encryption (AE).

When using authenticated encryption, an adversary may be able to modify a ciphertext using the techniques described above, but such modification would be detected by the authentication algorithm, and decryption will fail with an error. The decrypted message at that point should be discarded, preventing the use of tampered data.

There are many different methods to implement authenticated encryption. The most common approach is to use an authentication algorithm to authenticate the ciphertext produced by a cipher. Here I will describe two very popular authentication algorithms:

Poly1305, which is often used in conjunction with the stream cipher ChaCha20 to form ChaCha20-Poly1305;
and Galois/Counter Mode (GCM), which is often used with the block cipher AES to form AES-GCM.

These authentication algorithms work by computing a hash of the ciphertext, which is then stored alongside the ciphertext. This hash is not a regular hash, but it’s a keyed hash. A regular hash is a function that takes as input some data and returns a fixed-size bit string:

$$\operatorname{hash}: data \rightarrow bits$$

A keyed hash instead takes two inputs: a secret key and some data, and produces a fixed-size bit string:

$$\operatorname{keyed-hash}: (key, data) \rightarrow bits$$

The output of the keyed hash is more often called Message Authentication Code (MAC), or authentication tag, or even just tag.

During decryption, the same authentication algorithm is run again on the ciphertext, and a new tag is produced. If the new tag matches the original tag (that was stored alongside the ciphertext), then decryption succeeds. Else, if the tags don’t match, it means that the ciphertext was modified (or the tag was modified), and decryption fails. This gives us a way to detect tampering and gives us the opportunity to reject ciphertexts that were not produced by the secret key.

The secret key passed to the keyed hash function is not necessarily the same secret key used for the encryption. In fact, both ChaCha20-Poly1305 and AES-GCM operate on a subkey derived from the key used for encryption.

Poly1305

Poly1305 is a keyed hash function proposed by Daniel J. Bernstein in 2004. It works by using polynomials evaluated modulo the prime 2¹³⁰ - 5, hence the name.

The key to Poly1305 is a 256-bit string, and it’s split into two halves:

the first half (128 bits) is called $r$;
the second half (128 bits) is called $s$.

We’ll see later how this key is generated when Poly1305 is used to implement authenticated encryption. For now, let’s assume that the key is a random (unpredictable) bit string provided as an input.

The first half $r$ is also clamped by setting some of its bits to 0. This is a performance-related optimization that some Poly1305 implementations can take advantage of when doing multiplication using 64-bit registers. Clamping is performed by applying the following hexadecimal bitmask:

0ffffffc0ffffffc0ffffffc0fffffff

The message to authenticate is split into chunks of 128 bits each: $m_1$, $m_2$, $m_3$, … $m_n$. If the length of the message is not a multiple of 128 bits, then the last block may be shorter. The authentication tag is then calculated as follows:

Interpret $r$ and $s$ as two 128-bit little-endian integers.
Initialize the Poly1305 state $a_0$ to the integer 0. As we shall see later, this state will need to hold at most 131 bits.
For each block $m_i$:
- Interpret the block $m_i$ as a little-endian integer.
- Compute $\overline{m}_i$ by appending a 1-bit to the end of the block $m_i$. If $m_i$ is 128 bits long, then this is equivalent to computing $\overline{m}_i = 2^{128} + m_i$. In general, if the length of the block $m_i$ in bits is $\operatorname{len}(m_i)$, then this is equivalent to $\overline{m}_i = 2^{\operatorname{len}(m_i)} + m_i$.
  
  This step ensures that the resulting block $\overline{m}_i$ is always non-zero, even if the original block $\overline{m}_i$ is zero. This is important for the security of the algorithm, as explained later.
- Compute the new state $a_i = (a_{i-1} + \overline{m}_i) \cdot r \pmod{2^{130} - 5}$. Note that, because the operation is modulo $2^{130} - 5$, the result will always fit in 130 bits.
Once each block has been processed, compute the final state $a_{n+1} = a_n + s$. Note that the state $a_n$ is at most 130 bits long, and $s$ is at most 128 bits long, hence the result will be at most 131 bits long.
Truncate the final state $a_{n+1}$ to 128 bits by removing the most significant bits.
Return the truncated final state $a_{n+1}$ as a little-endian byte string.

What this method is doing is computing the following polynomial in $r$ and $s$:

$$\begin{align*} tag & = ((((((\overline{m}_1 \cdot r) + \overline{m}_2) \cdot r) + \cdots + \overline{m}_n) \cdot r) \bmod{(2^{130} - 5)}) + s \\ & = (\overline{m}_1 r^n + \overline{m}_2 r^{n-1} + \cdots + \overline{m}_n r) \bmod{(2^{130} - 5)} + s \end{align*}$$

$r$ and $s$ are secrets, and they come from the Poly1305 key. Note that if we didn’t add $s$ at the end, then the resulting polynomial would be a polynomial in $r$, and one could use polynomial root-finding methods to figure out $r$ from the authentication tag, without knowledge of the key. Therefore it’s important that $s$ is non-zero.

In Python, this is what a Poly1305 implementation could look like (disclaimer: this is for learning purposes, and not necessarily secure or optimized for performance):

def iter_blocks(message: bytes):
    """
    Splits a message in blocks of 16 bytes (128 bits) each, except for the last
    block, which may be shorter.
    """
    start = 0
    while start < len(message):
        yield message[start:start+16]
        start += 16

def poly1305(key: bytes, message: bytes):
    assert len(key) == 32  # 256 bits

    # Prime for the evaluation of the polynomial
    p = (1 << 130) - 5

    # Split the key into two parts r and s
    r = int.from_bytes(key[:16], 'little')  # 128 bits
    s = int.from_bytes(key[16:], 'little')  # 128 bits
    # Clamp r
    r = r & 0x0ffffffc0ffffffc0ffffffc0fffffff

    # Initialize the state
    a = 0

    # Update the state with every block
    for block in iter_blocks(message):
        # Append a 1-bit to the end of each block
        block = block + b'\1'
        # Convert the block to an integer
        c = int.from_bytes(block, 'little')
        # Update the state
        a = ((a + c) * r) % p

    # Add s to the state and truncate it to 128 bits, removing the most
    # significant bits and keeping only the least significant 128 bits
    a = (a + s) & ((1 << 128) - 1)

    # Convert the state from an integer to a 16-byte string (128 bits)
    return a.to_bytes(16, 'little')

And here is an example of how that code could be used:

key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')
msg = b'I had a very nice day today at the beach'
print(poly1305(key, msg).hex())

This returns b0c4cb74b3089e9a982e3baa90c1bb5f, which is the same result that we would get using OpenSSL:

openssl mac \
    -macopt hexkey:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
    -in <(echo -n 'I had a very nice day today at the beach') \
    poly1305

A few things to note:

The same key cannot be reused to construct two distinct tags. In fact, suppose that we use the same hash key to compute tag1 = Poly1305(key, msg1) and tag2 = Poly1305(key, msg2). Then, because $s$ is the same for both, we could subtract the two tags (tag1 - tag2) to remove the $s$ part and obtain a polynomial in $r$. From there, we could use algebraic methods to figure out $r$. Once we have $r$, we can use either one of the tags and compute $s$, therefore recovering the full secret key.

Similarly, if the keys were generated using a predictable algorithm (for example, incrementally: key[i+1] = key[i] + 1), it would still be possible to use a similar approach to figure out the secret key.

For this reason, Poly1305 keys must be unique and unpredictable. Generating Poly1305 keys randomly or pseudo-randomly is an acceptable approach. Authentication functions like Poly1305 are called one-time authenticators because they can be used only one time with the same key.
If we didn’t add the 1-bits at the end of each block (in other words, if we used the $m_i$ blocks instead of $\overline{m}_i$), then encrypting a message full of zero bits would be the equivalent of encrypting an empty message. Adding the 1-bits is a way to ensure that the length of the message always has an effect on the output.

Use of Poly1305 with ChaCha20 (ChaCha20-Poly1305)

Let’s see how we can combine ChaCha20 and Poly1305 to construct an authenticated cipher. To recap:

ChaCha20 is a stream cipher;
Poly1305 is a one-time authenticator;
ChaCha20, like most ciphers, requires the use of a unique nonce to allow key reuse.

Putting the two together gives birth to ChaCha20-Poly1305. Here I’m going to describe how to implement it as standardized in RFC 8439.

The inputs to the ChaCha20-Poly1305 encryption function are:

a 256-bit secret key;
a 96-bit nonce;
a variable-length plaintext message.

The outputs from the ChaCha20-Poly1305 encryption function are:

a variable-length ciphertext (same length as the input plaintext);
a 128-bit authentication tag.

The ChaCha20-Poly1305 decryption function will accept the same secret key, nonce, ciphertext, and authentication tag as the input, and produce either the plaintext or an error as the output. The error is returned in case the authentication fails.

Diagram of data flow during encryption with ChaCha20-Poly1305

ChaCha20-Poly1305 works in the following way:

The ChaCha20 stream cipher is initialized with the 256-bit secret key and the 96-bit nonce.
The stream cipher is used to encrypt a 256-bit string of all zeros. The result is the Poly1305 subkey.

If you recall how a stream cipher works, you should know that encrypting using a stream cipher is equivalent to performing the XOR of a random bit stream with the plaintext. Here the plaintext is all zeros, so the process of generating the Poly1305 subkey is equivalent to grabbing the first 256 bits from the ChaCha20 bit stream.

We previously saw that the Poly1305 subkey must be unpredictable and unique in order for Poly1305 to be secure. The use of ChaCha20 with a unique nonce ensures that: because ChaCha20 is a stream cipher, its output will be random and unpredictable. Therefore, with this construction, the subkey will be unpredictable even if the nonce is predictable.
The stream cipher is used to encrypt another 256-bit string. The result is discarded. This is equivalent to advancing the stream cipher state by 256 bits.

This step may seem weird, and in fact is not needed for security purposes, but it’s a mere implementation detail. This step is here because ChaCha20 has an internal state of 512 bits. In the previous step we obtained the first 256 bits of the state, and this next step is to discard the rest of the state to start with a fresh state. There is no particular reason for requiring a fresh state. The reason why RFC 8439 does that is because… spoiler alert: ChaCha20 is a block cipher under the hood. Its block size is 512 bits. If you read the RFC, you’ll see that it asks to call the ChaCha20 block encryption function once, grab the first 256 bits, and discard the rest. Here I’m using ChaCha20 as a stream cipher, so I have to include this extra step to discard the bits.
The plaintext is encrypted using the stream cipher.

Note that this is done without resetting the state of the cipher. We are continuing to use the same stream cipher instance that was used to generate the Poly1305 subkey.
The ciphertext is padded with zeros to make its length a multiple of 16 bytes (128 bits) and is authenticated using Poly1305, via the subkey generated in step 2.

This step may be done in parallel to the previous one, that is: every time we generate a chunk of ciphertext, we feed it to the Poly1305 authentication function.

Why pad the ciphertext before passing it to Poly1305? After all, ChaCha20 is a stream cipher, and Poly1305 can accept arbitrary-sized messages. Again, this is an detail of RFC 8439 and padding does not serve any specific purpose.
The length of the ciphertext (in bytes) is fed into the Poly1305 authenticator. This length is represented as a 64-bit little-endian integer padded with 64 zero bits.

The reason why the length is represented as 64 bits and padded (instead of representing it as 128 bits) will be clearer later: what I have given you so far is a simplified view of ChaCha20-Poly1305 and authenticated encryption in general. I will give you the full picture when talking about associated data later on, and at that point this step will be slightly modified.
The ciphertext from ChaCha20 and the authentication tag from Poly1305 are returned.

The decryption algorithm works in a very similar way: ChaCha20 is initialized in the same way, the subkey is generated in the same way, the Poly1305 authentication tag is calculated from the ciphertext in the same way. The only difference is that ChaCha20 is used to decrypt the ciphertext (instead of encrypting the plaintext) and that the input authentication tag is compared to the calculated authentication tag before returning.

Here is a Python implementation of ChaCha20-Poly1305, based on the implementations of ChaCha20 and Poly1305 from pycryptodome (usual disclaimer: this code is for educational purposes, and is not necessarily secure or optimized for performance):

from Crypto.Cipher import ChaCha20
from Crypto.Hash import Poly1305

def chacha20poly1305_encrypt(key, nonce, message):
    assert len(key) == 32  # 256 bits
    assert len(nonce) == 12  # 96 bits

    # Initialize the ChaCha20 cipher with the key and nonce
    cipher = ChaCha20.new(key=key, nonce=nonce)

    # Derive the Poly1305 subkey using the ChaCha20 cipher
    subkey = cipher.encrypt(b'\0' * 32)  # 256 bits
    subkey_r = subkey[:16]
    subkey_s = subkey[16:]

    # Initialize the Poly1305 authenticator with the subkey
    authenticator = Poly1305.Poly1305_MAC(r=subkey_r, s=subkey_s, data=None)

    # Discard the rest of the internal ChaCha20 state
    cipher.encrypt(b'\0' * 32)  # 256 bits

    # Encrypt the message
    ciphertext = cipher.encrypt(message)

    # Authenticate the ciphertext
    authenticator.update(ciphertext)
    # Pad the ciphertext with zeros (to make it a multiple of 16 bytes)
    if len(ciphertext) % 16 != 0:
        authenticator.update(b'\0' * (16 - len(ciphertext) % 16))
    # Authenticate the length of the associated data (0 for simplicity)
    authenticator.update((0).to_bytes(8, 'little'))  # 64 bits
    # Authenticate the length of the ciphertext
    authenticator.update(len(ciphertext).to_bytes(8, 'little'))  # 64 bits
    # Generate the authentication tag
    tag = authenticator.digest()

    return (ciphertext, tag)

def chacha20poly1305_decrypt(key, nonce, ciphertext, tag):
    assert len(key) == 32  # 256 bits
    assert len(nonce) == 12  # 96 bits
    assert len(tag) == 16  # 128 bits

    # Initialize the ChaCha20 cipher and the Poly1305 authenticator, in the
    # same exact way as it was done during encryption
    cipher = ChaCha20.new(key=key, nonce=nonce)

    subkey = cipher.encrypt(b'\0' * 32)
    subkey_r = subkey[:16]
    subkey_s = subkey[16:]
    authenticator = Poly1305.Poly1305_MAC(r=subkey_r, s=subkey_s, data=None)

    cipher.encrypt(b'\0' * 32)

    # Generate the authentication tag, like during encryption
    authenticator.update(ciphertext)
    if len(ciphertext) % 16:
        authenticator.update(b'\0' * (16 - len(ciphertext) % 16))
    authenticator.update((0).to_bytes(8, 'little'))
    authenticator.update(len(ciphertext).to_bytes(8, 'little'))
    expected_tag = authenticator.digest()

    # Compare the input tag with the generated tag. If they're different, the
    # plaintext must not be returned to the caller
    if tag != expected_tag:
        raise ValueError('authentication failed')

    # The two tags match; decrypt the plaintext and return it to the caller
    # Note that, because ChaCha20 is a symmetric cipher, there is no difference
    # between the encrypt and decrypt method: here we are reusing the same
    # exact code used during decryption
    message = cipher.encrypt(ciphertext)

    return message

And here is how it can be used:

key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')
nonce = bytes.fromhex('0123456789abcdef01234567')
message = b'I wanted to go to the beach, but now I changed my mind'

ciphertext, tag = chacha20poly1305_encrypt(key, nonce, message)
decrypted_message = chacha20poly1305_decrypt(key, nonce, ciphertext, tag)
assert message == decrypted_message

print(f'ciphertext: {ciphertext.hex()}')
print(f'       tag: {tag.hex()}')
print(f' plaintext: {decrypted_message}')

Running it produces the following output:

ciphertext: 5d9b09cc5d90ca9ddff2d3470cfd6b563c5158e952bfae6acf1ebf9a3b968a488a41969567ef5ccfe05dcf9e548567028ff374a754af
       tag: dac3c05d261920e278ceb22e2800aa95
 plaintext: b'I wanted to go to the beach, but now I changed my mind'

This is the same output we would obtain by using the ChaCha20-Poly1305 implementation from pycryptodome directly:

from Crypto.Cipher import ChaCha20_Poly1305

key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')
nonce = bytes.fromhex('0123456789abcdef01234567')
message = b'I wanted to go to the beach, but now I changed my mind'

cipher = ChaCha20_Poly1305.new(key=key, nonce=nonce)
ciphertext, tag = cipher.encrypt_and_digest(message)

print(f'ciphertext: {ciphertext.hex()}')
print(f'       tag: {tag.hex()}')

As already stated, it is extremely important that the nonce passed to ChaCha20-Poly1305 is unique. It may be predictable, but it must be unique. If the same nonce is reused twice or more, we can:

Decrypt arbitrary messages without using the secret key, if we can guess at least one message from its ciphertext.

This can be done using the techniques described at the beginning of this article: by recovering the random bit string from the XOR of the ciphertext with the guessed message.
Recover the Poly1305 subkey, and, at that point, tamper with ciphertexts and forge new, valid authentication tags.

This can be done by using algebraic methods on the polynomial of the authentication tag.

There is also a variant of ChaCha20-Poly1305, called XChaCha20-Poly1305, that features an extended 192-bit nonce (the X stands for ‘extended’). This is described in an RFC draft but so far it hasn’t been accepted as a standard yet. I won’t cover XChaCha20 in detail here, because it’s slightly more complex and does not add much to the topic of this article, but XChaCha20-Poly1305 has better security properties than ChaCha20-Poly1305, so you should prefer it in your applications if you can use it. The reason why XChaCha20-Poly1305 has better properties than ChaCha20-Poly1305 is that, having a longer nonce, the probability of generating two random nonces with the same value are much lower.

Galois/Counter Mode (GCM)

Let’s now take a look at Galois/Counter Mode (GCM). This is commonly used with the Advanced Encryption Standard (AES), to construct the authenticated cipher AES-GCM. One main difference between Poly1305 and GCM is that Poly1305 can work with any stream or block cipher, while GCM is designed to work with block ciphers with a block size of 128 bits.

GCM was proposed by David McGrew and John Viega in 2004 and is standardized in NIST Special Publication 800-38D as well as RFC 5288. It takes its name from Galois fields, also known as finite fields, which in turn get their name from the French mathematician Évariste Galois, who introduced the concept of finite fields as we know them today.

As we did before with Poly1305, we are going to first see how the keyed hash function used by GCM works, and then we will see how to use it to construct an authenticated cipher like AES-GCM on top of it. Before we can do that though, we need to understand what are finite fields, and what specific type of finite fields are used in GCM.

Finite Fields (Galois Fields)

What is a field? A field is a mathematical structure that contains a bunch of elements, and those elements can interact with each other using addition and multiplication. For both these operations there’s an identity element and an inverse element. Addition and multiplication in a field must obey the usual properties that we’re used to: commutativity, associativity, and distributivity.

A well-known example of a field is the field of fractions. Here is why fractions form a field:

the elements of the field are the fractions;
addition is well-defined: if we add two fractions, we get a fraction out (example: $5/3 + 3/2 = 19/6$);
multiplication is also well-defined: if we multiply two fractions, we get a fraction out (example: $1/2 \cdot 8/3 = 4/3$);
the additive identity element is 0: if we add 0 to any fraction, we get the same fraction back;
the additive inverse element is the negated fraction (example: $5/1$ is the additive inverse of $-5/1$ because $5/1 + (-5/1) = 0$);
the multiplicative identity element is 1: multiplying any fraction by 1 yields the same fraction back;
the multiplicative inverse element is what we get if we swap the numerator with the denominator (example: $3/2$ is the multiplicative inverse of $2/3$ because $3/2 \cdot 2/3 = 1$)—except for 0, which does not have a multiplicative inverse.
and so on…

On top of addition, multiplication, and inverse elements, we can define derived operations like subtraction and division. Subtracting $a$ from $b$ is equivalent to adding $a$ to the additive inverse of $b$: $a - b = a + (-b)$. Similarly, division can be defined in terms of multiplication with multiplicative inverses ($a / b = a b^{-1}$).

Fields are a generalization of structures where addition, multiplication, subtraction, and division behave according to the rules that we’re used to. Field elements do not necessarily need to be numbers.

An example of something that is not a field is the integers. That’s because integers don’t have multiplicative inverses (for example, there’s no integer that multiplied by 5 makes the result equal to 1). However, there is a way to turn the integers into a field: if we take the integers and a prime number p, then we can construct the field of integers modulo p.

When we work in the integers modulo a prime p, whenever we see p appear in any of our expressions, we can replace it with 0. In other words, in such a field, p and 0 are two different ways to write the same element–they are two different representations of the same element.

Here is an example: in the field of integers modulo 7, the expression 5 + 3 equals 1 because

5 + 3 evaluates to 8;
8, by definition, is 7 + 1;
if 7 and 0 are the same element, then 7 + 1 is equal to 0 + 1
0 + 1 evaluates to 1

What we have just seen is that 8 is just a different representation of 1, just like 7 is a different representation of 0. Different symbols, same object. Just like, in programming languages, we can have multiple variables point to the same memory location: here the numbers are like variables, and what they point to is what really matters.

In the field of integers modulo 7, the additive inverse for 5 is 2, because 5 + 2 = 7 = 0. If we manipulate the equation, we get that 5 = −2. In other words, 5 and −2 are two different representations of the same element, and for the same reason 2 and −5 are also two different representations of the same element. A similar story holds for multiplication: the multiplicative inverse for 5 is 3 because: 5 · 3 = 15 = 7 + 7 + 1 = 1, so we can write 5 = 3⁻¹ as well as 3 = 5⁻¹.

What we have just seen is an example of a finite field. It’s different from a general field because it contains a finite number of elements (unlike fractions, which do not have a limit). In the case of the integers modulo 7, the number of elements is 7, and the list of elements is: {0, 1, 2, 3, 4, 5, 6}, or {−3, −2, −1, 0, 1, 2, 3}, or {0, 1, 2, 3, 2⁻¹, 3⁻¹, 6}, depending on what representation we like the most.

A few words about terminology, notation, and equivalences of finite fields

There can be many ways to construct a finite field (or even a general field). I have given an example using numbers, but a field does not necessarily need to be formed from numbers. We can also use vectors, matrices, polynomials, and anything you would like. As long as addition, multiplication, identity elements, and inverse elements are well-defined, you can get a field. Using programming terms, you can think of a field as an interface or a trait that can have arbitrary implementations.

An important result in algebra is that finite fields with the same number of elements are unique up to isomorphism. This means that if two finite fields the same number of elements, then there is an equivalence relation between the two. The number of elements of a field is therefore enough to define a field. It’s not enough to tell us what the elements of the field look like, or how they can be represented, but it’s enough to know how it behaves. To denote a field with $n$ elements, there are two major notations: $GF(n)$ and $\mathbb{F}_{n}$.

Another important result in algebra is that $n$ may be either a prime number, or a power of a prime. For example, we can have finite fields with 2 elements, or with 9 (= 3²) elements, but we cannot have a field with 6 (= 2·3) elements. For this reason, you will often find finite fields denoted as $GF(p^k)$ or $\mathbb{F}_{p^k}$, where $p$ is a prime and $k$ is an integer greater than 0. The prime $p$ is called characteristic of the field, while $n = p^k$ is called order of the field.

Some common fields also have their own notation: in particular, the field of integers modulo a prime $p$ is denoted as $Z/pZ$. This notation encodes the “building instructions” to construct the field, in fact:

$Z$ denotes the integers: $Z = \{\dots, -2, -1, 0, 1, 2, \dots\}$;
$pZ$ denotes the integers multiplied by $p$: $pZ = \{\dots, -2p, -p, 0, p, 2p\}$ (example: $2Z = \{\dots, -4, -2, 0, 2, 4, \dots\}$);
$A/B$ is a quotient. This is a way to define an equivalence relation between elements, and its meaning is: within $A/B$, all the elements of $B$ are equivalent to 0. In the case of $Z/pZ$, all the multiples of $p$ are equivalent to 0, which is indeed what happens with the integers modulo $p$. The way I described this equivalence relation earlier is by saying that multiples of $p$ are different representations for 0.

Note that the integers modulo a power of a prime ($Z/p^kZ$, with $k$ greater than 1) do not form a field. The problem is that elements in $Z/p^kZ$ sometimes do not have a multiplicative inverse. For example, in $Z/4Z$, the number 2 does not have a multiplicative inverse (there is no element that multiplied by 2 gives 1). A field $GF(p^k)$ with $k$ greater than 1 needs to be constructed in a different way. One such way is to use polynomials, as described in the next section.

Polynomial fields

Let’s now move our attention from integers to polynomials, like this one:

$$x^7 + 5x^3 - 9x^2 + 2x + 1$$

Polynomials are a sum of coefficients multiplied by a variable (usually denoted by the letter x) raised to an integral power.

Let’s restrict our view to polynomials that have integer coefficients, like the one shown above. Something that is not a polynomial with integer coefficients is $1/2 x^2 + x$, because it has a fraction in it.

Integers and polynomials with integer coefficients are somewhat similar to each other. They kinda behave the same in many aspects. One important property of integers is the unique factorization theorem: if we have an integer, there’s a way to write it as a multiplication of some primary factors. For example, the integer 350 can be factored as the multiplication of 2, 5, 5, and 7.

$$350 = 7 \cdot 5 \cdot 5 \cdot 2$$

This factorization is unique: we can change the order of the factors, but it’s not possible to obtain a different set of factors (there’s no way to make the number 3 appear in the factorization of 350, or to make the number 7 disappear).

Polynomials with integer coefficients also have a unique factorization. In the case of integers, We call the unique factors “prime numbers”; in the case of polynomials we have “irreducible polynomials”. And just like we can have a field of integers modulo a prime, we can have a field of polynomials modulo an irreducible polynomial.

Integers	Polynomials (with integer coefficients)
Unique factorization: $42 = 7 \cdot 3 \cdot 2$	Unique factorization: $x^3 - 1 = (x^2 + x + 1)(x - 1)$
Prime numbers: 2, 3, 5, 7, 11, …	Irreducible polynomials: $x + 1$, $x^2 - 2$, $x^2 + x + 1$, …
Integers modulo a prime number	Polynomials modulo an irreducible polynomial

Let’s take a look at how arithmetic in polynomial fields work. Let’s take, for example, the field of polynomials with integer coefficients modulo $x^3 + x + 1$, and try to compute the result of $(x^2 + 1)(x^2 + 2)$. If we expand the expression, we get:

$$(x^2 + 1)(x^2 + 2) = x^4 + 3x^2 + 2$$

This expression can be reduced. Reducing a polynomial expression is the equivalent of what we were doing with integers modulo a prime, when we were saying that 8 = 7 + 1 = 1 (mod 7). That “conversion” from 8 to 1 is the equivalent of the reduction that we’re talking about here.

To reduce $x^4 + 3x^2 + 2$, first note that $x^4 = x \cdot x^3$. Also note that $x^3 = x^3 + x + 1 - x - 1$. Here we have just added and removed the term $x + 1$: the result hasn’t changed, but now the irreducible polynomial $x^3 + x + 1$ appears in the expression, and so we can substitute it with 0. Putting everything together, we get:

$$\begin{align*} (x^2 + 1)(x^2 + 2) & = x^4 + 3x^2 + 2 \\ & = x \cdot x^3 + 3x^2 + 2 \\ & = x \cdot (x^3 + x + 1 - x - 1) + 3x^2 + 2 \\ & = x \cdot (0 - x - 1) + 3x^2 + 2 \\ & = -x^2 - x + 3x^2 + 2 \\ & = 2x^2 - x + 2 \end{align*}$$

It’s interesting to note that, if the polynomial field is formed by an irreducible polynomial with degree $n$, then all the polynomials in that field will all have degree less than $n$. That’s because if any $x^n$ (or higher) appears in a polynomial expression, then we can use the substitution trick I just showed to reduce its degree.

Binary fields

Let’s now look at polynomials where coefficients are from the field of integers modulo 2, meaning that they can be either 0 or 1. This is an example of such a polynomial:

$$x^7 + x^4 + x^2 + 1$$

or, in a more explicit form, where we can clearly see all the coefficients:

$$1 x^7 + 0 x^6 + 0 x^5 + 1 x^4 + 0 x^3 + 1 x^2 + 0 x^1 + 1 x^0$$

These are called binary polynomials. It’s interesting to note that if we ignore the variables and the powers, and keep only the coefficients, then what we get is a bit string:

$$(1 0 0 1 0 1 0 1)$$

This suggests that there’s an interesting duality between binary polynomials and bit strings. This means, in particular, that binary polynomials can be represented in a very compact and natural way on computers.

The duality between binary polynomials and bit strings also suggests that perhaps we can use bitwise operations to perform arithmetic on binary polynomials. And this turns out to be true, in fact:

binary polynomial addition can be computed using the XOR on the two corresponding bit strings;
binary polynomial multiplication can be computed using XOR, AND and bit-shifting.

Computers are pretty fast at performing these bitwise operations, and this makes binary polynomials quite attractive for use in computer algorithms and cryptography.

Arithmetic with binary polynomials

The arithmetic of such polynomials is quite interesting: in fact, because $1 + 1 = 0$ (modulo 2), then also $x^k + x^k = 0$, in fact:

$$1 \cdot x^k + 1 \cdot x^k = (1 + 1) x^k = 0 \cdot x^k = 0$$

It’s easy to see that addition modulo 2 is equivalent to the XOR binary operator. And addition of two binary polynomials is equivalent to the bitwise XOR of their corresponding bit strings:

$$\begin{array}{ccccc} (x^3 + x^2 + 1) & + & (x^2 + x) & = & x^3 + x + 1 \\ \updownarrow & & \updownarrow & & \updownarrow \\ (1101) & \oplus & (0110) & = & (1011) \end{array}$$

Multiplication of binary polynomials can also be implemented as a bitwise operation on bit strings. First, note that multiplying a polynomial by a monomial is equivalent to bit-shifting:

$$\begin{array}{ccccc} (x^3 + x + 1) & \cdot & x^2 & = & x^5 + x^3 + x^2 \\ \updownarrow & & \updownarrow & & \updownarrow \\ (1011) & \ll & 2 & = & (101100) \end{array}$$

Then note that multiplication of two polynomials can be expressed as the sum of multiplications by monomials:

$$(x^3 + 1)(x^2 + x + 1) = (x^3 + 1) \cdot x^2 + (x^3 + 1) \cdot x^1 + (x^3 + 1) \cdot x^0$$

Putting everything together, we have multiplications by monomials (equivalent to bit-shifts) and sums (equivalent to bitwise XOR). This suggests that multiplication can be implemented on top of bitwise XOR and bit-shifting.

Here is some Python code to implement binary polynomial multiplication, where each polynomial is represented compactly as an int:

def multiply(a, b):
    """
    Compute a*b, where a and b are two integers representing binary
    polynomials.

    a and b are expected to have their most significant bit set to
    the monomial with the highest power. For example, the polynomial
    x^8 is represented as the integer 0b10000.
    """
    assert a >= 0
    assert b >= 0

    result = 0
    while b:
        result ^= a * (b & 1)
        a <<= 1
        b >>= 1
    return result

Other than XOR and bit-shifting, this code also uses AND to “query” whether a certain monomial is present or not.

Here is an example of how to use the code:

a = 0b0101_0111              # x^6 + x^4 + x^2 + x + 1
b = 0b0001_1010              # x^4 + x^3 + x
c = multiply(a, b)
assert c == 0b0111_0110_0110 # x^10 + x^9 + x^8 + x^6 + x^5 + x^2 + x

Now that we have introduced binary polynomials, we can of course form binary polynomials modulo a binary irreducible polynomial. These form a finite field, which is more concisely called: binary field.

Note that in a binary field where the modulo is an irreducible polynomial of degree $n$, all polynomials in the field can be represented as $n$-bit strings, and all $n$-bit strings have a corresponding binary polynomial in the field.

Arithmetic in binary fields

If we have three integers $a$, $b$, and $p$, we can compute $(a + b) \bmod{p}$ or $a \cdot b \bmod{p}$ by performing the binary operation (addition or multiplication) and then taking the remainder of the division by $p$. This is a method that returns the results of addition or multiplication using a representation with the lowest number of digits possible.

What if instead of having 3 integers we have three binary polynomials $A$, $B$, and $P$ and we want to compute $(A + B) \bmod{P}$ or $A \cdot B \bmod{P}$? It turns out that these operations can be implemented with code that is even easier than the integer counterpart: no division needs to be involved!

Let’s start with addition: we have already seen that addition with binary polynomials can be implemented with a simple XOR operation. This means that if the degree of $A$ and $B$ is lower than the degree of $P$, then the result of $A + B$ is also going to have degree less than $P$, hence no reduction is needed. We can use the result as-is, without any transformation: adding two binary field elements can be implemented with a single XOR operation.

With multiplication the story is different: the product $A \cdot B$ may have degree equal to or higher than $P$. For example, if $A = B = x$ and $P = x^2 + 1$, the product $A \cdot B$ is equal to $x^2$, which has the same degree as $P$. We need to find a way to efficiently reduce the higher-degree terms of this product. To see one way to do that, note that we can write $P$ like this:

$$P = x^n + Q$$

where $n$ is the degree of $P$ (the maximum power of $P$) and $Q$ is another binary polynomial, with degree strictly lower than $n$. Rearranging the equation, we get:

$$x^n = P + Q$$

Note that subtraction and addition are the same operations in a binary field. Because $P$ equals 0, we can write:

$$x^n = Q$$

This equivalence gives us a way to eliminate higher-level terms that appear during multiplication: whenever we see an $x^n$ appearing in the result, we can remove that term and add $Q$ instead. One way to do that, using binary strings, is to discard the highest bit (the one corresponding to $x^n$) and XOR with the binary string corresponding to $Q$.

Another way to do it is to just add $P$ (XOR by the binary string corresponding to $P$). This is equivalent to adding 0, results in the more compact representation that we’re interested in.

We could use similar tricks to eliminate terms like $x^{n+1}$, but these tricks are not necessary if we eliminate $x^n$ terms as soon as they appear in an iterative way.

Here is some Python code for multiplication in binary fields that uses the “add $P$” trick just described:

def multiply(a, b, p):
    """
    Compute a*b modulo p, where a, b and c are three integers representing
    binary polynomials.

    a, b and p are expected to have their most significant bit set to the
    highest power monomial. For example, the polynomial x^8 is represented as
    0b10000.
    """
    bit_length = p.bit_length() - 1
    assert a >= 0 and a < (1 << bit_length)
    assert b >= 0 and b < (1 << bit_length)

    result = 0
    for i in range(bit_length):
        result ^= a * (b & 1)
        a <<= 1
        a ^= p * ((a >> bit_length) & 1)
        b >>= 1
    return result

This code is essentially the same as the binary polynomial multiplication code we had before, except for this line in the for loop:

a ^= p * ((a >> bit_length) & 1)

This line is what “adds $P$” whenever adding the shifted $A$ would result in a $x^n$ term to appear.

Again, we achieved implementing multiplication using only XOR, AND and bit-shifting.

Note that the binary polynomial $P$ here does not necessarily need to be an irreducible polynomial for this algorithm to work. However, the resulting algebraic structure won’t be a field unless $P$ is irreducible. A similar story holds for integers: we can have integers modulo a non-prime number, but that’s not a field.

The GHASH keyed hash function

GCM uses a binary field. The irreducible binary polynomial that defines the binary field used by GCM is:

$$P = x^{128} + x^7 + x^2 + x + 1$$

We will call this field the GCM field. Note that this polynomial has degree 128, hence the GCM field elements can be represented as 128-bit strings, and each 128-bit string has a corresponding element in the GCM field.

The keyed hash function used by GCM is called GHASH and takes as input a 128-bit key. We will call this key $H$. This key is interpreted as an element of the GCM field.

The message to authenticate is split into blocks of 128 bits each: $M_1$, $M_2$, $M_3$, … $M_n$. If the length of the message is not a multiple of 128 bits, then the last block is padded with zeros. Each block of message is also interpreted as an element of the GCM field.

Here is how the authentication tag is computed from $H$ and the padded message blocks $M_1$, …, $M_n$:

The initial state (a GCM field element) is initialized to 0: $A_0 = 0$.
For every block of message $M_i$, the next state $A_i$ is computed as $A_i = (A_{i-1} + M_i) \cdot H \bmod{P}$.
The final state $A_n$ is returned as a 128-bit string.

What this function is doing is computing the following polynomial in $H$:

$$\begin{align*} Tag & = (((M_1 \cdot H + M_2) \cdot H + \cdots M_n) \cdot H) \bmod{P} \\ & = (M_1 H^n + M_2 H^{n-1} + \cdots M_n H) \bmod{P} \end{align*}$$

This construction is somewhat similar to the one from Poly1305, although there are important differences:

In Poly1305, the elements of the tag polynomial are integers modulo a prime, in GHASH they are elements of a binary field.
GHASH does not perform any step to encode the length of the message, hence the tag for an empty message will be the same as the tag for a sequence of zero blocks. We will see later that GCM fixes this problem by appending the length of the message to the end of the input passed to GHASH.
Most importantly, the final $Tag$ polynomial is a polynomial in one unknown, and as such $H$ may be easily recoverable using algebraic methods. For this reason, GHASH is not suitable as a secure one-time authenticator. We will see that GCM fixes this problem by encrypting the output of GHASH.

Use of GCM with AES (AES-GCM)

GCM is the combination of a block cipher, Counter Mode (CTR), and the GHASH function that we have just seen. The block cipher is often AES. When we combine AES with GCM, the what we get is AES-GCM, which is described below. However the block cipher does not necessarily need to be AES: what is important is that the block size of the cipher is 128 bits, and that’s because GHASH only works on 128-bit blocks.

The inputs to the AES-GCM encryption function are:

a secret key (the length of the key depends on the variant of AES used: if AES-128, this will be 128 bits);
a 96-bit nonce;
a variable-length plaintext message.

The outputs of the AES-GCM encryption function are:

a variable-length ciphertext (same length as the input plaintext);
a 128-bit authentication tag.

The AES-GCM decryption function will accept the same secret key, nonce, ciphertext, and authentication tag as the input, and produce either the plaintext or an error as the output. The error is returned in case the authentication fails.

Diagram of data flow during encryption with AES-GCM

AES-GCM works in the following way:

The GHASH subkey $H$ is generated by encrypting a zero-block: $H = \operatorname{Encrypt}(key, \underbrace{000\dots0}_\text{128 bits})$.
The block cipher AES is initialized in Counter Mode (AES-CTR) with the key, the nonce, and a 32-bit, big-endian counter starting at 2.
The plaintext is encrypted using the instance of AES-CTR just created.
The GHASH function is run with the following inputs:
- the subkey $H$, computed in step 1;
- the ciphertext padded with zeros to make its length a multiple of 16 bytes (128 bits), concatenated to the length (in bits) of the ciphertext represented as a 128-bit big-endian integer.
The result is a 128-bit block $S = \operatorname{GHASH}(H, ciphertext || padding || length)$.
The AES-CTR counter is set to 1.
The block $S$ is then encrypted using AES-CTR. The result of the encryption is the authentication tag.

Note that, because $S$ matches the block size of the cipher, this encryption won’t cause the counter value 2 to be reused.
The ciphertext and authentication tag are returned.

Here is how AES-GCM and GHASH can be implemented in Python, using the AES implementation from pycryptodome (usual disclaimer: this code is for educational purposes, and it’s not necessarily secure or optimized for performance):

from Crypto.Cipher import AES

def multiply(a: int, b: int) -> int:
    """
    Compute a*b in the GCM field, where a and b are two integers representing
    elements of the GCM field.

    a and b are expected to have their least significant bit set to the highest
    power monomial. For example, the polynomial x^125 is represented as 0b100.
    """
    bit_length = 128
    q = 0xe1000000000000000000000000000000
    assert a >= 0 and a < (1 << bit_length)
    assert b >= 0 and b < (1 << bit_length)

    result = 0
    for i in range(bit_length):
        result ^= a * ((b >> 127) & 1)
        a = (a >> 1) ^ (q * (a & 1))
        b <<= 1
    return result

def pad_block(data: bytes) -> bytes:
    """
    Pad data with zero bytes so that the resulting length is a multiple of 16
    bytes (128 bits).
    """
    if len(data) % 16 != 0:
        data += b'\0' * (16 - len(data) % 16)
    return data

def iter_blocks_padded(data: bytes):
    """
    Split the given data into blocks of 16 bytes (128 bits) each, padding the
    last block with zeros if necessary.
    """
    start = 0
    while start < len(data):
        yield pad_block(data[start:start+16])
        start += 16

def ghash(subkey: bytes, message: bytes) -> bytes:
    subkey = int.from_bytes(subkey, 'big')
    assert subkey < (1 << 128)

    state = 0
    for block in iter_blocks_padded(message):
        block = int.from_bytes(block, 'big')
        state = multiply(state ^ block, subkey)

    return state.to_bytes(16, 'big')

def aes_gcm_encrypt(key: bytes, nonce: bytes, message: bytes):
    assert len(key) in (16, 24, 32)
    assert len(nonce) == 12

    # Initialize a raw AES instance and encrypt a 16-byte block of all zeros to
    # derive the GHASH subkey H
    cipher = AES.new(mode=AES.MODE_ECB, key=key)
    h = cipher.encrypt(b'\0' * 16)

    # Encrypt the message with AES in CTR mode, with the counter composed by
    # the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32 bits)
    # integer, starting from 2
    cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=2)
    ciphertext = cipher.encrypt(message)

    # Compute the GHASH of the ciphertext plus the ciphertext length in bits
    s = ghash(h, pad_block(ciphertext) + (len(ciphertext) * 8).to_bytes(16, 'big'))
    # Encrypt the GHASH value using AES in CTR mode, with the counter composed
    # by the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32
    # bits) integer set at 1. The GHASH value fits in one block, so the counter
    # won't be increased during this round of encryption
    cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=1)
    tag = cipher.encrypt(s)

    return (ciphertext, tag)

def aes_gcm_decrypt(key: bytes, nonce: bytes, ciphertext: bytes, tag: bytes):
    assert len(key) in (16, 24, 32)
    assert len(nonce) == 12
    assert len(tag) == 16

    # Compute the GHASH subkey, the GHASH value, and the authentication tag, in
    # the same exact way as it was done during encryption
    cipher = AES.new(mode=AES.MODE_ECB, key=key)
    h = cipher.encrypt(b'\0' * 16)

    s = ghash(h, pad_block(ciphertext) + (len(ciphertext) * 8).to_bytes(16, 'big'))
    cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=1)
    expected_tag = cipher.encrypt(s)

    # Compare the input tag with the generated tag. If they're different, the
    # plaintext must not be returned to the caller
    if tag != expected_tag:
        raise ValueError('authentication failed')

    # The two tags match; decrypt the plaintext and return it to the caller.
    # Note that, because AES-CTR is a symmetric cipher, there is no difference
    # between the encrypt and decrypt method: here we are reusing the same
    # exact code used during decryption
    cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=2)
    message = cipher.encrypt(ciphertext)

    return message

And here is how the code can be used:

key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef')
nonce = bytes.fromhex('0123456789abcdef01234567')
message = b'I went to the zoo yesterday but not today'

ciphertext, tag = aes_gcm_encrypt(key, nonce, message)
print(f'ciphertext: {ciphertext.hex()}')
print(f'       tag: {tag.hex()}')
decrypted_message = aes_gcm_decrypt(key, nonce, ciphertext, tag)
print(f' plaintext: {decrypted_message}')
assert message == decrypted_message

This snippet produces the following output:

ciphertext: e0c32db2962f9b729c69028d9a1fdfb2c93839fc1188f314c58ee97fd6a242404953bb208df609a33c
       tag: 9fa6fe2f77a0c98282868924ace0e4ec
 plaintext: b'I went to the zoo yesterday but not today'

This is the same output we would obtain by using the AES-GCM implementation from pycryptodome directly:

from Crypto.Cipher import AES

cipher = AES.new(mode=AES.MODE_GCM, key=key, nonce=nonce)
ciphertext, tag = cipher.encrypt_and_digest(message)
print(f'ciphertext: {ciphertext.hex()}')
print(f'       tag: {tag.hex()}')

Nonce reuse is catastrophic for AES-GCM in two ways:

Because the ciphertext produced by AES-GCM is just a variant of AES-CTR, nonce reuse with GCM can have the same consequences as nonce reuse with AES-CTR, or any other stream cipher: if someone is able to guess the plaintext, they can recover the random stream, and use that to decrypt other messages (or portions of them).
If the same nonce is used twice or more, the GHASH subkey $H$ will always be the same. Even if the output of GHASH is encrypted in step 7, we can use the XOR of two authentication tags to “cancel” the encryption and obtain a polynomial in $H$. From there, we can use algebraic methods to recover $H$. This gives us the ability to forge new, valid authentication tags.

It’s worth mentioning that there’s a variant of AES-GCM, called AES-GCM-SIV, (Synthetic Initialization Vector) specified in RFC 8452. This differs from AES-GCM in that it uses a little-endian version of GHASH called POLYVAL (which is faster on modern CPUs), and in that it allows nonce reuse without the two catastrophic consequences that I mentioned above.

(Nonce reuse with AES-GCM-SIV however still presents a problem, just not as serious as the two ones above: specifically, it breaks ciphertext indistinguishability.)

Authenticated Encryption with Associated Data (AEAD)

The way I have described authenticated encryption, and in particular the constructions ChaCha20-Poly1305 and AES-GCM, is accurate, but incomplete. What I have told you is that when you use an authenticated encryption cipher, the ciphertext is checked for integrity and authenticity. But we can use the same technique to authenticate anything, not just ciphertexts: we can, for example, authenticate some plaintext data, or authenticate a piece of plaintext data and a piece of ciphertext altogether.

When we use a method to authenticate a plaintext message only, what we get is a Message Authentication Code (MAC). We don’t use the word “encryption” in this context, because the confidentiality of the message is not ensured (only its authenticity).

When we use a method to authenticate both a ciphertext and a plaintext message, what we get is Authenticated Encryption with Associated Data (AEAD). In this construction, there are two messages involved: one to be encrypted (resulting in a ciphertext), and one to be kept in plaintext. The plaintext message is called “associated data” (AD) or “additional authenticated data” (AAD). Both the ciphertext and the associated data are authenticated at encryption time, so their integrity and authenticity will be enforced.

The inputs to the encryption function of an AEAD cipher are, generally speaking:

a key;
a nonce;
the additional data;
the message to encrypt.

The outputs of the encryption are:

the ciphertext;
the authentication tag.

Note that there’s only one authentication tag that covers both the additional data and the ciphertext.

The inputs to the decryption function are:

the key used for encryption;
the nonce used for encryption;
the additional data used for encryption;
the ciphertext.

And the output of the decryption is either an error or the decrypted message.

It’s important to note that the associated data must be both at encryption time and decryption time. Changing a single bit of it will make the entire decryption operation fail.

Both ChaCha20-Poly1305 and AES-GCM (and their variants, XChaCha20-Poly1305 and AES-GCM-SIV) are AEAD ciphers. Here’s how they implement AEAD:

When the Poly1305 or GHASH authenticator is first initialized, they are fed the additional data, padded with zeros to make its size a multiple of 16 bytes (128 bits).
Then the padded ciphertext is fed into the authenticator.
The length of the additional data and the length of the ciphertext are represented as two 64-bit integers, concatenated, and fed into the authenticator.

Diagram of data flow during encryption with ChaCha20-Poly1305, including the Associated Data (AE)

Diagram of data flow during encryption with AES-GCM, including the Associated Data (AE)

If the additional data is empty, then what you get are exactly the constructions that I described earlier in this article.

Authenticated Encryption with Associated Data is useful in situations where you want to encode some metadata along with your encrypted data. For example: an identifier for the resource that is encrypted, or the type of data encrypted (text, image, video, …), or some information that indicates what key and algorithm was used to encrypt the resource, or maybe the expiration of the data. The associated data is in plaintext so systems that do not have access to the secret key can gather some properties about the encrypted resource. It must however be understood that the associated data cannot be trusted until it’s verified using the secret key. Systems that analyze the associated data must be designed in such a way that, if the associated data is tampered, nothing bad will happen, and such tampering attempt will be detected sooner or later.

A word of caution

Something very important to understand is that when using authenticated encryption ciphers like ChaCha20-Poly1305 or AES-GCM, decryption can in theory succeed even if the verification of the authentication tag fails.

For example, we can decrypt a ciphertext encrypted with ChaCha20-Poly1305 by using ChaCha20 and ignoring the authentication tag. Similarly, we can decrypt a ciphertext encrypted with AES-GCM by using AES-CTR and, again, ignoring the authentication tag. This possibility opens the doors to all the nasty scenarios that we have seen at the beginning of this article, removing all the benefits of authenticated encryption.

Perhaps the most important thing to remember when using authenticated encryption is: never use decrypted data until you have verified its authenticity.

Why am I emphasizing this? Because some AE or AEAD implementations do return plaintext bytes before verifying their authenticity.

The code samples that I have provided do the following: they first calculate the authentication tag, compare it to the input tag, and only if the comparison succeeds they perform the decryption. This is a simple approach, but it may be expensive when encrypting large amounts of data (for example: several gigabytes). The reason why this approach is expensive is that, if the ciphertext is too large, it may not fit all in memory, and the ciphertext would have to be read from the storage device twice: once for calculating the tag, and once for decrypting the ciphertext. Also, chances are that by the time the application has computed the tag, the underlying ciphertext may have changed without detection.

What some authenticated encryption implementations do when dealing with large amounts of data is that they calculate the tag and perform the decryption in parallel. They read the ciphertext chunk-by-chunk, and pass each chunk to both the authenticator and the decryption function, returning a chunk of decrypted bytes to the caller at each iteration. Only at the end, when the full ciphertext has been read, the authenticity is checked, and the application may return an error only at that point. With such implementations, it is imperative that the exit status of the application is checked before using any of the decrypted bytes.

An implementation that works like that (returning decrypted bytes before authentication is complete) is GPG. Here is an example of the output that GPG produces when decrypting a tampered message:

gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
This is a very long message.
gpg: WARNING: encrypted message has been manipulated!

The decrypted message (“This is a very long message”) got printed, together with a warning, and the exit status is 2, indicating that an error occurred. It is important in this case that the decrypted message is not used in any way.

Other implementations avoid this problem by simply not encrypting large amounts of data. If given a large file to encrypt, the file is first split into multiple chunks of a few KiB, then each chunk is encrypted independently, with its own nonce and authentication tag. Because each chunk is small, authentication and decryption can happen in memory, one before the other. If a chunk was tampered, decryption would stop, returning truncated output, but never tampered output. It’s still important to check the exit status of such an implementation, but the consequences are less catastrophic than before. The drawback of this approach is that the total size of the ciphertext increases, because each chunk requires a nonce, an authentication tag, and some information about the position of the chunk (to prevent the chunks from being reordered). Storing the nonces or the positions can be avoided by using an algorithm to generate them on the fly, but storing the tag cannot be avoided.

The method of splitting that I have just described (of splitting long messages into chunks that are individually encrypted and authenticated) is used for example in TLS, as well as the command line tool AGE.

Summary and final considerations

At the beginning of this article we have seen some risks of using bare encryption ciphers: one of them in particular was malleability, that is: the property that ciphertexts may be modified without detection.

This problem was addressed by using Authenticated Encryption (AE) or Authenticated Encryption with Associated Data (AEAD), which are methods to provide integrity and authenticity in addition to confidentiality when encrypting data.

We have seen the details of the two most popular authenticated encryption ciphers and briefly mentioned some of their variants. Their features are summarized here:

Cipher	Cipher Type	Key Size	Nonce Size	Nonce Reuse	Tag Size
ChaCha20-Poly1305	Stream, AEAD	256 bits	96 bits	Catastrophic	128 bits
XChaCha20-Poly1305	Stream, AEAD	256 bits	192 bits	Catastrophic	128 bits
AES-GCM	Stream, AEAD	128, 192, 256 bits	96 bits	Catastrophic	128 bits
AES-GCM-SIV	Stream, AEAD	128 or 256 bits	96 bits	Reduced risk	128 bits

Authenticated encryption is used in most of our modern protocols, including TLS, S/MIME, PGP/GPG, and many more. Failure to implement authenticated encryption correctly has lead to some serious issues in the past.

Whenever you’re using encryption, ask yourself: how is integrity and authentication verified? And remember: it’s essential to verify the authenticity of data before using it.

I hope you enjoyed this article! As usual, if you have any suggestions or spotted some mistakes, let me know in the comments or by contacting me!

on March 09, 2023 06:35 PM

A brief history of my experience with Linux

Aaron Rainbolt

From an early age, I’ve been very interested in computers. They were fascinating and profoundly powerful devices, capable of doing things that were (and in some ways still are) beyond my understanding. Like many of us, I started on Windows, and like many of us who started out computing when we were kids, I did a lot of damage to my first computer. I still haven’t grown out of breaking computers, I’ve just gotten a lot better at planning how I’m going to break them and what to do in order to fix them after the fact.

Fast forward several years. I’ve heard of Linux, read books about how to do some things like web development on Linux, and am thinking about trying Linux. I wanted to do music recording at the time too (something I’m still planning on doing). I didn’t have a working Internet connection, so I got in the car with my friend and went to a place that had Internet in order to download some Linux distros to try out.

The first distro I ever used was HP QuickWeb (Splashtop OS), which came with my Windows 7 laptop. I knew nothing about Linux at the time, I just found this QuickWeb thing included with my computer and decided to try it out in order to keep things isolated from my Windows installation. (In retrospect, this probably didn’t work at all, but I didn’t know that…) Basically it was nothing more than a Linux kernel, a simple and glitchy desktop, and a severely outdated copy of Firefox that I didn’t know better than to use. From this insecure and buggy mess, my journey into Linux began.

After fighting with QuickWeb’s browser for a bit, I learned about a distro called KXStudio, which was supposed to be geared towards music production. (KXStudio has since switched from being a distro to being a third-party repo that you use on top of Kubuntu, which severely confused me, since I didn’t know the difference between a repo and a distro at the time.) After a lot of searching, I managed to find KXStudio 14.04, from back when KXStudio was still a distro. I downloaded it, along with a few other distros that I ultimately didn’t end up getting to work, and also grabbed VirtualBox so that I could run the distros in a VM.

Let’s just say that combining a buggy distro with a total newbie resulted in chaos.

I went through quite a few “stages” learning how Linux worked, everything from “Where on earth is my C: drive?” to “Great, I just broke my audio, now what?”. It was pretty interesting. I also had to deal with the glitchiness and complexity of VirtualBox, as well as the fact that the distro I was using was not exactly newbie-friendly. The whole thing was a total mess. A very fun total mess. Before long, I had decided to install Linux on some physical hardware that I had available (which led to even more chaos, especially when UEFI came into the picture), and after days of headaches, many reinstalls, and tons of breakage, I was finally getting good enough at Linux usage to start using it as my daily driver on three different computers (a desktop and two laptops).

My third distro was ChaletOS 16.04. ChaletOS was essentially a modded Xubuntu variant made to look and feel a lot like Windows. It also has the Wine Windows Compatibility Layer preinstalled, making it compatible with a lot of Windows software out of the box. I didn’t use Wine very much - I was more interested in the beautiful UI, mostly smooth operation, and powerful features it provided. I was starting to get into more advanced Linux usage, and was for the first time using a sorta-halfway supported OS (all the ones I had used up until then were already EOL or very nearly EOL, if I remember correctly). I also had a working Internet connection, which greatly improved my ability to download software and research things. At this point I was almost entirely “sold” on Linux, and was only still using Windows on one laptop. That one laptop was important because it had a development environment on it that ran on Windows 7 inside a Hyper-V virtual machine inside Windows 8.0 Pro. I was reluctant to switch to Linux there because I knew Hyper-V wouldn’t run on Linux and I didn’t want to mess up the Windows 7 VM’s activation status by moving from Hyper-V to QEMU.

While using ChaletOS on my other laptop (not the one with the important development environment), I tried a bunch of different distros in VMs, including really weird ones like BasicLinux (don’t ever use it, it’s a nightmare) and Calculate Linux (Gentoo derivative, that was a total fail because hilariously I couldn’t figure out what the package manager’s name was). And then, in the midst of my distro-hopping adventures, I stumbled across two distros that almost immediately became my favorites. It was around the time of April 2020. Ubuntu 20.04 and its flavors has just been released recently. And it was during that time I discovered Kubuntu and Lubuntu.

I can’t remember which one I tried first, but I remember quite well some of my initial experiments with them. After playing with both Kubuntu and Lubuntu in VMs, I decided to install Lubuntu onto a flash drive and boot my non-development laptop from it. The extremely smooth experience Lubuntu 20.04 provided, the beautiful and simple UI, everything was almost perfect. Kubuntu was really nice, but it was a bit heavier and the laptop I was using for testing only had 4 GB RAM and no internal hard drive (it always booted from a USB stick, or from an SD card using a hack), so a lightweight OS was important to me. I used that OS a lot, and even booted my important development laptop from my Lubuntu USB stick to see what would happen (it worked quite well).

While I ended up loving Lubuntu, I was somewhat partial to Kubuntu since it was the basis of KXStudio, and the extreme power of the KDE desktop was a serious plus that Lubuntu’s LXQt desktop didn’t have. I had ended up having a slight mess with my Windows 7 VM on the development laptop which resulted in me having to re-activate it (long story short, don’t expect to restore a Hyper-V VM from a backup of only the .vhd files without the rest of the VM config!), and as a result, I was no longer so scared of having to move my VM to Linux using QEMU. I took the plunge and wiped Windows 8 with Kubuntu 20.04 on my main development laptop, then got my Windows 7 VM set up inside QEMU (where it actually worked better than it did on Hyper-V). Everything was working exactly as I had hoped (except for one small hiccup that resulted in an OS reinstall due to a combination of inexperience and paranoia, but that was overcome without too much trouble in the long run).

After a long bit of life lived in 20.04-land (and a brief stint with Lubuntu and Ubuntu Studio 21.10), Ubuntu 22.04 was released. I had finally mostly gotten a grip on how Ubuntu and Linux really worked, and was interested in trying to help with Ubuntu’s development. I had looked into helping in the past, but hadn’t actually gone ahead and tried to help. I joined the #lubuntu-devel channel with the nick “Guest3” and asked about the possibility of writing software documentation for Lubuntu. I surprisingly got replied to almost immediately by two members of the Lubuntu team, switched my IRC nick from Guest3 to arraybolt3, and before long was helping with testing, documentation writing, and debugging. (It all just kinda happened - I was there and stuff was happening and I ended up getting to try to help and it all worked really well.) I ended up helping with the development of Lubuntu and a couple of the other Ubuntu flavors during the Kinetic Kudu development cycle and the 22.04.1 release, and six hectic months of excitement later, Ubuntu 22.10 and its flavors were released.

That first step into the Ubuntu development world was about 10 months ago. We are now nearing the release of Ubuntu 23.04, and have since released 20.04.5 and 22.04.2. The joy of getting to contribute to an OS I’ve been using for years has been amazing, and I look forward to continuing to contribute in the future.

That was about four years of computing packed into a short article. Hope you found this enjoyable and helpful, and see you later!

on March 09, 2023 11:21 AM

March 07, 2023

It’s like…

Serge Hallyn

Let’s say you had one of those huge keyrings, with 50 keys on it. (Let’s say 41 keys – number chosen either randomly or bc its the current number of capabilities) On the plus side, when you want to let someone use one, you can magically, instantly make a copy to give to that person. On the down side, that person can do the same (give a copy to *anyone*), once you hand it to them. Further on the down side, well, you have 41 keys.

Now if you want to send the valet to take your car out, you have to give him a key to the elevator, key to the garage entry door, key to the car door, key to the car ignition, and a key to the garage-exit. You’ve done this yourself plenty so you know what keys to give.

But other parts of the building were designed by other people, who have placed gates in various places you’ve never visited, and know nothing about. So now you need the maid to go clean the fifth bedroom. How do you know what keys to give her?

You can start her on her way with the key to the elevator and the key to the bedroom. But she needed a key to pass a gate in the first hallway. So she comes back… all she knows to say is EPERM! She can’t even tell you where the gate is. So you walk with her to the first gate (strace), see the gate, figure out which key it needs, give it to her. Send her on her way.

Of course she hits another gate, comes back, and says EPERM!

Now, you’re not a quitter. And this is important. So you’ll keep trying until you figure out all the keys she needs. Unfortunately, there are 30 other people – chauffeurs, mechanics, janitors, butlers, maids, some delivery people, a gardener… They all need access to various places.

It gets worse. Some gates are only locked at certain times. So you think you’ve gotten them sorted out, but at 2am it turns out half of them are newly blocked. Oh, and next week the person in charge of the second floor decides a few more gates are really needed. Hopefully their newsletter will tell you where, and which keys they’ll use – but they’re busy, so probably not.

So, sure, you’re not usually a quitter, but it sure will be temping to give the full 41 keys to anyone who comes by.

on March 07, 2023 09:46 PM

March 06, 2023

Kubuntu Manual 22.04.2 Release

Kubuntu General News

Hello everyone! It’s a great day with a new release of the Kubuntu Manual to match the recently-released Kubuntu 22.04.2 update. Thank you for the community members who provided feedback and filed bug reports to our GitHub project! You can find the new releases either on the GitHub releases page or our support page.

on March 06, 2023 03:47 PM

March 04, 2023

Xubuntu Minimal Visual Tour

Sean Davis

Xubuntu Minimal provides a lighter-weight alternative to the full-size Xubuntu desktop. The core desktop experience allows you to build your very own version of Xubuntu with the tricky configuration already done for you. Scroll through the screenshots below to see Xubuntu Minimal in action. When you&aposre done, download Xubuntu Minimal from cdimage.ubuntu.com to test it for yourself!

on March 04, 2023 11:14 AM

March 03, 2023

Xubuntu Development Update March 2023

Sean Davis

Xubuntu Minimal has arrived for Xubuntu 23.04

After seven long years, Xubuntu Minimal (formerly Xubuntu Core), has become an official sub-project built and published on Canonical infrastructure. 🎉️

Introducing Xubuntu core « Xubuntu

Xubuntu – Xubuntu is an elegant and easy-to-use operating system.

The Xubuntu community

Xubuntu Core was first announced way back in May, 2015. That was a simpler time.

Xubuntu Minimal is a slimmed down version of Xubuntu that doesn&apost come with the additional features of a modern desktop operating system. You get the basics: Xfce, Xubuntu defaults, a file manager, and a few other essential components. Build your own experience atop Xubuntu&aposs stable and tested minimal environment.

Xubuntu Minimal currently clocks in at 1.7 GiB (1.8 GB), while the standard Xubuntu desktop image is 2.8 GiB (3.0 GB). When we started this project over seven years ago, the image fit on a standard CD-ROM at less than 700 MB. Since then, everything has gotten bigger and reducing the image size will require some clever adjustments. If you&aposd like to check Xubuntu Minimal out, daily ISO builds can be found on cdimage.ubuntu.com.

⚠️

There is a known issue with both the Xubuntu desktop and minimal images where default settings are not being correctly loaded. This should be resolved in tomorrow&aposs daily builds (March 4, 2023).

Flatpak removed for Xubuntu 23.04

Following a joint decision between the Ubuntu flavors, no official flavor will include Flatpak integration in their installation starting with the upcoming April release, 23.04 "Lunar Lobster". 23.04 would have been the first Xubuntu release to ship with Flatpak integration, so this change will not affect any user. Anybody that is already using Flatpak will not be affected either, as the package will not be uninstalled if you have interacted with Flatpak previously.

If you want to enable Flatpak in Xubuntu, the process is straightforward. Check out my recent article, Enable Flatpak in Xubuntu, for instructions on enabling Flatpak and adding the Flathub repository to your system.

Xubuntu is now on Mastodon

Have your social networks felt a little too... centralized, lately? Come to Mastodon and enjoy interacting with Xubuntu&aposs new social media presence, @xubuntu@floss.social. While you&aposre at it, you might also be interested in following Xfce (@xfce@floss.social), Lubuntu (@LubuntuOfficial@fosstodon.org), Kubuntu (@kubuntu@fosstodon.org), or Ubuntu Studio (@ubuntustudio@mastodon.art), to get you started with a few projects.

Package updates

There was a smattering of relevant package updates in Xubuntu 23.04 this month. You can find the full list of package updates on GitHub by comparing commits to the xubuntu-manifest project.

baobab 43.0 > 44 beta
gnome-disk-utility 43.0 > 44 Beta
gnome-font-viewer 43.0 > 44 Beta
gnome-sudoku 43.1 > 44 Beta
libreoffice 7.4.3 > 7.5.1 RC2
libxfce4ui 4.18.1 > 4.18.2
parole 4.16.0 > 4.18.0
synaptic 0.91.2 > 0.91.3
thunar 4.18.3 > 4.18.4
xfce4-notifyd 0.7.2 > 0.7.3
xfce4-panel 4.18.1 > 4.18.2
xfce4-power-manager 4.18.0 > 4.18.1
xfce4-session 4.18.0 > 4.18.1
xfce4-settings 4.18.0 > 4.18.2
xfce4-systemload-plugin 1.3.1 > 1.3.2
xubuntu-desktop 2.246 > 2.248

Thanks!

That&aposs all from me this month! If you&aposve come this far, you&aposre looking for something to do. Why not join in to the Ubuntu 23.04 Testing Week that&aposs going on now?

If you like the work that I do, considering sponsoring me. You can find links for how to support me and contribute to Xfce, Xubuntu, and elementary OS at bluesabre.org/donate. Thank you!

on March 03, 2023 12:44 PM

March 01, 2023

Plasma 5.27.2 for Kubuntu 22.10 available via PPA

Kubuntu General News

We have had many requests to make Plasma 5.27.2 available in our backports PPA for Kubuntu 22.10 (Kinetic Kudu).

As usual with our PPAs, there is the caveat that the PPA may receive additional updates and new releases of KDE Plasma, Gear (Apps), and Frameworks, plus other apps and required libraries. Users should always review proposed updates to decide whether they wish to receive them.

To add the PPA and upgrade, do:

sudo add-apt-repository ppa:kubuntu-ppa/backports && sudo apt full-upgrade -y

We hope keen adopters enjoy using Plasma 5.27.2!

Note that Plasma 5.27 is also available for our current development version, Lunar Lobster, to become Kubuntu 23.04.

on March 01, 2023 09:10 PM

February 28, 2023

FOSS Activites in February 2023

Utkarsh Gupta

Here’s my (forty-first) monthly but brief update about the activities I’ve done in the F/L/OSS world.

Debian

This was my 50th month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas ‘19! \o/

There’s a bunch of things I do, both, technical and non-technical. Here are the things I did this month:

Uploads

ruby-delayed-job (4.1.9-1~bpo11+1) - Backport to bullseye.
ruby-delayed-job-active-record (4.1.6-3~bpo11+1) - Backport to bullseye.
ruby-globalid (0.6.0-1~bpo11+1) - Backport to bullseye.
ruby-tzinfo (2.0.4-1~bpo11+2) - Backport to bullseye.
rails (2:6.1.7+dfsg-3~bpo11+1) - Backport to bullseye.
ruby-commonmarker (0.23.6-1~bpo11+1) - Backport to bullseye.
ruby-csv (3.2.2-1~bpo11+1) - Backport to bullseye.
ruby-task-list (2.3.2-2~bpo11+1) - Backport to bullseye.
ruby-i18n (1.10.0-2~bpo11+1) - Backport to bullseye.
ruby-mini-magick (4.11.0-1~bpo11+1) - Backport to bullseye.
ruby-net-ldap (0.17.0-1~bpo11+1) - Backport to bullseye.
ruby-roadie-rails (3.0.0-1~bpo11+1) - Backport to bullseye.
ruby-roadie (5.1.0-1~bpo11+1) - Backport to bullseye.
ruby-sanitize (6.0.0-1~bpo11+1) - Backport to bullseye.
ruby-nokogiri (1.13.1+dfsg-2~bpo11+1) - Backport to bullseye.
ruby-mini-portile2 (2.8.0-1~bpo11+2) - Backport to bullseye.
ruby-webrick (1.7.0-3~bpo11+2) - Backport to bullseye.
ruby-zip (2.3.0-2~bpo11+1) - Backport to bullseye.
gem2deb (2.1~bpo11+1) - Backport to bullseye.
ruby-actionpack-action-caching (1.2.2-1~bpo11+1) - Backport to bullseye.
ruby-nokogiri (1.13.5+dfsg-2~bpo11+1) - Backport to bullseye.
redmine (5.0.4-2~bpo11+1) - Backport to bullseye.
rails (2:6.1.7+dfsg-3~bpo11+2) - Backport to bullseye.
ruby-roadie-rails (3.0.0-1~bpo11+2) - Backport to bullseye.
redmine (5.0.4-4~bpo11+1) - Backport to bullseye.
redmine (5.0.4-5~bpo11+1) - Backport to bullseye.
ruby-web-console (4.2.0-1~bpo11+1) - Backport to bullseye.
libyang2 (2.1.30-2) - Adding DEP8 test for yangre.
redmine (5.0.4-3) - Add patch to stop unnecessary recursive chown’ing (Fixes: #1022816, #1022817).
redmine (5.0.4-4) - Set DH_RUBY_IGNORE_TESTS to all (Fixes: #1031308).
python-jira (3.4.1-1) - New upstream version, v3.4.1.

Others

Looked up some Release team documentation.
Sponsored php-font-lib and php-dompdf-svg-lib for William.
Granted DM rights for php-dompdf.
Mentoring for newcomers.
Reviewed micro bits for Nilesh, new uploads and changes.
Ruby sprints.
Bug work (on BTS and #debian-ruby) for rails and redmine.
Moderation of -project mailing list.

A huge thanks to Freexian for sponsoring my Debian work and Entrouvert for sponsoring the Redmine backports. :D

Ubuntu

This was my 25th month of actively contributing to Ubuntu. Now that I joined Canonical to work on Ubuntu full-time, there’s a bunch of things I do! \o/

I mostly worked on different things, I guess.

I was too lazy to maintain a list of things I worked on so there’s no concrete list atm. Maybe I’ll get back to this section later or will start to list stuff from the fall, as I was doing before. :D

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.

And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support).

This was my forty-first month as a Debian LTS and thirty-second month as a Debian ELTS paid contributor.
I worked for 24.25 hours for LTS and 28.50 hours for ELTS.

LTS CVE Fixes and Announcements:

Fixed CVE-2022-47016 for tmux and uploaded to buster via 2.8-3+deb10u1.
But decided to not roll the DLA for the package as the CVE got rejected upstream.
Issued DLA 3359-1, fixing CVE-2019-13038 and CVE-2021-3639, for libapache2-mod-auth-mellon.
For Debian 10 buster, these problems have been fixed in version 0.14.2-1+deb10u1.
Issued DLA 3360-1, fixing CVE-2021-30151 and CVE-2022-23837, for ruby-sidekiq.
For Debian 10 buster, these problems have been fixed in version 5.2.3+dfsg-1+deb10u1.
Worked on ruby-rails-html-sanitize and added notes to the security-tracker.
TL;DR: we need newer methods in ruby-loofah to make the patches for ruby-rails-html-sanitize backportable.
Started to look at other set of packages meanwhile.

ELTS CVE Fixes and Announcements:

Issued ELA 813-1, fixing CVE-2017-12618 and CVE-2022-25147, for apr-util.
For Debian 8 jessie, these problems have been fixed in version 1.5.4-1+deb8u1.
For Debian 9 stretch, these problems have been fixed in version 1.5.4-3+deb9u1.
Issued ELA 814-1, fixing CVE-2022-39286, for jupyter-core.
For Debian 9 stretch, these problems have been fixed in version 4.2.1-1+deb9u1.
Issued ELA 815-1, fixing CVE-2022-44792 and CVE-2022-44793, for net-snmp.
For Debian 8 jessie, these problems have been fixed in version 5.7.2.1+dfsg-1+deb8u6.
For Debian 9 stretch, these problems have been fixed in version 5.7.3+dfsg-1.7+deb9u5.
Helped facilitate RabbitMQ’s update queries by one of our customers.
Started to look at other set of packages meanwhile.

Other (E)LTS Work:

Triaged ruby-loofah, ruby-sinatra, tmux, ruby-sidekiq, libapache2-mod-auth-mellon, jupyter-core, net-snmp, and apr-util, rabbitmq-server.
Helped and assisted new contributors joining Freexian (LTS/ELTS/internally).
Answered questions (& discussions) on IRC (#debian-lts and #debian-elts) and Matrix.
Participated and helped fellow members with their queries via private mail and chat.
General and other discussions on LTS private and public mailing list.
Attended the monthly LTS meeting.

Until next time.
:wq for today.

on February 28, 2023 05:41 AM

February 24, 2023

Snowstorms, Kittens and Shattered dreams

Scarlett Gately Moore

Long ago I applied for my dream job at a company I have wanted to wok for since its beginning and I wasn’t ready technically. Fast forward to now, I am ready! A big thank you goes out to Blue Systems for that. So I go out and find the perfect role and start the application process. The process was months long, but was going very well, the interviews and I passed the technical with flying colors. I got to the end where the hiring lead told me he was submitting my offer… I was so excited, so much so, I told my husband and parents “I got the job!” I know, I jinxed myself there. Soon I receive the “There was a problem”.. One obscure assessment called GIA came back not so good. I remember that day, we were in the middle of a long series of winter storms and I when I took the test, my kitten decided right then it was me time. I couldn’t very well throw her out into the snowstorm, so I continued on the best I could. It is my fault, it clearly states to be distraction free. So I speak again to the hiring lead and we both feel with my experience and technical knowledge and abilities we can still move forward. I still had hope. After some time passes, I asked for an update and got the dreaded rejection. I am told it wasn’t just the GIA, but that I am not a good overall fit for the company. In one fell swoop my dreams are dashed and final, for this and all roles within that company. I wasn’t given a reason either. I am devastated, heart broken, and shocked. I get along with everyone, I exceed the technical requirements, and I work well in the community. Dream door closed.

I will not let this get me down. I am moving on. I will find my place where I ‘fit in’.

With that said, I no longer have the will, passion, or drive to work on snaps anymore. I will leave instructions with Jonathon as to what needs to be done to move forward. The good news is my core22 kde-neon extension was merged into upstream snapcraft, so whomever takes over will have a much easier time knocking them out. @kubuntu-council I will do whatever it takes to pay back the money for the hardware you provided me to do snaps, I am truly sorry about this.

What does my future hold? I will still continue with my Debian efforts. In fact, I have ventured out from the KDE umbrella and joined the go-team. I am finalizing my packaging for

https://github.com/charmbracelet/gum

and it’s dependencies: roff, mango, mango-kong. I had my first golang patch for a failing test and have submitted it upstream. I will upload these to experimental while the freeze is on.

I will be moving all the libraries in the mycroft team to the python umbrella as they are useful for other things and mycroft is no more.

During the holidays I was tinkering around with selenium UI testing and stumbled on some accessibility issues within KDE, so I think this is a good place for me to dive into for my KDE contributions.

I have been approached to collaborate with OpenOS on a few things, time permitting I will see what I can do there.

I have a possible gig to do some websites, while I move forward in my job hunt.

I will not give up! I will find my place where I ‘fit in’.

Meanwhile, I must ask for donations to get us by. Anything helps, thank you for your consideration.

https://gofund.me/a9c36b87

on February 24, 2023 02:12 PM

Lubuntu 22.04.2 LTS is Released!

Lubuntu Blog

Thanks to all the hard work from our contributors, Lubuntu 22.04.2 LTS has been released. With the codename Jammy Jellyfish, Lubuntu 22.04 is the 22nd release of Lubuntu, the eighth release of Lubuntu with LXQt as the default desktop environment. Support lifespan Lubuntu 22.04 LTS will be supported for 3 years until April 2025. Our […]

on February 24, 2023 12:37 AM

February 20, 2023

New hotcorners feature introduced in 23.04

David Mohammed

With the beta release of 23.04 around the corner, it seems a good moment for another introduction to one of the innovations for Lunar; a new implementation of the hotcorners feature. Hotcorners was completely rewritten and redesigned, to replace the current applet. There are significant differences, both from the user’s perspective and from a technical point of view. What stayed the same is that...

Source

on February 20, 2023 06:27 PM

February 17, 2023

ucaresystem core 4.4.0 : Now published for Ubuntu Kinetik and Lunar

Salih Emin

The new release of ucaresystem core is released and available for Ubuntu Kinetik (22.10) and Ubuntu Lunar (23.04) The all time classic uCareSystem is now release for Ubuntu Kinetik (22.10) and Ubuntu Lunar (23.04). Also, in the same repository you will find an app that you can use to quickly fetch info about your OSContinue reading

on February 17, 2023 04:24 PM

February 15, 2023

Netplan v0.106 is now available

Lukas Märdian

I’m happy to announce that Netplan version 0.106 is now available on GitHub and is soon to be deployed into an Ubuntu/Debian/Fedora installation near you! Six months and 65 commits after the previous version, this release is brought to you by 4 free software contributors from around the globe.

Highlights

Highlights of this release include the new netplan status command, which queries your system for IP addresses, routes, DNS information, etc… in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine readable YAML/JSON format).

Furthermore, we implemented a clean libnetplan API which can be used by external tools to parse Netplan configuration, migrated away from non-inclusive language (PR#303) and improved the overall Netplan documentation. Another change that should be noted, is that the match.macaddress stanza now only matches on PermanentMACAddress= on the systemd-networkd backend, as has been the case on the NetworkManager backend ever since (see PR#278 for background information on this slight change in behavior).

Changelog

New ‘netplan status’ CLI by @slyon in #290
API: implement APIs from the new specification by @daniloegea in #298
Check and fix non-inclusive language by @daniloegea in #303
Documentation improvements (using Diátaxis & RTD) by @slyon
Match by PermanentMACAddress by @rlaager in #278
Netplan api iterator by @daniloegea in #306
API: update netplan_delete_connection() to avoid spawning another process by @slyon in #322
NetworkManager 1.40 compat & file permission fixes (LP#1862600, LP#1997348) by @slyon in #300
Migrate from (deprecated) nose to pytest by @daniloegea in #302
parse: Add the filepath to OVS ports netdefs by @daniloegea in #295
Check if the interface name is too long (LP#1988749) by @daniloegea in #313
doc/examples: remove unnecessary route for IPv6 on-link gateways by @sbraz in #312
Memory leak CI action by @daniloegea in #321
tests:base:ethernets: Improve stability of autopkgtests by @slyon in #223

Bug fixes:

Fix some memory leaks by @daniloegea in #297
parser: plug a memory leak by @daniloegea in #309
src:parse: plug memory leaks in nullable handling by @slyon in #319
Fix ‘netplan ip leases’ crash (LP#1996941) by @daniloegea in #301
tests: mock calls to systemctl by @daniloegea in #314
ctests: fix an integer conversion issue by @daniloegea in #315
docs: small fix on netplan-set doc by @daniloegea in #316
parser: return the correct error on failure (LP#2000324) by @daniloegea in #308
apply: Fix crash when OVS is stopped (LP#1995598) by @daniloegea in #307
networkd: make sure VXLAN is in the right section (LP#2000713) by @daniloegea in #310
cli:set: update only specific origin-hint if given (LP#1997467) by @slyon in #299
vxlan: convert some settings to tristate (LP#2000712) by @daniloegea in #311
parser: check for route duplicates (LP#2003061) by @daniloegea in #320

on February 15, 2023 01:41 PM

February 10, 2023

Integer shift gotcha

Colin King

Left shifting values seems simple, but the following code contains a bug:

The literal value 1 is actually a signed int, so the promotion to a uint64_t type will sign extend the value before assigning to x, causing x to be 0xffffffff80000000.

The fix is simple, just make the literal value 1 an unsigned int using 1U instead of 1.

on February 10, 2023 11:27 AM

February 01, 2023

Ubuntu 2022v1 secure boot key rotation and friends

Julian Andres Klode

This is the story of the currently progressing changes to secure boot on Ubuntu and the history of how we got to where we are.

taking a step back: how does secure boot on Ubuntu work?

Booting on Ubuntu involves three components after the firmware:

shim
grub
linux

Each of these is a PE binary signed with a key. The shim is signed by Microsoft’s 3rd party key and embeds a self-signed Canonical CA certificate, and optionally a vendor dbx (a list of revoked certificates or binaries). grub and linux (and fwupd) are then signed by a certificate issued by that CA

In Ubuntu’s case, the CA certificate is sharded: Multiple people each have a part of the key and they need to meet to be able to combine it and sign things, such as new code signing certificates.

BootHole

When BootHole happened in 2020, travel was suspended and we hence could not rotate to a new signing certificate. So when it came to updating our shim for the CVEs, we had to revoke all previously signed kernels, grubs, shims, fwupds by their hashes.

This generated a very large vendor dbx which caused lots of issues as shim exported them to a UEFI variable, and not everyone had enough space for such large variables. Sigh.

We decided we want to rotate our signing key next time.

This was also when upstream added SBAT metadata to shim and grub. This gives a simple versioning scheme for security updates and easy revocation using a simple EFI variable that shim writes to and reads from.

Spring 2022 CVEs

We still were not ready for travel in 2021, but during BootHole we developed the SBAT mechanism, so one could revoke a grub or shim by setting a single EFI variable.

We actually missed rotating the shim this cycle as a new vulnerability was reported immediately after it, and we decided to hold on to it.

2022 key rotation and the fall CVEs

This caused some problems when the 2nd CVE round came, as we did not have a shim with the latest SBAT level, and neither did a lot of others, so we ended up deciding upstream to not bump the shim SBAT requirements just yet. Sigh.

Anyway, in October we were meeting again for the first time at a Canonical sprint, and the shardholders got together and created three new signing keys: 2022v1, 2022v2, and 2022v3. It took us until January before they were installed into the signing service and PPAs setup to sign with them.

We also submitted a shim 15.7 with the old keys revoked which came back at around the same time.

Now we were in a hurry. The 22.04.2 point release was scheduled for around middle of February, and we had nothing signed with the new keys yet, but our new shim which we need for the point release (so the point release media remains bootable after the next round of CVEs), required new keys.

So how do we ensure that users have kernels, grubs, and fwupd signed with the new key before we install the new shim?

upgrade ordering

grub and fwupd are simple cases: For grub, we depend on the new version. We decided to backport grub 2.06 to all releases (which moved focal and bionic up from 2.04), and kept the versioning of the -signed packages the same across all releases, so we were able to simply bump the Depends for grub to specify the new minimum version. For fwupd-efi, we added Breaks.

(Actually, we also had a backport of the CVEs for 2.04 based grub, and we did publish that for 20.04 signed with the old keys before backporting 2.06 to it.)

Kernels are a different story: There are about 60 kernels out there. My initial idea was that we could just add Breaks for all of them. So our meta package linux-image-generic which depends on linux-image-$(uname -r)-generic, we’d simply add Breaks: linux-image-generic (« 5.19.0-31) and then adjust those breaks for each series. This would have been super annoying, but ultimately I figured this would be the safest option. This however caused concern, because it could be that apt decides to remove the kernel metapackage.

I explored checking the kernels at runtime and aborting if we don’t have a trusted kernel in preinst. This ensures that if you try to upgrade shim without having a kernel, it would fail to install. But this ultimately has a couple of issues:

It aborts the entire transaction at that point, so users will be unable to run apt upgrade until they have a recent kernel.
We cannot even guarantee that a kernel would be unpacked first. So even if you got a new kernel, apt/dpkg might attempt to unpack it first and then the preinst would fail because no kernel is present yet.

Ultimately we believed the danger to be too large given that no kernels had yet been released to users. If we had kernels pushed out for 1-2 months already, this would have been a viable choice.

So in the end, I ended up modifying the shim packaging to install both the latest shim and the previous one, and an update-alternatives alternative to select between the two:

In it’s post-installation maintainer script, shim-signed checks whether all kernels with a version greater or equal to the running one are not revoked, and if so, it will setup the latest alternative with priority 100 and the previous with a priority of 50. If one or more of those kernels was signed with a revoked key, it will swap the priorities around, so that the previous version is preferred.

Now this is fairly static, and we do want you to switch to the latest shim eventually, so I also added hooks to the kernel install to trigger the shim-signed postinst script when a new kernel is being installed. It will then update the alternatives based on the current set of kernels, and if it now points to the latest shim, reinstall shim and grub to the ESP.

Ultimately this means that once you install your 2nd non-revoked kernel, or you install a non-revoked kernel and then reconfigure shim or the kernel, you will get the latest shim. When you install your first non-revoked kernel, your currently booted kernel is still revoked, so it’s not upgraded immediately. This has a benefit in that you will most likely have two kernels you can boot without disabling secure boot.

regressions

Of course, the first version I uploaded had still some remaining hardcoded “shimx64” in the scripts and so failed to install on arm64 where “shimaa64” is used. And if that were not enough, I also forgot to include support for gzip compressed kernels there. Sigh, I need better testing infrastructure to be able to easily run arm64 tests as well (I only tested the actual booting there, not the scripts).

shim-signed migrated to the release pocket in lunar fairly quickly, but this caused images to stop working, because the new shim was installed into images, but no kernel was available yet, so we had to demote it to proposed and block migration. Despite all the work done for end users, we need to be careful to roll this out for image building.

another grub update for OOM issues.

We had two grubs to release: First there was the security update for the recent set of CVEs, then there also was an OOM issue for large initrds which was blocking critical OEM work.

We fixed the OOM issue by cherry-picking all 2.12 memory management patches, as well as the red hat patches to the loader we take from there. This ended up a fairly large patch set and I was hesitant to tie the security update to that, so I ended up pushing the security update everywhere first, and then pushed the OOM fixes this week.

With the OOM patches, you should be able to boot initrds of between 400M and 1GB, it also depends on the memory layout of your machine and your screen resolution and background images. So OEM team had success testing 400MB irl, and I tested up to I think it was 1.2GB in qemu, I ran out of FAT space then and stopped going higher :D

other features in this round

Intel TDX support in grub and shim
Kernels are allocated as CODE now not DATA as per the upstream mm changes, might fix boot on X13s

am I using this yet?

The new signing keys are used in:

shim-signed 1.54 on 22.10+, 1.51.3 on 22.04, 1.40.9 on 20.04, 1.37~18.04.13 on 18.04
grub2-signed 1.187.2~ or newer (binary packages grub-efi-amd64-signed or grub-efi-arm64-signed), 1.192 on 23.04.
fwupd-signed 1.51~ or newer
various linux updates. Check apt changelog linux-image-unsigned-$(uname -r) to see if Revoke & rotate to new signing key (LP: #2002812) is mentioned in there to see if it signed with the new key.

If you were able to install shim-signed, your grub and fwupd-efi will have the correct version as that is ensured by packaging. However your shim may still point to the old one. To check which shim will be used by grub-install, you can check the status of the shimx64.efi.signed or (on arm64) shimaa64.efi.signed alternative. The best link needs to point to the file ending in latest:

$ update-alternatives --display shimx64.efi.signed
shimx64.efi.signed - auto mode
  link best version is /usr/lib/shim/shimx64.efi.signed.latest
  link currently points to /usr/lib/shim/shimx64.efi.signed.latest
  link shimx64.efi.signed is /usr/lib/shim/shimx64.efi.signed
/usr/lib/shim/shimx64.efi.signed.latest - priority 100
/usr/lib/shim/shimx64.efi.signed.previous - priority 50

If it does not, but you have installed a new kernel compatible with the new shim, you can switch immediately to the new shim after rebooting into the kernel by running dpkg-reconfigure shim-signed. You’ll see in the output if the shim was updated, or you can check the output of update-alternatives as you did above after the reconfiguration has finished.

For the out of memory issues in grub, you need grub2-signed 1.187.3~ (same binaries as above).

how do I test this (while it’s in proposed)?

upgrade your kernel to proposed and reboot into that
upgrade your grub-efi-amd64-signed, shim-signed, fwupd-signed to proposed.

If you already upgraded your shim before your kernel, don’t worry:

upgrade your kernel and reboot
run dpkg-reconfigure shim-signed

And you’ll be all good to go.

deep dive: uploading signed boot assets to Ubuntu

For each signed boot asset, we build one version in the latest stable release and the development release. We then binary copy the built binaries from the latest stable release to older stable releases. This process ensures two things: We know the next stable release is able to build the assets and we also minimize the number of signed assets.

OK, I lied. For shim, we actually do not build in the development release but copy the binaries upward from the latest stable, as each shim needs to go through external signing.

The entire workflow looks something like this:

Upload the unsigned package to one of the following “build” PPAs:
- https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa for non-embargoed updates
- https://launchpad.net/~ubuntu-security-embargoed-shared/+archive/ubuntu/grub2 for embargoed updates
Upload the signed package to the same PPA
For stable release uploads:
- Copy the unsigned package back across all stable releases in the PPA
- Upload the signed package for stable releases to the same PPA with ~<release>.1 appended to the version
Submit a request to canonical-signing-jobs to sign the uploads.

The signing job helper copies the binary -unsigned packages to the primary-2022v1 PPA where they are signed, creating a signing tarball, then it copies the source package for the -signed package to the same PPA which then downloads the signing tarball during build and places the signed assets into the -signed deb.

Resulting binaries will be placed into the proposed PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed
Review the binaries themselves
Unembargo and binary copy the binaries from the proposed PPA to the proposed-public PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed-public.

This step is not strictly necessary, but it enables tools like sru-review to work, as they cannot access the packages from the normal private “proposed” PPA.
Binary copy from proposed-public to the proposed queue(s) in the primary archive

Lots of steps!

WIP

As of writing, only the grub updates have been released, other updates are still being verified in proposed. An update for fwupd in bionic will be issued at a later point, removing the EFI bits from the fwupd 1.2 packaging and using the separate fwupd-efi project instead like later release series.

on February 01, 2023 01:40 PM

January 30, 2023

Ronin

Stuart Langridge

In 1701, Asano Naganori, a feudal lord in Japan, was summoned to the shogun’s court in Edo, the town now called Tokyo. He was a provincial chieftain, and knew little about court etiquette, and the etiquette master of the court, Kira Kozuke-no-Suke, took offence. It’s not exactly clear why; it’s suggested that Asano didn’t bribe Kira sufficiently or at all, or that Kira felt that Asano should have shown more deference. Whatever the reasoning, Kira ridiculed Asano in the shogun’s presence, and Asano defended his honour by attacking Kira with a dagger.

Baring steel in the shogun’s castle was a grievous offence, and the shogun commanded Asano to atone through suicide. Asano obeyed, faithful to his overlord. The shogun further commanded that Asano’s retainers, over 300 samurai, were to be dispossessed and made leaderless, and forbade those retainers from taking revenge on Kira so as to prevent an escalating cycle of bloodshed. The leader of those samurai offered to divide Asano’s wealth between all of them, but this was a test. Those who took him up on the offer were paid and told to leave. Forty-seven refused this offer, knowing it to be honourless, and those remaining 47 reported to the shogun that they disavowed any loyalty to their dead lord. The shogun made them rōnin, masterless samurai, and required that they disperse. Before they did, they swore a secret oath among themselves that one day they would return and avenge their master. Then each went their separate ways. These 47 rōnin immersed themselves into the population, seemingly forgoing any desire for revenge, and acting without honour to indicate that they no longer followed their code. The shogun sent spies to monitor the actions of the rōnin, to ensure that their unworthy behaviour was not a trick, but their dishonour continued for a month, two, three. For a year and a half each acted dissolutely, appallingly; drunkards and criminals all, as their swords went to rust and their reputations the same.

A year and a half later, the forty-seven rōnin gathered together again. They subdued or killed and wounded Kira’s guards, they found a secret passage hidden behind a scroll, and in the hidden courtyard they found Kira and demanded that he die by suicide to satisfy their lord’s honour. When the etiquette master refused, the rōnin cut off Kira’s head and laid it on Asano’s grave. Then they came to the shogun, surrounded by a public in awe of their actions, and confessed. The shogun considered having them executed as criminals but instead required that they too die by suicide, and the rōnin obeyed. They were buried, all except one who was not present and who lived on, in front of the tomb of their master. The tombs are a place to be visited even today, and the story of the 47 rōnin is a famous one both inside and outside Japan.

You might think: why have I been told this story? Well, there were 47 of them. 47 is a good number. It’s the atomic number of silver, which is interesting stuff; the most electrically conductive metal. (During World War II, the Manhattan Project couldn’t get enough copper for the miles of wiring they needed because it was going elsewhere for the war effort, so they took all the silver out of Fort Knox and melted it down to make wire instead.) It’s strictly non-palindromic, which means that it’s not only not a palindrome, it remains not a palindrome in any base smaller than itself. And it’s how old I am today.

Yes! It’s my birthday! Hooray!

A glowing message board reading 'BDAY BASH 47'

I have had a good birthday this year. The family and I had delightful Greek dinner at Mythos in the Arcadian, and then yesterday a bunch of us went to the pub and had an absolute whale of an afternoon and evening, during which I became heartily intoxicated and wore a bag on my head like Lord Farrow, among other things. And I got a picture of the Solvay Conference from Bruce.

A framed picture of the Solvay Conference 1927, which is a bunch of stern-looking male physicists and Marie Curie arranged like a school photo

This year is shaping up well; I have some interesting projects coming up, including one will-be-public thing that I’ve been working on and which I’ll be revealing more about in due course, a much-delayed family thing is very near its end (finally!), and in general it’s just gotta be better than the ongoing car crash that the last few years have been. Fingers crossed; ask me again in twelve months, anyway. I’ve been writing these little posts for 21 years now (last year has more links) and there have been ups and downs, but this year I feel quite hopeful about the future for the first time in a while. This is good news. Happy birthday, me.

Me wearing a peach-coloured card gift bag on my head in the pub

on January 30, 2023 03:44 PM

FOSS Activites in January 2023

Utkarsh Gupta

Here’s my (fortieth) monthly but brief update about the activities I’ve done in the F/L/OSS world.

Debian

This was my 49th month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas ‘19! \o/

There’s a bunch of things I do, both, technical and non-technical. Here are the things I did this month:

Uploads

redmine (5.0.4-1) - Fixing bug #1022818, #1026048, and #1027340.
libyang2 (2.1.30-2) - Adding DEP8 test for yangre.

Others

Proposed tomcat9 bullseye -pu via 9.0.43-2~deb11u5.
Helped Otto with review of mariadb from NEW.
Sponsored php-font-lib for William.
Advocated William for becoming DD, uploading.
Granted some DM rights.
Mentoring for newcomers.
Reviewed libgit2 bits, new uploads and changes.
Moderation of -project mailing list.

A huge thanks to Freexian for sponsoring my Debian work. :D

Ubuntu

This was my 24th month of actively contributing to Ubuntu. Now that I joined Canonical to work on Ubuntu full-time, there’s a bunch of things I do! \o/

I mostly worked on different things, I guess.

Debian (E)LTS

And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support).

This was my fortieth month as a Debian LTS and thirty-first month as a Debian ELTS paid contributor.
I worked for 43.25 hours for LTS and 25.00 hours for ELTS.

LTS CVE Fixes and Announcements:

Issued DLA 3281-1, fixing CVE-2022-47950, for swift.
For Debian 10 buster, these problems have been fixed in version 2.19.1-1+deb10u1.
Issued DLA 3295-1, fixing CVE-2022-24785 and CVE-2022-31129, for node-moment.
For Debian 10 buster, these problems have been fixed in version 2.24.0+ds-1+deb10u1.
Issued DLA 3296-1, fixing CVE-2023-24038, for libhtml-stripscripts-perl.
For Debian 10 buster, these problems have been fixed in version 1.06-1+deb10u1.
Issued DLA 3297-1, fixing CVE-2022-48281, for tiff.
For Debian 10 buster, these problems have been fixed in version 4.1.0+git191117-2~deb10u6.
Issued DLA 3298-1, fixing CVE-2020-8161, CVE-2020-8184, CVE-2022-44570, CVE-2022-44571, and CVE-2022-44572, for ruby-rack.
For Debian 10 buster, these problems have been fixed in version 2.0.6-3+deb10u2.
Issued DLA 3300-1, fixing CVE-2022-47951, for glance.
For Debian 10 buster, these problems have been fixed in version 2:17.0.0-5+deb10u1.
Issued DLA 3301-1, fixing CVE-2022-47951, for cinder.
For Debian 10 buster, these problems have been fixed in version 2:13.0.7-1+deb10u2.
Issued DLA 3302-1, fixing CVE-2022-47951, for nova.
For Debian 10 buster, these problems have been fixed in version 2:18.1.0-6+deb10u2.
Issued DLA 3303-1, fixing CVE-2022-25648, CVE-2022-46648, and CVE-2022-47318, for ruby-git.
For Debian 10 buster, these problems have been fixed in version 1.2.8-1+deb10u1.
Started to look at other set of packages.

ELTS CVE Fixes and Announcements:

Issued ELA 784-1, fixing CVE-2022-25648, CVE-2022-46648, and CVE-2022-47318, for ruby-git.
For Debian 9 stretch, these problems have been fixed in version 1.2.8-1+deb9u1.
Issued ELA 785-1, fixing CVE-2022-44570 and CVE-2022-44571, for ruby-rack.
For Debian 9 stretch, these problems have been fixed in version 1.6.4-4+deb9u4.
Issued ELA 787-1, fixing CVE-2022-45442, for ruby-sinatra.
For Debian 9 stretch, these problems have been fixed in version 1.4.7-5+deb9u2.
Helped facilitate Erlang’s and RabbitMQ’s update; cf: ELA 754-1.
Started to look at other set of packages.

Other (E)LTS Work:

Triaged node-moment, modsecurity-apache, ruby-git, ruby-sinatra, gpac, cargo, git, openjdk-11, swift, libxpm, lilypond, openjdk-8, modsecurity, netdata, nim, rust-cargo, sgt-puzzles, apache2, wireshark, libhtml-stripscripts-perl, redis, tomcat8, tiff, ruby-rack, tmux, ruby-rack, ruby-sidekiq, libapache2-mod-auth-mellon, jupyter-core, net-snmp, and rabbitmq-server.
Marked CVE-2023-{0358,2314{3-5}}/gpac as EOL for buster.
Marked CVE-2022-46176/cargo as no-dsa in buster.
Marked CVE-2022-4{4617,6285,883}/libxpm as no-dsa for buster, stretch, and jessie.
Marked CVE-2020-17354/lilypond as ignored for buster.
Marked CVE-2022-48279/modsecurity as no-dsa for buster.
Marked CVE-2023-2249{6,7}/netdata as no-dsa for buster.
Marked CVE-2021-46872/nim as no-dsa for buster.
Marked CVE-2022-46176/rust-cargo as no-dsa in buster.
Marked TEMP-1028986-7037E6/sgt-puzzles as no-dsa for buster.
Marked CVE-2006-20001 and CVE-2022-3{6760,7436}/apache2 as postponed for stretch and jessie.
Marked CVE-2023-22458/redis as not-affected for stretch and jessie.
Marked CVE-2022-45143/tomcat8 as postponed for stretch and jessie.
Marked CVE-2022-44572/ruby-rack as not-affected for stretch.
Marked CVE-2022-47950/swift as not-affected for stretch.
Auto EOL’d node-debug, nim, netty, ruby-git, firefox-esr, linux, swift, radare2, gpac, virtualbox, shiro, sgt-puzzles, pdns-recursor, sofia-sip, libgit2, wireshark, amanda, libhtml-stripscripts-perl, pkgconf, libapache-session-ldap-perl, golang-yaml.v2, nvidia-graphics-drivers, xen, rails, ruby-rack, assimp, thunderbird, cinder, glance, nova, editorconfig-core, chromium, ruby-globalid, spip, opusfile, pgpool2, and ruby-sanitize.
Helped and assisted new contributors joining Freexian (LTS/ELTS/internally).
Answered questions (& discussions) on IRC (#debian-lts and #debian-elts) and Matrix.
Participated and helped fellow members with their queries via private mail and chat.
General and other discussions on LTS private and public mailing list.
Attended the monthly LTS meeting.

Until next time.
:wq for today.

on January 30, 2023 05:41 AM

January 26, 2023

How to speed up your next build 5-20x with Firebuild?

Balint Reczey

TL;DR: Just prefix your build command (or any command) with firebuild:

firebuild <build command>

OK, but how does it work?

Firebuild intercepts all processes started by the command to cache their outputs. Next time when the command or any of its descendant commands is executed with the same parameters, inputs and environment, the outputs are replayed (the command is shortcut) from the cache instead of running the command again.

This is similar to how ccache and other compiler-specific caches work, but firebuild can shortcut any deterministic command, not only a specific list of compilers. Since the inputs of each command is determined at run time firebuild does not need a maintained complete dependency graph in the source like Bazel. It can work with any build system that does not implement its own caching mechanism.

Determinism of commands is detected at run-time by preloading libfirebuild.so and interposing standard library calls and syscalls. If the command and all its descendants’ inputs are available when the command starts and all outputs can be calculated from the inputs then the command can be shortcut, otherwise it will be executed again. The interception comes with a 5-10% overhead, but rebuilds can be 5-20 times, or even faster depending on the changes between the builds.

Can I try it?

It is already available in Debian Unstable and Testing, Ubuntu’s development release and the latest stable version is back-ported to supported Ubuntu releases via a PPA.

How can I analyze my builds with firebuild?

Firebuild can generate an HTML report showing each command’s contribution to the build time. Below are the “before” and “after” reports of json4s, a Scala project. The command call graphs (lower ones) show that java (scalac) took 99% of the original build. Since the scalac invocations are shortcut (cutting the second build’s time to less than 2% of the first one) they don’t even show up in the accelerated second build’s call graph. What’s left to be executed again in the second run are env, perl, make and a few simple commands.

The upper graphs are the process trees, with expandable nodes (in blue) also showing which command invocations were shortcut (green). Clicking on a node shows details of the command and the reason if it was not shortcut.

Could I accelerate my project more?

Firebuild works best for builds with CPU-intensive processes and comes with defaults to not cache very quick commands, such as sh, grep, sed, etc., because caching those would take cache space and shortcutting them may not speed up the build that much. They can still be shortcut with their parent command. Firebuild’s strength is that it can find shortcutting points in the process tree automatically, e.g. from sh -c 'bash -c "sh -c echo Hello World!"' bash would be shortcut, but none of the sh commands would be cached. In typical builds there are many such commands from the skip_cache list. Caching those commands with firebuild -o 'processes.skip_cache = []' can improve acceleration and make the reports smaller.

Firebuild also supports several debug flags and -d proc helps finding reasons for not shortcutting some commands:

...
FIREBUILD: Command "/usr/bin/make" can't be short-cut due to: Executable set to be not shortcut, {ExecedProcess 1329.2, running, "make -f debian/rules build", fds=[{FileFD fd=0 {FileOFD ...
FIREBUILD: Command "/usr/bin/sort" can't be short-cut due to: Process read from inherited fd , {ExecedProcess 4161.1, running, "sort", fds=[{FileFD fd=0 {FileOFD ...
FIREBUILD: Command "/usr/bin/find" can't be short-cut due to: fstatfs() family operating on fds is not supported, {ExecedProcess 1360.1, running, "find -mindepth 1 ...
...

make, ninja and other incremental build tool binaries are not shortcut because they compare the timestamp of files, but they are fast at least and every build step they perform can still be shortcut. Ideally the slower build steps that could not be shortcut can be re-implemented in ways that can be shortcut by avoiding tools performing unsupported operations.

I hope those tools help speeding up your build with very little effort, but if not and you find something to fix or improve in firebuild itself, please report it or just leave a feedback!

Happy speeding, but not on public roads!

on January 26, 2023 09:06 AM

January 19, 2023

New edge tiling introduced in 23.04

David Mohammed

In the past few months, we’ve been exploring the options to create a more extended version of the built-in edge tiling, that we have been using for so many years. Ubuntu Budgie’s Window Shuffler already offered (a.o.) quarter & half tiling, activated by keyboard shortcuts, and controlling windows from the keyboard has been a priority since Shuffler first saw the daylight. For those who prefer...

Source

on January 19, 2023 03:24 PM

January 07, 2023

What are the most important improvements that Debian need to make?

Jonathan Carter

“What are the most important improvements that Debian need to make?” – I decided to see what all the fuss is about and asked ChatGPT that exact question.

It’s response:

The list of potential improvements for Debian that I provided is based on my understanding of the challenges and opportunities that software projects like Debian typically face, as well as my understanding of the goals and values of the Debian project. It is not based on any specific information about the current priorities or needs of the Debian project or community.

There are many areas where the Debian project could make improvements, and the most important ones will depend on the priorities and needs of the project and its community. Some potential areas for improvement could include:

Improving the user experience: Debian is known for its stability and security, but it is not always the easiest distribution to use, especially for new users. Improving the user experience and making Debian more accessible to a wider audience could be an important goal.

Keeping up with the rapid pace of change in the software ecosystem: With over 50,000 packages, Debian is a large distribution, and it can be difficult to keep up with the constant stream of updates and new software being released. Developing tools and processes to help manage this complexity and ensure that new packages are tested and integrated smoothly could be an important focus.

Maintaining the stability and security of the distribution: Ensuring that Debian remains a stable and secure platform is a top priority for the project. This includes testing new packages thoroughly before they are included in the distribution, responding quickly to security vulnerabilities, and investing in infrastructure and processes to support these efforts.

Ensuring the long-term sustainability of the project: Debian is a volunteer-driven project, and it can be challenging to attract and retain contributors over the long term. Developing strategies to support and encourage participation and collaboration within the community could be an important focus.

Enhancing the tools and infrastructure that support the project: The Debian project relies on a wide range of tools and infrastructure to support its development and distribution efforts. Improving and modernizing these resources could help to make the project more efficient and effective.

I find that to be a great response, and I’m curious where it came from, so I asked “How did you come to that conclusion?”, it replied:

The list of potential improvements for Debian that I provided is based on my understanding of the challenges and opportunities that software projects like Debian typically face, as well as my understanding of the goals and values of the Debian project. It is not based on any specific information about the current priorities or needs of the Debian project or community.

Well, one thing is for sure, the IRC chat bots of the future are going to be much more complex than the ones I grew up with :-)

on January 07, 2023 12:19 PM

January 05, 2023

Year of The Broken Desktop

Alan Pope

This morning I attempted to start work on my desktop PC and couldn’t. The screen is black, it doesn’t want to wake up the displays. I used the old REISUB trick to restart, and it boots, but there’s no output on the display. I did some investigation and this post is mainly to capture my notes and so others can see the problem and perhaps debug and fix it. The setup is an Intel Skull Canyon NUC connected to an external GPU enclosure which contains an NVIDIA GeForce RTX 2060.

on January 05, 2023 09:00 AM

January 02, 2023

What to do about hotlinking

Stuart Langridge

Hotlinking, in the context I want to discuss here, is the act of using a resource on your website by linking to it on someone else’s website. This might be any resource: a script, an image, anything that is referenced by URL.

It’s a bit of an anti-social practice, to be honest. Essentially, you’re offloading the responsibility for the bandwidth of serving that resource to someone else, but it’s your site and your users who get the benefit of that. That’s not all that nice.

Now, if the “other person’s website” is a CDN — that is, a site deliberately set up in order to serve resources to someone else — then that’s different. There are many CDNs, and using resources served from them is not a bad thing. That’s not what I’m talking about. But if you’re including something direct from someone else’s not-a-CDN site, then… what, if anything, should the owner of that site do about it?

I’ve got a fairly popular, small, piece of JavaScript: sorttable.js, which makes an HTML table be sortable by clicking on the headers. It’s existed for a long time now (the very first version was written twenty years ago!) and I get an email about it once a week or so from people looking to customise how it works or ask questions about how to do a thing they want. It’s open source, and I encourage people to use it; it’s deliberately designed to be simple¹, because the target audience is really people who aren’t hugely experienced with web development and who can add sortability to their HTML tables with a couple of lines of code.

The instructions for sorttable are pretty clear: download the library, then put it in your web space and include it. However, some sites skip that first step, and instead just link directly to the copy on my website with a <script> element. Having looked at my bandwidth usage recently, this happens quite a lot², and on some quite high-profile sites. I’m not going to name and shame anyone³, but I’d quite like to encourage people to not do that, if there’s a way to do it. So I’ve been thinking about ways that I might discourage hotlinking the script directly, while doing so in a reasonable and humane fashion. I’m also interested in suggestions: hit me up on Mastodon at @sil@mastodon.social or Twitter⁴ as @sil.

Move the script to a different URL

This is the obvious thing to do: I move the script and update my page to link to the new location, so anyone coming to my page to get the script will be wholly unaffected and unaware I did it. I do not want to do this, for two big reasons: it’s kicking the can down the road, and it’s unfriendly.

It’s can-kicking because it doesn’t actually solve the problem: if I do nothing else to discourage the practice of hotlinking, then a few years from now I’ll have people hotlinking to the new location and I’ll have to do it again. OK, that’s not exactly a lot of work, but it’s still not a great answer.

But more importantly, it’s unfriendly. If I do that, I’ll be deliberately breaking everyone who’s hotlinking the script. You might think that they deserve it, but it’s not actually them who feel the effect; it’s their users. And their users didn’t do it. One of the big motives behind the web’s general underlying principle of “don’t break the web” is that it’s not reasonable to punish a site’s users for the bad actions of the site’s creators. This applies to browsers, to libraries, to websites, the whole lot. I would like to find a less harsh method than this.

Move the script to a different dynamic URL

That is: do the above, but link to a URL which changes automatically every month or every minute or something. The reason that I don’t want to do this (apart from the unfriendly one from above, which still applies even though this fixes the can-kicking) is that this requires server collusion; I’d need to make my main page be dynamic in some way, so that links to the script also update along with the script name change. This involves faffery with cron jobs, or turning the existing static HTML page into a server-generated page, both of which are annoying. I know how to do this, but it feels like an inelegant solution; this isn’t really a technical problem, it’s a social one, where developers are doing an anti-social thing. Attempting to solve social problems with technical measures is pretty much always a bad idea, and so it is in this case.

Contact the highest-profile site developers about it

I’m leaning in this direction. I’m OK with smaller sites hotlinking (well, I’m not really, but I’m prepared to handwave it; I made the script and made it easy to use exactly to help people, and if a small part of that general donation to the universe includes me providing bandwidth for it, then I can live with that). The issue here is that it’s not always easy to tell who those heavy-bandwidth-consuming sites are. It relies on the referrer being provided, which it isn’t always. It’s also a bit more work on my part, because I would want to send an email saying “hey, Site X developers, you’re hotlinking my script as you can see on page sitex.example.com/sometable.html and it would be nice if you didn’t do that”, but I have no good way of identifying those pages; the document referrer isn’t always that specific. If I send an email saying “you’re hotlinking my script somewhere, who knows where, please don’t do that” then the site developers are quite likely to put this request at the very bottom of their list, and I don’t blame them.

Move the script and maliciously break the old one

This is: I move the script somewhere else and update my links, and then I change the previous URL to be the same script but it does something like barf a complaint into the console log, or (in extreme cases based on suggestions I’ve had) pops up an alert box or does something equally obnoxious. Obviously, I don’t wanna do this.

Legal-ish things

That is: contact the highest profile users, but instead of being conciliatory, be threatening. “You’re hotlinking this, stop doing it, or pay the Hotlink Licence Fee which is one cent per user per day” or similar. I think the people who suggest this sort of thing (and the previous malicious approach) must have had another website do something terrible to them in a previous life or something and now are out for revenge. I liked John Wick as much as the next poorly-socialised revenge-fantasy tech nerd, but he’s not a good model for collaborative software development, y’know?

Put the page (or whole site) behind a CDN

I could put the site behind Cloudflare (or perhaps a better, less troubling CDN) and then not worry about it; it’s not my bandwidth then, it’s theirs, and they’re fine with it. This used to be the case, but recently I moved web hosts⁵ and stepped away from Cloudflare in so doing. While this would work… it feels like giving up, a bit. I’m not actually solving the problem, I’m just giving it to someone else who is OK with it.

Live with it

This isn’t overrunning my bandwidth allocation or anything. I’m not actually affected by this. My complaint isn’t important; it’s more a sort of distaste for the process. I’d like to make this better, rather than ignoring it, even if ignoring it doesn’t mean much, as long as I’m not put to more inconvenience by fixing it. We want things to be better, after all, not simply tolerable.

So… what do you think, gentle reader? What would you do about it? Answers on a postcard.

and will stay simple; I’d rather sorttable were simple and relatively bulletproof than comprehensive and complicated. This also explains why it’s not written in very “modern” JS style; the best assurance I have that it works in old browsers that are hard to test in now is that it DID work in them and I haven’t changed it much ↩
in the last two weeks I’ve had about 200,000 hits on sorttable.js from sites that hotlink it, which ain’t nothin’ ↩
yet, at least, so don’t ask ↩
if you must ↩
to the excellent Mythic Beasts, who are way better than the previous hosts ↩

on January 02, 2023 06:10 PM

December 30, 2022

Full Circle Magazine #188

Full Circle Magazine

This month:
* Command & Conquer
* How-To : Python, ~~Blender~~ and Latex
* Graphics : Inkscape
* Everyday Ubuntu
* Micro This Micro That
* Review : Kubuntu 22.10
* Review : Ubuntu Cinnamon 22.04
* Ubports Touch : OTA-24
* ~~Tabletop Ubuntu~~
* Ubuntu Games : Dwarf Fortress (Steam Edition)
plus: News, My Story, The Daily Waddle, Q&A, and more.

Get it while it’s hot: https://legacy.fullcirclemagazine.org/issue-188/

on December 30, 2022 05:33 PM

December 15, 2022

Adding giscus Comments

Alan Pope

My blog at popey.com/blog has gone through a number of iterations since I started it back in the 1990’s. First it was created using Microsoft FrontPage, and hosted on some free web space at CiX, and has morphed into a self-hosted WordPress site, Nikola static site, and now a Hugo static site. At various times I’ve had comment systems available underneath posts. Over the years I’ve used Microsoft FrontPage extensions (which, amusingly used to crash the entire Web Host at CiX back in the day), WordPress comments, and Disqus to facilitate visitor interaction.

on December 15, 2022 04:00 PM

December 14, 2022

Selenium + AT-SPI = GUI Testing

Harald Sitter

At KDE we have multiple levels of quality assurance ranging from various degrees of a humans testing features to fully automated testing. Indeed automated testing is incredibly important for the continued quality of our software. A big corner stone of our testing strategy are so called unit tests, they test a specific piece of our software for its behavior in isolation. But for many aspects of our software we need a much higher level view, testing pieces of Plasma’s application launcher in isolation is all good and well but that won’t tell us if the entire UI can be easily navigated using the keyboard. For this type of test we require a different testing approach altogether. A couple months ago I’ve set set out to create a testing framework for this use case and I’m glad to say that it has matured enough to be used for writing tests. I’d like to walk you through the technical building blocks and a simple example.

Let us start of by looking at the architecture at large. So… there’s Selenium which is an incredibly popular, albeit web-oriented, testing framework. Its main advantages for us are its popularity and that it sports a server-client split. This means we can leverage the existing client tooling available for Selenium without having to write anything ourselves, we only need to grow a server. The server component, called a WebDriver, implements the actual interaction with UI elements and is generic enough to also apply to desktop applications. Indeed so thought others as well: there already exists Appium – it extends Selenium with more app-specific features and behaviors. Something for us to build upon. The clients meanwhile are completely separate and talk to the WebDriver over a well defined JSON REST protocol, meaning we can reuse the existing clients without having to write anything ourselves. They are available in a multitude of programming languages, and who knows maybe we’ll eventually get one for writing Selenium tests in QML

That of course doesn’t explain how GUI testing can work with this on Linux. Enter: AT-SPI. AT-SPI is an accessibility API and pretty much the standard accessibility system for use on Linux. Obviously its primary use is assistive technologies, like the screen reader Orca, but to do its job it essentially offers a toolkit-independent way of introspecting and interacting with GUI applications. This then gives us a way to implement a WebDriver without caring about the toolkit or app specifics. As long as the app supports AT-SPI, which all Qt apps do implicitly, we can test it.

Since all the client tooling is independent of the server all we needed to get GUI testing going was a WebDriver that talks to AT-SPI.

That is what I set out to write and I’m happy to announce that we now have an AT-SPI based WebDriver, and the first tests are popping into existence already. There is also lovely documentation to hold onto.

So, without further ado. Let us write a simple test. Since the documentation already writes one in Python I’ll use Ruby this time around so we have some examples of different languages. A simple candidate is KInfoCenter. We can test its search functionality with a couple of lines of code.

First we need to install selenium-webdriver-at-spi, clone it, cmake build it, and cmake install it. You’ll also need to install the relevant client libraries. For ruby that’s simply running gem install appium_lib.

Then we can start with writing our test. We will need some boilerplate setup logic. This is more or less the same for every test. For more details on the driver setup you may also check the wiki page.

  def setup
    @appium_driver = Appium::Driver.new(
      {
        'caps' => { app: 'org.kde.kinfocenter.desktop' },
        'appium_lib' => {
          server_url: 'http://127.0.0.1:4723',
          wait_timeout: 10,
          wait_interval: 0.5
        }
      }, true
    )
    @driver = @appium_driver.start_driver
  end

The driver will take care of starting the correct application and make sure that it is actually running correctly. Next we’ll write the actual test. Let’s test the search. The first order of business is using a tool called Accerciser to inspect the AT-SPI presentation of the application. For more information on how to use this tool please refer to the wiki. Using Accerciser I’ve located the search field and learned that it is called ‘Search’. So, let’s locate it and activate it, search for the CPU module:

  def test_search
    search = driver.find_element(:name, 'Search')
    search.click
    search.send_keys('cpu')

Next let us find the CPU list item and activate it:

    cpu = driver.find_element(:class_name, '[list item | CPU]')
    assert(cpu.displayed?)
    cpu.click

And finally let’s assert that the page was actually activated:

    cpu_tab = driver.find_element(:class_name, '[page tab | CPU]')
    assert(cpu_tab.displayed?)

To run the complete test we can use the run wrapper: selenium-webdriver-at-spi-run ./kinfocentertest.rb (mind that it needs to be +x). If all has gone well we should get a successful test.

Finished in 1.345276s, 0.7433 runs/s, 1.4867 assertions/s.

1 runs, 2 assertions, 0 failures, 0 errors, 0 skips
I, [2022-12-14T13:13:53.508516 #154338]  INFO -- : tests done
I, [2022-12-14T13:13:53.508583 #154338]  INFO -- : run.rb exiting true

This should get you started with writing a test for your application! I’ll gladly help and review your forthcoming tests.
For more detailed documentation check out the writing-tests wiki page as well as the appium command reference.

Of course the work is not done. selenium-webdriver-at-spi is very much still a work in progress and I’d be glad for others to help add features as they become needed. The gitlab project is the place for that.

The complete code of the example above:

#!/usr/bin/env ruby
# frozen_string_literal: true

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2022 Harald Sitter <sitter@kde.org>

require 'appium_lib'
require 'minitest/autorun'

class TestKInfoCenter < Minitest::Test
  attr_reader :driver

  def setup
    @appium_driver = Appium::Driver.new(
      {
        'caps' => { app: 'org.kde.kinfocenter.desktop' },
        'appium_lib' => {
          server_url: 'http://127.0.0.1:4723',
          wait_timeout: 10,
          wait_interval: 0.5
        }
      }, true
    )
    @driver = @appium_driver.start_driver
  end

  def teardown
    driver.quit
  end

  def test_search
    search = driver.find_element(:name, 'Search')
    search.click
    search.send_keys('cpu')

    cpu = driver.find_element(:class_name, '[list item | CPU]')
    assert(cpu.displayed?)
    cpu.click

    cpu_tab = driver.find_element(:class_name, '[page tab | CPU]')
    assert(cpu_tab.displayed?)
  end
end

on December 14, 2022 03:59 PM

About Planet Ubuntu

Planet Ubuntu is a window into the world, work and lives of Ubuntu developers and contributors.

If you are an Ubuntu Member, and would like your blog aggregated here, please see the PlanetUbuntu wiki page.
Feeds