posts on Loren's blog

Shut Up and Take My Money

loren@lorendb.dev (Loren Burkholder) — Thu, 05 Mar 2026 23:13:52 -0500

I recently had the opportunity to purchase some software in an IT role. Everything went well until I needed a modification to my billing. Easy enough, according to the software’s website. Just email us and we’ll apply the modification. So I shot off an email asking to apply a modification to my billing.

I received a friendly response from a salesman: “How are you planning to use our product?” OK, fine, he can get his metrics. I’m not opposed to that. So I shot off another email, briefly describing my use case. Additionally, I mentioned that we have subscribed to this software; we’re running a trial period for it and plan to fully roll it out before long, barring any major hiccups.

The salesman’s next reply took me off guard. “I think our product will work well for you. Let’s set up a sales call.” I’m sorry, what?! I already told you that I have bought your product; you don’t have to sell it to me again.

Let’s go back in time to an entirely different yet hauntingly familiar incident.

Some time ago, I was asked to look into purchasing email security software. This seemed like a straightforward task. However, I was dumbfounded to see that it was impossible to purchase email security software… unless you were willing to go through a sales call. You couldn’t even find pricing numbers on product websites. For at least one company, I filled out the contact form with an invalid phone number so I could ask for pricing without worrying that my phone would start ringing off the hook.

In both of these cases, the software in question is aimed squarely at IT professionals. Lest we forget, the stereotypical IT worker is an introverted nerd who hates talking to people. If you are trying to sell something to an introvert who hates talking to people, your selling point should be a self-service checkout, so the introvert can avoid sales calls.

I’m not an idiot; neither am I incapable of research. When I need to purchase software, I research my options, compare their features (and price), and decide which best meets my needs. Then I go searching for an “add to cart” button. If I am met instead with “Schedule a call,” I am not pleased. I already decided I was going to give you money; you don’t have to sell your product to me again.

If I have a question about your product during the research phase, I’ll first refer to the internet. If I can’t surface anything useful myself, I may see if an LLM knows the answer. Only if neither the internet nor an LLM can help me will I send your sales team an email, because I know they will hound me for money and try to set up sales calls that I don’t want. A self-service sale should be much more meaningful than a sale that is the result of a sales team, because it means the customer liked your product without you breathing down their neck about it. In fact, your “Features” page on your website should be the entirety of your sales team.

Shut up and take my money.

Getting Bitten by Poor Naming Schemes

loren@lorendb.dev (Loren Burkholder) — Thu, 18 Dec 2025 23:50:43 -0500

I recently came into possession of an old Dell Precision T3610 workstation and promptly installed Proxmox to add it to my Proxmox cluster. After performing some ludicrously silly RAM and storage upgrades (how about 96 GB of DDR3, plus a 13-disk array of 500 GB SSDs?), I decided I wanted to max out the CPU as well.

The Precision T3610 shipped with an Intel Xeon E5-1650 v2. According to the linked Intel product page, this CPU uses the FCLGA2011 socket. Easy enough, I thought to myself. Just find the best CPU that supports FCLGA2011, make sure you have the latest BIOS installed, and everything should be all hunky dory. So I did some research and landed on the Xeon E7-8890 v4. It’s several years newer than the E5-1650 v2, has a whopping 24 cores (and hyperthreading bumps it to 48 logical cores!), and can support having not one, not two, but eight of itself installed in a single motherboard! Most crucially, the Intel product page says it uses the FCLGA2011 socket. When I stumbled across one of these monsters on eBay for just $15, I snapped it up.

Cue my massive shock and disappointment when, a few days later, I found myself unable to install the E7-8890 v4 in my T3610. The new CPU, despite being the same physical size as the old CPU, had extra contacts on the bottom and had a different physical keying. What? I thought Intel said this was the same socket!

Some amount of research later, I discovered that Intel’s LGA2011 socket has many variations. One of these variations is also called Socket R (or LGA2011-0). The T3610, and by extension the old E5-1650 v2 CPU, uses Socket R. The newer E7-8890 v4, meanwhile, uses a different variation called Socket R2 (or LGA2011-1). As if this wasn’t confusing enough, there’s even a third variation of the LGA2011 socket! I’ll refer you to the Wikipedia page for more info on that.

This is obviously not a great naming scheme. Why not use unique numbers for each version of the socket instead of tacking on a suffix? But the real kicker here is that Intel itself doesn’t seem to be able to keep up with its own naming scheme! It appears that its CPU specifications pages refer to all variants of the LGA2011 socket as FCLGA2011. This leaves folks like myself wondering what went wrong when their new-to-them CPUs don’t fit in their motherboards.

So where does that leave me? Well, I now have a fancy paperweight. I could have returned the CPU, but return shipping costs would have been half of what I paid for the CPU itself, so I’m hanging onto it for now in case I ever come into possession of a server with a Socket R2 motherboard that could use a nicer CPU. At least it wasn’t a super expensive CPU, so all in all, this isn’t the worst learning experience ever.

The Advantage of a Fishtank PC Case

loren@lorendb.dev (Loren Burkholder) — Thu, 06 Feb 2025 17:16:41 -0500

My PC is built in a fishtank-style case, namely the Lian Li O11 Vision.

The Lian Li O11 Vision. What a nice case!

Despite being built as a collab with the PC Masterrace subreddit (aka PCMR), this case and others with the same fishtank style are widely derided by many Redditors. While glass panels give you a great view of your internal components and RGB lighting, not everybody likes RGB, and having the top and front made of glass limits the effective airflow of the case, leaving your PC susceptible to running hotter. However, I think I have identified the biggest advantage that fishtank cases have over more traditional cases.

It is easy to forget to occasionally clean out your PC; when you do finally open the case, the results can be horrifying.

With a fishtank case, this will not happen. As you use the PC, you will always have a front-row seat to the internals. In order to keep your PC presentable, you will of necessity need to clean the thing out every now and again. Despite having my PC for just three months, I’ve already popped the side panels off to dust the glass once.

If you still hate fishtank cases but want to make sure you see any dust or just prevent dust outright, might I suggest trying an open frame case or an immersion cooling case?

The Case for Custom Email Domains

loren@lorendb.dev (Loren Burkholder) — Sun, 26 Jan 2025 22:02:01 -0500

I recently switched my primary email to be hosted under my own domain, instead of using Outlook.com mail. It turns out there both advantages and disadvantages to doing so. I’ll try to fairly address both sides, but (spoiler alert) I think that the pros heavily outweigh the cons.

Before we begin, I’d like to note that I personally am using Purelymail’s hosting, as it is cheap ($10/yr) and has glowing reviews from its customers. However, I am not affiliated with Purelymail, nor will I discourage you from using some other service.

Also, I will refer to all the well-known free email providers as Big Email for the remainder of this article.

Reasons to use a custom domain

My biggest reason for moving to my own domain was to assert ownership over my online accounts. It’s not uncommon to read stories of people whose Big Email accounts were inexplicably permabanned with no chance of recovering them. With a custom domain, even if my mail provider goes dark or bans my account, I can just point my DNS records at another mail service and maintain access to all my accounts that rely on sending authentication emails.

Switching to a custom domain is obviously cool because you can control the entire email address, not just the username.

If you want to get some real hands-on experience, you can run your own mail server! All you need is a cheap VPS and a willingness to get your hands dirty.

A VPS (virtual private server) is required here because residential ISPs typically block email servers being run from your home.

On the other hand, services like Purelymail will run the mail server for you for very cheap, taking most of the burden off your shoulders without

Reasons to avoid a custom domain

Setting up a custom domain can be a bit tricky if you aren’t a technically inclined person (although if you’re reading my blog, you likely are savvy enough). On the other hand, Big Email provides a pain-free solution that anybody can use. A counterpoint to this argument: when you buy a domain from Porkbun, you are offered turnkey email hosting for $2/inbox/mo. Porkbun handles all the DNS records for you.

While custom domains do save you from Big Email lockouts, the onus is still on you to make sure that you don’t fall victim to domain hijacking.

Big Email typically provides much more robust spam detection and reporting features than smaller providers will.

Conclusion

There are certainly valid reasons to avoid custom email domains, but I think that if you already own a domain, you should seriously consider using it for your email as well.

Fixing VR's screen door effect with the new iPad

loren@lorendb.dev (Loren Burkholder) — Sat, 15 Jun 2024 03:45:00 -0400

While I do not own a VR headset, as a technophile, I am very intrigued by the technology and like to follow the developments that happen in the VR space. As a result, I’ve become aware of the screen door effect: when you look closely at content rendered on a VR headset, you can see the gaps between pixels. Given that pixels are laid out in a grid, this ends up giving the feeling of looking through a screen door. Newer headsets have mitigated this problem somewhat by using higher resolution displays, but the problem is still there. You can see an example of this on the Bigscreen Beyond, Apple Vision Pro, and Meta Quest 3 in this image from Karl Guttag:

The effect is especially noticeable here on the Quest 3.

The Vision Pro has been hailed by some as eliminating the screen door effect, but Karl Guttag debunked that theory: the Vision Pro has the same screen door effect as any other headset. It’s not noticeable because Apple intentionally left the optics slightly out of focus; this makes the effect difficult to see at the cost of some sharpness.

You may wonder, then, if we’ll ever eliminate this effect. After all, if the incredibly pixel-dense Vision Pro displays can’t fix it, will we ever build dense enough displays to fix it? To answer this question, let’s take a look at the new iPad Pro.

Tandem OLED

One of the features of the latest iPad Pro is the new Tandem OLED display. Apple has leveraged the transparency of OLED panels by stacking two of them in one display. While this is expensive technology, it does have the benefit of greatly boosting the potential screen brightness: when the two OLED panels work together, you can double your potential brightness. However, I’m not interested in making VR headsets excessively bright (although that could possibly be useful for some VR games). Instead, I think tandem OLED displays can fix the screen door effect by filling in the gaps between pixels.

For reference, here is Apple’s magnification of the Vision Pro display (interestingly, I couldn’t find a similar image for any other OLED VR headset):

Not bad, considering that those pixels are just 7.5 micrometers across.

As far as I can tell, the iPad display has simply stacked two panels directly on top of each other. This provides no benefits to VR. Instead, I propose offsetting the top panel a bit to fill the empty space of the lower panel with the pixels of the other. For starters, here’s what it would look like if you simply offset an identical panel vertically and horizontally by half a pixel’s width:

There's far less dead space already.

But we can go further with this. What if you created a custom pixel layout for the second layer? I’m no expert on OLED screens, so I can’t say whether this is actually a practical layout, but let’s suppose that we can create a display like this:

This might work, although I'm a little concerned that the blue subpixels would complicate the wiring traces.

When layered on top of the Vision Pro’s pixel layout, we get a display with nearly zero black areas:

This is looking better and better...

And this is all assuming we’re sticking with the Vision Pro’s pixel layout. Other headsets will have their own subpixel layouts that may be more or less conducive to creating a secondary offset layout.

If you want to play around with your own pixel layouts, you can download the XCF file that I used to create these images here.

Will we see this technology adopted any time soon? I hestitate to say that we will; after all, the tandem OLED configuration is expensive enough that it could easily become cost-prohibitive for many people. However, I think it’s only a matter of time until somebody either comes up with this idea independently or steals it from this post, and the higher quality display might be worth the money if it’s installed on a premium edition headset.

Maybe it is the year of the Linux desktop

loren@lorendb.dev (Loren Burkholder) — Thu, 23 May 2024 19:30:00 -0400

A while back I wrote about the challenges that are preventing mass Linux adoption. I’ve been mulling on this and related topics for a while, and yet I haven’t really progressed far beyond my closing thoughts from then:

However, I think there’s another way we could push Linux faster. If some enterprising person launches a Linux computer startup that focuses heavily on marketing, strikes deals with retailers to promote their computers, and ships a polished distro that will make the transition from Windows as easy as possible, I think there is a real possibility that Linux on the desktop could happen sooner than we expect. Unfortunately, I don’t know of any tech billionares who are looking for a cool side project, but if that describes you, you’re welcome to talk to me about it!

Well, it just so happens that a tech billionare is publicly contemplating the potential of Linux:

Maybe it is time to move consumer desktops to Linux
— Elon Musk (@elonmusk) May 20, 2024

That’s right, the man that some adore as “Technoking” and others deride as “Space Karen” now is talking about switching the general public over to Linux on the desktop. Granted, he didn’t actually say “We’re building a Linux PC”, but when Elon Musk publicly talks about crazy ideas, sometimes they actually happen (example).

A little background

Before you dismiss this tweet as “another meme from Elon”, let’s take a look at the historical context. In February, Elon bought a new Windows laptop and was frustrated when he discovered that Microsoft requires you to sign in with a Microsoft account:

Just bought a new PC laptop and it won’t let me use it unless I create a Microsoft account, which also means giving their AI access to my computer! This is messed up.

There used to be an option to skip signing into or creating a Microsoft account.

Are you seeing this too?
— Elon Musk (@elonmusk) February 25, 2024

He went so far as to complain at Satya Nadella in a (probably futile) attempt to get Microsoft to ditch the sign-in requirement.

By the way, if you want to bypass Microsoft account sign-in on Windows, you can easily bypass it during setup. When asked to sign in, type “no@thankyou.com” for the email address and anything you want in the password field. Since this email address has been blacklisted, the sign-in will fail and you will be dropped back to a local account creation screen.

Elon’s recent tweet suggesting that Linux should take over was prompted by Microsoft’s announcement of Recall, an AI assistant that will log everything you did on your PC for the last three months and then feed that into Copilot to try to help you find more things. Besides the massive security implications of Recall, the entire concept seems dystopian, making the Apple “Crush” ad seem downright appealing by comparison. You may not have any features of humanity left, but at least Apple (theoretically) isn’t feeding everything you do into the hungry maw of data-collecting servers (although that could start changing if the rumors of a new OpenAI-powered Siri are true).

The part where I make wild speculations

At this point, there are two possible outcomes: either Elon makes sure to keep all of Windows’ antifeatures turned off and just lives with it, or he starts a computer company out of frustration. The second scenario is the interesting one here.

I’m going to assume such a computer company would focus on laptops initially, but pretty much all of the below commentary applies equally to desktop PCs.

If Elon decides to start a computer company, one thing is certain: it will be divisive solely because it is owned by Elon Musk. On the one hand, ardent fans will all but smash their existing computers as they rush to daily-drive Elon’s laptop instead. On the other hand, some will go out of their way to avoid Elon’s laptops, possibly even switching their operating system:

This is a coworker's reaction to Elon's Linux tweet.

Of course, I’m not planning to ditch Linux if Elon starts a Linux computer company, and I certainly hope nobody else does either.

Beyond the initial online arguments that are grounded only in personal opinions of Elon, what would a potential product look like and what would it do for Linux? Most likely, any laptop produced by Elon would be done as a subsidiary of X, which means it would likely sprout a (paywalled) xAI integration. This could be good or bad, depending on how it’s executed. Done well, it ends up being a useful AI assistant that doesn’t push itself in your face. Done poorly, you end up with a final product that is no better than Windows.

Would the laptop sell well? I hesitate to make a call on that. While the laptop would face an uphill battle to gain widespread acceptance while being an Elon Musk product, it would have the advantage of a high-profile launch, with accompanying media coverage. Imagine all the news outlets that covered the Apple Vision Pro now covering a Linux laptop, and you’ll get an idea of how many potential users could be reached. As long as the laptop has a solid enough user experience to get positive reviews, that translates into possibly more media coverage Linux laptops have ever received before, helping sell the laptop to people who aren’t using their opinion of Elon to decide whether or not to buy.

According to Statcounter, Linux desktop share sits at 3.74%, with ChromeOS at 6.18% (thanks to its widespread adoption in public schools) and macOS at 22.59%. For Linux to reach a point where software makers begin to treat it as an equal player, it needs to at least get close to equal macOS’s market share. My not-at-all-scientific mental estimations put the required number of conversions well into the millions, making this no easy task.

Fortunately, it’s not a task that Elon would have to achieve himself. With the all the publicity surrounding this laptop, existing companies like System76 that already have a solid product would have a golden opportunity to push themselves into the public eye, and if public demand for any Linux-powered laptop rose enough, the big OEMs could potentially start joining the trend. After all, it’s not like most people unconditionally love Windows or anything, and between an all-seeing AI Big Brother, an ad-riddled interface, and a history of being just unstable enough to bluescreen at the worst possible times, the Windows of the future just might be a Windows that nobody uses.

What are the chances that this happens?

Frankly, I’m not holding my breath. While Elon probably has the money to start a computer company several times over without batting an eye, it’s not clear whether he actually wishes to do so, as his attention is already spread quite thinly between all his other endeavors. With that being said, Elon’s distrust of others to do a good job might push him to build a computer that he feels he can trust with his personal information.

What do I mean by “Elon’s distrust of others to do a good job”? As far as I can tell, he views himself as the only person he can trust to hold his companies to idealistic standards. It’s why he bought Twitter; he ultimately didn’t trust the existing Twitter leadership to provide a politically fair platform, so he put himself in charge instead. It also could partly explain why he still retains a significant amount of control and involvement at SpaceX; his goal for that company is to enable Martian colonies, and if somebody else was in charge of SpaceX, they might veer from that goalpost and focus on the existing launch market.

Regardless of whether Elon starts a computer company, his tweet has likely already had a positive impact. As of 5/23/2024, Twitter reports 43,000 views on that tweet; many of those views are probably from non-Linux users. This exposure alone has probably convinced at least a few people to try Linux, and at the end of the day, that’s a good thing.

Does Skylon have a place in the world of Starship?

loren@lorendb.dev (Loren Burkholder) — Fri, 12 Apr 2024 22:09:06 -0400

As a space nerd, I love learning about the craziest rocket concepts that have come up over the years (and there certainly have been quite a lot). One category of concept intrigues me in particular: single-stage-to-orbit, or SSTO. This type of spacecraft can reach orbit without the need for detachable boosters or other staged hardware. This is quite easy to achieve on a body like the Moon, given its low gravity and lack of atmosphere. SSTOs are also feasible on Mars, as Mars has lower gravity and a very thin atmosphere. However, Earth’s higher gravity and thicker atmosphere make SSTOs fairly tricky to pull off; nevertheless, it’s doable, and so there have been numerous designs proposed.

Many SSTO designs have a traditional rocket form factor, with the exception that the fuel tank is conical rather than cylindrical (the DC-X is a good example of this). However, the spaceplane form factor is also common and arguably sports some benefits: at takeoff, you can use the atmosphere to your advantage as it generates lift, and you can just land on a runway instead of having to fool around with parachutes or retropropulsive landings (a la Falcon 9). In fact, as far as I can tell, the spaceplane-style SSTO is the more popular SSTO style among Kerbal Space Program players; browsing the KSP subreddit shows both spaceplanes and rockets, but spaceplanes are far more common.

Of course, what goes in KSP doesn’t necessarily go in real life. While Kerbin’s gravity and atmospheric pressure are similar to Earth’s, it is only about a tenth of the size of Earth, making it far easier to get to orbit. This makes it very easy to take a large fuel tank, slap a Vector on the bottom, and get 4 km/s of delta-V, which is enough to get to orbit and back with a single stage.

Skylon

Skylon stands out among spaceplane SSTO designs for various reasons. For starters, its SABRE engines can run in an airbreathing mode, which reduces the amount of liquid oxygen that Skylon has to carry (which in turn allows it to carry more hydrogen fuel), but also function just fine when burning hydrolox. Furthermore, it’s the only SSTO spaceplane that is in active development; while not much news is forthcoming from its developer Reaction Engines, it appears that they are still working on SABRE and Skylon.

However, Skylon has been in development since the 1980s. It has its roots in the HOTOL spaceplane; while the HOTOL project was ultimately cancelled by the British government, the people behind it didn’t give up the idea of a spaceplane with a dual-mode engine, eventually starting Reaction Engines to develop Skylon as a sort of second iteration on the concept. Since then, we’ve seen the end of the space shuttle and the rise of Falcon 9, with Starship looming on the horizon. We’re no longer in an era where rocket launches are prohibitively expensive; in fact, if you just want to send a CubeSat into orbit, you can do so quite cheaply. SpaceX currently estimates a launch cost of $300,000 for a 1U CubeSat weighing up to fifty kilograms on one of their rideshare missons. In other words, if you have three hundred grand burning a hole in your pocket, you can get sweet bragging rights by launching your own satellite! And it will only get cheaper: when SpaceX starts offering rideshare launches on Starship, costs will likely drop significantly due to Starship’s improved efficiency (not to mention its reusable second stage, which represents an enormous savings on its own).

At this point, then, should we be throwing in the towel on Skylon? Does it even have a place in today’s landscape, or does Starship completely obsolete Skylon?

By the numbers

I’m not going to use mass-to-orbit as a deciding factor, because Starship would beat every other rocket out there, and besides, just because we need super-heavy-lift rockets for some payloads doesn’t mean we should use them for every single satellite that is ever launched. Instead, I’ll try to make a more fair comparison based on the payload fraction and launch costs of each vehicle. Please keep in mind that while I’ve tried to get accurate numbers here, you should treat everything said here as a rough calculation, as Skylon is still operation in the theoretical realm and the numbers given for Starship are likely not extremely precise. Additionally, SpaceX is continually iterating Starship for more performance, so we could easily see the numbers change in another year or so. I’ve also converted every number to freedom units, so there could be slight precision losses from that as well.

According to its manual, Skylon has a payload capacity of 16.5 tons, while SpaceX says that Starship can launch up to 150 tons. Given Skylon’s 345 ton mass and Starship’s 5000 ton mass (Wikipedia), we can calculate a payload fraction of 4.8% for Skylon and 3% for Starship. While this seems to point to Skylon, the cost per launch tells a different story: Skylon’s estimated cost per launch is $10 millon. While we don’t know the exact numbers for Starship, Elon Musk has claimed that costs could fall as low as a few million dollars per launch. Even if Starship cost the same to launch as Skylon, it still has a clear win in cost per pound to orbit: in this scenario, Skylon would cost $302.39 per pound to orbit, while Starship would cost just $33.33 per pound to orbit.

Since numbers don’t lie, you’d be forgiven if you immediately wrote off Skylon at this point. However, I think there’s one more thing we need to consider.

Apples to oranges

Skylon and Starship are both orbital launch vehicles that aim to decrease the cost to launch while promoting reusability, but that’s about where the similarities end, as Skylon leans into the SSTO spaceplane design while Starship embraces traditional rocket designs alongside retropropulsive landing. Clearly, we can’t properly compare these vehicles without weighing these differences, and I think this is where Skylon gets an opportunity to shine.

Traditional reentry capsules (think Apollo) tend to pull many Gs on reentry, which causes a certain level of discomfort to the crew. I couldn’t find numbers on the projected G-forces on either Starship or Skylon, but Skylon has a big advantage in this department: its wings. As a craft that is capable of gliding, Skylon is capable of taking a much shallower reentry path that reduces the overall stress on the vehicle. In an age where spacecraft are being proposed for use as long-range passenger transport vehicles and various companies are developing space stations for commercial or tourist use, Skylon could become the perfect crew transport vehicle to reduce stress on passengers, both at launch and reentry. Given that Reaction Engines claimed an intended crew capacity of 30 or 40, Skylon could become the luxury space travel option, while a trip on Starship would be a bit more like flying economy: less expensive and less comfortable. And it’s not necessarily just people; Skylon could potientially enable us to launch equipment that is much more delicate to further scientific research.

It’s up to Reaction Engines now

With all this in mind, I’m optimistic that there is still a place for Skylon in Starship’s world. At this point, we just need Reaction Engines to pull through and actually build the thing. Until then, we’ll just have to keep flying the traditional rockets.

Introducing the Matrix Codebrowser

loren@lorendb.dev (Loren Burkholder) — Wed, 27 Mar 2024 18:54:14 -0400

Today marks the public introduction of the Matrix Codebrowser. It’s an instance of codebrowser that is hosting the sources for mtxclient, libQuotient, and olm (with more projects to come in the future).

Testimonials

But it’s almost surreal to see my own code in a page that I usually see when I need to understand Qt’s inner workings.

kitsune, March 27, 2024

What is it good for?

If you’ve never used a codebrowser instance before, you probably are wondering “why would you make such a big deal about a public source mirror?” Well, codebrowser doesn’t just mirror a repository to a web interface. It also uses compilation information to detect and display all the symbols in a source file. For example, try mousing over some symbols in the QScopedValueRollback<> source from the official codebrowser instance.

This ability to easily navigate the content of a library’s codebase is the raison d’être of codebrowser. I occasionally have a need to see exactly what a certain Qt function is doing under the hood; while I could grep through the GitHub mirror, it’s far faster to look up the function in codebrowser, and I can also quickly dive into function calls from there.

More to come

The current public version of the code only supports C++ codebases; however, the official instance has seen massive work by KDAB to add support for Rust and Dart code. Once KDAB’s work is made public, I plan to add more repositories like matrix-rust-sdk and vodozemac.

You can also submit requests! I’m currently mainly focused on Matrix-related libraries, since that is likely the most useful code I can support, but if you want to see your favorite client or server added, I’ll be happy to add it (as long as it’s within reason). Just leave a comment below or come say hi on Matrix.

The year of Linux on the desktop, and why we can't have nice things

loren@lorendb.dev (Loren Burkholder) — Tue, 19 Mar 2024 15:32:00 -0400

Ever since the dawn of time (er, Linux), enthusiastic users have predicted the advent of the year of Linux on the desktop. Hardly a year has gone by without somebody proclaiming it to be such a year. And yet Linux only recently crossed 4% of desktop market share worldwide. What gives?

On the surface, it’s easy to point to different reasons. Maybe it’s the overwhelming sea of distros, or the lack of official support for Adobe (or whatever proprietary Windows-only software you use), or the gaps in the user experience that leave you poking around in the terminal, wondering where it all went wrong. Maybe it’s even the lack of prebuilt Linux PCs from major manufacturers. These are all valid arguments, but I think there’s a deeper problem here.

Heads up - this post is a bit of a rant. Proceed at your own risk; I provide absolutely no guarantee that you will agree with the stated sentiments.

Also, pedants likely will notice that I’m not treating ChromeOS as Linux. This is intentional; the Linux experience that everybody wants to see become popular doesn’t include Google accounts and spyware throughout the OS. Instead, I’m using the term Linux to refer to the more traditional distros.

Opensource is not capitalism

It’s no secret that opensource software is a very socialist way to do things. For example, companies whose product is opensource tend to make their money by selling hosted versions of their software or providing paid support for their products. Unfortunately for Linux, this directly conflicts with the capitalist economy that most of the world runs on. Why is this unfortunate for Linux? Because it drives down contributions.

Now, I can hear you screaming at me already: “But everybody hires kernel developers nowadays! Even Microsoft funds kernel developers!” And you’re right! However, you need to remember a few things. First, developers from, say, Microsoft are only concerned about kernel development that directly impacts Microsoft. This practically means that Microsoft’s kernel involvement is limited to making WSL possible and supporting their custom Linux distro that they use for Azure. Beyond that, they have little incentive to get involved. This means that while the kernel does get plenty of attention, there’s also potential for resources to be focused in areas that don’t directly benefit many users.

Second, and perhaps more importantly, Linux is not just the kernel. While I don’t want to fan the flames of the GNU/Linux warriors, their arguments do point to the underlying issue here: while the kernel provides a rock-solid base for your operating system, there are dozens of other components that make up a Linux OS (init system, desktop environment, network connection managers, etc.). Every one of those components must itself receive maintenance in order for your system to run smoothly.

The question, then, is “who maintains all of this software?” Sure, there are plenty of volunteers out there who are looking to fix bugs on their systems or just have a fun time writing code, but you can’t count on a volunteer to provide forty hours of programming a week. To build a truly polished and reliable product, paid development is nearly essential. You may disagree with me, but I’d like to point out that much of the polished Linux software we love has had developers behind it paid to work on it: besides the kernel itself, there’s systemd, many KDE components, much of Pop_OS!, and many other projects, not to mention applications like Firefox, Element, and Visual Studio Code, all of which are direct products of companies instead of being volunteer projects.

At this point, you’re probably wondering where I’m going with this. Well, if you’ll bear with me, I promise it will all make sense in a bit.

This is why we can’t have nice things

If so many companies are already investing in Linux and the wider opensource ecosystem, why hasn’t the world switched to Linux? To understand that, let’s go back to what I said earlier about Linux kernel development:

Developers from, say, Microsoft are only concerned about kernel development that directly impacts Microsoft.

And therein lies the rub. While we see many companies embracing opensource, nearly all of them end up being like, say, Signal: they provide value to the ecosystem and create an incredible product, but at the end of the day, they are only investing into the apps part of the Linux ecosystem. I’m certainly not trying to downplay what companies like Signal are doing, but at the end of the day, the core system components are left without support.

This is not to say that core software is without support; again, volunteer effort goes a long way to keep the lights on, and companies like Blue Systems (who work on KDE) and System76 (who have their own distro and have contributed massively to the entire opensource stack, including Coreboot) are doing amazing things to supplement volunteer work with commercially-funded development. However, the number of companies paying developers to work on projects like KDE is vastly lower than the number of companies paying developers to work on the kernel.

In my opinion, this lack of commercial support is what leads to the rough edges in the Linux experience. If you want to see what I mean, the best way is to try a volunteer-built distro like Debian or Manjaro against a commercially-built distro like Zorin OS or Pop_OS!. The distros backed by for-profit companies have had much more work put into making the best experience possible, whereas the volunteer distros largely depend on upstream development to identify and fix rough edges.

However, with most opensource companies focused on building only one or two apps rather than funding an ecosystem, we can’t fix the problem overnight.

The bright light is… Valve?

Interestingly enough, the trail to fixing this ecosystem problem may have been blazed by a gaming company. That’s right, Valve is back to save the day again, but to understand why, we must first look away from Valve.

If commercially-backed development only happens when a company has an incentive to provide that development, we need to figure out what companies would directly benefit from a better Linux desktop experience. We don’t have to look far, though. As I mentioned above, System76 has poured lots of time and effort into developing the best Linux experience for their hardware, going so far as to create their own distro (and they’re building their own desktop environment, too!). But it doesn’t have to be just System76. It could be any company selling a Linux PC.

Unfortunately, there’s not a lot of those out there today. But that could change in the future! Specifically, Lenovo is a prime candidate to become a major provider of Linux systems. A few months ago, the user mindcrime revealed in a Hacker News thread that Lenovo prefers Linux:

Lenovo secretly want Linux to catch on big-time, because they HATE the amount of leverage Microsoft have over them due to the ubiquity of Windows on so many of their devices. But they have to be careful to not embrace Linux too openly for fear of Microsoft retaliation. So they’re in a bit of a catch-22 situation. […]

- mindcrime

If mindcrime is right, Lenovo would be eager to produce Linux hardware the moment Linux seemed to be gaining significant market share (and indeed, they already do make some Linux machines). However, Lenovo is going to need something more than a report of 4% Linux usage worldwide from Statcounter before they pull the trigger on Linux devices. And for that something, we need to look to Valve.

Valve has a long history with Linux. For ten years, they’ve used it as the base of SteamOS, and their games have supported Linux for many years as well. However, Valve wasn’t content to stop there. They currently are supporting development of KDE to run on the Steam Deck. Futhermore, they’ve invested heavily into development of Proton to allow Windows-only games to run on Linux, and according to ProtonDB, fully 40% of games released on Steam are known to be partially or entirely playable on Linux.

The thing is, Proton is just a soft fork of Wine, which is aimed at running any Windows software on Linux, and most (if not all) Proton code is ultimately upstreamed into Wine proper. In other words, Valve’s work on Proton is just another ecosystem investment. Like their work on KDE, their Wine contributions have the capability to unlock Linux for everyone; if and when Wine gets good enough to run nearly all Windows software, including major products like Adobe or Office, with few to no glitches, we’ll have one less blocker for potential new Linux users, and it’ll be due in no small part to Valve.

How does this relate to Lenovo? In my opinion, there are two major blockers for Lenovo (or other OEMs) to adopt Linux: app support and ease of use. Valve is helping on the app support front by developing Proton and on the ease of use front by contributing to KDE. Without major OEM contributions, I foresee Valve being the primary driver of these efforts until OEMs start pushing Linux big-time. In other words, Valve may provide the impetus that drives Linux popularity up to the point that Lenovo feels ready to publicly move toward Linux.

It’s also worth pointing out that we shouldn’t expect OEMs to freeload on Linux. Granted, there will probably be smaller system integrators that will simply OEM-install Ubuntu with their own wallpaper and call it a day, but larger system integrators will almost certainly invest money into the ecosystem. How can I predict that so confidently? Because OEMs already spend enormous amounts of money on Windows.

Let’s take a closer look at the numbers. According to Gartner (table 3), in 2023, Lenovo alone shipped 59.7 million PCs. I think it’s safe to assume that most of them were running Windows, but to be conservative, let’s say just 40 million of those PCs shipped with an active copy of Windows (maybe the other 20 million were running ChromeOS). I wasn’t able to find what large OEMs pay per Windows license (drop a comment below if you know!), but let’s choose an absurdly low number for sake of argument. Assuming Lenovo pays just $5 per Windows OEM key, that works out to $200 million going to Microsoft per year. That’s enough to hire 1,000 developers and pay them $200,000 per year to work on the Linux ecosystem. If we adjust the OEM license key price to even $20 (which still seems like a very low estimate), the developer count jumps to 4,000. And $200,000 is a pretty good salary; I’m sure there are many developers who would be happy to work full-time on opensource for much less than that.

Granted, Lenovo will probably take the opportunity to recoup some of the licensing costs as profit instead of reinvesting that money, and they also won’t entirely abandon Windows, but imagine the effect that even 100 full-time developers could have on your favorite desktop environment. And now remember that there are other manufacturers that will likely jump on board; looking at Gartner’s data, the top five Windows PC manufacturers shipped a combined 185.8 million PCs in 2023. Given a $20 OEM license cost and assuming just half of that money gets redirected to Linux development at $150,000 per developer, OEMs could hire over twelve thousand developers! A developer army of that size would undoubtedly make short work of bug trackers everywhere.

So which year will Linux take over?

It’s tough to predict when Linux will truly take off. I do think that it has a strong chance to become dominant in the future once Microsoft switches Windows to a cloud service, as many people won’t want to hand over their PC to Microsoft or pay a subscription to use Windows (although Google will probably attempt to push ChromeOS into the gap left by Microsoft). However, could it rise sooner than that?

Microsoft’s cloud PC ambitions are likely many years away from being fully realized, so we must look to alternate ways that Linux could rise. As I mentioned above, Valve’s investments are pushing the Linux ecosystem into a place where it can serve most users’ needs. For example, many gamers are learning that Linux actually works just fine on their Steam Decks and can even run their games on their PCs. This could definitely help push Linux upward; however, we’re still missing a major piece of the puzzle.

If you walk into your local Best Buy, Walmart, or Costco and have a look at the various computers displayed, you’ll notice an overwhelming selection of Windows machines with a few Macs and Chromebooks thrown in. There are no machines with Linux installed. Ditto for web retailers. While you can find Linux machines on some websites, they are almost never shown as front-page products, and many retailers just don’t carry Linux machines at all. This means that most users don’t even get a chance to evaluate Linux as an option.

For those who are interested in Linux, then, there are two options: learn how to create a virtual machine or figure out how to boot from a Linux USB and install it. To be fair, the Linux installation experience has had an incredible amount of work put into it, and it’s extremely polished and easy to use (in my opinion, better than the Windows experience), but it’s still beyond the range of many people’s technical ability.

Linux will suffer from this deemphasizing effect until some company steps up to provide Linux computers that are promoted at common retailers. To be honest, Valve could save us again here: if they built and sold a line of gaming PCs preloaded with a desktop-oriented version of SteamOS, desktop Linux would likely get into the hands of hundreds of thousands of gamers. Valve wouldn’t even have to build the PCs; even if some OEM had Valve slap a Steam logo on a gaming PC with SteamOS installed, we’d likely see a massive uptick in Linux users.

Of course, stores like Walmart tend to stock low- to mid-range PCs intended for casual use rather than dedicated gaming, so the Valve approach might not work well for getting Linux PCs into stores. Unfortunately, this is going to be a little harder to overcome, but I think there’s another problem we’ve been overlooking this whole time: today’s prebuilt Linux systems are almost all high-end workstations. For example, with the exception of the Meerkat mini PC, System76 doesn’t sell desktops or laptops for less than $1,000. With many consumers being constrained by a tight budget, then, System76 is out of the question. However, if we could get something like the Pinebook Pro (but perhaps built with an i3 or similar processor) into the hands of consumers, it would be very appealing. I think that a low-end Linux laptop could easily be built for $300 or less without majorly sacrificing build quality; after all, the Pinebook Pro has a magnesium alloy chassis and is still only a little over $200.

Finally, I think it’s worth looking at what I’m going to call the Hail Mary approach. In this approach, a major OEM like Lenovo would invest millions of dollars into a fast-track Linux development campaign, with the goal of creating a distro with a frictionless experience in just a year’s time. From the development side, I think this would be possible; if they used an already solid distro for the base (say, Fedora or openSUSE) and picked Plasma for the UI, they are already at least 85% of the way to a beginner-friendly product. Their year of development could mainly focus on removing rough edges, implementing a few missing features, and generally making sure the experience was absolutely competitive with Windows.

However, from a business perspective, the Hail Mary approach is extremely risky. Unless it was extremely well cloaked, Microsoft would quickly learn that Lenovo was about to ditch Windows. If Microsoft decided to immediately cut off business relations, Lenovo would be left unable to ship either Windows machines or polished Linux machines for quite a while. This kind of risk means that a Hail Mary will almost certainly not happen.

With all things considered, then, I predict that while the year of the Linux desktop has a high chance of happening, it will almost certainly not happen in the next three to five years. For that to happen, we’d need Microsoft to go all-in on Windows 365 within a few years from now, as well as large efforts from at least one major player to develop the Linux ecosystem.

A few ways we can game the system

First off, if you can, please consider contributing to your favorite opensource product to help fill the gap. For example, I have made a few contributions to KDE in the form of code; you may prefer to pick a different desktop environment, work on an application, or even contribute by helping with non-programming tasks (documentation, bug triage, etc.) or donating money.

However, I think there’s another way we could push Linux faster. If some enterprising person launches a Linux computer startup that focuses heavily on marketing, strikes deals with retailers to promote their computers, and ships a polished distro that will make the transition from Windows as easy as possible, I think there is a real possibility that Linux on the desktop could happen sooner than we expect. Unfortunately, I don’t know of any tech billionares who are looking for a cool side project, but if that describes you, you’re welcome to talk to me about it!

Hang in there!

Regardless of the difficulties that we face, Linux will almost certainly get its time in the limelight within the next ten years. It’s already showing steady growth, especially in India, where it commands 15% market share, and it ultimately beats its opponents in nearly every way. Unlike Windows, it’s lean and mean (and developer-friendly); unlike macOS, it’s completely under your control and allows freedom of hardware choice; unlike ChromeOS (and Windows and macOS), it’s privacy-respecting, so your daily computer usage won’t be sent off to Big Tech’s servers. Finally, despite its reliance on volunteer contributions, it provides a remarkably stable and polished experience already. With this combination of features, it’s just a matter of time before people start to switch, and once global Linux adoption reaches a certain critical mass, OEM adoption and contribution will do the rest.

C++ Spotlight: FINAL CUT terminal GUI

loren@lorendb.dev (Loren Burkholder) — Sat, 02 Mar 2024 15:20:00 -0500

This is the first post in what will hopefully be a long series where I take a look at different C++ libraries and utilities that are uniquely awesome. Today, we’re looking at FINAL CUT, a library for implementing a GUI in the terminal. It promises an API that is very similar to Qt’s API and plenty of functionality, and it doesn’t disappoint.

Getting started

I was happy to see that openSUSE packages FINAL CUT. A quick sudo zypper in libfinal-devel later, I had the library installed on my system. Getting it into my C++ test project, however, was not quite as fun. It appears that FINAL CUT doesn’t provide a CMake interface out of the box, so I had to manually add the library to my CMake file:

cmake_minimum_required(VERSION 3.5)

project(cpp-playground LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_library(final final)

add_executable(cpp-playground main.cpp)
target_link_libraries(cpp-playground PRIVATE final)

It’s not a huge dealbreaker or anything, but it’s always nice to be able to use find_package instead of find_library, given that find_package does so much more.

With CMake set up, let’s try the basic dialog example linked in the FINAL CUT wiki:

#include <final/final.h>

int main(int argc, char *argv[])
{
    finalcut::FApplication app(argc, argv);
    finalcut::FDialog dialog(&app);
    dialog.setText("A dialog");
    const finalcut::FPoint position{25, 5};
    const finalcut::FSize size{30, 10};
    dialog.setGeometry(position, size);
    finalcut::FWidget::setMainWidget(&dialog);
    dialog.show();
    return app.exec();
}

Running that code will show a window in your terminal; you can drag it around with your mouse and click on the button in the titlebar to open the menu!

That background would make my eyes hurt if it stayed there too long...

Let’s build something useful!

Demos are great, but what if we could build something that has an actual use? Let’s build a proof-of-concept system monitor application that will show individual resources in separate windows. We’ll start with a main dialog that shows the app name and version:

class MainDialog : public finalcut::FDialog
{
public:
    MainDialog(finalcut::FApplication *app)
        : finalcut::FDialog(app)
    {
        setText("finaltop");
        const finalcut::FPoint position{4, 2};
        const finalcut::FSize size{17, 3};

        m_about = new finalcut::FLabel("finaltop v0.0.1", this);
        m_about->setGeometry({3, 2}, {20, 5});

        setGeometry(position, size);
    }

private:
    finalcut::FLabel *m_about;
};

For reference, our main() function looks like this:

int main(int argc, char *argv[])
{
    finalcut::FApplication app(argc, argv);

    MainDialog mainDialog(&app);
    mainDialog.show();
    finalcut::FWidget::setMainWidget(&mainDialog);

    return app.exec();
}

Note the call to finalcut::FWidget::setMainWidget(); this marks our MainDialog as the primary dialog for the application. Closing this dialog will shut down the entire app.

Now let’s get down to business and implement individual resource dialogs, beginning with adding an uptime counter to the main dialog. Since Linux adheres to the “everything is a file” design, we can easily parse this from /proc/uptime:

std::string getUptime()
{
    std::ifstream proc_uptime("/proc/uptime");
    double uptime;
    proc_uptime >> uptime;
    int hours, minutes, seconds;
    hours = static_cast<int>(uptime / 3600);
    minutes = static_cast<int>((uptime - hours * 3600) / 60);
    seconds = static_cast<int>(uptime - hours * 3600 - minutes * 60);
    return std::to_string(hours) + "h " + std::to_string(minutes) + "m " + std::to_string(seconds) + "s";
}

We could dump this string directly to the dialog in the constructor, but that’s no fun. Let’s utilize timers to update a label every second. First, we need to create a label and register a timer for it in the constructor:

// m_uptime is a finalcut::FLabel * and m_uptimeTimer is an int
m_uptime = new finalcut::FLabel("Uptime: " + getUptime(), this);
m_uptime->setGeometry({3, 2}, {30, 1});
m_uptimeTimer = addTimer(1000);

Next, we’ll add a function to MainDialog that will handle the timer:

void onTimer(finalcut::FTimerEvent *e) override
{
    if (e->getTimerId() == m_uptimeTimer)
    {
        m_uptime->setText("Uptime: " + getUptime());
        // Note that FINAL CUT will not automatically redraw controls, likely
        // because rendering to the terminal is a pretty expensive operation.
        m_uptime->redraw();
    }
}

With that done, we now have a nifty dynamic uptime counter! Here’s what it looks like on my machine:

Next, we’ll build a CPU meter. For sake of brevity and simplicity, I won’t implement a per-core monitor; instead, I’ll just use the first line of /proc/stat to get overall CPU usage. With that being said, the logic to extend CPU monitoring to cover all cores would be simple; while the first line of /proc/stat is the overall CPU usage, the next number of lines provide per-core monitoring, so we could reuse the parsing logic to extract the per-core usage values and create a dynamically-sized list of progress bars to monitor each of them.

First, we’ll add a CPU widget class that displays a finalcut::FProgressBar, as well as the CPU name from /proc/cpuinfo:

class CpuDialog : public finalcut::FDialog
{
public:
    CpuDialog(finalcut::FWidget *parent)
        : finalcut::FDialog(parent)
    {
        setText("CPU");
        setGeometry({4, 9}, {60, 7});

        m_cpuMeter = new finalcut::FProgressbar{this};
        m_cpuMeter->setGeometry({3, 2}, {50, 1});
        parseCpuUsage();

        m_cpuInfo = new finalcut::FLabel{this};
        m_cpuInfo->setGeometry({3, 4}, {60, 1});

        std::ifstream cpuinfo("/proc/cpuinfo");
        std::string line;
        while (std::getline(cpuinfo, line))
            if (line.starts_with("model name"))
                break;
        m_cpuInfo->setText(line.substr(line.find(':') + 2));

        m_cpuUsageTimer = addTimer(500);
    }

    void onTimer(finalcut::FTimerEvent *e) override
    {
        if (e->getTimerId() == m_cpuUsageTimer)
        {
            parseCpuUsage();
            m_cpuMeter->redraw();
        }
    }

private:
    finalcut::FProgressbar *m_cpuMeter;
    finalcut::FLabel *m_cpuInfo;
    int m_cpuUsageTimer;

    double m_totalCpuLastTick = -1;
    double m_activeCpuLastTick = -1;
};

Now we just have to plug in live data from /proc/stat. Here’s the parser function I ended up building (this was not directly AI-written but it’s inspired by an AI-generated prototype function):

void parseCpuUsage()
{
    std::ifstream proc_stat("/proc/stat");
    std::string cpuLine;
    std::getline(proc_stat, cpuLine);
    std::string cpu;
    int user, nice, system, idle, iowait, irq, softirq, steal, guest, guestnice;
    std::stringstream cpuStream(cpuLine);
    cpuStream >> cpu >> user >> nice >> system >> idle >> iowait >> irq >> softirq >> steal >> guest >> guestnice;
    double totalTime = user + nice + system + idle + iowait + irq + softirq + steal + guest + guestnice;
    double activeTime = user + nice + system + irq + softirq + steal + guest + guestnice;
    m_cpuMeter->reset();
    m_cpuMeter->setPercentage(((activeTime - m_activeCpuLastTick) / (totalTime - m_totalCpuLastTick)) * 100);
    m_totalCpuLastTick = totalTime;
    m_activeCpuLastTick = activeTime;
}

I won’t explain much about this function; you can read man 5 proc if you want to learn more about how /proc/stat formats data. I will point out, however, that I am storing the time values in doubles. Using ints in this case would result in the displayed value clamping to either 0% or 100%. Also of note is the call to FProgressBar::reset(); by default, calling setPercentage() will add to the value of the progress bar rather than overwrite it.

The final result looks pretty good! Here’s a screenshot with Speedometer running in the background to provide a load:

Since memory information in Linux is exposed through /proc/meminfo in a similar manner to /proc/cpuinfo, implementing a memory use widget is not a lot more work than a bit of copy-and-paste. For brevity, I won’t show the implementation on this page, but you can find it here. Here’s how my final implementation looks:

If you have swap enabled, this dialog will also show a swap meter.

Final thoughts (ha ha)

Apart from the annoying all-caps name, FINAL CUT is very fun and intuitive to work with. I’ve had a lot of fun working with it so far, and I might keep developing this system monitor out into a fully-fledged system monitor just as an excuse to keep working with FINAL CUT. If you want to play around with the code, I’ve uploaded it to GitHub as finaltop.

Looking ahead, I have a small list of libraries that I’d like to try out in the future, but if you have any recommendations, please let me know!

Ollama is now packaged for openSUSE

loren@lorendb.dev (Loren Burkholder) — Tue, 27 Feb 2024 20:20:00 -0500

TL;DR: I’ve packaged Ollama for openSUSE! As of February 27, 2024, it’s available for Tumbleweed (Leap will hopefully pick it up at some point). Just run sudo zypper ref && sudo zypper in ollama to get it! Please note that it will not use your GPU for now; if you need GPU support, keep installing Ollama from https://ollama.com instead. Now for the main post…

As somebody who has been a local LLM enthusiast ever since the advent of the Llama models [citation needed], I have been using Ollama for some time now. However, Ollama has one small problem:

Oh noes, it's an install script!

That’s right, it’s 2024 and we still haven’t learned that Flatpaks and AppImages are the best way to distribute a Linux application that will work on any distro (yes, I am part of the Snap hater community). Instead, the Ollama authors have created a script that will download Ollama, create a systemd service, and even install CUDA drivers if necessary.

Now, to be fair, Flatpaks and AppImages aren’t intended to be used for systemd services. Also, while Ollama’s website doesn’t inform you of the security risks of running a random script from the internet, it at least gives you a link to the script source, so you can read over it and make sure it’s not going to do anything nefarious.

By the way, in case you don’t know, you can validate any install script even if the source isn’t linked. Instead of directly piping the script into sh, instead pipe it into a temporary file. You can then read that file to make sure it looks safe before actually running the script.

The obvious best practice for distributing Linux software, however, is to provide system packages. Some projects (like Docker) provide their package repositories; however, in most cases, software should be distributed through the main repository. Since the Ollama devs are not doing this themselves (which is completely understandable; most developers don’t maintain their own packages), I decided to take it upon myself to refresh my knowledge of RPM and the Open Build Service and package Ollama for openSUSE myself.

It can’t be that hard, right?

I’ve packaged a few things for openSUSE before, but it’s been a little while, so I decided to start from scratch. I cloned my home repository to my system and created a new package:

$ osc checkout home:LorenDB
$ cd 'home:LorenDB'
$ osc mkpac ollama

This just creates a blank folder. You’d think it would at least create a .spec file (or “specfile”), but no, you have to do the whole thing yourself. So I used this RPM packaging tutorial from Fedora (which openSUSE links to from their wiki) and copied the sample specfile over to my local repository, adapting the project name, license, and a few other things as needed. I also created a _service file to automatically fetch the Ollama source from GitHub. Then I ran osc build to build the package. It failed, which I was completely expecting. After all, Ollama is written in Go, so I needed to adapt the specfile to actually build Ollama.

osc is the command-line tool for interacting with Open Build Service repositories and packages. It’s really powerful; for example, when I call osc build as mentioned above, it will spin up a VM just like the VMs used on the OBS runners to build packages.

Unexpected difficulties

Given that I am unfamiliar with the Go workflow, I copied a bunch of stuff from the Hugo specfile and rebuilt the package.

It didn’t build.

The build output told me that Go was unable to access https://proxy.golang.org, which is used for the Go package registry. This happens because Go wants to download all of Ollama’s dependencies, but OBS’s runners aren’t internet-connected. After talking with the extremely helpful folks in #packaging:opensuse.org, I learned that OBS actually has a mechanism in place to handle Go packages. With the power of _service files, you can build an archive of Go packages that will be passed into the runner. This way, openSUSE doesn’t have to package every Go package in existence just to satisfy arbitrary Go-based packages. All you have to do to enable this is add the following service to your _service file:

<service name="go_modules" mode="manual">
    <param name="compression">xz</param>
</service>

I found that a few more tweaks were required to make this work; I had to add the resulting vendor.tar.xz to my specfile as a source and untar it during setup. I also had to add -mod=vendor to the GOFLAGS environment variable so Go would know to use my bundled modules.

A quick acknowledgement here: I referenced the Arch Linux Ollama PKGBUILD for some stuff related to the Go build process. Thanks, Arch folks!

Having done all this, I rebuilt the package.

It didn’t build.

Git shenanigans

Ollama does not provide LLM execution capability by itself. Rather, it acts as a powerful API wrapper for the wonderful llama.cpp. This means that building Ollama needs to include a build of llama.cpp. Ollama achieves this by packaging llama.cpp as a git submodule. Unfortunately, the build script attempts to fetch the llama.cpp submodule from GitHub on every build. I was able to disable this by adding export OLLAMA_SKIP_PATCHING=1 to the build process. However, the build still failed, as the tar_scm service that I used to download the source does not include the submodule in the source tarball by default.

…or so I thought when Ollama failed to build llama.cpp, complaining that llm/llama.cpp was not a git repository. I added the following to my tar_scm service:

<param name="submodules">enable</param>

but llama.cpp still wouldn’t build! After some sleuthing, I realized that the problem was a little bit more insiduous than I had thought. While tar_scm had been downloading the llama.cpp submodule all along, it was also silently discarding the .git directory from both the top-level Ollama repo and the llama.cpp submodule. This meant that when Ollama tried to run some git commands on llama.cpp to get the code in the proper state, git would fail, thereby crashing the build. To fix this, I had to add this parameter to tar_scm:

<param name="package-meta">yes</param>

With all this done, I rebuilt the package… and it successfully built!

systemd and user accounts

However, I wasn’t out of the woods yet. I still had to properly package the systemd service so users don’t have to open a new terminal and run ollama serve every time they want to talk to an AI. Bundling the service was pretty easy; I extracted the contents of the service file from the Ollama script, put them in a dedicated ollama.service file, added that file to the specfile, and then added the following instructions to my specfile:

%pre -f %{name}.pre
%service_add_pre %{name}.service

%post
%service_add_post %{name}.service

%preun
%service_del_preun %{name}.service

%postun
%service_del_postun %{name}.service

This installs and uninstalls the Ollama service when you install and uninstall the package. Unfortunately, it doesn’t autostart the service, but that’s a feature for another day.

I also had to create the ollama user account, since Ollama wants to run the server as a dedicated user. I achieved this by adding a file called ollama-user.conf:

#Type Name   ID GECOS      Home directory Shell
u     ollama -  "Ollama" /usr/share/ollama    -

and adding that file to my specfile:

%{sysusers_requires}

# ...

%build
%sysusers_generate_pre %{SOURCE3} %{name} %{name}-user.conf

With this done, I pushed the package to OBS (I had been working on the package locally until now) and spun up a fresh openSUSE Tumbleweed VM to test the package on a clean system. Unfortunately, I had left an error in ollama-user.conf, so installation failed. A quick fix later, I successfully installed Ollama from an RPM package for the first time!

My joy was short-lived, however, as I quickly realized that the systemd service was failing.

Some investigation revealed that while I’d successfully specified that the ollama user’s home folder was /usr/share/ollama, that folder is not automatically created. The fix ended up being as simple as registering /usr/share/ollama in the specfile:

%files
# %{_datadir} expands to "/usr/share" and %{name} expands to "ollama"
%attr(-, ollama, ollama) %{_datadir}/%{name}

Note the %attr(-, ollama, ollama); I initially failed to set ollama as the owner of /usr/share/ollama, so Ollama was unable to access its own home folder! Also of note is that after receiving feedback on my package, I changed the home folder to /var/lib/ollama.

With this fix in place, I again installed the package on my test VM… and after running sudo systemctl start ollama, I was able to run the Ollama systemd service!

Getting it into the openSUSE repositories

Now I just needed to upstream the package to the openSUSE:Factory repository, which is the default repository for all packages that ship with openSUSE Tumbleweed. However, you can’t directly submit a package to Factory.

Instead, openSUSE packaging uses development repositories. These are repositories that are officially run by openSUSE but not directly packaged in openSUSE. Instead, they serve as a staging area to verify that packages build and look sane. Once a package passes in the devel repository, it can be submitted to Factory. In my case, Ollama fit best in the science:machinelearning project, so I submitted my Ollama package there. Around 12 hours later, my request was accepted!

The final step to adding a package to Factory is to write to the openSUSE Factory mailing list to announce your new package, and then submit a request with a link back to that email. Unfortunately, my first two attempts at submitting my package were rejected due to various minor problems in the specfile. However, after submitting my third request, I recieved this exciting email:

This was the most exciting email I've recieved in a little while.

That’s right, Ollama was finally accepted into Factory! My packaging saga was finally over!

…at least, it’s over for now. Ollama releases happen more or less weekly, so I’ll probably have to spend a few minutes every weekend pulling down the latest sources and sending them off to Factory, but the hard work is done.

Some limitations of the package

If every cloud has a silver lining, then every silver lining must have a cloud… right? At any rate, this package is not a perfect solution for everyone. If you wish to use CUDA or ROCm to accellerate your LLMs by running them on the GPU instead of the CPU, you will need to install Ollama from source. openSUSE can’t package a CUDA or ROCm version of Ollama, since neither of those frameworks are available in the main openSUSE repositories. ROCm is open source, so it could eventually land in the main repository, but CUDA is nonfree, so it’s out of the question for now. I’m not ruling out the possibility of eventually releasing an RPM that links against CUDA or ROCm, but something will have to change in order for that to happen.

However, if you’re just looking for a quick way to run an LLM on your machine and don’t need extremely fast results, then enjoy running Ollama from the openSUSE repositories!

Some sort of a conclusion

Despite the various frustrations of building the Ollama RPM, I really enjoyed this little adventure into packaging and plan to keep getting involved. I’ll continue to maintain the Ollama package, and I’m also planning to take maintenance of a few other packages. Furthermore, I’ve created a few more packages! Specifically, I’ve packaged Supernote’s SN Pro font (package), and I’m working on submitting a package for dness.

At this point, I also want to thank Richard Rahl and smolsheep for their help. Whenever I asked questions in #packaging:opensuse.org, they were always quick to patiently answer my questions and point out problems with my package. People like them are what makes contributing to open source projects awesome. Thanks, y’all!

How to reject a feature request

loren@lorendb.dev (Loren Burkholder) — Wed, 21 Feb 2024 21:40:00 -0500

Open-source maintainers go through a lot. They have to deal with users who feel entitled to having their bugs fixed before the other bugs, contributers who submit pull requests of subpar quality, and loads and loads of duplicate issues. Given all this, it’s understandable that maintainers sometimes feel overwhelmed or lash out at those they feel are being unreasonable (for example, Linus Torvalds famously stepped down to address his abusive behavior, and still occasionally sends a heated email).

However, it is in everybody’s best interest that maintainters stay calm and respectful. Abusive maintainers will quickly lead to projects being abandoned by both contributers and users, which reduces the quality of open-source software in general.

A quick note here: I don’t currently maintain any large open-source projects. However, I’ve watched and experienced feature request rejections in various projects, and I feel like this is something that everybody should be reminded of periodically.

It is inevitable that a user will someday request a feature that you don’t like. For example, I have observed users repeatedly requesting that nheko gain the capability to show link previews. The maintainer has repeatedly rejected those requests on grounds that it reduces privacy (by giving your IP address to any URL that somebody sends you, even if you don’t open the URL yourself).

The wrong way to reject a feature request

You may be tempted to tell the user off for suggesting such a clearly flawed and stupid feature. Why are you wasting my time? I’m not going to spend my evening working on a feature that nobody but you will ever use! If you want it so bad, why don’t you just fork the project and implement it yourself?

Here’s the problem: you’re establishing yourself as an opinionated, self-centered person who will only implement features that you will use yourself. Now, to be clear, it’s not wrong to implement features just for yourself. This is referred to as annoyance-driven development. However, a good maintainer will put some effort into implementing features to make life easier for others.

The right way to reject a feature request

Instead, you should politely reject the request with a clear explanation of why you think it’s a bad idea. It’s fine to suggest forking the project, but you should refrain from doing that in most cases. Instead, invite them to contribute the feature themselves.

You should never reject a feature across the board unless it has serious ethical, privacy, or security concerns or if it would require a ridiculous amount of refactoring for a feature that would affect a very small percentage of your users. Here’s why: even if you refuse to implement the feature yourself, leaving that feature as a possibility could lead to a new contributor. Once they have added the feature, you don’t need to touch the code again (unless it breaks) or use that feature at all. However, some of your other users have gained a new feature, and your software has become more compelling for new users.

When it’s OK to be a bit grouchy

Sometimes users persist in being unreasonable. Let’s look at a real-life situation where a feature request was declined, but the user kept pestering the maintainer about it.

It turns out that there is a tiny group of Linux users who believe that D-Bus is bloated, a security risk, and otherwise a bad idea. One of them, @kyronsaif, opened an issue asking if D-Bus could be dropped as a dependency of nheko, where it is currently used to show notifications on Linux and expose a useful API for other applications. The maintainer, @deepbluev7, replied with a calm, rational explanation of why nheko depends on D-Bus and why the dependency won’t be dropped. Case closed… until it wasn’t.

The next day, user @Tcll opened another issue asking to remove D-Bus. This issue is less of a request and more of a tirade against D-Bus. This can be observed in multiple places. First, @Tcll says:

I haven’t found any alternative implementations of dbus (I personally don’t care about integrations enough to look)

This indicates a lack of effort put into the issue. While @Tcll clearly cares enough to create an issue, there is little to no effort put into giving reasonable alternatives to D-Bus. Later, @Tcll says:

it breaks nemo [editor’s note: nemo is a Linux file manager]

and

other apps automatically try to integrate with it to do more than they should have to

Neither of these are issues with D-Bus, and neither of these have anything to do with nheko using D-Bus.

@deepbluev7 responded with an emphatic message reiterating the reasoning from the previous issue and then closed the issue. @Tcll fired back with a message filled with incorrect claims (“sad that the user has to be forced to build this because Linux just can’t be easy to use”), lack of willingness to put forth any personal effort (“don’t ask me to spend time learning your code structure though, you already know your structure”), and personal attacks on @deepbluev7 (“way to be a bad dev letting your personal preferences get in the way of others…”).

For the sake of brevity, I won’t continue to provide a play-by-play of the entire issue thread, since most of it continued in this vein. However, I will point out that every time @deepbluev7 replied, his replies continued to contain rational reasons for rejecting the feature request. His answers do start sounding more and more frustrated:

Any user can do anything they want with Nheko (apart from what the GPL disallows), but no user can force me to do anything or restrict me from spending my time on more useful things. FOSS is about collaboration, not about making others do stuff for you. And it is common and even beneficial in FOSS to just fork and implement what you like there and then have upstream pull it, when they think it is useful. I think you are seriously misunderstanding something here.

but he continues to retain a respectful tone throughout, even mentioning an alternative solution for @Tcll:

I agree. Maintaining multiple implementations on how to do everything in Nheko as well as ways to switch between those at runtime is probably much more overhead to maintain, than just having multiple clients. There are actually several other clients, some of them might fit your requirements better!

As the discussion begins to taper off, @Tcll acknowledges @deepbluev7’s handling of the situation:

at least that’s a more reasonable answer than “build it yourself” or other flack like I commonly get

at least this was a much more respectful issue, and I love it when things go this way even if neither of us can reach a conclusion, I’d rather it end with respect than enemies being made out of it

Clearly, some of @deepbluev7’s responses would not be appropriate if this were a corporate support desk, but this isn’t a corporate support desk. This is an individual volunteer whose personal time is being taken up by this discussion, and the resulting frustration is completely understandable. However, at no point did @deepbluev7 veer into the realm of blatant name-calling or violent attacks.

In conclusion

Maintainers must retain a certain level of professionalism in dealing with users. However, as long as you aren’t running an open-source company, it’s fine if you aren’t entirely professional. Nobody expects a random guy fron Nebraska to behave as if he were the support staff for a Big Tech company, and neither do they expect you to behave that way. Just stay respectful to your users, even if they are a bit unreasonable, and you’ll find that others will respect you for it.

Also, if @deepbluev7, @kyronsaif, or @Tcll read this post, please don’t take offense at being used as an example!

Fixing Hugo's RSS Feed

loren@lorendb.dev (Loren Burkholder) — Mon, 19 Feb 2024 23:15:00 -0500

I’ve built this blog using Hugo, which I love for its sheer simplicity. However, I noticed that the autogenerated RSS feed wasn’t showing any author information. It turns out that I copy-pasted an outdated hugo.toml template that used an older method of specifying author information. This led to the following changes in my hugo.toml:

[params]
  # before
  author = "Loren Burkholder"
  
  # after
  author.name = "Loren Burkholder"
  author.link = "https://lorendb.dev"
  author.email = "loren@lorendb.dev"

However, the feed still had some problems. The author for each post was formatted as loren@lorendb.dev (Loren Burkholder), which is pointlessly verbose. Furthermore, while I try to add tags to each post I make, Hugo was blissfully ignoring them in the RSS feed.

The author name formatting was fairly easy to fix; I just needed to copy the default RSS template to layouts/_default/rss.xml and change the author field from this:

{{- with $authorEmail }}<author>{{ . }}{{ with $authorName }} ({{ . }}){{ end }}</author>{{ end }}

to this:

{{- with $authorName }}<dc:creator>{{ . }}</dc:creator>{{ end }}

This will render each post with the author set to Loren Burkholder. Much better! You will note that I changed the <author> tag to <dc:creator>. This is because the RSS spec requires the <author> tag to use the me@example.com (My Name) format. <dc:creator>, however, allows you to use just a name without violating the spec. Unfortunately, it looks like Akregator can’t handle a feed without <author>, and Tiny Tiny RSS doesn’t observe the <dc:creator> field, so I had to put back the original author field as well.

The tags were slightly trickier. Since I’m a total Hugo scripting noob, I ended up copying a general tag rendering snippet from the Hugo forums and adapting it to my RSS template:

{{ if isset .Params "tags" -}}
    {{ range $index, $name := sort .Params.tags -}}
    <category>{{ $name }}</category>
    {{- end}}
{{- end }}

You can find the entire RSS template here if you want to use it in your own Hugo site.

An update from the next day

The morning after I published this post, I noticed that Hugo is also truncating the contents of RSS feeds, so anybody using a feed reader on this blog would only see the first paragraph or so of each post. This is an absolute crime in the realm of feed readers; it forces people to leave their feed reader for an external site to read the post. Being lazy, I copied somebody else’s approach of telling Hugo to use the post .Content instead of the .Summary. However, I also found out (the hard way) that you must keep the call to transform.XMLEscape; otherwise, your RSS feed will probably break when others try to add it to their feed readers.

I also realized that my original <author> implementation didn’t meet RSS spec, so I modified it to come back in line with the spec.

Update (2/24/2024)

I’ve realized from this HN thread that using <dc:creator> requires a modification to the <rss> element:

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">

This will actually import the appropriate RSS namespace to provide <dc:creator>. Without this, some readers like Thunderbird would fail.

Lapce for C++ Development

loren@lorendb.dev (Loren Burkholder) — Sun, 18 Feb 2024 23:08:00 -0500

On February 18, Lapce hit number one on Hacker News. I was immediately intrigued to see a new cross-platform IDE show up that was not built on Electron. In fact, I was intrigued enough to download Lapce and take it for a spin (in fact, I’m writing this post in Lapce).

My review will be influenced heavily by the fact that I nearly exclusively work with C++. I sometimes use QML in conjunction with C++, and I also really enjoy using D, but the fact remains that in both personal and work programming, I generally use C++. Therefore, I will mainly evaluate Lapce based on its performance as a C++ IDE. Currently, I daily-drive Qt Creator, which is a fairly polished IDE that eschews fancy visuals in favor of building a highly functional IDE for folks like myself who work with Qt. This means that Lapce will have a high bar to clear for me to actually consider adopting it as my default IDE.

Without further ado, let’s get into the review!

Getting started

Lapce has a download button on their website, so I figured I’d try their Linux download. I was expecting to maybe get links to .deb or .rpm files or maybe an AppImage; instead, they provide a .tar.gz that contains a single binary file. Rather unorthodox, but it actually worked fine on openSUSE Tumbleweed, so obviously they’ve built it right. However, I’d prefer to see them distribute either an AppImage (which can be properly installed using AppImageLauncher) or a .flatpakref file that points to a Flatpak repository. After all, Lapce is already available on Flathub.

After running the binary from the .tar.gz, I decided to see if they had a native openSUSE package available; I was pleasantly surprised to find that there is a lapce package available in the default Tumbleweed repo (for the uninitiated, openSUSE puts most software in the openSUSE-Tumbleweed-Oss repo, but some software is in other official repositories; for example, many Matrix clients are in network:messaging:matrix). I installed the package and found that it did not remember my settings from the .tar.gz version. This was highly disappointing; software on Linux should always store settings in the same paths. However, upon further inspection, I found that this is actually a good thing: while openSUSE bundles the nightly version, the .tar.gz download is the latest stable build, so they store their configuration in separate folders to avoid a nightly build breaking your development workflow.

The UI

My first impressions when actually running the IDE can be summed up thusly: Lapce is trying to create a more-or-less drop-in replacement for Visual Studio Code, but written in Rust. And as long as it’s not using Electron, I count that as a win.

However, I’d like to get a bit more specific about the UI. I never have really appreciated the Visual Studio Code layout; I much prefer how Visual Studio or Qt Creator organize the UI. Now, Lapce does a pretty good job at looking OK, but I do have a fair number of nitpicks. I’ve categorized them to make them seem slightly more coherent.

Actually editing text

It’s shockingly easy to lose focus on the editor, and when you do, keyboard inputs won’t work like you would expect. For example, if you use the resize handle on the file view, the editor will stop recieving input until you click somewhere else in the UI. Even global shortcuts (like Save or the command palette) don’t work.

Lapce doesn’t support the normal middle-click paste functionality that is generally supported on Linux. This is a huge deal-breaker for me, as I use this all the time. Another essential text editing feature that is not implemented is dragging selected text. I suppose this may be because Lapce is using a custom UI toolkit that may not have implemented text dragging yet.

Theme problems

The default theme has very similar colors for the general background and the editor background. This, when combined with the minimalist design, makes it rather hard to differentiate between your file content and the file sidebar, especially when you are working with a Markdown file that is mainly white text.

File sidebar issues

The file sidebar has a number of issues that I want to point out. First, the sidebar is laid out in two sections. At the top is open editors; at the bottom is your filesystem view. I am used to this being the other way around; unfortunately, I can’t find any way to change this. To make matters worse, you can’t even resize the two sections. The default sizing leaves me able to see only three open files at a time. Moving on, there are only two icons used in the sidebar by default: one for folders and one for files. Thankfully, there is one extension that adds new icons (Material-themed) to Lapce, which add unique and colorful icons for many file types, but the default experience makes it extremely difficult to find files. To make matters worse, if a file has a name long enough to start overflowing the file explorer, Lapce will shrink and then remove the icon for the file before it will elide the file name. This makes it harder to distinguish files while providing no real benefit in terms of file name readability.

The command palette

The command palette is unusual, to say the least. To begin with, it uses Ctrl+Shift+P, which is a bit unwieldy, and simultaneously uses… F1?! F1 is recognized everywhere as the de facto Help key, not a command palette key! I tried rebinding to the more familiar and easier Ctrl+K, but it didn’t work. It turns out that Ctrl+K is used for shortcut layering. Fine, I thought, I’ll use Ctrl+K Ctrl+K. No dice. I eventually settled on Ctrl+Shift+K, but it feels like a suboptimal solution. Once I had my shortcuts in place, I found that your shortcut(s) will only open the command palette; to close it, you have to use Esc. This is bad design; menus should be closable with the same key that opens them (see: the Windows Start menu). Once you get the command palette open, it types a colon at the beginning, which limits you to commands. To search files, you have to remove the colon. Given that my main use for this sort of global search bar feature is quickly jumping to files or symbols, I’m not impressed. Finally, if you click on the command palette bar, you can’t type anything into it, not even Esc. You have to click outside the palette to close it and then open it with the shortcut.

Miscellaneous UI issues

While the file sidebar shows your open editors, there is also a tab bar at the top of the window with your open files. To me, this is a waste of precious vertical space that could be used to show more code. After all, in IDEs, vertical space is precious, and I don’t want to sacrifice visible lines of code to add a redundant open files view. On that note, Lapce scores poorly on visible lines of code. It shows just 52 lines at a time, while Visual Studio Code manages 64 and Qt Creator gets 66 (although I may have changed font sizes in the other two IDEs; I don’t remember). After changing the font size from the default 13 points down to a still-readable 12, Lapce still only has 58 visible lines of code.

There is a general lack of tooltips throughout the UI that makes it difficult to know what every button will do. Again, this may be due to the custom UI toolkit.

Markdown editing

This post was written in Lapce as I worked with the IDE; this gave me a great opportunity to try out Lapce’s Markdown support. Overall, it looks pretty nice. There’s good syntax highlighting, code blocks that have a content language specified have appropriate syntax highlighting applied to the contents, and there is even support for the TOML front matter that Hugo generates. However, I do have a few complaints:

Generally, IDEs don’t use word-wrapping by default, since it would make code extremely difficult to read. However, when you are working with Markdown, you’re not working with carefully formatted code. You’re working with paragraphs of text. And when you are working with paragraphs of text, word-wrapping makes it easier to read. Visual Studio Code word-wraps Markdown files for this reason. However, Lapce does not word-wrap Markdown. I’d love to see this fixed.

Inline code is very difficult to distinguish from regular text; the backticks are highlighted but the contents are not. I’d love to see the contents have either a different text color or a unique background color.

List handling is so-so; while most editors will auto-generate the next list item when you press Enter, Lapce does not.

Finally, Lapce has no Markdown preview support. To me, this isn’t a huge deal, but for some people it would probably be a deal-breaker.

C++ development

Upon opening a C++ project, I immediately noticed that Lapce supports C++ syntax out of the box. Nearly every character in every file has coloring applied to it. Some symbols such as braces, brackets, parentheses, and operators remain the default text color, but every identifier and value in the source has a color applied. As somebody who appreciates the value of colors to better communicate what’s going on in code, I am a huge fan of this. (In fact, I have changed the Qt Creator dark syntax theme to add more color; you can get it here.)

Unfortunately, there was not much more than syntax highlighting available. After searching fruitlessly for a menu entry to allow me to manage extensions, however, I found the extensions button at the top of the file view. Sure enough, there was a clangd extension available, so I installed it and… nothing changed, except for an error popup declaring that the clangd extension couldn’t retrieve an artifact. What? I eventually learned that it was trying (and apparently failing) to download clangd. Fortunately, you can configure it with the path to your already-installed clangd. After doing so, I was rewarded with a fairly usable C++ experience. Here’s how Lapce looks with the code for Alpaka open:

However, it wasn’t all smooth sailing. I immediately realized that Lapce displays inlay hints when it has enough information available to do so. I find inlay hints incredibly annoying, so I had to uncover the setting to turn them off. Another unfortunate realization was that the C++ syntax highlighting became less colorful with clangd enabled; however, there is still plenty of color in the IDE, so I don’t feel like that is a huge problem. Code completion was another rough patch; while Lapce gave me plenty of completions from clangd, I found that it wasn’t completely intelligent about doing so. For example, typing case Foo: in a switch statement would trigger a completion popup suggesting a variable. Normally, I would expect to be able to type a newline before any completion popups display; with Lapce, however, I have to press Esc after typing the colon.

It quickly became obvious to me that Lapce is not designed to create an extremely powerful C++ experience. As a simple example, while Qt Creator allows you to switch between headers and source files, there is no such functionality in Lapce. However, for a more impactful example, let’s look at the C++ build environment support. In IDEs like Lapce and Visual Studio Code, you are not given project support. You are instead given support for run configurations, which (in the case of Lapce) are TOML files that can contain commands for running apps. In my opinion, when you already have build systems like CMake, Meson, and Cargo that are designed to handle this for you, I shouldn’t have to specify this stuff myself. I should be able to simply have Lapce run cmake --build on my code when I ask it to build my app.

With all this being said, I realize that I’m giving Lapce a bit of a hard time here. It’s not intended to be a supercharged C++ IDE; it aims to be a replacement for Visual Studio Code, and it does well at that. It’s just that Visual Studio Code and similar editors have introduced a paradigm that I don’t appreciate.

Final assessment

Lapce is a promising IDE, and for a hobby project it’s at an impressive level of completion. Since it is clearly marked as being in beta, I am willing to forgive a lot of problems as bugs that are planned to be fixed before a 1.0 release.

However, I will not be switching to Lapce due to its overall lackluster C++ experience. Qt Creator provides a much more powerful experience for me; in fact, the only real gripe I have with Qt Creator is that it doesn’t support languages other than C++ and QML very well. This could technically be fixed by creating plugins for other languages, but Qt Creator’s plugin interface is not very well documented.

I do plan to keep an eye on Lapce. I dislike Visual Studio Code for its use of Electron and its invasive telemetry. If Lapce improves enough and gets a D plugin, I’ll probably switch my D developement to Lapce so I can ditch Visual Studio Code once and for all.

Finally, I want to give a shoutout to the Lapce developers for creating such a cool (and gorgeous) app. If any of the developers reads this, please forgive the ranty tone of voice and take this as constructive criticism :)

Let's Ban SMS 2FA

loren@lorendb.dev (Loren Burkholder) — Sun, 11 Feb 2024 16:08:00 -0500

There, I said it.

Now that I’ve got that out of the way, let’s talk a bit about why we should actually ban SMS 2FA.

A quick note: I’ve rewritten this post to present a more balanced view of the topic. You can find the old version on the Internet Archive.

The problems

SMS 2FA is something of a paradox in and of itself. Two-factor authentication is generally positioned as a way to increase your security; however, SMS messaging is inherently insecure, since it is unencrypted and SMS messages are publicly broadcast wirelessly. This means that bad actors who are physically near to you can trivially snoop on 2FA codes that are texted to you.

However, unencrypted message broadcasting is not the only concern here. SIM-swapping is another method that can allow an attacker to gain access to your SMS messages; in this sort of attack, the attacker calls your phone company, claims that they are you, and asks to swap your phone number over to their SIM card. While the FCC is taking action to fight SIM-swapping, it is still very much a relevant attack; in fact, last month (January 2024) the U.S. Securities and Exchange Commission had its Twitter (yes, I’m still calling it Twitter) account hacked in a SIM swap attack.

To be fair, SIM swapping is not a trivial attack. It generally requires social engineering to complete, which makes it less useful for drive-by attackers. However, eSIM swapping is starting to gain popularity. This attack is less complex to complete, as it doesn’t require social engineering the phone company. You just need to get the victim’s login credentials, log into their phone provider account, and swap the eSIM.

Companies don’t care

You might be wondering why this is such a big problem. Sure, SMS is offered as a 2FA method on many sites, but you don’t need to enable it, right? Well, it turns out that platforms like PayPal and Amazon don’t care that SMS 2FA is insecure. If you want to set up any form of 2FA for your account, you are forced to set up SMS as a fallback option. This makes their 2FA support little more than a farce, given that anybody attacking your Amazon or PayPal account can theoretically override your 2FA by intercepting your SMS. The reasoning for this? “If you lose your 2FA device, we’ll text you a code to recover access to your account.”

(A bit more on the account recovery: given a valid email address, PayPal will happily send you a text to verify a password reset, and Amazon lets you reset your password with just a phone number. This means that as long as you know that somebody has an Amazon account that may be linked to their phone number, intercepting their SMS could give you full access to their Amazon account.)

Even for the services that aren’t forcing SMS 2FA down your throat, SMS is still presented as an option on many sites. Its ease of use undoubtedly has led many people to say “Oh, I’ll just use SMS since it’s easier”, which leaves those people more vulnerable to attack.

Having said this, I must offer a commendation to Twitter. In 2023, they disabled SMS 2FA for anyone not subscribed to Twitter Blue, citing its insecurity as the reason. I’d prefer if Twitter had disabled SMS 2FA across the board, but even if they left it as an incentive for people to pay for Twitter, it’s still a welcome change.

It isn’t that hard to fix

Now, some people might start complaining that changing their SMS 2FA deployment will be a huge technical problem. No, it won’t. Companies have several options here:

Use TOTP; adding TOTP support is going to be fairly trivial, since it only requires an algorithm to calculate a code based on the original secret value (you’ll probably also want to throw in a QR code generator, but that’s technically optional). And it’s not like you’ll have to roll your own implemention; GitHub is full of TOTP libraries (and apps). This does require the user to install a TOTP app, but there are many trusted TOTP apps available, so that shouldn’t be a big problem.
Use passkeys, the latest and greatest in 2FA security. Passkeys are (as far as I can tell) sort of a software-defined WebAuthn key that operates like a hardware security key (as an aside, apparently Yubico stopped making these in blue). They are supported by modern browsers and phone operating systems, not to mention password managers like Bitwarden, and are generally designed to be extremely simple to use. Users generally shouldn’t need to install any extra apps to use passkeys.
If your 2FA flow is hardcoded in such a way that you have to use an SMS-like flow where you send the user a message containing a one-time code which the user must repeat back to you, you can at least swap SMS for a different service. Some platforms support email, but you could also use something like end-to-end encrypted Matrix chats, or you could even bake a 2FA sending service into your app the way GitHub does. If you really can’t drop SMS, at least try to upgrade messages to RCS whenever possible. (If you are using Twilio Verify, you are already being automatically upgraded to RCS.)

What we can do about it

How can we combat SMS 2FA? It turns out that there’s a surprising amount of things you can do.

Disable SMS 2FA for any account that will let you turn it off. Obviously, you’ll have to keep SMS 2FA on for Amazon and PayPal, but at least reduce your usage of SMS 2FA as much as possible.
Write to companies that force SMS 2FA and ask them to remove it.
Tell others about the risks of SMS 2FA. The more people know about this risk, the less the chance that one of them will fall victim to a SIM swap.
If you work at a company that has SMS 2FA deployed, ask them to remove it from your product. This is especially true if you work at PayPal, Amazon, or anywhere else that could be a likely target for attackers. This includes banks (yes, my bank forces SMS 2FA, and no, I’m not happy about it).
If you live in the US, write to your congressman to urge him to introduce legislation banning the use of unencrypted message-based 2FA (and mention the SEC Twitter hack to show that the government is susceptible to this attack). If you don’t live in the US, write to whoever represents you in your government (e.g. EU residents can write their MEP). While this may seem like a bit of a long shot, it’s possible that your message could result in an actual law. EU residents are especially likely to succeed, given the EU’s historical stance on legislating sane tech laws (i.e. the GDPR and the DMA).
Share this post, or if you have your own blog, write your own post on this subject! The more publicity this issue gets, the better.

The part where I backtrack a bit

Having said all of this, is SMS 2FA completely evil? The answer is actually complex. First, while SMS 2FA is insecure, it is better than no 2FA at all, since it can function as a deterrent to a casual hacker. Second, it is user-friendly. TOTP, passkeys, and physical security keys all tend to be more effort to learn to use than a simple SMS verification scheme. To successfully remove SMS 2FA, we’ll have to educate the general public on how to use the alternatives or create a truly simple-to-use alternative. I do think that passkeys could become a good solution, however, as device and browser makers have put a lot of effort into making them a seamless option.

Conclusion

I hope you enjoyed reading this little rant here. Hopefully, you were inspired to take action in some way to help remedy the SMS 2FA problem. If you sent a letter to your government representative or started a discussion at your company to try to remove SMS 2FA, please leave a comment below! If this post gathers enough attention and people start logging their actions below, I’ll create a page tracking all our SMS 2FA shenanigans to help keep tabs on the situation.

Archiving Steam games for fun and profit

loren@lorendb.dev (Loren Burkholder) — Wed, 03 Jan 2024 23:27:24 -0500

tldr: if you want to create an archive of all your Steam games, check out download-steam-games on my GitHub. It has minimal documentation, but you should be able to get it running without too much trouble.

I recently decided it would be cool to download some beta versions of KSP2 so I can revisit them in a year or so and laugh about how many terrible bugs there were in the original version of the game. I decided that it would make sense to download the game backup directly to my server, so I sshed into the server, installed Steam from the system repositories, and started a new ssh -X session. However, when I launched Steam, I realized that my server’s connection to the internet is much too slow to run ssh -X in any sort of usable state. In fact, it was so slow that it apparently timed something out on Steam’s end and made it practically impossible to log in to Steam.

Undaunted, I started looking into SteamCMD. It turns out that while SteamCMD is intended for game server administrators, it will happily download any Steam game. It didn’t take me long to install it and use it to download a game. However, it’s a bit of a pain running through a set of instructions in SteamCMD again and again for downloading multiple games, so I set out to script it.

But why would you do this?

Why wouldn’t I do this? If you think I’m crazy, just go read r/DataHoarder for a while.

Seriously, while I don’t expect Steam to disappear tomorrow, there is precedent for games to be removed from the platform; many of the original games on the platform have been removed. The oldest game still available (counting by which date it was made available on Steam) is the original Counterstrike with a game ID of 10. Games 1 through 9 aren’t on the platform anymore, and I’m sure they are far from the only games deleted from the platform. Archiving games allows me to avoid losing games that are removed from Steam. Also, sometimes enthusiasts like to be able to access old versions of games; one of the top posts of all time on r/DataHoarder is about somebody who hoarded an old Minecraft alpha, giving Minecraft enthusiasts a chance at taking a more complete look at the evolution of the game.

Update (2/23/2024): it turns out that I am wrong on this point. Games may be delisted from Steam, but they are not deleted. If you own the game, you can still download it. Also, since Steam numbers games by tens, the original Counterstrike is the first listed game on Steam. Apologies for this!

A naïve bash script

My first approach was to create a bash script to wrap SteamCMD with a nice syntax. Of course, I also wanted to bypass the interactive prompt. Let’s go through what I ended up building:

STEAM_ACCOUNT=
GAME_NAME=
GAME_STEAM_ID=
GAME_FORCE_WINDOWS=
GAME_BETA=

These variables control some basic stuff. STEAM_ACCOUNT is the account you logged into SteamCMD with; while SteamCMD caches credentials, you still have to issue the login <username> command every time you start it, so the script has to know what username to use. GAME_NAME doesn’t have to be the actual game name; it’s just used to generate the filename for the final game archive. GAME_STEAM_ID is the numerical ID of the game you want to download; for example, KSP2 is 954850.

GAME_FORCE_WINDOWS deserves a more detailed explanation. While I run Linux, some games do not have Linux builds available. For those games, I would like to save a backup of the Windows build instead. This will let me run the game via Proton later. Conveniently, SteamCMD allows you to override what platform you download the game for by setting @sSteamCmdForcePlatformType <platform>. GAME_FORCE_WINDOWS is used later as part of the SteamCMD script contents; therefore, if I want to add a Windows override, I can set GAME_FORCE_WINDOWS to @sSteamCmdForcePlatformType windows.

Finally, we have GAME_BETA. SteamDB shows beta version identifiers for games; you can use those to download earlier versions of some games. This is something I want to use, given that the original purpose of this project was to archive early KSP2 builds.

The next bit of the script is simply logic to set each of these variables based on command line parameters. I’ve decided to skip it for brevity, as it is trivial bash logic. Moving on, we get this:

cat > ~/.download-$GAME_NAME.txt << EOF
$GAME_FORCE_WINDOWS
force_install_dir $HOME/Steam/downloads/$GAME_NAME
login $STEAM_ACCOUNT
app_update $GAME_STEAM_ID $GAME_BETA validate
quit
EOF

SteamCMD can be controlled by scripts. This is how we’re going to get around its interactive interface.

The first line of the script is the aforementioned platform override. After that, we use force_install_dir to force SteamCMD to download the game to a known location. In this version of the script, I’ve hardcoded it to ~/Steam/downloads, since I’ve installed SteamCMD to ~/Steam. Then we log in and issue the actual command to grab the app; once the app has downloaded, we quit SteamCMD.

PREVIOUS_DIR=$(pwd)
cd ~/Steam
~/Steam/steamcmd.sh +runscript $HOME/.download-$GAME_NAME.txt
cd $PREVIOUS_DIR
rm ~/.download-$GAME_NAME.txt

Here we actually execute the script and then delete it to prevent clutter.

PREVIOUS_DIR=$(pwd)
cd ~/Steam/downloads
tar --use-compress-program=pigz -cf $PREVIOUS_DIR/$GAME_NAME.tar.gz $GAME_NAME
cd $PREVIOUS_DIR

rm -r $HOME/Steam/downloads/$GAME_NAME

The final step is to compress the game folder into a single .tar.gz file. Again, we’re dumping the final archive into ~/Steam/downloads for now. The only really unusual part of this is that I’m telling tar to use pigz to do the gzip compression. This is because pigz will use all CPU cores, which is important when you’re archiving many gigabytes of game data.

This is all fine and good, but you still have to manually run the command for each game. To mitigate this, I created a simple script that read a vaguely CSV-like file which listed games to download. That was definitely better than nothing, but it still felt pretty hacky.

Building it the right way

I decided to port my two scripts into one app. I chose to write the app in D; in retrospect, Python would probably have been a good choice simply because it comes preinstalled on pretty much every distro, but I much prefer D for pretty much anything. My ported script is pretty similar to the functionality shown above; however, I added various improvements. Paths are no longer hardcoded, it’s possible to download games for any and all platforms (Windows, macOS, and Linux), and game data is stored as a JSON file instead of in a weird custom-ish format.

The app, which I’m calling download-steam-games (how original), is available on GitHub. Building it is a simple matter of running dub build. After that, you need to run dub run -- --add-game to add a game to the download list. After you’ve added some games, you can use dub run to run the application in download mode. It will download all your games for every platform you’ve specified, .tar.gz them, and put them in the output folder of your choosing.

Limitations

The app currently is somewhat limited: there’s no functionality to reconfigure the settings, you can’t specify beta versions with --add-game, and there’s no progress reporting during download. However, I think it’s robust enough to actually use in production. I hope to address all of these issues soon, and while I’m at it, I’d like to add a feature to add a timestamp to the archive name so you can download many versions of the same game over time.

Conclusion

This is a fun project that is helping get me hooked on data hoarding. If you start using my app to help hoard your own games, please leave a comment below! I’d love seeing how many gigabytes (or terabytes) of games you’re hoarding.

Updates

Shortly after writing this post, I’ve added some logging to the app. It ain’t pefect, but it ain’t terrible either.

Thoughts on KSP2 For Science!

loren@lorendb.dev (Loren Burkholder) — Tue, 26 Dec 2023 09:20:21 -0500

Kerbal Space Program 2 had its first major update, called For Science!, released a week ago (as of my writing this). As a KSP nerd, I downloaded it and ran it that day. Since then, I’ve done some messing around with it, and I thought I’d write down my impressions.

The good

There’s so much to love about the For Science! update. As the first Early Access milestone, it brings a new gameplay mode called Exploration mode, which is similar to the Science mode from the original Kerbal Space Program. And boy, has science been improved! In KSP1, science was often cumbersome; to collect all your science points, you needed to track every time you entered a new biome and manually run all of your experiments. In KSP2, you just add one science part to your craft (the Science Jr. is a great example) and start flying. Science collection runs automatically for you, meaning that you won’t have to wonder if your orbit just crossed another biome. Granted, some contracts require you to take a Kerbal on EVA and manually make an observation, but in general, science is very passive.

And speaking of contracts, that system is also improved. I haven’t personally played KSP1 in career mode (or at least not any farther than one or two launches), but I’ve heard that KSP1 contracts were often bland: you might be asked to launch a vehicle above 1000 meters. In KSP2, you immediately get believable contracts like proving that you can get to space, then orbiting, then going on EVA, and so forth and so on.

Another huge part of For Science! is the addition of a thermals system. The KSP2 team has created a beautiful reentry effect. Here’s the thermals on reentry from Minmus (this was after a number of aerobraking passes, so my actual velocity wasn’t too high):

Isn't it just gorgeous?

While some people have reported bugs with thermals, I’m happy to report I haven’t encountered any yet.

There’s a lot more to cover, but I’ll cut this section short in interest of keeping this post readable. However, I’ll at least mention a few: wobbly rockets are fixed (!!!), there are many more easter eggs (or discoverables, as they are now called), the boat dock has had a facelift, and buoyancy works, just to name the most obvious ones.

The bad

KSP2 For Science! hasn’t been all sunshine and roses, however. There have been some bugs pointed out, but I’ve personally experienced two of them.

The first is orbital decay. In real life, given enough time, any orbit will decay to the point that the orbiting object crashes. In KSP and KSP2, orbital decay is not implemented; players don’t want to have to manually boost their satellites and space stations again and again. However, at release, KSP2 players found that if you orbited the right planets or moons at a low enough altitude, your orbit would decay. The KSP2 team diagnosed and fixed the issue, or so I thought. However, my first Minmus mission in For Science! proved otherwise. Orbital decay is still an issue. Thankfully, you can prevent orbital decay by dropping into timewarp when possible, but in my opinion, this should be a top-priority bug for the KSP2 team.

The other major bug that I’ve experienced is disappearing orbital lines. Normally, when you open the map view, you see a line that marks your orbit. However, KSP2 has a bug that will sometimes set a craft’s state to “Landed” when it is clearly not landed. When a craft is in the Landed state, it does not have orbital lines drawn. I experienced this during an extremely simple mission to low Kerbin orbit and back; there were no quicksaves or quickloads in the mission. At some point during ascent, I realized that my orbital lines were gone. Due to the simplicity of the mission, I was able to complete it without orbital lines, but for more complex missions, you would have to quicksave and quickload to continue. However, sometimes reloading the game doesn’t fix the bug; in that case, you have to shut down the game and manually edit the craft file to override the vessel state.

The ugly

I have a few things to point out here that fall into the “ugly” category. First, camera behavior is buggy. When you switch to the map view and back, the camera always resets its view. This isn’t too bad by itself, but the default camera view for large craft will often be clipped into the middle of the vehicle. This means that every time you switch to the map and back, you have to zoom out a bunch to see where you even are.

Second, sometimes clouds render as black during the night. I will add a massive disclaimer that I only saw this once, and I didn’t try to reproduce it, but the fact remains that as I landed, the clouds were completely black, which didn’t happen before For Science!

The final ugly thing? Jebediah Kerman is no longer selected by default as the first kerbonaut on any mission! What is this world coming to?!

The takeaway

Is KSP2 ready for mass adoption by anyone and everyone? Until orbital decay and disappearing map lines are fixed, no. I can’t recommend a game that has such serious gameplay-affecting bugs. However, I am optimistic that the KSP2 team is working on fixes for these and other problems. Who knows? I have a friend who’s waiting for KSP2 to have the bugs fixed before he dives into the Kerbal universe for the first time; maybe in a few months I can tell him that KSP2 is ready!

If you work on KSP2

If any KSP2 devs read this, first, thank you for making a great game! I personally have entirely switched from KSP1 to KSP2. However, it’d be awesome if you’d pull the right strings to get those bugs fixed. :)

Grouphosting

loren@lorendb.dev (Loren Burkholder) — Fri, 22 Dec 2023 15:32:27 -0500

I’ve been diving into the world of selfhosting recently, but there’s a catch: it’s not just me. I’m doing it with some friends. This, of course, is fun, but I think the term “selfhosting” is a bit of a misnomer here; furthermore, I think that there is a powerful new type of selfhosting here that could change the way people selfhost.

Grouphosting

I’ve been calling this concept “grouphosting”. What is grouphosting? It’s fairly similar to selfhosting, except there are multiple administrators. What advantages does this have? The immediately obvious answer is that server problems can (hopefully) be resolved more quickly; the more administrators you have in your group, the more likely it is that one of them will be available when the inevitable server problem occurs.

However, there’s another powerful feature that is available only to grouphosters: resilient backups on a budget. In commercial environments where money is not a big problem and preserving your data is vital, sysadmins set up 3-2-1 backups for maximum resiliency, which includes geographically separate data copies. Some companies already have multiple locations available to locate backup servers; others simply purchase storage from providers like Backblaze or AWS to give themselves a separate datacenter. However, for individual selfhosters, geographically redundant might mean “My server rack in the basement syncs to an old NAS in my attic.”

When you grouphost, you are involving multiple people who live at separate geographical locations. Therefore, even if not everybody in the group is capable of hosting a server at their place, you hopefully will end up with at least two people in your group who are capable and willing to put a server in their house. At this point, all you need to do is provide the servers and you’ve achieved resilient backups without paying a cloud hosting company!

Closely related to this is the concept of load balancing. For the purposes of this blog post, let’s say you’re deploying Matrix. Matrix doesn’t need a lot of resources for an instance that will only serve one user; a few gigs of RAM and a fairly low-power CPU would suffice. However, assuming you’re selfhosting on a physical machine at your house, you probably have more than just a few gigs of RAM - even the Raspberry Pi 5 comes with a minimum of 4 GB. If you’re using an old desktop or a dedicated server, you almost certainly will have 8 GB or more. This is plenty of resources to run a Matrix instance with more than just one user. By grouphosting, you can keep your machine just to a Matrix server, but now you’ll be utilizing its extra capacity with accounts for your entire group, and in return you’ll get access to the other services the group is hosting.

I don’t trust my friends on my server!

I understand exactly where you’re coming from. It’s good to keep integrity of your system at the forefront; if you aren’t comfortable with grouphosting, it’s your right to refuse others access to your server.

What if I can’t find anybody at all?

That’s OK! Some people may need to just selfhost. I just think that it makes sense to grouphost when possible.

I bought a domain that is based around my name. Does the whole group have to use it?

If they want to, sure; if they don’t want to, tell them to buy their own domain. After all, you can get domains for a lot of great TLDs (including .com, .xyz, .dev, and .foo, to name a few) for around $10 per year or less, so it won’t break the bank to share the cost of a domain.

Of course, you can always host your own stuff (like a blog) under any domain you want.

OK, you’ve got me convinced. How should I start?

If you know anybody who selfhosts, ask them, but be prepared for rejection: they may not wish to migrate to a new domain. Otherwise, talk to all your friends that you would trust with your server. Once you’ve got a few people on board, buy yourself a server or two. I personally scored a well-equipped Dell PowerEdge R520, a server rack, and a few other things for $650 on Craigslist. Sites like eBay and The Server Store are also great resources for finding cheap used servers. Of course, you could also find cheap desktop PCs instead of rackmounted servers. ServeTheHome is an awesome resource for finding good home server hardware, and they even have a dedicated forum section for sharing great deals on server hardware.

Once you’ve got a server or two and have put them at people’s homes, it’s pretty similar to selfhosting. Just make sure to give the right people administrative access to your installed services. You should also probably write down some details of everything you install; that way, if anything crashes, any administrator can read your documentation to help troubleshoot the problem.

Once you’re done setting up your server(s), don’t just rest on your laurels. Be on the lookout for other potential grouphosters. If you have done your grouphosting right, you should be able to loop in a new grouphoster with minimal effort. Ideally, you should even be able to put the new guy’s server to use, if he has one.

Fin

I hope that at least some part of this made you go “yeah, that actually makes sense”. While I doubt this post will have a huge impact, it might start some other person out into their own grouphosting, and if that happens, I’m happy. :)

Hello World!

loren@lorendb.dev (Loren Burkholder) — Fri, 22 Dec 2023 10:47:01 -0500

I’m setting up this blog to try to start writing down the occasional thing that might be useful to others.