Linux

source ~/.bash_history

Perhaps not the same definition of "broken" that you're looking for, but when I first started using Linux, I was using Kubuntu as my first distro have some brief experimenting with Manjaro.

Anyway, back then, I for some reason had the Skype snap installed. Can't recall why I had it to begin with, but I decided later on that ofc I didn't need Skype, and of course uninstalled the snap.

A few days later, I was met with some storage issues, where I had a limited amount of storage left on my SSD. I'm sitting there a little confused since I swore I was using less storage, but I did a thorough cleaning of my computer by deleting files I didn't necessarily need, and uninstalling any programs that I hardly ever used. That seemed to do the job, even if it was less storage space...

Until the next day, when the storage was full again. After getting some help from someone, I found that Skype, despite being uninstalled, was still running in the background, and found that there were residual files. The residual stuff running in the background was trying to communicate with what I had uninstalled, and logged multiple errors per second in a plaintext file that ended up being 176GB.

Whether I did something wrong or if there was something up with the snap, I still don't know as this was over a year ago and I was still learning the ropes of Linux at the time.

Connect via ssh to my home server from work

Using a cli torrent client to download stuff

Decide I need a VPN.

Install VPN again from CLI

Run VPN which disconnects my ssh connection

Even when I get home, the server is headless so I have to locate a keyboard and mouse before I can fix.

Not strictly Linux related, but in college I was an IT assistant. One day I was given a stack of drives to run through dariks boot and nuke.

I don't remember exactly what happened, but I think midway through, my laptop shut off.

Guess who picked the wrong drive to wipe with DBAN :)

sudo rm -f /lib /usr/share/backup/blah blah.tar.gz

Note the space.

I set up 2FA via a hardware security key (a yubikey) for login, sudo etc. I then tried to switch security keys, removing the old pam files and adding a new one. But I didn't tidy the pam files up before logging in, and there was effectively no way to log in, since editing the pam files required sudo access to edit in the first place. So basically the whole system required access to a pluggable authentication module that it no longer had any ability to recognize. It was honestly pretty funny. I did manage to recover my data by booting from a live system and decrypting my drive from there.

I've also accidentally removed my desktop environment twice while trying to update Python versions and then cleaning up old packages, but that's kinda not that big deal and is just a facepalm moment.

Writing and running a script to delete the first 2 characters from all files and folders recursively.

It started backtracking to my home folder. :/

sudo apt autoremove

Who ever made this shit and then decided to always show you this message that you should do it. What a dick

Many many years ago I wanted to clean up my freshly installed Slackware system by removing old files.

find / -mtime +30 -exec rm -f {};

Bad idea.

It was my first time using a Linux GUI. I was comfortable with CLI, but it was my first time having it installed on a laptop instead of just sshing into a server somewhere.

So naturally, instead of learning how the GUI worked, I tried changing it to be exactly like Windows. I was doing things like making it so I could double click shell scripts and other code files and they would run instead of opening them up in an editor. I think you see where this is going, but I sure as hell didn't.

Well, one of my coworkers comes over and asks me to run this code on this device we were developing. We were still in the very early stages of development, we didn't even have git set up, so he brought the code over on a USB stick. I pop it into my laptop. I went to check it once by opening it in an editor by double clicking on it... Only it ran the code that was written for our device on my laptop instead of opening in an editor.

To this day, I have no idea what it did to fuck my laptop so bad. I spent maybe an hour trying to figure out what was wrong, but I was so inexperienced with Linux, that I decided to just reinstall the OS. I had only installed it the day before anyway, so I wasn't losing much.

One day on my main Arch installation I created a container inside a directory, and "booted" into it by using systemd-nspawn. When I was done with it I decided to do a rm -rf / inside the container just to be funny. Then I noticed that my DE on the host froze and I couldn't do anything. Then I realized that systemd-nspawn mounts some important host's directories on the container, and I deleted those when I did the rm -rf /. I didn't lose anything, but it was scary.

One time I rebooted. The system never recovered.

So I am sort of an embedded developer, and I like to mess around with weird configurations. So the craziest experiment I did was trying to reflash a rasberry pi from a system running in the pi's RAM. It honestly might have worked, but during the prep work I forgot to resize the filesystem before mucking with the paritions and had to reflash the normal way before I could try again. Ended up just turning it into a pihole instead, but I still learned a lot about pivot_root

Installed python3 before it was made the native python on the dist. Half broke everything, including apt & python. So I uninstalled it, and then everything was broken. Finally got python3 reinstalled, and lived with it kindof working & awful distribution updates.

I have finally freed myself of that prison last month, by nuking everything and starting fresh.

I've literally done the rm -rf / thing. I thought I was in a different subdirectory, but I was in / and did rm -rf .

When it didn't return after half a second, I looked at the command again and hit CTRL+C about 20 times in the span of 3 seconds.

I had to rebuild the install, but luckily didn't lose anything in /home.

I can't even remember how I did this, but overwriting the partition table on the main production server at our small startup (back when "the server" would usually live on the premises of the startup). I remember my boss starting to hyperventilate from panic while I reconstructed it from memory / notes, and all the filesystems came back and he calmed down.

Same job, they gave me a little embedded-systems unit for me to use to build a prototype on. I hooked it up, nothing worked. I brought it back to them.

Hey, this one doesn't work.

Huh... that's weird, it was working before. Did you break it?

I don't think so. Can I have one that works?

They literally told me, as they were handing me the second one: Okay, here's another one. Don't break it.

I figured it out literally seconds after breaking the second one... I was hooking it up to 12 volts of power when it needed 5. Second dead computer. Explaining that and that I needed a third one now was fun.

I once did an apt-get upgrade in the middle of when debian testing was recompiling all packages and moving to a new gcc version. I get it, using testing invites stuff like this. But come on, there should at least be a way to warn people beforehand.

I set up a progressive backup of my home folder... to my home folder. By the time I got home that day it was impossible to log in because there was no room to create a login record. Had to fix that by deleting the backup file using a live CD.

Oh, i have a brilliant one:

A few years ago i spent a lot of time converting .flac-files into .ogg-files in order to put on my oldschool iPod. As I did a lot of repetitive typing - entering $dir / for file in flac ; do convert etc / mkdir -p $somewhere/$artist/$album / mv $somewhere/.ogg->$new_dir/ and so on - I thought: "hm lets just write a loop over loops for all the artists here and then all the albums and at the same time create the nested directories somewhere else... hm actually in the home directory.... and later love everything on the iPod at once."

so i was in my music folder with the artists-folders i wanted to convert. i did something wrong

So i did my complicated script directly in the shell. I made something wrong and instead of creating a folder "~/artist/album" I created 3 folders in my current working directory: "~", "artist" and "album". hmph dammit gotta try again... but first : i have to clean up these useless folders in the current dir. so i type of course this: "$ rm -r ~ artist album " after about 5 seconds of wondering why it took so long i realized my error. o_O I stopped the running command, but it was (of course) too late and i bricked my current installation. All the half-deleted config files made or impossible to start normally and extremely tedious to repair it by hand, so i reinstalled.

I uninstalled Python.

I was playing around with Pygame of all things, and it wasn't behaving as the (apparently out of date) documentation was saying it should, so I figured I'd just uninstall and reinstall Python.

EVERYTHING borked. APT wouldn't even work.

I cant remember anymore... Let me explain ... My first computer was with at-the-time-very-new windows xp, using primary for games, after some time it got bloated with stuff so i had to reinstall again and again over time. Then i discovered redhat,centos and debian... I started heavily distro hopping. My passion for software grew to the point that I was installing new software on daily basis, just to explore new things. But nothing seemed stable enough, ubuntu, fedora, sabayon, gentoo, arch... And their derivatives all broke under my fingers to the point that i had to do more fixing than discovering new software, I took it as a challenge and continue. At around the time of university I discovered NixOS, as with any new technology I went head on with it. It took a lot of trial and error since at the time there were no documentation for any of it. I spent months reading the code, but I never gave up, since what I have found was a gem. I found the OS that is resistant to my curiosity, I just cant seem to be able to break it. Now I use NixOS everywhere that I can, even on my work computer. I do not need to reinstall after initial installation. Well... only when hardware fails...

Tried to convert Ubuntu to Debian by replacing the repos in sources.list and apt dist-upgrading. 💣 Teenagers...

I had issues with a new version of glibc that prevented me from working on music in Ardour on Manjaro. I then proceeded to force-downgrade glibc (in the hopes of letting me get back to work) and that broke sudo and some other things, which I found out after rebooting. That was an interesting learning experience. Now I snapshot before I do stupid stuff. :]

Nooo I have so many.. This one I can explain in English:

Xubuntu but blind

So, this is ~2016. Ubuntu is hip and a handful of my students use it. On my PCs I only use Debian and Suse. So to help them better I take out an old ASUS laptop and install Ubuntu on it. Try out Xubuntu instead.

At that time I was also huge into alternative keyboard layouts. I had a slightly modified Neo keyboard layout installed when I switched to Xubuntu.

Here the fun starts because the obscure internal graphics card built into the laptop didn't have driver support under Xubuntu. Black screen but I could hear it working. This was the hardest driver fix I ever did. No monitor and a keyboard layout I wasn't used to, under a Linux distro I wasn't used to. And I also was at the university library, so no hardware support or Debian stick in reach.

Not quite catastrophic but:

I'm in the process of switching my main server over from windows to Linux

I went with Deb 12 and it all works smoothly but I don't have enough room to back up data to change the drive formats so they're still NTFS. I was looking at my main media HDD and thought "oh, I'll at least delete those windows partitions and leave the main partition intact."

I found out the hard way that NTFS partitions can't just reclaim space like that. It shuffles all the data when you change the partition. It's currently 23 hours into the job and it's 33% done.

I did this to reclaim 30 MB of space on a 14 TB drive.

sudo rm -r /run/timeshift

Accidentally executed a JPEG (on an NTFS partition) and the shell started going crazy. reboot was not successful =[

I used to work at this place that had a gigantic QNX install. I don't know if QNX that we used back then had any relation to q&x now They certainly don't look very close.

It was in the '90s and they had it set up so that particular nodes handled particular jobs. One node to handle boot images and serve as a net boot provider, one node handled all of the arcnet to ethernet communication, one node handled all the serial to mainframe, a number of the nodes were main worker nodes that collected data and operated machinery and diverters. All of these primary systems were on upper-end 386s or 486s ,they all had local hard disks.

The last class of node they called slave nodes. They were mainly designed for user data ingest, data scanning stations, touch screen terminals, simple things that weren't very high priority.

These nodes could have hard discs in them, and if they did, they would attempt to boot from them saving the net boot server a few cycles.

If for some reason they were unable to boot from their local hard drive, They would netboot format their local hard drive and rewrite their local file system.

If they were on able to rewrite their local file system they could still operate perfectly fine purely off the net boot. The Achilles heel of the system was that you had no idea that they had net booted unless you looked into the log files. If you boot it off your local hard drive of course your root file system would be on your local hard drive. If you had net booted, and it could not rebuild your local file system, your local root file / was actually the literal partition on the boot server. Because of the design of the network boot, nothing looked like it was remotely mounted.

SOP for problems on one of the slave nodes was to wipe the hard disk and reboot, in the process it would format the hard drive and either fix itself or show up as unreliable and you could then replace the disc or just leave the disc out of it. Of course If the local disk had failed and the box had already rebooted off netboot without a technician standing there to witness it, rm -Rf would wipe out the master boot node.

I wasn't the one that wiped it, but I fully understand why the guy did.

Turns out we were on a really old version of QNX, we were kind of a remote warehouse mostly automated. They just shut us down for about a week. Flew a team out. Rebuilt the system from newer software, and setup backups.

A regular update I guess...

I recently broke the networking stack by uninstalling ca-certificates

I was using a slightly risky command to delete unneeded packages, and for some reason ca-certificates was on the list

At least the fix was simple. Boot the rescue iso and reinstall them

I'm not sure how funny this will be, but here's how I broke my system twice in a single case. Step by step:

1. Migrated from Manjaro KDE to EndeavourOS KDE. Kept the previous home directory.
2. After a few updates, there was a problem with Plasma. Applications were not starting from the panels or the .desktop files (they worked from the terminal. The terminal emulator was in startup and worked that way)
3. After a few google searches, found out that downgrading glibc would do something, so downgraded... Worked for a while
4. While using pacman -Syu, I always checked for warnings (foolishly thinking that the downgraded and ignored glibc would cause a pacman warning if it broke dependencies) and there were none. So, the updated OS stopped working due to unmatched glibc. BREAK 1
5. To fix it, I opened one of my multiple boots (another EndeavourOS) and made a script using pacman -Ql and cp to copy new glibc related files into the broken system (because I was too lazy to learn how to do it the correct way with pacman and chroot didn't work because glibc is needed by bash).
6. Turned out the script I made was wrong and I hadn't checked the intermediate output from pacman -Ql, which was telling cp to copy the whole /etc /usr and other directories. (just if I hadn't given the -r to cp) BREAK 2

In the end, I just made a new installation, this time with a new home and hand-picked whatever settings I wanted from the previous home, Viva la multi-HDD

First, the classical typo in a bash script:

set FOLDER=/some/folder

rm -rf ${FODLER}/

which is why I like to add a set -u at the begining of a script.

The second one is not with a Linux box but a mainframe running AIX:

If on Linux killall java kills all java processes, on AIX it just ignore the arguments and kill all processes that the user can kill. Adios the CICS region 😬 (on the test env. thankfully)

The first time I wanted to try Linux I did by installing elementary OS in dual boot mode (with windows) and everything went well, I played with it a bit and then I returned to Windows..

So, few days after that I realize that I have a lot of space in the Linux partition and I didn't have plans to use it anymore so I go to drive's & partition's manager on windows to delete my elementary OS partition..

Oh Lord when I restarted my PC, grub was showing nonsenses and I couldn't boot on windows again, I was in panic, I spent the rest of the day trying to fix grub to boot windows. At the end of the day I did it and save all my files and I uninstall grub properly, but what a day 😂

Back when I started using Linux, I really wanted something that was super different from windows (I used Gnome 3 for like 3 years). I decided one day to try out Fedora cause, hey, I can live on the bleeding edge.

Second day I had it installed, I was having issues with the audio. Decided to try reinstalling pulse. Apt autoremoved it and somehow completely nuked the entire GUI. Stuck in terminal mode, I found that I had no ethernet to connect to, nor could I figure out how to connect to a wifi network with a password or download packages to a USB. After a couple hours, I gave up, wiped the drive, and went back to Mint.

Nowadays I'm happier in my little comfort zone.

First time trying Linux I went with an arch install because I Googled "best version of Linux" and went with arch. Followed a guide to the point of drive formatting and I decided to go with a setup with drive encryption. I didn't understand what I was doing, ended up locking myself out of my hard drives and couldn't get windows to reinstall on them. I used a MacBook for a week until I installed Ubuntu and managed to wipe and reset my drives and reinstalled. Needless to say I am going to read up a little more before I try that again.

CTRL-C-ing apt because it looked stuck for more than 10 minutes. I don't recommend doing it.