Could it be a failing/failed drive? Maybe try making a new bootable flash drive and see what happens.
this post was submitted on 29 Jul 2023
14 points (88.9% liked)
Selfhosted
40006 readers
716 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
I could format it and try but the moment I do I lose the ability to debug the issue and learning from the problem. So I may temporary solve it but it may happen again. I wonder if someone knows what I can check/test/run to identify what broke.
Valid points. I was thinking of using a second drive. That way you’d still have this one to troubleshoot and learn from.
Sorry to hear that, booting problems suck and are horrible to debug.
The next step I would try would be to boot an other install, like a liveusb or a raspbian, on the same usb port, to completely eliminate a hardware problem if it boots properly.
If it is a software problem, it seems to happen very early in the boot process, so my bet would be a corrupted initramfs/initrd (or what is equivalent on a Pi). No idea how you could debug and fix that on Ubuntu, though (especially on a Pi where /boot is… different). Maybe installing an other Ubuntu on an other disk or stick, then copy the boot files (making a backup of the original files to restore them if it doesn't help, no need to pile up possible causes of trouble). Throwing it here in case other people know better.
Thanks for the quick reply 👍
The next step I would try would be to boot an other install, like a liveusb or a raspbian, on the same usb port, to completely eliminate a hardware problem if it boots properly.
Good advice. I moved the usb drive from another (working) PI and attached it to the same USB port. It boots correctly. It is not the USB port nor power.
If it is a software problem, it seems to happen very early in the boot process, so my bet would be a corrupted initramfs/initrd (or what is equivalent on a Pi). No idea how you could debug and fix that on Ubuntu, though (especially on a Pi where /boot is… different).
I believe it is something like that. Or it is not mounting the drive correctly and not finding it, or it is something else. I just wish there was a better (or any) error printed on the console. I tried to attach a keyboard to get to a shell with no success. I honestly could just reformat the drive and use a clean install, but it is the last resource. I would like to understand what happened so I can learn from it and avoid it in the future (or learn a path to fix it).
I believe it is something like that. Or it is not mounting the drive correctly and not finding it, or it is something else.
Yes, that's what I had in mind. I already had similar problems with initramfs, because it was responsible to load the drivers needed for mounting the disk where the kernel was, so a bad initramfs caused the boot to fail from the get go failing to mount the partition.
That being said, I've looked at my Pi, and I have no idea what would serve as an initramfs in all those Pi specific files in /boot, if any. If you really want to understand what happened, I guess a possible path would be to find resources on the web explaining in details the Pi boot process, since it's different from usual linux boot process (it's not just the Pi either, I played with other ARM devices, like the stuff from pine64, and they all had their own way to boot).
Hi, just writing it here in case you were curious of the cause of the problem.
I finally had some time today to work on it. What I tried was to copy the content of the system boot partition of another PI, one that I configured at the same time/day, and compared file by file with the content of the partition in the broken one. Now there were some diff in the some binary files as expected, but what surprised me was that one file was ONLY present in the working one... initrd.img XD Now, don't ask me why the hell the file was not there. Maybe it got corrupted somehow, since nothing touches this partition as far as I am aware aside at boot time. Luckily, there was .bak file present in the partition which I renamed and... it worked.
Lesson learned: I now have a copy of the boot partition of every PI I managed (it is only 167MB pre tar-zip, so it is not a cost) and I have it backed up safely on another system. Should another file get corrupted in the future (maybe this time without a .bak) I have an older working copy if it, and I can restore the service without the need to format everything.
Thank you so much for your help and advice!
Ahah, so that's what initrd are called on Pi. :P Good catch!
Funny enough, I don't have such file on mine, the only *.img I have is the kernel, kernel8.img. I guess it's OS specific.
The .bak things sound like an interrupted update, or something. Like if the updater moved the current initrd as a backup file, then started building the new initrd and crashed or was rebooted before completion. That's what I really dislike about automatic updates, I prefer to be sure to know it's running, and see the output. :)
Congrats on sorting it out!
This might be completely diferent thing and it could be fixed already. Few months ago there was an update (debian bullseye i think) that messed up usb IDs on rasberries. It was mainsail os for 3D printer in my case and there was a fix that I never got working so I reinstalled OS and didnt update after that. I believe it was something with uuid, but better check yourself and never trust my words lol I hope someone knows more about that because it reminds me on the pain I had
view more: next ›