Chris: Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.
Wes: My name is Wes.
Brent: And my name is Brent.
Chris: Well, hello, gentlemen. Coming up on the show today, it's a big one.
Chris: Kent Overstreet, the creator of BcacheFS, joins us to give us a full update
Chris: on what's going on, what's new, what's next, and a surprising upside of getting
Chris: kicked out of the kernel.
Chris: And then we're going to round it all out with some great picks,
Chris: some boosts, actually like a really loaded pick section, and a lot more.
Chris: So before we get to all of that, let's give the old time appropriate greetings
Chris: to our virtual lug. Hello, Mumble Room.
Chris: Hello. Good afternoon. Hello. Just a couple of you in the on-air and a group
Chris: of you up there in the quiet listening as well. Mumbarum always cooking.
Chris: We're always happy to have everybody in there. And a big good morning to our
Chris: friends over at Defined Networking.
Chris: Defined.net slash unplugged. Go check out Managed Nebula.
Chris: It's built on top of the open source Nebula project. They handle the tricky
Chris: bits that you might not want to think about.
Chris: 100 devices totally for free.
Chris: No credit card required. Defined.net slash unplugged. this
Chris: is a mesh vpn optimized for speed simplicity and
Chris: really great security but ultimately it kind
Chris: of goes a little bit beyond when you're picking your network infrastructure if you've
Chris: been in tech for a minute you know that there's also the company behind the
Chris: product and what they're all about and what their goals are and i think the
Chris: way they're configured the way they're designed all of that stuff ultimately
Chris: ends up getting represented long term in the product and that's what I think to find has been really,
Chris: really judicious in how they have built and designed themselves to really align
Chris: with how I would like a company that's building this type of tool.
Chris: And I think you would also probably align with that if you're a listener to
Chris: this show. So it goes just beyond the software and the fact this is a truly
Chris: free project that's totally self-hostable if you'd like.
Chris: It's optimized for speed, simplicity. It uses less battery on your devices,
Chris: less network resources.
Chris: You can host it completely or you can use their managed solution. But beyond just that.
Chris: It's about the company architecture. It's about the funding model.
Chris: It's about all these things that go into a long-term sustainable product that
Chris: you can build a network on top of.
Chris: And they have been solid since 2017. They had to be.
Chris: They had to go in production day one for Slack and their global infrastructure.
Chris: Now, now it's incredible what they power.
Chris: Rivian cars on the road, all the metrics they send back to Rivian HQ,
Chris: that's secured by Nebula.
Chris: Think about all of those road, all those machines on the road communicating over Nebula.
Chris: They have to keep that secure that would be such a scandal if anything ever happened for
Chris: rivian it's really impressive it has
Chris: an incredible level of resilience speed and scalability that you can tune to
Chris: your needs from a home lab to an enterprise so go get started 100 hosts absolutely
Chris: free support the show no credit card required go to defined.net slash unplugged
Chris: well you know it is actually very scary Very close.
Chris: So we wanted to give everybody a
Chris: heads up that your time to submit to the Holiday Home Lab is running out.
Chris: We want your holiday home lab submission. Submit your home lab for Linux Unplugged
Chris: holiday special coming up.
Chris: We're looking for your creative or efficient or chaotic, overbuilt, just beginning setups.
Chris: We want everything from the community and we'll feature some of the winners on our show.
Chris: We'll have probably a batch that make it on the show. We'll have a batch that
Chris: we'll maybe do in special editions and all kinds of stuff. We have a form over
Chris: at LinuxUnplugged.com slash holiday.
Chris: Your time is almost out because we have picked our live record date,
Chris: which we would invite you to join us.
Chris: It'll be on Friday, December 19th, 2025, when we record the holiday home lab special.
Chris: So if you'd like to join us for the live event, especially if you've submitted
Chris: your home lab Friday, the 19th, we'll have
Chris: the exact details and time at jupiterbroadcasting.com slash calendars.
Chris: We get closer, but more importantly, because I sometimes forget that bit.
Chris: We always put it in the podcasting 2.0 feed. So if you have a podcasting to
Chris: do app, you'll see it in there when we're going to be live.
Chris: It'll be, I think our last live episode, maybe not, but one of our last live episodes of the year.
Chris: We, we, I don't know if we have after Christmas quite dialed in yet,
Chris: but it's getting pretty close.
Chris: And so you have, um, well, you don't, you got to get in before the 19th because we need days.
Chris: So I guess we should have probably told you a deadline, not just when we're going to do it live.
Chris: So I'm going to say the deadline is the 17th. that gives us at least two days
Chris: to go through all the submissions right yeah.
Wes: That seems reasonable.
Chris: So december 17th we're going to cut off no guarantees yeah so there's your cutoff
Chris: and then the live show will be on the 19th and we'll hopefully have uh more
Chris: details on exactly when probably we could probably just safely assume our regular time seems.
Wes: Like it sure why not.
Chris: Yeah there you works for me it's good it's gonna be fun it's gonna be a lot
Chris: of fun and we've had some great submissions.
Wes: You can't have a holiday without a home lab.
Chris: No, you've got to play your Christmas mixes and stuff. Well,
Chris: last week we played some clips to just get you up to date on the BcacheFS current
Chris: status and the kernel and honestly our real-time reactions because a lot of
Chris: that developed while we,
Chris: in both times there were major developments while we were traveling.
Wes: Yeah.
Chris: So we were like processing all of that.
Wes: Attempting to travel or attend a conference or, you know, not think about file systems.
Chris: So if you missed that last week, you might want to go grab that context.
Chris: So that way you have like sort of the full context for today's conversation when Kent joins us.
Chris: And as the year rounds out, we thought this is a topic we should talk about
Chris: because there's been a lot of changes.
Chris: Bcache FS was kicked out of the Linux kernel. We talked about that last week.
Chris: But I think it's best to get the information straight from the horse's mouth.
Chris: And in this case, that's Kent Overstreet.
Chris: So we're going to get into this in the way that only Linux Unplugged does with
Chris: the details and the technical stuff.
Chris: And the best source, of course, is Kent. He's the creator of Bcache FS.
Chris: And he's a rare, rare breed of developer.
Chris: And I think as you listen to our chat with him, this is going to become evident to you.
Chris: I wanted to mention something that maybe isn't obvious, though.
Chris: Kent is creating what I consider to be maybe the most modern file system,
Chris: period, right now. And he's making it GPL and available for Linux.
Chris: And he's doing it from what he calls a glamping tent attached to the back of
Chris: a truck in South America via a Starlink connection, living off of the support of his patrons.
Chris: And he is developing every single day. He committed, this is a Sunday morning,
Chris: and he committed three hours ago.
Wes: He sure did.
Chris: And then he had released a major version, which we will get into,
Chris: with major new features a few hours before our chat that day,
Chris: because we talked to Kent on Thursday.
Wes: Yeah.
Chris: And he had just released a massive, massive update.
Chris: And he's just constantly working. He's constantly communicating.
Chris: And he's doing this while creating this project while being nomadic and keeping
Chris: his cost of living as low as possible.
Chris: So I just think he's an incredibly interesting developer from that standpoint.
Chris: We're very lucky to have him.
Chris: And just the insights here are great. So if you're not familiar with BcashFS,
Chris: buckle up. And if you are, there's a lot to love in here, too.
Chris (Interview): Ken Overstreet, welcome back to the show. It's great to have you, sir.
Kent: Great to be here.
Chris (Interview): And also, congrats. BcashFS 1.33 came out this morning, I think.
Kent: Metadata version reconcile.
Chris (Interview): Huge, right?
Kent: Tied it perfectly.
Chris (Interview): I think we should start right there, and then we'll kind of get into other stuff.
Chris (Interview): But let's talk about the new reconcile system, because reading through the show
Chris (Interview): notes, it It looks like it was, it looks like one of those things on the surface
Chris (Interview): seems like a small amount of work.
Chris (Interview): But then when you dig into the details, it was like almost a total refactoring of type.
Kent: Total, but it touched all kinds of stuff. Yeah. Like had to get metadata plugged
Kent: in, all kinds of just little bits here and there.
Kent: Because whereas before, this grew out of Rebalance, which just did background
Kent: compression and background target.
Kent: Now we're aiming for strict correctness so that we can actually check the validity
Kent: of all stored data against the IO path options.
Kent: So if you're doing correctness checking, that tends to be a different level.
Kent: So big project.
Chris (Interview): And it's actually done now so um it's congratulations just reading through that i just like.
Kent: This is a.
Chris (Interview): Big piece of work and it's pretty cool that it landed right before you were
Chris (Interview): able to join us now as a somebody who's using bcachefs now will i need to will
Chris (Interview): i need to prepare or make any changes before i update.
Kent: Uh the only thing to be aware of is uh there
Kent: will be an upgrade process uh when you
Kent: mount on the new version one of the things that should be coming soon
Kent: but didn't land for this release is we should be able to do these
Kent: upgrades in the background and we're still doing
Kent: it at mount time and we still haven't got this communication with systemd going
Kent: to let systemd know that we're actually doing stuff at mount and it just uh
Kent: thinks that we've timed out so that's that's a little bit of an annoyance but
Kent: just just let it go until the kernel log says it's finished start the d-message,
Kent: whatever you're watching,
Kent: and reboot it again if you need to because of the systemd thing and it'll be fine.
Kent: We had a lot of that back in like the 614 upgrade for back pointers or i had
Kent: to redo something some minor detail of the back pointers actually had to rewrite
Kent: half of everyone's metadata.
Wes (Interview): I think i remember upgrading from that one yeah.
Kent: Yeah yeah no one noticed anything besides going hey why were my mouse taking
Kent: so long people would kick it a couple times and it would just finally finished
Kent: and so that was fun this is not as big an upgrade but okay okay.
Chris (Interview): That puts it in perspective all right so let's walk it back a little bit it's
Chris (Interview): been just about a year or more since we talked i can't quite recall to be honest
Chris (Interview): but it feels like it's been five years i mean time flies it's.
Wes (Interview): Just shy uh 693 days.
Chris (Interview): Oh okay oh so okay yeah it's been.
Kent: A bit so.
Chris (Interview): Um things have changed a Let's talk a little bit about user base,
Chris (Interview): who's using it these days.
Chris (Interview): Uh, maybe how just off the top of your head, I know this is kind of a wild one,
Chris (Interview): but how, how you say things have changed since 2024, just kind of a,
Chris (Interview): the best recap that you can recall.
Kent: The biggest, biggest changes that things have really stabilized.
Kent: Even like six months ago, you can probably three months ago,
Kent: like back in July, I was still in the midst of, you know, the,
Kent: the bugs are coming in about as fast as I could handle them and things that, you know,
Kent: I could see the trend back in, like, 6.14, but there was a noticeable inflection
Kent: point, like 6.16, 6.17, and now it's like things are starting to feel calm.
Kent: Past couple months were a giant slog with getting Reconcile out the door,
Kent: but I'm feeling pretty good about where we're at.
Kent: And we're getting really close to taking the experimental label off.
Kent: I pegged that for Christmas.
Wes (Interview): What a lovely present, huh?
Kent: Yeah, yeah. Yeah, there's like almost nothing left.
Wes (Interview): Yeah, I was going to ask, you mentioned sort of, you know, assessing stability
Wes (Interview): and here getting ready to take off that experimental label.
Wes (Interview): What actually are you looking for? Because I know you're helping people one-on-one.
Wes (Interview): You've also got like actual, you know, bugs getting filed upstream.
Wes (Interview): Probably you have like automated tests running.
Kent: Lots and lots of automated testing, yeah. There's a whole cluster of machines off in Finland.
Kent: Big 80 core R machines with a quarter terabyte Rm each. And I run those machines hard.
Kent: So i'm constantly watching the test dashboard which is never
Kent: as clean as i would like it but i'm always
Kent: watching like triaging and okay that one can
Kent: wait no that's actually something serious i need to jump on now and uh most
Kent: of it is just how are user feedback am i are the bug reports coming in how severe
Kent: are those bug reports how quickly are we dealing with them are we finding stuff that it's like,
Kent: oh that repair path needs a ton of work or is it like oh that's a that's like
Kent: a trivial 10 or 20 line fix,
Kent: So it's a lot of things.
Chris (Interview): I'm thinking back, it was really still pretty experimental at the time.
Chris (Interview): It feels like now we're at a point where if I deploy it, I'm probably not going
Chris (Interview): to see any big format changes or anything that's really going to be disruptive
Chris (Interview): to my workflow. And that's huge, right?
Kent: Yeah. One of the things with format changes is I've actually gotten less conservative
Kent: with doing format changes since we're no longer upstream. We're DKMS.
Chris (Interview): Interesting. Okay.
Kent: Because, yeah, we no longer have users running a whole bunch of different versions.
Kent: And maybe rebooting it to older kernels.
Kent: It's very easy now for users to just stay on the latest version.
Kent: And as long as I'm not pushing out regressions and we've been doing great on
Kent: regressions, then there's no reason for users to have to roll back to older
Kent: versions of the PCacheFS, which we can do.
Kent: It was an expensive downgrade and then re-upgrade in the kernel.
Kent: And now it's like, if we have to change something and there's a legitimate reason,
Kent: like reconcile, we've got another one coming for per-device fragmentation LRUs.
Kent: That'll fix some issues with copy GC and tiering.
Kent: Then it's just not as big a deal, especially once in the next couple of weeks
Kent: we could do those upgrades in the background. Then it'll be completely smooth.
Wes (Interview): Yeah, then we really probably won't notice, huh?
Kent: Yeah.
Chris (Interview): So, Kent, when all of this was going down with the discussion,
Chris (Interview): will they or won't they extract BcashFS, did this benefit even occur to you during all of that?
Kent: No, that one actually did not. The whole DKMS transition went much,
Kent: much smoother than I expected.
Kent: I had a lot of concerns and fears at the time how things were going to turn
Kent: out, and it turned out actually better than I could have expected.
Chris (Interview): How so? I mean, besides obviously what we just talked about,
Chris (Interview): but in other ways as well?
Kent: Yeah, I didn't seem to lose users like I expected.
Kent: The people who actually care stuck with it. We were able to get the DKMS packages
Kent: done, all the automation, everything we needed.
Kent: That came together really fast because a lot of people jumped in and helped out.
Kent: It's shocker in hindsight, not so obvious going in, but it's a lot easier for
Kent: people to help out with a non-core kernel file system stuff.
Kent: And there's a lot of people who are interested and want to help out,
Kent: but they're understandably intimidated by diving into a 100,000 line C code
Kent: base, but like packaging stuff, yeah, that was a lot easier.
Chris (Interview): I hadn't even thought about that, Kent, so I actually appreciate you bringing
Chris (Interview): that up. I guess what I have been stuck on,
Chris (Interview): I guess it seems like it's a good thing for Bcache in the short term,
Chris (Interview): but it's not a great thing for Linux in the long term.
Chris (Interview): I have really expressed on the show, I really think Linux needs a sophisticated,
Chris (Interview): modern file system that's GPL and built in. I think it's imperative.
Kent: Yeah, I don't think the Linux file system world is in great shape,
Kent: and I feel for those guys.
Kent: But, you know, a lot of this stuff, there's fairly deep-seated problems,
Kent: and I wasn't able to make any headway, and I can't do anything if no one wants
Kent: to, if everyone is stuck in their ways and doesn't want to change.
Chris (Interview): Is it momentum of just history and people that have their fiefdoms?
Chris (Interview): Is it just that? It's built fighting through that?
Kent: That's some of it. Yeah. There's a lot of reasons.
Kent: You know, the kernel is a big old project and, you know, basic processes,
Kent: how we do things tend to, you know, in any big institution tend to ossify and
Kent: become harder to change.
Kent: And the Linux kernel came of age, you know, really back in the 90s,
Kent: early 2000s, with everything being done on the mailing list,
Kent: patch-based workflow, didn't even use source control for years and years famously.
Kent: And the one place where it really lags today is like automated testing, QA.
Kent: So one of the pain points for me of being in the kernel is I do rely very heavily on automated testing.
Kent: And if you're in the kernel, you're expected to stay on mainline,
Kent: rebase onto RC1s every time RC1 comes out.
Kent: And that's a good thing. You want people to be dogfooding and catching issues like that.
Kent: If the file system people are the only people doing really comprehensive stress
Kent: and torture testing, and there's
Kent: other subsystems in the kernel that aren't testing the code at all,
Kent: that's really painful, especially if you're like me, where I'm just one guy
Kent: with 100,000 lines of code, and if.
Kent: Things aren't working right, it'd be like every other release.
Kent: My test dashboard would blow up, and it'd be a couple weeks where it'd be tracking
Kent: people down, and I can't get work done if my test dashboard is a whole bunch
Kent: of red, and I can't tell if my own patches are buggy or not.
Kent: Think about that for the kernel as a whole. It boggles my mind trying to get
Kent: work done in that environment without automated testing and not being able to
Kent: get good feedback on what code is working or not.
Kent: They're spending way too much time chasing regressions and other people's code and it's not fun.
Wes (Interview): It does make me reflect just on all the
Wes (Interview): pros and cons to the coupling to the
Wes (Interview): kernel if you are coupled you get the advantage that it's there by default but
Wes (Interview): if you're uncoupled there's a lot of pros although also of course some cons
Wes (Interview): I was just curious how you thought it was going being out of the kernel besides
Wes (Interview): DKMS I know there were some symbols that were no longer exported that had to
Wes (Interview): get Vendored like that kind of thing.
Wes (Interview): Do you anticipate being able to keep the current model working without too much of us?
Kent: I'm curious how that's gonna go The the core kernel interfaces that file systems
Kent: depend on and be cache of us They've gotten a lot less journey than than they used to be,
Kent: so I don't expect things to be too painful but,
Kent: The kernel is not very friendly to out-of-tree projects and codes, so it's a question mark.
Chris (Interview): It's fair. That's fair. It doesn't make me feel much better,
Chris (Interview): but it's a fair answer. I'm sure you probably feel very much the same.
Kent: We shall see.
Chris (Interview): Yeah.
Kent: Yeah.
Chris (Interview): All right. Well, I really appreciate you addressing that. That actually gives
Chris (Interview): me, I think, a better understanding overall of the situation.
Chris (Interview): I mean, as us as users, I've been really impressed how some of my favorite distributions
Chris (Interview): have just made it. But they've really stepped up to make it pretty simple for
Chris (Interview): me to just keep using BcacheFS.
Kent: Yeah, the NixWest and Arch people have been awesome to work with.
Kent: I haven't had to do anything for those distros besides keep in contact and make
Kent: sure we're on the same page.
Kent: And those guys hang out in the BcashFS IRC channel and we give each other a
Kent: shot whenever something comes up. And generally things are just chill and awesome.
Chris (Interview): Okay, so you made a hint about a lab. And I've heard you kind of made reference
Chris (Interview): to some pretty significant large BcashFS deployments.
Chris (Interview): It's just, I don't know, off the top of your head, what are some of the larger
Chris (Interview): that you're aware of that people have been experimenting with or using in production?
Kent: Oh, actually, I'm not sure what references I've made because I honestly do not know.
Kent: I hear from people when stuff breaks and if things are working great,
Kent: then like there have been quite a few times when, you know,
Kent: the rare regression, user impacting regression comes through.
Kent: And I find out that there's actually a lot more users than I thought there were.
Kent: Like the Debian brouhaha, where the maintainer shipped a package and broke the
Kent: build and then sat on updates.
Kent: And there was a bug in mount where users couldn't pass the degraded option.
Kent: And I start getting all these bug reports from users I've never heard of that
Kent: I just had a drive die and I can't mount my file system.
Kent: So, yeah, I have no kind of telemetry or statistics, and I would really love
Kent: to get telemetry and just see all the data I possibly can when something breaks,
Kent: but I don't have that yet.
Chris (Interview): Okay, okay.
Wes (Interview): Have you run into any interesting systems, though, or heard from folks reporting
Wes (Interview): successful or otherwise deployments?
Kent: Oh, there's definitely some big file systems out there. There's been,
Kent: for quite a while now, file systems north of 100 terabytes.
Kent: I mentioned the 6.14 backpointers upgrade. That was specifically because people
Kent: are running into scalability issues, and at 100 terabytes, FS checks were taking more than a day.
Kent: And now we can scale up. We tested scalability up to like 10 or 12 petabytes,
Kent: and no one's hitting scalability issues anymore.
Chris (Interview): Okay, well, I'm nowhere near those numbers, but I've kind of this year started
Chris (Interview): building my home lab, rebuilding my home lab.
Chris (Interview): It's really gonna be a project for next year. And I think I want to do Bcash FS.
Chris (Interview): And I just sort of wanted to pick your brain where you think Bcash FS is for
Chris (Interview): home labbers, if that's maybe a good use case, some things I should maybe try or know about.
Chris (Interview): Anything like that, you know? Like if I got you on the line,
Chris (Interview): I gotta be selfish and I gotta ask.
Kent: Yeah, yeah. Oh, I love working with the home labbers. Those are the guys that
Kent: hang out in the IRC channel and that,
Kent: We have the most productive working relationships because those are the guys
Kent: that like to tinker and investigate and discover how things break.
Kent: And that's where we find a lot of the weird corner cases. And that's where the
Kent: polish comes from is working with those guys.
Chris (Interview): Okay. So if I'm going down the route.
Kent: Go for it.
Chris (Interview): Yeah. I mean, just any tips, anything I need to worry about?
Chris (Interview): Is it just deploy, any mount options? I want any pro tips you got for me.
Wes (Interview): One angle might be like RAID 5.6. Is that available yet? like how she structures the array maybe if.
Chris (Interview): You got any tips.
Kent: Yeah uh aside from erasure coding we'll
Kent: get to that okay everything should be pretty fire and forget if
Kent: you've seen like the documentation on uh like how
Kent: you set iopath options you know all that stuff can be set per
Kent: directory application level uh target
Kent: options you can set up whatever kind
Kent: of tiering or caching setup or pin
Kent: different directories to different devices there there
Kent: really shouldn't be much to configure uh actually the
Kent: one thing that i should mention is uh there is an outstanding bug on the bug
Kent: tracker that looks like it might be a bug in z standard the title of the bug
Kent: is uh data corruption after writing hundreds of gigabytes and it's only used
Kent: with c standards so yeah maybe.
Wes (Interview): Choose a different compression.
Kent: Not the first c standard bug that we've tripped across right right and one of
Kent: the hardening projects that's like next up on the to-do list will enable us
Kent: to like pinpoint and flick up those guys like a test case or something and,
Kent: hardness against uh compression algorithm library bugs or compression algorithm
Kent: bugs and a couple other things in the data update path so yeah aside from that
Kent: i mean you should be good to go you.
Chris (Interview): Mentioned erasure coding though you mentioned something about erasure.
Kent: Yeah that's coming uh hopefully like first half of next year there's not much
Kent: left uh people have been running erasure coding and meant not much to say about
Kent: it besides the fact that it's fast and seems to work well,
Kent: The only thing that's missing there, which the reason you shouldn't use it yet,
Kent: unless you really know the gotcha, is we're missing like resilver or like the
Kent: erasure coding version of re-replicate.
Kent: But drive dies and you've got degraded stripes.
Kent: You can still read your data, but we don't have any kind of path to resilver those stripes.
Wes (Interview): Does your recent reconcile work impact the implementation of that?
Kent: It does, actually. uh that kind
Kent: of gave us the hooks uh and
Kent: the engine that's going to drive all that resilvering stuff and i reserve space
Kent: and some metadata on this format stuff for exactly where erasure coding is going
Kent: to plug into that when i got the rest of the design sketched out so it's it's
Kent: looking like it's not going to be that much work but i got i got two more hardening
Kent: projects to do before i can think about that stuff.
Chris (Interview): So when you say hardening projects i think of security hardening But do you
Chris (Interview): mean a different type of hardening?
Kent: Yes. I mean, the file system, like this is file system land.
Kent: The file system jobs is to never lose your data.
Kent: I always tell people, I don't care what happens to your file system.
Kent: I don't care if it, whose fault it is, if it's like user error or hardware error.
Kent: As long as there's data on your file system, it's the file system's job to get
Kent: back anything that can be gotten back.
Kent: And repair it and get it working again fully read-write with no user intervention.
Kent: That's the goal here. And we're,
Kent: I mean, judging from the log reports that I'm seeing, we're basically there.
Kent: I just got a bug report like last week, two device replicated file system,
Kent: tons of Btree node errors on only on one device.
Kent: Like you glance at the log at first and you think that must be a BcacheFS bug.
Kent: But then you look further and all the errors are like got wrong Btree node,
Kent: but on one device, the other device is good.
Kent: And the amount is like there there's no way
Kent: a bcashfs bug could like swap
Kent: out all your btree nodes with the wrong btree node only
Kent: on one device that's just not possible so my suspicion is that a device mapper
Kent: like a configuration got screwed up or something and shifted all the offset
Kent: of every data on that device so that's the only thing i can imagine that could do that oh man so.
Chris (Interview): Okay.
Wes (Interview): But it's those kinds of situations.
Chris (Interview): You want things to just work from you want to be able to get your data back
Chris (Interview): from that situation Yes.
Kent: Yeah, yeah makes sense. Yeah, I am I am very very insistent and very hard line
Kent: even more hard line than the ZFS folks on this,
Kent: Like if you compare a bcache fx and ZFS ZFS was designed for enterprise deployments
Kent: where you can depend on replication You can depend on hardware that doesn't screw up flush and FUA,
Kent: uh and if you're running it in those kind of situations it should always work,
Kent: but if you're running at single device mode on garbage hardware there are failure
Kent: modes that zfs explicitly does not protect against and you'll be sending your
Kent: drive to a drive recovery service,
Kent: bcashfs handles those like i don't care if you metaphorically let your let your
Kent: file system on fire it's we're gonna get your data back.
Chris (Interview): We love that because we love garbage hardware i was gonna to say this is where
Chris (Interview): it sounds like it fits our home webs anyway we're all about garbage
Chris (Interview): make it last as long as possible right so it sounds like i mean i shouldn't
Chris (Interview): even be asking this since you just released 1.33 this morning but it sounds
Chris (Interview): like the next bit of focus is some of this hardening work that's kind of where you're going next.
Kent: Yeah the other one is kind of cool too uh because it's again a garbage hardware
Kent: thing we want to be able to run on devices where maybe flush and fuel it is
Kent: just broken and we can't do anything about it like cheap sd cards,
Kent: or usb controllers usb controllers often have buggy flush to have seen the bug
Kent: reports this idea came out of meeting with valve a couple weeks ago what i'm
Kent: going to be doing is we've got scrub now and then in recovery we've got a journal
Kent: so we know what the most recently written data was,
Kent: Why not just run a targeted scrub on only the last second of data we wrote?
Kent: And then we can detect specifically and prove,
Kent: it was broken flush fua in that case and
Kent: even better we've also got whole file system rollback from the journal rewind
Kent: thing i remember that that whole thing so if we detect that we could just roll
Kent: your file system back to the last consistent point and you don't users don't
Kent: have to care then we can run on garbage hardware and it does not matter ah.
Wes (Interview): That sounds great.
Chris (Interview): I love that But okay, so am I to get the impression perhaps that Valve is interested
Chris (Interview): in a more robust file system for future use?
Kent: They might be.
Chris (Interview): Yeah, that's interesting. I mean, if they're paying attention,
Chris (Interview): they probably should be. So that's good to know.
Wes (Interview): That was going to make me...
Kent: More people should be paying attention to file systems in my opinion.
Chris (Interview): Yeah, I agree with you. I'm glad you are paying so much attention and I'm glad
Chris (Interview): it sounds like maybe Valve is consulting with somebody that knows their stuff.
Wes (Interview): Makes me curious. How's the support been like both in terms of,
Wes (Interview): You mentioned some of the volunteers for DKMS, but also just,
Wes (Interview): you know, monetary support as well.
Wes (Interview): You mentioned during some of the turbulent times in the kernel that some of
Wes (Interview): that back and forth was generating bad press that was maybe making it harder
Wes (Interview): to find consistent support. Has anything changed there?
Kent: Uh things have gotten gotten stable since
Kent: since valve has jumped in and supporting yeah there
Kent: there's it still really should be funded better if you better if you have any
Kent: listeners with big pockets this is a project that really should have like three
Kent: or five or six engineers working on it and so to me working it's ridiculous
Kent: hours seven days a week that would be awesome uh,
Kent: but it's it's stable and good and and i'm not i'm not sweating it like i like i was anymore Oh.
Wes (Interview): Good. Glad to hear that.
Chris (Interview): That's good. And that's mostly, is that due to Valve helping there?
Chris (Interview): Is that stabilize things a bit?
Kent: Yes. Yes.
Chris (Interview): That's good to hear.
Kent: And before Valve, a European funding agency, NLNet, they jumped in and helped
Kent: out. And that was a big help.
Chris (Interview): Yeah, it really would be nice if a few more folks would step up.
Chris (Interview): A few companies that are going to probably no doubt one day make a lot of money
Chris (Interview): off of BcashFS should step in now, right?
Kent: No doubt.
Chris (Interview): Yeah.
Kent: No doubt.
Chris (Interview): Hey, does that cross your mind? You are, you know, you're probably,
Chris (Interview): you know, I'm sure you're not living, you know, luxurious. You're just,
Chris (Interview): you know, you're getting by on this.
Chris (Interview): And someday some companies are going to probably sell network attached devices that sell for $30,000.
Kent: Yeah, but I'm also thinking long term here. Like one of the things about the
Kent: whole criminal community is they've gotten too far down the route of just chasing
Kent: where the money is going.
Kent: And things in file system land got very project manager focused without enough,
Kent: you know, engineering focus on long-term plans.
Kent: Like, how are we going to architect things so we can do what we want,
Kent: not just a year from now, but 10 years from now.
Kent: And right now, Linux kernel funding is very much dominated by,
Kent: you know, the big cloud providers.
Kent: And the big cloud providers don't care about reliability the same way the rest of us do.
Wes (Interview): Yeah, that's kind of the whole cloud thing, right? They're like,
Wes (Interview): well, you'll just spin up more of these throwaway instances or whatever.
Kent: Yeah. If a node dies, it's no big deal if you've got everything replicated across the cluster.
Kent: But a lot of us actually do care about reliability of the single node.
Kent: So I honestly don't care that much about the big cloud providers or the big
Kent: money part of the tech industry because they're not pushing technology forward
Kent: in the field that I care about.
Wes (Interview): Yeah, they have a totally different approach to storage and how they manage
Wes (Interview): this stuff that just doesn't really make sense for those of us trying to run regular-sized arrays.
Kent: But there are people out there who do need a file system that will never fail,
Kent: who do need the core part of the system to be as reliable as humanly possible.
Kent: So I'm not thinking about big cloud providers. I'm thinking about like...
Kent: I didn't even realize this until years ago i saw it
Kent: on my github page there's a little sticker something about
Kent: i had code uh on running on mars it is someone had noticed that everyone who
Kent: had got code into the linux kernel and probably some time frame or whatever
Kent: well nasa's running linux on the mars rovers which apparently they are that
Kent: means i've got code on mars that's so cool so,
Kent: yeah but they don't just need the kernel they actually need you know a file
Kent: system that's reliable enough that it's not going to take down the rover if
Kent: it crashes even if you're running in a high radiation environment where bit
Kent: errors are just common and unexpected that really can recover from anything yeah.
Wes (Interview): Hopefully you can save that data before you transmit it and have a chance of sending it.
Kent: Out yeah.
Chris: Unraid.net slash unplugged. You want to build your own dream server?
Chris: Well, unraid 7.2 makes it easier than ever.
Chris: Go unleash your hardware.
Chris: Now with a fully responsive web GUI, unraid now works beautifully across all your devices.
Chris: You can set there and build your favorite application stack from your couch if you want.
Chris: I think what you're really going to like too if you do the ZFS thing is it also
Chris: adds ZFS RAID Z expansion support.
Chris: That means you can now grow your ZFS pools with having to start over.
Chris: Man, that's great to see. And for those that maybe have a spare USB or external
Chris: hard drive, Unraid 7.2 introduces support for Extended 2, 3,
Chris: 4, and NTFS as well as Extended Fat.
Chris: And so if you've got grandpa's photos like I do on an old NTFS drive somebody
Chris: gave you, you can just now instead of having to build that Windows box or try
Chris: to load that NTFS driver on your Linux desktop, just plug it into your Unraid.
Chris: There's also a new API. It's officially here. It's real. It's beautiful.
Chris: People are building stuff on top of it, and it's open source.
Chris: Fully integrated. It gives you secure, programmable access to system data for
Chris: building dashboards, automations, or, you know, your own external apps.
Chris: It even supports external authentication via OIDC, or OIDC, as you called.
Chris: It's massive. I mean, 7.2's already had well over 25,000 downloads,
Chris: lots of applications coming out.
Chris: You can get a free 30-day trial and support the show when you go to unray.net
Chris: slash It's the OS that grows with your skills.
Chris: And 7.2 introduces the new Unraid API, and it's chef's kiss.
Chris: Check it out, support the show, and get a free 30-day trial.
Chris: Unraid.net slash unplugged.
Chris (Interview): Okay, so speaking of architecture stuff, you seem pretty open to the inclusion
Chris (Interview): of Rust code in BcacheFS in the future. Can you touch on that?
Kent: Oh, I'm pushing hard for that.
Chris (Interview): Tell me about it.
Kent: Well, I mean, Rust is a sea change moment for systems programming.
Kent: For decades, how we wrote systems code didn't change much. you just had to deal
Kent: with, you had to know that your code is perfect and be able to prove to yourself
Kent: that your code is perfect.
Kent: But the dream for a long time has been code programs with embedded correctness proofs.
Kent: Things like CompCert, the first C compiler that is written in a language cock
Kent: where you can prove that it always generates code that matches the specification.
Kent: Wouldn't it be great if all code is like that?
Chris (Interview): Especially file system code. Yeah.
Kent: I mean, I'd love that because I hate debugging.
Chris (Interview): Yeah.
Kent: So the question has been, is that stuff ever going to be practical?
Kent: And Rust actually solves a huge chunk of the of the problem for you know in
Kent: a language that we can actually use in the system space that's huge and there's
Kent: deep reasons why Rust is designed the way it is and why it's well designed.
Kent: Rust is about as close as you can get to a pure functional language in an imperative
Kent: language as you can get, and you could argue that it's just as good as a pure
Kent: functional language for all intents and purposes.
Kent: And not just that what's also coming on the
Kent: horizon uh there's been a big development in uh how you actually make embedded
Kent: correctness proofs practical and not something that you that make you spend
Kent: 10 times as long to write something like concert which is uh dependent types,
Kent: and idris is the language that's been exploring this it's
Kent: a it's a haskell offshoot and it's one
Kent: of those things were in hindsight if you're a kind
Kent: of programming language nerd or a computer science nerd you go
Kent: oh that was obvious because there's this
Kent: thing called rice's theorem which is an offshoot of or kind of a corollary uh
Kent: to the halting problem the halting problem says you know in in a try and complete
Kent: language you can't prove or know ahead of time if a program completes You just
Kent: have to run it and see what happens.
Kent: Rice's theorem extends that to, in a Turing-Complete language.
Kent: Any property of the program, anything that program does, is either a direct
Kent: consequence of the syntax, or it's undecidable. You just can't know.
Kent: So the direct consequence of the syntax thing is interesting because type systems
Kent: are syntactical. You can reason about them syntactically.
Kent: So what that kind of says is any practical advance in proving properties of
Kent: programs has to be a type system thing.
Kent: And that's what Idris and dependent types are doing.
Kent: And it actually makes this stuff ergonomic. And I think one way or the other,
Kent: that's going to make it into Rust someday.
Kent: I mean, we could even do it because Rust has really, really good macros.
Kent: We could probably add this stuff to Rust as procedural macros.
Kent: And so my game plan is, you know, this might be like five, ten years out,
Kent: five years before the Rust rewrite is mostly done.
Kent: That won't be the big advance for BcacheFS. The big advance will be dependent types.
Chris (Interview): Can you explain more?
Kent: So there's file systems are giant. state machines.
Kent: There's tons of things that just have to be consistent.
Kent: Like in the B-tree code, like all the locking stuff for the B-tree,
Kent: iterators, lots of higher level stuff.
Kent: You can't model that stuff with the type systems of today.
Kent: I have to write these very long and very heavy assertions in debug mode that
Kent: just methodically hey, check, is the entire state of this object correct?
Wes (Interview): So you have a giant state space to explore that you have to explore manually
Wes (Interview): because the type system, the compiler can't because you can't fully describe
Wes (Interview): the complexity to the type system as is. Is that right?
Kent: Yeah, yeah. So anytime we mutate like the state of the objects that describe
Kent: what locks we hold on the B-tree, we have to call these verify functions,
Kent: the search that check, is this a valid state?
Kent: All that stuff with dependent types, that becomes stuff that the compiler can check for us.
Chris (Interview): Yeah, okay. That's great.
Wes (Interview): Well, is there anything you're trying to do now? Obviously, dependent types
Wes (Interview): takes getting to Rust and Rust getting that functionality, but are there things
Wes (Interview): you're doing now practically today to try to prepare the code base in your approach?
Chris (Interview): Years ahead, basically, yeah.
Kent: Leading into something that I mentioned in the chain log, yeah.
Kent: So, maybe a year ago, Well, rewind, C finally got the cleanup attribute,
Kent: which lets you run a destructor for an object when it falls out of scope.
Kent: Now, this is basically sudo RAII, like C++ and Rust have.
Kent: And then the kernel wrapped this up into a nicer interface, the class thing.
Kent: And I found that, I started refactoring code, and I realized as I was doing
Kent: it, you kind of have to get rid of gotos, because go-to's don't play nicely with the RII stuff.
Wes (Interview): Sure, yeah.
Kent: So doing the whole conversion, going all the way, has real benefits.
Kent: You kind of have to. You don't want to be in this weird intermediate state where
Kent: you're trying to make sure the two don't conflict.
Kent: I got kind of far along and I realized, okay, this is actually making the code
Kent: fairly close to what the Rust version would be.
Kent: Uh it's not going to be a full rewrite for a lot of the code it's going to be
Kent: mostly syntactic so i've been pushing on this stuff for pretty hard for the
Kent: past six months and we're pretty far along uh like six months ago when i started
Kent: i i grew up and word counted and bcashfs had 20,
Kent: 2 600 go to's in the code base which was about the same number as xfs so a pretty
Kent: standard for a C code base of that size.
Kent: And now we're down to about 600.
Wes (Interview): Wow.
Kent: A lot of files don't have any at all.
Kent: So the next step is gonna be, and we've already got some Rust code and some
Kent: Btree iterator, Btree API and transaction interface bindings.
Kent: Those need to be improved and fleshed out. And then we can start thinking about,
Kent: just taking a file and just doing a, not a rewrite, but just a translation.
Kent: If we've got the bindings, then it should start to become straightforward.
Kent: It starts to become straightforward with giant air quotes because there's always
Kent: things that comes up and there will be a lot of interface work.
Kent: But it's actually looking tractable to convert a 100,000 line code base to Rust
Kent: or at least large swaths of it. And that's freaking awesome.
Chris (Interview): Kent, I think it always impresses me. And you just demonstrated it here.
Chris (Interview): It's like you have a grasp of the really small, nuanced details that need to
Chris (Interview): be addressed. And at the same time, you're holding space in your head for five-year
Chris (Interview): changes and really big macro stuff.
Chris (Interview): That's a pretty rare talent.
Kent: I've been doing this for a long time at this point.
Chris (Interview): Well, it's really impressive. Okay, so I think the listeners are probably wondering,
Chris (Interview): what is your Linux setup like? Are you constantly breaking your system with
Chris (Interview): new builds? Do you kind of have a standard box?
Chris (Interview): Do you have a specific distro you go to? You know, the kind of things they might be interested in.
Kent: Oh, I don't faff with that stuff much anymore. I got too much work to do to
Kent: screw around like I used to.
Kent: So the big workstation, which is still up in Canada, hosted by a friend in his
Kent: Colo facility, that's a Debian installation that I haven't reinstalled in like probably 15 years.
Chris (Interview): Oh, that's impressive.
Kent: Situation yes that's a proper dead read yep yeah yeah so that's running uh not
Kent: bcash fs yeah it's uh that's running ext4 on top of bcash on top of md raid 6,
Kent: yeah and that setup hasn't changed in yeah forever i've swapped out every drive
Kent: individually to like grow the array but that's literally the same array that
Kent: i've had for like 15 years so.
Chris (Interview): That's like a home base box that you you ssh into or whatever and it's kind of like your master.
Kent: Machine Yeah, yeah.
Chris (Interview): Okay.
Kent: Yeah.
Chris (Interview): And then do you have like a laptop or something with you that's sort of your own on the go machine?
Kent: Yep. And the laptop is running BcacheFS and NixOS.
Wes (Interview): Excellent.
Chris (Interview): Ah, ha, ha.
Wes (Interview): Hey, just like Glenn.
Chris (Interview): Hey, very nice. Yeah, I mean, NixOS has made it very easy to run BcacheFS. It's really great.
Chris (Interview): So we're all three of us to them.
Wes (Interview): Elvis Jericho as well, right? Like from before it was in the kernel to in kernel
Wes (Interview): to after kernel, like all of it has been seamless, which is great.
Kent: Yeah, Elvis Jericho hangs out in the channel, and he's been interested in the
Kent: System D stuff, and we've been talking about ideas about how to sort that stuff
Kent: out and get proper keep lives.
Kent: What I want to do is there's Plymouth, I think, has some kind of mechanism for
Kent: giving you information about your boot up.
Kent: I can't remember off the top of my head what exactly is there but what I want to do is,
Kent: plumb proper progress indicators up to Plymouth,
Kent: in the not too distant future, we shouldn't need it for upgrades but if there's
Kent: major file system damage and we have to fall back to B3 node scan we want to
Kent: tell the user what we're doing and how long it's going to take or if a drive
Kent: is going bad and we know that,
Kent: we want to be able to alert the user So yeah.
Chris (Interview): That would be really nice. Okay. Now, uh, no judgment zone. Do you have a desktop
Chris (Interview): environment? That's your go-to desktop environment on that laptop?
Kent: Sway.
Chris (Interview): How fun. All right. Why Sway?
Kent: Well, before Sway, I was using Awesome for years and years.
Chris (Interview): Nice.
Kent: And I loved that, but it was X11 only, and it was never going to come to Wayland.
Kent: And I finally did the Wayland switch for whatever reason.
Kent: I don't have strong opinions on it. Wayland's been fine.
Kent: But yeah, Sway was the Wayland option, and it's actually been really clean.
Kent: I like how it does configuration, and it's pretty cool.
Chris (Interview): So your Nix OS configuration, is it like a configuration you've brought with
Chris (Interview): you from machine to machine, or do you start fresh when you've got a new laptop?
Chris (Interview): Like the first time you set it up, did you start fresh, or is this something you drag along with you?
Kent: Yeah, I try to keep my configs in a Git repository, which I am very bad about keeping up to date.
Kent: So I generally just move the whole SSD to the new machine.
Chris (Interview): That's such a fun system guy thing to do. That's great.
Kent: How could I be laziest?
Chris (Interview): Hey as long as the disc's still good right.
Kent: It's all of it yeah that's.
Chris (Interview): What matters it makes it easier well gentlemen do you have any other questions for ken oh.
Wes (Interview): Did you want to talk about maybe um bcache fs top that was maybe something.
Chris (Interview): Oh yeah yeah new tool or improve tool.
Kent: Oh yeah yeah actually that's been out there for for a while but some improvements
Kent: land yeah so bcache fs top is this little thing that just shows you counters
Kent: for for your file system it's the simplest dumbest thing ever but let's see
Kent: what your file system is doing it's like yeah do you think it's kind of thing we love we.
Wes (Interview): Love it i like.
Kent: All this i mean.
Wes (Interview): There's a lot of stuff in here copy gc wait a whole bunch of error information io read io write oh.
Kent: Yeah some of those are old and probably don't make much sense it should be swapped
Kent: out for better ones but i've been doing a lot of work on like the the reconcile
Kent: and data update path events and trace points so the big thing that i just did was uh,
Kent: I was already moving in this direction, trying to hold to it,
Kent: but I finally did a giant codebase refactoring so that now it's done and things
Kent: will never get out of sync.
Kent: Every counter has a corresponding trace point. So the idea is that you see what
Kent: your file system is doing, you look at FSTOP, that will show you like what events,
Kent: what code pass, like what is it doing?
Kent: And then if you want to drill down and find out anything more,
Kent: the name of the counter is the name of the trace point and the trace point gives
Kent: you information about that specific event.
Wes (Interview): Wow oh that's so great wow smooth that's simple enough we could actually we
Wes (Interview): could actually do it maybe with a little lm help but we can get there.
Kent: And if any of those trace points isn't showing you the information that you
Kent: need in order to understand what's going on, it's dead easy now to just,
Kent: you know, everything is pretty printer-based.
Kent: We can just dump a couple fields in or dump another variable in there,
Kent: and it's like a five-line patch.
Kent: So you can just hop on the IRC channel and say, hey, I'm trying to figure out
Kent: what this is doing. And it could be like, here's a five-line patch for you,
Kent: and it'll be in the next release.
Wes (Interview): That is i love how much just from the developer perspective how much is sort
Wes (Interview): of the modern development affordances and workflow you are trying to impose
Wes (Interview): on this otherwise very austere and like you know.
Kent: Hard to.
Wes (Interview): Use environment and it seems like it's really paying dividends i mean from the
Wes (Interview): the file system as a database to insisting on automated testing to all the stuff
Wes (Interview): you're just talking about.
Kent: Yeah and a lot of the recent stuff as the project has grown a lot of the through
Kent: a polish and the stuff that i really like to see has come from just working with users,
Kent: a lot of you know developers think of support as a
Kent: chore but what i've realized is that you
Kent: know actually basically any support issue
Kent: that comes up there's something that we as engineers can do to just make the
Kent: system you know easier to use or tell you exactly what's going on or make the
Kent: issue go away entirely so doing support it motivates you to to just improve
Kent: that kind of polish and you get lots of really fruitful interactions about,
Kent: hey, the system is doing something wonky and you just got to be talking to people and,
Kent: Yeah.
Chris (Interview): I feel like, you know, when I think about Bcache's contribution to Linux over
Chris (Interview): the next 20 years, you are in a very unique period of time right now where it
Chris (Interview): would be great to have a lot of engineers at the same time.
Chris (Interview): You're really you're very hands on. I mean, Kent, you're you're not only like
Chris (Interview): doing the day to day work, but you are communicating with users.
Chris (Interview): You're writing comprehensive release notes that I think are better than most
Chris (Interview): projects release. and then you're communicating with folks at us at the same time.
Chris (Interview): And it's really, it's a special thing that's happening right now that 20 years
Chris (Interview): from now is going to people be using a file system they may not even know that's
Chris (Interview): had this kind of intention and energy put into it. But it's really great.
Kent: It's a lot to juggle, but it's a lot of fun too. And let me flip some of those
Kent: questions around too because you've been playing with BcacheFS a bit.
Kent: What do you like about it? What do you find modern and interesting about it?
Chris (Interview): I think I'm really kind of excited just having a file system that I can trust
Chris (Interview): at the caliber that I would trust ZFS,
Chris (Interview): but with features that feel more approachable, that it seems more reasonable
Chris (Interview): on my type of systems and is built in to the kernel or is easy enough with like
Chris (Interview): Nix OS to run as a DKMS module.
Chris (Interview): And like you add all that stuff up and it's essentially, it feels like the true
Chris (Interview): Linux competitor to ZFS that we've been waiting for since Sun Microsystems was a company.
Kent: That's been the dream. That's what I and like funders and everyone I've been
Kent: talking about for years and years.
Chris (Interview): And not to take away from ZFS. They can have it. It's great, right?
Kent: Oh, totally. Totally. ZFS deserves so much accolades and respect for showing
Kent: us what a modern file system should be and demonstrating that it was even possible.
Chris (Interview): Right, right. And it's made a lot of people happy. It's great.
Chris (Interview): It's just we need something in Linux too, right?
Chris (Interview): We need something that's GPL'd and something that's created kind of in the Linux
Chris (Interview): ethos inside the Linux community for Linux, which is exactly what you're doing.
Kent: And there's also legitimate technical advancements to be done beyond what ZFS did.
Kent: Uh probably not a lot of non-file system nerds know about like the inner workings
Kent: of this stuff but zfs was actually a very conservative design in a lot of ways technically,
Kent: for very smart reasons if you're trying to do a very ambitious feature set you
Kent: don't want to reimagine everything all at once right but the the core design
Kent: it very much is in line with the original unix file systems of like radix trees
Kent: of blocks hanging off the inode and like all the on-desk data structures,
Kent: it's block-based, it's not extant-based.
Kent: And we knew 20 years ago, 30 years ago, XFS did extents, that extents are the way to go.
Kent: The trouble was that extents and snapshots are a very difficult combination,
Kent: and one that took me years to figure out.
Kent: And there's lots of other stuff, like the whole file system as a database thing.
Kent: I could talk for a while about that one, but that was BcacheFS's biggest contribution
Kent: that's really simplified the design and made a lot of feature work.
Kent: Dead easy compared to what it would have been in previous systems.
Kent: I'll give you one example.
Kent: Towards the end of Reconcile, I was debugging with a user who was,
Kent: there was some stuff with Btree nodes not going on the pending list or coming
Kent: off the pending list or something.
Kent: But he mentioned a slow evacuate, and he was running on rotating disks.
Kent: And I went, hang on, the main Reconcile implementation scans the key space in
Kent: logical key order, like I know number offset within the file but for rotating
Kent: disks you really want to be doing it in like physical LBA order,
Kent: the disk is laid out in the disks,
Kent: and now Fitzroff in the channel pointed out that hang on
Kent: the old style evacuate actually you mentioned this when I was already doing
Kent: the fix but you mentioned that the old style evacuate code is already walking
Kent: back pointers do it physical LBA order this is going to be a real performance
Kent: regression and evacuate is something that actually you do want to run at a reasonable pace.
Wes (Interview): Yes.
Kent: So...
Kent: I had to add extra indexes for indexing pending reconciled work by physical LBA order,
Kent: plummet all into the triggers, add DFS check passes to make sure that that stuff
Kent: is correct, hook it into the back pointers code.
Kent: I was able to get all that done in a day, start to finish, from when I was thinking
Kent: about the design to passing tests and working.
Kent: That's like not everything is like that but that's right yeah what you can sometimes
Kent: do when you've got a good solid foundation and you know like of course the database
Kent: engineers are going that's just adding a table like we could do that and like yeah.
Chris (Interview): Yeah we follow that yeah.
Kent: Wow that's awesome.
Wes (Interview): You know you said that i think that like a good a good solid foundation that's
Wes (Interview): kind of what bcashfs feels like i'm using here with compression and the built-in
Wes (Interview): encryption and we we use and like ButterFS on the show but it just feels like
Wes (Interview): BcashFS is primed to be sort of like a,
Wes (Interview): less sharp version that you could just sort of always trust.
Kent: Something that you can debug.
Chris (Interview): Yeah, yeah.
Wes (Interview): Yeah, that tells you where it's wrong, that you know you can at least recover
Wes (Interview): data off of, and where the features that are in, you know you can rely on them.
Chris (Interview): Yeah.
Kent: Yeah, that's kind of what I'm going for.
Chris (Interview): I think it's one of the projects that we follow with the most interest and excitement.
Chris (Interview): So thank you for just all of the work you've done, especially recently.
Chris (Interview): It just seems like it's really been going from strength to strength.
Kent: Yeah, the community feedback, doing it for the normal people,
Kent: that's what I'm in this for.
Kent: I don't care about what Red Hat or Facebook are doing.
Kent: I'm just writing code that's solid and reliable and makes people happy. That's fun.
Chris (Interview): Yeah, it is. Yeah. And there will be just every day more and more people using it every day.
Chris (Interview): And it's going to solve more problems as people learn about it.
Chris (Interview): And we're going to keep spreading the word too.
Wes (Interview): If we do have users who give it a go after this and they do run into issues,
Wes (Interview): what are the best ways for folks
Wes (Interview): to actually interface with you and either get support or file blocks?
Kent: So jump on the irc channel uh you
Kent: can file a bug but it's generally a lot quicker to get stuff resolved if you
Kent: can join the irc channel then we could just you know talk talk people through
Kent: like gathering what information we need and just having a normal conversation
Kent: makes things a lot a lot quicker and then we can bring them into the community
Kent: and get them to do more testing and file file more bugs excellent.
Chris (Interview): Thank you kent.
Kent: Pleasure.
Chris: 1password.com slash unplugged. That's the number 1password and then unplugged
Chris: and that's all lowercase.
Chris: Go take the first steps to better security for your team by securing credentials
Chris: and protecting every application, even the unmanaged one. So go learn more.
Chris: You need to go to 1password.com slash unplugged.
Chris: This is a real challenge. There's a lot of assets to manage.
Chris: I mean, you have hardware, all the different devices from mobile to desktop to laptops and more.
Chris: But there's also identities, of course, and there's applications.
Chris: And there's more and more of
Chris: those all the time that just spin up that you might not even know about.
Chris: It's a lot. It creates a mountain of security risk. And you can conquer that
Chris: mountain of security risk with 1Password extended access management.
Chris: It's a big problem. Lots of people report this is their biggest challenge in
Chris: IT, just a SaaS sprawl, you could say, which creates shadow IT,
Chris: accounts, services you might not even know your users are using.
Chris: And it's not hard to see why the users get more work done.
Chris: I remember this when both Dropbox and Slack rolled out. It was a big user-up
Chris: adoption, and it created friction between IT and the users.
Chris: That's something that 1Password Extended Access Management really smooths out.
Chris: And one of the ways you are empowered is with Trelica.
Chris: Trelica by 1Password can discover and secure all your apps, managed or not.
Chris: That means you're going to get an inventory of every app in use at your company.
Chris: Trelica has pre-populated app profiles, so you can get an idea of the SaaS risks.
Chris: You can get an idea of who has access to what if there's redundancies.
Chris: You can really optimize your spend with tools like that, but probably most importantly,
Chris: you can enforce best security practices across every app your employees use.
Chris: You can actually manage the shadow IT. You can securely onboard and off-board
Chris: employees, and you can make sure you're meeting compliance goals.
Chris: Trelica by OnePass provides a complete solution for SaaS access governance.
Chris: It's just one of the ways that extended access management helps teams strengthen
Chris: compliance and security.
Chris: So go check it out and support the show. You go to 1Password.com slash unplugged.
Chris: You know about their award-winning password manager. It's trusted by millions of users.
Chris: This goes way beyond just passwords.
Chris: 1Password Extended Access Management. It's powerful, and it gives you a complete
Chris: oversight of the sprawling landscape of SaaS apps.
Chris: Whether your company knows about them or not, that's a big deal.
Chris: Go learn more. Support the show. go to the number one password.com slash unplugged.
Chris: That's one password.com slash unplugged.
Chris: You're going to change the way you look at managing it.
Chris: And it's a great way to support the show. Go take the first steps to better
Chris: security for your team by securing credentials and protecting every application.
Chris: Even the unmanaged shadow it one password.com slash unplugged.
Chris: That's one password.com slash unplugged.
Chris: Join crowdhealth.com and use the promo code unplugged. It is open enrollment
Chris: time, the season where the health insurance companies are going to hope you'll
Chris: just sign up again for overpriced premiums and confusing fine print that never
Chris: seems to work in your favor.
Chris: And the prices just seems to get worse all the time. It's horrible.
Chris: I had a tough call to make, especially as a small business owner,
Chris: what direction I was going to go. My wife also owns her own small business.
Chris: And we looked at all the options. We tried the traditional routes.
Chris: It wasn't working for us.
Chris: I was so grateful. over three years ago when I found CrowdHealth.
Chris: I think it's time to stop playing the health insurance game.
Chris: You can join CrowdHealth. It's a community of people funding each other's medical bills directly.
Chris: No middleman, no networks, no nonsense. But don't just take my word for it.
Chris: Go trust yourself and go take control of your future with CrowdHealth.
Chris: It's a healthcare alternative for people who make their own decisions.
Chris: Health insurance is, it's really stressful. It's confusing and you never really get what you want.
Chris: And when you see the prices, you wonder what's ever gonna stop this train.
Chris: Well, this is how we take the power back. And it's incredible how well it works.
Chris: And they have a beautiful app that makes it really simple, straightforward.
Chris: Of course, they have a website too.
Chris: Chat, customer support's all right there. Really great leadership team. I've talked to the CEO.
Chris: I've talked to people around there just casually and, of course,
Chris: through the course of doing business with them now for the ads.
Chris: And I'm really impressed. And not only do I like what I've seen over the last
Chris: three years, but I like the momentum.
Chris: I feel like I've picked a winner here. I think it's something you should really
Chris: check out because it works well for me it works well for my wife and I,
Chris: and you can get healthcare for under $100 you get access to a team of health
Chris: bill negotiators low cost prescriptions lab testing tools as well as a database
Chris: of low cost high quality doctors that get vetted by CrowdHealth it's an insurance alternative,
Chris: and if something major happens you pay the first $500 then the crowd steps in to fund the rest.
Chris: It's really the way it should work now. And of course, you'll join the crowd.
Chris: It's a group of members just like you who want to help pay for each other's
Chris: unexpected medical events.
Chris: You'll be impressed of how well it works, too.
Chris: The system is betting out there that you're just going to keep buying the same overpriced insurance.
Chris: And man, are they just making a boatload of money. And it gets so complicated
Chris: now. And if these subsidies expire, prices are going to go sky high.
Chris: If you join CrowdHealth and use our promo code UNPLUGGED, you can get started
Chris: for $99 for your first three months. That's fantastic.
Chris: So far, CrowdHealth members have saved over $40 million in health care expenses
Chris: because they just refused to overpay for health care.
Chris: Now, CrowdHealth is not insurance. You should opt out and take your power back.
Chris: This is how we win. This is how we change it.
Chris: Joincrowdhealth.com, promo code UNPLUG, take your power back,
Chris: and come join the crowd with me.
Chris: I think you're going to be really impressed. Joincrowdhealth.com and use the
Chris: promo code UNPLUG, and you will get your first three months for $99,
Chris: which is incredible. Use the promo code unplugged at joincrowdhealth.com.
Brent: While we have a special baller booster today, it is our dear PJ.
Brent: Jeff Boosin, a row of McDucks, 22,222.
Brent: Jeff says, okay, Brent. Brent, okay. Call me out for not boosting while I'm
Brent: out running errands and can't defend myself.
Brent: A guy only has so many sats to boost. That's true.
Brent: Always down to boost a good soapbox episode, though. I should make some soapbox
Brent: art for merch or something.
Wes: Oh, great idea.
Chris: That'd be good. Could you get me, if I'm on a soapbox, could it be me?
Chris: In a soapbox with a dress shirt and shorts my traditional style for the summer
Chris: and then like maybe holding like the like the a flag for Linux Unplugged it's
Chris: like rah you know and I'm like crying to the sky with my,
Chris: my rant I guess I don't know I wanted to you know invoke a real image your quest.
Wes: For your desktop freedom.
Chris: Yeah I think I just came up with a prompt yes.
Brent: You did can.
Chris: Someone just run that for.
Wes: An animated Jeff.
Chris: Thank you PJ for being our baller booster sir and glad you're listening.
Chris: Appreciate that support. You know, also just a quick note.
Chris: PJ mines those sats most, I don't know if these ones in particular,
Chris: but often what he boosts in, he mines with solar energy.
Chris: He uses clean solar energy to mine his sats and then he boosts in the show.
Chris: How about that? How cool is that?
Brent: For Jeff, those sats are as local as possible. They're made on his roof.
Chris: Yeah.
Wes: And, you know, Jeff sends us all kinds of value in all kinds of ways.
Chris: Magnolia Mayhem comes in with 2,123 sats. That's two, one, two, three.
Chris: Ah, I'm back at it, like an old pair of jeans. That's the last time I let my
Chris: filthy clanker touch my Nix config. Uh-oh.
Chris: Yeah, every now and then it goes bad, man. It can go real bad.
Chris: Wipe out a user account or two on you.
Wes: How would you know anything about that?
Chris: Wouldn't know anything about that. Thank you for the boost. Thank you,
Chris: Magnolia. Appreciate you.
Wes: Southern Fried Sasa comes in with a row of ducks. Oh, it's for me.
Chris: For Wes. Oh.
Wes: If the family wants to rid themselves of that radioactive artifact.
Chris: Oh, yeah. I don't know if the listeners thought this.
Wes: This was from the members feed.
Chris: Yeah, the members feed. You told us about something you discovered over the holiday.
Wes: Yeah, radioactive personnel identification marker from the World War II era.
Wes: Southern Sassa goes on to say, Most gun and knife shows will have at least one
Wes: guy who pedals in World War II memorabilia.
Chris: That's true.
Wes: No shipping required. Also, I had to tune into the video stream to see the infamous
Wes: kiddies. brent your co-stars are the true stars.
Chris: That's true sorry hey i completely agree and good idea.
Wes: There thank you.
Chris: Uh during our chat with uh kent for most of it,
Chris: brent's cat cosmo was completely wrapped around him like a scarf just completely
Chris: wrapped around stylish yeah yeah it even purrs thank you southern fraud appreciate you well.
Brent: So ham sent in 2000 sats,
Brent: Just a clarification on Git here. I don't mean the people shouldn't learn or be taught Git.
Brent: Rather, that it's basically impossible to teach people Git because of its incredible
Brent: complexity and just downright weird UI and UX.
Brent: For example, just take a look at, and they link us to a blog here,
Brent: it looks like, maybe a man page.
Brent: And tell me if that page is real or fake.
Wes: Yeah, you should check this out because it's a great site.
Chris: Okay, all right. What do you mean if it's real or fake? What do you mean?
Chris: What are you talking about?
Wes: Well, just take a peek. It's going to show you a man page.
Chris: Okay. Oh, git saw command. Is this real? Git exhibits, which blob?
Chris: Okay, I would have no idea.
Chris: I'm going to guess this is not a real command.
Wes: No, it's a generator of fake commands.
Chris: This is funny.
Wes: The point is they look shockingly like actual git.
Chris: That's a pretty great way to draw that point. That's pretty clever. I appreciate that.
Chris: You know, I was just, while Brent was reading that boost, I was just browsing GitHub.com.
Chris: It's just fascinating how nowhere on there does it say Git. It says GitHub.
Chris: Very little, but nothing about Git. Nothing. It's just...
Wes: No, not until you kind of get into the details. And if you use their,
Wes: like, you know, GH tool, even less.
Chris: Yeah. It's interesting.
Brent: You actually, with the web interface, you almost don't need to know Git if you're
Brent: just, like, managing a project or something like that.
Chris: Yeah. Yeah. Hey, there he is. It's Gene Bean, guys. He came in with a row of ducks,
Chris: It's as though I use iOS as my daily driver, I always keep some kind of phone
Chris: around running Lineage OS or something similar.
Chris: Given a reasonable price, I'd absolutely love for that device to be one that
Chris: shipped with Graphene OS.
Wes: Oh, yeah.
Chris: Mm-hmm. So that'd be fun. The kicker for me is that I need it to not be as pricey
Chris: as a flagship phone so that I can justify it as a secondary device.
Wes: Fair enough. Fair enough.
Chris: You know, as crazy as it sounds, there's a market for a secondary phone device.
Chris: I mean, I carry two phones.
Chris: And I agree you don't need the second phone to be the most cutting-edge phone.
Wes: They're small computers. And how many computers do we have already?
Chris: Maybe that's where the new Yala phone will fill a gap. Thank you,
Chris: Gene. It's always great to hear from you. Appreciate you very much.
Wes: Hybrid sarcasm comes in with 10,000 sets.
Wes: Just completed my holiday home lab submission. Well, thank you. That's great.
Chris: All right.
Wes: Be gentle, he implores us. Also, who's in the lead for the Boosties 2025 prize?
Chris: Oh, okay. All right. And remember, there's going to be a giveaway, right?
Chris: Hybrids mentioned this, a giveaway for a Jupiter Party membership.
Chris: Now, we don't want to give away the winners or the totals.
Chris: So we looked it up, we got the current numbers, and then Wes put the names through a randomizer.
Chris: So this is who's currently in the top four for the most boost to this show in 2025.
Chris: Black Host, Our Podcast, Adversary 17, and The Dude Abides.
Chris: All in the range of top four. That list was in no particular order. It was randomized.
Wes: And we are, I guess, in line with the Holiday Home Lab. I think the boosties will coincide, right?
Chris: I think that's the way we'll do it. We'll do it together. So it will be one
Chris: big spectacular episode for the holidays.
Wes: Which would mean that you have until, say, maybe the 18th or 19th to get any
Wes: boosts in if you do want to.
Chris: Be at the top.
Wes: Yeah. Reclaim your standing.
Chris: Thank you, hybrid.
Brent: Well, Daoma, Daoma boosts in 2,000 sets.
Brent: Lately, I've been able to leverage more open source tools, namely Ansible and
Brent: Open Tofu, at work as a sys engineer.
Brent: I find myself thinking about it constantly and working much later hours while
Brent: still feeling that I have a good work-life balance, even though I'm working for free, basically.
Brent: Should I find other work that would better utilize this passion?
Brent: Is it possible to just pick up an IT sysgig on the side, for instance?
Brent: I'm not a dev, so I struggle to understand
Brent: how I can contribute beyond submitting a PR on GitHub for a typo.
Chris: Well, infrastructure is a great skill set. You don't necessarily have to be
Chris: a developer, and of course they do blur more and more.
Chris: Side gig question, yes. I think small businesses are always looking for help.
Chris: They need somebody who can do part-time work because they can't afford a full-time
Chris: IT person often. So if you know a small business in your area...
Chris: It could be somebody that you patron and you've noticed they have computer problems.
Chris: That has literally worked for me in the past.
Chris: And then going back to essentially working for free, I often find myself when
Chris: I'm in the learning phase, I've got a lot of energy to just really commit to this.
Chris: And maybe sometimes an employer benefits from that. But I'm also benefiting.
Wes: Yeah, I think you can probably frame it as like an investment in yourself.
Wes: And there's a certain value, at least I have, I imagine you do too,
Wes: on, you know, curiosity, learning, and the sort of the fun play discovery aspects
Wes: of making this stuff work.
Chris: What's that Dunder Mifflin curve too about learning stuff, you know,
Chris: I'm talking about where at the beginning, you don't know what you don't know,
Chris: and you start to think you're an expert.
Chris: And then as you learn more, you realize you have so much more to learn,
Chris: and then you kind of go down that curve and become an expert again.
Chris: And I think, too, like you could burn through a bit of that phase during this,
Chris: kind of learn what you don't know, have a better understanding,
Chris: a wider context, and then absolutely make a career out of it.
Brent: For me, this is also something to be said for having a way to apply the things you're learning.
Brent: So sometimes the work environment is that place. And if you're going to be learning
Brent: it on your own, maybe you don't have the same opportunity to solve a problem
Brent: or implement a new solution.
Wes: I do think you're right. Infra is often something open source projects need.
Wes: Think it's probably a little tricky to get involved because you're probably
Wes: going to have to hang around a little while to figure that out.
Wes: It's not always advertised but I'm just thinking, you know, the Podverse folks
Wes: are hard at work on their latest versions and I know there's some folks who
Wes: are part of our community who help out on their infraside and that's...
Chris: Podverse is a great shout out. It's a great idea.
Wes: So I don't know about the particular communities that you're involved with necessarily
Wes: but there's probably something out there somewhere that needs some love.
Chris: Mm-hmm.
Wes: Retro Gear boosts in with 2,998 sats.
Wes: No, using Albi.
Chris: Oh, cool. Oh, yeah, connected to Fountain now. That's a beta feature.
Wes: Neat.
Chris: Yep, yep, Albi Hub. Mm-hmm, mm-hmm. Nosterwall connected.
Wes: Okay, Retro says, Gens, Proxmox NixOS actually works pretty well.
Wes: This is the sort of hybrid, right? It's like Proxmox, but it's running on NixOS.
Chris: I've been seeing a bit more buzz around this, so I'm interested to hear what he has to say.
Wes: I'm running a few VMs on there that can't be done with modules or Docker containers.
Chris: Ah.
Wes: Cheers.
Chris: I wonder what those would be.
Wes: Well, so basically, I think just like stuff you might want to run in a VM.
Wes: So as an alternative, right, instead of doing the setup we were doing with Home
Wes: Assistant, you could write Home Assistant on Proxmox, still running on Nix.
Chris: Right, right. Interesting.
Brent: That's a lot of layers, but it could solve some problems.
Chris: Yeah.
Wes: I mean, you do it. You probably want to have all the Proxmox stuff,
Wes: right? Like the API and all the...
Chris: Yeah, and if you're like baked into the Proxmox backup system and you know that
Chris: system, that makes a ton of sense.
Wes: Appreciate the experience report.
Chris: That's useful i'd love to hear more of them.
Brent: Well turd ferguson here sends us a boost 14 444 cents,
Brent: Hey, boys, did you see that Bazite pushed a petabyte of ISOs in the last 30 days?
Brent: It looks like they also just crossed 40,000 active weekly users.
Brent: Do you think everyone is loading these on Steam Decks or something?
Chris: Hmm. If they're hitting 40,000 active users, there must be some Bazite,
Chris: active Bazite users out in the audience.
Chris: I also would love some feedback on if you put it on a Steam Deck. I have my Steam Deck.
Chris: I have not been using as much the last few weeks because we went back to the
Chris: Switch for a smoother Breath of the Wild playback.
Chris: Priorities. I mean, that'd be a good time to reload that thing.
Chris: I'd love some field experience out there.
Chris: I kind of imagine it's people that are trying to do a Steam-like experience
Chris: on non-deck hardware. That would be my gut.
Brent: That makes sense to me as well.
Wes: Although it would work just fine if you just game a fair amount and also need a workstation.
Brent: Or if you've got, sort of like Jeff does, if you have a living room computer
Brent: that you want to have a similar experience, that might be a nice option.
Wes: True.
Chris: I think that makes Bazite clearly the winner of the UBlue projects.
Chris: That's great for them and probably great for all the other projects, too.
Chris: Well, there you go. Thank you, everybody, that boosted this show.
Chris: We really do appreciate it. And all of you who stream SATs.
Chris: 22 of you stream those SATs, and you collectively stacked 37,664 SATs.
Chris: That would be one of our largest boosts for this episode.
Chris: That would be the largest boost if you collected it all together. So thank you very much.
Chris: When you combine that with our messengers, we stacked 98,395 sats just under
Chris: that 100k sats line for this episode.
Chris: Not our best week, but we do appreciate everybody who contributes to the show
Chris: with their time, their talent, or their treasure, be it through a membership or through a boost.
Chris: We are here every week for you doing the show that only Linux Unplugged can
Chris: do. and that's because we're focused on our audience and not clickbait or things
Chris: that might play well on shorts or YouTubes or things like that.
Chris: We're just trying to make great quality content for our audience.
Chris: And thank you everybody who boosts with something like Fountain FM or AlbiHub
Chris: or one of those. We really do appreciate you.
Chris: Fountain does just keep getting easier with lots of features.
Chris: Fountain.fm for all of the hosted infrastructure.
Chris: Podcastapps.com for those new apps. And of course, if you'd like to support
Chris: with a membership, That's linuxunplugged.com slash membership and jupyter.party for all the shows.
Chris: I gave you a lot of URLs, but I trust you. You're pretty clever. You're pretty clever.
Chris: All right we got an overloaded pick section we should probably move through
Chris: this pretty quick so let's start with the one you probably shouldn't use dot
Chris: files installer dot files installer everybody what.
Wes: Is going on.
Chris: Here is this your.
Brent: Way of not using home manager.
Chris: No no no no no no no no i'm not using this although i did use it to back up
Chris: my dot files that was great um so it's really for people that publish different
Chris: different communities publish these curated .files.
Chris: So you can pull down one file that has all these configurations for popular
Chris: apps and desktop environments.
Chris: And it's really, it's a way, it's way beyond theming. It's a whole system vibe.
Chris: And you can have lots of different sources of these. You can have the problem
Chris: where you're overriding things.
Chris: What this application does is it lets you manage the various sources,
Chris: turn things on and off, back up your own things, and essentially try to prevent
Chris: any kind of loss that would mess your system up entirely while you're YOLOing
Chris: everybody's different configurations down.
Chris: And so it does try to prioritize data safety, offering a robust backup functionality
Chris: before you start just YOLOing different configs down.
Chris: And then you can also publish your own ones if you want to help other people
Chris: just YOLO into their config.
Chris: And it is a nice GTK application. It makes it very clear how you could possibly
Chris: be messing your system up.
Wes: You're making me like it.
Chris: I know, right? I know. I mean, I don't think anybody should do this.
Chris: Don't do this, blah, blah, blah. Safety first, all that.
Chris: Don't do as we do. But if you just had a system where you didn't care a whole
Chris: bunch or it was easy to roll back, this does make it kind of easy to roll back.
Wes: Or you intentionally want an experiment.
Wes: Okay, I'll try it.
Chris: Yeah, say you wanted to get yourself on Hyperlender or Sway or one of these,
Chris: and you don't really know how the hell to set up anything that looks nice.
Chris: Sometimes these can be a nice starting place.
Wes: That's a good point.
Chris: So the Stop Files installer is GPL3. I believe it's on Flathub as well.
Chris: Okay, now you ready for this next one?
Wes: Yeah, this one I kind of like.
Chris: This one for me, when macOS started getting good, now it's gone the other direction,
Chris: But, you know, when the first releases,
Chris: it was pretty rough, but they started getting to a point where a lot of independent
Chris: developers started getting on board and creating these purpose-built applications
Chris: that just make life a little bit simpler and easier for a lot of users on macOS.
Wes: Right. They could just focus on this, like, one little task and do that well.
Wes: And so it was pretty polished and it had all the, like, you know,
Wes: features that would come with working on it for a little while.
Chris: And it would kind of become, like, the go-to application for doing this.
Chris: Well, I hope that's SSH Studio.
Chris: It's a GUI that's GTK4, Libidwadia GUI.
Chris: That is an easy-to-use config editor for your SSH setup.
Chris: And it gives you what they call a visual host editor to edit common fields like
Chris: the host name, the host, the users, the identity files, any forwarding agent
Chris: setup. It does inline validation.
Chris: Of course, it helps you search across your machines. But I think the bigger
Chris: feature in here is really probably for folks who would be using a tool like
Chris: this is maybe the SSH key management stuff.
Chris: And it's if you connect to a handful of hosts, if this is an area that has sort
Chris: of been troubling for you.
Chris: Like this exact kind of example of these purpose-built applications that as
Chris: a platform grows in user base, we just start to see a demand for it.
Chris: And so in a way, I wanted to note it because I feel like it's important in that
Chris: way, but B, it's actually kind of a useful app. It's pretty lean, pretty mean.
Wes: Yeah, I mean, it just kind of shows you what's going on. It could be a nice
Wes: way to investigate a system.
Wes: It could also be a good way to discover some of the options that do exist for
Wes: your SSH config and connections if you haven't taken advantage of those.
Wes: And it's easy to install with flatback.
Chris: GPL 3.
Wes: Nice.
Chris: Last but not least is what I'm calling the honorable mensch section of the picks
Chris: because it's something we talk about too much.
Wes: And you couldn't resist talking about it some more.
Chris: They just came out with a great release. I found it personally very useful for
Chris: tracking my total cost of ownership of my vehicles.
Chris: We've talked about LubeLogger before, but a big version just dropped,
Chris: and I want to let you know about it because I think these are great.
Chris: First up, a new feature, inspections, pretty much what you'd expect is custom
Chris: forms where you can go inspect your own vehicle and create reusable inspection
Chris: forms, which is awesome for me.
Wes: That's great.
Chris: And create your own inspection checklist and then log that in LubeLocker. Huge, huge.
Chris: Also now support for households. So this is one of the most requested features of LubeLocker.
Chris: It's the ability to allow users to inherit vehicles in a garage and limit what
Chris: actions they can perform on those vehicles, kind of like a permission structure.
Chris: And so you no longer have to add a user and each vehicle. Instead,
Chris: you can add them to a household.
Chris: And then you just add that car, you add them to the household,
Chris: they automatically get access to the vehicles in your garage.
Chris: And they also created a viewer only, so a read-only user.
Chris: They are handling, I think, the integration of AI in a pretty reasonable way.
Chris: They say we know it's controversial. So what we're doing is instead of adding
Chris: AI directly into LubeLogger, we're asking for an API key. and then we've decided
Chris: to create an MCP server for LubeLogger that you spin up separately,
Chris: probably like a container or something like that.
Wes: Right.
Chris: And then you would use that as a bridge between AI agents. So instead of having
Chris: it built directly into main LubeLogger, it's like a sidecar.
Wes: Yeah, I mean, it seems reasonable enough, right? Basically, it's just repackaging
Wes: probably the existing API into the MCP standard.
Chris: Now, there are some features in there that are kind of neat.
Chris: It allows you to add fuel records from receipts, odometer records from a picture.
Wes: Oh, that's nice.
Chris: Yeah, so you just take a picture of your dashboard, Update the odometer reading.
Wes: You might actually do it.
Chris: Or like when we take our trips to any event that we drive to,
Chris: we have to track all the receipts.
Chris: And so like when we're on a road trip, we will sometimes just have like just
Chris: a massive bundle of receipts when we're driving across the continent.
Chris: And this time around, I was using, I think I don't quick scan and paperless,
Chris: you know, to try to capture it.
Wes: A little more real time.
Chris: A little more, but something like this for my maintenance records and stuff
Chris: like that, where I could just bring it right in. Just anything that makes that
Chris: smoother is really, really, really great.
Chris: And also invoices and those types of things to like for repairs.
Chris: It's still an experimental stage, but that's an, I think, an idea where it's
Chris: implemented not, so they're not slamming it in your face.
Chris: You can run this sidecar and you have opt in. And then, by the way,
Chris: once you do opt in, here's an actual couple of practical, useful things.
Wes: Not bad.
Chris: I think they nailed that. so I think that deserved an honorable mention huge
Chris: release for Lublogger and it has been,
Chris: it's really been a game changer for me because it helped me understand that
Chris: like this vehicle is getting, actually two of my vehicles are now beginning
Chris: to cost more than I want to spend on them in maintenance and you can track that
Chris: kind of stuff in a way that isn't tedious,
Chris: so I have not been historically great at that and Lublogger has made that very easy,
Chris: So thank you to them, and congratulations. Also, thank you to Kent for taking the time to join us.
Chris: He's a busy guy. He's got a lot of demands on his time, and we really appreciate
Chris: him taking the time to chat with us.
Chris: And we'll put some links to the new release in the show notes.
Chris: And don't forget, he's also on Patreon. So if you want to support his work directly,
Chris: that's how he's getting it done.
Chris: We'll try to put a link to that in the show notes, too, so people can do that.
Chris: And Wes, besides those show notes being over at linuxunplugged.com slash 644,
Chris: if they're in a mobile client that has a few fancy features,
Chris: they might get some bonus stuff. Some stuff that makes the show even better for them.
Wes: Yeah, cloud chapters for one. That's right. JSON delivered live right to your podcatcher.
Chris: And don't forget about those beautiful transcripts that are derived from individual
Chris: tracks of each host. So it is the most pristine transcription we can possibly generate for you.
Chris: And for the clients that support it, we are also doing attribution.
Chris: So it tells you which speaker is speaking.
Chris: That isn't easy, but we do it for you. And it's out there.
Chris: And hopefully, you know, as we just do more episodes, we're going to create
Chris: a huge data set that we'll use for something one day.
Chris: I don't know what. And then last but not least, as we round out the year.
Chris: We sure would love to have you join us live. JBLive.tv on a Sunday.
Chris: Make it a Tuesday on a Sunday, 10 a.m. Pacific, 1 p.m. Eastern.
Wes: Hey, that is also technically a magical feed feature.
Chris: That's true. It's in those feeds. And then don't forget December 19th live.
Chris: It is the home lab holiday special and you need to get your submissions in by that Wednesday.
Chris: So that would be the 17th. Get your submissions in at Linux unplugged.com slash holiday.
Chris: Thank you so much for joining us on this week's episode. We sure appreciate
Chris: you listening and we'll see you right back here next Tuesday as in Sunday.