WEBVTT 00:00:11.249 --> 00:00:15.789 Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. 00:00:15.989 --> 00:00:16.649 My name is Wes. 00:00:16.849 --> 00:00:17.689 And my name is Brent. 00:00:18.209 --> 00:00:21.709 Well, hello, gentlemen. Coming up on the show today, it's a big one. 00:00:21.809 --> 00:00:26.269 Kent Overstreet, the creator of BcacheFS, joins us to give us a full update 00:00:26.269 --> 00:00:31.109 on what's going on, what's new, what's next, and a surprising upside of getting 00:00:31.109 --> 00:00:32.169 kicked out of the kernel. 00:00:32.329 --> 00:00:34.909 And then we're going to round it all out with some great picks, 00:00:35.269 --> 00:00:39.449 some boosts, actually like a really loaded pick section, and a lot more. 00:00:39.449 --> 00:00:43.069 So before we get to all of that, let's give the old time appropriate greetings 00:00:43.069 --> 00:00:44.929 to our virtual lug. Hello, Mumble Room. 00:00:48.253 --> 00:00:52.853 Hello. Good afternoon. Hello. Just a couple of you in the on-air and a group 00:00:52.853 --> 00:00:55.793 of you up there in the quiet listening as well. Mumbarum always cooking. 00:00:56.133 --> 00:00:59.473 We're always happy to have everybody in there. And a big good morning to our 00:00:59.473 --> 00:01:01.433 friends over at Defined Networking. 00:01:01.553 --> 00:01:05.473 Defined.net slash unplugged. Go check out Managed Nebula. 00:01:05.733 --> 00:01:09.973 It's built on top of the open source Nebula project. They handle the tricky 00:01:09.973 --> 00:01:11.573 bits that you might not want to think about. 00:01:11.833 --> 00:01:13.833 100 devices totally for free. 00:01:14.293 --> 00:01:17.693 No credit card required. Defined.net slash unplugged. this 00:01:17.693 --> 00:01:21.253 is a mesh vpn optimized for speed simplicity and 00:01:21.253 --> 00:01:24.493 really great security but ultimately it kind 00:01:24.493 --> 00:01:27.753 of goes a little bit beyond when you're picking your network infrastructure if you've 00:01:27.753 --> 00:01:33.373 been in tech for a minute you know that there's also the company behind the 00:01:33.373 --> 00:01:37.233 product and what they're all about and what their goals are and i think the 00:01:37.233 --> 00:01:40.333 way they're configured the way they're designed all of that stuff ultimately 00:01:40.333 --> 00:01:46.053 ends up getting represented long term in the product and that's what I think to find has been really, 00:01:46.193 --> 00:01:51.013 really judicious in how they have built and designed themselves to really align 00:01:51.013 --> 00:01:53.833 with how I would like a company that's building this type of tool. 00:01:53.993 --> 00:01:56.433 And I think you would also probably align with that if you're a listener to 00:01:56.433 --> 00:01:59.453 this show. So it goes just beyond the software and the fact this is a truly 00:01:59.453 --> 00:02:01.913 free project that's totally self-hostable if you'd like. 00:02:02.733 --> 00:02:06.393 It's optimized for speed, simplicity. It uses less battery on your devices, 00:02:06.393 --> 00:02:07.553 less network resources. 00:02:07.853 --> 00:02:11.433 You can host it completely or you can use their managed solution. But beyond just that. 00:02:12.698 --> 00:02:15.258 It's about the company architecture. It's about the funding model. 00:02:15.378 --> 00:02:19.098 It's about all these things that go into a long-term sustainable product that 00:02:19.098 --> 00:02:20.618 you can build a network on top of. 00:02:20.818 --> 00:02:23.258 And they have been solid since 2017. They had to be. 00:02:23.478 --> 00:02:26.278 They had to go in production day one for Slack and their global infrastructure. 00:02:26.738 --> 00:02:29.138 Now, now it's incredible what they power. 00:02:29.798 --> 00:02:33.298 Rivian cars on the road, all the metrics they send back to Rivian HQ, 00:02:33.518 --> 00:02:34.938 that's secured by Nebula. 00:02:35.438 --> 00:02:39.958 Think about all of those road, all those machines on the road communicating over Nebula. 00:02:40.238 --> 00:02:43.218 They have to keep that secure that would be such a scandal if anything ever happened for 00:02:43.218 --> 00:02:45.918 rivian it's really impressive it has 00:02:45.918 --> 00:02:50.038 an incredible level of resilience speed and scalability that you can tune to 00:02:50.038 --> 00:02:53.718 your needs from a home lab to an enterprise so go get started 100 hosts absolutely 00:02:53.718 --> 00:02:59.438 free support the show no credit card required go to defined.net slash unplugged 00:02:59.438 --> 00:03:05.498 well you know it is actually very scary Very close. 00:03:05.598 --> 00:03:07.278 So we wanted to give everybody a 00:03:07.278 --> 00:03:11.718 heads up that your time to submit to the Holiday Home Lab is running out. 00:03:22.541 --> 00:03:26.841 We want your holiday home lab submission. Submit your home lab for Linux Unplugged 00:03:26.841 --> 00:03:28.181 holiday special coming up. 00:03:28.321 --> 00:03:32.921 We're looking for your creative or efficient or chaotic, overbuilt, just beginning setups. 00:03:33.141 --> 00:03:36.781 We want everything from the community and we'll feature some of the winners on our show. 00:03:37.281 --> 00:03:39.901 We'll have probably a batch that make it on the show. We'll have a batch that 00:03:39.901 --> 00:03:43.981 we'll maybe do in special editions and all kinds of stuff. We have a form over 00:03:43.981 --> 00:03:46.181 at LinuxUnplugged.com slash holiday. 00:03:46.661 --> 00:03:50.861 Your time is almost out because we have picked our live record date, 00:03:50.961 --> 00:03:52.581 which we would invite you to join us. 00:03:52.761 --> 00:03:59.001 It'll be on Friday, December 19th, 2025, when we record the holiday home lab special. 00:03:59.321 --> 00:04:02.181 So if you'd like to join us for the live event, especially if you've submitted 00:04:02.181 --> 00:04:05.341 your home lab Friday, the 19th, we'll have 00:04:05.341 --> 00:04:08.961 the exact details and time at jupiterbroadcasting.com slash calendars. 00:04:09.061 --> 00:04:12.721 We get closer, but more importantly, because I sometimes forget that bit. 00:04:12.881 --> 00:04:15.901 We always put it in the podcasting 2.0 feed. So if you have a podcasting to 00:04:15.901 --> 00:04:17.861 do app, you'll see it in there when we're going to be live. 00:04:18.241 --> 00:04:23.921 It'll be, I think our last live episode, maybe not, but one of our last live episodes of the year. 00:04:24.061 --> 00:04:26.881 We, we, I don't know if we have after Christmas quite dialed in yet, 00:04:26.881 --> 00:04:28.621 but it's getting pretty close. 00:04:28.881 --> 00:04:34.541 And so you have, um, well, you don't, you got to get in before the 19th because we need days. 00:04:34.761 --> 00:04:37.961 So I guess we should have probably told you a deadline, not just when we're going to do it live. 00:04:38.241 --> 00:04:43.941 So I'm going to say the deadline is the 17th. that gives us at least two days 00:04:43.941 --> 00:04:45.961 to go through all the submissions right yeah. 00:04:45.961 --> 00:04:46.701 That seems reasonable. 00:04:46.701 --> 00:04:52.221 So december 17th we're going to cut off no guarantees yeah so there's your cutoff 00:04:52.221 --> 00:04:56.721 and then the live show will be on the 19th and we'll hopefully have uh more 00:04:56.721 --> 00:05:01.541 details on exactly when probably we could probably just safely assume our regular time seems. 00:05:01.541 --> 00:05:03.541 Like it sure why not. 00:05:03.541 --> 00:05:06.421 Yeah there you works for me it's good it's gonna be fun it's gonna be a lot 00:05:06.421 --> 00:05:07.961 of fun and we've had some great submissions. 00:05:07.961 --> 00:05:09.821 You can't have a holiday without a home lab. 00:05:12.892 --> 00:05:16.112 No, you've got to play your Christmas mixes and stuff. Well, 00:05:16.192 --> 00:05:21.332 last week we played some clips to just get you up to date on the BcacheFS current 00:05:21.332 --> 00:05:25.392 status and the kernel and honestly our real-time reactions because a lot of 00:05:25.392 --> 00:05:26.312 that developed while we, 00:05:26.472 --> 00:05:30.072 in both times there were major developments while we were traveling. 00:05:30.532 --> 00:05:30.632 Yeah. 00:05:31.092 --> 00:05:33.012 So we were like processing all of that. 00:05:33.212 --> 00:05:38.032 Attempting to travel or attend a conference or, you know, not think about file systems. 00:05:38.172 --> 00:05:40.852 So if you missed that last week, you might want to go grab that context. 00:05:40.852 --> 00:05:44.892 So that way you have like sort of the full context for today's conversation when Kent joins us. 00:05:45.352 --> 00:05:48.312 And as the year rounds out, we thought this is a topic we should talk about 00:05:48.312 --> 00:05:50.032 because there's been a lot of changes. 00:05:50.152 --> 00:05:54.572 Bcache FS was kicked out of the Linux kernel. We talked about that last week. 00:05:55.192 --> 00:05:58.292 But I think it's best to get the information straight from the horse's mouth. 00:05:58.332 --> 00:05:59.752 And in this case, that's Kent Overstreet. 00:05:59.972 --> 00:06:03.252 So we're going to get into this in the way that only Linux Unplugged does with 00:06:03.252 --> 00:06:04.852 the details and the technical stuff. 00:06:06.092 --> 00:06:10.672 And the best source, of course, is Kent. He's the creator of Bcache FS. 00:06:10.852 --> 00:06:13.692 And he's a rare, rare breed of developer. 00:06:14.312 --> 00:06:18.172 And I think as you listen to our chat with him, this is going to become evident to you. 00:06:18.892 --> 00:06:21.472 I wanted to mention something that maybe isn't obvious, though. 00:06:23.030 --> 00:06:27.110 Kent is creating what I consider to be maybe the most modern file system, 00:06:27.490 --> 00:06:30.850 period, right now. And he's making it GPL and available for Linux. 00:06:31.910 --> 00:06:35.750 And he's doing it from what he calls a glamping tent attached to the back of 00:06:35.750 --> 00:06:41.650 a truck in South America via a Starlink connection, living off of the support of his patrons. 00:06:42.350 --> 00:06:46.810 And he is developing every single day. He committed, this is a Sunday morning, 00:06:46.930 --> 00:06:48.110 and he committed three hours ago. 00:06:48.250 --> 00:06:48.930 He sure did. 00:06:48.930 --> 00:06:52.610 And then he had released a major version, which we will get into, 00:06:52.730 --> 00:06:57.270 with major new features a few hours before our chat that day, 00:06:57.330 --> 00:06:59.050 because we talked to Kent on Thursday. 00:06:59.290 --> 00:06:59.590 Yeah. 00:07:00.330 --> 00:07:04.310 And he had just released a massive, massive update. 00:07:04.450 --> 00:07:07.210 And he's just constantly working. He's constantly communicating. 00:07:07.210 --> 00:07:12.750 And he's doing this while creating this project while being nomadic and keeping 00:07:12.750 --> 00:07:14.230 his cost of living as low as possible. 00:07:14.690 --> 00:07:17.830 So I just think he's an incredibly interesting developer from that standpoint. 00:07:17.830 --> 00:07:19.070 We're very lucky to have him. 00:07:19.630 --> 00:07:22.330 And just the insights here are great. So if you're not familiar with BcashFS, 00:07:22.570 --> 00:07:24.370 buckle up. And if you are, there's a lot to love in here, too. 00:07:24.950 --> 00:07:28.470 Ken Overstreet, welcome back to the show. It's great to have you, sir. 00:07:29.030 --> 00:07:29.730 Great to be here. 00:07:30.170 --> 00:07:36.130 And also, congrats. BcashFS 1.33 came out this morning, I think. 00:07:36.610 --> 00:07:37.810 Metadata version reconcile. 00:07:38.490 --> 00:07:39.230 Huge, right? 00:07:39.590 --> 00:07:40.270 Tied it perfectly. 00:07:40.570 --> 00:07:43.110 I think we should start right there, and then we'll kind of get into other stuff. 00:07:43.250 --> 00:07:46.690 But let's talk about the new reconcile system, because reading through the show 00:07:46.690 --> 00:07:50.250 notes, it It looks like it was, it looks like one of those things on the surface 00:07:50.250 --> 00:07:51.810 seems like a small amount of work. 00:07:51.810 --> 00:07:56.610 But then when you dig into the details, it was like almost a total refactoring of type. 00:07:56.850 --> 00:08:00.790 Total, but it touched all kinds of stuff. Yeah. Like had to get metadata plugged 00:08:00.790 --> 00:08:03.830 in, all kinds of just little bits here and there. 00:08:04.090 --> 00:08:10.050 Because whereas before, this grew out of Rebalance, which just did background 00:08:10.050 --> 00:08:11.290 compression and background target. 00:08:11.290 --> 00:08:16.630 Now we're aiming for strict correctness so that we can actually check the validity 00:08:16.630 --> 00:08:19.530 of all stored data against the IO path options. 00:08:19.950 --> 00:08:24.790 So if you're doing correctness checking, that tends to be a different level. 00:08:26.030 --> 00:08:26.990 So big project. 00:08:26.990 --> 00:08:34.150 And it's actually done now so um it's congratulations just reading through that i just like. 00:08:34.150 --> 00:08:34.670 This is a. 00:08:34.670 --> 00:08:38.090 Big piece of work and it's pretty cool that it landed right before you were 00:08:38.090 --> 00:08:43.330 able to join us now as a somebody who's using bcachefs now will i need to will 00:08:43.330 --> 00:08:46.170 i need to prepare or make any changes before i update. 00:08:46.170 --> 00:08:49.790 Uh the only thing to be aware of is uh there 00:08:49.790 --> 00:08:52.450 will be an upgrade process uh when you 00:08:52.450 --> 00:08:55.510 mount on the new version one of the things that should be coming soon 00:08:55.510 --> 00:08:58.350 but didn't land for this release is we should be able to do these 00:08:58.350 --> 00:09:01.070 upgrades in the background and we're still doing 00:09:01.070 --> 00:09:06.110 it at mount time and we still haven't got this communication with systemd going 00:09:06.110 --> 00:09:09.830 to let systemd know that we're actually doing stuff at mount and it just uh 00:09:09.830 --> 00:09:14.690 thinks that we've timed out so that's that's a little bit of an annoyance but 00:09:14.690 --> 00:09:18.550 just just let it go until the kernel log says it's finished start the d-message, 00:09:18.650 --> 00:09:19.370 whatever you're watching, 00:09:19.930 --> 00:09:24.030 and reboot it again if you need to because of the systemd thing and it'll be fine. 00:09:25.255 --> 00:09:30.335 We had a lot of that back in like the 614 upgrade for back pointers or i had 00:09:30.335 --> 00:09:35.135 to redo something some minor detail of the back pointers actually had to rewrite 00:09:35.135 --> 00:09:36.435 half of everyone's metadata. 00:09:38.075 --> 00:09:40.555 I think i remember upgrading from that one yeah. 00:09:40.555 --> 00:09:45.895 Yeah yeah no one noticed anything besides going hey why were my mouse taking 00:09:45.895 --> 00:09:50.375 so long people would kick it a couple times and it would just finally finished 00:09:50.375 --> 00:09:57.535 and so that was fun this is not as big an upgrade but okay okay. 00:09:57.535 --> 00:10:02.515 That puts it in perspective all right so let's walk it back a little bit it's 00:10:02.515 --> 00:10:06.555 been just about a year or more since we talked i can't quite recall to be honest 00:10:06.555 --> 00:10:10.175 but it feels like it's been five years i mean time flies it's. 00:10:10.175 --> 00:10:12.855 Just shy uh 693 days. 00:10:12.855 --> 00:10:15.355 Oh okay oh so okay yeah it's been. 00:10:15.355 --> 00:10:16.835 A bit so. 00:10:16.835 --> 00:10:20.975 Um things have changed a Let's talk a little bit about user base, 00:10:20.995 --> 00:10:22.635 who's using it these days. 00:10:23.115 --> 00:10:26.215 Uh, maybe how just off the top of your head, I know this is kind of a wild one, 00:10:26.295 --> 00:10:29.535 but how, how you say things have changed since 2024, just kind of a, 00:10:29.595 --> 00:10:31.435 the best recap that you can recall. 00:10:31.555 --> 00:10:34.855 The biggest, biggest changes that things have really stabilized. 00:10:35.995 --> 00:10:39.835 Even like six months ago, you can probably three months ago, 00:10:40.015 --> 00:10:44.295 like back in July, I was still in the midst of, you know, the, 00:10:44.295 --> 00:10:47.435 the bugs are coming in about as fast as I could handle them and things that, you know, 00:10:48.673 --> 00:10:54.033 I could see the trend back in, like, 6.14, but there was a noticeable inflection 00:10:54.033 --> 00:10:59.413 point, like 6.16, 6.17, and now it's like things are starting to feel calm. 00:11:00.053 --> 00:11:04.093 Past couple months were a giant slog with getting Reconcile out the door, 00:11:04.253 --> 00:11:07.073 but I'm feeling pretty good about where we're at. 00:11:07.553 --> 00:11:10.393 And we're getting really close to taking the experimental label off. 00:11:10.753 --> 00:11:12.153 I pegged that for Christmas. 00:11:12.853 --> 00:11:14.793 What a lovely present, huh? 00:11:15.653 --> 00:11:19.433 Yeah, yeah. Yeah, there's like almost nothing left. 00:11:20.133 --> 00:11:23.093 Yeah, I was going to ask, you mentioned sort of, you know, assessing stability 00:11:23.093 --> 00:11:25.973 and here getting ready to take off that experimental label. 00:11:26.153 --> 00:11:29.333 What actually are you looking for? Because I know you're helping people one-on-one. 00:11:29.333 --> 00:11:32.353 You've also got like actual, you know, bugs getting filed upstream. 00:11:32.353 --> 00:11:34.153 Probably you have like automated tests running. 00:11:34.493 --> 00:11:38.913 Lots and lots of automated testing, yeah. There's a whole cluster of machines off in Finland. 00:11:39.593 --> 00:11:43.933 Big 80 core R machines with a quarter terabyte Rm each. And I run those machines hard. 00:11:44.533 --> 00:11:47.353 So i'm constantly watching the test dashboard which is never 00:11:47.353 --> 00:11:50.533 as clean as i would like it but i'm always 00:11:50.533 --> 00:11:53.273 watching like triaging and okay that one can 00:11:53.273 --> 00:11:58.493 wait no that's actually something serious i need to jump on now and uh most 00:11:58.493 --> 00:12:04.813 of it is just how are user feedback am i are the bug reports coming in how severe 00:12:04.813 --> 00:12:10.553 are those bug reports how quickly are we dealing with them are we finding stuff that it's like, 00:12:11.113 --> 00:12:15.553 oh that repair path needs a ton of work or is it like oh that's a that's like 00:12:15.553 --> 00:12:17.173 a trivial 10 or 20 line fix, 00:12:18.370 --> 00:12:20.130 So it's a lot of things. 00:12:20.470 --> 00:12:24.590 I'm thinking back, it was really still pretty experimental at the time. 00:12:24.770 --> 00:12:28.750 It feels like now we're at a point where if I deploy it, I'm probably not going 00:12:28.750 --> 00:12:31.390 to see any big format changes or anything that's really going to be disruptive 00:12:31.390 --> 00:12:33.650 to my workflow. And that's huge, right? 00:12:34.330 --> 00:12:38.530 Yeah. One of the things with format changes is I've actually gotten less conservative 00:12:38.530 --> 00:12:42.470 with doing format changes since we're no longer upstream. We're DKMS. 00:12:42.670 --> 00:12:43.290 Interesting. Okay. 00:12:43.470 --> 00:12:47.990 Because, yeah, we no longer have users running a whole bunch of different versions. 00:12:48.370 --> 00:12:49.910 And maybe rebooting it to older kernels. 00:12:50.330 --> 00:12:54.110 It's very easy now for users to just stay on the latest version. 00:12:54.370 --> 00:12:58.310 And as long as I'm not pushing out regressions and we've been doing great on 00:12:58.310 --> 00:13:02.630 regressions, then there's no reason for users to have to roll back to older 00:13:02.630 --> 00:13:04.610 versions of the PCacheFS, which we can do. 00:13:06.449 --> 00:13:12.209 It was an expensive downgrade and then re-upgrade in the kernel. 00:13:12.369 --> 00:13:16.129 And now it's like, if we have to change something and there's a legitimate reason, 00:13:16.429 --> 00:13:22.709 like reconcile, we've got another one coming for per-device fragmentation LRUs. 00:13:22.709 --> 00:13:25.349 That'll fix some issues with copy GC and tiering. 00:13:26.549 --> 00:13:29.949 Then it's just not as big a deal, especially once in the next couple of weeks 00:13:29.949 --> 00:13:33.549 we could do those upgrades in the background. Then it'll be completely smooth. 00:13:33.829 --> 00:13:36.229 Yeah, then we really probably won't notice, huh? 00:13:36.449 --> 00:13:36.909 Yeah. 00:13:37.389 --> 00:13:41.249 So, Kent, when all of this was going down with the discussion, 00:13:41.589 --> 00:13:47.349 will they or won't they extract BcashFS, did this benefit even occur to you during all of that? 00:13:47.809 --> 00:13:52.409 No, that one actually did not. The whole DKMS transition went much, 00:13:52.469 --> 00:13:53.869 much smoother than I expected. 00:13:54.549 --> 00:13:57.409 I had a lot of concerns and fears at the time how things were going to turn 00:13:57.409 --> 00:14:00.549 out, and it turned out actually better than I could have expected. 00:14:01.069 --> 00:14:03.229 How so? I mean, besides obviously what we just talked about, 00:14:03.289 --> 00:14:04.129 but in other ways as well? 00:14:04.749 --> 00:14:08.849 Yeah, I didn't seem to lose users like I expected. 00:14:09.769 --> 00:14:14.229 The people who actually care stuck with it. We were able to get the DKMS packages 00:14:14.229 --> 00:14:17.509 done, all the automation, everything we needed. 00:14:17.689 --> 00:14:20.669 That came together really fast because a lot of people jumped in and helped out. 00:14:21.709 --> 00:14:27.689 It's shocker in hindsight, not so obvious going in, but it's a lot easier for 00:14:27.689 --> 00:14:30.789 people to help out with a non-core kernel file system stuff. 00:14:31.409 --> 00:14:34.809 And there's a lot of people who are interested and want to help out, 00:14:34.849 --> 00:14:39.829 but they're understandably intimidated by diving into a 100,000 line C code 00:14:39.829 --> 00:14:43.849 base, but like packaging stuff, yeah, that was a lot easier. 00:14:44.069 --> 00:14:47.069 I hadn't even thought about that, Kent, so I actually appreciate you bringing 00:14:47.069 --> 00:14:49.509 that up. I guess what I have been stuck on, 00:14:50.454 --> 00:14:53.534 I guess it seems like it's a good thing for Bcache in the short term, 00:14:53.694 --> 00:14:56.374 but it's not a great thing for Linux in the long term. 00:14:57.214 --> 00:15:01.974 I have really expressed on the show, I really think Linux needs a sophisticated, 00:15:02.374 --> 00:15:06.294 modern file system that's GPL and built in. I think it's imperative. 00:15:07.074 --> 00:15:10.354 Yeah, I don't think the Linux file system world is in great shape, 00:15:10.354 --> 00:15:11.914 and I feel for those guys. 00:15:11.914 --> 00:15:17.634 But, you know, a lot of this stuff, there's fairly deep-seated problems, 00:15:17.634 --> 00:15:23.054 and I wasn't able to make any headway, and I can't do anything if no one wants 00:15:23.054 --> 00:15:26.754 to, if everyone is stuck in their ways and doesn't want to change. 00:15:26.954 --> 00:15:31.694 Is it momentum of just history and people that have their fiefdoms? 00:15:31.694 --> 00:15:34.294 Is it just that? It's built fighting through that? 00:15:34.654 --> 00:15:39.674 That's some of it. Yeah. There's a lot of reasons. 00:15:39.674 --> 00:15:45.014 You know, the kernel is a big old project and, you know, basic processes, 00:15:45.234 --> 00:15:50.854 how we do things tend to, you know, in any big institution tend to ossify and 00:15:50.854 --> 00:15:51.854 become harder to change. 00:15:52.834 --> 00:15:56.694 And the Linux kernel came of age, you know, really back in the 90s, 00:15:56.794 --> 00:16:01.034 early 2000s, with everything being done on the mailing list, 00:16:01.174 --> 00:16:06.274 patch-based workflow, didn't even use source control for years and years famously. 00:16:07.474 --> 00:16:12.014 And the one place where it really lags today is like automated testing, QA. 00:16:13.214 --> 00:16:20.894 So one of the pain points for me of being in the kernel is I do rely very heavily on automated testing. 00:16:21.094 --> 00:16:25.254 And if you're in the kernel, you're expected to stay on mainline, 00:16:25.354 --> 00:16:28.394 rebase onto RC1s every time RC1 comes out. 00:16:28.574 --> 00:16:33.434 And that's a good thing. You want people to be dogfooding and catching issues like that. 00:16:35.096 --> 00:16:40.016 If the file system people are the only people doing really comprehensive stress 00:16:40.016 --> 00:16:41.996 and torture testing, and there's 00:16:41.996 --> 00:16:45.656 other subsystems in the kernel that aren't testing the code at all, 00:16:47.096 --> 00:16:52.176 that's really painful, especially if you're like me, where I'm just one guy 00:16:52.176 --> 00:16:54.096 with 100,000 lines of code, and if. 00:16:56.656 --> 00:17:00.956 Things aren't working right, it'd be like every other release. 00:17:01.516 --> 00:17:06.056 My test dashboard would blow up, and it'd be a couple weeks where it'd be tracking 00:17:06.056 --> 00:17:11.476 people down, and I can't get work done if my test dashboard is a whole bunch 00:17:11.476 --> 00:17:14.936 of red, and I can't tell if my own patches are buggy or not. 00:17:18.896 --> 00:17:23.996 Think about that for the kernel as a whole. It boggles my mind trying to get 00:17:23.996 --> 00:17:27.956 work done in that environment without automated testing and not being able to 00:17:27.956 --> 00:17:30.936 get good feedback on what code is working or not. 00:17:30.936 --> 00:17:38.316 They're spending way too much time chasing regressions and other people's code and it's not fun. 00:17:38.316 --> 00:17:41.016 It does make me reflect just on all the 00:17:41.016 --> 00:17:43.956 pros and cons to the coupling to the 00:17:43.956 --> 00:17:47.016 kernel if you are coupled you get the advantage that it's there by default but 00:17:47.016 --> 00:17:51.556 if you're uncoupled there's a lot of pros although also of course some cons 00:17:51.556 --> 00:17:55.476 I was just curious how you thought it was going being out of the kernel besides 00:17:55.476 --> 00:17:59.996 DKMS I know there were some symbols that were no longer exported that had to 00:17:59.996 --> 00:18:01.496 get Vendored like that kind of thing. 00:18:01.636 --> 00:18:05.396 Do you anticipate being able to keep the current model working without too much of us? 00:18:06.436 --> 00:18:13.576 I'm curious how that's gonna go The the core kernel interfaces that file systems 00:18:13.576 --> 00:18:17.856 depend on and be cache of us They've gotten a lot less journey than than they used to be, 00:18:18.376 --> 00:18:22.216 so I don't expect things to be too painful but, 00:18:23.650 --> 00:18:29.250 The kernel is not very friendly to out-of-tree projects and codes, so it's a question mark. 00:18:29.630 --> 00:18:32.390 It's fair. That's fair. It doesn't make me feel much better, 00:18:32.430 --> 00:18:36.350 but it's a fair answer. I'm sure you probably feel very much the same. 00:18:36.450 --> 00:18:36.630 We shall see. 00:18:37.110 --> 00:18:37.390 Yeah. 00:18:37.630 --> 00:18:37.850 Yeah. 00:18:38.270 --> 00:18:42.630 All right. Well, I really appreciate you addressing that. That actually gives 00:18:42.630 --> 00:18:45.630 me, I think, a better understanding overall of the situation. 00:18:45.950 --> 00:18:49.670 I mean, as us as users, I've been really impressed how some of my favorite distributions 00:18:49.670 --> 00:18:52.970 have just made it. But they've really stepped up to make it pretty simple for 00:18:52.970 --> 00:18:54.410 me to just keep using BcacheFS. 00:18:55.528 --> 00:18:58.648 Yeah, the NixWest and Arch people have been awesome to work with. 00:18:58.848 --> 00:19:02.768 I haven't had to do anything for those distros besides keep in contact and make 00:19:02.768 --> 00:19:04.528 sure we're on the same page. 00:19:04.648 --> 00:19:10.848 And those guys hang out in the BcashFS IRC channel and we give each other a 00:19:10.848 --> 00:19:13.928 shot whenever something comes up. And generally things are just chill and awesome. 00:19:14.268 --> 00:19:20.448 Okay, so you made a hint about a lab. And I've heard you kind of made reference 00:19:20.448 --> 00:19:24.588 to some pretty significant large BcashFS deployments. 00:19:24.588 --> 00:19:27.408 It's just, I don't know, off the top of your head, what are some of the larger 00:19:27.408 --> 00:19:30.248 that you're aware of that people have been experimenting with or using in production? 00:19:30.408 --> 00:19:37.208 Oh, actually, I'm not sure what references I've made because I honestly do not know. 00:19:38.348 --> 00:19:44.648 I hear from people when stuff breaks and if things are working great, 00:19:44.808 --> 00:19:50.288 then like there have been quite a few times when, you know, 00:19:50.768 --> 00:19:53.968 the rare regression, user impacting regression comes through. 00:19:53.968 --> 00:19:57.548 And I find out that there's actually a lot more users than I thought there were. 00:19:58.148 --> 00:20:05.128 Like the Debian brouhaha, where the maintainer shipped a package and broke the 00:20:05.128 --> 00:20:06.408 build and then sat on updates. 00:20:06.548 --> 00:20:12.328 And there was a bug in mount where users couldn't pass the degraded option. 00:20:13.148 --> 00:20:16.588 And I start getting all these bug reports from users I've never heard of that 00:20:16.588 --> 00:20:20.108 I just had a drive die and I can't mount my file system. 00:20:21.808 --> 00:20:27.308 So, yeah, I have no kind of telemetry or statistics, and I would really love 00:20:27.308 --> 00:20:31.888 to get telemetry and just see all the data I possibly can when something breaks, 00:20:32.128 --> 00:20:33.768 but I don't have that yet. 00:20:34.088 --> 00:20:34.888 Okay, okay. 00:20:35.148 --> 00:20:38.548 Have you run into any interesting systems, though, or heard from folks reporting 00:20:38.548 --> 00:20:40.388 successful or otherwise deployments? 00:20:40.628 --> 00:20:43.788 Oh, there's definitely some big file systems out there. There's been, 00:20:43.928 --> 00:20:48.428 for quite a while now, file systems north of 100 terabytes. 00:20:50.229 --> 00:20:54.829 I mentioned the 6.14 backpointers upgrade. That was specifically because people 00:20:54.829 --> 00:21:00.529 are running into scalability issues, and at 100 terabytes, FS checks were taking more than a day. 00:21:01.389 --> 00:21:06.949 And now we can scale up. We tested scalability up to like 10 or 12 petabytes, 00:21:09.529 --> 00:21:12.329 and no one's hitting scalability issues anymore. 00:21:14.309 --> 00:21:19.009 Okay, well, I'm nowhere near those numbers, but I've kind of this year started 00:21:19.009 --> 00:21:22.009 building my home lab, rebuilding my home lab. 00:21:22.109 --> 00:21:25.089 It's really gonna be a project for next year. And I think I want to do Bcash FS. 00:21:25.869 --> 00:21:30.189 And I just sort of wanted to pick your brain where you think Bcash FS is for 00:21:30.189 --> 00:21:34.649 home labbers, if that's maybe a good use case, some things I should maybe try or know about. 00:21:34.829 --> 00:21:36.889 Anything like that, you know? Like if I got you on the line, 00:21:37.069 --> 00:21:38.469 I gotta be selfish and I gotta ask. 00:21:38.809 --> 00:21:42.949 Yeah, yeah. Oh, I love working with the home labbers. Those are the guys that 00:21:42.949 --> 00:21:45.029 hang out in the IRC channel and that, 00:21:46.023 --> 00:21:49.983 We have the most productive working relationships because those are the guys 00:21:49.983 --> 00:21:54.003 that like to tinker and investigate and discover how things break. 00:21:55.103 --> 00:21:58.923 And that's where we find a lot of the weird corner cases. And that's where the 00:21:58.923 --> 00:22:00.503 polish comes from is working with those guys. 00:22:01.363 --> 00:22:03.983 Okay. So if I'm going down the route. 00:22:04.103 --> 00:22:04.783 Go for it. 00:22:05.023 --> 00:22:08.023 Yeah. I mean, just any tips, anything I need to worry about? 00:22:08.123 --> 00:22:12.283 Is it just deploy, any mount options? I want any pro tips you got for me. 00:22:12.303 --> 00:22:18.003 One angle might be like RAID 5.6. Is that available yet? like how she structures the array maybe if. 00:22:18.003 --> 00:22:18.743 You got any tips. 00:22:18.743 --> 00:22:21.543 Yeah uh aside from erasure coding we'll 00:22:21.543 --> 00:22:24.663 get to that okay everything should be pretty fire and forget if 00:22:24.663 --> 00:22:28.063 you've seen like the documentation on uh like how 00:22:28.063 --> 00:22:30.783 you set iopath options you know all that stuff can be set per 00:22:30.783 --> 00:22:33.923 directory application level uh target 00:22:33.923 --> 00:22:36.743 options you can set up whatever kind 00:22:36.743 --> 00:22:39.643 of tiering or caching setup or pin 00:22:39.643 --> 00:22:43.103 different directories to different devices there there 00:22:43.103 --> 00:22:46.083 really shouldn't be much to configure uh actually the 00:22:46.083 --> 00:22:50.983 one thing that i should mention is uh there is an outstanding bug on the bug 00:22:50.983 --> 00:22:56.003 tracker that looks like it might be a bug in z standard the title of the bug 00:22:56.003 --> 00:23:00.323 is uh data corruption after writing hundreds of gigabytes and it's only used 00:23:00.323 --> 00:23:03.263 with c standards so yeah maybe. 00:23:03.263 --> 00:23:04.323 Choose a different compression. 00:23:04.323 --> 00:23:08.223 Not the first c standard bug that we've tripped across right right and one of 00:23:08.223 --> 00:23:11.243 the hardening projects that's like next up on the to-do list will enable us 00:23:11.243 --> 00:23:16.683 to like pinpoint and flick up those guys like a test case or something and, 00:23:17.703 --> 00:23:22.803 hardness against uh compression algorithm library bugs or compression algorithm 00:23:22.803 --> 00:23:28.263 bugs and a couple other things in the data update path so yeah aside from that 00:23:28.263 --> 00:23:30.143 i mean you should be good to go you. 00:23:30.143 --> 00:23:32.183 Mentioned erasure coding though you mentioned something about erasure. 00:23:32.183 --> 00:23:38.003 Yeah that's coming uh hopefully like first half of next year there's not much 00:23:38.003 --> 00:23:43.363 left uh people have been running erasure coding and meant not much to say about 00:23:43.363 --> 00:23:45.823 it besides the fact that it's fast and seems to work well, 00:23:46.543 --> 00:23:50.763 The only thing that's missing there, which the reason you shouldn't use it yet, 00:23:50.923 --> 00:23:57.423 unless you really know the gotcha, is we're missing like resilver or like the 00:23:57.423 --> 00:23:59.523 erasure coding version of re-replicate. 00:23:59.803 --> 00:24:02.663 But drive dies and you've got degraded stripes. 00:24:02.923 --> 00:24:08.643 You can still read your data, but we don't have any kind of path to resilver those stripes. 00:24:09.303 --> 00:24:13.683 Does your recent reconcile work impact the implementation of that? 00:24:13.883 --> 00:24:16.983 It does, actually. uh that kind 00:24:16.983 --> 00:24:20.223 of gave us the hooks uh and 00:24:20.223 --> 00:24:24.183 the engine that's going to drive all that resilvering stuff and i reserve space 00:24:24.183 --> 00:24:28.503 and some metadata on this format stuff for exactly where erasure coding is going 00:24:28.503 --> 00:24:33.383 to plug into that when i got the rest of the design sketched out so it's it's 00:24:33.383 --> 00:24:37.243 looking like it's not going to be that much work but i got i got two more hardening 00:24:37.243 --> 00:24:39.163 projects to do before i can think about that stuff. 00:24:39.163 --> 00:24:44.243 So when you say hardening projects i think of security hardening But do you 00:24:44.243 --> 00:24:46.063 mean a different type of hardening? 00:24:46.683 --> 00:24:50.643 Yes. I mean, the file system, like this is file system land. 00:24:50.843 --> 00:24:54.183 The file system jobs is to never lose your data. 00:24:54.803 --> 00:24:58.463 I always tell people, I don't care what happens to your file system. 00:24:58.623 --> 00:25:03.483 I don't care if it, whose fault it is, if it's like user error or hardware error. 00:25:03.763 --> 00:25:07.683 As long as there's data on your file system, it's the file system's job to get 00:25:07.683 --> 00:25:09.683 back anything that can be gotten back. 00:25:10.743 --> 00:25:15.583 And repair it and get it working again fully read-write with no user intervention. 00:25:15.883 --> 00:25:18.743 That's the goal here. And we're, 00:25:19.826 --> 00:25:23.246 I mean, judging from the log reports that I'm seeing, we're basically there. 00:25:23.626 --> 00:25:28.446