WEBVTT 00:00:11.417 --> 00:00:16.197 Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. 00:00:16.377 --> 00:00:17.017 My name is Wes. 00:00:17.237 --> 00:00:18.077 And my name is Jeff. 00:00:18.697 --> 00:00:22.837 Hello, gentlemen. Coming up on the show this week, one piehole, 00:00:22.997 --> 00:00:26.857 two VPNs, and zero public exposure. I'm pretty proud of this one. 00:00:27.197 --> 00:00:30.997 Then, it's our pitch to ditch your GUI-only monitoring system, 00:00:31.137 --> 00:00:34.797 and why we rolled out Prometheus and Grafana. 00:00:35.037 --> 00:00:37.417 And then we're going to round the show out with some great boosts, 00:00:37.577 --> 00:00:40.157 some great picks, and a whole lot more. So before we go any further, 00:00:40.357 --> 00:00:43.137 time-appropriate greetings to our virtual lug. Hello, Mumble Room. 00:00:43.677 --> 00:00:45.817 Hello, Wes, and hello, Brent. 00:00:46.137 --> 00:00:50.217 Hi. Yeah, you can join us in the Mumble Room or at jblive.tv. 00:00:50.837 --> 00:00:55.617 Make it a Tuesday on a Sunday. We have the times at jupiterbroadcasting.com slash calendar. 00:00:55.817 --> 00:00:59.197 And a big good morning to our friends over at Defined Networking. 00:00:59.497 --> 00:01:03.797 Go check out Nebula VPN. They have a full managed product, 100 devices, 00:01:04.017 --> 00:01:04.937 no credit card required. 00:01:05.097 --> 00:01:08.037 Support the show, defined.net slash unplugged. 00:01:08.517 --> 00:01:11.777 It is a great service. And, you know, when I've thought about it a lot, 00:01:11.837 --> 00:01:15.697 I talk about how Slack used it and they launched it in 2017 to build out the 00:01:15.697 --> 00:01:18.697 security around the Slack global empire. And I talk about how Rivian uses it 00:01:18.697 --> 00:01:21.677 for real time analytics for the cars to do securely on the road. 00:01:22.805 --> 00:01:27.825 And those are all really big-scale projects. But recently, I've appreciated 00:01:27.825 --> 00:01:33.045 how great Nebula is on a 1, 2, 3 node network. 00:01:33.205 --> 00:01:39.705 And the fact that I can set up a on-demand mesh network that has name resolution 00:01:39.705 --> 00:01:41.305 and everything. We'll talk more about this. 00:01:42.045 --> 00:01:47.085 And there's no big tech login. There's no third-party hosted admin dashboard. 00:01:48.105 --> 00:01:52.825 Nothing like that. It's just two machines using cryptographic keys talking to each other. 00:01:52.945 --> 00:01:54.245 It's just a couple of text files, really. 00:01:54.845 --> 00:02:00.265 It's so powerful for small home lab stuff, and it's so scalable to massive enterprise stuff. 00:02:00.405 --> 00:02:03.525 And you can try it out with our fully managed product and support the show by 00:02:03.525 --> 00:02:06.485 going to define.net slash unplugged. 00:02:06.545 --> 00:02:10.525 You're going to like it a lot. And I'll tell you, I've been using it on extremely 00:02:10.525 --> 00:02:12.145 limited bandwidth connections. 00:02:12.745 --> 00:02:16.265 And it's so much better, and it's so much more resource sensitive. 00:02:16.445 --> 00:02:22.705 It's way lighter. It's way lighter. Check it out, define.net slash Nebula. 00:02:22.945 --> 00:02:25.825 And thank you to Define for sponsoring the Unplugged program. 00:02:28.525 --> 00:02:34.245 All right, you know we've got to mention it. Planet Nix and Scale23x are 39 days away. 00:02:35.125 --> 00:02:39.665 That means 33 days until Brent needs to be going down the road, at least. 00:02:40.105 --> 00:02:41.445 Let's just round that to 30. 00:02:41.725 --> 00:02:45.505 And six, I believe, or five, actually, more Linux Unplugs, maybe, 00:02:45.605 --> 00:02:47.445 until we need to be on the road ourselves. 00:02:47.605 --> 00:02:47.905 Wow. 00:02:47.905 --> 00:02:48.625 So it's coming up. 00:02:48.825 --> 00:02:50.545 I think we better get in the Nixie mood. 00:02:50.825 --> 00:02:54.465 Yeah, and I am really looking forward. Planet Nix has a theme this year. 00:02:54.605 --> 00:02:55.925 It's where builders come together. 00:02:56.225 --> 00:03:00.465 And our Nix coverage is supported again by Phlox, who's focused on making reproducible 00:03:00.465 --> 00:03:02.065 dev environments actually usable. 00:03:02.125 --> 00:03:04.445 And it's a fantastic tool. 00:03:04.605 --> 00:03:10.085 So check out Phlox and come see us at Scale and Planet Nix. You do need to register at Scale. 00:03:10.765 --> 00:03:14.525 And you can take 40% off that registration with our promo code UNPLG, 00:03:14.725 --> 00:03:20.105 U-N-P-L-G. and we'd love to see you there. One other item. 00:03:22.031 --> 00:03:26.391 The meetup page is now live. The details are not yet locked in. 00:03:26.531 --> 00:03:28.571 The date, time, location likely to change. 00:03:29.531 --> 00:03:32.131 But you can join the meetup and you'll be the first to get updates. 00:03:32.431 --> 00:03:37.571 And if you are intending to join us at the meetup, please consider signing up 00:03:37.571 --> 00:03:39.931 for the meetup. Please. We'd love to see you there. 00:03:40.251 --> 00:03:43.811 Last time we had about 80 more people than we expected. Great problem to have. 00:03:44.571 --> 00:03:48.071 It was very stressful on the restaurant staff. And they thankfully could open 00:03:48.071 --> 00:03:51.871 up. They had to open up another wing for us, which they were able to do. 00:03:52.491 --> 00:03:55.651 but this time we wanted to give them a great heads up. So if you're planning 00:03:55.651 --> 00:03:58.191 to make it and I want to bring a guest, there's room for that too. 00:03:58.531 --> 00:04:00.391 Just let us know. And we'll plan accordingly. 00:04:00.871 --> 00:04:04.511 Meetup.com slash Jupiter broadcasting link in the show notes to the direct meetup. 00:04:05.191 --> 00:04:07.411 We'd really appreciate it. If you could make it, if you're in the area, 00:04:07.991 --> 00:04:10.751 even if you can't go to the event, you're welcome to join us at the meetup. 00:04:11.431 --> 00:04:16.451 We did get one submission for that. I saw for a swag idea that we could hopefully 00:04:16.451 --> 00:04:20.531 have together for scale and Linux fest. It was a nice one. 00:04:20.931 --> 00:04:24.091 I'll show it to you boys after the show but I'd like to see a few more send 00:04:24.091 --> 00:04:27.431 them in to unplugged at jupiterbroadcasting.com or tag Wes in Matrix, 00:04:29.191 --> 00:04:33.451 and let us know we'll try to put one together pretty soon so we all have a uniform 00:04:33.451 --> 00:04:37.951 that we can identify each other with and have easy conversation hey I know you 00:04:37.951 --> 00:04:39.331 you listen to the show are. 00:04:39.331 --> 00:04:40.031 We getting hats. 00:04:41.411 --> 00:04:44.671 Ooh you know I'm a hat guy now you know I'm a hat guy now that's right, 00:04:47.918 --> 00:04:52.478 Well, what is, in a name, gentlemen, in short, convenience, right? 00:04:52.838 --> 00:04:56.478 When you set up your home lab or your enterprise network, whatever it is, 00:04:56.578 --> 00:05:00.058 it is eventually inevitable that you need good name resolution. 00:05:00.338 --> 00:05:04.378 I suspect for you there might be a spousal approval factor in the mix for that too. 00:05:04.598 --> 00:05:08.058 Yeah, and also just a memory factor. It gets hard to remember, 00:05:08.118 --> 00:05:10.418 especially the mesh network VPNs and the LAN IPs. 00:05:11.298 --> 00:05:14.318 And, of course, I have to go and make it hard, and I have multiple mesh networks 00:05:14.318 --> 00:05:16.578 now, multiple locations. 00:05:17.618 --> 00:05:22.118 some behind double carrier grade NAT a couple of them behind double carrier 00:05:22.118 --> 00:05:27.658 grade NAT so I had to go and make it hard on myself and I want sensible name 00:05:27.658 --> 00:05:32.418 resolution that works on the LAN and works across the various mesh networks, 00:05:33.538 --> 00:05:38.598 so I can just you know connect by machine name to all of them and then I need 00:05:38.598 --> 00:05:43.318 something also that does fast forwarding out to the internet and then can cache 00:05:43.318 --> 00:05:44.938 that so then future queries are faster. 00:05:44.938 --> 00:05:47.458 And then was it something you wanted to, like, I don't know, 00:05:47.518 --> 00:05:51.318 do you have some of these services that depend on other services in a way where, 00:05:51.318 --> 00:05:53.198 like, DNS is how they find each other? 00:05:53.638 --> 00:05:55.938 Yeah, and there's a lot of things I've set up are just by name now. 00:05:57.198 --> 00:06:03.478 So, you know, I had a basic pie hole going on my tail net, and I had a basic pie hole going on my LAN. 00:06:04.578 --> 00:06:06.078 But then we set up my wife's clinic. 00:06:06.258 --> 00:06:12.338 And was the tail net pie hole was running, like, as a container on a VPS or something? 00:06:12.338 --> 00:06:16.438 Yeah, and it just only had an interface on the Tailnet. So it was just acting 00:06:16.438 --> 00:06:17.738 as name resolution for the Tailnet. 00:06:17.858 --> 00:06:21.378 And then I kind of combined that with MagicDNS and sort of had the whole Tailnet thing solved. 00:06:21.638 --> 00:06:23.678 Then I had to go set up another network and all of that. 00:06:24.298 --> 00:06:28.458 And I also just kind of wanted to take another look at this and see if I couldn't do this better. 00:06:28.938 --> 00:06:33.998 But when I had set it up for the Tailnet only on the VPN, I took a shortcut. 00:06:34.698 --> 00:06:38.678 And instead of having to worry about exposing a pie hole to the internet... 00:06:40.126 --> 00:06:43.266 I just only bound it to the tail net interface. 00:06:44.306 --> 00:06:49.106 So I didn't have to worry about public IP and the internet, you know, 00:06:49.186 --> 00:06:52.146 banging on my pie hole server that's on a VPS because it couldn't talk to it. 00:06:52.506 --> 00:06:56.306 But if I wanted to make this pie hole usable across multiple mesh networks, 00:06:56.646 --> 00:07:00.626 it meant undoing that sort of convenience in security I had and coming up with 00:07:00.626 --> 00:07:03.386 a better security architecture to go across multiple networks. 00:07:04.886 --> 00:07:08.366 That's where it got a little kind of more complicated because 00:07:08.366 --> 00:07:12.006 i went from the easy way to the hard way and so 00:07:12.006 --> 00:07:14.726 there's multiple uh layers i kind of took 00:07:14.726 --> 00:07:18.126 to this and i kind of like to hear your guys's feedback on this so the first 00:07:18.126 --> 00:07:23.726 step i took is i wasn't sure if this is the right call but i essentially put 00:07:23.726 --> 00:07:29.886 the pie hole container on host networking so it could see all the interfaces 00:07:29.886 --> 00:07:32.406 And then in the configuration, 00:07:32.866 --> 00:07:38.126 I limited by application configuration to only bind to the tail net and the 00:07:38.126 --> 00:07:41.706 Nebula VPN interfaces and to not bind to the WAN interface. 00:07:42.026 --> 00:07:44.846 So at an application configuration layer, I did that. 00:07:45.126 --> 00:07:51.906 And then at another layer, I also set up ACLs with IP tables, 00:07:52.166 --> 00:07:55.746 just real basic IP tables that blocks all traffic on port 53. 00:07:55.746 --> 00:07:59.486 So like just in case, you know, for a moment, like when PyHole's starting up, 00:07:59.646 --> 00:08:03.166 if for a brief moment it bound to port 53 on the WAN interface, 00:08:03.566 --> 00:08:06.826 this would essentially prevent that from happening. Or if I make a config change 00:08:06.826 --> 00:08:10.566 mistake in the future, it prevents it from exposing it to the public internet. 00:08:11.706 --> 00:08:13.906 And so that's sort of the... 00:08:15.147 --> 00:08:18.507 multi-layer approach in a way and then all the communications just happening 00:08:18.507 --> 00:08:22.687 over the mesh vpns i'm not communicating with the buy hole at all no admin interface 00:08:22.687 --> 00:08:24.207 nothing over any public interface, 00:08:25.087 --> 00:08:29.707 how do you feel i did is that too risky would you brent would you be comfortable 00:08:29.707 --> 00:08:31.747 with that deployment i suppose i. 00:08:31.747 --> 00:08:37.807 Mean to me that feels probably more fine than anything that i've probably deployed 00:08:37.807 --> 00:08:43.167 in the past so it seems okay but really i'm not the pro or anything like that 00:08:43.167 --> 00:08:48.147 but But what I'm getting from you is that this is upping your peace of mind with this. 00:08:48.287 --> 00:08:51.387 But there's also some hesitation. So I'm curious to hear what Wes has to say. 00:08:51.887 --> 00:08:54.907 I do think it seems totally reasonable. You could, you know, 00:08:55.447 --> 00:08:58.587 get with the times and use NF tables already. No, I'm just kidding. 00:08:59.047 --> 00:09:02.487 I did actually, that actually considered, I was like, ah, this is what I know. But yeah. 00:09:02.687 --> 00:09:08.187 I think from background discussions, I picked up maybe you were using a sidecar before? 00:09:08.467 --> 00:09:09.467 A tailscale sidecar, yeah. 00:09:09.987 --> 00:09:14.087 So I think maybe in like another version, if you were going like fully, 00:09:14.107 --> 00:09:19.767 you know, application mesh native could be to just double down on the sidecar. 00:09:19.847 --> 00:09:20.667 Do a nebulous sidecar. 00:09:20.787 --> 00:09:24.247 Yeah. Like have it serve those two interfaces just in its own containerized 00:09:24.247 --> 00:09:25.127 networking environment. 00:09:25.127 --> 00:09:25.707 I like that. 00:09:25.847 --> 00:09:28.827 Where things might get more complicated depending on exactly what you want and 00:09:28.827 --> 00:09:32.387 convenience, et cetera. What matters to you is what you're doing with that host. 00:09:32.567 --> 00:09:36.367 And is that host then wanting to query the pie hole? And are you going to let 00:09:36.367 --> 00:09:40.027 that happen over local host? Or in this scenario, you'd either need to replumb 00:09:40.027 --> 00:09:43.687 stuff and forward it or rely on it only querying it over the mesh, 00:09:43.727 --> 00:09:45.627 which would probably be fine, but maybe you don't want to do that. 00:09:45.767 --> 00:09:51.527 The host is also on the tail net. So there's that too. But yeah, that is a tricky part. 00:09:52.547 --> 00:09:57.147 Technically, the host OS can't talk to it over the network, which hasn't been an issue yet. 00:09:58.383 --> 00:10:01.023 But so that's the basic, that's the core network setup, okay? 00:10:01.163 --> 00:10:05.103 And then what I decided to do was I turned off the tailscale MagicDNS stuff 00:10:05.103 --> 00:10:12.923 and didn't like the results because I do not have DNS entries for every machine on my tail net. 00:10:13.023 --> 00:10:16.643 And that's what MagicDNS was solving for me. So my sort of compromise solution 00:10:16.643 --> 00:10:24.143 was I re-enabled MagicDNS and then I added this pie hole as the upstream DNS server for MagicDNS. 00:10:24.503 --> 00:10:26.903 And I think that worked seemingly pretty well. 00:10:27.403 --> 00:10:29.303 and then I enabled the DNS. 00:10:29.503 --> 00:10:33.003 Right, so in that setup, TailScale will answer sort of right away for the TailScale 00:10:33.003 --> 00:10:36.063 host and then forward to your setup for anything it doesn't know about where 00:10:36.063 --> 00:10:37.403 you have to find your own manual entries. 00:10:37.483 --> 00:10:39.563 And that's where you'll find entries for the Nebula devices. 00:10:39.863 --> 00:10:40.123 Nice. 00:10:40.323 --> 00:10:45.743 And then you can configure the Nebula lighthouse to suggest a DNS server to the clients. 00:10:46.990 --> 00:10:51.150 And, um, that is, that's a really simple, it's like two lines of configuration 00:10:51.150 --> 00:10:53.150 on the lighthouse and you just give it the DNS server. 00:10:53.550 --> 00:10:57.450 And then, so that's also helping the Nebula clients discover who they're supposed 00:10:57.450 --> 00:10:58.610 to talk to for name resolution. 00:10:58.950 --> 00:11:02.250 And since I only have like three nodes on this little tiny, maybe four nodes 00:11:02.250 --> 00:11:07.190 now on this little tiny network, I'll talk more about super easy to just add the entries manually. 00:11:07.250 --> 00:11:10.330 And I don't, if this is going to be for a private clinic, so I don't think I'll be adding more hosts. 00:11:10.450 --> 00:11:13.670 One thing we should play with, which I haven't yet, but I'd like to get more 00:11:13.670 --> 00:11:19.090 into is doing either delegation or maybe using an API to trigger updates because 00:11:19.090 --> 00:11:21.970 Nebula lighthouses can serve DNS. 00:11:22.330 --> 00:11:22.410 Yeah. 00:11:22.890 --> 00:11:26.350 So you could also, depending on if you want to, maybe the static has advantages 00:11:26.350 --> 00:11:29.070 too, of course, but you could also maybe set it up so... 00:11:29.070 --> 00:11:29.190 Yeah. 00:11:29.590 --> 00:11:32.930 You know, the pie hole would just query Nebula and be able to answer for the 00:11:32.930 --> 00:11:34.670 Nebula host without you having to hard code it. 00:11:34.750 --> 00:11:39.630 The advantage was on the pie hole DNS server, now I also have a bunch of entries 00:11:39.630 --> 00:11:41.810 for the devices that are on my LANs. 00:11:41.950 --> 00:11:46.150 So it hosts here at the studio and hosts at the RV are also on this DNS server. 00:11:46.270 --> 00:11:50.730 So all the machines, if you're on the LAN, whichever LAN you're on, 00:11:50.870 --> 00:11:54.710 or whichever Mesh VPN network you're on, we all can resolve the same host names now. 00:11:55.070 --> 00:11:57.790 So that's kind of why I didn't go that direction. 00:11:58.490 --> 00:12:01.610 But I think that would be an easier setup if you just had a couple of machines. 00:12:01.870 --> 00:12:04.890 Well, I meant like integrating the two, like keeping the pie hole, 00:12:05.390 --> 00:12:08.090 just letting Nebula answer for the host it knows about. 00:12:08.310 --> 00:12:11.190 Oh, okay. And then would it upstream to the pie hole when it does? 00:12:11.290 --> 00:12:13.750 I see. Yeah, I like, okay. Oh my God, change. 00:12:13.750 --> 00:12:18.430 How do you feel, Chris, about the need for internet access here? 00:12:18.530 --> 00:12:23.990 Because occasionally you don't actually have access whenever a storm comes by 00:12:23.990 --> 00:12:25.430 or you're traveling, that kind of thing. 00:12:25.530 --> 00:12:29.230 So your name resolution internally on your local network would be affected. 00:12:29.230 --> 00:12:31.110 Is that a correct understanding? 00:12:31.630 --> 00:12:38.510 I did. Yeah. And so for that, I kept my pie hole on my LAN. And it forwards now to this guy. 00:12:38.710 --> 00:12:39.090 Nice. 00:12:39.370 --> 00:12:41.890 But for the most part, because that pie hole has been around so long, 00:12:41.950 --> 00:12:45.590 I have all these same DNS entries already. So, but I did keep it for that reason. 00:12:46.925 --> 00:12:52.905 And I'm very happy now. It adds complexity to have two mesh networks and, 00:12:52.905 --> 00:12:58.365 you know, multiple LANs, but it's seamless now to the end user now that I've done this. 00:12:58.565 --> 00:13:04.465 So I'm pretty happy, and the latency is pretty good even for LTE connections, really. 00:13:05.045 --> 00:13:07.805 Well, you know, it kind of makes sense, too. It's like, it'd be one thing if 00:13:07.805 --> 00:13:10.605 you didn't have the existing infrastructure and all that, but because you kind 00:13:10.605 --> 00:13:14.345 of have hosts that are positioned to fit into both of these networks or could 00:13:14.345 --> 00:13:16.225 bridge them, like it doesn't actually. 00:13:16.225 --> 00:13:18.425 You didn't have to stand up a bunch of new infrastructure. You kind of just 00:13:18.425 --> 00:13:22.245 had to reprovision some of it to better work with your new setup. 00:13:22.465 --> 00:13:25.225 I would like to actually ask, so if you want to boost in or send us a contact, 00:13:25.525 --> 00:13:29.225 if you were building this from scratch, so I already had a pie hole going. 00:13:29.525 --> 00:13:32.105 But if you out there, listener, were building this from scratch, 00:13:32.105 --> 00:13:34.565 what would you have used to do this name resolution? 00:13:34.885 --> 00:13:37.885 Because it did cross my mind. Like, maybe this is just a stupid DNS mask thing. 00:13:37.965 --> 00:13:39.265 I just set up a simple DNS mask. 00:13:39.885 --> 00:13:43.725 But then I like the idea of a little bit of ad blocking for the systems as well. 00:13:43.725 --> 00:13:45.525 That's nice. That's a nice feature that comes with it. 00:13:46.225 --> 00:13:49.165 And you can do dns mask configuration with. 00:13:49.165 --> 00:13:53.105 Pot all right because it uses like a forks version that was my conclusion yeah 00:13:53.105 --> 00:13:57.425 i was like yeah well i might i kind of get and i know how to use it yep and 00:13:57.425 --> 00:14:00.965 it's worked fine for me and it's survived multiple major upgrades now so it's 00:14:00.965 --> 00:14:05.025 past those tests as well so it's a good project but i would be curious like 00:14:05.025 --> 00:14:07.385 i think you probably would use tectidium or technically that would least. 00:14:07.385 --> 00:14:11.145 Be not or like because i know can do sort of like delegate zones where it will 00:14:11.145 --> 00:14:14.825 say like hey for anything in this sub you know maybe you have like dot nebula 00:14:14.825 --> 00:14:18.345 demands or whatever, go query this server for those and then return those. 00:14:18.485 --> 00:14:21.885 It also has some plugin capability, which I haven't really explored. Um, 00:14:23.215 --> 00:14:25.255 Or, you know, there's a lot of good options these days. 00:14:25.295 --> 00:14:27.935 Yeah, I saw some people that were solving this with AdGuard. 00:14:28.895 --> 00:14:31.095 Okay, yeah. Yeah, you could totally use Bind, of course. 00:14:31.495 --> 00:14:33.895 So I'd just be interested to know how people are solving this. 00:14:35.055 --> 00:14:37.975 I would also like to know if anybody has a way to solve this declaratively, 00:14:38.015 --> 00:14:40.175 you know, so that would also be a winner in my book. 00:14:40.635 --> 00:14:41.855 But while we were talking about 00:14:41.855 --> 00:14:45.235 Nebula, you've been working on something that's kind of slick, Wes Pano. 00:14:46.015 --> 00:14:50.055 Yeah, it was just an idea we had while we were toying around with setting up 00:14:50.055 --> 00:14:51.255 the clinic the other week. 00:14:51.255 --> 00:14:56.755 was, well, what if you just had like a low-key, you know, not crazy production 00:14:56.755 --> 00:15:00.555 scale, not being like a whole control plane for Nebula necessarily, 00:15:01.155 --> 00:15:05.175 but just something to make printing new host certs easier. 00:15:05.515 --> 00:15:09.595 Yeah, could you explain that a little bit? So if I'm not using the managed product, 00:15:09.915 --> 00:15:11.855 there's sort of some cert exchanges that have to happen. 00:15:12.375 --> 00:15:16.435 Yeah, right. So you have to, you're basically managing a CA. 00:15:16.735 --> 00:15:16.775 Right? 00:15:16.835 --> 00:15:20.875 So you have your own certificate authority, and then to get hosts onto the network, 00:15:21.535 --> 00:15:25.455 they generate their own private key but then you kind of have to sign the public 00:15:25.455 --> 00:15:28.815 part of that and that's how they get blessed with a host name and an IP address 00:15:28.815 --> 00:15:31.855 on the network and then that's how anything trusts them, 00:15:32.675 --> 00:15:35.635 when you try to communicate with something you need to be able to present that 00:15:35.635 --> 00:15:39.035 public side that is signed by the CA that they all mutually trust and. 00:15:39.035 --> 00:15:43.315 The beauty is the simplicity is it's really coming down to files you're moving 00:15:43.315 --> 00:15:47.455 around that have keys in them and that is the totality of the infrastructure 00:15:47.455 --> 00:15:48.955 actually required to get this working, 00:15:49.895 --> 00:15:54.395 And if you sit with the amazingness of that for a moment, it really is very impressive. 00:15:54.555 --> 00:15:56.495 These machines are discovering themselves. 00:15:56.675 --> 00:15:58.635 You need a lighthouse, but they're discovering or use a public one. 00:15:59.035 --> 00:16:04.455 And they're communicating and creating a mesh VPN just by exchanging these key files. 00:16:04.715 --> 00:16:08.275 Yep. And, you know, just simple concepts of groups and you have stuff signed 00:16:08.275 --> 00:16:10.095 by the right thing and it kind of all just works. But... 00:16:11.582 --> 00:16:14.342 For simple static networks, that works pretty well. But, you know, 00:16:14.562 --> 00:16:19.202 I'd been playing around with my sidecar mesh setup on NixOS. 00:16:19.582 --> 00:16:21.862 And especially for like the demos I was doing and testing it out, 00:16:21.982 --> 00:16:22.682 it was pretty convenient. 00:16:22.862 --> 00:16:26.922 You know, products like Tailscale or Netbird, they have this UX you get with 00:16:26.922 --> 00:16:30.482 basically all you need is one secret, right? Like an API key, 00:16:30.622 --> 00:16:31.962 and you can put that in somewhere. 00:16:32.162 --> 00:16:36.642 And then when the client launches, it goes and uses that to an API and then can onboard itself. 00:16:36.782 --> 00:16:41.202 And I was just like, well, I wonder if we could get that same workflow with Nebula. 00:16:41.822 --> 00:16:47.902 So NACME, or ACME for Nebula, is my little attempt at that. It's super early 00:16:47.902 --> 00:16:49.482 days. I need to do a bunch more testing. 00:16:49.942 --> 00:16:52.562 Eventually, it'd be great to do renewals too. But right now, 00:16:52.702 --> 00:16:56.102 it's at the initial testing stage of just being able to, you run a little server, 00:16:56.302 --> 00:17:01.462 you configure an API key that's bound to certain groups, and then you have a 00:17:01.462 --> 00:17:06.282 little client that can run and go get a new host onboarded. 00:17:06.602 --> 00:17:09.242 And so if you configure that, I also want to set up a bunch of this stuff, 00:17:09.242 --> 00:17:13.442 especially with nick's side but configure it to run before nebula uh you could 00:17:13.442 --> 00:17:17.482 have it go item potently check to see if it needs to configure the the host 00:17:17.482 --> 00:17:20.802 for the first time set up the keys and everything and then have nebula start 00:17:20.802 --> 00:17:23.462 and be ready to go or at least that's the idea. 00:17:24.102 --> 00:17:29.442 Yes this is really neat so it's automated certificate minting and it gives you 00:17:29.442 --> 00:17:33.442 essentially like you said it's a like an api key type type exchange. 00:17:33.442 --> 00:17:36.502 The goal too would be like it's sort of best effort right it's meant for like 00:17:36.502 --> 00:17:39.502 home lab or you know stuff where you maybe you're not going to go the full like 00:17:39.502 --> 00:17:41.382 crazy it automation it's. 00:17:41.382 --> 00:17:43.802 Great for like a small business network like we were just setting. 00:17:43.802 --> 00:17:46.522 And the whole thing with nebula right is like there are some 00:17:46.522 --> 00:17:49.482 trade-offs you have less with those certs and the way it's kind of like more 00:17:49.482 --> 00:17:53.562 like a jwt kind of style of trust you know you don't necessarily have this one 00:17:53.562 --> 00:17:57.222 database that determines all of the truth right in a sort of less eventually 00:17:57.222 --> 00:18:01.602 consistent way but the upside is nebula will just keep working right as long 00:18:01.602 --> 00:18:06.422 as the certs aren't expired like there's no the control plane nothing happens nothing goes down what. 00:18:06.422 --> 00:18:09.122 Freaked me out recently was the idea that maybe, 00:18:10.316 --> 00:18:16.996 My Google account could be suspended because, so PayPal decided to flag my account 00:18:16.996 --> 00:18:18.436 for like re-verification. 00:18:19.076 --> 00:18:21.916 And it's a very complicated process. It's not just like a regular, 00:18:22.096 --> 00:18:24.616 it's like a very in-depth, multiple types of documentation. 00:18:24.616 --> 00:18:26.216 What did you do? Nothing. 00:18:26.356 --> 00:18:26.856 I don't know, man. 00:18:26.876 --> 00:18:27.416 That's usually what it is. 00:18:27.496 --> 00:18:32.556 I don't know. But it occurred to me that if our Google Workspace account payment 00:18:32.556 --> 00:18:36.556 got bounced because of PayPal, then I might not be able to authenticate to my tail net anymore. 00:18:37.516 --> 00:18:39.136 And that freaked me out a little bit. 00:18:39.296 --> 00:18:39.516 Yeah. 00:18:39.516 --> 00:18:42.876 And that's where I was like, oh, the simplicity of these keys and the fact that 00:18:42.876 --> 00:18:47.376 they'll work for as long as I issue these keys for is very reassuring. 00:18:47.616 --> 00:18:50.376 And so my thought was this was like worst case, you know, like even if this 00:18:50.376 --> 00:18:54.316 is down, you can still manually add things like this is just a convenience functionality 00:18:54.316 --> 00:18:55.656 to make it easier to onboard hosts. 00:18:55.656 --> 00:19:00.016 So this is not setting up routing. This is not setting up the networking layer 00:19:00.016 --> 00:19:03.636 stuff. This is just keys to get you going to then build that stuff. 00:19:03.796 --> 00:19:04.016 Exactly. 00:19:04.596 --> 00:19:07.396 That's really cool. NACME. I like the name, too. It's very clever. 00:19:07.496 --> 00:19:08.336 I think that could take off. 00:19:08.996 --> 00:19:14.896 so uh we'll put a link in the show notes it's on wes's github n-a-c-m-e mit 00:19:14.896 --> 00:19:20.876 licensed indeed indeed uh and uh version 0.10 was released just recently. 00:19:20.876 --> 00:19:25.596 Yeah we'll see i should cut a new version it's move it's you know it's moving 00:19:25.596 --> 00:19:27.356 fast and uh needs more tests. 00:19:30.795 --> 00:19:34.635 I want to say thank you to our members and our boosters. Next week, 00:19:34.655 --> 00:19:36.355 I'm calling it a birthday episode, boys. 00:19:36.575 --> 00:19:38.915 I don't know what we're going to do, but check this out. 00:19:39.175 --> 00:19:44.255 So Brian and I started podcasting in January, right around my actual birthday, 00:19:44.255 --> 00:19:48.595 almost exactly 20 years ago to the day for next week's episode. 00:19:49.455 --> 00:19:52.775 20 years of podcasting on my birthday next week. 00:19:52.915 --> 00:19:56.275 So if that's not a long-term commitment to the space, I don't know what is. 00:19:56.455 --> 00:19:57.855 So send a birthday boost. 00:19:58.055 --> 00:20:01.735 We'd love that. or become a member and use the promo code bootleg. 00:20:01.855 --> 00:20:04.095 We have a couple of, well, we have a handful of redemptions left. 00:20:05.415 --> 00:20:09.735 And, you know, you become a member at the party or a core contributor and support 00:20:09.735 --> 00:20:10.835 the show at a great discount. 00:20:11.315 --> 00:20:14.455 And if you'd like to get your company or product in front of the world's largest 00:20:14.455 --> 00:20:16.595 and best Linux audience, show me an email. 00:20:16.975 --> 00:20:19.935 Chris at jupiterbroadcasting.com. This space could be yours. 00:20:20.275 --> 00:20:24.575 And thank you to everybody who supports the show. We greatly appreciate it. 00:20:27.683 --> 00:20:30.883 Now, Chris, for the last two episodes, you've been talking about deploying, 00:20:30.883 --> 00:20:35.723 you know, a bunch of new machines at a clinic, making sure you have the responsibility 00:20:35.723 --> 00:20:39.243 for keeping your wife's business happy from a tech perspective. 00:20:39.723 --> 00:20:44.863 And now you're putting up, you know, some infrastructure that you also need 00:20:44.863 --> 00:20:50.143 to be working at all times. I would imagine now is the time to make sure all 00:20:50.143 --> 00:20:52.943 that stuff yells at you whenever it's not in good health. 00:20:52.943 --> 00:20:54.003 Right he. 00:20:54.003 --> 00:20:57.883 Wanted to uh just leave a business card with my phone number on it but i didn't 00:20:57.883 --> 00:20:58.803 think that was a great idea. 00:20:58.803 --> 00:21:02.723 And part of me is like well we should do this why it's still fresh in the mind 00:21:02.723 --> 00:21:08.563 because this stuff fades oh yeah and i thought well if we're going to do this for hadia's clinic, 00:21:09.383 --> 00:21:12.643 um maybe i should do this for my own infrastructure and then i thought wouldn't 00:21:12.643 --> 00:21:17.003 it be great if we could build something that if we ever did this you know on 00:21:17.003 --> 00:21:19.543 occasion for audience members or whoever, 00:21:20.003 --> 00:21:23.363 wouldn't it be nice if we could also offer to monitor their stuff and i could 00:21:23.363 --> 00:21:26.703 build something that was pretty flexible like this and i'm sure you boys are 00:21:26.703 --> 00:21:28.343 familiar with uptime kuma, 00:21:29.163 --> 00:21:34.123 we actually use it at jb alex set it up for us a while ago and we like it it's 00:21:34.123 --> 00:21:38.983 pretty simple and it alerts us when something goes offline like one of our websites 00:21:38.983 --> 00:21:43.843 via telegram bot and it creates a nice dashboard super easy to self-host and 00:21:43.843 --> 00:21:48.563 they have a demo i'll put a link in the show notes And it does monitoring for HTTP, TCP. 00:21:49.023 --> 00:21:54.843 It can search website keywords, check WebSockets, do ping, check DNS records, stuff like that. 00:21:55.023 --> 00:21:56.663 Very easy, straightforward to get going. 00:21:56.883 --> 00:22:00.863 Yes. So obviously that was the first thing I decided because this is what I 00:22:00.863 --> 00:22:02.003 have the most experience with. 00:22:02.423 --> 00:22:06.783 And I thought this will be the way to go. Can you guess what the problem was? 00:22:07.003 --> 00:22:11.303 Well, I think I know because it's been a longstanding issue we've had with the project. 00:22:11.423 --> 00:22:12.403 In fact. What's that? 00:22:12.603 --> 00:22:14.943 A lack of declarative configuration. 00:22:15.463 --> 00:22:19.323 Yeah, man, it's just really gooey. Well, I had to set up something like, 00:22:19.323 --> 00:22:25.323 you know, 45 hosts and services to monitor and I was sitting there creating 00:22:25.323 --> 00:22:30.423 all the entries and I'm like, this is going to take me to two days to do. 00:22:31.530 --> 00:22:34.890 And also because I wanted them to be actionable and all this stuff. So it's like, oh, my God. 00:22:35.030 --> 00:22:40.230 And then I also wanted tiered escalating alerts. So first ping me via notify. 00:22:40.710 --> 00:22:43.610 So because I have a lot of stuff coming in via NTFY these days. 00:22:44.190 --> 00:22:47.210 And so that's sort of a feed of just checking on my systems. 00:22:47.230 --> 00:22:48.750 And if I'm available, I check it. 00:22:49.150 --> 00:22:54.390 But if I'm not presently thinking about my infrastructure, my systems, I don't check it. 00:22:54.590 --> 00:22:57.850 So I needed something that would break through to Telegram and kind of, you know, kick it up. 00:22:57.850 --> 00:22:58.110 Yeah. 00:22:58.250 --> 00:23:01.730 And so I wanted and I wanted it based on different thresholds and trends. 00:23:01.950 --> 00:23:06.190 And so when I started getting into I need to add, let's just say 45 ish host 00:23:06.190 --> 00:23:11.530 combination services, maybe more and more complex alerting with a little bit more nuance. 00:23:13.395 --> 00:23:17.135 I really started to hit two different walls. Like my GUI exhaustion kicked in. 00:23:17.315 --> 00:23:19.435 Like if I was adding like five systems, I would just done it. 00:23:19.895 --> 00:23:23.675 And then trying to get complex logic around alerts started to get frustrating. 00:23:24.395 --> 00:23:28.055 Yeah, you combine those two. That could be pretty annoying, especially if you 00:23:28.055 --> 00:23:31.695 have to like go configure it and then go run the test and then go see if it 00:23:31.695 --> 00:23:34.895 did the thing you want and then go repeat that cycle a whole bunch. 00:23:35.555 --> 00:23:40.515 Yeah, man. So I decided to break the seal on something that I have never, 00:23:40.515 --> 00:23:45.035 I've never bothered learning. I've never wanted to embrace because of the overhead. 00:23:45.035 --> 00:23:47.575 It's been a long time coming on the show. I'm excited for this. 00:23:47.595 --> 00:23:53.035 It has. Ladies and gentlemen, I have finally deployed my first Prometheus instance. 00:23:53.395 --> 00:23:53.815 Woohoo! 00:23:54.115 --> 00:23:58.155 And of course, once you have a Prometheus instance, you want pretty dashboards 00:23:58.155 --> 00:23:59.475 and you want all the details. 00:23:59.675 --> 00:24:04.255 So along with that, I have also finally deployed my first Grafana instance. 00:24:04.795 --> 00:24:07.355 Yeah, something tells me you really deployed the Grafana and then you just got 00:24:07.355 --> 00:24:08.815 the Prometheus, so you had something to fill it. 00:24:09.375 --> 00:24:12.375 It feels like that deserves a round of applause, really, there, 00:24:12.475 --> 00:24:13.955 Chris. Give yourself, yeah. 00:24:14.175 --> 00:24:14.475 Thank you. 00:24:14.735 --> 00:24:15.375 Everybody, yeah. 00:24:15.755 --> 00:24:19.595 I mean, uptime Kuma is good, but I needed a little bit more than that. 00:24:19.635 --> 00:24:23.415 And I also have kind of a complex situation that I thought Prometheus was a 00:24:23.415 --> 00:24:24.175 little better at solving. 00:24:24.575 --> 00:24:26.795 I have set up a federated configuration. 00:24:27.515 --> 00:24:32.295 And I had a problem where I have my Odroid on my home lab that doesn't have 00:24:32.295 --> 00:24:35.695 a lot of available resources. as it has some, but now it's doing frigging a bunch of other stuff. 00:24:36.515 --> 00:24:39.855 And I'm on LTE on two different ends of the connection, not all of them, 00:24:39.955 --> 00:24:41.795 but two different ends are on LTE networks. 00:24:41.935 --> 00:24:45.315 So they're slow and I can't just be blasting a bunch of data. 00:24:45.435 --> 00:24:49.275 My original idea was I'll set up a central VPS connected to all the mesh networks, 00:24:49.275 --> 00:24:52.715 and then it will monitor everything. And I just go to the VPS dashboard. 00:24:52.915 --> 00:24:56.795 But then I actually started running the math on that. And I realized the overhead 00:24:56.795 --> 00:25:00.615 would be somewhere between 40 megabytes to 100 megabytes a day, 00:25:01.395 --> 00:25:05.095 best case, plus the overhead and latency it adds and slows down the whole LTE 00:25:05.095 --> 00:25:06.375 connection while it's doing that stuff. 00:25:06.555 --> 00:25:09.695 And that's only going to grow as you add more stuff that you monitor, surely. 00:25:10.295 --> 00:25:17.315 Exactly. So the solution was a federated setup where I had local monitoring on my home lab system, 00:25:18.372 --> 00:25:24.392 monitoring on a VPS, and then light remote monitoring on my wife's clinic network. 00:25:24.792 --> 00:25:29.392 And I set up a Prometheus system with a black box exporter that does some additional 00:25:29.392 --> 00:25:33.852 like HTTP, TCP, ping checks, and can do some API authentication for me. 00:25:33.972 --> 00:25:36.672 And that all feeds into Grafana to give me dashboards. 00:25:36.852 --> 00:25:40.592 And then that all talks to alert manager. So let me zoom out. 00:25:41.812 --> 00:25:46.712 Prometheus is running on two different systems. plus a little Prometheus client 00:25:46.712 --> 00:25:52.092 is running on my wife's clinic and I have a Prometheus integration now running on Home Assistant. 00:25:52.332 --> 00:25:52.712 Oh yeah. 00:25:52.872 --> 00:25:53.372 So this is collecting. 00:25:53.372 --> 00:25:54.012 You knew that was coming. 00:25:54.112 --> 00:25:56.912 Yeah, buddy. And that's really useful. I could talk more about that. 00:25:57.012 --> 00:26:01.272 But the Prometheus agent is essentially collecting all of the metrics, 00:26:01.452 --> 00:26:06.332 the CPU, the disk usage, and it allows me to trend and alert on these over time. 00:26:06.772 --> 00:26:11.532 And with Home Assistant, I'll just mention this quickly, that integration is 00:26:11.532 --> 00:26:16.712 pretty awesome because you can export a lot of different things from automation 00:26:16.712 --> 00:26:18.772 details, log, all these types of things. 00:26:19.012 --> 00:26:22.792 I mean, it basically has a whole bunch of stats in its own stats engine to begin with, right? 00:26:22.932 --> 00:26:23.352 Oh, yeah. 00:26:23.472 --> 00:26:25.232 You can kind of just dump those out to Prometheus. 00:26:25.552 --> 00:26:29.412 It's a sensor machine, yeah. And so the way I use that now is I am using, 00:26:29.412 --> 00:26:31.512 via Prometheus data export into 00:26:31.512 --> 00:26:35.872 Grafana, I have dashboards on how long my different climate entities run. 00:26:36.332 --> 00:26:39.232 So since it's winter, we have some electric heat out in different areas, 00:26:39.232 --> 00:26:41.772 and I want to make sure that they're not running excessive because that would 00:26:41.772 --> 00:26:42.972 tell me the heater isn't keeping up. 00:26:43.132 --> 00:26:45.772 I want to know how long we're running electric heat. I want to compare that 00:26:45.772 --> 00:26:48.832 to our intake from solar. And I want to have all that on one dashboard. 00:26:49.072 --> 00:26:54.672 And now I have just not only a beautiful display, but good historical data that I can work with. 00:26:54.872 --> 00:26:58.912 And all of this is pulled in on my local Prometheus incident that's running on my home lab. 00:26:59.712 --> 00:27:03.272 Then I have Prometheus running on the VPS. That's doing some remote checks to 00:27:03.272 --> 00:27:05.412 make sure that the remote systems are up. 00:27:05.552 --> 00:27:09.692 And it has some logic to understand that if the mesh network is down, don't freak out. 00:27:10.052 --> 00:27:12.932 don't alert about every single host. You know, I wanted some of this tiered 00:27:12.932 --> 00:27:15.772 logic in there so it knows if one network connection is down, 00:27:15.832 --> 00:27:17.392 then everything that's on that network is down. 00:27:18.559 --> 00:27:20.519 which is going to hopefully save me a lot of notifications. 00:27:20.779 --> 00:27:23.919 Yeah, and you've saved yourself that first night, right? Where the network goes 00:27:23.919 --> 00:27:25.339 down and suddenly everything's on fire. 00:27:25.519 --> 00:27:29.039 And then everything it's observing about the VPS itself, the mesh networks, 00:27:29.039 --> 00:27:33.759 and the wife's clinic is getting federated back to my Grafana instance. 00:27:34.039 --> 00:27:39.699 So I have just one dashboard to view everything. And that's running on my local instance. Oh, man. 00:27:40.079 --> 00:27:45.519 And the difference is, according to my calculations, boys, the difference is 00:27:45.519 --> 00:27:49.779 about two megabytes a day of data usage. nicely done yeah that's. 00:27:49.779 --> 00:27:50.659 At least a power of 10. 00:27:50.659 --> 00:27:55.819 It's it was a nice little savings and it's less it's less running on my home lab or on the vps i do. 00:27:55.819 --> 00:27:59.559 Like that you kind of went from zero to not uh you know. 00:27:59.559 --> 00:28:03.939 Yeah unsophisticated setup i think the real deal breaker was, 00:28:05.359 --> 00:28:09.059 everything could be done declaratively and i could do this sort of hybrid federated 00:28:09.059 --> 00:28:12.859 setup and those flexible yeah coming together those two things like it's like 00:28:12.859 --> 00:28:16.679 to create all the dashboards I didn't create a single dashboard in the GUI. 00:28:16.739 --> 00:28:20.559 There's so many community examples that you can modify and get started with. 00:28:20.659 --> 00:28:24.099 And then you drop them in a folder and that becomes a dashboard and Bob's your 00:28:24.099 --> 00:28:26.059 uncle. You got all the things and bells and whistles you'd want. 00:28:26.219 --> 00:28:28.519 And then I have alert manager, last piece. 00:28:28.779 --> 00:28:34.099 I have alert manager running on the VPS and everything forwards to that. 00:28:34.239 --> 00:28:37.999 So I only have to have one alert instance and it can communicate with notify. 00:28:37.999 --> 00:28:41.279 It can communicate with telegram because I set up a little bot thing and all of that. 00:28:41.799 --> 00:28:45.739 and it handles all that stuff. And it will also let me know if any of the remote 00:28:45.739 --> 00:28:48.099 hosts are down or if any of the mesh networks go down. 00:28:49.038 --> 00:28:54.818 And the results have been, there's about 31 services that I'm getting real-time 00:28:54.818 --> 00:28:58.698 visibility into their performance and their metrics, all in beautiful Grafana dashboards. 00:28:59.198 --> 00:29:02.958 And I also have real-time alerts anytime any service goes down. 00:29:03.318 --> 00:29:06.998 And I'm now very carefully monitoring storage, which is very tight these days. 00:29:07.198 --> 00:29:10.138 And so I have different thresholds for storage alerts. And I have different 00:29:10.138 --> 00:29:13.878 thresholds for like if home assistance at 80% CPU for X amount of time. 00:29:13.998 --> 00:29:17.778 And if it gets to 85 or 90 for X amount of time, we do different styles of escalation. 00:29:17.778 --> 00:29:22.238 and then there's follow-up for when things recover i get a recovery alert so 00:29:22.238 --> 00:29:23.438 that's all working really beautiful. 00:29:23.438 --> 00:29:24.738 Yeah i'm curious how you tested it. 00:29:24.738 --> 00:29:35.358 The entire well by shutting things down i did excellent all right docker compose down you know. 00:29:35.358 --> 00:29:39.578 The next version of this is you give your agent access and you ask it to stop a random service. 00:29:39.578 --> 00:29:43.618 Right just chaos monkey oh my god that'd be fun basically just give west your credentials, 00:29:45.218 --> 00:29:48.178 yeah that's to most of it um but so i measured 00:29:48.178 --> 00:29:51.238 the whole stack across all the machines 400 megabytes of 00:29:51.238 --> 00:29:54.838 ram totally fine not even 00:29:54.838 --> 00:29:57.538 making really an impact at all in my home lab system and i was worried because i'd heard some 00:29:57.538 --> 00:30:00.538 stories about grafana but i really really 00:30:00.538 --> 00:30:03.398 like the home assistant integration with prometheus it's 00:30:03.398 --> 00:30:06.158 something else i've put off for a while if you have been tempted to 00:30:06.158 --> 00:30:09.418 try this the insights are fantastic and then 00:30:09.418 --> 00:30:12.478 the other thing that's really fun is frigate which 00:30:12.478 --> 00:30:15.558 i also set up recently also has an 00:30:15.558 --> 00:30:18.398 api where i can export all the 00:30:18.398 --> 00:30:21.318 information to prometheus and grafana 00:30:21.318 --> 00:30:24.518 so i have details on how my coral is doing 00:30:24.518 --> 00:30:28.678 inference wise and my different cameras and their connectivity and i can and 00:30:28.678 --> 00:30:33.098 their detection frames per second the camera's overall health when the automations 00:30:33.098 --> 00:30:36.738 execute and how frequently for like arming and disarming the recording if they 00:30:36.738 --> 00:30:41.098 are available at all is all coming into this dashboard so i have essentially 00:30:41.098 --> 00:30:42.878 a camera health dashboard now. 00:30:43.919 --> 00:30:46.219 It's so great because you have dashboards. 00:30:46.219 --> 00:30:46.959 For your dashboards. 00:30:47.119 --> 00:30:50.959 I do. And a lot of the RAR projects, it's great. Oh, I'm like, 00:30:51.079 --> 00:30:52.839 I'm inception dashboards over here. 00:30:52.999 --> 00:30:57.079 A lot of the RAR projects out there that we don't talk about also have APIs 00:30:57.079 --> 00:31:00.539 and health API endpoints that also plug into this. 00:31:00.619 --> 00:31:03.439 So you can get all that kind of information in there. 00:31:04.219 --> 00:31:09.299 And it, it really struck me like how, now I get it, why people go through setting all this up. 00:31:09.379 --> 00:31:13.079 And yes, it's a lot of YAML and all of that, but there's so many great examples out there. 00:31:13.739 --> 00:31:21.519 And I now have an enterprise grade monitoring stack that I just never even thought I would get into. 00:31:23.080 --> 00:31:27.800 Zero dollars spent, you know, one day I've set up really a lot of documentation 00:31:27.800 --> 00:31:30.620 and maybe five to eight hours of fiddling to get it all working. 00:31:31.440 --> 00:31:36.160 And it really makes me feel a lot better about my self-hosted infrastructure. 00:31:36.500 --> 00:31:39.980 I have built quite a little empire now of things I depend on and my family depends 00:31:39.980 --> 00:31:41.540 on and my wife and whatnot. 00:31:41.960 --> 00:31:45.420 And I probably haven't been monitoring it as seriously as I should. 00:31:45.800 --> 00:31:49.340 And I just thought that, eh, it's fine. But honestly, it does feel a lot better 00:31:49.340 --> 00:31:53.000 because I'm getting insights before things go wrong. 00:31:53.080 --> 00:31:55.620 Now I'm getting ideas of trends. 00:31:55.880 --> 00:31:58.080 So stuff I know, Oh, this is actually something I need to address. 00:31:58.460 --> 00:32:02.020 And I know something's a problem before the wife has to tell me it's a problem. 00:32:02.260 --> 00:32:05.140 So I know if jellyfin isn't working for some reason or et cetera, 00:32:05.420 --> 00:32:08.200 or I know if her system isn't backing up before she has to tell me. 00:32:09.060 --> 00:32:13.920 So if you have a self-hosted setup, definitely uptime Kuma is a pretty good 00:32:13.920 --> 00:32:18.740 starting point. But if you have the ambition and the time Prometheus and Grafana 00:32:18.740 --> 00:32:22.700 get a double recommendation from me, I waited way too long. 00:32:23.080 --> 00:32:29.160 There is a real learning curve, but the visibility you get out of it, it's just so useful. 00:32:29.480 --> 00:32:33.480 Plus, like I've revisualized my entire infrastructure once again. 00:32:33.660 --> 00:32:37.480 All the stuff I've built, like I know how many services and how many hosts and 00:32:37.480 --> 00:32:39.100 I know how they're doing. 00:32:39.320 --> 00:32:42.480 I have a much more concrete picture once again of everything I've done over 00:32:42.480 --> 00:32:43.420 the last five, six years. 00:32:43.620 --> 00:32:47.500 Well, I think as you're discovering, right, a lot of things publish Prometheus metrics. 00:32:47.860 --> 00:32:48.000 Yeah. 00:32:48.000 --> 00:32:54.020 So actually, as Tiny points out in our chat room, Uptime Kuma itself can publish to Prometheus. 00:32:54.060 --> 00:32:55.580 Yes, I thought about that for a second. 00:32:55.760 --> 00:32:59.100 And Nebula does as well. I don't know if you've integrated that yet. 00:32:59.780 --> 00:33:04.440 Hmm. Hmm. Hmm. Should look to see what Tailscale can do too, 00:33:04.480 --> 00:33:05.340 but that would be really useful. 00:33:05.840 --> 00:33:08.720 Yeah, I'm pretty happy with this. I did have to kind of, you know, 00:33:08.780 --> 00:33:09.680 be tight on the retention. 00:33:09.900 --> 00:33:13.980 So I don't have, I think I have like 30 or 60 days. I couldn't go crazy because 00:33:13.980 --> 00:33:14.820 of storage constraints. 00:33:15.720 --> 00:33:17.660 And I did, I will admit, Wes... 00:33:20.447 --> 00:33:26.007 I feel a little guilty, but I did a lot of it with Docker. And the reason was, 00:33:26.107 --> 00:33:27.427 and it's always what gets me, 00:33:30.047 --> 00:33:36.387 is a lot of the community add-ons and plugins all assume you're using the Docker instance. 00:33:36.827 --> 00:33:39.187 And it's one of those things where it's like, well, yeah, I could come up with 00:33:39.187 --> 00:33:41.787 a way to declaratively do that with Nix every single time. 00:33:42.887 --> 00:33:48.687 Or I just start the container and it works, right? And so all of the OS-level 00:33:48.687 --> 00:33:51.447 stuff to make it work is all declarative, obviously. 00:33:52.107 --> 00:33:56.687 But then, like, Prometheus, Grafana, and Alert Manager, they're all in, 00:33:56.707 --> 00:33:57.927 like, one big Docker compose. 00:33:58.287 --> 00:33:58.647 Ah, yeah. 00:33:59.787 --> 00:34:00.947 I felt bad about that. 00:34:01.047 --> 00:34:04.487 No, I mean, as long as you're dabbling, I think, a little bit and seeing that 00:34:04.487 --> 00:34:06.927 Nix can play very nicely with these things. 00:34:07.107 --> 00:34:09.427 And it was a no-brainer for the Nginx. Like, some of the stuff, 00:34:09.547 --> 00:34:13.547 I had to add some more stuff behind a reverse proxy and get SSL certs for it 00:34:13.547 --> 00:34:15.247 that I didn't have before just so I could keep consistent. 00:34:15.827 --> 00:34:18.127 So I wasn't doing IP for some stuff and name for other stuff. 00:34:18.247 --> 00:34:21.327 I wanted names for everything. So I just got all the DNS set up anyways. 00:34:22.167 --> 00:34:26.567 And that was really, really nice to just configure all of the Nginx stuff via Nix. 00:34:26.787 --> 00:34:29.467 And, I mean, it should be easy, right? Like any of your Nix hosts, 00:34:29.467 --> 00:34:33.067 you can just add in a bit to have it run a Prometheus exporter for the node 00:34:33.067 --> 00:34:35.127 and send the node metrics too. 00:34:35.707 --> 00:34:37.787 And I know between the three of us, 00:34:38.826 --> 00:34:43.466 I'm also the Git Luddite, but the other thing I liked and appreciated about 00:34:43.466 --> 00:34:48.506 having declarative config for my monitoring setup is that I could use Git to manage that. 00:34:48.826 --> 00:34:52.626 And if I F it up in the future, you know, I've got some recoverability there. 00:34:52.726 --> 00:34:56.746 So the more I can define via text, I feel like the safer it is to experiment 00:34:56.746 --> 00:34:59.146 and the easier it is to roll back. 00:34:59.986 --> 00:35:02.986 And so that's, I just, that's something I just, food of thought. 00:35:03.126 --> 00:35:05.586 One of the lessons I took away from the setup is that gave me a little bit of 00:35:05.586 --> 00:35:08.066 comfort level to experiment with something that I didn't fully understand yet. 00:35:08.066 --> 00:35:09.066 that has a big learning curve. 00:35:09.446 --> 00:35:12.566 And now that you have it captured, you can watch as it evolves too, 00:35:12.686 --> 00:35:15.286 right? So you have rollbacks, but you can also then go, well, 00:35:15.346 --> 00:35:17.026 what did I tweak? Was that what broke it? 00:35:17.286 --> 00:35:21.266 And I did catch a couple of things already, I will say too. I had some choral 00:35:21.266 --> 00:35:24.946 performance degradation issues that I tracked down to Wi-Fi actually, 00:35:25.146 --> 00:35:28.806 but it started a whole process of like breaking down where the problem was at. 00:35:28.966 --> 00:35:33.026 What is the maintenance going to be like for this? Like how likely are you if 00:35:33.026 --> 00:35:38.266 you're just booting up a new container for some kind of new system you're playing 00:35:38.266 --> 00:35:43.046 with that may last a long time are you likely to have that to the dashboard here in 00:35:43.426 --> 00:35:48.506 the process of setting it all up or is this just going to fade a little bit 00:35:48.506 --> 00:35:51.746 and then you'll have the problem where you've got a bunch of services that aren't 00:35:51.746 --> 00:35:54.226 actually integrated in this how do you think that's going to go. 00:35:54.226 --> 00:35:57.966 That's a great question that's a great question um because 00:35:57.966 --> 00:36:01.346 my current thinking so i haven't thought a lot about that because 00:36:01.346 --> 00:36:06.126 i've been thinking i need to really freeze the state of my odroid and i need 00:36:06.126 --> 00:36:10.006 to stop adding stuff because every single effing thing i add i need to migrate 00:36:10.006 --> 00:36:15.666 to a one liter pc one day and i just went and made it a lot more complicated 00:36:15.666 --> 00:36:20.466 right so i have been thinking i was actually going to hit the pause button for a while, 00:36:20.966 --> 00:36:25.186 until i get to that migration but you raise a good point of like what if i find 00:36:25.186 --> 00:36:29.666 it in my new favorite self-hosted app and i get it all set up do i throw it 00:36:29.666 --> 00:36:33.806 into the monitoring system and i I think my answer for that is I don't know if you guys do this. 00:36:34.951 --> 00:36:40.171 This is probably just a Chris thing, but I have, I have two tiers for self-hosted 00:36:40.171 --> 00:36:41.531 applications. You know where I'm going? 00:36:41.671 --> 00:36:42.231 I think so. 00:36:44.771 --> 00:36:47.791 And it's like, if I'm just playing around or if it's just something for me or 00:36:47.791 --> 00:36:50.191 really, you know, maybe I want to, I don't know, for whatever reason, 00:36:50.191 --> 00:36:54.531 I'll just put it on a port and I'll just go to the local IP and I'll put the 00:36:54.531 --> 00:36:55.731 colon in the browser like an animal. 00:36:57.311 --> 00:37:01.031 but then when it becomes like oh this is something that's serious then i go 00:37:01.031 --> 00:37:05.011 ahead and i set up the reverse proxy and i get an ssl cert for it and i even 00:37:05.011 --> 00:37:06.331 register a dns name because. 00:37:06.331 --> 00:37:08.431 Some stuff just doesn't survive to that level you know. 00:37:08.431 --> 00:37:13.231 Right so i think that is the threshold in which i now need to say and i'm going 00:37:13.231 --> 00:37:14.491 to add it to the monitoring system. 00:37:14.491 --> 00:37:17.431 There'll probably be a natural point where you realize that it's down and you 00:37:17.431 --> 00:37:19.391 wanted it to be out but then oh right okay. 00:37:19.391 --> 00:37:24.291 Better ahead that other people do this let us know send a booster contact form 00:37:24.291 --> 00:37:27.311 and let us know if you guys do this I just wonder if it's a sus thing. 00:37:27.771 --> 00:37:31.891 Because I've definitely seen some other folks that we know be like, oh, why do you do that? 00:37:32.171 --> 00:37:35.391 I think my other question I have for you guys is, do you honestly think I did 00:37:35.391 --> 00:37:37.631 it overkill? Do you think I went too far with this? 00:37:37.991 --> 00:37:39.991 I think, is that implicit in your question there, Brent? 00:37:40.111 --> 00:37:41.971 How many more pie holes would you have run? 00:37:42.431 --> 00:37:43.091 Yeah, right? 00:37:43.311 --> 00:37:48.351 I guess it is a little bit, because there seems like there's such a gap between 00:37:48.351 --> 00:37:52.391 where you started solving this problem versus where you ended up. 00:37:54.051 --> 00:37:59.491 but as long as it's well your requirements were quite specific as well right 00:37:59.491 --> 00:38:04.031 so I think had you loosen those requirements especially with the notifying you 00:38:04.031 --> 00:38:05.171 know the tiered notifications, 00:38:05.691 --> 00:38:09.871 that probably would have made this much much easier for you however you probably 00:38:09.871 --> 00:38:13.551 would have hated your life every day of the monitoring system after that so 00:38:13.551 --> 00:38:15.531 I think if you're looking long 00:38:15.531 --> 00:38:18.591 term which it sounds like you are with this kind of monitoring then I'm, 00:38:19.472 --> 00:38:24.052 It sounds like you made the right choice, because your digital life and your 00:38:24.052 --> 00:38:27.692 maintenance of that life will just get better. 00:38:28.172 --> 00:38:31.672 You know, I saw this morning over on the BcacheFS subreddit, 00:38:31.672 --> 00:38:33.392 a couple different folks working 00:38:33.392 --> 00:38:37.332 on both for Telegraph and then for Prometheus, BcacheFS collectors. 00:38:37.872 --> 00:38:38.352 Cool. 00:38:38.752 --> 00:38:40.432 So a little incentive for you in the future. 00:38:40.552 --> 00:38:42.212 So you don't think I've overdone it? 00:38:43.452 --> 00:38:45.272 Not if you install BcacheFS, no. 00:38:45.272 --> 00:38:49.432 I mean, if I could pull BcacheFS metrics in, that would be pretty neat. 00:38:49.512 --> 00:38:50.532 I think that justifies your setup. 00:38:50.572 --> 00:38:55.212 But you see, I'm a little worried about that. But because, again, 00:38:56.072 --> 00:39:00.352 it feels it's like one of these things where when it's when it's all declared, 00:39:00.352 --> 00:39:04.332 I can kind of re pick it up and read through it and understand it. 00:39:04.432 --> 00:39:08.312 Where when it's the GUI, I have to really dig through it and really, 00:39:08.512 --> 00:39:12.492 really, really have to grind again to get it figured out again. 00:39:12.692 --> 00:39:14.712 I don't know. Maybe that's just me convincing myself. 00:39:14.712 --> 00:39:18.612 Well, no, and I mean, there's a lot of pieces, but I think one of the benefits 00:39:18.612 --> 00:39:22.392 of a Prometheus-style setup is, right, you're building on top of time series, 00:39:22.432 --> 00:39:25.532 and that's a fairly universal format for a lot of things. 00:39:25.672 --> 00:39:26.152 Yeah, yeah, yeah. 00:39:26.572 --> 00:39:30.052 Wes didn't answer your simple question. Do you think that he went overkill? 00:39:31.132 --> 00:39:33.532 No, I mean, how many services did you say you had? 00:39:33.792 --> 00:39:38.592 Around 37, and then, you know, five or six hosts in there, something like that. Yeah. 00:39:38.752 --> 00:39:41.552 I mean, it seems like maybe if anything, it's more of like, um, 00:39:41.692 --> 00:39:45.792 like a reckoning with the level of infrastructure you're already providing and 00:39:45.792 --> 00:39:48.812 that it deserves a similar class of monitoring. 00:39:49.132 --> 00:39:51.012 Yeah. And it's like, it's not only is it running a clinic, right? 00:39:51.072 --> 00:39:53.752 But the home assistant stuff is really integrated into the function of the home 00:39:53.752 --> 00:39:58.072 to a degree that like prevents freezing and, and, and other damage occurring. 00:39:58.272 --> 00:39:59.352 So it's pretty significant. 00:39:59.912 --> 00:40:04.492 Uh, yeah, I think I probably was underdoing it. Yeah, I think I might've been. 00:40:05.112 --> 00:40:08.792 And yeah, it did need, it did need a better solution. I would be curious to 00:40:08.792 --> 00:40:10.632 know if maybe there was a better way to go though. 00:40:10.792 --> 00:40:13.352 So, and how people are doing it. Always love to hear that. 00:40:13.652 --> 00:40:17.112 I'm not opposed to coming up with a better way. Could always make for a good segment. 00:40:19.816 --> 00:40:23.836 All right. Well, check this out. Linux Unplugged has been here for over 12 years. 00:40:23.996 --> 00:40:25.616 And I think I figured out why. 00:40:25.816 --> 00:40:29.016 Right. We focus on a few things. And I think this is one of our strengths. 00:40:29.356 --> 00:40:32.236 Real use cases for Linux. We can get value out of it. 00:40:32.636 --> 00:40:35.896 Free software that's actually free. And we talk about the differences there. 00:40:36.196 --> 00:40:40.636 And we try to focus on self-hosting that's practical and just works and not 00:40:40.636 --> 00:40:44.736 like the hype stuff. And I think you'll also find that we have honest conversations 00:40:44.736 --> 00:40:48.016 that try to help you make sense of the big shifts in the Linux landscape. 00:40:48.016 --> 00:40:49.896 And you can look over the 12-year history of the show. 00:40:50.751 --> 00:40:55.671 And we don't chase the outrage. We don't chase the hype. We don't go for the drama clickbait. 00:40:56.191 --> 00:41:00.491 We just try to focus on the signal there. And so when you support the Unplugged 00:41:00.491 --> 00:41:03.551 program, you're keeping something that's a bit rare alive. 00:41:03.771 --> 00:41:07.171 It's focused, thoughtful Linux podcasts that tries to stay in its lane, 00:41:07.351 --> 00:41:10.651 respects your time, and treats the community like adults. 00:41:11.311 --> 00:41:14.031 And that's probably not as common as it should be. 00:41:14.511 --> 00:41:19.031 So this here show, it runs on value for value, time, talent, or treasure. 00:41:19.471 --> 00:41:22.971 listening and sharing the show, spreading the word, time, participating in the 00:41:22.971 --> 00:41:25.191 community, helping create maybe show swag. 00:41:25.491 --> 00:41:28.011 That's time. It could be a little bit of talent in there too, right? 00:41:28.351 --> 00:41:32.571 Also your feedbacks, corrections, things like that. Also helpful. 00:41:32.731 --> 00:41:33.931 And of course, treasure. 00:41:34.751 --> 00:41:38.571 Boosts, membership, direct support at meetups, all of those things make a big 00:41:38.571 --> 00:41:40.451 difference right now. The reason why I'm talking to you right now is because 00:41:40.451 --> 00:41:41.671 we don't have a sponsor for this slot. 00:41:41.911 --> 00:41:46.831 So every bit helps the show continue and ideally thrive and grow. 00:41:47.431 --> 00:41:51.531 Better coverage, bigger experiments, more room to explore what's next before 00:41:51.531 --> 00:41:54.871 it's obvious without all the hype. I mean, look at the history of the show. 00:41:55.151 --> 00:41:58.811 So if it's helped you understand Linux better or avoid bad tech decisions or 00:41:58.811 --> 00:42:01.571 feel more confident about running your own systems, consider supporting the 00:42:01.571 --> 00:42:04.071 show to keep that going. You can send us a boost. 00:42:04.331 --> 00:42:07.771 You can become a core contributor or a Jupyter.party member. 00:42:07.791 --> 00:42:11.951 Or, of course, if you use the promo code bootleg while it lasts, 00:42:12.051 --> 00:42:12.911 you get it at a great price. 00:42:13.351 --> 00:42:18.131 That's linuxunplugged.com membership to support this show directly or jupiter.party. 00:42:18.131 --> 00:42:20.391 You get the perks and you keep the show going. 00:42:20.511 --> 00:42:23.711 And of course you can send us a boost to support each episode directly. 00:42:24.031 --> 00:42:26.931 Thank you everybody who does that. It makes all the difference. 00:42:30.051 --> 00:42:35.131 Well, AJ wrote in this week, long-time listener, Mobile Linux Survivor reporting 00:42:35.131 --> 00:42:37.071 in here. Hey, Chris, JB Crew. 00:42:37.571 --> 00:42:41.551 Been a long-time member, watching and listening since about the Matt Hartley 00:42:41.551 --> 00:42:44.711 era. So not quite a Lunduk graybeard, but almost. 00:42:46.371 --> 00:42:51.671 I'm extremely jaded about Linux phones, and for some good reasons. 00:42:52.371 --> 00:42:56.591 I backed the Librem 5, lawyers got involved there, and still no phone. 00:42:56.591 --> 00:43:00.851 I owned a PinePhone and a PinePhone Pro, which were underwhelming at best. 00:43:01.031 --> 00:43:05.011 So I assume we're all pretty burned on mobile Linux by now. 00:43:05.391 --> 00:43:10.031 But then I heard about the FLX1. Last spring, I learned about it. 00:43:10.151 --> 00:43:12.711 It's the FLX1 from Fury Labs. 00:43:12.991 --> 00:43:17.091 Didn't expect much there, but I believe in the idea of mobile Linux. 00:43:17.931 --> 00:43:22.311 So I backed it. A month or two later, though, the FLX1 was cancelled. 00:43:22.651 --> 00:43:25.251 So cue that purism-era PTSD. 00:43:26.191 --> 00:43:29.151 But here's where it gets a little weird in a good way. 00:43:29.411 --> 00:43:34.371 Fury Labs announced a replacement device, the FLX1S for slim. 00:43:34.931 --> 00:43:40.011 It offers refunds or a spot in the new queue. And so I stayed in, 00:43:40.231 --> 00:43:42.351 fully expecting another disappointment. 00:43:43.440 --> 00:43:48.800 But then they delivered. My FLX1S just arrived January 2nd, and I've been daily 00:43:48.800 --> 00:43:50.280 driving it since that weekend. 00:43:50.720 --> 00:43:54.180 Not testing, not tinkering, but daily driving it. 00:43:54.460 --> 00:43:58.480 So here's a little report. Does it actually work? Yes, calls seem to work. 00:43:58.580 --> 00:44:02.120 SMS and MMS, mobile data, GPS, most Bluetooth works. 00:44:02.460 --> 00:44:05.420 Many Android apps also via WayDroid. 00:44:05.780 --> 00:44:11.060 They use a fork called Andromeda. The battery lasts about a full day with normal 00:44:11.060 --> 00:44:15.020 use as well. The software stack, as I understand it, is FOSH, 00:44:15.480 --> 00:44:19.780 customized Debian base built on the Helium project. 00:44:20.240 --> 00:44:23.100 Those details might be slightly off, but that's the gist I understand. 00:44:23.340 --> 00:44:27.580 Is it perfect? Nope. Is it real? Shockingly, yes, it's real. 00:44:28.040 --> 00:44:31.860 There's a compromise, of course, but they're shrinking pretty quickly. 00:44:32.080 --> 00:44:35.400 Some issues get fixed day to day, not month to month. 00:44:35.700 --> 00:44:39.760 I even submitted a bug fix that'll ship by default on the next release. 00:44:39.760 --> 00:44:42.900 that alone felt wildly refreshing compared 00:44:42.900 --> 00:44:45.600 to my previous experiences so why am i 00:44:45.600 --> 00:44:49.980 reaching out well i have zero affiliation no financial interest and no incentives 00:44:49.980 --> 00:44:55.260 i just genuinely am a happy customer which feels rare enough to mention fury 00:44:55.260 --> 00:45:00.140 labs has restored some of my hope in mobile linux they're active in the matrix 00:45:00.140 --> 00:45:05.040 room with a small but engaged community and i really think you should get your hands on a device, 00:45:05.560 --> 00:45:10.120 maybe invite someone from that team onto the show and talk about what they did 00:45:10.120 --> 00:45:11.740 differently this time around. 00:45:12.220 --> 00:45:17.440 Oh, that's an interesting suggestion. Boy, I would appreciate a contact if you have one, AJ. 00:45:17.840 --> 00:45:21.580 That is a good report, right? Isn't it nice to hear that possibly a Linux phone 00:45:21.580 --> 00:45:23.820 out there that people are happy with and gets the basics done? 00:45:23.960 --> 00:45:28.120 And I think about how much I could do in a web browser if I didn't have an app. 00:45:28.280 --> 00:45:32.020 And I start to think maybe it's not crazy maybe it's maybe the dream is possible 00:45:32.020 --> 00:45:36.600 aj makes me believe again faith restored thank you aj appreciate that report 00:45:36.600 --> 00:45:40.380 great example of value contribute to the show right there with a in the field 00:45:40.380 --> 00:45:43.640 report another great example of course is la boost, 00:45:48.212 --> 00:45:52.332 And Daja's back with a row of McDucks as our baller booster this week. 00:45:55.852 --> 00:45:58.972 And he writes, I just wanted to share a little Linux and self-hosted success 00:45:58.972 --> 00:46:01.212 story. Oh, here we go. I love these. 00:46:01.272 --> 00:46:01.372 Yeah. 00:46:01.932 --> 00:46:04.892 Thanks to what I've learned from the crew, our community, and our community 00:46:04.892 --> 00:46:09.012 over the last six years, I was able to migrate off of a GCP, 00:46:09.292 --> 00:46:14.592 oh, a Google Cloud, to a combo of our own Infra and Colo, resulting in monthly 00:46:14.592 --> 00:46:16.172 savings of this boost amount. 00:46:16.552 --> 00:46:16.992 Wow. 00:46:17.932 --> 00:46:23.732 But it was in dollars instead of sats that's a big number and all foss you guys 00:46:23.732 --> 00:46:27.712 are awesome thank you love to hear that that that really you know that kind 00:46:27.712 --> 00:46:30.872 of stuff makes our day also regarding the on-site you guys did at his clinic 00:46:30.872 --> 00:46:34.792 it was super interesting and it's always those small gotchas that get you. 00:46:34.792 --> 00:46:36.592 Ain't that the truth it's always networking. 00:46:36.592 --> 00:46:40.312 That is such a great boost also i mean because it's just really great to hear 00:46:40.312 --> 00:46:43.952 that we made a little difference there but also uh appreciate the signal on 00:46:43.952 --> 00:46:47.152 that type of topic it's the first time we've ever done something like that so 00:46:47.152 --> 00:46:48.472 we always appreciate the feedback. 00:46:48.472 --> 00:46:51.832 Optical gre comes in. 00:46:51.832 --> 00:46:52.952 With 21. 00:46:52.952 --> 00:46:55.652 703 satoshis, 00:46:58.352 --> 00:47:03.252 i forgot where i live so i need west to check his map for me and then relay 00:47:03.252 --> 00:47:07.552 that information to brent so he can help me with the many unfinished projects 00:47:07.552 --> 00:47:09.152 if he ever works his way through here. 00:47:09.152 --> 00:47:13.192 Uh-oh here we go don't tell me with. 00:47:13.192 --> 00:47:13.812 A good time. 00:47:13.812 --> 00:47:16.952 Did you uh yes actually oh i. 00:47:16.952 --> 00:47:17.772 Keep it in my back pocket. 00:47:17.772 --> 00:47:21.452 There it is nice okay watch out watch out i don't want that on camera because 00:47:21.452 --> 00:47:25.392 that did sharpen the edges yeah all right do we have a location yes. 00:47:25.392 --> 00:47:28.792 We do uh 21703 looks like, 00:47:29.970 --> 00:47:33.630 A postal code from Frederick County, Maryland. 00:47:33.970 --> 00:47:34.270 Oh. 00:47:34.850 --> 00:47:35.170 Wow. 00:47:35.350 --> 00:47:35.750 There you go, Brent. 00:47:36.030 --> 00:47:38.450 Well, what you have to do with these messages is also tell me, 00:47:38.530 --> 00:47:42.190 you know, some temptations as to why I should come through the area. Well, mostly food. 00:47:42.210 --> 00:47:43.690 Well, you swing by on your way to the Capitol. 00:47:43.950 --> 00:47:46.350 If they have gluten-free pizza, then you got me. 00:47:46.990 --> 00:47:50.350 Just, he has a few allergies he'd like you to know about. If you consider those 00:47:50.350 --> 00:47:52.930 and work them into your boost, there's probably a good chance you'll stop by. 00:47:53.030 --> 00:47:56.450 Oh, and if you have a plug, outdoor plug, he can make 120 work. 00:47:56.950 --> 00:48:00.170 If you have good cat snacks, that's usually a big help. 00:48:00.390 --> 00:48:02.230 Liking cats helps, too. That is true. 00:48:03.150 --> 00:48:09.130 Well, Gene Bean sent in, this is just a little row of ducks, 2,222. 00:48:09.810 --> 00:48:14.690 He says, can you share that Nix config for the clinic? I'd love to get some ideas off there. 00:48:14.830 --> 00:48:18.530 We could, yeah. I think we would probably just want to do a quick sanitization 00:48:18.530 --> 00:48:22.650 check, since it is for a clinic and all of that. But I think we could give a 00:48:22.650 --> 00:48:23.370 look at that after the show. 00:48:23.470 --> 00:48:23.670 Totally. 00:48:23.670 --> 00:48:27.050 And if it passes the sniff test, we'll just put it in the show notes for this 00:48:27.050 --> 00:48:29.350 episode. Does that make sense? Yeah. 00:48:30.210 --> 00:48:33.150 So I guess the answer is, if the answer is yes, it'll be in the show notes, 00:48:33.230 --> 00:48:34.870 Gene Bean. Good question. 00:48:35.590 --> 00:48:36.910 Show notes for 651. 00:48:37.070 --> 00:48:41.150 You know, I've talked, yeah, linuxonplugged.com slash 651. I've thought about this. 00:48:41.750 --> 00:48:45.970 There's not a lot that's going to be revealed because anything that's like a 00:48:45.970 --> 00:48:49.750 secret gets stored outside the main config that goes in the repository. 00:48:51.410 --> 00:48:56.290 But, it is the type of thing that if I had access to someone's network, 00:48:56.290 --> 00:48:59.770 I would use the hell out of this to get everywhere I wanted to go. 00:49:00.190 --> 00:49:04.090 And I just, like, this for me, when I, I don't mean to be this guy, 00:49:04.170 --> 00:49:08.330 but like when I was hired to do penetration testing, this, I would have loved 00:49:08.330 --> 00:49:09.730 a map like this. I would have. 00:49:10.892 --> 00:49:14.652 This would be like, oh, you just gave me the job for easy, free, you know, basically. 00:49:16.152 --> 00:49:18.732 First I'll own his couple of pie holes and then I will. 00:49:18.932 --> 00:49:23.292 Right. It's really, well, it just gives you time to research and it's easier 00:49:23.292 --> 00:49:26.232 than ever to drop these configs into a machine and say, hey, 00:49:26.312 --> 00:49:29.612 machine, what's the first thing I could pick on, right? Like you got to think 00:49:29.612 --> 00:49:31.252 about the tools that are available to people now. 00:49:31.672 --> 00:49:36.232 And so it crosses my mind that there is a level of information that's being exposed. 00:49:36.852 --> 00:49:42.232 And so I have some consideration there. But it relies on breaking into the infrastructure. 00:49:42.512 --> 00:49:47.352 And then most of what gets exposed is just internal non-routable IP addresses 00:49:47.352 --> 00:49:50.972 and things like that, or perhaps where secrets get stored and whatnot. 00:49:51.352 --> 00:49:55.332 But it's something I think about, and I would love the audience's thoughts about it as well. 00:49:55.392 --> 00:49:58.372 And if you guys are concerned, I know there's a culture around sharing your 00:49:58.372 --> 00:50:01.472 Nix configs and your Ansible configs, and I like that. And I use that. 00:50:01.472 --> 00:50:03.032 We've benefited a lot from it. 00:50:03.452 --> 00:50:07.172 And I know I could do a sanitized version. So, I mean, maybe we'll try that. 00:50:07.172 --> 00:50:11.152 But then I'm not – I prefer if I'm going to put it up on GitHub, 00:50:11.492 --> 00:50:15.492 I would prefer to actually use it. 00:50:16.492 --> 00:50:21.032 You know, like – and then it's – anyways. Good question. Check the show notes, 00:50:21.132 --> 00:50:24.452 Gene, and I would love people's feedback on that. Thank you very much. 00:50:25.372 --> 00:50:29.832 Is it my turn now? I don't remember. Sifeseeker comes in with – I got all distracted. 00:50:30.012 --> 00:50:33.272 Comes in with 2,500 sats. 00:50:35.319 --> 00:50:38.499 Hi guys, I've been kicking around the idea of a NixOS router, 00:50:38.699 --> 00:50:40.279 and an example would be great, Leo, there you go. 00:50:41.879 --> 00:50:47.899 Would you be willing to share your config? The router config is interesting, right? It's in there. 00:50:48.019 --> 00:50:52.899 I will also mention that Bearded Tech in our community has a really cool NixOS-based router project. 00:50:53.239 --> 00:50:55.539 And if you're actually thinking about using it for your home router... 00:50:55.539 --> 00:50:56.779 Yeah, this might be something to consider. 00:50:56.819 --> 00:50:58.019 That might be something worth looking at. 00:50:58.019 --> 00:51:00.179 We were kind of doing a bunch of stuff all at the same time, 00:51:00.319 --> 00:51:03.419 so we took a peek, but we kind of wanted to start a little more minimal and 00:51:03.419 --> 00:51:06.279 work our way up. But it looks great, especially if you just want, 00:51:06.379 --> 00:51:08.699 like, a standalone router that is NixOS powered. 00:51:08.879 --> 00:51:14.819 Yes. And you've got to remember, we were building something that was a VM first, VM server first. 00:51:15.059 --> 00:51:17.159 It wasn't going to be the router first. 00:51:17.359 --> 00:51:21.539 Yeah, yeah, right. Exactly. Yeah. All right. But, yeah, all right. 00:51:21.639 --> 00:51:25.099 We'll take a look at that, Cypher. Thank you very much. I guess there's some demand. 00:51:25.519 --> 00:51:28.079 Hybrid Sarcasm comes in with 10,000 SATs. 00:51:33.668 --> 00:51:36.068 You asked for some feedback regarding actual budget. 00:51:36.208 --> 00:51:36.328 Yeah. 00:51:36.528 --> 00:51:41.048 It's been a pretty good replacement for you need a budget with spousal approval so far. 00:51:41.688 --> 00:51:44.888 That's $9 a month I don't have to spend on a cloud service. 00:51:45.148 --> 00:51:49.368 You'll also appreciate that the actual devs have a sustainable funding model 00:51:49.368 --> 00:51:52.848 for their core contributors, and they are looking to expand it to others. 00:51:53.048 --> 00:51:54.588 And then we've got a link we'll put in the show notes. 00:51:54.788 --> 00:51:56.528 Oh, interesting. I'm looking. I'm trying. There it is. Yeah. 00:51:56.628 --> 00:51:58.688 So they have in their documentation here. Hmm. 00:51:59.408 --> 00:52:04.428 Thank you. I did not know that. That's a good little bit of information there. I appreciate that, Mr. 00:52:04.448 --> 00:52:04.488 Zark. 00:52:04.588 --> 00:52:05.328 I appreciate the report. 00:52:05.328 --> 00:52:07.588 Are you saying those are the actual, actual devs? 00:52:08.568 --> 00:52:09.868 No, the actual, actual, actual. 00:52:10.008 --> 00:52:10.348 No, yeah. 00:52:10.388 --> 00:52:16.148 Oh, actually, okay. Well, adversaries came in with 8,441 sats. 00:52:20.368 --> 00:52:24.108 Adversaries responding to our question last week, Chris, you were asking for 00:52:24.108 --> 00:52:26.688 Wi-Fi analyzers for Android. 00:52:27.568 --> 00:52:31.088 adversaries says unify makes a great wi-fi analyzer 00:52:31.088 --> 00:52:34.208 app called wi-fi man it doesn't require unify 00:52:34.208 --> 00:52:37.448 gear to work it just uses your phone's radios and 00:52:37.448 --> 00:52:41.588 i can second this one this is the application i've been using for about the 00:52:41.588 --> 00:52:47.388 last year i used it just this week to that's good to know hey hey you didn't 00:52:47.388 --> 00:52:50.428 give me time to answer your question nor did you ask me while you were working 00:52:50.428 --> 00:52:57.088 on this project so oh wow dude anyways it's fantastic i use it just this week uh to I fixed my parents' 00:52:57.228 --> 00:53:00.248 Wi-Fi, and it's got some sweet features. 00:53:00.388 --> 00:53:03.208 So I would say put this on your phone, play with it. 00:53:03.308 --> 00:53:07.808 It's pretty amazing. It took me a few uses to discover all of the different 00:53:07.808 --> 00:53:10.528 crazy features that are hidden in it. It's really quite good. 00:53:10.968 --> 00:53:14.628 This makes you the buddy that shows up with a water hose after I put the fire 00:53:14.628 --> 00:53:15.888 out. You realize that, right? 00:53:16.108 --> 00:53:16.468 You're welcome. 00:53:16.768 --> 00:53:18.688 All right. That makes you – that's what you – I. 00:53:18.688 --> 00:53:19.468 Could start a new fire. 00:53:20.228 --> 00:53:26.708 No doubt we will. No doubt about that at all. Anonymous comes in with 2,021 sats. 00:53:27.468 --> 00:53:31.528 No message, just value. Thank you very much. And then Tomato comes in with a row of duckles. 00:53:32.875 --> 00:53:36.675 And writes, I loved this old network segment. I'd be curious to hear if Brent 00:53:36.675 --> 00:53:38.075 started to automate his van yet. 00:53:38.275 --> 00:53:41.215 Mine is completely unautomated. I'm not sure where to start. 00:53:41.655 --> 00:53:43.655 Oh, well, he needs to start with sensors, right, Brent? 00:53:44.195 --> 00:53:48.975 Sensors is great. Yeah, I did the opposite this week and pulled my lithium batteries 00:53:48.975 --> 00:53:54.415 out of my van. So I've unautomated everything only because it got really, 00:53:54.515 --> 00:53:55.555 really, really, really cold. 00:53:55.775 --> 00:53:58.915 And this is not very good for them. So I feel like this week I went backwards. 00:53:58.915 --> 00:54:03.355 but I'm going to kind of build all the automation here in the workshop, 00:54:03.355 --> 00:54:07.595 just as winter's here, then I can just plunk it in the van, you know? 00:54:08.315 --> 00:54:13.415 But I would say, Chris, you've got much more opinions on this than I do, 00:54:13.515 --> 00:54:16.655 but I would say start with the problems that you feel like you want to solve 00:54:16.655 --> 00:54:18.475 or have visibility into, right? 00:54:18.555 --> 00:54:22.255 If you want exterior temperatures versus interior temperatures, 00:54:22.575 --> 00:54:23.795 and that's really important to you, start there. 00:54:24.095 --> 00:54:27.795 If you want to, I don't know, have some other solution to a problem, 00:54:27.795 --> 00:54:29.435 That's always the best place to start. 00:54:29.715 --> 00:54:34.015 And plus one sensors. And then if you really want to see what's capable and 00:54:34.015 --> 00:54:37.655 way, way far out there, check out smartyvan.com. 00:54:38.295 --> 00:54:44.795 He's also a YouTuber and he has created some really inspired automations around van living. 00:54:44.955 --> 00:54:49.655 I mean, absolutely high end tech stuff that you could build from for years. 00:54:49.655 --> 00:54:53.895 He's also released some code. He has examples and automations and video tutorials 00:54:53.895 --> 00:55:01.135 and all of that. So it's smartyvan.com, S-M-A-R-T-Y, van.com. 00:55:01.235 --> 00:55:02.475 And you get some good inspiration there. 00:55:02.595 --> 00:55:07.975 I think my next step, if you're curious, is likely getting some visibility and 00:55:07.975 --> 00:55:11.475 automation around keeping these batteries charged and healthy. 00:55:12.290 --> 00:55:16.250 One of them is how to keep them warm while they're charging in the winter or, 00:55:16.250 --> 00:55:19.050 you know, in the after winter season. 00:55:19.530 --> 00:55:23.270 So that's the main problem I have that I'm going to solve that will get me, 00:55:23.430 --> 00:55:28.890 you know, some open hardware experience and also diving more into what Home 00:55:28.890 --> 00:55:31.570 Assistant can do to automate all this. So that's my next step. 00:55:31.710 --> 00:55:34.010 But write in and let me know what problem you're solving. 00:55:34.310 --> 00:55:37.630 I would also add, like, if you want something to rabbit hole into for a while 00:55:37.630 --> 00:55:42.050 before you get into all this, go learn the ESP platform. So that is a skill 00:55:42.050 --> 00:55:43.750 that will pay dividends years. 00:55:44.050 --> 00:55:44.850 And relays, relays. 00:55:44.970 --> 00:55:45.110 Yeah. 00:55:45.430 --> 00:55:46.210 ESVs and relays. 00:55:46.690 --> 00:55:49.150 And the little sensors and there's kits you can get on, you know, 00:55:49.230 --> 00:55:51.990 the big box websites for super cheap and all of that. 00:55:52.110 --> 00:55:53.310 And buy some epoxy too. 00:55:54.430 --> 00:55:55.310 While you're at it. 00:55:56.710 --> 00:56:00.990 All right. Thank you, everybody who boosted the show. We do appreciate you very much. 00:56:01.070 --> 00:56:05.330 And, of course, shout out to our SAT streamers as well. We had 26 of you stream 00:56:05.330 --> 00:56:10.210 sats, and collectively you came in with 26,866 sats, which does technically 00:56:10.210 --> 00:56:13.370 make our streamers the baller booster again this week. Thank you, everybody. 00:56:13.590 --> 00:56:18.810 When you combine that with our boosters, we raised a total of 109,354 sats. 00:56:19.010 --> 00:56:22.930 Pretty humble, but we're very appreciative and gives us an opportunity to make 00:56:22.930 --> 00:56:24.750 our birthday episode a banger. 00:56:27.590 --> 00:56:31.370 There's real, real easy ways to boost in these days. Fountain FM is making it 00:56:31.370 --> 00:56:36.290 easier and easier, including making it just all kind of dollar-based, simple stuff. 00:56:36.410 --> 00:56:39.470 And, of course, there's the entire awesome self-hosted infrastructure. 00:56:39.470 --> 00:56:41.190 You can find that when you go 00:56:41.190 --> 00:56:44.290 to newpodcastapps.com. You'll go down that rabbit hole. You get AlbiHub. 00:56:44.450 --> 00:56:46.830 It's really awesome. And, of course, we have the membership program, 00:56:47.350 --> 00:56:52.330 linuxunplugged.com slash membership, or jupiter.party for the whole dang network. 00:56:54.245 --> 00:56:56.245 All right. Would you guys like a few picks? 00:56:56.585 --> 00:56:58.145 Yeah, what'd you get in your bag today? 00:56:58.345 --> 00:57:03.445 Well, you've heard me mention my Hypervibe, which is a NixOS-based Hyperland 00:57:03.445 --> 00:57:05.545 desktop and still rocking it. 00:57:05.605 --> 00:57:09.365 Got it running on three machines these days and it's in a great state and I like it a lot. 00:57:09.525 --> 00:57:12.985 But perhaps you are an Arch person. 00:57:13.585 --> 00:57:15.845 Well, Rich Arch has a Hypervibe spin. 00:57:16.045 --> 00:57:19.925 They say we at Rich Arch Project are re-releasing our Hypervibe spin. 00:57:20.205 --> 00:57:26.005 We now have taken the Hypervibe configs and enriched them with the Noctelis shell on Hyperland. 00:57:26.585 --> 00:57:30.605 You can try it in a VM, and it includes some screenshots, or some instructions 00:57:30.605 --> 00:57:32.905 and a screenshot, which we'll put a link to in the show notes. 00:57:33.925 --> 00:57:37.665 It's better looking than the way I have it configured. I'll tell you that. It's really nice. 00:57:37.725 --> 00:57:42.645 So now you're going to re-Nixify, Hypervibe-ify, the rich Arch Hypervibe-based config? 00:57:42.805 --> 00:57:47.065 I like the way you think. You basically start with a base Linux, 00:57:47.605 --> 00:57:50.025 or a base Arch ISO, just base install. 00:57:50.545 --> 00:57:53.345 And then he has some kickoff scripts that you can curl onto that basic system 00:57:53.345 --> 00:57:58.005 and turn it into a Hypervibe desktop based off of what I set up to run on Nix. 00:57:58.225 --> 00:58:01.105 But with Arch. Kind of neat. Thank you, Rich, for sending that in. 00:58:01.225 --> 00:58:01.445 That's great. 00:58:01.625 --> 00:58:05.245 It's beautiful. It's beautiful. And then I've got one that. 00:58:06.545 --> 00:58:07.585 This looks really nice. 00:58:07.905 --> 00:58:10.745 This is nice to see. It's good to have another one of these. 00:58:11.005 --> 00:58:14.165 I've talked about Junction before. Now we're going to talk about Switchyard, 00:58:14.405 --> 00:58:19.585 a modern rules-based URL launcher that replaces your default browser. 00:58:20.025 --> 00:58:24.245 So wrap your noodle around this. Instead of having one browser as your default, 00:58:24.485 --> 00:58:27.225 you set switchyard as the default browser. 00:58:27.505 --> 00:58:32.205 And then when you click a link, it brings up a little window and it lets you 00:58:32.205 --> 00:58:34.505 choose which browser you want to open in. 00:58:34.725 --> 00:58:38.765 But on top of that, they have added a really nice graphical interface, 00:58:38.925 --> 00:58:43.425 a GTK graphical interface, where you can have rules to just automatically send 00:58:43.425 --> 00:58:45.505 some URLs right to a particular browser. 00:58:45.645 --> 00:58:49.665 And this is exactly how I work. And this is why I really appreciate this Because 00:58:49.665 --> 00:58:51.765 there's some stuff I always open in Firefox. 00:58:52.125 --> 00:58:55.265 There's one site and only one site I use Brave for right now. 00:58:55.385 --> 00:58:57.145 And then there's other stuff I open in Zen. 00:58:58.579 --> 00:59:02.259 And it's very, it's always that stuff. So this is really, really great. 00:59:02.459 --> 00:59:05.779 It's a super fast app and it has a simple configuration. 00:59:05.839 --> 00:59:10.019 If you do want to do it by text, they have a flat pack and a Nixflake ready 00:59:10.019 --> 00:59:12.359 to go. So I thought I might get your approval on that one too. 00:59:12.539 --> 00:59:16.659 Oh yeah, absolutely. And it's written in Go, GPL 3.0. 00:59:16.819 --> 00:59:20.479 Yeah. So this is so nice if you do live the multi-browser lifestyle. 00:59:20.859 --> 00:59:26.119 And I do. And I probably would say Firefox is 90% everything. 00:59:26.119 --> 00:59:28.979 but then there's those, or maybe like if I'm going to do a Google Meet, 00:59:29.099 --> 00:59:30.079 I might actually do that in Chrome. 00:59:30.479 --> 00:59:34.599 And I maybe don't use Google Chrome for literally anything else on that machine, 00:59:34.619 --> 00:59:35.899 but I use it for Google Meet. 00:59:36.339 --> 00:59:39.519 And it's nice to have something. I just click a link and switch art, it'll send it to that. 00:59:39.619 --> 00:59:42.759 But if I don't have any rules set, it gives you a really lean, 00:59:42.859 --> 00:59:47.059 mean, fast UI and you big icons and you just select the browser you do want 00:59:47.059 --> 00:59:49.939 to open and it sends a link to that browser. So it's, ha ha. 00:59:50.179 --> 00:59:52.739 I think especially, right, they kind of keyed in on work or other stuff, 00:59:52.839 --> 00:59:55.179 but even just maybe you're doing, you're in a mode where you're doing show notes. 00:59:55.179 --> 00:59:58.779 You want to make sure you, you know, you're clinking some of the Brent sends. 00:59:58.839 --> 00:59:59.879 You want it to open the right spot. 01:00:00.059 --> 01:00:02.419 Yeah. Something that's in private browsing mode for sure. 01:00:02.639 --> 01:00:04.579 Well, yeah, no JavaScript either. 01:00:04.859 --> 01:00:05.179 Containerized. 01:00:05.359 --> 01:00:05.679 Definitely. 01:00:07.232 --> 01:00:09.772 All right. Well, that's pretty much the end. I just want to remind everybody 01:00:09.772 --> 01:00:10.792 that the meetup page is up. 01:00:10.872 --> 01:00:13.592 If you're going to be in the Pasadena area around March 5th, 01:00:13.612 --> 01:00:17.592 we'll be at Planet Nixon scale and hanging out with our buddies from Phlox. 01:00:17.692 --> 01:00:20.012 We'd love to see you there. Meetup.com slash Jupiter Broadcasting. 01:00:20.012 --> 01:00:23.252 We'll get the details locked in soon for all of that. 01:00:23.432 --> 01:00:26.352 I'm very much looking forward to it. It's going to be nice, especially as it's 01:00:26.352 --> 01:00:30.572 very cold right now. I'm picturing the nice sunshine. It's a beautiful time to be in Pasadena. 01:00:30.812 --> 01:00:31.312 Seeing a wonderful audience. 01:00:31.632 --> 01:00:36.152 Mm-hmm. Mm-hmm. Good crew down there, too. Wes, is there some pro tips we could 01:00:36.152 --> 01:00:37.132 leave with them? you know things 01:00:37.132 --> 01:00:40.912 where they could get more data more information around the show like. 01:00:40.912 --> 01:00:42.872 Some sort of enriched xml file. 01:00:42.872 --> 01:00:43.972 Yeah something links. 01:00:43.972 --> 01:00:45.752 To uh text and json files. 01:00:45.752 --> 01:00:46.532 Could be yeah. 01:00:46.532 --> 01:00:47.452 With like chapter. 01:00:47.452 --> 01:00:51.512 Information and transcript information yeah could have all of that. 01:00:51.512 --> 01:00:54.432 Like an srt for however you want to consume this. 01:00:54.432 --> 01:00:57.612 It's that if you've got a um podcast client there's more and more of them that 01:00:57.612 --> 01:01:00.392 supports transcripts we have that in the feed for you and of course if you have 01:01:00.392 --> 01:01:03.612 a podcasting toodle client you get all kinds of stuff like the cloud chapters 01:01:03.612 --> 01:01:08.512 like the live item entry pending information and a whole bunch of good stuff. 01:01:08.672 --> 01:01:09.752 And of course we are live. 01:01:12.932 --> 01:01:16.532 Yeah, we love it. If you make it a Tuesday on a Sunday, join us Sunday at 10 01:01:16.532 --> 01:01:20.432 a.m. Pacific, 1 p.m. Eastern, jupiterbroadcasting.com slash calendar for your time. 01:01:21.072 --> 01:01:23.432 If you want to, I don't know, read about what we talked about. 01:01:23.592 --> 01:01:24.572 You want more show? I don't know. 01:01:25.092 --> 01:01:31.852 Linuxunplugged.com. This was episode, geez, 651. So Linuxunplugged.com slash 651. 01:01:32.132 --> 01:01:36.432 We get together every Sunday with our Mumble room. That information's on our website as well. 01:01:36.732 --> 01:01:40.112 You can get in there, get a low latency Opus stream. We tell you about it. Try it out. 01:01:40.432 --> 01:01:44.192 And last but not least, we have that Matrix room going 24-7. 01:01:44.392 --> 01:01:47.352 You can find details to that. It's a great community. And if you're already 01:01:47.352 --> 01:01:48.812 in the Federation, why not join us? 01:01:49.092 --> 01:01:51.592 Thanks so much for joining us on this week's episode of Unplugged. 01:01:51.692 --> 01:01:53.612 See you right back here next Sunday!