Chris: Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.
Wes: My name is Wes.
Brent: And my name is Jeff.
Chris: Hello, gentlemen. Coming up on the show this week, one piehole,
Chris: two VPNs, and zero public exposure. I'm pretty proud of this one.
Chris: Then, it's our pitch to ditch your GUI-only monitoring system,
Chris: and why we rolled out Prometheus and Grafana.
Chris: And then we're going to round the show out with some great boosts,
Chris: some great picks, and a whole lot more. So before we go any further,
Chris: time-appropriate greetings to our virtual lug. Hello, Mumble Room.
Mumble: Hello, Wes, and hello, Brent.
Chris: Hi. Yeah, you can join us in the Mumble Room or at jblive.tv.
Chris: Make it a Tuesday on a Sunday. We have the times at jupiterbroadcasting.com slash calendar.
Chris: And a big good morning to our friends over at Defined Networking.
Chris: Go check out Nebula VPN. They have a full managed product, 100 devices,
Chris: no credit card required.
Chris: Support the show, defined.net slash unplugged.
Chris: It is a great service. And, you know, when I've thought about it a lot,
Chris: I talk about how Slack used it and they launched it in 2017 to build out the
Chris: security around the Slack global empire. And I talk about how Rivian uses it
Chris: for real time analytics for the cars to do securely on the road.
Chris: And those are all really big-scale projects. But recently, I've appreciated
Chris: how great Nebula is on a 1, 2, 3 node network.
Chris: And the fact that I can set up a on-demand mesh network that has name resolution
Chris: and everything. We'll talk more about this.
Chris: And there's no big tech login. There's no third-party hosted admin dashboard.
Chris: Nothing like that. It's just two machines using cryptographic keys talking to each other.
Wes: It's just a couple of text files, really.
Chris: It's so powerful for small home lab stuff, and it's so scalable to massive enterprise stuff.
Chris: And you can try it out with our fully managed product and support the show by
Chris: going to define.net slash unplugged.
Chris: You're going to like it a lot. And I'll tell you, I've been using it on extremely
Chris: limited bandwidth connections.
Chris: And it's so much better, and it's so much more resource sensitive.
Chris: It's way lighter. It's way lighter. Check it out, define.net slash Nebula.
Chris: And thank you to Define for sponsoring the Unplugged program.
Chris: All right, you know we've got to mention it. Planet Nix and Scale23x are 39 days away.
Chris: That means 33 days until Brent needs to be going down the road, at least.
Wes: Let's just round that to 30.
Chris: And six, I believe, or five, actually, more Linux Unplugs, maybe,
Chris: until we need to be on the road ourselves.
Brent: Wow.
Chris: So it's coming up.
Wes: I think we better get in the Nixie mood.
Chris: Yeah, and I am really looking forward. Planet Nix has a theme this year.
Chris: It's where builders come together.
Chris: And our Nix coverage is supported again by Phlox, who's focused on making reproducible
Chris: dev environments actually usable.
Chris: And it's a fantastic tool.
Chris: So check out Phlox and come see us at Scale and Planet Nix. You do need to register at Scale.
Chris: And you can take 40% off that registration with our promo code UNPLG,
Chris: U-N-P-L-G. and we'd love to see you there. One other item.
Chris: The meetup page is now live. The details are not yet locked in.
Chris: The date, time, location likely to change.
Chris: But you can join the meetup and you'll be the first to get updates.
Chris: And if you are intending to join us at the meetup, please consider signing up
Chris: for the meetup. Please. We'd love to see you there.
Chris: Last time we had about 80 more people than we expected. Great problem to have.
Chris: It was very stressful on the restaurant staff. And they thankfully could open
Chris: up. They had to open up another wing for us, which they were able to do.
Chris: but this time we wanted to give them a great heads up. So if you're planning
Chris: to make it and I want to bring a guest, there's room for that too.
Chris: Just let us know. And we'll plan accordingly.
Chris: Meetup.com slash Jupiter broadcasting link in the show notes to the direct meetup.
Chris: We'd really appreciate it. If you could make it, if you're in the area,
Chris: even if you can't go to the event, you're welcome to join us at the meetup.
Chris: We did get one submission for that. I saw for a swag idea that we could hopefully
Chris: have together for scale and Linux fest. It was a nice one.
Chris: I'll show it to you boys after the show but I'd like to see a few more send
Chris: them in to unplugged at jupiterbroadcasting.com or tag Wes in Matrix,
Chris: and let us know we'll try to put one together pretty soon so we all have a uniform
Chris: that we can identify each other with and have easy conversation hey I know you
Chris: you listen to the show are.
Wes: We getting hats.
Chris: Ooh you know I'm a hat guy now you know I'm a hat guy now that's right,
Chris: Well, what is, in a name, gentlemen, in short, convenience, right?
Chris: When you set up your home lab or your enterprise network, whatever it is,
Chris: it is eventually inevitable that you need good name resolution.
Wes: I suspect for you there might be a spousal approval factor in the mix for that too.
Chris: Yeah, and also just a memory factor. It gets hard to remember,
Chris: especially the mesh network VPNs and the LAN IPs.
Chris: And, of course, I have to go and make it hard, and I have multiple mesh networks
Chris: now, multiple locations.
Chris: some behind double carrier grade NAT a couple of them behind double carrier
Chris: grade NAT so I had to go and make it hard on myself and I want sensible name
Chris: resolution that works on the LAN and works across the various mesh networks,
Chris: so I can just you know connect by machine name to all of them and then I need
Chris: something also that does fast forwarding out to the internet and then can cache
Chris: that so then future queries are faster.
Wes: And then was it something you wanted to, like, I don't know,
Wes: do you have some of these services that depend on other services in a way where,
Wes: like, DNS is how they find each other?
Chris: Yeah, and there's a lot of things I've set up are just by name now.
Chris: So, you know, I had a basic pie hole going on my tail net, and I had a basic pie hole going on my LAN.
Chris: But then we set up my wife's clinic.
Wes: And was the tail net pie hole was running, like, as a container on a VPS or something?
Chris: Yeah, and it just only had an interface on the Tailnet. So it was just acting
Chris: as name resolution for the Tailnet.
Chris: And then I kind of combined that with MagicDNS and sort of had the whole Tailnet thing solved.
Chris: Then I had to go set up another network and all of that.
Chris: And I also just kind of wanted to take another look at this and see if I couldn't do this better.
Chris: But when I had set it up for the Tailnet only on the VPN, I took a shortcut.
Chris: And instead of having to worry about exposing a pie hole to the internet...
Chris: I just only bound it to the tail net interface.
Chris: So I didn't have to worry about public IP and the internet, you know,
Chris: banging on my pie hole server that's on a VPS because it couldn't talk to it.
Chris: But if I wanted to make this pie hole usable across multiple mesh networks,
Chris: it meant undoing that sort of convenience in security I had and coming up with
Chris: a better security architecture to go across multiple networks.
Chris: That's where it got a little kind of more complicated because
Chris: i went from the easy way to the hard way and so
Chris: there's multiple uh layers i kind of took
Chris: to this and i kind of like to hear your guys's feedback on this so the first
Chris: step i took is i wasn't sure if this is the right call but i essentially put
Chris: the pie hole container on host networking so it could see all the interfaces
Chris: And then in the configuration,
Chris: I limited by application configuration to only bind to the tail net and the
Chris: Nebula VPN interfaces and to not bind to the WAN interface.
Chris: So at an application configuration layer, I did that.
Chris: And then at another layer, I also set up ACLs with IP tables,
Chris: just real basic IP tables that blocks all traffic on port 53.
Chris: So like just in case, you know, for a moment, like when PyHole's starting up,
Chris: if for a brief moment it bound to port 53 on the WAN interface,
Chris: this would essentially prevent that from happening. Or if I make a config change
Chris: mistake in the future, it prevents it from exposing it to the public internet.
Chris: And so that's sort of the...
Chris: multi-layer approach in a way and then all the communications just happening
Chris: over the mesh vpns i'm not communicating with the buy hole at all no admin interface
Chris: nothing over any public interface,
Chris: how do you feel i did is that too risky would you brent would you be comfortable
Chris: with that deployment i suppose i.
Brent: Mean to me that feels probably more fine than anything that i've probably deployed
Brent: in the past so it seems okay but really i'm not the pro or anything like that
Brent: but But what I'm getting from you is that this is upping your peace of mind with this.
Brent: But there's also some hesitation. So I'm curious to hear what Wes has to say.
Wes: I do think it seems totally reasonable. You could, you know,
Wes: get with the times and use NF tables already. No, I'm just kidding.
Chris: I did actually, that actually considered, I was like, ah, this is what I know. But yeah.
Wes: I think from background discussions, I picked up maybe you were using a sidecar before?
Chris: A tailscale sidecar, yeah.
Wes: So I think maybe in like another version, if you were going like fully,
Wes: you know, application mesh native could be to just double down on the sidecar.
Chris: Do a nebulous sidecar.
Wes: Yeah. Like have it serve those two interfaces just in its own containerized
Wes: networking environment.
Chris: I like that.
Wes: Where things might get more complicated depending on exactly what you want and
Wes: convenience, et cetera. What matters to you is what you're doing with that host.
Wes: And is that host then wanting to query the pie hole? And are you going to let
Wes: that happen over local host? Or in this scenario, you'd either need to replumb
Wes: stuff and forward it or rely on it only querying it over the mesh,
Wes: which would probably be fine, but maybe you don't want to do that.
Chris: The host is also on the tail net. So there's that too. But yeah, that is a tricky part.
Chris: Technically, the host OS can't talk to it over the network, which hasn't been an issue yet.
Chris: But so that's the basic, that's the core network setup, okay?
Chris: And then what I decided to do was I turned off the tailscale MagicDNS stuff
Chris: and didn't like the results because I do not have DNS entries for every machine on my tail net.
Chris: And that's what MagicDNS was solving for me. So my sort of compromise solution
Chris: was I re-enabled MagicDNS and then I added this pie hole as the upstream DNS server for MagicDNS.
Chris: And I think that worked seemingly pretty well.
Chris: and then I enabled the DNS.
Wes: Right, so in that setup, TailScale will answer sort of right away for the TailScale
Wes: host and then forward to your setup for anything it doesn't know about where
Wes: you have to find your own manual entries.
Chris: And that's where you'll find entries for the Nebula devices.
Brent: Nice.
Chris: And then you can configure the Nebula lighthouse to suggest a DNS server to the clients.
Chris: And, um, that is, that's a really simple, it's like two lines of configuration
Chris: on the lighthouse and you just give it the DNS server.
Chris: And then, so that's also helping the Nebula clients discover who they're supposed
Chris: to talk to for name resolution.
Chris: And since I only have like three nodes on this little tiny, maybe four nodes
Chris: now on this little tiny network, I'll talk more about super easy to just add the entries manually.
Chris: And I don't, if this is going to be for a private clinic, so I don't think I'll be adding more hosts.
Wes: One thing we should play with, which I haven't yet, but I'd like to get more
Wes: into is doing either delegation or maybe using an API to trigger updates because
Wes: Nebula lighthouses can serve DNS.
Chris: Yeah.
Wes: So you could also, depending on if you want to, maybe the static has advantages
Wes: too, of course, but you could also maybe set it up so...
Chris: Yeah.
Wes: You know, the pie hole would just query Nebula and be able to answer for the
Wes: Nebula host without you having to hard code it.
Chris: The advantage was on the pie hole DNS server, now I also have a bunch of entries
Chris: for the devices that are on my LANs.
Chris: So it hosts here at the studio and hosts at the RV are also on this DNS server.
Chris: So all the machines, if you're on the LAN, whichever LAN you're on,
Chris: or whichever Mesh VPN network you're on, we all can resolve the same host names now.
Chris: So that's kind of why I didn't go that direction.
Chris: But I think that would be an easier setup if you just had a couple of machines.
Wes: Well, I meant like integrating the two, like keeping the pie hole,
Wes: just letting Nebula answer for the host it knows about.
Chris: Oh, okay. And then would it upstream to the pie hole when it does?
Chris: I see. Yeah, I like, okay. Oh my God, change.
Brent: How do you feel, Chris, about the need for internet access here?
Brent: Because occasionally you don't actually have access whenever a storm comes by
Brent: or you're traveling, that kind of thing.
Brent: So your name resolution internally on your local network would be affected.
Brent: Is that a correct understanding?
Chris: I did. Yeah. And so for that, I kept my pie hole on my LAN. And it forwards now to this guy.
Brent: Nice.
Chris: But for the most part, because that pie hole has been around so long,
Chris: I have all these same DNS entries already. So, but I did keep it for that reason.
Chris: And I'm very happy now. It adds complexity to have two mesh networks and,
Chris: you know, multiple LANs, but it's seamless now to the end user now that I've done this.
Chris: So I'm pretty happy, and the latency is pretty good even for LTE connections, really.
Wes: Well, you know, it kind of makes sense, too. It's like, it'd be one thing if
Wes: you didn't have the existing infrastructure and all that, but because you kind
Wes: of have hosts that are positioned to fit into both of these networks or could
Wes: bridge them, like it doesn't actually.
Wes: You didn't have to stand up a bunch of new infrastructure. You kind of just
Wes: had to reprovision some of it to better work with your new setup.
Chris: I would like to actually ask, so if you want to boost in or send us a contact,
Chris: if you were building this from scratch, so I already had a pie hole going.
Chris: But if you out there, listener, were building this from scratch,
Chris: what would you have used to do this name resolution?
Chris: Because it did cross my mind. Like, maybe this is just a stupid DNS mask thing.
Chris: I just set up a simple DNS mask.
Chris: But then I like the idea of a little bit of ad blocking for the systems as well.
Chris: That's nice. That's a nice feature that comes with it.
Wes: And you can do dns mask configuration with.
Chris: Pot all right because it uses like a forks version that was my conclusion yeah
Chris: i was like yeah well i might i kind of get and i know how to use it yep and
Chris: it's worked fine for me and it's survived multiple major upgrades now so it's
Chris: past those tests as well so it's a good project but i would be curious like
Chris: i think you probably would use tectidium or technically that would least.
Wes: Be not or like because i know can do sort of like delegate zones where it will
Wes: say like hey for anything in this sub you know maybe you have like dot nebula
Wes: demands or whatever, go query this server for those and then return those.
Wes: It also has some plugin capability, which I haven't really explored. Um,
Wes: Or, you know, there's a lot of good options these days.
Chris: Yeah, I saw some people that were solving this with AdGuard.
Chris: Okay, yeah. Yeah, you could totally use Bind, of course.
Chris: So I'd just be interested to know how people are solving this.
Chris: I would also like to know if anybody has a way to solve this declaratively,
Chris: you know, so that would also be a winner in my book.
Chris: But while we were talking about
Chris: Nebula, you've been working on something that's kind of slick, Wes Pano.
Wes: Yeah, it was just an idea we had while we were toying around with setting up
Wes: the clinic the other week.
Wes: was, well, what if you just had like a low-key, you know, not crazy production
Wes: scale, not being like a whole control plane for Nebula necessarily,
Wes: but just something to make printing new host certs easier.
Chris: Yeah, could you explain that a little bit? So if I'm not using the managed product,
Chris: there's sort of some cert exchanges that have to happen.
Wes: Yeah, right. So you have to, you're basically managing a CA.
Chris: Right?
Wes: So you have your own certificate authority, and then to get hosts onto the network,
Wes: they generate their own private key but then you kind of have to sign the public
Wes: part of that and that's how they get blessed with a host name and an IP address
Wes: on the network and then that's how anything trusts them,
Wes: when you try to communicate with something you need to be able to present that
Wes: public side that is signed by the CA that they all mutually trust and.
Chris: The beauty is the simplicity is it's really coming down to files you're moving
Chris: around that have keys in them and that is the totality of the infrastructure
Chris: actually required to get this working,
Chris: And if you sit with the amazingness of that for a moment, it really is very impressive.
Chris: These machines are discovering themselves.
Chris: You need a lighthouse, but they're discovering or use a public one.
Chris: And they're communicating and creating a mesh VPN just by exchanging these key files.
Wes: Yep. And, you know, just simple concepts of groups and you have stuff signed
Wes: by the right thing and it kind of all just works. But...
Wes: For simple static networks, that works pretty well. But, you know,
Wes: I'd been playing around with my sidecar mesh setup on NixOS.
Wes: And especially for like the demos I was doing and testing it out,
Wes: it was pretty convenient.
Wes: You know, products like Tailscale or Netbird, they have this UX you get with
Wes: basically all you need is one secret, right? Like an API key,
Wes: and you can put that in somewhere.
Wes: And then when the client launches, it goes and uses that to an API and then can onboard itself.
Wes: And I was just like, well, I wonder if we could get that same workflow with Nebula.
Wes: So NACME, or ACME for Nebula, is my little attempt at that. It's super early
Wes: days. I need to do a bunch more testing.
Wes: Eventually, it'd be great to do renewals too. But right now,
Wes: it's at the initial testing stage of just being able to, you run a little server,
Wes: you configure an API key that's bound to certain groups, and then you have a
Wes: little client that can run and go get a new host onboarded.
Wes: And so if you configure that, I also want to set up a bunch of this stuff,
Wes: especially with nick's side but configure it to run before nebula uh you could
Wes: have it go item potently check to see if it needs to configure the the host
Wes: for the first time set up the keys and everything and then have nebula start
Wes: and be ready to go or at least that's the idea.
Chris: Yes this is really neat so it's automated certificate minting and it gives you
Chris: essentially like you said it's a like an api key type type exchange.
Wes: The goal too would be like it's sort of best effort right it's meant for like
Wes: home lab or you know stuff where you maybe you're not going to go the full like
Wes: crazy it automation it's.
Chris: Great for like a small business network like we were just setting.
Wes: And the whole thing with nebula right is like there are some
Wes: trade-offs you have less with those certs and the way it's kind of like more
Wes: like a jwt kind of style of trust you know you don't necessarily have this one
Wes: database that determines all of the truth right in a sort of less eventually
Wes: consistent way but the upside is nebula will just keep working right as long
Wes: as the certs aren't expired like there's no the control plane nothing happens nothing goes down what.
Chris: Freaked me out recently was the idea that maybe,
Chris: My Google account could be suspended because, so PayPal decided to flag my account
Chris: for like re-verification.
Chris: And it's a very complicated process. It's not just like a regular,
Chris: it's like a very in-depth, multiple types of documentation.
Wes: What did you do? Nothing.
Chris: I don't know, man.
Wes: That's usually what it is.
Chris: I don't know. But it occurred to me that if our Google Workspace account payment
Chris: got bounced because of PayPal, then I might not be able to authenticate to my tail net anymore.
Chris: And that freaked me out a little bit.
Brent: Yeah.
Chris: And that's where I was like, oh, the simplicity of these keys and the fact that
Chris: they'll work for as long as I issue these keys for is very reassuring.
Wes: And so my thought was this was like worst case, you know, like even if this
Wes: is down, you can still manually add things like this is just a convenience functionality
Wes: to make it easier to onboard hosts.
Chris: So this is not setting up routing. This is not setting up the networking layer
Chris: stuff. This is just keys to get you going to then build that stuff.
Wes: Exactly.
Chris: That's really cool. NACME. I like the name, too. It's very clever.
Chris: I think that could take off.
Chris: so uh we'll put a link in the show notes it's on wes's github n-a-c-m-e mit
Chris: licensed indeed indeed uh and uh version 0.10 was released just recently.
Wes: Yeah we'll see i should cut a new version it's move it's you know it's moving
Wes: fast and uh needs more tests.
Chris: I want to say thank you to our members and our boosters. Next week,
Chris: I'm calling it a birthday episode, boys.
Chris: I don't know what we're going to do, but check this out.
Chris: So Brian and I started podcasting in January, right around my actual birthday,
Chris: almost exactly 20 years ago to the day for next week's episode.
Chris: 20 years of podcasting on my birthday next week.
Chris: So if that's not a long-term commitment to the space, I don't know what is.
Chris: So send a birthday boost.
Chris: We'd love that. or become a member and use the promo code bootleg.
Chris: We have a couple of, well, we have a handful of redemptions left.
Chris: And, you know, you become a member at the party or a core contributor and support
Chris: the show at a great discount.
Chris: And if you'd like to get your company or product in front of the world's largest
Chris: and best Linux audience, show me an email.
Chris: Chris at jupiterbroadcasting.com. This space could be yours.
Chris: And thank you to everybody who supports the show. We greatly appreciate it.
Brent: Now, Chris, for the last two episodes, you've been talking about deploying,
Brent: you know, a bunch of new machines at a clinic, making sure you have the responsibility
Brent: for keeping your wife's business happy from a tech perspective.
Brent: And now you're putting up, you know, some infrastructure that you also need
Brent: to be working at all times. I would imagine now is the time to make sure all
Brent: that stuff yells at you whenever it's not in good health.
Chris: Right he.
Wes: Wanted to uh just leave a business card with my phone number on it but i didn't
Wes: think that was a great idea.
Chris: And part of me is like well we should do this why it's still fresh in the mind
Chris: because this stuff fades oh yeah and i thought well if we're going to do this for hadia's clinic,
Chris: um maybe i should do this for my own infrastructure and then i thought wouldn't
Chris: it be great if we could build something that if we ever did this you know on
Chris: occasion for audience members or whoever,
Chris: wouldn't it be nice if we could also offer to monitor their stuff and i could
Chris: build something that was pretty flexible like this and i'm sure you boys are
Chris: familiar with uptime kuma,
Chris: we actually use it at jb alex set it up for us a while ago and we like it it's
Chris: pretty simple and it alerts us when something goes offline like one of our websites
Chris: via telegram bot and it creates a nice dashboard super easy to self-host and
Chris: they have a demo i'll put a link in the show notes And it does monitoring for HTTP, TCP.
Chris: It can search website keywords, check WebSockets, do ping, check DNS records, stuff like that.
Wes: Very easy, straightforward to get going.
Chris: Yes. So obviously that was the first thing I decided because this is what I
Chris: have the most experience with.
Chris: And I thought this will be the way to go. Can you guess what the problem was?
Wes: Well, I think I know because it's been a longstanding issue we've had with the project.
Chris: In fact. What's that?
Wes: A lack of declarative configuration.
Chris: Yeah, man, it's just really gooey. Well, I had to set up something like,
Chris: you know, 45 hosts and services to monitor and I was sitting there creating
Chris: all the entries and I'm like, this is going to take me to two days to do.
Chris: And also because I wanted them to be actionable and all this stuff. So it's like, oh, my God.
Chris: And then I also wanted tiered escalating alerts. So first ping me via notify.
Chris: So because I have a lot of stuff coming in via NTFY these days.
Chris: And so that's sort of a feed of just checking on my systems.
Chris: And if I'm available, I check it.
Chris: But if I'm not presently thinking about my infrastructure, my systems, I don't check it.
Chris: So I needed something that would break through to Telegram and kind of, you know, kick it up.
Wes: Yeah.
Chris: And so I wanted and I wanted it based on different thresholds and trends.
Chris: And so when I started getting into I need to add, let's just say 45 ish host
Chris: combination services, maybe more and more complex alerting with a little bit more nuance.
Chris: I really started to hit two different walls. Like my GUI exhaustion kicked in.
Chris: Like if I was adding like five systems, I would just done it.
Chris: And then trying to get complex logic around alerts started to get frustrating.
Wes: Yeah, you combine those two. That could be pretty annoying, especially if you
Wes: have to like go configure it and then go run the test and then go see if it
Wes: did the thing you want and then go repeat that cycle a whole bunch.
Chris: Yeah, man. So I decided to break the seal on something that I have never,
Chris: I've never bothered learning. I've never wanted to embrace because of the overhead.
Wes: It's been a long time coming on the show. I'm excited for this.
Chris: It has. Ladies and gentlemen, I have finally deployed my first Prometheus instance.
Wes: Woohoo!
Chris: And of course, once you have a Prometheus instance, you want pretty dashboards
Chris: and you want all the details.
Chris: So along with that, I have also finally deployed my first Grafana instance.
Wes: Yeah, something tells me you really deployed the Grafana and then you just got
Wes: the Prometheus, so you had something to fill it.
Brent: It feels like that deserves a round of applause, really, there,
Brent: Chris. Give yourself, yeah.
Chris: Thank you.
Brent: Everybody, yeah.
Chris: I mean, uptime Kuma is good, but I needed a little bit more than that.
Chris: And I also have kind of a complex situation that I thought Prometheus was a
Chris: little better at solving.
Chris: I have set up a federated configuration.
Chris: And I had a problem where I have my Odroid on my home lab that doesn't have
Chris: a lot of available resources. as it has some, but now it's doing frigging a bunch of other stuff.
Chris: And I'm on LTE on two different ends of the connection, not all of them,
Chris: but two different ends are on LTE networks.
Chris: So they're slow and I can't just be blasting a bunch of data.
Chris: My original idea was I'll set up a central VPS connected to all the mesh networks,
Chris: and then it will monitor everything. And I just go to the VPS dashboard.
Chris: But then I actually started running the math on that. And I realized the overhead
Chris: would be somewhere between 40 megabytes to 100 megabytes a day,
Chris: best case, plus the overhead and latency it adds and slows down the whole LTE
Chris: connection while it's doing that stuff.
Wes: And that's only going to grow as you add more stuff that you monitor, surely.
Chris: Exactly. So the solution was a federated setup where I had local monitoring on my home lab system,
Chris: monitoring on a VPS, and then light remote monitoring on my wife's clinic network.
Chris: And I set up a Prometheus system with a black box exporter that does some additional
Chris: like HTTP, TCP, ping checks, and can do some API authentication for me.
Chris: And that all feeds into Grafana to give me dashboards.
Chris: And then that all talks to alert manager. So let me zoom out.
Chris: Prometheus is running on two different systems. plus a little Prometheus client
Chris: is running on my wife's clinic and I have a Prometheus integration now running on Home Assistant.
Wes: Oh yeah.
Chris: So this is collecting.
Wes: You knew that was coming.
Chris: Yeah, buddy. And that's really useful. I could talk more about that.
Chris: But the Prometheus agent is essentially collecting all of the metrics,
Chris: the CPU, the disk usage, and it allows me to trend and alert on these over time.
Chris: And with Home Assistant, I'll just mention this quickly, that integration is
Chris: pretty awesome because you can export a lot of different things from automation
Chris: details, log, all these types of things.
Wes: I mean, it basically has a whole bunch of stats in its own stats engine to begin with, right?
Chris: Oh, yeah.
Wes: You can kind of just dump those out to Prometheus.
Chris: It's a sensor machine, yeah. And so the way I use that now is I am using,
Chris: via Prometheus data export into
Chris: Grafana, I have dashboards on how long my different climate entities run.
Chris: So since it's winter, we have some electric heat out in different areas,
Chris: and I want to make sure that they're not running excessive because that would
Chris: tell me the heater isn't keeping up.
Chris: I want to know how long we're running electric heat. I want to compare that
Chris: to our intake from solar. And I want to have all that on one dashboard.
Chris: And now I have just not only a beautiful display, but good historical data that I can work with.
Chris: And all of this is pulled in on my local Prometheus incident that's running on my home lab.
Chris: Then I have Prometheus running on the VPS. That's doing some remote checks to
Chris: make sure that the remote systems are up.
Chris: And it has some logic to understand that if the mesh network is down, don't freak out.
Chris: don't alert about every single host. You know, I wanted some of this tiered
Chris: logic in there so it knows if one network connection is down,
Chris: then everything that's on that network is down.
Chris: which is going to hopefully save me a lot of notifications.
Wes: Yeah, and you've saved yourself that first night, right? Where the network goes
Wes: down and suddenly everything's on fire.
Chris: And then everything it's observing about the VPS itself, the mesh networks,
Chris: and the wife's clinic is getting federated back to my Grafana instance.
Chris: So I have just one dashboard to view everything. And that's running on my local instance. Oh, man.
Chris: And the difference is, according to my calculations, boys, the difference is
Chris: about two megabytes a day of data usage. nicely done yeah that's.
Wes: At least a power of 10.
Chris: It's it was a nice little savings and it's less it's less running on my home lab or on the vps i do.
Wes: Like that you kind of went from zero to not uh you know.
Chris: Yeah unsophisticated setup i think the real deal breaker was,
Chris: everything could be done declaratively and i could do this sort of hybrid federated
Chris: setup and those flexible yeah coming together those two things like it's like
Chris: to create all the dashboards I didn't create a single dashboard in the GUI.
Chris: There's so many community examples that you can modify and get started with.
Chris: And then you drop them in a folder and that becomes a dashboard and Bob's your
Chris: uncle. You got all the things and bells and whistles you'd want.
Chris: And then I have alert manager, last piece.
Chris: I have alert manager running on the VPS and everything forwards to that.
Chris: So I only have to have one alert instance and it can communicate with notify.
Chris: It can communicate with telegram because I set up a little bot thing and all of that.
Chris: and it handles all that stuff. And it will also let me know if any of the remote
Chris: hosts are down or if any of the mesh networks go down.
Chris: And the results have been, there's about 31 services that I'm getting real-time
Chris: visibility into their performance and their metrics, all in beautiful Grafana dashboards.
Chris: And I also have real-time alerts anytime any service goes down.
Chris: And I'm now very carefully monitoring storage, which is very tight these days.
Chris: And so I have different thresholds for storage alerts. And I have different
Chris: thresholds for like if home assistance at 80% CPU for X amount of time.
Chris: And if it gets to 85 or 90 for X amount of time, we do different styles of escalation.
Chris: and then there's follow-up for when things recover i get a recovery alert so
Chris: that's all working really beautiful.
Wes: Yeah i'm curious how you tested it.
Chris: The entire well by shutting things down i did excellent all right docker compose down you know.
Wes: The next version of this is you give your agent access and you ask it to stop a random service.
Chris: Right just chaos monkey oh my god that'd be fun basically just give west your credentials,
Chris: yeah that's to most of it um but so i measured
Chris: the whole stack across all the machines 400 megabytes of
Chris: ram totally fine not even
Chris: making really an impact at all in my home lab system and i was worried because i'd heard some
Chris: stories about grafana but i really really
Chris: like the home assistant integration with prometheus it's
Chris: something else i've put off for a while if you have been tempted to
Chris: try this the insights are fantastic and then
Chris: the other thing that's really fun is frigate which
Chris: i also set up recently also has an
Chris: api where i can export all the
Chris: information to prometheus and grafana
Chris: so i have details on how my coral is doing
Chris: inference wise and my different cameras and their connectivity and i can and
Chris: their detection frames per second the camera's overall health when the automations
Chris: execute and how frequently for like arming and disarming the recording if they
Chris: are available at all is all coming into this dashboard so i have essentially
Chris: a camera health dashboard now.
Chris: It's so great because you have dashboards.
Wes: For your dashboards.
Chris: I do. And a lot of the RAR projects, it's great. Oh, I'm like,
Chris: I'm inception dashboards over here.
Chris: A lot of the RAR projects out there that we don't talk about also have APIs
Chris: and health API endpoints that also plug into this.
Chris: So you can get all that kind of information in there.
Chris: And it, it really struck me like how, now I get it, why people go through setting all this up.
Chris: And yes, it's a lot of YAML and all of that, but there's so many great examples out there.
Chris: And I now have an enterprise grade monitoring stack that I just never even thought I would get into.
Chris: Zero dollars spent, you know, one day I've set up really a lot of documentation
Chris: and maybe five to eight hours of fiddling to get it all working.
Chris: And it really makes me feel a lot better about my self-hosted infrastructure.
Chris: I have built quite a little empire now of things I depend on and my family depends
Chris: on and my wife and whatnot.
Chris: And I probably haven't been monitoring it as seriously as I should.
Chris: And I just thought that, eh, it's fine. But honestly, it does feel a lot better
Chris: because I'm getting insights before things go wrong.
Chris: Now I'm getting ideas of trends.
Chris: So stuff I know, Oh, this is actually something I need to address.
Chris: And I know something's a problem before the wife has to tell me it's a problem.
Chris: So I know if jellyfin isn't working for some reason or et cetera,
Chris: or I know if her system isn't backing up before she has to tell me.
Chris: So if you have a self-hosted setup, definitely uptime Kuma is a pretty good
Chris: starting point. But if you have the ambition and the time Prometheus and Grafana
Chris: get a double recommendation from me, I waited way too long.
Chris: There is a real learning curve, but the visibility you get out of it, it's just so useful.
Chris: Plus, like I've revisualized my entire infrastructure once again.
Chris: All the stuff I've built, like I know how many services and how many hosts and
Chris: I know how they're doing.
Chris: I have a much more concrete picture once again of everything I've done over
Chris: the last five, six years.
Wes: Well, I think as you're discovering, right, a lot of things publish Prometheus metrics.
Chris: Yeah.
Wes: So actually, as Tiny points out in our chat room, Uptime Kuma itself can publish to Prometheus.
Chris: Yes, I thought about that for a second.
Wes: And Nebula does as well. I don't know if you've integrated that yet.
Chris: Hmm. Hmm. Hmm. Should look to see what Tailscale can do too,
Chris: but that would be really useful.
Chris: Yeah, I'm pretty happy with this. I did have to kind of, you know,
Chris: be tight on the retention.
Chris: So I don't have, I think I have like 30 or 60 days. I couldn't go crazy because
Chris: of storage constraints.
Chris: And I did, I will admit, Wes...
Chris: I feel a little guilty, but I did a lot of it with Docker. And the reason was,
Chris: and it's always what gets me,
Chris: is a lot of the community add-ons and plugins all assume you're using the Docker instance.
Chris: And it's one of those things where it's like, well, yeah, I could come up with
Chris: a way to declaratively do that with Nix every single time.
Chris: Or I just start the container and it works, right? And so all of the OS-level
Chris: stuff to make it work is all declarative, obviously.
Chris: But then, like, Prometheus, Grafana, and Alert Manager, they're all in,
Chris: like, one big Docker compose.
Wes: Ah, yeah.
Chris: I felt bad about that.
Wes: No, I mean, as long as you're dabbling, I think, a little bit and seeing that
Wes: Nix can play very nicely with these things.
Chris: And it was a no-brainer for the Nginx. Like, some of the stuff,
Chris: I had to add some more stuff behind a reverse proxy and get SSL certs for it
Chris: that I didn't have before just so I could keep consistent.
Chris: So I wasn't doing IP for some stuff and name for other stuff.
Chris: I wanted names for everything. So I just got all the DNS set up anyways.
Chris: And that was really, really nice to just configure all of the Nginx stuff via Nix.
Wes: And, I mean, it should be easy, right? Like any of your Nix hosts,
Wes: you can just add in a bit to have it run a Prometheus exporter for the node
Wes: and send the node metrics too.
Chris: And I know between the three of us,
Chris: I'm also the Git Luddite, but the other thing I liked and appreciated about
Chris: having declarative config for my monitoring setup is that I could use Git to manage that.
Chris: And if I F it up in the future, you know, I've got some recoverability there.
Chris: So the more I can define via text, I feel like the safer it is to experiment
Chris: and the easier it is to roll back.
Chris: And so that's, I just, that's something I just, food of thought.
Chris: One of the lessons I took away from the setup is that gave me a little bit of
Chris: comfort level to experiment with something that I didn't fully understand yet.
Chris: that has a big learning curve.
Wes: And now that you have it captured, you can watch as it evolves too,
Wes: right? So you have rollbacks, but you can also then go, well,
Wes: what did I tweak? Was that what broke it?
Chris: And I did catch a couple of things already, I will say too. I had some choral
Chris: performance degradation issues that I tracked down to Wi-Fi actually,
Chris: but it started a whole process of like breaking down where the problem was at.
Brent: What is the maintenance going to be like for this? Like how likely are you if
Brent: you're just booting up a new container for some kind of new system you're playing
Brent: with that may last a long time are you likely to have that to the dashboard here in
Brent: the process of setting it all up or is this just going to fade a little bit
Brent: and then you'll have the problem where you've got a bunch of services that aren't
Brent: actually integrated in this how do you think that's going to go.
Chris: That's a great question that's a great question um because
Chris: my current thinking so i haven't thought a lot about that because
Chris: i've been thinking i need to really freeze the state of my odroid and i need
Chris: to stop adding stuff because every single effing thing i add i need to migrate
Chris: to a one liter pc one day and i just went and made it a lot more complicated
Chris: right so i have been thinking i was actually going to hit the pause button for a while,
Chris: until i get to that migration but you raise a good point of like what if i find
Chris: it in my new favorite self-hosted app and i get it all set up do i throw it
Chris: into the monitoring system and i I think my answer for that is I don't know if you guys do this.
Chris: This is probably just a Chris thing, but I have, I have two tiers for self-hosted
Chris: applications. You know where I'm going?
Wes: I think so.
Chris: And it's like, if I'm just playing around or if it's just something for me or
Chris: really, you know, maybe I want to, I don't know, for whatever reason,
Chris: I'll just put it on a port and I'll just go to the local IP and I'll put the
Chris: colon in the browser like an animal.
Chris: but then when it becomes like oh this is something that's serious then i go
Chris: ahead and i set up the reverse proxy and i get an ssl cert for it and i even
Chris: register a dns name because.
Wes: Some stuff just doesn't survive to that level you know.
Chris: Right so i think that is the threshold in which i now need to say and i'm going
Chris: to add it to the monitoring system.
Wes: There'll probably be a natural point where you realize that it's down and you
Wes: wanted it to be out but then oh right okay.
Chris: Better ahead that other people do this let us know send a booster contact form
Chris: and let us know if you guys do this I just wonder if it's a sus thing.
Chris: Because I've definitely seen some other folks that we know be like, oh, why do you do that?
Chris: I think my other question I have for you guys is, do you honestly think I did
Chris: it overkill? Do you think I went too far with this?
Chris: I think, is that implicit in your question there, Brent?
Wes: How many more pie holes would you have run?
Chris: Yeah, right?
Brent: I guess it is a little bit, because there seems like there's such a gap between
Brent: where you started solving this problem versus where you ended up.
Brent: but as long as it's well your requirements were quite specific as well right
Brent: so I think had you loosen those requirements especially with the notifying you
Brent: know the tiered notifications,
Brent: that probably would have made this much much easier for you however you probably
Brent: would have hated your life every day of the monitoring system after that so
Brent: I think if you're looking long
Brent: term which it sounds like you are with this kind of monitoring then I'm,
Brent: It sounds like you made the right choice, because your digital life and your
Brent: maintenance of that life will just get better.
Wes: You know, I saw this morning over on the BcacheFS subreddit,
Wes: a couple different folks working
Wes: on both for Telegraph and then for Prometheus, BcacheFS collectors.
Chris: Cool.
Wes: So a little incentive for you in the future.
Chris: So you don't think I've overdone it?
Wes: Not if you install BcacheFS, no.
Chris: I mean, if I could pull BcacheFS metrics in, that would be pretty neat.
Wes: I think that justifies your setup.
Chris: But you see, I'm a little worried about that. But because, again,
Chris: it feels it's like one of these things where when it's when it's all declared,
Chris: I can kind of re pick it up and read through it and understand it.
Chris: Where when it's the GUI, I have to really dig through it and really,
Chris: really, really have to grind again to get it figured out again.
Chris: I don't know. Maybe that's just me convincing myself.
Wes: Well, no, and I mean, there's a lot of pieces, but I think one of the benefits
Wes: of a Prometheus-style setup is, right, you're building on top of time series,
Wes: and that's a fairly universal format for a lot of things.
Chris: Yeah, yeah, yeah.
Brent: Wes didn't answer your simple question. Do you think that he went overkill?
Wes: No, I mean, how many services did you say you had?
Chris: Around 37, and then, you know, five or six hosts in there, something like that. Yeah.
Wes: I mean, it seems like maybe if anything, it's more of like, um,
Wes: like a reckoning with the level of infrastructure you're already providing and
Wes: that it deserves a similar class of monitoring.
Chris: Yeah. And it's like, it's not only is it running a clinic, right?
Chris: But the home assistant stuff is really integrated into the function of the home
Chris: to a degree that like prevents freezing and, and, and other damage occurring.
Chris: So it's pretty significant.
Chris: Uh, yeah, I think I probably was underdoing it. Yeah, I think I might've been.
Chris: And yeah, it did need, it did need a better solution. I would be curious to
Chris: know if maybe there was a better way to go though.
Chris: So, and how people are doing it. Always love to hear that.
Chris: I'm not opposed to coming up with a better way. Could always make for a good segment.
Chris: All right. Well, check this out. Linux Unplugged has been here for over 12 years.
Chris: And I think I figured out why.
Chris: Right. We focus on a few things. And I think this is one of our strengths.
Chris: Real use cases for Linux. We can get value out of it.
Chris: Free software that's actually free. And we talk about the differences there.
Chris: And we try to focus on self-hosting that's practical and just works and not
Chris: like the hype stuff. And I think you'll also find that we have honest conversations
Chris: that try to help you make sense of the big shifts in the Linux landscape.
Chris: And you can look over the 12-year history of the show.
Chris: And we don't chase the outrage. We don't chase the hype. We don't go for the drama clickbait.
Chris: We just try to focus on the signal there. And so when you support the Unplugged
Chris: program, you're keeping something that's a bit rare alive.
Chris: It's focused, thoughtful Linux podcasts that tries to stay in its lane,
Chris: respects your time, and treats the community like adults.
Chris: And that's probably not as common as it should be.
Chris: So this here show, it runs on value for value, time, talent, or treasure.
Chris: listening and sharing the show, spreading the word, time, participating in the
Chris: community, helping create maybe show swag.
Chris: That's time. It could be a little bit of talent in there too, right?
Chris: Also your feedbacks, corrections, things like that. Also helpful.
Chris: And of course, treasure.
Chris: Boosts, membership, direct support at meetups, all of those things make a big
Chris: difference right now. The reason why I'm talking to you right now is because
Chris: we don't have a sponsor for this slot.
Chris: So every bit helps the show continue and ideally thrive and grow.
Chris: Better coverage, bigger experiments, more room to explore what's next before
Chris: it's obvious without all the hype. I mean, look at the history of the show.
Chris: So if it's helped you understand Linux better or avoid bad tech decisions or
Chris: feel more confident about running your own systems, consider supporting the
Chris: show to keep that going. You can send us a boost.
Chris: You can become a core contributor or a Jupyter.party member.
Chris: Or, of course, if you use the promo code bootleg while it lasts,
Chris: you get it at a great price.
Chris: That's linuxunplugged.com membership to support this show directly or jupiter.party.
Chris: You get the perks and you keep the show going.
Chris: And of course you can send us a boost to support each episode directly.
Chris: Thank you everybody who does that. It makes all the difference.
Brent: Well, AJ wrote in this week, long-time listener, Mobile Linux Survivor reporting
Brent: in here. Hey, Chris, JB Crew.
Brent: Been a long-time member, watching and listening since about the Matt Hartley
Brent: era. So not quite a Lunduk graybeard, but almost.
Brent: I'm extremely jaded about Linux phones, and for some good reasons.
Brent: I backed the Librem 5, lawyers got involved there, and still no phone.
Brent: I owned a PinePhone and a PinePhone Pro, which were underwhelming at best.
Brent: So I assume we're all pretty burned on mobile Linux by now.
Brent: But then I heard about the FLX1. Last spring, I learned about it.
Brent: It's the FLX1 from Fury Labs.
Brent: Didn't expect much there, but I believe in the idea of mobile Linux.
Brent: So I backed it. A month or two later, though, the FLX1 was cancelled.
Brent: So cue that purism-era PTSD.
Brent: But here's where it gets a little weird in a good way.
Brent: Fury Labs announced a replacement device, the FLX1S for slim.
Brent: It offers refunds or a spot in the new queue. And so I stayed in,
Brent: fully expecting another disappointment.
Brent: But then they delivered. My FLX1S just arrived January 2nd, and I've been daily
Brent: driving it since that weekend.
Brent: Not testing, not tinkering, but daily driving it.
Brent: So here's a little report. Does it actually work? Yes, calls seem to work.
Brent: SMS and MMS, mobile data, GPS, most Bluetooth works.
Brent: Many Android apps also via WayDroid.
Brent: They use a fork called Andromeda. The battery lasts about a full day with normal
Brent: use as well. The software stack, as I understand it, is FOSH,
Brent: customized Debian base built on the Helium project.
Brent: Those details might be slightly off, but that's the gist I understand.
Brent: Is it perfect? Nope. Is it real? Shockingly, yes, it's real.
Brent: There's a compromise, of course, but they're shrinking pretty quickly.
Brent: Some issues get fixed day to day, not month to month.
Brent: I even submitted a bug fix that'll ship by default on the next release.
Brent: that alone felt wildly refreshing compared
Brent: to my previous experiences so why am i
Brent: reaching out well i have zero affiliation no financial interest and no incentives
Brent: i just genuinely am a happy customer which feels rare enough to mention fury
Brent: labs has restored some of my hope in mobile linux they're active in the matrix
Brent: room with a small but engaged community and i really think you should get your hands on a device,
Brent: maybe invite someone from that team onto the show and talk about what they did
Brent: differently this time around.
Chris: Oh, that's an interesting suggestion. Boy, I would appreciate a contact if you have one, AJ.
Chris: That is a good report, right? Isn't it nice to hear that possibly a Linux phone
Chris: out there that people are happy with and gets the basics done?
Chris: And I think about how much I could do in a web browser if I didn't have an app.
Chris: And I start to think maybe it's not crazy maybe it's maybe the dream is possible
Chris: aj makes me believe again faith restored thank you aj appreciate that report
Chris: great example of value contribute to the show right there with a in the field
Chris: report another great example of course is la boost,
Chris: And Daja's back with a row of McDucks as our baller booster this week.
Chris: And he writes, I just wanted to share a little Linux and self-hosted success
Chris: story. Oh, here we go. I love these.
Wes: Yeah.
Chris: Thanks to what I've learned from the crew, our community, and our community
Chris: over the last six years, I was able to migrate off of a GCP,
Chris: oh, a Google Cloud, to a combo of our own Infra and Colo, resulting in monthly
Chris: savings of this boost amount.
Brent: Wow.
Chris: But it was in dollars instead of sats that's a big number and all foss you guys
Chris: are awesome thank you love to hear that that that really you know that kind
Chris: of stuff makes our day also regarding the on-site you guys did at his clinic
Chris: it was super interesting and it's always those small gotchas that get you.
Wes: Ain't that the truth it's always networking.
Chris: That is such a great boost also i mean because it's just really great to hear
Chris: that we made a little difference there but also uh appreciate the signal on
Chris: that type of topic it's the first time we've ever done something like that so
Chris: we always appreciate the feedback.
Wes: Optical gre comes in.
Chris: With 21.
Wes: 703 satoshis,
Wes: i forgot where i live so i need west to check his map for me and then relay
Wes: that information to brent so he can help me with the many unfinished projects
Wes: if he ever works his way through here.
Chris: Uh-oh here we go don't tell me with.
Brent: A good time.
Chris: Did you uh yes actually oh i.
Wes: Keep it in my back pocket.
Chris: There it is nice okay watch out watch out i don't want that on camera because
Chris: that did sharpen the edges yeah all right do we have a location yes.
Wes: We do uh 21703 looks like,
Wes: A postal code from Frederick County, Maryland.
Chris: Oh.
Brent: Wow.
Chris: There you go, Brent.
Brent: Well, what you have to do with these messages is also tell me,
Brent: you know, some temptations as to why I should come through the area. Well, mostly food.
Wes: Well, you swing by on your way to the Capitol.
Brent: If they have gluten-free pizza, then you got me.
Chris: Just, he has a few allergies he'd like you to know about. If you consider those
Chris: and work them into your boost, there's probably a good chance you'll stop by.
Chris: Oh, and if you have a plug, outdoor plug, he can make 120 work.
Wes: If you have good cat snacks, that's usually a big help.
Chris: Liking cats helps, too. That is true.
Brent: Well, Gene Bean sent in, this is just a little row of ducks, 2,222.
Brent: He says, can you share that Nix config for the clinic? I'd love to get some ideas off there.
Chris: We could, yeah. I think we would probably just want to do a quick sanitization
Chris: check, since it is for a clinic and all of that. But I think we could give a
Chris: look at that after the show.
Wes: Totally.
Chris: And if it passes the sniff test, we'll just put it in the show notes for this
Chris: episode. Does that make sense? Yeah.
Chris: So I guess the answer is, if the answer is yes, it'll be in the show notes,
Chris: Gene Bean. Good question.
Wes: Show notes for 651.
Chris: You know, I've talked, yeah, linuxonplugged.com slash 651. I've thought about this.
Chris: There's not a lot that's going to be revealed because anything that's like a
Chris: secret gets stored outside the main config that goes in the repository.
Chris: But, it is the type of thing that if I had access to someone's network,
Chris: I would use the hell out of this to get everywhere I wanted to go.
Chris: And I just, like, this for me, when I, I don't mean to be this guy,
Chris: but like when I was hired to do penetration testing, this, I would have loved
Chris: a map like this. I would have.
Chris: This would be like, oh, you just gave me the job for easy, free, you know, basically.
Wes: First I'll own his couple of pie holes and then I will.
Chris: Right. It's really, well, it just gives you time to research and it's easier
Chris: than ever to drop these configs into a machine and say, hey,
Chris: machine, what's the first thing I could pick on, right? Like you got to think
Chris: about the tools that are available to people now.
Chris: And so it crosses my mind that there is a level of information that's being exposed.
Chris: And so I have some consideration there. But it relies on breaking into the infrastructure.
Chris: And then most of what gets exposed is just internal non-routable IP addresses
Chris: and things like that, or perhaps where secrets get stored and whatnot.
Chris: But it's something I think about, and I would love the audience's thoughts about it as well.
Chris: And if you guys are concerned, I know there's a culture around sharing your
Chris: Nix configs and your Ansible configs, and I like that. And I use that.
Wes: We've benefited a lot from it.
Chris: And I know I could do a sanitized version. So, I mean, maybe we'll try that.
Chris: But then I'm not – I prefer if I'm going to put it up on GitHub,
Chris: I would prefer to actually use it.
Chris: You know, like – and then it's – anyways. Good question. Check the show notes,
Chris: Gene, and I would love people's feedback on that. Thank you very much.
Chris: Is it my turn now? I don't remember. Sifeseeker comes in with – I got all distracted.
Chris: Comes in with 2,500 sats.
Chris: Hi guys, I've been kicking around the idea of a NixOS router,
Chris: and an example would be great, Leo, there you go.
Chris: Would you be willing to share your config? The router config is interesting, right? It's in there.
Chris: I will also mention that Bearded Tech in our community has a really cool NixOS-based router project.
Chris: And if you're actually thinking about using it for your home router...
Wes: Yeah, this might be something to consider.
Chris: That might be something worth looking at.
Wes: We were kind of doing a bunch of stuff all at the same time,
Wes: so we took a peek, but we kind of wanted to start a little more minimal and
Wes: work our way up. But it looks great, especially if you just want,
Wes: like, a standalone router that is NixOS powered.
Chris: Yes. And you've got to remember, we were building something that was a VM first, VM server first.
Wes: It wasn't going to be the router first.
Chris: Yeah, yeah, right. Exactly. Yeah. All right. But, yeah, all right.
Chris: We'll take a look at that, Cypher. Thank you very much. I guess there's some demand.
Wes: Hybrid Sarcasm comes in with 10,000 SATs.
Wes: You asked for some feedback regarding actual budget.
Chris: Yeah.
Wes: It's been a pretty good replacement for you need a budget with spousal approval so far.
Wes: That's $9 a month I don't have to spend on a cloud service.
Wes: You'll also appreciate that the actual devs have a sustainable funding model
Wes: for their core contributors, and they are looking to expand it to others.
Wes: And then we've got a link we'll put in the show notes.
Chris: Oh, interesting. I'm looking. I'm trying. There it is. Yeah.
Chris: So they have in their documentation here. Hmm.
Chris: Thank you. I did not know that. That's a good little bit of information there. I appreciate that, Mr.
Brent: Zark.
Wes: I appreciate the report.
Brent: Are you saying those are the actual, actual devs?
Wes: No, the actual, actual, actual.
Chris: No, yeah.
Brent: Oh, actually, okay. Well, adversaries came in with 8,441 sats.
Brent: Adversaries responding to our question last week, Chris, you were asking for
Brent: Wi-Fi analyzers for Android.
Brent: adversaries says unify makes a great wi-fi analyzer
Brent: app called wi-fi man it doesn't require unify
Brent: gear to work it just uses your phone's radios and
Brent: i can second this one this is the application i've been using for about the
Brent: last year i used it just this week to that's good to know hey hey you didn't
Brent: give me time to answer your question nor did you ask me while you were working
Brent: on this project so oh wow dude anyways it's fantastic i use it just this week uh to I fixed my parents'
Brent: Wi-Fi, and it's got some sweet features.
Brent: So I would say put this on your phone, play with it.
Brent: It's pretty amazing. It took me a few uses to discover all of the different
Brent: crazy features that are hidden in it. It's really quite good.
Chris: This makes you the buddy that shows up with a water hose after I put the fire
Chris: out. You realize that, right?
Brent: You're welcome.
Chris: All right. That makes you – that's what you – I.
Wes: Could start a new fire.
Chris: No doubt we will. No doubt about that at all. Anonymous comes in with 2,021 sats.
Chris: No message, just value. Thank you very much. And then Tomato comes in with a row of duckles.
Chris: And writes, I loved this old network segment. I'd be curious to hear if Brent
Chris: started to automate his van yet.
Chris: Mine is completely unautomated. I'm not sure where to start.
Chris: Oh, well, he needs to start with sensors, right, Brent?
Brent: Sensors is great. Yeah, I did the opposite this week and pulled my lithium batteries
Brent: out of my van. So I've unautomated everything only because it got really,
Brent: really, really, really cold.
Brent: And this is not very good for them. So I feel like this week I went backwards.
Brent: but I'm going to kind of build all the automation here in the workshop,
Brent: just as winter's here, then I can just plunk it in the van, you know?
Brent: But I would say, Chris, you've got much more opinions on this than I do,
Brent: but I would say start with the problems that you feel like you want to solve
Brent: or have visibility into, right?
Brent: If you want exterior temperatures versus interior temperatures,
Brent: and that's really important to you, start there.
Brent: If you want to, I don't know, have some other solution to a problem,
Brent: That's always the best place to start.
Chris: And plus one sensors. And then if you really want to see what's capable and
Chris: way, way far out there, check out smartyvan.com.
Chris: He's also a YouTuber and he has created some really inspired automations around van living.
Chris: I mean, absolutely high end tech stuff that you could build from for years.
Chris: He's also released some code. He has examples and automations and video tutorials
Chris: and all of that. So it's smartyvan.com, S-M-A-R-T-Y, van.com.
Chris: And you get some good inspiration there.
Brent: I think my next step, if you're curious, is likely getting some visibility and
Brent: automation around keeping these batteries charged and healthy.
Brent: One of them is how to keep them warm while they're charging in the winter or,
Brent: you know, in the after winter season.
Brent: So that's the main problem I have that I'm going to solve that will get me,
Brent: you know, some open hardware experience and also diving more into what Home
Brent: Assistant can do to automate all this. So that's my next step.
Brent: But write in and let me know what problem you're solving.
Chris: I would also add, like, if you want something to rabbit hole into for a while
Chris: before you get into all this, go learn the ESP platform. So that is a skill
Chris: that will pay dividends years.
Brent: And relays, relays.
Chris: Yeah.
Brent: ESVs and relays.
Chris: And the little sensors and there's kits you can get on, you know,
Chris: the big box websites for super cheap and all of that.
Wes: And buy some epoxy too.
Brent: While you're at it.
Chris: All right. Thank you, everybody who boosted the show. We do appreciate you very much.
Chris: And, of course, shout out to our SAT streamers as well. We had 26 of you stream
Chris: sats, and collectively you came in with 26,866 sats, which does technically
Chris: make our streamers the baller booster again this week. Thank you, everybody.
Chris: When you combine that with our boosters, we raised a total of 109,354 sats.
Chris: Pretty humble, but we're very appreciative and gives us an opportunity to make
Chris: our birthday episode a banger.
Chris: There's real, real easy ways to boost in these days. Fountain FM is making it
Chris: easier and easier, including making it just all kind of dollar-based, simple stuff.
Chris: And, of course, there's the entire awesome self-hosted infrastructure.
Chris: You can find that when you go
Chris: to newpodcastapps.com. You'll go down that rabbit hole. You get AlbiHub.
Chris: It's really awesome. And, of course, we have the membership program,
Chris: linuxunplugged.com slash membership, or jupiter.party for the whole dang network.
Chris: All right. Would you guys like a few picks?
Wes: Yeah, what'd you get in your bag today?
Chris: Well, you've heard me mention my Hypervibe, which is a NixOS-based Hyperland
Chris: desktop and still rocking it.
Chris: Got it running on three machines these days and it's in a great state and I like it a lot.
Chris: But perhaps you are an Arch person.
Chris: Well, Rich Arch has a Hypervibe spin.
Chris: They say we at Rich Arch Project are re-releasing our Hypervibe spin.
Chris: We now have taken the Hypervibe configs and enriched them with the Noctelis shell on Hyperland.
Chris: You can try it in a VM, and it includes some screenshots, or some instructions
Chris: and a screenshot, which we'll put a link to in the show notes.
Chris: It's better looking than the way I have it configured. I'll tell you that. It's really nice.
Wes: So now you're going to re-Nixify, Hypervibe-ify, the rich Arch Hypervibe-based config?
Chris: I like the way you think. You basically start with a base Linux,
Chris: or a base Arch ISO, just base install.
Chris: And then he has some kickoff scripts that you can curl onto that basic system
Chris: and turn it into a Hypervibe desktop based off of what I set up to run on Nix.
Chris: But with Arch. Kind of neat. Thank you, Rich, for sending that in.
Wes: That's great.
Chris: It's beautiful. It's beautiful. And then I've got one that.
Wes: This looks really nice.
Chris: This is nice to see. It's good to have another one of these.
Chris: I've talked about Junction before. Now we're going to talk about Switchyard,
Chris: a modern rules-based URL launcher that replaces your default browser.
Chris: So wrap your noodle around this. Instead of having one browser as your default,
Chris: you set switchyard as the default browser.
Chris: And then when you click a link, it brings up a little window and it lets you
Chris: choose which browser you want to open in.
Chris: But on top of that, they have added a really nice graphical interface,
Chris: a GTK graphical interface, where you can have rules to just automatically send
Chris: some URLs right to a particular browser.
Chris: And this is exactly how I work. And this is why I really appreciate this Because
Chris: there's some stuff I always open in Firefox.
Chris: There's one site and only one site I use Brave for right now.
Chris: And then there's other stuff I open in Zen.
Chris: And it's very, it's always that stuff. So this is really, really great.
Chris: It's a super fast app and it has a simple configuration.
Chris: If you do want to do it by text, they have a flat pack and a Nixflake ready
Chris: to go. So I thought I might get your approval on that one too.
Wes: Oh yeah, absolutely. And it's written in Go, GPL 3.0.
Chris: Yeah. So this is so nice if you do live the multi-browser lifestyle.
Chris: And I do. And I probably would say Firefox is 90% everything.
Chris: but then there's those, or maybe like if I'm going to do a Google Meet,
Chris: I might actually do that in Chrome.
Chris: And I maybe don't use Google Chrome for literally anything else on that machine,
Chris: but I use it for Google Meet.
Chris: And it's nice to have something. I just click a link and switch art, it'll send it to that.
Chris: But if I don't have any rules set, it gives you a really lean,
Chris: mean, fast UI and you big icons and you just select the browser you do want
Chris: to open and it sends a link to that browser. So it's, ha ha.
Wes: I think especially, right, they kind of keyed in on work or other stuff,
Wes: but even just maybe you're doing, you're in a mode where you're doing show notes.
Wes: You want to make sure you, you know, you're clinking some of the Brent sends.
Wes: You want it to open the right spot.
Chris: Yeah. Something that's in private browsing mode for sure.
Wes: Well, yeah, no JavaScript either.
Chris: Containerized.
Brent: Definitely.
Chris: All right. Well, that's pretty much the end. I just want to remind everybody
Chris: that the meetup page is up.
Chris: If you're going to be in the Pasadena area around March 5th,
Chris: we'll be at Planet Nixon scale and hanging out with our buddies from Phlox.
Chris: We'd love to see you there. Meetup.com slash Jupiter Broadcasting.
Chris: We'll get the details locked in soon for all of that.
Chris: I'm very much looking forward to it. It's going to be nice, especially as it's
Chris: very cold right now. I'm picturing the nice sunshine. It's a beautiful time to be in Pasadena.
Wes: Seeing a wonderful audience.
Chris: Mm-hmm. Mm-hmm. Good crew down there, too. Wes, is there some pro tips we could
Chris: leave with them? you know things
Chris: where they could get more data more information around the show like.
Wes: Some sort of enriched xml file.
Chris: Yeah something links.
Wes: To uh text and json files.
Chris: Could be yeah.
Wes: With like chapter.
Chris: Information and transcript information yeah could have all of that.
Wes: Like an srt for however you want to consume this.
Chris: It's that if you've got a um podcast client there's more and more of them that
Chris: supports transcripts we have that in the feed for you and of course if you have
Chris: a podcasting toodle client you get all kinds of stuff like the cloud chapters
Chris: like the live item entry pending information and a whole bunch of good stuff.
Chris: And of course we are live.
Chris: Yeah, we love it. If you make it a Tuesday on a Sunday, join us Sunday at 10
Chris: a.m. Pacific, 1 p.m. Eastern, jupiterbroadcasting.com slash calendar for your time.
Chris: If you want to, I don't know, read about what we talked about.
Chris: You want more show? I don't know.
Chris: Linuxunplugged.com. This was episode, geez, 651. So Linuxunplugged.com slash 651.
Chris: We get together every Sunday with our Mumble room. That information's on our website as well.
Chris: You can get in there, get a low latency Opus stream. We tell you about it. Try it out.
Chris: And last but not least, we have that Matrix room going 24-7.
Chris: You can find details to that. It's a great community. And if you're already
Chris: in the Federation, why not join us?
Chris: Thanks so much for joining us on this week's episode of Unplugged.
Chris: See you right back here next Sunday!