Clips: Enhanced BPF is another hotness of Linux for the last couple of years.
Clips: And when the patches were first added to Linux, the lead developer,
Clips: Alexei Storitov, said this allows you to do crazy things.
Clips: This is normally the words you don't tell Linus when you want him to accept
Clips: patches into the kernel, but fortunately the patches were accepted.
Clips: Enhanced BPF puts a virtual machine in the kernel that we can program from user space.
Chris: Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.
Wes: My name is Wes.
Brent: And my name is Brent.
Chris: Well, hello, gentlemen. Today, we're digging into a superpower that is inside
Chris: all of our Linux kernels.
Chris: We're going to talk about how eBPF works and how anyone can take advantage of it.
Chris: Then we're going to round out the show with some great feedback,
Chris: some picks, and a lot more. It is a special out-of-time episode.
Chris: As you listen to this right now, we are at Planet Nix and Scale.
Chris: So this is an episode in between.
Chris: You're going to get all our Planet Nix and Scale coverage coming soon,
Chris: but we wanted to take a moment in between episodes and do something kind of
Chris: fun and really dig in and get technical.
Chris: But first, I want to say a big good morning to our friends at TailScale.
Chris: TailScale.com slash unplugged. They are the easiest way to connect your devices
Chris: and services to each other wherever they are.
Chris: It is modern networking. It's a flat mesh network that is protected by...
Wes: Waggaard.
Chris: That's right. And it is so fast, so quick to get going, and it gives you superpowers.
Chris: Not only do you get a flat mesh network across complex networks,
Chris: so maybe you have multiple data centers or VPSs, or you've got mobile devices,
Chris: or you've got double carrier grade NAT, it'll smooth all of that out.
Chris: That's fantastic. But then there's also a whole suite of tools that make it
Chris: really convenient to use.
Chris: Sort of like AirDrop for your entire Tailnet, including like your Android device
Chris: and your Linux device, so you can send files around.
Chris: They will manage your SSH keys through your Tailnet for you,
Chris: so you can just log into all your individual devices. You don't have to manually
Chris: copy keys around like an animal.
Chris: And they also offer more advanced features, so you can set up ACLs and really
Chris: manage the system and lock down only certain things to certain people.
Chris: And when you try it now, when you go to tailscale.com slash unplugged,
Chris: you get it for free up to 100 devices and three users.
Chris: No credit card required. I mean, you can really cook with 100 devices,
Chris: and then maybe you'll discover it's great to bring to work too.
Chris: Thousands of companies like Instacart, Hugging Face, Duolingo,
Chris: Jupyter Broadcasting, and many others use Tailscale, and we love it.
Chris: Try one for yourself. Go get yourself a little Tailscale right now.
Chris: You're going to love the way it tastes, and you're going to love how easy it
Chris: is to get going. If you've got five minutes, you'll probably get it running on three devices.
Chris: I have no inbound ports on any of my firewalls. Talescale.com slash unplugged.
Chris: Well, like I mentioned, we are at Planet Nix and Scale right now.
Chris: But we did think it was sort of a perfect out-of-time episode because we really
Chris: first got excited about eBPF at Scale back in 2019?
Wes: Yeah, a million years ago.
Chris: And it was a great presentation that really just brought home to us how powerful this was going to be.
Wes: Yeah, you heard from the man himself, Brendan Gregg. observability,
Wes: performance, tracing, guru, eBPF was still kind of new then.
Wes: He's kind of well-known. You've maybe seen his famous video online using D-Trace
Wes: to show how hard drives don't like it when you yell at them.
Wes: So, you know, has deep insight into this area and was, even in 2019 and earlier,
Wes: was already getting excited about eBPF.
Wes: So it's kind of neat to look back now as like a whole giant marketplace of eBPF-based,
Wes: observability tools now exist.
Chris: And the name is sort of a misnomer, right? Because it sounds like a packet filter.
Chris: And so you think, well, how? Okay, what is this a firewall, guys?
Chris: No, no, no. That's why we wanted to play that intro clip. It is so much more.
Chris: It is really a VM inside the kernel that can run simple code that you can create
Chris: and craft that is protected.
Chris: And so since this is a prerecord, we're not going to have your boost this week,
Chris: but we do want to know if you like this type of deep dive into this particular topic.
Chris: So boost those in and we'll bank them for when we come back.
Chris: But let's talk about the extended Berkeley packet filter.
Wes: Yeah. So it does do some networking stuff still to this day.
Wes: But it did, as you say, start out as a packet filter introduced in 1992 to efficiently
Wes: filter network packets in BSD operating systems. and.
Chris: Pf sense and other firewall products like open sense of you know they've been
Chris: using bpf as part of their core product i that's one of the reasons i was an early pf sense user.
Wes: And you know before too long within the same decade it made its way over to
Wes: linux in the form of tcp dump and already you're seeing this thing right where
Wes: you can kind of use user space to help better observe what's going on on your system,
Wes: And so if you've ever written sort of an expression to filter things or look
Wes: at packets using TCP dump, well, you're using a language that then gets compiled
Wes: down to BPF bytecode and executed.
Chris: That bytecode right there is kind of the magic, right? Because it turns out
Chris: that this thing is essentially capable of running this bytecode.
Chris: So it's not just a packet filter.
Wes: Yeah, and that's like how the implementation works. As it started out,
Wes: it was a very simple virtual machine.
Wes: And don't think virtual machine like QEMU necessarily are like simulating a full computer.
Wes: the point is it's like a very limited restricted bytecode
Wes: that can only do certain things relevant to at first filtering packets
Wes: and that lets you make sure that like it's not going to do anything crazy it
Wes: can't go into infinite loops all kinds of other nice things and optimize
Wes: it and then be able to load it
Wes: in and have you know you run the program it supplies
Wes: the packet and anything else that you need is input to it
Wes: and then the machine executes and ultimately that's how you tell like
Wes: do i accept the packet or do i drop the packet uh but
Wes: at first it was a you know a very limited
Wes: i think i had like a two two registers to use super limited thing
Wes: but ebpf extended bpf was introduced in linux 318 this was like 2014 so bpf
Wes: had been around for a while there'd been various developments but ebpf really
Wes: kicked things off in the linux side of things bpf hadn't you know caught up
Wes: with the times in some ways it was still 32 bit it had some of those register limitations.
Wes: So they upgraded to 64 bit registers added more
Wes: instructions they added the verifier to
Wes: the kernel which is a big part of it that lets you analyze bpf programs
Wes: to make sure they're safe no infinite loops they don't do invalid memory access
Wes: there's a checker process right because it's like you're loading something from
Wes: user space into the kernel that's a big security concern so you you want to
Wes: make sure that you have and just be on security operationally too right you
Wes: don't want it to be able to crash the kernel so.
Chris: We actually started to see this stuff land you know progressively in linux 3
Chris: 18 and beyond so this is actually like you said 2014 that's a this has been landing for a while.
Wes: Yeah definitely and then so that was kind of like the raw stuff and then 2018
Wes: and beyond people started adding more tools like the bcc but the bpf compiler
Wes: collection bpf trace which we'll talk about too uh the company psyllium and
Wes: their their products have a bunch of like ebpf stuff for,
Wes: Kubernetes offerings plus we got better like compiler support there's something
Wes: called Kori or compile once run everywhere so like compilers have better support
Wes: to be able to make you know you can compile your BPF program and not have to
Wes: worry about as much necessarily depending on what you're doing with it about
Wes: how compatible it'll be with the kernel where you compiled it versus where you're running it so.
Chris: The idea being like it should be compatible across multiple versions of Linux
Chris: as long as they have this correct implementation.
Wes: Right okay This has been so successful that newer versions of Dtrace on Linux,
Wes: They're basically just some extra user space stuff that uses eBPF and other
Wes: kernel primitives under the hood.
Chris: Really? Yeah. Huh.
Wes: And eBPF was also ported to Windows.
Chris: Yeah, I heard about this. They're getting a lot of good stuff over there.
Wes: One important thing with the extended part is they were also able to make the
Wes: instruction set be more sympathetic to modern hardware, and they implemented
Wes: just-in-time compilation as well.
Wes: So that meant eBPF can be really fast.
Wes: They've also made it so that there's relatively stable APIs we've kind of been
Wes: talking about, so that as kernels change that, you can use eBPF to hook into internals.
Wes: If you do that, there's no guarantees, right? But that's kind of on the tin.
Wes: But there is some stable interfaces that you can use, which is nice.
Wes: So the other important point to know is there's kind of various things it can do.
Wes: There's XDP, which is Express Data Path, which intercepts packets basically
Wes: at like the earliest point that you can.
Wes: you have limitations on what you can do but it can be super low level and performant
Wes: and so you can see this sometimes maybe unlike responding to ddos attacks where
Wes: you're getting flooded with traffic that you have you know maybe you can identify
Wes: various ways that you can program in here so you could essentially.
Chris: Ideally catch it before it begins to overwhelm the system because you're catching
Chris: it much further in like the driver stack.
Wes: Yeah that's the idea and it doesn't do as much as like the very general and
Wes: powerful but you know full-featured linux kernel networking stack you can kind
Wes: of just be like, well, no, if it looks like this at all, just cut it off immediately.
Chris: Yeah, if I'm getting DDoSed, I don't want the network stack trying to figure
Chris: out what to do with all this traffic because that's what's going to take me down.
Wes: Yeah, one way to say it is limited context but maximum performance.
Chris: I like that.
Wes: Okay, so then you can also do various types of K-probes.
Chris: I'm sorry?
Wes: Yeah, K-probes. And this is dynamic instrumentation. That's why it sounds quite probing.
Wes: You can hook almost any kernel function.
Wes: but you don't there's no like clear definition of what you're going to get you
Wes: have to go look at the function you're hooking into it's all going to be dependent
Wes: on that but no kernel internals any kernel function almost i'm sure there are
Wes: some limitations that's pretty a lot of them yeah i.
Chris: Mean that could be like from you know keyboard input to network traffic to disk
Chris: io i mean that's there's all kinds of things.
Wes: There so that's where a lot of some of the power comes from yeah but that may
Wes: or may not be stable there's no guarantee about it being stable across kernel
Wes: versions, can change at any time. People can update the signatures.
Wes: That's one of the things that's been happening in the Rust discussion is the
Wes: kernel, many developers expect to be able to make a change like that and because
Wes: they have one big code base, they can update it everywhere and be able to do
Wes: a refactoring like that. And, but the other one,
Wes: Trace points. This is an important one. Trace points. Predefined stable instrumentation.
Chris: Ah.
Wes: Low overhead, but only available where explicitly added by kernel developers.
Chris: So, okay. So a trace point would be like a spot you hook in and start getting
Chris: metrics or information out of.
Chris: But the developer of the subsystem has to implicitly support that. Yeah.
Wes: You have to add that in versus totally dynamic with a K-probe.
Chris: Okay.
Wes: But the upside is you get a structured data specific to each trace point,
Wes: right? So the trace point tells you what it is.
Chris: Yeah, yeah, okay.
Wes: And they're maintained across kernel versions, so you can kind of rely on them for more long-term use.
Chris: I have a feeling this might be relevant later. Okay, good to know.
Wes: Yeah, I think that's probably like the quick, high-level version of what eBPF
Wes: is and kind of what it can do.
Chris: And it is so, so, so simple, but yet so powerful is what I love about it.
Chris: And we have a couple of examples in the show, and I think some hands-on stuff
Chris: that people could take away.
Chris: And maybe we should start with BCC tools themselves.
Wes: Yeah, because we've both had a chance to play with those.
Chris: Yeah, I was, you know, I was joking around with Wes and be like,
Chris: it'd be kind of great to know where in the system when I open up this directory
Chris: that's a fuse mounted directory,
Chris: like where is the actual delay happening and what part of the system is sitting around waiting.
Chris: And through this process of trying to get to that, I came across tools that
Chris: let me look at my disk IO analysis or let me look at the network traffic and
Chris: really kind of using these different tools, putting them together.
Chris: You can actually start to get a really good picture of where the delay was happening in the system.
Chris: And, you know, for me, it was like, this is going to be amazing.
Chris: I had discovered by running it on my desktop, just as when I was experimenting
Chris: with this stuff, that, oh, yeah, I've still got this errands app that we talked
Chris: about on the show that I'm not currently using like today.
Chris: But I guess when I close it, it doesn't actually close. It doesn't leave an icon.
Chris: I had no idea it was running. But using these tools, I started seeing this application
Chris: that was hitting my disk every so often. And I'm like, I don't recognize this.
Chris: and I discovered I actually had that running the entire time.
Chris: And it was kind of useful.
Wes: Yeah, it kind of skips through a lot of the boundaries and other limitations
Wes: that can sometimes pop up when you're trying to look at your system.
Wes: So it can be surprisingly insightful.
Wes: They do have a nice tutorial, which we'll link. They also, I like that they
Wes: have a set of things to run first.
Wes: Like before you go to these tools, make sure you check all like the regular
Wes: system monitoring tools first because we'll see as a theme, like,
Wes: you know, your H-tops and B-tops and all kinds of things.
Wes: kind of get a broad look and you could see some specifics whereas you could
Wes: you can make broad eppf programs but by default they're going to be a lot more
Wes: specific when you're looking at like one thing.
Chris: Like disk.
Wes: Latency or something specific to the file system.
Chris: Or networking yeah yeah that's a good point um all right so we could talk about
Chris: some of the commands we tried i know that uh file top and tcp top we played
Chris: around with um i did not get a chance to play around with ext4 slower and XFS
Chris: slower, but this is an interesting way you can use it too.
Wes: Uh-huh, yeah. It'll tell you when, you know, there's things happening in your
Wes: file system that are causing latency more than, like, a programmable amount
Wes: you can pass on the command line.
Wes: Or they've got ZFS dist, which traces ZFS reads, writes, opens,
Wes: fsyncs, and then summarizes the latency of all that in a histogram for you.
Chris: That's really cool.
Wes: Yeah, right? And then two really useful basic ones are exec snoop and open snoop.
Chris: Yeah, OpenSnoop, isn't that kind of like, oh, there's like a Mac app that monitors
Chris: traffic? Is that what it was? OpenSnoop something else?
Wes: Yeah, no. So OpenSnoop, you're thinking of, there's a Linux one,
Wes: I think, right? OpenSnitch? Yeah, just one thing.
Chris: Snitch. Snitch. That's, yes.
Wes: OpenSnoop tracks file opens.
Chris: Okay.
Wes: So you can run this and it hooks into the kernel via BPF. And then anything
Wes: that's running on your system that opens files, it'll just print out in your
Wes: console right in front of you.
Chris: I did play with this one. I did play with this OpenSnoop one.
Chris: You're right. Yeah, that's really cool because I remember I launched,
Chris: you could launch particular things and then just watch all the crap that happens on your system.
Wes: Right. And especially, you know, you can filter it, you can have it look at
Wes: specific processes or you could, you know, grab output or whatever.
Wes: So you can filter on a busy system, but I think it's especially useful on a
Wes: system you think should be, you know, relatively quiet and just as a way to
Wes: see, like, what is actually happening in the background?
Wes: And for short-lived processes that might be hard to see in a summary-type program
Wes: that are still doing things, it'll still print in OpenSnoop.
Wes: That's where ExecSnoop does a similar thing, but for process opens.
Wes: So again, for things where it's hard to see in a general tool,
Wes: but you really want to see at a nitty-gritty level what is happening in being
Wes: Exec on this box, ExecSnoop is handy for that.
Chris: I have a sense Exec Snoop has come up actually in real life for you as a handy tool.
Wes: Yeah.
Chris: Do you want to share a war story with us?
Wes: I do. So a few years ago, I was adminning a VPS box. I wasn't totally in charge
Wes: of, but had some, you know, administration over. And...
Wes: Things were seemingly broken. I'd noticed some messed up statistics first,
Wes: right? The metrics were a little odd.
Wes: And I went on and I started just kind of doing regular updates and poking around at the system.
Wes: And I noticed that I wasn't able to actually update the initRAMFS,
Wes: right? I was doing all the, it was an Ubuntu box. I had everything going.
Wes: But there was some like dynamic library linking problem that was happening after
Wes: I started digging into the output and tried rebuilding it way too many times.
Wes: And I'm like, what, why? Why can't I get it?
Chris: Yeah, well, I can't get my initRAMFS updated.
Wes: The kernel's here. Everything else in the update's fine.
Chris: But you can't boot in that new kernel if you can't get that updated.
Wes: Yeah. So this was maybe closer to when I really started playing with these tools
Wes: a little bit more seriously and having them installed on more boxes.
Wes: And so they were already available.
Wes: And I used OpenSnoop because I wanted to see exactly what was being opened and
Wes: touched by the update initRAMFS process.
Chris: Right.
Wes: And then that led me to look at some weird file paths.
Wes: And I also started, then I did exec snoop, and that showed similar file paths running on the system.
Chris: Oh, uh-oh.
Wes: And then I was able to figure out that there was, in fact, a crypto miner running on the box.
Brent: Oh, no.
Wes: But it had loaded a custom kernel module to hide itself from tools like PS and top and htop.
Brent: That's pretty slick.
Wes: But not from exec snoop.
Chris: Clever. So that's why it was breaking when you were trying to update in that in an FS.
Wes: Yes. Something it did to the system. I never quite figured out exactly what
Wes: it had changed, but it had messed with some of the files on the system in ways
Wes: that meant that the linking was no longer cleanly happening.
Chris: Yeah.
Chris: So that's a good way to stack those two tools.
Chris: You know, OpenSnoop and ExecSnoop are really a couple of nice tools you can stack together.
Wes: And there's a whole bunch of the, this is part of BCC, the BPF compiler collection,
Wes: which includes a whole bunch of other stuff.
Wes: But these are the tools that they ship by default, which leverages the framework
Wes: they've built to implement these via eBPF.
Wes: And so there's like, yeah, all kinds of stuff, file system specific stuff.
Wes: They've got things for, you know, looking at network connections,
Wes: TCP connections. They've got stuff for monitoring databases. It's broad.
Chris: I don't know if it's necessarily best practice, but you could say that with
Chris: these tools, you could be fairly confident that you had cleaned up the crypto
Chris: miner and the infection.
Chris: You know, because you can actually watch at a much more intimate level what's
Chris: happening at the system.
Wes: Definitely useful. Yeah.
Chris: I'm not saying it's probably a best practice, you know, but probably just wipe
Chris: the box. Yeah, but it is sort of nice if for some reason you don't have that
Chris: option, you can use these tools to kind of confirm that stuff isn't coming back after a reboot.
Wes: Right. And the more, you know, depending
Wes: on, but you can see exactly where they're hooking and modify the,
Wes: because a lot of this is implemented via a combo of C because that's the part
Wes: that gets compiled and like it's like a limited subset of C that does the BPF
Wes: stuff that gets converted into a loadable thing for the kernel.
Wes: but there's a bunch of python utilities around it to wrap it so you could fork
Wes: that copy it modify it if you need it even more or to try to observe specific
Wes: things once you identified like a particular problem or threat now.
Chris: That's really going deep,
Chris: onepassword.com slash unplugged. That's the number one password.com slash unplugged, all lowercase.
Chris: And you're going to want to go there because this is something that if I had,
Chris: when I still worked in IT, I think it would have sustained me for many, many more years.
Chris: The reality is your end users don't, and I mean, without exception,
Chris: work on only company-owned devices, applications, and services.
Chris: Maybe you get lucky and they mostly do, but I don't find that to be the reality today.
Chris: And so the next question becomes, how do you actually keep your company's data
Chris: safe when it's sitting on all of these unmanaged apps and devices?
Chris: Well, that's where 1Password has an answer. It's extended access management.
Chris: 1Password extended access management helps you secure every sign-on for every
Chris: app on every device because it solves the problems that your traditional IAMs
Chris: and MDMs just can't touch.
Chris: And it's the first security solution that brings unmanaged devices and applications
Chris: and identities, like even vendor, under your control.
Chris: It ensures that every credential is strong and protected, every device is known
Chris: and healthy, and every app is visible.
Chris: This is some powerful stuff, and it's available for companies that use Okta
Chris: or Microsoft Entra, and it's in beta for Google Workspace customers too.
Chris: OnePassword changed the game for password management, and now they're taking
Chris: everything they've learned there and expanding it to the login level and the application level.
Chris: You know what a difference it makes when people have proper password management.
Chris: Now let's have proper login management and authorization.
Chris: 1Password also has regular third-party audits and the industry's largest bug
Chris: bounty. They exceed the standards set by others.
Chris: They really do. So go secure every app, every device, and every identity.
Chris: Even the unmanaged ones. You just go to 1Password.com slash unplugged.
Chris: All over case. that's onepassword.com.
Brent: Slash unplugged,
Brent: now Wes it sounds like BCC is an abstraction to EBPF what's going on under the hood here.
Wes: Yeah so there are tools that you
Wes: can just run like we've been talking about but how did those tools come to be
Wes: well that's where some of those abstractions and a framework comes in that allows
Wes: basically embedding C code directly within Python scripts and then And BZC's
Wes: tools also sort of handle making sure you've got the whole tool chain available.
Wes: So it uses LLVM and Clang to compile things. It handles verification of stuff.
Wes: It handles loading it into the kernel for you and attaching it to the right
Wes: hooks, basically through like Python method calls instead of you having to run
Wes: the right commands in the bash shell.
Wes: So you kind of get like a unified approach where you can write kernel level
Wes: programs without necessarily having to deal with like all the other stuff.
Chris: You know I was thinking gosh this really seems like going way beyond what my
Chris: skill set would be able to manage but then I thought,
Chris: python code i bet an llm would get me 90 of the way there nice these days and
Chris: then i could probably just finish it off.
Wes: And there's a fair amount of like existing tools that you
Wes: can either feed into an llm and ask about or
Wes: modify or try and you know hack on yourself yeah and so here's a simple example
Wes: that we can play with it uses xdp okay and so there's a simple c program that
Wes: has a function called xdp drop all okay uh and that's going to attach to the
Wes: xdp hook and so we get like a little data summary of the packet info,
Wes: which we're not going to care about.
Wes: All we're going to do is return XDP drop, which is a magic value that tells
Wes: the kernel, hey, just drop this.
Chris: Just drop all.
Wes: So it's going to drop everything. And then in the Python, that's it C-wise. That's all we do.
Wes: And then in the Python world, there's a little setup to make,
Wes: hey, I want a new BPF thing or whatever.
Wes: And then we tell it to load in our little blob of C.
Wes: And then we attach it to the interface we want. That's two lines of Python.
Wes: And then it prints a nice little message and starts working.
Chris: And what it should do is drop all network traffic?
Wes: Yeah, so we specified a specific interface.
Chris: Yeah, so you say on this interface, just drop everything.
Wes: And that's because we attached it to that specific interface.
Chris: And this is like a quick kill switch. So what we're going to do as a test here
Chris: is I've set up a ping with an audible sound so we can see when we're getting a result.
Chris: So we can see we're pinging the box right now. And then at some point, Wes is going to kill.
Chris: He's actually going to run that kill script, and it's going to drop all traffic.
Chris: and we'll hear it drop off. Whenever you're ready, Mr. Payne.
Wes: Three, two, one.
Chris: There it goes. Yeah, it takes almost no time at all. It happens almost immediately.
Wes: Now, and of course, in my regular SSH session, I can't stop it,
Wes: but I do have a sneaky console here, so we should be able to take a peek.
Brent: Wes always has a sneaky console.
Chris: All right, I'm leaving the ping going, so if it resumes.
Wes: We should hear it. I'm hitting Control-C now.
Chris: Okay, out there it goes. Look at that.
Wes: Yeah. So like that wasn't that much work. This is just an Ubuntu 24.04 box.
Wes: So like, you know, all the stuff you needed was in the repos already.
Chris: I'm thinking, you know, like that's just a it's it's not a total practical example,
Chris: but it's a quick example of the power you have there and you're executing that from user space.
Wes: Yeah, you do need, you know, root permissions to be able to load it into the kernel.
Chris: But then you're executing things inside the kernel that just immediately cutting
Chris: off traffic to that interface.
Wes: Yes. And I didn't have to build a per kernel kernel module with the right headers
Wes: and like have to worry about messing up my implementation in a way that's going to crash the box.
Chris: You didn't even have to create like a IP tables like that rejects traffic because
Chris: that would actually be much further down in the processing stack if you're using
Chris: something like IP tables.
Chris: Yeah, that's cool. That's fun.
Wes: So in reality, you'd want, right, you would do some sort of filtering where
Wes: you would look at the data structure you get from XDP to figure out like,
Wes: oh, is this one I want to block or let it keep going through processing?
Wes: But yeah, this is your basic. It's not hello world. It's goodbye world.
Chris: Yeah. It's goodbye world. Exactly.
Wes: But the main point to show that was that this BCC framework is one way if you
Wes: want to start developing custom tools.
Wes: But you can get even more ad hoc than that.
Chris: All right.
Wes: Because there's BPF trace.
Chris: Yeah. I want to talk about this.
Wes: And this is basically sort of like the closest thing to D-Trace for Linux if
Wes: you don't want to use the Oracle tool, which I believe Gentoo now has.
Chris: Oh, all right. So BPF trace is sort of what I know of D-Trace is like this ultimate
Chris: tool when you're debugging or trying to figure out where your system has gone sideways.
Chris: again like i was talking about like why is this one fuse directory taking so
Chris: long to open and so i imagine this is a similar type of.
Wes: Yeah one way to think about is you basically get like a nice targeted little
Wes: scripting language to write tracing programs oh that use the trace points in the kernel.
Chris: Wow okay all right.
Wes: And there are some other options there's a program called ply that i
Wes: haven't tried but looks also nice um but bpf trace
Wes: is quite popular uh so we talked about right
Wes: there's different there's xdp there's the k prob stuff these are the trace points
Wes: which is the the easy stable ones because they're defined and they give you
Wes: a known data structure so you can do bpf trace dash l or the bcc tools has tplist
Wes: which is also just lists all the trace points it also shows you can do other
Wes: user space stuff but we're not going to talk about that today so.
Chris: There is so tplist i guess that makes it a lot easier if tplist is going to
Chris: give you all of the kernel trace points so you know what you have to work with that's really useful.
Wes: Right And it tells you the shape of them, too. You get the sort of data structure.
Chris: Okay.
Wes: Um...
Wes: Let me pull it up here.
Chris: He's got it there on the machine.
Wes: Yeah, so here's sort of like, here's some stuff about block dirty buffer.
Wes: And it tells you you get a dev device, you get a sector and a size.
Wes: And so you know those are the things you can work with. And then it has a name
Wes: and you just tell it that you want to trace that thing.
Wes: So this came up because, you know, we talked a little bit about BCC having file system specific tools.
Wes: Like, oh, I want to see watch for slow operations from ButterFS, say, right?
Chris: Right.
Wes: Well, they haven't yet. I suppose I should step up or someone should step up
Wes: and add these, but there isn't yet a BcacheFS version of these tools, right?
Chris: There we go. That'd be cool.
Wes: Obviously, probably what you want to do is just, you know, fork the existing
Wes: ButterFS one and modify it and make it work right for BcacheFS.
Chris: Sure.
Wes: But if you want to be ad hoc about it, BPF Trace can handle this too.
Wes: So I did TP list and then grepped that for things that said BcacheFS.
Chris: So you got all of the kernel interfaces regarding trace points,
Chris: as I should say, trace points regarding BcacheFS.
Wes: So here's just like a list of some of what the trace points look like.
Chris: I see. Okay. So now, okay, so what I'm seeing here on Wes's screen is like an
Chris: output, BcacheFS colon, and I can get data update, rebalance, extent.
Chris: There's a lot of different options here. Essentially, just what is the file system up to?
Wes: And so these would all be things that Kent or team have put in explicitly so
Wes: that there's a way to like easily and with low overhead watch these things.
Chris: So since these trace points already exist, you can use...
Chris: bcc to write your own kind of monitoring.
Wes: Yeah or bpf trace which is right.
Chris: Yeah thank you.
Wes: Um there's a lot of terms there's a lot of bpf terms
Wes: so here's one that stood out uh copy gc
Wes: wait and bcashfs is a copy on write
Wes: file system and it does this bucket based allocation and so
Wes: it has what's called a copying garbage collector so as
Wes: you're copying files it'll handle moving things around so
Wes: that it can then like get good bucket allocation and
Wes: defragment things on the fly that kind of thing so copy
Wes: gc wait is a trace point that tells you when
Wes: you're when bcachefs is waiting for copy gc
Wes: to complete so it can be a reason that your
Wes: file system is being slow or not responding especially if you're moving or copying
Wes: files around so you can see on your screen uh just a little tiny script uh sudo
Wes: bpf trace dash e and then you pass it a little string and we tell it we want
Wes: to use the trace point bcachefs copy gc wait and then we have a little script here that,
Wes: one of the inputs is the device and then it has little stuff that extracts the
Wes: major and minor number from that you don't have to do that and then all it does
Wes: is print what device we're waiting on and how much we're waiting yeah and so
Wes: it's you know i don't know 10 lines of code.
Chris: And you get a nice kind of structured output it's easy to human readable it
Chris: tells you the total flushes the total buffers flushed you get you know the total
Chris: output whatever whatever the stat is you're watching.
Wes: And then so on my system i'm just doing it on you know one rudifest disc so
Wes: i knew which disc it would be it was just kind of interesting to see like i
Wes: went around and i dd'd some big files and copied them around and i could see
Wes: like oh yeah right the operation completes and then immediately the print tells
Wes: me that oh we waited that much and.
Chris: The practical use here is you know in a rate array,
Chris: It might be useful to discover that one of your disks is significantly underperforming the others.
Chris: It could be indicative of a larger problem. It could be indicative of why you're
Chris: having performance issues.
Wes: So, like, it turns out that like this, there's actually probably a fair amount
Wes: of situations where single trace points, just on their own, might be something
Wes: you want to look at, right?
Wes: So other ones for BcacheFS, write buffer flush. That's an important event.
Wes: Or journal writes. You might be wanting to know stuff about how the journal works.
Wes: but you can also use the scripting language and the
Wes: fact that eBPF supports basic data structures like maps
Wes: and histograms to make more
Wes: complicated combined programs so i had an llm i
Wes: fed it the output of that tplist stuff that told it the available trace all
Wes: the traces you had and what the structures looked like so you had to write the
Wes: right code that's clever and uh bpf trace makes it easy to have stuff that runs
Wes: at an interval right so you can set up all the traces And then every five seconds,
Wes: it can print out a little summary that it's done.
Wes: And so each time it traces, it can update a variable, and then it can calculate latencies or deltas.
Wes: So this is a quick one that it did that looks at that right buffer flush trace
Wes: point, and it samples it over every five seconds, and then it tells you how many flushes happened,
Wes: how many buffers were flushed, how many were skipped, how many were fast path
Wes: flushes, and then the total amount of data that was flushed.
Wes: So just like a quick way to get, you know, an accurate little like every five
Wes: second little printout.
Chris: And what's great is like, as
Chris: far as I know, this is really putting no measurable strain on the system.
Wes: It's one of the more performant ways to do it. Yeah.
Chris: So you can really get deep insights without some of the overhead you sometimes
Chris: get by that kind of monitoring.
Wes: So then to kind of further stress just how far this whole having an AI help
Wes: me out would go, I had an idea.
Wes: I did ultimately tone it back a bit, but basically like, what if I wanted to
Wes: monitor some of the mesh network traffic that was going on?
Wes: There's a lot of options for that, but can this do this too?
Wes: And so it built me a little script that you give it an interface name,
Wes: like tailscale0, say, right?
Wes: and then it'll do a similar thing where every five seconds it'll just look at
Wes: that interface and it'll count sent packets sent bytes receive packets receive
Wes: bytes and it'll do a send latency histogram for you on the sent packets yeah.
Chris: So you get this you actually i mean it's not a gui but it's it's it's a bar
Chris: graph on the command line.
Wes: Yeah right and so the combination of the built-ins ebpf and
Wes: then the built-ins into bpf trace which
Wes: has some of this stuff to do nice histogram display and stuff from
Wes: their print command um it's you know this is a little bit longer some of it's
Wes: kind of because there's a print statement for each thing we're printing it's
Wes: like five different things we're measuring right so it takes up some space on
Wes: the screen but like it's a single page of code here so it's not a it's not crazy
Wes: to start trying to understand it.
Chris: Right i've never even seen some of this stuff but i under i could like the trace
Chris: points they all make sense they're all it's just real plain english really it's
Chris: really easy and then you have the print and the time interval.
Wes: And so under the hood, it is tracing something called NetDevStartTransmit.
Chris: Okay.
Wes: So then it filters, it gets data, it filters on interface name from that data,
Wes: and then it has a little counter it's keeping, so it increments the counter.
Wes: And then it has a thread ID that you can use, and so it gets the current timestamp
Wes: in nanoseconds and stores that in a map based on its thread ID.
Wes: And then it also does a trace point for sent packets with NetDevTransmit,
Wes: and then it grabs the start time if it exists for
Wes: its thread id and then it can compute how
Wes: long it took to send from that and then that's where it can use
Wes: the histogram stuff to print out a nice little thing now that it's tracking
Wes: latency and then it has another uh trace point to use uh net if receive skb
Wes: which tracks incoming packets and then it just has a little bit of code here
Wes: to kind of tie it together every five seconds and do the printout and then it
Wes: clears all the counters so that it can do the next cycle.
Chris: That's a neat little magic, like, superpower, a pocket of power that's in the
Chris: Linux kernel there for this.
Wes: Now, it does mean, right, like, you can't use it if you don't have trace points you care about.
Wes: And so you have to start understanding some kernel internals and,
Wes: like, what trace points might matter and what they mean.
Wes: But if you have a specific problem or a mystery on your system that you're trying
Wes: to look into and you're willing to try to chase down hypotheses using these
Wes: tools, I think you could get pretty far.
Chris: Yeah, that's for sure. And then, you know, there are actual complete products
Chris: out there that are dipping into this stuff.
Chris: I was reading online that there's actually several Kubernetes products that
Chris: are tapping into eBPF to do things under the hood.
Chris: Also, Falco, which is a real-time security monitoring application you can run on Linux.
Chris: It says, Falco uses eBPF to monitor system calls and network events directly
Chris: in the kernel, enabling rapid detection of anomalous behavior with low latency.
Chris: This kernel-level approach is faster and less resource-intensive than the user
Chris: space monitoring tools.
Wes: There's also user space tracing you can do. So if you set it up right,
Wes: you can trace JVM events or Ruby things or Python stuff.
Wes: And so you have products now, too, that will use things like open telemetry
Wes: or other tracing standards,
Wes: and you can have a trace then that can have both the kernel side via the ePPF
Wes: stuff and the user space stuff without necessarily having to do as much custom
Wes: observability implementation in the code base because they can kind of do more dynamic.
Wes: And you might be able to then see stuff, right, where like, oh,
Wes: there was a problem on the kernel layer, and then that's why we started seeing
Wes: increased latency in the application layer.
Wes: But you can use that. There's a lot of open source products or open core things,
Wes: and there's just a lot of sort of primitives in one layer above primitives that
Wes: are probably already on your kernel.
Chris: I wonder if it's possible people out there listening have already been using
Chris: eBPF for a while and this is all old news to them.
Chris: Who didn't tell us? Like, have
Chris: you used it in what applications and what utilities has it been for you?
Chris: I mean, we've gone through a few examples here, but like Wes's story with the
Chris: VPS and my story with tracking down like a background desktop application,
Chris: there's just these also very practical day-to-day uses for this stuff.
Chris: that's nothing more than just using some of the pre-existing tools that you
Chris: just run and tap into this, and you don't have to write any kind of scripting code.
Wes: Yeah, I mean, it's been around for long enough now that it's well-packaged in most distros.
Brent: Can you use eBPF to figure out why I have so many tabs open?
Wes: Well, maybe we can add a trace point to Firefox to keep track at least.
Chris: Jupiterbroadcasting.com slash river river is the most trusted spot in the u.s
Chris: for individuals and businesses to buy send receive bitcoin and they make it
Chris: easy in three simple steps jupiterbroadcasting.com slash river hey.
Wes: I just got set up with them and it was in fact very easy.
Chris: I think their best low-key feature actually
Chris: has to do with cash just as an aside so bitcoin is
Chris: dipping as we record and if oh man this is
Chris: this was the moment so they have a 3.8 percent
Chris: interest cash savings account and they pay the interest
Chris: out in sats so you put your cash in there and it's
Chris: fdic insured 3.8 percent interest and
Chris: then when the when the price dips on bitcoin you can use that balance
Chris: to smash buy and so you can you can essentially dca with that 3.8 percent interest
Chris: and then you can smash buy when the price dips it's just brilliant and river
Chris: is a really great company i had a call with their community management person
Chris: and was really impressed with what they said.
Chris: And if you're in Canada and you're looking for another trusted source to get
Chris: sats, bitcoinwell.com slash Jupiter.
Chris: There you go. That's our tip.
Brent: Well, since this week we are hoarding our boost for next week's episode,
Brent: we decided to do a little dive into the old mailbag.
Brent: Zach sent us a little note here. I'm behind on listening to episodes.
Brent: But the NixOS episode from last month, Linux Unplugged 5.9.8,
Brent: was really interesting.
Brent: I wanted to throw out some thoughts responding to a comment that was sent in
Brent: about wanting a NixOS-like system, but being able to use the old traditional system admin tools.
Brent: For context, I've tried out NixOS, but quickly got to the point where I would
Brent: need to dive in and figure out those flakes.
Brent: At that point, it just became one more thing on that list to do.
Brent: Since then, I've gotten into bootable containers.
Brent: I've specifically been using the Universal Blue ecosystem with Bootsy,
Brent: but have wanted to try out Vanilla OS and Common Arch that also use OCI images
Brent: for delivery and customization.
Brent: Listening to that NixOS episode, I felt like many of those talking points that
Brent: were given in favor of NixOS were also very in line with the benefits that I've
Brent: seen in using OCI images to build out customized OS images.
Brent: I don't use any of the headliner distros like aurora or
Brent: bluefin although i have tried a few and overall they were quite
Brent: well put together but i've taken their more
Brent: low-level image builds and layered my own customizations
Brent: on top of them it's been a fantastic way to manage everything as when i need
Brent: a new server or desktop all i have to do is install my image and everything's
Brent: ready to go i will add the caveat that there may be some learning if you haven't
Brent: already been familiar with building OCI containers,
Brent: but I live in a world where professionally and for hobbies, that's what I'm doing.
Brent: So that wasn't a huge burden for me to learn.
Brent: Thanks for the show. Really have been enjoying it and wanted to send this little
Brent: message in just in case someone else might find some use.
Wes: Well, cool. Thank you for sharing. We love success stories like this.
Wes: I wonder if you have any of it on GitHub or anything too.
Chris: If people are curious. Good question. I really
Chris: feel like this nails something that we've been feeling and chatting a
Chris: lot about behind the scenes is uh the really
Chris: really brilliant thing that the universal blue folks
Chris: have tapped into is this existing knowledge set around how oci images work and
Chris: how you can layer things and really tapped into the whole devops workflow that
Chris: people live every day and now you can use it that skill set that you've learned
Chris: and you can use it to customize your Linux desktop. I mean, that is just really powerful.
Chris: It's a different approach than Nix, right? Whereas Nix, you're building it from
Chris: the ground up. You would use Nix to maybe generate those OCI images or something like that.
Chris: But both, like he says, are essentially accomplishing similar goals.
Wes: Yeah, and a lot of the immutability side of things and the image-based approach, right?
Wes: Like starting to think of your system as a cohesive whole that you deliver in
Wes: those coherent units rather than ad hoc imperative updates.
Chris: Now, admittedly, I don't use them as much, but Brent, when I think of an image-based
Chris: system, I don't necessarily think of flexibility.
Brent: Yeah, I was kind of wondering about this. I've had a few people tell me,
Brent: oh, yeah, images is totally the way to go.
Brent: Of course, we've been playing with Nix and NixOS at the same time,
Brent: and the thing I keep tripping on with images is...
Brent: You build them once and then you use them many times. But the way,
Brent: Chris, I think you and I have been using at least NixOS on the desktop is your
Brent: Nix config is like constantly evolving.
Brent: Therefore, the next time you go to install it somewhere else,
Brent: it's always that new updated copy.
Brent: And I would imagine one of the downsides with building an image is,
Brent: well, you have that intermediate step.
Brent: You have to go ahead and build your image every time you want that newest, freshest version.
Brent: Am I understanding that correctly? Is that a proper downside here?
Wes: I know some of the silver blue types, right, they're still using a lot of RPM
Wes: OS tree, which has ways to apply things live.
Wes: So it might depend a little bit on how exactly what you're putting in those
Wes: images, or maybe you have like a dynamic switch image thing.
Wes: But yeah, I do wonder about that.
Chris: And then you also got to remember, it depends on what you're doing.
Chris: You know, it might just be maybe a lot of your applications are flat packs and
Chris: you're installing those and updating those separately anyways.
Wes: Or you're using, you know, various containers or other things for your dynamic environments.
Chris: I'll take Liam's email here. He says, good day, gents. In the most recent LUP,
Chris: you mentioned not having received much feedback about multi-monitor setup.
Chris: So he sent us a picture of his.
Chris: I've got a laptop screen at 1080 at 144 hertz. That's my primary display,
Chris: you know, for things like Skype.
Chris: Then I have two externals, a 32-inch 2K.
Chris: I do think that's the sweet spot, he says, at 60 hertz. And then my rightmost
Chris: monitor is in portrait mode.
Chris: That's where I can see more lines of code. All of this works without issue on XFCE.
Chris: In case you do navigate to the image, yeah, that's a treadmill desk.
Chris: I'm on my second one since the start of the pandemic this.
Wes: Is a nice setup.
Chris: Listeners since Sidebyte, Lunduk and Last Days and Early Tech Snap member since
Chris: 2022 annual member since December that's awesome thank you very much Liam and
Chris: I like your setup I am as you boys know big fan of the vertical monitor,
Chris: For our show docs or, you know, like reviewing a config file or just having a terminal up there.
Wes: Journal D.
Chris: Yep. The vertical monitor is a productivity hack.
Chris: If you can afford the luxury of, because you're not, it's not great.
Chris: But one of the other things I will use the vertical monitor for is I stack two chats.
Chris: Like if I have two different work chats or something, I'll stack them on the
Chris: vertical monitor. Works really great.
Wes: Right. Yeah. It doesn't fit every activity. So it's sort of a less generally useful screen at times.
Chris: Yeah. Yeah. It is more limited, so it's more of a luxury. I do have to admit that.
Brent: I have a bit of a question here for us. Did we ever divulge what our current setups are?
Brent: Chris, I know you've gone into details, but I don't think Wes and I have actually shared that.
Chris: Wes, I bet you kind of move around depending on, because you're mostly using laptops.
Wes: I do a lot of laptop. I've got a dual monitor set up in the sort of main office-y
Wes: bit, and then I have just a single where I do laptop with one screen in my living room sort of desk.
Chris: You got a monitor at the living? That's nice.
Wes: That's for the like casual work where I want to do stuff, but with the TV on.
Chris: Yeah. Okay, Brent, what's your setup?
Brent: Well, I don't actually do dual monitor anymore.
Brent: I am now doing the tri-monitor thing. Chris, I know you're like a quad monitor
Brent: guy, but I did steal your little tip. You've been trying to get me to use this for years.
Brent: I have same as Liam here on the right-hand side, a vertical monitor.
Brent: It's just an old monitor I've had forever. it doesn't,
Brent: matter though because it's just chats or like currently
Brent: i have our episode recorder there i
Brent: have the window on which we're connected together
Brent: and then i also have the jb chat even though we're not doing
Brent: it live for some reason it just makes me feel better and it's
Brent: perfect for that and so i've got three monitors running usually for me off a
Brent: laptop and got this big massive 20 well i say massive although liam beat me
Brent: on this one but it's this 27 inch like 4k dell monitor right as my main laptop
Brent: as my main monitor in front of me and that's been a pretty sweet setup.
Wes: I'm on to him.
Chris: He asked.
Wes: This question because he wanted to.
Chris: Answer exactly and i'm here for yeah
Chris: i love it um you know i've been i know i mentioned this
Chris: recently on the show but i have i definitely have always
Chris: been a multi-monitor person for many many many years now but i
Chris: have been enjoying the single extra extra
Chris: ridiculously wide screen it's actually a curved wide
Chris: screen at home and uh and i use gnome there as well and i i really like i mean
Chris: you can you can lay out three full windows side by side on the screen it's just
Chris: really you get a lot of room and when you start getting that kind of room you
Chris: don't really need a second screen as much and i will say it is a much easier way to do desktop linux.
Wes: I do think that'll be in one of my future monitor purchases for sure.
Chris: If you go with like an intel or an amd gpu right now maybe one day in the video
Chris: But Intel and AMD and a single monitor, I promise you, your life will be easier
Chris: than if you try to be a quad monitor guy. It's not an easy path.
Wes: But what if I want to chain DisplayPort together?
Chris: Yeah. Okay. Yeah. Oh, man. And, like, I don't know what my deal is because I've
Chris: only been using computers for, like, 30-plus years.
Chris: But, like, I still have major strugs trying to make sure that the correct monitor
Chris: is the default, like, bioscreen monitor.
Brent: Oh, gosh.
Chris: And I don't know maybe this isn't how it works,
Chris: hand to the mixer if it doesn't change i get it just set up right and then like
Chris: a couple of days go by and the next time i turn the machine on like the different
Chris: monitors lighting up and i've gone through and i've i i thought i did the proper
Chris: ordering you know in the plugs and it changes i swear that happens i don't know
Chris: maybe maybe i'm making it up.
Brent: You need to like use uuids for your monitor setup.
Chris: Yeah well it's at the bios level like you know props to plasma by the time i
Chris: get to Plasma, it's always fine.
Chris: And I've been actually having really good success with locking my screens.
Chris: They go to sleep. They wake back up. Everything's fine. All the orientations are correct.
Chris: And when it boots, it always nails it. So Plasma has been just doing great with my multi-moder setup.
Chris: But my BIOS, right? Or like when the system's booting, the console where you're
Chris: just seeing the output, like that is a moving target for some reason.
Chris: And I don't think that's technically how this works but maybe it's me you know
Chris: moving monitors around i don't know you.
Wes: Could take the next year off.
Chris: And just.
Wes: Rewrite your bios.
Chris: Oh i thought maybe i'd do a study where i just empirically like write every
Chris: move i make down okay this one's in this plug boot yeah you write that step one i.
Brent: Think maybe you should study the cameras that you have in the studio because
Brent: every time i show up i tend to shuffle all your cables around for your monitors.
Chris: That is true that is true,
Chris: Okay, well, the pick this week has to be something that gets you up and running
Chris: with BPF right away, right?
Chris: Like, we don't want to sit here and talk about eBPF this whole episode and not
Chris: give you a great tool to check out.
Chris: And this week, it's Network Top. It helps you monitor traffic from your network using BPF.
Wes: Yeah, we should be clear. This one is classic BPF, but it's still handy.
Wes: And that means it is broadly useful, too.
Chris: Yeah.
Wes: It's built in Rust.
Chris: Oh, it is.
Wes: Yeah, so it's super performant. It's easy to use.
Wes: And it means you get to use like the TCP dump style syntax in a nice little Tui.
Wes: And so I sent you an example. What do you think of it?
Chris: Well, first of all, it's very readable, right? Because it is,
Chris: like you said, you're getting charts in a terminal UI.
Chris: And I mean, it's immediately understandable. I'm trying to figure out what you're
Chris: doing here because I see ginormous,
Chris: like you have almost no traffic and then it jumps up to almost 26.3 megabytes
Chris: a second and it hovers there for about 40 seconds and then it drops down to absolutely nothing.
Wes: Yeah so you'll you see how there's okay there's an input section at the top
Wes: and then under that there's rules yeah you'll see what it says all on the left
Wes: and then to the right it says host and then a specific ip address.
Chris: So you're you're looking at all traffic from just this host.
Wes: Yes um and so those are two separate rules so basically it's it's looking at
Wes: one interface and just watching it totally by default and then you can add bpf
Wes: rules that it'll add as additional things that it'll monitor at the same time
Wes: and then you can use the arrow keys to toggle between the views for different uh rules okay and,
Wes: Um, there was no traffic because, uh, I SSH to another machine here at the studio.
Wes: We hadn't really been talking, right? I exchanged a few packets for that, but that was all.
Wes: Um, and then I did a net cap back to my laptop, just reading from random.
Wes: Uh, and so that was the big traffic spike. And then I killed it just to watch it all drop off.
Wes: But you know, like it could, you can do, you can do like specific ports that you want to watch.
Wes: You can do specific IPs, host names, TCP states, like whatever you can do with
Wes: TCP dump. Basically you can put those rules in here.
Chris: And you're getting a command line TUI, a terminal user interface.
Chris: So it's really easy to understand. Like, these rules Wes are talking about are
Chris: clearly delineated in the UI. It's really simple syntax.
Wes: So maybe there's something you're trying to hunt for. You find it here,
Wes: and then you can go do the TCP dump to capture the traffic and do some more analysis.
Chris: And it is MIT licensed, so it is free to use.
Chris: And then there is a bonus pick, BPF Tune.
Wes: Yeah, well, you kind of hinted at this a bit. And I happen to know you turned
Wes: it on for at least one of your systems.
Chris: I have it running on my home workstation.
Wes: Despite the fact that this is indeed a GPL 2.0 with Linux Syscall Note licensed
Wes: Oracle open source project.
Chris: BPF Tune aims to provide lightweight, always-on auto-tuning of system behavior.
Chris: The key benefits it provides is by using BPF observability features,
Chris: it continuously monitors and adjusts system behavior. because we can observe
Chris: the system behavior at a fine grain.
Chris: We can then tune at a finer grain too. So like individual socket policies,
Chris: individual device policies.
Wes: Yeah, right. So think of all those things like CTL can tweak.
Wes: It's using BPF hooks to watch your system and then automatically go tweak those.
Chris: Yeah. Now I haven't really noticed a difference, but I've only had it on for a couple of days.
Chris: But the idea is brilliant. It's just so brilliant.
Chris: Like here's the system. I'm monitoring. OK, now I'll just go make adjustments
Chris: here to make essentially maybe like ease pressure on the system or something like that.
Chris: I don't know if anybody has experience with this, because I just started,
Chris: but it seems like a fantastic idea.
Chris: BPF tune. Chris will have a link to these in the show notes.
Chris: And in Nix, it's like just one, it's like, you know, service enable BPF auto
Chris: tune. And then you just, you're good.
Chris: There's more you can do, but that's, it's really simple. So I was like,
Chris: all right, I'm just going to turn this on.
Chris: And I just love the concept of the system auto tuning itself.
Wes: Yeah, and they kind of make the case here too, if you're doing the cattle not
Wes: pets approach and how many of your systems ever really even have a human person
Wes: who might be on it who could tune it.
Wes: Okay, maybe you can hand-tune the database, but you're not going to hand-tune
Wes: all your dynamic web workers.
Chris: This could be great for my Odroid.
Wes: But if some of them are longer lived, this king can make sure they're running all right.
Chris: I should put this on my Odroid. I really should. Little Odroid's doing a lot
Chris: of work and I never really check in on it. Just sitting there being a little yeoman about it.
Chris: Well, we will have links to this and everything else. There's a lot of links this week.
Chris: Linuxunplugged.com slash 605. And remember, we want to know if you enjoy these deep dives.
Chris: We can get into the weeds here a bit too much, but if this is the kind of stuff
Chris: you like, let us know because there's, it's honestly for us as creators,
Chris: it's a little, creators, it's a little scary to do topics like this.
Chris: I know that sounds stupid, but it is. It's just a little scary.
Chris: So we always like to know what your thoughts are.
Chris: We'll be back at our regular live time.
Chris: Tuesdays as in Sunday Sunday at 12 p.m. Pacific, 3 p.m. Eastern.
Chris: Now if you want more show remember our members get the full bootleg which is
Chris: clocking in at like an hour and 20 minutes or something right now and this is a short one,
Chris: and of course you can get details of that at linuxunplugged.com Thank you so
Chris: much for joining us on this week's episode of the Unplugged program we just
Chris: really appreciate your time for listening if you want to share it with somebody
Chris: we always like that too word of mouth is the best advertising for a podcast
Chris: we always appreciate that,
Chris: thank you so much for being here and we'll see you right back here next Tuesday
Chris: as in Sunday which isn't Tuesday at all.