Welcome to the NGI Zero Podcast where we talk to the people who are building the next generation
internet.
Hi, Ronny Lam.
And I'm Tessel Renzenbrink.
We're both from NLnet, a foundation which supports people who are working on a free
and open internet.
For season two of this podcast, we will be focusing on digital sovereignty through free
and open source technologies.
Our guests today are the team behind LibreQoS, a Quality of Experience & Smart Queue Management
system designed for ISP networks.
We have three guests and that's Robert Chacón, he's the chief executive officer, Herbert
Wolferson, chief product officer, and Frank Borsik, who is the chief operating officer.
Welcome all and thank you for joining us in this podcast.
Thank you for having us.
Yeah, thank you.
Hi, welcome all.
LibreQoS started during the COVID pandemic, right?
Can you tell us the origin story?
Sure.
So, I operate a small ISP here in West Texas and Southern New Mexico here in the States.
And we primarily service rural folks, low income families, families that otherwise wouldn't
have connectivity.
So, we noticed a trend during the COVID pandemic that usage went through the roof.
A lot of access points and choke points along the network were experiencing intermittent
packet loss, which is a phenomenon known as bufferbloat.
And it was causing some issues in terms of people's ability to do online learning, people's
ability to do work from home.
And we realized that we needed to do something and it wasn't practical to swap every router
on the network and put FQ-CoDel on it.
So, instead, we started working on a script that just basically did queuing across the
network and it implemented FQ-CoDel, the algorithm that was co-authored by Dave Täht, one of
our team members.
And it was greatly helpful in terms of reducing the jitter and latency and the buffer bloat
on the network.
So, we noticed that gamers stopped having issues, people who work from home are having
more stable experience.
It was helping with VoIP calling and things like that.
So, we decided to kind of expand on that script a little bit and then we open sourced it.
So, we uploaded it to GitHub and then the whole community has formed around it since
that time.
Herbert has joined us, Frank's joined us, Dave, one of the original authors of the algorithm,
has now joined us.
And it's been a really rewarding experience to see it help ISPs around the world.
But you see, you say that you use it for people in their homes.
So, where do you exactly place this?
Is this in the router in the home or is this in the router at the ISP?
That's exactly right.
It's sandwiched between two routers at the ISP level.
So, typically, an ISP or a smaller ISP might have, for example, a border router, something
that does BGP, NAT, firewall, things like that.
And they might have a core distribution router that then sends the traffic to the different
sites, whether those are towers, OLTs, things like that, that distribute across a fiber
access network or across fixed wireless.
So, it's basically an x86 appliance.
It could be any server or desktop computer that has an x86 CPU.
And it just sits as a sort of transparent bridge between the border gateway and the
core distribution router.
And you had to do this, of course, at every core distribution router.
Yes, I mean, typically for most smaller ISPs, there will only be one.
So, there's only like in the case of my network, there's only one LibreQoS box.
But it's able to shape traffic for up to tens of thousands of users per box.
So, we see some networks now that are deploying LibreQoS at up to 100 gigabits per second.
Other networks are much smaller and might only need a few hundred megabits.
But it scales up to that point.
Amazing. And the box is still there?
Yes, the same one since we deployed it has been running smooth.
It's given us an incentive to really optimize the code on our end so that, you know, as the
network's grown, we've been able to keep the same equipment.
But a lot of smaller operators will, for example, use an old server they have lying around.
There's some in the African continent that have been extremely resourceful and found
equipment that we didn't expect to work, but it did.
For example, CTN networks in Malawi has had great success with it.
But it's been really, really encouraging to see people use hardware of all types to deploy it.
Yeah, exactly. I was thinking about the African continent a friend of mine.
He's delivering internet there for years now, and they have to do it with whatever network they can
get access to to connect to the internet.
And solutions like this really help to share that internet with everybody, I think.
Absolutely, especially in Africa.
I mean, the connectivity costs there are just astronomical compared to what we're used to in
North America or in much of Europe.
Many folks there will pay, I think it's in the ballpark of, you know, two, three thousand
dollars USD per month for 200 megabits of connectivity, something that just wouldn't
make sense in other markets.
But there, you know, maybe it does make sense.
So they need to really fairly distribute that traffic.
If you only have 200 megabits among thousands of users, it is viable, but you have to have a
queuing mechanism in place to make sure that traffic is fairly distributed and that one user
doesn't hog all of the traffic.
Say a video game or something, right?
So that's that's what LibreQoS offers.
So those networks, I think, are able to be not just more profitable and to scale further, but
also to make sure that everyone has an equitable access to technology and especially folks that
depend on it for things like not just work from home, but also telemedicine.
We have a number of clients on my own network that are permanently disabled and cannot leave
their homes very easily.
And they heavily depend on telecommunications for their ability to see
their doctor for routine
checkups.
So I'm sure that that applies to many ISPs around the world that have those kind of clients.
So we hope it's helping to improve equity around the world.
We would be happy to connect with your friend, Ronny.
Yeah, sure.
Let's do it afterwards.
Yeah.
And that was really what brought me on board was I saw Robert's script being talked about.
And the ISP I've been working with in central Missouri is kind of unusual that we also are very
community-focused.
We specialize in trying to get wireless service out to the parts of Missouri that nobody's going
to run fiber to, for example, in the middle of a swamp.
We're never going to be particularly profitable because there's frequently one house per three
miles.
But somebody's got to help them.
And a lot of them were really struggling, especially in the pandemic, because suddenly
kids were supposed to do homeschooling.
And I kid you not, they still have party lines, the old phone system where you pick up a phone
and can hear neighbors talking in some of these neighborhoods.
So getting any sort of wireless out there was a game changer to begin with.
But then LibreQoS, I already knew about FQ-CoDel.
I'd used it in some other projects.
I was super happy to see that Robert was getting Cake, which is an improved version of the same
thing, in.
So one day we put it in place, tried it out, and discovered that the number of people calling to
complain went down by a massive amount.
And that was enough.
I'm one of those weird obsessive programmers that once I've got a good idea, I love to keep
writing code.
So I ended up porting a bunch of it to Rust, got in touch with Robert, merged it, and we
combined our efforts to turn it into something a little more friendly than a giant script.
I've just been blown away seeing how much this is helping around the world rather than just in
my little neck of the woods.
Maybe this is something that you don't want to share, but how many instances are running now?
We can definitely share it.
The number is huge, and it's north of 500 ISPs so far.
And it can be even more, but we don't know for sure.
Because prior to the current versions, ISPs were able to not share any information with us.
So there were installations in the past.
So, but it's growing.
On the market, there are basically five other competing solutions.
We are the only one that's open source.
We are probably the second biggest right now, as for the number of customers.
We're in what, 44 countries?
No, no, no.
It's actually 58 countries and 28 US states on top of that.
It's even more right now because these are the numbers from December of last year.
Wow.
Didn't you also claim to be on seven continents?
It's close.
So there is no LibreQoS instance at Antarctica yet.
Not yet.
We are getting there.
Because in January 2026, there will be a first Antarctic Network Operator Group meeting.
So we hope to go there.
That would be amazing.
It would, yeah.
I would like to go back to what you said, Herbert, about small communities,
underserved communities that have no internet or less good internet access.
Can you, if I understand it correctly, LibreQoS really focuses on that demographic.
Can you explain one of you why you find this so important?
Yeah.
Oddly enough, we started with that as a goal and discovered it worked well for big networks too.
So we have ended up helping a bunch of big networks as well, but we,
you know, our hearts are very much with the small underserved.
People who need the help.
The main problem is you have a limited supply of internet out to the demand that can be either
geographically very spread out, in which case you're using wireless links that may not be the
most reliable because they get less reliable over distance.
Supply can be kind of limited sometimes because none of the big companies really want to spend
what it would cost to run fiber out to be local to these remote areas.
And, you know, when you talk to some of the providers in Africa,
they have really long hops to get out to an internet exchange to talk to the rest of the world.
But even in Missouri, which is far more developed, you are remarkably long way from a good IX
in physical terms when there's something like a swamp and a mountain between you and where the
fiber is because you either have to find a way to run the fiber all the way around or do what
one provider did here, which was run fiber through the swamp and replace it every year
until they went bankrupt, which didn't work out.
So the problem is you've sort of got these automatic choke points that are in place
and demand on, you know, pretty much every ISP oversubscribes.
And if you always had exactly as much capacity as all your customers could use if they all went
full speed at the same time, you'd be bankrupt because you'd be buying huge amounts of capacity
and never using most of it. But oversubscription gets kind of nasty when the bottleneck is how much
you can physically acquire as opposed to how much you can afford to acquire.
So if you've got a limited amount of bandwidth to divide out, what Cake really shines at is
optimizing the experience for the people who are using it, giving them what they can.
And it focuses heavily on identifying interactive sessions. So for example, your Zoom call,
your Voice over IP phone call gets significant better treatment than little Jimmy in the basement
deciding that it's time to download the latest game from Steam. And because flows get
independently tracked within each client and then tracked relative to one another, you get this
wonderful setup that, you know, Jimmy in the basement doesn't ruin your business meeting
by starting the download because relative to Jimmy, your interactive meeting flows get prioritized.
But also in the grand scheme of things, if you're overprovisioned, you're trying to use
more total bandwidth than you really have available, it penalizes the big bulk flows
far before it penalizes the interactive flows where you notice. So your videos don't skip
your meetings are good. You can still watch YouTube, Netflix, whatever streaming service.
But people's downloads do slow down very slightly, but you don't really notice if a 10 gig download
is going to take five minutes longer. Whereas you absolutely notice if the video meeting you're
having is jerking, Zoom is popping up the famous your internet connection is having troubles
message. That's what generates calls. That's what makes people unhappy. And that's what makes things
unusable in these underserved areas. Does that make sense? It really does. Yeah. And this is also
an advantage for gamers, I think. Absolutely. We have some folks on our own network that actually
make their living off of Twitch streaming and video gaming. For example, there's one person that
has a testimony on my ISP's website. They do a 24 hour live stream of a Turtle Tank. And that is
their primary income because it's a very popular stream. But they also do video gaming on Twitch.
And they're able to use their connection to its fullest extent, even when they're completely
maxing it out with multiple streams on top of each other, because of the Cake algorithm,
because we're able to use that. It really provides like Herbert's explaining the
fair priority in terms of latency to the applications that need it while kind of
setting aside the ones that maybe don't bulk flows, things like a torrent download or something like
if you're downloading maybe a Linux ISO or something that doesn't need to be
prioritized in terms of its latency. But it can have a huge impact in terms of someone's income
or if they're doing telemedicine and anything like that, it's a really big impact.
Yeah, exactly. Wow. So that's for the smaller ISPs. We are currently onboarding the biggest one so far.
And they have 800,000 plus subscribers. And they are covering something like eight US states.
So this will be the latest addition. That is good. Yeah.
Rumor is that there is Klingon inside LibreQoS.
That's actually Dave's fault, but I implemented it for him. It's not in the most recent version
of the user interface because we got fed up of so many people calling us and asking us what the
heck that was. But sharing screenshots with all your ISP customer data on them is really painful.
So we added a little thing that redacts customer information so that you can take screenshots.
And version one of that translated everybody's name to a hash table and then pulled Klingon
phrases out saying things like your brother has a smooth head.
But it's not there anymore. It's not. Although I've had enough requests to bring it back
that it'll probably reappear pretty soon. But I also like that people are calling
what is this on my screen? Maybe you should put it in there for that.
Yes. Yeah, those people have to be educated.
I mean, yeah, we were trying to explain it on social media, but not everybody read it.
Yeah, I guess I mistakenly thought that everybody who wanted to use LibreQoS was
going to be on the Star Trek. Disappointment. Disappointment.
I was wondering, could you paint a picture of the fields that you are operating in? And can you tell
us something about your competitors? Sure. So there are a number of, as they're called,
Quality of Experience software vendors that produce software somewhat similar to LibreQoS.
Many of them use a lot of the same tools that LibreQoS is based on. So LibreQoS is based on
the Linux kernel. It's based on additional features such as Linux traffic control system,
eBPF, XDP, things like the HTB, the hierarchical token bucket. So many of the other solutions
leverage a lot of these same open source tools, but are proprietary and closed source. So a big
difference with LibreQoS is that we are contributing back upstream. So we're contributing
back our contributions to the Linux kernel, to the XDP, eBPF projects, and other projects in order
to make sure that the benefits of not just fair queuing, but the ability to move traffic around
the Linux kernel become available to other projects. So we're really making sure that that's
available to others. But a lot of the computing solutions include, I believe, Backwind, Preseem,
Paraqum, and there are a few others out there that do similar work. But we're the only open
source solution at this time. We're the only pronounceable one too.
If you don't do it in Klingon, but yeah. Why did you choose to open source LibreQoS?
My dream when we open sourced it was, I mean, it was maybe ambitious, but my hope was one day
someone in a remote part of the world is going to download this and it's going to have the same
impact for my ISP or for their ISP that they have for my ISP. It's going to have some impact that
allows them to grow and to compete against these huge monopolies that exist in their markets.
I didn't know that that would ever happen, but I thought, let's take a chance. There might be one
person out there who will run this script. Instead, it's been a completely different story than what we
expected. So we're not just seeing one or two ISPs deploying this. It's in the hundreds. Most
recently we found in the Philippines, especially there are hundreds of ISPs that are very small,
one to five operator ISPs that are deploying the software and doing workshops. We actually found
out after the fact. So we saw some photos posted online of these workshops with 20 plus people all
wearing LibreQoS shirts and we're like, what's going on here? So it was just a very welcome thing
to see how this is helping their ISPs. They're spreading the word. They're doing these workshops
on their own initiative and helping these other ISPs in their same country to benefit from it.
So I think it's been great to see that kind of coming to fruition in a way we couldn't anticipate.
Wow. That's an amazing story. So because LibreQoS is also a business, so it wasn't planned to be
a business, but it got so popular that you made it into a business or how did that work?
Yeah, I think originally, I mean, the idea was just make this available to everyone. And then
over time, you realize that there are some inherent support needs and you need some full-time
developers on this project to really do what it warrants. So we decided to try to find some way to
thread things to where we could both support the open source side of it and then also support the
ISPs that are larger in scale that want to use this. So we've added an optional feature. It's
the LibreQoS Long-Term Stats or LTS. And that's basically an optional add-on that allows you to
track statistics long-term in a cloud-hosted way or in the next few months, it'll be available on
premises as well. But it's completely optional feature. The core fundamental technology that
we provide, the queuing, the traffic shaping, the accounting, all of that is fundamentally free and
open source in GPLv2. So we found that that was a compromise we could make that would allow us to
continue to support these efforts around the world and have a nice little extra feature for
the larger ISPs that might want to make use of it. And in the end, there will be more competitors
that will arrive to the market because it's really an lucrative market. There will be a lot of money
to be earned. And we understand that we need to help so many people and money will help basically
to do it. So Robert, for example, can make a choice to give the paid product that we have
for free to someone that really needs it. So we have this kind of control and we can do
these kind of things in order to help people in some countries.
Yeah, we've also talked about trying to keep the pricing low enough that we're effectively keeping
the other players in the market honest. Not necessarily a nice thing to do in some ways,
but I firmly believe that the whole market, much of the internet needs this. It's a good idea to
make sure that it's widely accessible even if you don't choose us. So there's something to be said
for that. My wife told me to stop feeling guilty about needing to eat. So that brought me on board
with the idea that not everything can be open source and I can't give everything away.
So my take on it is when we first tried talking to big rich ISPs about a pay it forward,
subsidize the people who don't have anything type model, that got nowhere. Nobody liked that and we
were almost ashamed that we were getting more donations from Africa than we were from the
United States at one point. So we're trying to work around that and make sure that we can at
least survive in a way that we're comfortable and still give everything away to the people who
really need it. And the core is absolutely going to remain free. Even if we ended up rewriting all
of it, I'm pretty sure core version two would be free because it needs to be free if people need it.
Do we have any other plans for the future? Well, right now we're working on version two of what
we're calling LTS. It's going to be called Insight and it's a very exciting product. I think people
will be excited to see it, but it leverages all of the queuing and the traffic analytics that we're
doing in LibreQos to provide network operators with a lot more insight in terms of what's happening
at a high level in their network. We think it's something that's going to improve the,
not just the usability of the software, but reduce the time it takes to troubleshoot a customer
issue. So for example, if a customer calls in, you basically press a button and it tells you right
away what's going on for that customer's connection. So those tools are meant to help operators as they
scale to be able to save time and to really make sure their customers have a great experience.
And again, the core product stays free and open source, but we have a lot of plans in terms of
improving the product. We don't want to give too much away, but Frank and Herb, are there any
particular parts you want to add to that? What I'm kind of proud of is we've been deliberately
skewing the stats we gather towards what's requested by some of the much smarter than me
doctoral people who work on the core of Cake and future queuing standards to try and give them the
data they need to advance the field as a whole. And quite a lot of them are tied into Linux Kernel.
So I'm hoping that by gathering this giant amount of data, part of what we can do is make life
better for everyone by giving, I'm not sure if I'm NDA'd, so I'm trying not to say their names,
by sending the data in that direction so that they can work on it and do research. And I find
that exciting. I'm not smart enough to do the research. My focus is very much implementing
the cool stuff they come up with, but if they don't have the data, I don't think they could do that.
And it's just exciting to me that we're potentially improving everything.
Yeah, exactly. And it would be very cool to have insights into what LibreQoS is actually
doing in your network and how it is improving the network connections of the end users.
Yeah, I mean, it makes a huge difference in terms of just the day to day operation of an ISP. I think
when I first started out, we had a huge difficulty with customer calls and has a problem. Okay, what
do I do? A trace route? Do I do just ping test, ICMP? It really was not very intuitive to
troubleshoot any issues. I mean, at this point, if you run LibreQoS, just the open source component
of it, you're able to click in on a customer's profile and see all their traffic flowing by you
in real time. You're able to see, okay, well, they're connected to this ASN. The round trip time to
this ASN is higher or lower than the average. Maybe this entire class of traffic is having trouble.
It can really make a difference in terms of the subscriber knowing that you can see what's
going on in a way that helps them. You can troubleshoot in a lot more proactive ways.
We think it really is helping, especially in parts of the world where maybe latency is
varying quite a bit. For example, the Asian continent and many parts of Africa, there's a
tremendous difference in latency between endpoints just because of constricted back calls and fiber
links. We think it helps a lot there, especially in South Africa. We've seen a large benefit to
people being able to see those insights, to be able to see that maybe their connection to
cloud fliers taking the wrong path, that kind of thing. Those insights can really help to improve
the customer experience. Oh, goodness. Do you want to share the story about the
fiber cuts to Africa and the sheer number of messages we suddenly had?
Yeah. It thankfully kept the ISPs that already had it deployed very happy. The ones that were
seeing that all of the bandwidth across the board was dropping to really abysmal levels were able to
continue to run. Even on their 200 meg, 500 meg links, they were equitably distributing the
traffic among all clients, which is something that even if you had more bandwidth, wasn't
necessarily happening when that fiber cut affected you. The fair queuing, I think, really saved the
day for a lot of those ISPs and we got a lot more emails from folks that wanted to try it out.
If there is one continent that's not performing well, that's Europe.
We don't have so many ISPs from Europe and it's really sad for me because they need help.
We see so many speed tests. They are really terrible. We can see high latencies and people
think that when they are deploying fiber, then everything is solved, but that's not true.
We see fiber ISPs in need to use LibreQoS or any other Quality of Experience solutions out there.
We need to get more Europeans on board. That's kind of interesting too. If you look at our
approximate map for where we've had anonymous stats reported, we've got a pretty solid wall
down Eastern Europe. A few in the periphery. It just seems like the middle doesn't like us.
That's because we don't provide native translations. Nobody's heard of us. I don't
know. For some reason, we're very much in a circle around Europe, even including small
islands off Scotland, but not so much in the middle. Basically, the biggest number in Europe
is in Poland, then Czech Republic, but not many in Germany. In Poland, you have so many fiber
and they still need help. It's really interesting. This leads me to the question because Robert and
Herbert, you both run an ISP. Can you explain something about the field of ISPs? What is it
like? Are they monopolies? Are they playing fair? Are they being nice to their customers?
I would say that's the whole reason I started my ISP. I was a student at the local university and
I always wanted to do something to compete against these big monopolies. At the time,
in my city, there was effectively one operator that controlled 90-95 plus percent of the market.
A lot of elderly folks, for example, would be approached by a sales representative and told,
okay, well, you can have this internet service for $50 per month. Sounds great. They weren't
told the asterisk part, which is that after, I think, 12 months, their rate would go up,
not just for the internet service, but for everything. It would go up by not just a few
percentage points. It was going from, for example, $50 per month to up to $270 per month.
Those folks are typically, maybe in my region, not fully English speaking. They maybe are
transplants from other countries. These folks are very much taken advantage of.
I think seeing that was really disheartening to me. I knew that a lot of elderly folks,
especially who were on fixed incomes, were going to be affected by that. I knew some personally.
We decided to start an ISP that would compete against those approaches to business. I think
we've succeeded in our market because of that. I think we've shown, built trust with the community
and shown that you can offer internet at fair rates. Often, just because a region has connectivity
doesn't mean that it's accessible or affordable. I think that's a really big dimension of things.
Just like Frank was saying with having interchanges and exchanges across Europe,
it's important that we have those across the US and across the entire world because they improve
the competitive potential of people in those markets. If there's no IX, there's really no
opportunity for someone to start an ISP like we did. We think that it's important not just to
have a third or fourth option, but also because many of the first three options that people have
are not necessarily within their financial means. I'm going to try really hard not to get myself
deported here. Missouri is a little strange. I live in Columbia, Missouri, which is a city
slap bang in the middle of the state. My city was one of the first to build a fiber municipal network.
We actually built out fiber to every house within city limits. A week before it went live,
the state government prohibited municipal networks and the whole thing has been rotting,
never turned on ever since. They waited until it was built too, so they've spent the money.
The fiber is still there. It just doesn't go anywhere. The state is pretty much divided between
every area that has one cable, has one of two cable providers and one of two phone companies.
They very carefully demarcate where their zones are, so when you move into a house you can pick
one of those two. There's plenty of small ISPs filling in the gaps and they're absolutely vital
because the telcos have been extremely selective. They won't build any of the good stuff out to
rural areas where density is below a certain amount, but they're also very selective in towns.
In Columbia here you can get two and a half gig fiber from one of the phone companies on one street
and not the street adjacent to it because one street everybody's middle class, affluent.
The other street is public housing and doesn't have access to this kind of thing. So my ISP,
guess which of those two streets we serve? It's not the one with great internet. Somebody's got
to do it though because they've got kids, those kids are in school, those kids need to learn.
So I like to say that ISPs, especially the small ones, are the salt of the earth because they are
really delivering something like, you know, internet today is something like water or electricity like
50 years ago or 70 years ago. Yeah, it's really interesting to hear that, well,
both these stories that you tell are quite gruesome.
Yeah, to some extent even some countries in the global south can have it better than the
developed countries. People can't believe that, but I was in Africa with LibreQoS for the first
time last August and it was visible. So even though they don't have so much, they can work
with it and provide better service. One of our customers can have better service with
really small internet packages than the fiber ISPs in the US and it's crazy.
Yeah, you mean because they actually try to provide good internet rather than just make money.
Yeah, and they are trying to leverage the tools, right? For example, LibreQoS.
But what can regular people do in their homes, in their offices to get rid of
bufferbloat, latency, jitter? There are a few options. For example, many folks will run OpenWRT.
Dave's a big proponent of it and OpenWRT is firmware that you can load onto many recent and
older model routers. It allows you to run the algorithms such as Cake and FQ-CoDel that we use
for LibreQoS and it can make a big impact. It's not enough on its own in some cases. There can be
congestion just on the network level for your ISP, but it can make a big improvement for many folks.
So we definitely recommend that folks flash their router firmware with OpenWRT if it's something
that's possible. And there are also non-open source solutions that can really help. For example,
simple thing is to have a MikroTik router and you can turn on SQM, which is Smart Queue Management,
and you will have this solved. And there are other open WRT-based companies that are selling
routers. So you can really get quite a lot of solutions these days. And like as of late,
there is a US company Alta Labs and they implemented Cake. So that's another
good router company that people can get if they don't feel like working with OpenWRT on their own.
Cool. Yeah. That's good information, I think. And also let's try to get Europe more educated
on this for you. Yeah. So we tried at FOSDEM this year, thanks to NLnet. And I was surprised to see
many, many people coming to me to speak about us. And they said they are following LibreQoS online
and they are trying to talk with their ISPs and everything. So we see it growing also in Europe.
Oh, cool. So people are asking their own ISPs like, can you install LibreQoS? Oh, that's the best way
of getting them to know you. From the bottom up? Yeah, exactly. So I was also wondering,
because LibreQoS received NGI Zero funding, how did it help you to improve it?
I should probably answer that because I think I received a fair chunk of it. Where it helped me
was it allowed me to set aside a few weeks at a time to do a deep dive into a very necessary
piece of software improvement, as opposed to trying to fit it in between day job and playing
with my wife and three-year-old. I could actually set aside whole days, work uninterrupted, and know
that I was still going to be able to pay the bills. That makes just such an enormous difference.
All the proposals have been very targeted that whatever we request funding for is going into
the open source section. It'll never touch the other part. It'll be given away free and it's
broad enough that it's going to help a lot of people. For example, in the next funding round,
we have a pretty ambitious target to create hitless rate changes. What that means is,
through the API or the interface, you can change details about somebody's connection
without causing their connection to burp at all. Other than potentially some discards if the speed
limit goes down. I'm not a magician, but I can make it a lot better. It's a hugely requested
feature, partly because some of the big ISPs we've talked to, like one in Poland,
was complaining that sometimes it can take three seconds to change everything on his system.
During that three seconds, his friends would call him. His friends obviously dial faster than I do,
but I'll believe him. Implementing that is going to be a couple of weeks of hard work.
We're not listing the planning stuff. We've got it pretty much planned out how to do it. It's just
physically a matter of having the time to do it and knowing that I'm still putting yummy food
on the table and can relax and not stress out about trying to fit it in between other things.
I'm hugely grateful. It's been an absolute game changer. It's really helped LibreQoS because
suddenly we have these nice targeted items. We have some funding for it. We can go out and do it,
make it happen. Otherwise, we'd probably still be tugging along trying to get them in the spare time
and probably two, three years behind on where we are now.
Yeah. Thank you because this is literally you're making the internet better on a very low level.
It's really useful what you're working on. Thanks a lot for that.
Absolutely.
Yeah. Also, I was always thinking and maybe that's the thinking throughout the whole of Europe
is that bandwidth is free. If you have problems, throw more bandwidth at it.
Yeah. It's a misunderstanding of I think that the access technology, even if you have a fiber
to the X location type access network, it doesn't necessarily mean that you don't have choke points.
If you have an OLT that's only able to provide GPON type speeds, you're going to hit that gigabit
limit. Even if you're providing gigabit plans in theory, most of your subscribers are not
actually subscribed to that. They're typically subscribed to the 100 meg, 300 meg, 500 meg plans.
They can easily saturate that link with a single download of a video game or something in the
background or a Linux torrent, whatever it happens to be. That's something that's not really
considered. It's thought, okay, well, that's too high of a limit. They'll never hit that limit, but
we actually can saturate those limits. Things are well enough interconnected that you can.
Those choke points exist on any network, whether it's a small fixed wireless network in
Malawi or a larger fiber optic network in Western Europe. It's something that we need to address,
whether it's through LibreQoS or other means.
I think those are all the questions we had from our side. Is there anything that you would still
like to share with us? Well, something I wanted to say was to thank NLnet for all of its
contributions. Like Herbert has explained, we have many large projects that have to be done,
especially on the open source side, that are not really viable in the timescale that we would
otherwise fit them into. For example, the rewrite of the UI, which has really greatly improved
accessibility, especially for ISPs in the Philippines and in the developing world,
all of that was made possible by NLnet's contributions. We're extremely grateful for
that. It's been interesting to see how those things that we thought might have been
interesting or helpful have ended up being really critical. For example, the ability for people to
change their settings through the web UI has made it much more accessible to deploy this software,
even if you're not a seasoned Linux administrator. That's something that really opens it up to many
small ISPs around the world. Those are things that maybe NLnet saw before we did in terms of the
impact it would make around the world. We're grateful to see that coming to fruition and
helping all these small ISPs. We thank NLnet for all of its contribution.
Besides actual ISPs, LibreQoS found a way to improve internet also in other locations. For
example, we have helped a company that's giving connectivity to cruise ships and other vessels
already. That's a really interesting vertical that we hope to get in and help even more.
Also, there was a deployment in mining already in Finland. That was also really nice.
There are other use cases. We are told that people are using LibreQoS at LAN parties.
You know, LAN is coming back. People want to meet and play games. This is really cool.
Also, there are university campuses, for example. That's a really nice use case.
If your ISP is not cooperating, you need to fix the connectivity for a few thousand people
that are living and breathing and doing hard work at the campus. That's also something else.
We are always looking for new use cases that will be really helpful to the people.
I just wanted to mention you guys have the coolest stickers. Frank mailed me over a bunch of the
stickers you guys put together. I've been giving them out. My daughter thinks they're the best.
I have to admit a fair number of them have ended up splattered over my daughter, her stuff,
and our fridge. They are really cool, so thank you.
You're so welcome.
Yes. This year, we got two colors, which was a really nice touch from NLnet.
That's really great. Thank you so much for all these nice things you say. It's nice to hear that.
Thank you even much more for building LibreQoS and continuing to do that and helping people
to get better internet. I really like this story, so thanks a lot.
Yeah, thank you.
Absolutely.
Thank you.
Thank you all.
Thank you all much, guys.
Okay.
The end.
You make me think, Robert. Have you said thank you once?
Well, we are appreciative. What Frank was mentioning, this is not just helping the ISPs
like for the mining operation. They were finding ways that before you had to be following these
huge mining machines, miles underground or whatever, with a fiber optic cable and a little
control panel. If the machinery crushed something or it crushed the support and it collapsed,
you were at grave danger. Now, they're actually deploying it so that LibreQoS is keeping the
latency low enough that they say, okay, fine, we now trust that you can operate this machine from
miles above on the surface. All the risk is gone for a lot of these operators. It's potentially
saving some people from unnecessary danger. There's all kinds of use cases like this around
the world. That's why we're so appreciative of NLnet. It's not just going to help ISPs or people
who are gaming or something. It's going to help my client, the one that's paraplegic who depends on
telemedicine for 90% of her doctor's visits. It's making an impact. We just appreciate everything
that you all do to help move forward this mission and to make sure that these proprietary solutions
aren't the only option for people. Regardless of your ISP size, you can help people with this.
The other thing, basically, NLnet was one of the reasons that we were not forced to take
VC money, for example, because we were approached quite early. We were approached by VC funds
because they see that there is a chance to earn a lot of money in this space.
We can stay private. And honest. NLnet's allowing us to stay honest, right?
And honest, yes. If you get VCs in, the honesty changes.
It's a lot of pressure. We've seen companies like, I think,
ZeroTier. There's been companies that have just privatized their own open source
solutions that were developed by the community and we don't think that's the right thing to do.
You see these other solutions that we compete against, like Preseem and Paraqum. They're all
based on fundamental technology that was open source and none of those contributions are going
back upstream. So, NLnet's making sure that all this goes back upstream, like with Toke's work and
all the folks at Karlstad University that are working on the kernel. We're just really
grateful that this ecosystem can be fostered and it's thanks to you all.
Great, thank you.