1
00:00:00,000 --> 00:00:10,800
Welcome to the NGIZero podcast where we talk to the people who are building the next generation

2
00:00:10,800 --> 00:00:11,800
internet.

3
00:00:11,800 --> 00:00:15,080
I'm Ronny Lam and I'm Tessel Renzenbrink.

4
00:00:15,080 --> 00:00:19,760
We're both from NLnet, a foundation which supports people who are working on free and

5
00:00:19,760 --> 00:00:22,040
open source technologies.

6
00:00:22,040 --> 00:00:26,320
Our guests today are Julien Malka and Camille Mondon.

7
00:00:26,320 --> 00:00:32,080
Julien is a PhD student in Software Supply Chain Security at the Polytechnique Institute

8
00:00:32,080 --> 00:00:37,520
Télécom in Paris and he is also a NixOS contributor.

9
00:00:37,520 --> 00:00:43,360
Camille Mondon is a PhD student in Statistics at the Toulouse School of Economics.

10
00:00:43,360 --> 00:00:47,720
Together they work on a project that implements Clevis in NixOS.

11
00:00:47,720 --> 00:00:52,600
It makes it possible to remotely reboot servers with full disk encryption without needing

12
00:00:52,600 --> 00:00:54,920
a human in the loop.

13
00:00:54,920 --> 00:01:01,200
That project was extended to include Proxmox NixOS which we'll also be talking about today.

14
00:01:01,200 --> 00:01:05,280
Hi Julien and Camille, welcome.

15
00:01:05,280 --> 00:01:07,800
Hello, thank you for having us.

16
00:01:07,800 --> 00:01:09,560
Yes, hello.

17
00:01:09,560 --> 00:01:14,320
First of all, did we miss something in our introduction?

18
00:01:14,320 --> 00:01:17,880
No, that was quite perfect, yes.

19
00:01:17,880 --> 00:01:22,920
So what key issues do you see with the state of the internet today?

20
00:01:23,920 --> 00:01:31,720
Well, today I guess one of the biggest key issues is that the internet is quite owned

21
00:01:31,720 --> 00:01:37,680
by a monopoly of Big Data Corporations.

22
00:01:37,680 --> 00:01:47,720
And basically it makes it difficult for people like us that have particular interest in computer

23
00:01:47,720 --> 00:01:55,080
science and internet to make a living out of it while staying in the open source community.

24
00:01:55,080 --> 00:02:03,760
So I guess it makes it very important to have funding, public funding and institutions that

25
00:02:03,760 --> 00:02:11,240
stay strong and to still consider that the internet is a public service and should be

26
00:02:11,240 --> 00:02:15,040
more of a public good.

27
00:02:15,360 --> 00:02:19,600
For instance, just try to self-host your own mail server and you'll see the difficulty

28
00:02:19,600 --> 00:02:30,200
that you can meet while doing so because basically big companies have got their hold on these

29
00:02:30,200 --> 00:02:34,680
activities and self-hosting is not so easy.

30
00:02:34,680 --> 00:02:42,360
Yeah, like Camille said, I think internet is very, very far from being actually a common

31
00:02:42,680 --> 00:02:45,640
good or a public good and it doesn't belong to people.

32
00:02:45,640 --> 00:02:53,320
It's actually controlled by a very small number of very powerful entities, commercial entities

33
00:02:53,320 --> 00:03:05,160
usually that have their own agenda and that sometimes have political bias in this agenda

34
00:03:05,160 --> 00:03:07,440
to continue making profit.

35
00:03:07,520 --> 00:03:13,120
I think one of the key issues today is the issue of centralization, bringing back the

36
00:03:13,120 --> 00:03:19,920
power to the people, also involving empowering people to own their little share of the internet

37
00:03:19,920 --> 00:03:31,880
and not be controlled by a few companies that control search results, social medias, basically

38
00:03:31,880 --> 00:03:34,720
ideology bubbles, everything.

39
00:03:34,720 --> 00:03:39,280
How does your project contribute to addressing those issues?

40
00:03:39,280 --> 00:03:46,800
I think our projects come from the idea that both Camille and I wanted to have some piece

41
00:03:46,800 --> 00:03:54,800
of control over what software we use on the daily basis.

42
00:03:54,880 --> 00:04:03,600
We started building some kind of very small home lab micro data center to host our own

43
00:04:03,600 --> 00:04:15,480
services and we found that we were facing some difficulties in some areas and one of

44
00:04:15,480 --> 00:04:26,000
them was having an easy way to have full disk encryption on our servers without it being

45
00:04:26,000 --> 00:04:31,760
too much of a hassle, like when you have to reboot the server with full disk encryption,

46
00:04:31,760 --> 00:04:37,200
you physically have to do an action on the server remotely but you have to be there and

47
00:04:37,200 --> 00:04:38,200
do it.

48
00:04:38,920 --> 00:04:50,560
It was a blocker for us and that led to our first part of the project on Clevis.

49
00:04:50,560 --> 00:04:57,600
We found out that using NixOS we were able to much more efficiently maintain our services,

50
00:04:57,600 --> 00:05:05,640
our self-hosted services, but with the virtualization technology that we were using, Proxmox, we

51
00:05:05,640 --> 00:05:13,240
also found out that we had to go away from NixOS just for our hypervisors and that led

52
00:05:13,240 --> 00:05:15,240
to the second part of the project.

53
00:05:15,240 --> 00:05:25,920
And basically, our project contributes in two ways.

54
00:05:25,920 --> 00:05:33,280
First on the macro level, it's a project that is part of the alternative approach to the

55
00:05:33,280 --> 00:05:40,560
centralization of the internet that we discussed earlier because it is a project that is fully

56
00:05:40,560 --> 00:05:50,920
open source and funded by public institutions and also on a more individual centered level,

57
00:05:50,920 --> 00:05:59,440
it enables people like us without a big structure and just like people that have passion for

58
00:06:00,200 --> 00:06:09,600
computer science also to maintain their own infrastructure without the human power necessary

59
00:06:09,600 --> 00:06:15,760
to have quite a big infrastructure actually because with Proxmox or with Clevis you can

60
00:06:15,760 --> 00:06:22,800
really have a fleet of servers or many services and just because of the, I'd say the cleverness

61
00:06:22,880 --> 00:06:29,240
of these open source projects and the ideas behind them, you can just be like two or three

62
00:06:29,240 --> 00:06:33,160
people and have a really big infrastructure.

63
00:06:33,160 --> 00:06:37,080
Yeah, I can agree on that.

64
00:06:37,080 --> 00:06:44,840
We are doing the same for the NLnet network and yeah, it works great.

65
00:06:44,880 --> 00:06:54,920
Other people use other automation services for that and we do it all with the Nixos.

66
00:06:54,920 --> 00:07:00,960
What would be the use cases for remote unattended disk encryption?

67
00:07:00,960 --> 00:07:05,920
Well, disk decryption, I must say.

68
00:07:05,920 --> 00:07:12,280
I think the use case is that this decryption is a necessary security measure that everyone

69
00:07:12,280 --> 00:07:22,680
should adopt today but on laptops it's becoming easier and easier to do so with a lot of distributions 

70
00:07:22,680 --> 00:07:31,200
that have an easy toggle to do it but on servers you always have the problem that when the

71
00:07:31,200 --> 00:07:38,200
server boots you need to input your password to continue the boot process.

72
00:07:38,200 --> 00:07:45,920
It can be done usually physically, you have to be there and type the password with your

73
00:07:45,920 --> 00:07:52,200
keyboard for example but you also can do it remotely but it doesn't remove the fact that

74
00:07:52,200 --> 00:07:57,200
when the server reboots you have to be there and have a manual intervention.

75
00:07:57,200 --> 00:08:05,400
The idea is that with Clevis you have a framework that allows for automatic decryption of the

76
00:08:05,400 --> 00:08:12,400
disk using some kind of secret and the secret can be either locally on your device with the TPM

77
00:08:12,400 --> 00:08:23,400
or it can be also stored on other machines and so you have the idea that as long as you have one of your

78
00:08:23,400 --> 00:08:32,400
servers that is booted and up and running it can help the other server boot unattended.

79
00:08:32,400 --> 00:08:44,400
Yes I understand that and I think that's very helpful but isn't it true that after the key exchange

80
00:08:44,400 --> 00:08:49,400
the key will be in memory right?

81
00:08:49,400 --> 00:08:52,400
Yeah it's true.

82
00:08:53,400 --> 00:09:07,400
So for a VM with a hosting provider that hosting provider might also have access to your memory?

83
00:09:07,400 --> 00:09:16,400
Yeah that's true. So that's the whole question around trusted computing and the new security extensions

84
00:09:16,400 --> 00:09:25,400
and processors that will help us go further into building trust within an untrusted hypervisor.

85
00:09:25,400 --> 00:09:37,400
That's not exactly the use case here that we tackle mostly this is useful for booting bare metal

86
00:09:37,400 --> 00:09:44,400
and this was also the reason why when we were in a situation where the hypervisor was not running

87
00:09:44,400 --> 00:09:50,400
NixOS but we were running all our VMs on NixOS we were having this frustration of having our

88
00:09:50,400 --> 00:09:57,400
Clevis contribution not be that useful to us and that's why also we wanted to port the hypervisor

89
00:09:57,400 --> 00:10:08,400
under NixOS because now we can have this the hypervisor use this encryption and we have a cluster

90
00:10:08,400 --> 00:10:17,400
of several servers and they can help each other boot and decrypt the disk at boot using Clevis.

91
00:10:17,400 --> 00:10:23,400
What were the challenges to get Proxmox running on the NixOS?

92
00:10:23,400 --> 00:10:36,400
I guess the code base is pretty huge on Proxmox there's many packages that had to be incorporated in the same project

93
00:10:36,400 --> 00:10:53,400
and yes because of the way that NixOS function there was a lot of patching to do in the Perl code base of Proxmox

94
00:10:53,400 --> 00:11:02,400
and also the Rust ecosystem because they're currently moving from a Perl based project to Rust slowly

95
00:11:02,400 --> 00:11:09,400
and so that was quite time consuming I'd say.

96
00:11:09,400 --> 00:11:18,400
The Proxmox ecosystem is of course free and open source but it's not really designed to be run under different conditions

97
00:11:18,400 --> 00:11:28,400
than where it's been developed and it's been developed to run under Debian and so you have a lot of I would say

98
00:11:28,400 --> 00:11:42,400
things that are assumed to be there or paths that are assumed to be existing or the Rust ecosystem relies a lot on the Debian APT system

99
00:11:42,400 --> 00:11:53,400
so it's kind of we had to strip the software of its original home and make it a new home under NixOS.

100
00:11:54,400 --> 00:12:07,400
Was it hard to get your changes, your patches into the Proxmox code base? Was it hard to get them accepted?

101
00:12:08,400 --> 00:12:18,400
We didn't get any patch accepted or submitted to the Proxmox code base. We have some patches that are really NixOS

102
00:12:18,400 --> 00:12:25,400
relevant only for NixOS and then we have stored these patches locally.

103
00:12:25,400 --> 00:12:39,400
We have a few that we feel would be interesting to upstream especially Camille adapted a small part of the code base

104
00:12:39,400 --> 00:12:56,400
to be able to build under ARM or AH64 and so we feel even though the upstream project doesn't really build NixOS or packages for ARM

105
00:12:56,400 --> 00:13:07,400
we were able to do it without too much hassle just changing a sub part of the code and we are planning to submit this patch soon.

106
00:13:10,400 --> 00:13:19,400
And going back to Clevis, you said it's good if at least one of your services is running.

107
00:13:19,400 --> 00:13:39,400
Does that mean that your challenge can be accepted by any of a running service or must it be one specific server that answers the security challenge?

108
00:13:40,400 --> 00:13:48,400
Actually, if I understood the question correctly, I guess the Tong server can be...

109
00:13:48,400 --> 00:14:00,400
You can actually describe your own security protocol if you want either one server and one TPM.

110
00:14:00,400 --> 00:14:13,400
You can basically, using Shamir's secret sharing, describe your own security scheme, either any of the servers or all of them.

111
00:14:14,400 --> 00:14:24,400
Okay, clear. But what were the challenges there to get Clevis working?

112
00:14:25,400 --> 00:14:42,400
Yes. Well, this was actually very difficult because coming back to Proxmox, it was mostly about pioneering in the direction that people never actually ported Proxmox out of Debian.

113
00:14:42,400 --> 00:14:56,400
But for Clevis, the difficulty was very different because it was mostly about understanding the boot process and working in the first stages of the boot process in its RD.

114
00:14:56,400 --> 00:15:08,400
And since on NixOS, it works quite differently from the other distributions.

115
00:15:08,400 --> 00:15:35,400
And also, we spent a lot of time working on a test that would be able to prove that Clevis works efficiently and correctly when installing a new NixOS machine, configuring some Clevis keys and a Tong server, and then rebooting and checking that the partition is deciphered correctly.

116
00:15:35,400 --> 00:15:49,400
Yeah, I think as Camille said, we spent a lot of time designing a NixOS test, which is an integration test for a feature, which is a very nice part of the NixOS ecosystem.

117
00:15:49,400 --> 00:16:03,400
So you can have tests that spawn NixOS VM, run some commands on them, and then check what have some kind of assertion on the state of the VM.

118
00:16:03,400 --> 00:16:17,400
And we wanted to do a very, very extensive test where we could install a brand new NixOS machine, then provision, like do the partitioning, then test all.

119
00:16:17,400 --> 00:16:25,400
We have three kinds of disk encryption, one with zfs, one with bcachefs, and one with luks.

120
00:16:25,400 --> 00:16:30,400
And we tried these three different kinds of this encryption.

121
00:16:30,400 --> 00:16:41,400
And then we provisioned the machine so that it uses our Clevis module to decipher the disk at boot.

122
00:16:41,400 --> 00:17:05,400
And the tests were able to show that both that our module was working to decipher the disk when booting, but also if there was any kind of struggle or problem, it would roll back to a simple passphrase handling like usual.

123
00:17:05,400 --> 00:17:17,400
So we were able to prove during our inside our pull requests that we were not going to destroy anyone's setup with this change.

124
00:17:17,400 --> 00:17:22,400
And that's where that got us a long way into getting it accepted.

125
00:17:22,400 --> 00:17:24,400
So that is nice added information.

126
00:17:24,400 --> 00:17:43,400
This means that even if the tank server cannot be reached, your disk is not lost, but you fall back to passphrase on the console, which you of course have to physically access.

127
00:17:43,400 --> 00:17:44,400
Yeah, exactly.

128
00:17:44,400 --> 00:17:59,400
So with NixOS, you could always, if you do something wrong, you could always roll back to an earlier generation and have your disk be safe like this.

129
00:17:59,400 --> 00:18:08,400
But we also wanted to make sure that in any case, the worst case scenario is you fall back to the original behavior without Clevis.

130
00:18:08,400 --> 00:18:10,400
Yeah, yeah, sure.

131
00:18:10,400 --> 00:18:14,400
So I'm hearing NixOS a lot now.

132
00:18:14,400 --> 00:18:17,400
What is so special about NixOS?

133
00:18:17,400 --> 00:18:19,400
Why do you like it so much?

134
00:18:19,400 --> 00:18:34,400
Well, I guess from a quite outside point of view, because my main domain of expertise is not actually computer science, because I'm a PhD student in statistics.

135
00:18:34,400 --> 00:18:50,400
And so I actually use NixOS on a daily basis for my work, because basically, as I said, it makes me able to maintain quite a big infrastructure on my own.

136
00:18:50,400 --> 00:18:56,400
And to be sure, like when I do something, I don't have to do it again if my server crashes.

137
00:18:56,400 --> 00:19:19,400
For me, it's really a concrete way to make use of what the open source community has to offer and to speed up my work in statistics and to have a reprehensible environment.

138
00:19:19,400 --> 00:19:26,400
And me personally, when I discovered NixOS, I was already a Linux user for some time.

139
00:19:26,400 --> 00:19:30,400
And then a friend of mine showed me what NixOS is.

140
00:19:30,400 --> 00:19:40,400
And I started because we are both academics and we are both I am a theoretical computer scientist from my academic background.

141
00:19:40,400 --> 00:19:50,400
So I started reading the Eelco's, the creator of NixOS, PhD thesis and also other kind of resources I could find.

142
00:19:50,400 --> 00:19:58,400
And my conclusion, why this is how we should have done computer science forever.

143
00:19:58,400 --> 00:20:01,400
And I started to find it very elegant.

144
00:20:01,400 --> 00:20:23,400
And that was basically what pushed me to learn it, because it's also quite a difficult journey to learn Nix and NixOS where you need motivation not to say, oh, I don't really understand how I should do this, this and there is no documentation online.

145
00:20:23,400 --> 00:20:25,400
So I would just fall back to something else.

146
00:20:25,400 --> 00:20:36,400
So I continued in this process and I learned it and I started using it for my my different computing devices.

147
00:20:36,400 --> 00:20:42,400
So especially my servers, because I had a few servers, I was already self hosting stuff.

148
00:20:42,400 --> 00:20:56,400
And then it became interesting to me that it felt way less work to maintain all these services that I was using.

149
00:20:56,400 --> 00:21:04,400
And before that, I was mainly using container solutions.

150
00:21:04,400 --> 00:21:13,400
And they were breaking quite often on me and that was some work to keep them up to date and working.

151
00:21:13,400 --> 00:21:19,400
And when using NixOS, it felt that this was working just for free.

152
00:21:19,400 --> 00:21:32,400
It was really something that I learned over the year of using it, that it was way less taxing on me to maintain my services using NixOS.

153
00:21:32,400 --> 00:21:39,400
And to me, this is like you have the technology, Nix and NixOS are interesting technology,

154
00:21:39,400 --> 00:21:49,400
but you also have a very high value in the set of people that are packaging things into Nix package and maintaining these things,

155
00:21:49,400 --> 00:22:06,400
because they pour all their expertise into several into some topic, into packaging modules and software into NixOS

156
00:22:06,400 --> 00:22:12,400
so that people then can use them without needing all the knowledge to operate them.

157
00:22:12,400 --> 00:22:18,400
And that's what we did with Clevis. Clevis is not a very complicated technology.

158
00:22:18,400 --> 00:22:25,400
It's also not a very simple technology to set up if you don't know anything about it on other distribution.

159
00:22:25,400 --> 00:22:34,400
But on NixOS, you don't really need the domain specific knowledge that we had to acquire to write the module to operate it as a user.

160
00:22:34,400 --> 00:22:42,400
And that's something that is really interesting in the NixOS distribution, according to me.

161
00:22:43,400 --> 00:22:50,400
Yeah, here you have it. Both ways you can arrive to NixOS.

162
00:22:50,400 --> 00:22:57,400
The hard way like Julien did, like trying for many years many different things, containers,

163
00:22:57,400 --> 00:23:08,400
and having to try every technology possible and to reboot each machine after it has crashed and restarted from the beginning.

164
00:23:08,400 --> 00:23:14,400
And I, when I arrived and that Julien has already discovered NixOS,

165
00:23:14,400 --> 00:23:26,400
I actually only ever installed an Arch Linux, one Arch Linux machine in my life and used it for like something like 30 minutes before Julien told me,

166
00:23:26,400 --> 00:23:33,400
yes, you should switch to NixOS. It's a little more difficult just the first on the first days.

167
00:23:33,400 --> 00:23:43,400
But actually, I guess I kind of dodged the bullet and just started right away on the easy path.

168
00:23:44,400 --> 00:23:49,400
Yeah, it's a learning curve, but it's well worth it.

169
00:23:49,400 --> 00:23:56,400
Especially when you, like you said, when you want to maintain several services.

170
00:23:56,400 --> 00:24:11,400
You mentioned community. So, so do you already have a community behind Clevis or are you planning to build a community around it?

171
00:24:11,400 --> 00:24:18,400
What does that mean for long term sustainability of your of your projects?

172
00:24:18,400 --> 00:24:28,400
So according to me for Clevis, as we upstream our change into Nix package, there is no real notion of community.

173
00:24:28,400 --> 00:24:32,400
We gave it to the community. We plan to maintain it over time.

174
00:24:32,400 --> 00:24:37,400
But it's kind of now including it included into a bigger project.

175
00:24:37,400 --> 00:24:44,400
But for Proxmox NixOS, we have it as a standalone project for NixOS.

176
00:24:44,400 --> 00:24:55,400
We have it as a standalone project for now, planning to upstreaming into at some point, but not right now.

177
00:24:55,400 --> 00:25:06,400
And we have already so that there are quite a few people that are interested into the technology and participate to the project.

178
00:25:06,400 --> 00:25:33,400
And as it's a it's quite a big piece of software, it's kind of a relief that other people want to help and have their own idea and their own extensions or project into this inside this big projects so that they can do the project also belongs to other people and will evolve.

179
00:25:33,400 --> 00:25:36,400
Not just by our fact.

180
00:25:37,400 --> 00:25:40,400
Yeah. And I'll add that for Clevis.

181
00:25:40,400 --> 00:25:46,400
The point was kind of to make an initial proposal like here is how you can use it.

182
00:25:46,400 --> 00:25:55,400
Some examples. But the thing is, it's quite personal and how you would use Clevis because your each setup is quite different.

183
00:25:55,400 --> 00:26:13,400
So the idea was really to use the power of NixOS and Nix packages to like propose your use case and people can rely on it and then propose their own.

184
00:26:13,400 --> 00:26:21,400
And of course, for instance, people might want another kind of partition scheme or format.

185
00:26:21,400 --> 00:26:30,400
And maybe this is we didn't we only use the zfs and Luks and bcachefs.

186
00:26:30,400 --> 00:26:34,400
And if people want to add something else, then they contribute.

187
00:26:34,400 --> 00:26:38,400
And that's how it works, I guess.

188
00:26:39,400 --> 00:26:51,400
That also leads me to the question, because you started this project with the NGI Zero funding as Clevis.

189
00:26:51,400 --> 00:26:55,400
But then you morphed also into Proxmox.

190
00:26:55,400 --> 00:27:00,400
But both these projects already sound separately rather big.

191
00:27:00,400 --> 00:27:07,400
How did you manage to do so much in only one project, which are actually two projects?

192
00:27:08,400 --> 00:27:12,400
Long of hard night's work, I guess.

193
00:27:12,400 --> 00:27:20,400
No, no, but actually they were for us quite like they were our big project on a personal level.

194
00:27:20,400 --> 00:27:23,400
Like that's what we needed for our infrastructure.

195
00:27:23,400 --> 00:27:31,400
I guess that there was there were both the next logical steps because we had already some services deployed.

196
00:27:31,400 --> 00:27:39,400
We had already an supervisor with Proxmox on Debian and we needed to move to full disk encryption.

197
00:27:39,400 --> 00:27:49,400
And so, well, we took some time because we've been actually thinking about it for a long time.

198
00:27:49,400 --> 00:27:58,400
And so we had some ideas that already had some were already a little more mature than maybe like starting the project right away.

199
00:27:58,400 --> 00:28:08,400
And just going in the open and we yes, we knew what we wanted to do precisely, I guess.

200
00:28:08,400 --> 00:28:18,400
And also that's maybe more of response, but we did the Clevis part and I reached out to Michiel and he said to me,

201
00:28:18,400 --> 00:28:23,400
Oh, you want to do this? Let's just add it to your first project. It's easier.

202
00:28:23,400 --> 00:28:30,400
OK. Are there any next steps that you are thinking about?

203
00:28:30,400 --> 00:28:40,400
For the Proxmox project, we are we have quite a few people that reached out with suggestions for extensions.

204
00:28:40,400 --> 00:28:50,400
And that's basically what we we think we will do whenever we get a bit of time to do it.

205
00:28:50,400 --> 00:28:59,400
But the two extensions that we think about currently is implementing some kind of

206
00:28:59,400 --> 00:29:10,400
option for declarative configuration of the virtual machines on the Proxmox host.

207
00:29:10,400 --> 00:29:16,400
So so far, we just have a Proxmox instance running on NixOS.

208
00:29:16,400 --> 00:29:21,400
But there is minimal configuration that you can do through the NixOS module system.

209
00:29:21,400 --> 00:29:30,400
But when you want to create a new virtual machine, you have to use the web interface as people do when they use Proxmox.

210
00:29:30,400 --> 00:29:41,400
But because we are NixOS users, we also consider having another way to configure this virtual machine through the NixOS configuration itself,

211
00:29:41,400 --> 00:29:44,400
which some people have expressed interest in.

212
00:29:44,400 --> 00:29:54,400
And we also want to consider other storage layers for the virtual machine.

213
00:29:54,400 --> 00:30:05,400
So you have typically in Proxmox setup, you can have a safe cluster that acts as a layer for the storage of your virtual machine

214
00:30:05,400 --> 00:30:13,400
so that you can move some virtual machine from one physical machine to another super easily.

215
00:30:13,400 --> 00:30:18,400
And it helps you achieve what we call high availability.

216
00:30:18,400 --> 00:30:29,400
Like you get one node of this cluster that for some reason has a problem and shut down the virtual machines that are actually running on this node.

217
00:30:29,400 --> 00:30:35,400
They can migrate to another node so that you don't you don't get service interruption.

218
00:30:35,400 --> 00:30:40,400
So there is safe that is already included in NixOS.

219
00:30:40,400 --> 00:30:46,400
And we had suggestion for implementation of something called LinStore, which is another layer.

220
00:30:46,400 --> 00:30:49,400
And we are we are also considering this.

221
00:30:49,400 --> 00:31:01,400
Yeah. And also, I guess one concrete big step will be to also move our little homemade data center fully to our Proxmox NixOS.

222
00:31:01,400 --> 00:31:09,400
And it would be quite an interesting proof of a concept and of robustness of our implementation.

223
00:31:09,400 --> 00:31:19,400
And and yeah, usually we will really use the fact that safe allows like to disconnect one node, then may reinstall this node on Proxmox NixOS.

224
00:31:19,400 --> 00:31:25,400
Hope that that we actually don't lose any of our our infrastructure.

225
00:31:25,400 --> 00:31:32,400
But yes, that would be once this is done, we kind of achieved what we we wanted to do for a long time.

226
00:31:33,400 --> 00:31:43,400
So far, we've we've developed on a development cluster and we had like we had no problem so far.

227
00:31:43,400 --> 00:31:45,400
We think everything works correctly.

228
00:31:45,400 --> 00:31:57,400
But the next step is committing with our own hardware and move like migrate our production cluster, which has a lot of edge case and weird configuration.

229
00:31:57,400 --> 00:32:03,400
So we know that if there is something not working, we will find it at this point.

230
00:32:03,400 --> 00:32:12,400
So Proxmox cannot cannot only be used to build physical machines, but also for containers.

231
00:32:12,400 --> 00:32:24,400
I was just thinking, doesn't it make it easier to build NixOS based containers when there is a NixOS layer under it?

232
00:32:24,400 --> 00:32:29,400
Well, honestly, this is not this is not our use case.

233
00:32:29,400 --> 00:32:32,400
We don't we don't run containers.

234
00:32:32,400 --> 00:32:36,400
So we didn't really go too much in this direction.

235
00:32:37,400 --> 00:32:56,400
But we hope we hope that people that have different use case than ours can do their own experimentation and either report if something is not working as they as they think it should or contribute just to make it also work for their own use cases.

236
00:32:56,400 --> 00:33:00,400
There is quite a lot of stuff that you can make with Proxmox.

237
00:33:01,400 --> 00:33:04,400
And for sure, not everything is working today.

238
00:33:04,400 --> 00:33:11,400
Only things that we have tested because we know that we need it and that most people need it.

239
00:33:11,400 --> 00:33:24,400
But over time, we will get this this proportion of things working over the total bigger and bigger because other people we come and say, oh, this thing is not working.

240
00:33:24,400 --> 00:33:28,400
That's like I think it should and they will fix it or we will fix it.

241
00:33:28,400 --> 00:33:49,400
Yeah, and clearly, I think there's a lot of wonderful ideas yet to be yet to be had on on based on this Proxmox on well this hypervisor based on on NixOS and and then all of your containers and VMs that are also on NixOS.

242
00:33:49,400 --> 00:33:59,400
And I guess that's a big direction in towards smashing the state and be able to have like a fully fully reproducible.

243
00:33:59,400 --> 00:34:04,400
Well, fleet of VMs and containers.

244
00:34:04,400 --> 00:34:05,400
Yeah.

245
00:34:06,400 --> 00:34:10,400
Maybe coming back to the community question again.

246
00:34:11,400 --> 00:34:28,400
I mean, for for Proxmox, Proxmox, there is already a big community and I can only think that that that there are also people that are in the NixOS community that are also in the Proxmox community.

247
00:34:28,400 --> 00:34:38,400
So I can only imagine that that that there is a community waiting there to to support you.

248
00:34:38,400 --> 00:34:57,400
Yeah, that was actually quite a big surprise because the day that we released the project, we were really happy to see that many people were already like waiting for it and contacting reaching out to us to say, yeah, that's that's super useful.

249
00:34:57,400 --> 00:35:04,400
And so that like that's that makes us happy to have contributed to this project.

250
00:35:04,400 --> 00:35:12,400
Yeah, we were we were not expecting this kind of hype, but we had a lot of people reach out.

251
00:35:12,400 --> 00:35:18,400
I guess 300 stars on the repository in a few days.

252
00:35:18,400 --> 00:35:23,400
So I think there is, of course, a community of people that are really interested in that.

253
00:35:23,400 --> 00:35:31,400
I don't know how much people have actually taken the step to use it in their own production environment.

254
00:35:31,400 --> 00:35:47,400
But the idea is that everyone that is committed to to to use something that is not commercial because our project is not commercial and will not do any kind of support like the Proxmox project does.

255
00:35:48,400 --> 00:35:56,400
We will help them move to it and we will accept contribution and issues.

256
00:35:56,400 --> 00:36:04,400
So we would be very happy having this kind of little community around the project that make it grow.

257
00:36:04,400 --> 00:36:12,400
Yeah, and it was nice to to to see the impact of the of this project because mostly because we're in Ph.D.

258
00:36:12,400 --> 00:36:19,400
both in Ph.D. are some of our projects were like research oriented like packaging,

259
00:36:19,400 --> 00:36:27,400
like the Lattek editor or Zotero, which is for bibliography management.

260
00:36:27,400 --> 00:36:37,400
And while the hype was really bigger than on this very niche little projects and so very, very, very happy to to see that.

261
00:36:37,400 --> 00:36:48,400
Yeah, interesting. And how did NGI or NGI Zero help your project?

262
00:36:48,400 --> 00:36:55,400
I guess by by giving us the chance to have time to dedicate on this.

263
00:36:55,400 --> 00:37:14,400
We it's difficult when you when you're a Ph.D. student or to to to be able to dedicate some of your time to to open source and free software like you have to to prioritize things.

264
00:37:14,400 --> 00:37:20,400
And often like for me, open source is a priority.

265
00:37:20,400 --> 00:37:32,400
But being able to dedicate some time being financially supported is is luck that that we've been able to have thanks to NGI Zero.

266
00:37:32,400 --> 00:37:39,400
And probably we would have done this kind of project anyway at some point.

267
00:37:39,400 --> 00:37:51,400
But having this financial support gave us legitimacy and to allocate some time of some part of our time to do it and do it as soon as possible.

268
00:37:51,400 --> 00:38:02,400
Yes, clearly the impulse, they gave us the impulse to to work on it and to make something that is... yes to go,

269
00:38:03,400 --> 00:38:14,400
I'd say to the last step of the project, and not just like start it and then abandon it for and that was really nice to to to achieve something.

270
00:38:14,400 --> 00:38:31,400
And and also I say that's in a way quite I'd say financially interesting because in a way it's something that we we do on our free time.

271
00:38:31,400 --> 00:38:41,400
And and if it was if it were our main work, maybe we would get in a way paid more.

272
00:38:41,400 --> 00:38:50,400
But we maybe would do it with less passion because here it's really something that we use for our needs.

273
00:38:50,400 --> 00:38:53,400
So we really wanted to make it the best we could.

274
00:38:53,400 --> 00:39:05,400
And and so the motivation and the financial support is really a good I guess a good way to to support projects and maybe more efficient that what you would meet in a private company or something.

275
00:39:06,400 --> 00:39:14,400
And if you had to give any advice to other people who are considering to apply for funding, what what would you say to them?

276
00:39:14,400 --> 00:39:17,400
Yeah, I'd say do it apply.

277
00:39:20,400 --> 00:39:23,400
It's you you never know what what will happen.

278
00:39:23,400 --> 00:39:31,400
And so do try to apply the application process is not to taxing actually.

279
00:39:31,400 --> 00:39:35,400
It's not something that's going to take days and days, probably just a few hours.

280
00:39:36,400 --> 00:39:48,400
So you can if you can afford taking these few hours, writing your ideas down and trying something that's that may become a very nice experience.

281
00:39:48,400 --> 00:39:50,400
It's always worth it to do it.

282
00:39:50,400 --> 00:40:01,400
And I want to mention that when I applied first for for Clevis, I had some very insightful feedback from while being rejected.

283
00:40:01,400 --> 00:40:06,400
I had some very insightful feedback from from the NLnet team.

284
00:40:06,400 --> 00:40:13,400
And that was really interesting to give me a new angle and vision on what I wanted to do.

285
00:40:13,400 --> 00:40:19,400
So at the beginning, the project was based on another technology called Mondos and it was rejected.

286
00:40:19,400 --> 00:40:33,400
But the feedback helped me have a better overview on the ecosystem that were existing, was existing on this kind of software and to do actually a new a new application that went through.

287
00:40:33,400 --> 00:40:47,400
And it's really interesting to see that we have teams, a team of very, very informed, well informed experts that can give you feedback that will actually make a difference for your project.

288
00:40:47,400 --> 00:40:51,400
Yeah. And I'd just yes, do it.

289
00:40:51,400 --> 00:41:02,400
Do consider applying for an NGI Zero funding and and do it with the right idea that you're contributing to something that is funded by a public institution.

290
00:41:02,400 --> 00:41:10,400
And so you're you're contributing to to making Internet a common good.

291
00:41:10,400 --> 00:41:19,400
And so, yes, that's your project should be oriented in that direction.

292
00:41:19,400 --> 00:41:30,400
Yeah. And maybe then some advice from our side that would be the feedback is indeed very helpful.

293
00:41:30,400 --> 00:41:39,400
And maybe sometimes you don't get the feedback proactively.

294
00:41:39,400 --> 00:41:51,400
So always ask for the feedback because, yeah, we or they another grant grantor, not grantee.

295
00:41:51,400 --> 00:42:02,400
Yeah, they they they need to give you proper feedback about why you are why you were rejected and you can learn from it.

296
00:42:02,400 --> 00:42:12,400
Yeah. And really, at some point, we thought that we might not do do it and get not get funded because we we were first rejected.

297
00:42:12,400 --> 00:42:25,400
But that's where like thinking like really taking into account the feedback and like modifying the project in the way that Julien did.

298
00:42:25,400 --> 00:42:35,400
Well, it was really relevant because in the end we we we got accepted and and and and yeah, the project was indeed better.

299
00:42:35,400 --> 00:42:41,400
That's using the other technology. And now we do use a clevis that we even didn't know before.

300
00:42:41,400 --> 00:42:58,400
Well, when we first started the project. So I would like to add a little disclaimer for people who are listening because currently, at least the people who are assessing all the applications, they are very, very busy.

301
00:42:58,400 --> 00:43:04,400
And so getting feedback might be a little less likely now.

302
00:43:04,400 --> 00:43:08,400
Maybe it will again, you know, in the future be be be possible again.

303
00:43:08,400 --> 00:43:15,400
But I think at this very moment it will be a bit hard to to get for them to make time for it.

304
00:43:15,400 --> 00:43:22,400
Although everybody agrees that it is super important, but it's a bit busy at the moment.

305
00:43:23,400 --> 00:43:27,400
Yes, that's why we need more money in that in that in that domain.

306
00:43:27,400 --> 00:43:30,400
We need more more more people helping.

307
00:43:30,400 --> 00:43:37,400
And yes, that's really an important step to to get rid of this issue of centralized Internet.

308
00:43:37,400 --> 00:43:40,400
And yeah, yes, I agree.

309
00:43:40,400 --> 00:43:48,400
It would be really good if because, of course, NGI Zero is financially supported by the European Commission.

310
00:43:48,400 --> 00:43:53,400
And that's great. But also maybe nation states.

311
00:43:53,400 --> 00:43:58,400
Germany is now also putting some money or a lot of money into open source.

312
00:43:58,400 --> 00:44:07,400
It should be more normal for public institutions to support the public Internet.

313
00:44:07,400 --> 00:44:14,400
We cannot leave it to companies to build it, obviously, because it becomes a dystopia.

314
00:44:14,400 --> 00:44:19,400
It's also very sounds like investment strategy for for the public sector.

315
00:44:19,400 --> 00:44:29,400
Like you keep we are we are draining so much money, at least in France, being,

316
00:44:29,400 --> 00:44:39,400
I'd say, dependent on software that is proprietary and very expensive and keeps you locked into one kind of ecosystem.

317
00:44:39,400 --> 00:44:48,400
And and just putting money into building blocks for software or for the Internet that is a common good,

318
00:44:48,400 --> 00:45:03,400
that is free software that is built by passionate people that do not have in mind capitalistic goals or is just for me a very sound political strategy.

319
00:45:04,400 --> 00:45:14,400
And in a way, we save so much money in comparison when we like paying a very, very expensive well,

320
00:45:14,400 --> 00:45:26,400
a big, big money to the to a tech developer that would maybe just work on its specific use case for its specific company with a close source and code.

321
00:45:27,400 --> 00:45:34,400
And then and then no one can rely on it to to make well advances in society.

322
00:45:34,400 --> 00:45:47,400
Cool. Maybe coming back to your projects, how can the listeners contribute to bringing the projects you are working on further?

323
00:45:47,400 --> 00:45:52,400
Basically, I'd say do try Proxmox, do try Clavis.

324
00:45:52,400 --> 00:46:01,400
And we've already had some some feedback and we want more and like also contribution because, of course, we're still doing Ph.Ds.

325
00:46:01,400 --> 00:46:05,400
And yes, we need people to actually be involved in it.

326
00:46:05,400 --> 00:46:15,400
So because we cannot like think of all the use cases that people would imagine.

327
00:46:15,400 --> 00:46:26,400
And so, yes, we already have some warm messages and we want others and more.

328
00:46:26,400 --> 00:46:35,400
And yeah, yeah. And knowing that both of you are doing Ph.Ds, the grant only gives you money, not time.

329
00:46:35,400 --> 00:46:39,400
Right. Exactly. Only 24 hours in a day.

330
00:46:39,400 --> 00:46:54,400
And we spend some days, maybe a lot of these 24 hours working on on these on these on these projects and really happy to have done it because now we can move to next steps.

331
00:46:54,400 --> 00:46:56,400
We don't really know where for right now.

332
00:46:56,400 --> 00:47:05,400
But yeah, we still a lot of it's quite infinite, actually, because it's both a passion and work.

333
00:47:05,400 --> 00:47:14,400
So, yeah. Is there anything that that that that we that we missed in our conversation that that you need to add?

334
00:47:14,400 --> 00:47:28,400
I just would like to add that all this journey from Camille and I was fueled by the idea that we should be less dependent on software.

335
00:47:28,400 --> 00:47:40,400
We don't have control on software that is managed by companies that can access, analyze and sell our private data.

336
00:47:40,400 --> 00:47:47,400
And it was it was a journey to own our software again.

337
00:47:47,400 --> 00:47:57,400
And I would advise anyone listening to ask themselves, I might have, what kind of software do I rely on on a daily basis?

338
00:47:57,400 --> 00:48:09,400
And if you're using commercial software, are you at ease that the software might be that the data you're giving to the software might be

339
00:48:09,400 --> 00:48:21,400
harvested, analyzed, sold for purposes that are in no way aligned with your own purpose?

340
00:48:21,400 --> 00:48:37,400
Are you are at ease that software that you might using abusing on a daily basis is trying to control the way you think by showing you targeted contents?

341
00:48:37,400 --> 00:48:54,400
And you have no way to control this. Are you at ease that, and that was, I'm saying this because I was, I had this personal reflection that am I at ease that software media that I'm using is specifically designed to make me addicted to it.

342
00:48:54,400 --> 00:49:06,400
And at some point, if you're, if one of these answers is no, maybe do read about self hosting and free software.

343
00:49:06,400 --> 00:49:22,400
And also, it's more easy than you think to migrate some of your users, well, use cases to open source and self hosting like really what we, what I learned I was from I was starting from nothing.

344
00:49:22,400 --> 00:49:38,400
Basically, I was just enjoying computer science like from afar and in only a few few months I already had my media server my services.

345
00:49:38,400 --> 00:49:58,400
My services. As I said, only the mail server is quite difficult. But one day maybe we are in a society where it's possible to self host your basic computer site, well, internet needs and yes to have kind of a freedom and and basically yes.

346
00:49:58,400 --> 00:50:01,400
So keep on smashing the state.

347
00:50:01,400 --> 00:50:21,400
And that's the whole reason we are working on NixOS is because we believe NixOS can make it so that you don't need to be an expert in computer science or infrastructure or DevOps to be able to have your first feet into this.

348
00:50:21,400 --> 00:50:27,400
Your first foot into this new world of self hosting and owning your own software.

349
00:50:27,400 --> 00:50:44,400
Yeah, we couldn't say it better. I mean, this is what NGI stands for the next generation internet with which is more private, more secure, and more sustainable.

350
00:50:44,400 --> 00:50:46,400
And more user control.

351
00:50:46,400 --> 00:50:47,400
Exactly.

352
00:50:47,400 --> 00:50:50,400
Sovereignty.

353
00:50:50,400 --> 00:50:59,400
Yes, thanks a lot for that passionate call for a better internet and by extension a better world.

354
00:50:59,400 --> 00:51:05,400
Thank you very much Julien and Camille for this conversation. It was really enlightening.

355
00:51:05,400 --> 00:51:07,400
Thank you so much.

356
00:51:20,400 --> 00:51:22,400
Thank you.