Repository navigation
Expand file tree
/
Copy pathHow_Digital_Knowledge_Survives_Time.vtt
More file actions
1562 lines (1041 loc) · 64.6 KB
/
Copy pathHow_Digital_Knowledge_Survives_Time.vtt
File metadata and controls
1562 lines (1041 loc) · 64.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
WEBVTT
00:00.000 --> 00:05.160
you know um there is this totally universal human experience of like packing up to move
00:05.160 --> 00:09.180
or maybe you're just cleaning out a garage or something oh yeah the dreaded garage cleanup
00:09.180 --> 00:14.620
right and you take a cardboard box you fill it with just completely miscellaneous things and you
00:14.620 --> 00:19.600
grab this thick black marker and you just write the word stuff on the side of it or you know if
00:19.600 --> 00:24.220
you're feeling really organized that day maybe you write misc ah yes highly specific exactly
00:24.220 --> 00:29.540
but in that exact moment you know precisely what is in that box right because you just packed it
00:29.540 --> 00:35.260
yeah you know it's um the cables for the old tv the warranty for the blender or maybe your winter
00:35.260 --> 00:39.720
gloves and it makes perfect sense to you right then but then you shove it up in the attic exactly
00:39.720 --> 00:44.220
shove it in the attic and like five years go by yeah and suddenly you need those winter gloves
00:44.220 --> 00:49.200
so you climb up there you look at this wall of boxes you see the one that just says stuff
00:49.200 --> 00:58.340
and you realize something genuinely terrifying which is that marker that label you wrote it wasn't
00:58.340 --> 01:04.060
actually a description at all it was like a wager against your own future amnesia
01:04.060 --> 01:09.760
oh wow yeah that's a great way to put it and we almost always lose that bet right but what's wild
01:09.760 --> 01:14.960
to me is that we do this every single day not just with cardboard boxes in the attic but with our
01:14.960 --> 01:20.760
digital lives every time you hit save yes when you click save on a file or when you name a folder
01:20.760 --> 01:27.980
final draft version two you are making that exact same wager against time and usually losing usually
01:27.980 --> 01:33.300
losing so for you listening today whether you're just you know an everyday computer user trying to
01:33.300 --> 01:38.360
find a text spreadsheet from three years ago or maybe you're a senior software engineer who navigates
01:38.360 --> 01:43.620
these massive global open source libraries we are doing a deep dive into something that will
01:43.620 --> 01:50.140
fundamentally alter how you view saving your work it really changes your perspective it does we are
01:50.140 --> 01:55.540
looking at this incredibly dense single source today it's a monograph from november 2026 right
01:55.540 --> 02:00.320
titled the architecture of continuity exactly by an independent researcher who goes by the name
02:00.320 --> 02:08.000
Flyxion and our mission today is to basically map the entire life cycle of how human knowledge and work
02:08.000 --> 02:15.600
actually survive time starting from that tiny mundane act of saving a file on a personal laptop right all the
02:15.600 --> 02:20.380
way up to the mathematical and like the social foundations that keep our global scientific
02:20.380 --> 02:26.120
institutions from just completely collapsing it's basically a theory of everything for digital survival
02:26.120 --> 02:32.100
it really is i mean when you read it it reads almost like a mathematical proof of why human beings forget
02:32.100 --> 02:38.420
things yeah it's not just philosophy no not at all the author isn't just philosophizing about lost files
02:38.420 --> 02:43.860
or being messy the text uses really rigorous mathematics i mean we're talking graph theory
02:43.860 --> 02:51.380
probability computational complexity which sounds intimidating but it's so applicable it is because it
02:51.380 --> 02:56.580
demonstrates why things that we just normally brush off as annoying technical problems like oh a broken
02:56.580 --> 03:04.420
code library or a lost document or actually these really profound failures of human prediction and institutional memory
03:04.420 --> 03:10.820
right so i want to start at the absolute most basic level here the individual human yeah the individual
03:10.820 --> 03:17.380
human sitting at a computer because if i'm saving a file i usually feel like i'm doing future me a
03:17.380 --> 03:24.660
favor you know i'm storing information safely but Flyxion characterizes this as an asymmetry of saving
03:24.660 --> 03:31.380
yeah asymmetry and i was really struggling to wrap my head around this at first what is asymmetrical
03:31.380 --> 03:37.700
about it like what am i failing to predict about my own future when i just hit ctrl s well so the
03:37.700 --> 03:43.620
asymmetry is entirely rooted in the decay of your informational state okay unpack that for me so when
03:43.620 --> 03:48.820
you create a file and you save it you're making a prediction about a future encounter with that file right
03:48.820 --> 03:53.940
right but in that future encounter the reader is going to be lacking some of the knowledge that you
03:53.940 --> 03:59.300
have right now in the present even if the reader is me especially if the reader is you that's a crucial
03:59.300 --> 04:05.300
insight here the biological person might be the exact same human being but your context your
04:05.300 --> 04:11.300
informational state has decayed because time passed exactly at the moment of creation you know exactly
04:11.300 --> 04:17.700
why the file exists what problem it solves um what assumptions you were making even the stuff you tried
04:17.700 --> 04:24.500
and deleted right the invisible context yes but the returning reader future you knows almost none of
04:24.500 --> 04:30.740
this so the cost of making an artifact usable for the future like writing documentation or organizing
04:30.740 --> 04:36.660
folders that cost has to be paid at the exact moment when its necessity is least apparent to you
04:37.460 --> 04:42.820
because to present me it's obvious exactly to present you the file seems entirely self-explanatory
04:42.820 --> 04:47.940
why would you explain it to yourself man i always just chalk that up to personal laziness you know
04:47.940 --> 04:53.620
um it's like i tell myself oh i'll write the documentation later or i'll clean up the folder
04:53.620 --> 04:59.060
structure when i have some free time but you never do never but you know Flyxion frames this entirely
04:59.060 --> 05:05.220
differently it's not a moral failing it's an economic problem yes what the text calls the economy of
05:05.220 --> 05:11.780
attention right externalizing your memory so writing down that context organizing those folders adding tags
05:11.780 --> 05:19.220
is fundamentally alone against the future alone right when you are deep in a project your attention is this
05:19.220 --> 05:25.780
scarce incredibly valuable resource spending that attention on documentation is cheap right now
05:25.780 --> 05:30.420
but it feels like a total distraction from the actual work i mean it totally does you just want
05:30.420 --> 05:36.180
to finish the thing exactly so the cost of externalizing the context is born entirely by the
05:36.180 --> 05:42.100
writer in the present but the benefit is deferred it's highly uncertain and it's enjoyed by the reader in
05:42.100 --> 05:48.580
the future so the equilibrium state of our attention span basically guarantees we will under invest in
05:48.580 --> 05:55.300
metadata always every single time that is so wild yeah every time so essentially past me is packing a
05:55.300 --> 06:01.540
time capsule for an amnesiac future me but past me is incredibly cheap with their time that is the
06:01.540 --> 06:07.700
perfect analogy yes and because of that under investment we run into a major problem with how we even
06:07.700 --> 06:14.260
define a file as surviving right we do yeah we conflate a lot of things right the monograph argues
06:14.260 --> 06:20.100
we mix up three entirely independent concepts when we talk about digital survival uh physical
06:20.100 --> 06:25.860
referential and operational persistence yes the three persistences this is foundational can you break
06:25.860 --> 06:30.660
those down yeah because i thought if a file is on my hard drive it survived well that's only the first
06:30.660 --> 06:36.420
level sure that's physical persistence it's the baseline it just means a decoding procedure exists to find
06:36.420 --> 06:41.300
the literal bits on a storage medium so those zeros and ones are still there right the physical
06:41.300 --> 06:47.380
magnetic charges on a disk or the solid state traps they still hold the data and the theory someone
06:47.380 --> 06:53.860
could extract it okay but that doesn't mean i can find it exactly which brings us to referential
06:53.860 --> 07:00.580
persistence referential persistence means that an agent so a person or a search tool can actually find
07:00.580 --> 07:06.260
and verify that specific artifact within a reasonable bounded budget of time and effort okay let me
07:06.260 --> 07:12.500
make sure i've got this if i have a cardboard box of like 50 unlabeled external hard drives in my
07:12.500 --> 07:18.660
basement a nightmare scenario but short right a total nightmare and i need one specific tax document
07:18.660 --> 07:24.500
from 2012 the document technically has physical persistence the data hasn't melted assuming the
07:24.500 --> 07:31.060
drives still work yes right but for all practical purposes it lacks referential persistence because i just
07:31.060 --> 07:36.580
don't have the time or the budget to plug in all 50 drives mount them and manually search every single
07:36.580 --> 07:43.460
directory to find it exactly a backup system by itself only solves physical persistence it doesn't
07:43.460 --> 07:48.980
solve referential okay so what if i do have a great search tool and i find the file then you hit the
07:48.980 --> 07:56.340
highest level which is operational persistence operational right once you've physically preserved the file
07:56.340 --> 08:02.660
and you've successfully located it do you actually have the context required to use it or modify it
08:02.660 --> 08:09.540
correctly ah i see for example you might find a compiled software binary from 1998 okay you know
08:09.540 --> 08:14.500
exactly what it is you have it sitting right on your current desktop so you have physical and referential
08:14.500 --> 08:21.380
persistence i found it it's right there but it requires a dead operating system a specific software
08:21.380 --> 08:26.100
library that's no longer hosted anywhere on the internet and a graphics architecture that hasn't
08:26.100 --> 08:31.940
been manufactured in two decades oh wow so the artifact is found but it is completely totally
08:31.940 --> 08:38.820
unusable that is fascinating so the text really emphasizes that these three states are completely
08:38.820 --> 08:43.460
independent of each other completely independent completely you can have anyone without the others
08:43.460 --> 08:49.540
right so like i said a hard drive in a box has physical but not referential and that 1998 software
08:49.540 --> 08:56.100
has referential but not operational keeping something actually alive truly alive requires
08:56.100 --> 09:01.700
operational context that makes so much intuitive sense but fly on doesn't just stop at intuition
09:01.700 --> 09:07.060
right they actually model this mathematically they do yeah with the two-channel recovery model right
09:07.780 --> 09:13.620
so if human memory decays so reliably and our attention economy guarantees that we won't write enough
09:13.620 --> 09:19.060
down how do we mathematically model our actual chances of finding our work later well
09:19.060 --> 09:26.020
recovery relies on two distinct channels you have the memory channel and the record channel okay the
09:26.020 --> 09:31.620
memory channel is basically your biological brain it represents your unaided human recollection of
09:31.620 --> 09:38.020
where a file is what it does which is terrible yes mathematically flyxion models this channel
09:38.020 --> 09:45.460
is decaying exponentially over time exponentially yes it's modeled as uh e to the minus lambda delta
09:45.460 --> 09:52.980
okay translate the math for me sure lambda is your personal rate of forgetting and delta is the time
09:52.980 --> 10:02.420
delay so as delta time increases the value drops exponentially over a long enough horizon the memory
10:02.420 --> 10:08.420
channel effectively hits zero that is so relatable my brain is actively overwriting the directory path to
10:08.420 --> 10:13.060
that old project just to make room for like meeting schedules and what i need to buy at the grocery store
10:13.060 --> 10:20.340
exactly your brain has to optimize for the present so if my brain is zeroing out the record channel has
10:20.340 --> 10:26.740
to step in precisely the record channel represents your metadata the file names the search indexes the
10:26.740 --> 10:33.220
folder structures and its superpower is that it stays relatively stable over time it doesn't decay like
10:33.220 --> 10:39.860
wetware right it doesn't decay exponentially like your brain so the value of your metadata is practically
10:39.860 --> 10:45.700
zero at the exact moment of creation right because your memory channels at a hundred percent right when
10:45.700 --> 10:51.060
i save it i don't need the metadata because i literally just did it but as time approaches infinity
10:51.620 --> 10:57.140
and your memory goes to absolute zero the metadata becomes literally the only mechanism for recovery the
10:57.140 --> 11:03.380
only lifeline yes but there is an assumption baked into that idea of time passing right we tend to think
11:03.380 --> 11:09.700
that as time goes on it inevitably just gets harder and harder to recover an artifact like it's just this
11:09.700 --> 11:18.260
one-way street toward entropy usually yes but Flyxion points out that this model exhibits non-monotonicity
11:19.060 --> 11:26.500
yes non-monotonicity which means the recoverability doesn't always go down it can actually increase over
11:26.500 --> 11:31.940
time which sounds impossible how does that work it happens because the record channel relies on the retrieval
11:31.940 --> 11:38.260
context so the tools and the technology available to the person doing the searching if the tools improve
11:38.260 --> 11:44.180
the probability of recovery can actually trend upward okay can you give me an example yeah imagine
11:44.180 --> 11:51.780
scanning an old handwritten laboratory notebook in say 1995 okay so it's just a raw image file exactly
11:51.780 --> 11:59.540
just a tiff or a jpeg for 20 years that file was completely unsearchable if you forgot the file name
11:59.540 --> 12:07.300
you gave it it was effectively lost in the noise right but then suddenly optical character recognition
12:07.300 --> 12:14.980
ocr technology and these new machine learning models drastically improved in the 2020s oh i see i see right
12:15.780 --> 12:21.860
suddenly a modern search tool can just natively read the cursive handwriting inside that ancient image
12:22.660 --> 12:29.140
the artifact itself hasn't changed at all the pixels are identical but the retrieval context improved
12:29.540 --> 12:37.140
so the referential persistence spiked that is incredible yeah and that plays directly into a
12:37.140 --> 12:43.220
massive warning in the text about what Flyxion calls premature normalization yes premature normalization
12:43.220 --> 12:49.940
is a killer because if i had transcribed that 1995 notebook into just a basic plain text file back then
12:50.500 --> 12:55.540
you know throwing out the raw image because i couldn't easily index it at the time i would have ruined
12:55.540 --> 13:01.140
it right you would have permanently destroyed data yes i might have lost the diagrams or the little
13:01.140 --> 13:07.220
marginalia or just the weird context of how it was written on the page that future ai tools could
13:07.220 --> 13:13.780
have analyzed exactly by destroying the original raw data just to fit the temporary limitations of 1995
13:13.780 --> 13:19.540
search technology you are robbing the future of the chance to apply better tools to it later
13:19.540 --> 13:27.140
wow so normalizing data to fit current limitations permanently caps its future persistence it puts a
13:27.140 --> 13:33.140
ceiling on it forever okay but even if we keep all the raw data we still have to put a name on it and
13:33.140 --> 13:39.780
naming things is notoriously hard it's the hardest problem in computer science right inflation says naming
13:39.780 --> 13:46.180
is basically a tiny highly compressed message that you are sending to a future searcher yes and there is
13:46.180 --> 13:52.980
a mathematical limit to this called label capacity label capacity yeah fleischin posits this formula
13:53.780 --> 14:00.020
if you have a naming policy that uses a set of l distinct labels to categorize a total of n artifacts
14:00.020 --> 14:06.260
okay wait let me map this out as you go sure let's say i have a folder with a thousand files in it so n is
14:06.260 --> 14:12.820
one thousand okay n equals one thousand and let's say your naming policy only ever uses about 20 words to name
14:12.820 --> 14:21.380
those files like draft final tax project v2 typical yeah so l is 20 exactly the math says that a searcher
14:21.380 --> 14:26.820
who only knows the label will succeed in finding the specific artifact with a probability of l divided
14:26.820 --> 14:34.580
by n so 20 divided by 1 000 right which is two percent two percent chance of finding what you need on the
14:34.580 --> 14:42.180
first try that seems incredibly fragile if search is purely sequential it is extremely fragile a naming
14:42.180 --> 14:47.380
policy mathematically cannot perform better than its label capacity if you don't use enough distinct
14:47.380 --> 14:52.820
words your names completely lose their power to differentiate but search isn't completely broken
14:52.820 --> 14:57.700
in real life so we must be doing something else right we don't just rely purely on sequential naming
14:58.260 --> 15:05.700
no we rely on conventions conventions a convention is an agreed-upon rule that constrains the search space
15:05.700 --> 15:11.220
it encodes attributes into a structure so that you don't actually have to memorize them you just compute
15:11.220 --> 15:18.500
them okay so like a music library hierarchy perfect example yeah okay if you just dump 10 000 mp3 files into a
15:18.500 --> 15:24.900
single massive directory you have to remember the exact title of a song to find it and you won't i definitely
15:24.900 --> 15:32.660
won't but if you adopt a convention like you have a folder for the artist and inside that is a subfolder for the album and the file
15:32.660 --> 15:38.900
itself is named by track number and then song title you don't have to remember the file name at all exactly
15:38.900 --> 15:43.860
you just compute the path based on the structure you say to yourself i want the third track on this
15:43.860 --> 15:51.700
specific album you have replaced raw memory with logical calculation that is so elegant yeah but Flyxion
15:51.700 --> 15:58.020
also mentions this applies to what words we use too right naming things for what they are rather than how they
15:58.020 --> 16:05.460
are implemented yes because implementations change constantly but problem domains endure okay explain
16:05.460 --> 16:11.220
that difference so if you name a code file based on its underlying data structure let's say you name
16:11.220 --> 16:18.500
it hashmapprocessor.py okay in five years a new developer might rewrite that logic using a completely
16:18.500 --> 16:24.980
different data architecture it's not a hash map anymore now the name is actively lying to you right but if you
16:24.980 --> 16:32.420
name it for the enduring problem it solves like tax liability calculator dot py a future searcher
16:32.420 --> 16:37.700
will always know they are looking for that specific domain logic even if the tech stack has completely
16:37.700 --> 16:43.380
turned over three times since it was written because the concept of calculating taxes outlives the code
16:43.380 --> 16:50.260
used to do it precisely okay so the record channel is really this combination of naming structure and search
16:51.060 --> 16:56.100
but Flyxion points out this whole thing is a multiplicative bottleneck yes the chain of
16:56.100 --> 17:01.780
conditional probabilities right to recover something you have to form a query then you have to search an
17:01.780 --> 17:06.980
index then select a candidate from the results and finally interpret it and because these are
17:06.980 --> 17:13.140
multiplicative probabilities meaning you multiply the chances of each step succeeding together the entire
17:13.140 --> 17:21.700
system is bottlenecked by your absolute worst factor okay so if i have absolutely perfect naming conventions
17:21.700 --> 17:28.500
like 100 success rate on finding the file right but i have zero documentation on how to interpret the
17:28.500 --> 17:36.980
file once it's open my total recoverability crashes to zero exactly one time zero is zero fixing the naming
17:36.980 --> 17:43.300
even more won't help you at all you have to address the context gap finding the file again only
17:43.300 --> 17:50.900
satisfies referential persistence right the second level yes to achieve operational persistence to actually use the
17:50.900 --> 17:58.580
thing you have to climb what Flyxion calls the ladder of capability or the operation classes the ladder of
17:58.580 --> 18:04.580
capability i love that imagery let's climb it uh what's at the bottom at the very bottom is
18:04.580 --> 18:12.100
inspection okay can i just open a random file in a hex editor or a text editor and see the raw bytes
18:12.100 --> 18:19.620
that requires almost zero context right i just need a screen one rung up is execution can i actually run
18:19.620 --> 18:26.740
the script that requires more context yeah that requires knowing the operating system the language version
18:26.740 --> 18:33.700
maybe the processor architecture okay what's next then to reach correct use correct use so not just
18:33.700 --> 18:40.260
running it but getting it to do what i want exactly can you run it on the intended inputs and get a
18:40.260 --> 18:46.020
meaningful output you need to know what the program actually expects from you like does it expect dates
18:46.020 --> 18:52.340
in a european format or an american format right day month year versus month day year right i'll crash
18:52.340 --> 18:59.380
your program fast oh absolutely or does it expect floating point decimals or whole integers okay so that's
18:59.380 --> 19:07.620
correct use and climbing higher requires safe modification yes can i actually change the code
19:07.620 --> 19:12.740
without breaking the hidden assumptions that it relies on which is terrifying in old code extremely
19:13.380 --> 19:20.820
above that is reproduction can i use this artifact to exactly reconstruct a specific scientific result
19:20.820 --> 19:27.540
or a previous build and the very peak the absolute peak is evaluation can i independently
19:27.540 --> 19:32.740
judge whether the artifact is scientifically or technically correct for the purpose it claims to serve
19:33.380 --> 19:40.020
that requires total mastery of the context okay so to support a future user climbing all those rungs
19:40.020 --> 19:46.980
i have to preserve a massive amount of context read me files test suites inline comments yes
19:46.980 --> 19:54.980
but the text introduces a really severe computational trap here a trap yeah different operations on that ladder
19:54.980 --> 20:00.100
can be satisfied by different combinations of context like you can learn how to use a function
20:00.100 --> 20:05.380
by reading the written manual or by running the test suite or by reverse engineering the source code
20:05.380 --> 20:11.700
itself right okay there are multiple paths to understanding exactly but if you as the author want to
20:11.700 --> 20:17.540
figure out the absolute minimum cheapest set of documentation you can provide that will guarantee
20:17.540 --> 20:22.340
a future reader can climb that ladder so i don't have to write a thousand page manual right
20:22.340 --> 20:30.260
the math proves that finding that optimal minimal set is an np complete optimization problem wait really
20:30.260 --> 20:38.100
yes np complete yes Flyxion actually reduces this to the classic set cover problem in computer science okay
20:39.140 --> 20:45.620
let's ground the set cover problem for a second for everyone sure if i'm an engineer i know my users
20:45.620 --> 20:50.820
have dozens of different paths they might take to understand my code right right some might read the
20:50.820 --> 20:55.860
comments some might trace the execution line by line some might just glance at the api docs
20:56.500 --> 21:01.300
and i want to write the minimum amount of documentation that successfully covers
21:01.300 --> 21:09.140
all those possible user paths yes you're saying finding that perfect optimal overlap is mathematically
21:09.140 --> 21:14.980
intractable like computers can't even solve it efficiently it is computationally impossible
21:14.980 --> 21:22.420
to solve optimally as the system scales up that is mind-blowing it's a brilliant application of complexity
21:22.420 --> 21:28.980
theory to human behavior we often berate developers right we say they write redundant documentation or
21:28.980 --> 21:35.620
incomplete documentation why didn't you just write the docs properly exactly but the mathematical reality is
21:35.620 --> 21:43.540
that they are attempting to solve an np complete problem in their heads in real time it's impossible wow
21:43.540 --> 21:51.620
this is exactly why we rely on rigid conventions like mandatory manual pages or standardized doc string
21:51.620 --> 21:58.180
templates in python oh because conventions don't solve the optimization problem they just bypass it
21:58.180 --> 22:03.380
yes they dictate a standard good enough path so you don't have to calculate the optimal one
22:04.100 --> 22:09.860
you just fill out the template that is so cool and we lean on other concepts to bypass this too right like
22:09.860 --> 22:16.260
contracts or horror logic and software engineering yes the idea of contracts the idea there is that if
22:16.260 --> 22:20.900
you document what goes into a function the preconditions and what comes out the post conditions
22:20.900 --> 22:26.420
that that is sufficient context that's the theory yeah but Flyxion completely dismisses this as an
22:26.420 --> 22:33.060
illusion specifically for that fourth rung on the ladder safe modification yes the contract tells you the
22:33.060 --> 22:39.300
interface but it massively under determines the implementation what does that mean in practice it means if you
22:39.300 --> 22:46.340
only know what goes in and what comes out you have enough context for correct use you can run it
22:46.340 --> 22:52.980
but safe modification requires understanding the why okay so i was thinking about this like a restaurant
22:52.980 --> 22:59.460
menu oh i like that go on the contract is the menu right it tells me that if i pay 30 that's the
22:59.460 --> 23:05.860
precondition i will receive a cooked steak that's the post condition right now if i'm just eating the steak
23:05.860 --> 23:10.740
correct use the contract is totally fine i pay 30 dollars i get a steak simple enough but if i want
23:10.740 --> 23:17.300
to modify the recipe safe modification the contract is completely useless the menu doesn't tell me why
23:17.300 --> 23:23.220
the chef chose that specific cut of beef or what temperature the grill is set to or what happens to
23:23.220 --> 23:29.860
the chemical composition of the glaze if i swap out the salt for sugar that's a perfect analogy yes if you
23:29.860 --> 23:34.340
only have the contract you might change the code in a way that respects the inputs and outputs you
23:34.340 --> 23:40.100
still allow put a steak but you completely destroy a hidden performance constraint or a security
23:40.100 --> 23:46.580
assumption i make a terrible steak exactly the context required for modification exists entirely
23:46.580 --> 23:53.300
outside the artifact itself and this honestly this completely undercuts how we track changes in code today
23:53.300 --> 23:59.460
you mean commits and diffs specifically commits and diffs yeah think about a diff if you change a
23:59.460 --> 24:08.340
threshold in a statistical model from 0.05 to 0.01 okay a tiny number change the diff the textual
24:08.340 --> 24:14.420
difference that the system saves just shows the bytes changing a five became a one it is fundamentally
24:14.420 --> 24:20.340
devoid of intent because the computer doesn't know why i did it right the bytes changed are visually
24:20.340 --> 24:27.860
identical whether you are tightening a statistical threshold fixing a typo or tuning a parameter to fit a
24:27.860 --> 24:33.620
specific data set and we are seeing a massive push right now literally right now in the industry
24:33.620 --> 24:39.620
for ai tools that auto generate commit messages just by looking at those text diffs oh yes it's a huge
24:39.620 --> 24:45.700
trend but if intent is completely absent from the text difference those ai tools are essentially
24:45.700 --> 24:53.300
hallucinating documentary value right 200 they are hallucinating because if a tool looks at the change from
24:53.300 --> 25:00.740
0.05 to 0.01 and it generates a message that says changed threshold to 0.01 it has merely restated the
25:00.740 --> 25:05.860
physical difference which is useless right it hasn't supplied the intent the human who wrote the change
25:05.860 --> 25:11.540
has to step in and explicitly say changed threshold because the previous value caused false positives in
25:11.540 --> 25:18.580
the european data sets exactly you mathematically cannot compute intent from a diff and without that intent
25:18.580 --> 25:23.540
the operational persistence of the whole repository just decays a little bit with every single change
25:23.540 --> 25:30.260
death by a thousand cuts wow but okay everything we've talked about so far assumes it's just you
25:30.260 --> 25:35.620
you alone at your computer wrestling with your own amnesia and your own files the solitary developer right
25:35.620 --> 25:40.980
but the second you try to apply this operational context to a shared code base with other people
25:40.980 --> 25:48.020
a totally new problem emerges identity yes how does the system even know that you are you right
25:48.020 --> 25:53.460
establishing identity and permissions and Flyxion defines a digital identity in a really specific
25:53.460 --> 26:01.140
way it's not just a name it is a four-part relation a four-part relation put that down all right first
26:01.140 --> 26:09.700
it consists of a label that's your human readable username like coder dave okay second an identifier this is
26:09.700 --> 26:16.740
the hidden numeric id the computer actually uses behind the scenes like user id 4096 right the computer
26:16.740 --> 26:22.340
doesn't care about coder dave exactly third is a history of your actions everything you've done
26:22.340 --> 26:28.340
on the system and fourth is social recognition which is how the community actually views you and trusts
26:28.340 --> 26:34.980
you so conflating the human label with the hidden identifier leads to what the monograph calls the reuse
26:34.980 --> 26:42.020
hazard right the reuse hazard yes this causes so much chaos i can imagine if a system tracks ownership
26:42.020 --> 26:48.100
of files by that hidden numeric id and an employee leaves the company so their account is deleted
26:48.100 --> 26:55.540
right but later the system recycles their username for a new hire or worse it recycles the actual numeric
26:55.540 --> 27:02.020
id to a new user total disaster the new person instantly inherits all the ownership rights and
27:02.020 --> 27:08.420
the entire history of the old person because the computer just sees id 4096 exactly the system assumes the
27:08.420 --> 27:14.820
identity is continuous because the numeric id is the same but the human reality has completely fractured
27:14.820 --> 27:20.420
that is a massive security risk it's why modernist systems try to use immutable identifiers behind
27:20.420 --> 27:27.380
the scenes things that never get recycled but because we humans rely on the label the username
27:27.380 --> 27:33.860
for our referential persistence any recycling of namespaces creates these profound continuity risks
27:33.860 --> 27:38.820
because i'm looking at the screen and i think i'm trusting the old coder dave but it's actually a new
27:38.820 --> 27:45.700
guy exactly okay so once identity is established we get into permissions and the text dives deep into
27:45.700 --> 27:52.820
the traditional unix permission check ah yes the mode bits owner group and other right and there is a
27:52.820 --> 27:59.060
specific illustration in the book example 6.1 that sounds completely paradoxical when you first read
27:59.060 --> 28:06.820
it the mode 007 example yes the 007 paradox can you explain this because i read this three times
28:06.820 --> 28:13.380
and it broke my brain it's brilliant okay so in unix permissions are checked by assigning the acting
28:13.380 --> 28:19.460
user to exactly one class and the system evaluates you sequentially okay if you are the owner of the file
28:20.100 --> 28:24.980
the system puts you in the owner class and it only looks at the owner permissions it ignores the rest
28:24.980 --> 28:31.860
completely okay got it so in octal notation a permission mode of 007 means that the owner the
28:31.860 --> 28:38.740
first digit has zero permissions the group the second digit has zero permissions right but other
28:39.380 --> 28:45.220
the third digit meaning literally everyone else in the entire system has a seven which means full read
28:45.220 --> 28:51.860
write and execute permissions so the moment i the literal creator and owner of the file try to read my
28:51.860 --> 28:58.820
own file the system categorizes me into the owner class sees i have a zero for permissions and denies
28:58.820 --> 29:05.460
me access exactly meanwhile any random stranger logs onto the system falls into the other class and has
29:05.460 --> 29:11.700
full permission to read and modify it yes you own the house but you were the only person on the planet
29:11.700 --> 29:18.420
locked out of it that is so insane why would a system even allow that state to exist because it
29:18.420 --> 29:24.100
highlights a foundational theorem of governance in digital systems governance yes you are locked out
29:24.100 --> 29:31.460
of the house but because you are the owner you hold the meta permission the meta permission you have the
29:31.460 --> 29:36.500
inherent right to run a command to change the permissions and grant yourself access again oh
29:37.620 --> 29:44.340
true governance is born the exact moment a system separates the rules of access from the right to change those
29:44.340 --> 29:50.740
rules that is deep so i don't have the right to read the file but i have the right to change the rule
29:50.740 --> 29:56.340
about who could read the file exactly at the metal level and that separation applies to shared spaces as
29:56.340 --> 30:03.300
well through mechanisms like the sticky bit the sticky bit i've seen that term what does that do well if
30:03.300 --> 30:08.820
you have a shared directory where anyone can write a file like a public dropbox folder you have a major
30:08.820 --> 30:13.700
vulnerability because the right permission on a directory normally allows you to delete any file
30:13.700 --> 30:20.020
within it wait really yeah if i can write to a folder i can delete anything in it in basic unix yes
30:20.020 --> 30:26.020
so in a shared space anyone could delete everyone else's work that sounds like a terrible design for a
30:26.020 --> 30:32.820
shared folder it is so the sticky bit was invented as a targeted governance rule it restricts deletion in
30:32.820 --> 30:38.260
a shared directory so that you can only delete files that you specifically own oh that makes sense
30:38.260 --> 30:43.140
it explicitly separates the right to contribute to a shared space from the right to destroy
30:43.140 --> 30:49.140
the shared space contribution is not control contribution is not control i love that okay
30:49.140 --> 30:55.140
so permissions dictate who can touch the files locally but modern files don't live in isolation
30:55.140 --> 31:01.700
anymore not at all they rely on these massive labyrinthine networks of other people's files the operational
31:01.700 --> 31:07.940
persistence of your code is entirely dependent on the operational persistence of code you don't even control
31:08.260 --> 31:15.380
we are entering the dependency web a dependency web let's start small just a script on my machine
31:15.380 --> 31:20.340
sure a script is basically just an externalized procedure you write it down so you don't have
31:20.340 --> 31:25.860
to memorize a sequence of commands right right but scripts are incredibly fragile because they rely on
31:25.860 --> 31:31.060
implicit preconditions a script might assume you have a specific version of a database running in the
31:31.060 --> 31:36.260
background or that a certain environment variable is set on your machine right things that aren't actually in the
31:36.260 --> 31:43.380
code itself exactly those are undeclared dependencies well we try to mitigate that with idempotence right yes idempotent
31:43.940 --> 31:51.780
the mathematical property where f of f of x equals f of x right which basically means if a script fails halfway
31:51.780 --> 31:57.060
through you can just run it again from the top and it won't duplicate records or corrupt the state it just
31:57.060 --> 32:04.180
safely finishes what it started that's the ideal yeah but Flyxion points out a massive blind spot in how
32:04.180 --> 32:13.060
we combine these scripts composing two safe idempotent scripts isn't safe if they don't commute if they
32:13.060 --> 32:20.820
don't commute like in what order you run them exactly if script a relies on a state created by script b
32:21.460 --> 32:27.700
the execution order dictates the operational success you can have two perfectly safe scripts but run them
32:27.700 --> 32:34.340
in the wrong order and everything breaks and that fragility just explodes when we move from simple
32:34.340 --> 32:39.620
scripts to the massive dependency graph of modern software it's terrifying because you declare that
32:39.620 --> 32:47.540
your software needs library x but library x secretly relies on library y and library y requires a specific
32:47.540 --> 32:55.620
system library to be installed which leads us to dll hell dll hell dynamic link library conflicts
32:56.340 --> 33:03.700
it's the classic single slot namespace conflict what is a single slot conflict okay so imagine program a
33:03.700 --> 33:10.260
needs version one of a library and program b needs version two of that exact same library but your operating
33:10.260 --> 33:16.820
system only has one slot in memory or one location on disk for that library's specific name so they
33:16.820 --> 33:22.100
literally cannot coexist on the same machine exactly they are competing for this single slot if you
33:22.100 --> 33:28.500
install version two for program b program a instantly breaks reinvented package managers to solve this
33:28.500 --> 33:35.220
though right like npm for node or cargo for rust we did yeah they build these massive dependency trees
33:35.220 --> 33:40.180
to ensure every package gets the exact version it needs sometimes isolating them in their own little
33:40.180 --> 33:46.260
folders right but phylaxon pulls the rug out from under this entire concept yeah this part blew my mind
33:46.260 --> 33:51.860
resolving package dependencies isn't just a difficult engineering task it is mathematically intractable
33:51.860 --> 33:56.980
it's np complete it's np complete the text actually proves that finding a valid conflict-free set of
33:56.980 --> 34:02.980
dependencies is mathematically equivalent to the boolean satisfiability problem specifically 3sat
34:03.620 --> 34:09.140
okay hold on if i'm a software engineer and i'm just trying to fix a failing build on my laptop i'm not
34:09.140 --> 34:16.100
thinking about abstract logic puzzles how does a simple npm install command map directly to a 3sat
34:16.100 --> 34:22.660
problem it maps perfectly okay in 3sat you have a massive list of variables that must be assigned
34:22.660 --> 34:29.940
either true or false and you have a list of logical clauses that must be satisfied simultaneously okay a
34:29.940 --> 34:37.940
clause might dictate something like either variable a is true or variable b is false or variable c is true okay
34:37.940 --> 34:44.660
so a bunch of interconnected roles exactly in package management every version of every single package
34:44.660 --> 34:52.660
is a boolean variable is package a version 1.2 installed true or false oh and the dependencies
34:52.660 --> 35:00.180
are the clauses so if package a version 1.2 is installed so it's true then package b version 2.0 must be true
35:00.180 --> 35:09.540
but package c version 1.5 might require package b to be false because it needs an older version oh my god
35:10.340 --> 35:17.460
the logic grid just locks up it locks up completely every time you declare a requirement like i need
35:17.460 --> 35:23.700
version 1.2 or higher you are adding a massively complex set of clauses to the solver and as the