diff --git a/site/.gitignore b/site/.gitignore deleted file mode 100644 index 73af841..0000000 --- a/site/.gitignore +++ /dev/null @@ -1,5 +0,0 @@ -*.swp -*~ -.vscode -.DS_Store -venv \ No newline at end of file diff --git a/site/README.html b/site/README.html deleted file mode 100644 index b7c2a05..0000000 --- a/site/README.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Readme - - - - - -
- -

Readme

-
- Last modified: 2024-02-09 - -
-
-

My notes for everything

-

This is a repo where I store my notes and resources. I write my notes in markdown with vim, and use a custom GUI to view them. Some of them are taken while reading textbooks, and others are taken in real time during lectures.

-

Please note that much of the info on here is paraphrased from other sources, and I claim no ownership of it. This is also a work in progress, and I am constantly adding to and editing it. If you have any questions or comments, feel free to reach out.

-
- -
- - \ No newline at end of file diff --git a/site/algorithms/BFS.html b/site/algorithms/BFS.html deleted file mode 100644 index 3a8f95c..0000000 --- a/site/algorithms/BFS.html +++ /dev/null @@ -1,274 +0,0 @@ - - - - - - Breadth First Search Algorithm Implementation and Analysis - - - - - -
- -

Breadth First Search Algorithm Implementation and Analysis

-
- Last modified: 2025-01-01 - Category: algorithms -
-
-

Breadth First Search

-

Completely explore the vertices of a graph in order of their distance from the starting node.

-

There are three states of a vertex in BFS: -- Undiscovered: The vertex has not been seen yet. -- Discovered: The vertex has been seen, but its neighbors have not been explored yet. -- Explored: The vertex has been seen and its neighbors have been explored.

-

Algorithm

-
BFS(G, s):
-  mark all vertices as undiscovered
-
-  mark s as discovered
-  q = queue({s})
-  while q is not empty:
-    u = poll(q)
-    for each edge (u, v) in G:
-      if v is undiscovered:
-        mark v as discovered
-        add v to q
-    mark u as explored
-
-

Analysis

-

The outer while loop runs once for each vertex in the graph, and the inner for loop runs once for each edge of the current node. Remembering that the sum of the degrees of all vertices is equal to twice the number of edges in the graph, we have...

-

$$ -O(|V|) + O(\sum_{v \in V} deg(v)) = O(|V| + |E|) -$$

-

Lemmas

-
    -
  1. $BFS(s)$ visits a vertex $v$ if and only if there is a path from $s$ to $v$.
  2. -
  3. Edges into then-unexplored vertices form a tree rooted at $s$ (the BFS spanning tree).
  4. -
  5. Level $i$ in the tree are exactly all vertices $v$ stuch that the shortest path from $s$ to $v$ has $i$ edges.
  6. -
  7. All non-tree edges from $G$ connect vertices in the same level or adjacent levels.
  8. -
-

Difference in levels

-

Let $L(v)$ be the level of vertex $v$ in a BFS tree of interest.

-

Claim: -$$ -\forall (x, y) \in E, |L(x) - L(y)| \le 1 -$$

-

Proof: -Suppose $L(x) = i$ and $L(y) = j$. Without loss of generality, assume $x$ is explored before $y$.

-

Consider the iteration where we process $x$.

-

Case 1: $y$ is still undiscovered. Since there is an edge between $x$ and $y$, we will discover $y$ in the next iteration, and so $L(y) = i + 1$.

-

Case 2: $y$ is discovered. Then $y$ is already in the queue, somewhere before $x$. We know $L(y) \ge i$ because $x$ was discovered before $y$. Since the levels are non-decreasing, and $L(x) = i$, we have $L(y) \le i + 1$.

-

Thus, $|L(x) - L(y)| \le 1$.

-

Shortest paths

-

Claim:

-

For every vertex $v \in V$ reachable from $s$, $L(v)$ is the length of the shortest path from $s$ to $v$.

-

Proof:

-

Let $l(v)$ be the length of the shortest path from $s$ to $v$.

-

We have that $L(v) \ge l(v)$, since $L(v)$ is the length of a valid path from $s$ to $v$, so the shortest path must be at least as short.

-

Next, we must show that $L(v) \le l(v)$.

-

Let $v_0, v_1, \ldots, v_k$ be the shortest path from $s$ to $v$ (with $v_0 = s$).

-

$$ -\forall v \in BFS(G, s), L(v) = \text{length of the shortest path from } s \text{ to } v -$$

-
from collections import deque
-
-def bfs(graph, start):
-    visited = set()
-    queue = deque([start])
-    visited.add(start)
-    while queue:
-        vertex = queue.popleft()
-        print(vertex)
-        for neighbor in graph[vertex]:
-            if neighbor not in visited:
-                visited.add(neighbor)
-                queue.append(neighbor)
-
-

Or a reusable level-order iterator over a graph using BFS:

-
from collections import deque
-
-def level_order_traversal(graph, start):
-    queue = deque([(start, 0)])
-    while queue:
-        vertex, level = queue.popleft()
-        yield vertex, level
-        for neighbor in graph[vertex]:
-            queue.append((neighbor, level + 1))
-
-
-
Tags: complexity-analysis, graph-theory, graph-traversal, shortest-paths
-
- - \ No newline at end of file diff --git a/site/algorithms/DAGs.html b/site/algorithms/DAGs.html deleted file mode 100644 index 5733cd8..0000000 --- a/site/algorithms/DAGs.html +++ /dev/null @@ -1,234 +0,0 @@ - - - - - - Topological Ordering and Properties of Directed Acyclic Graphs - - - - - -
- -

Topological Ordering and Properties of Directed Acyclic Graphs

-
- Last modified: 2025-01-01 - Category: algorithms -
-
- -
-

Directed Acyclic Graphs (DAGs)

-

DAGs are pretty self explanatory, but their use cases are vast.

-

Topological Orderings

-

A topological ordering of a directed graph $G = (V, E)$ is a linear ordering of all its vertices such that for every directed edge $(v_i, v_j) \in E$, $v_i$ comes before $v_j$ in the ordering if $v_i < v_j$.

-

Lemma: If $G$ has a topological ordering, then $G$ is a DAG. -Proof: For contradiction, assume $G$ has a cycle $v_0, \ldots , v_k$, as well as a topological ordering.

-

We can order vertices $u_1, \ldots, u_n$ such that $\forall \text{ directed edges } i \to j$, we have $i < j$.

-

Take the smallest $u_i = v_j$ in the cycle mentioned previously. Then $v_{j - 1} \to v_{j}$ and $v_{j} \to v_{j + 1}$ violate our ordering, since $v_j$ was the minimum in the topological ordering (so both $v_{(j - 1) \mod k}$ and $v_{(j + 1) \mod k}$ are greater).

-

Lemma: If $G$ is a DAG, then $G$ has a source vertex ($indeg(v) = 0$). -Proof: Suppose for contradiction that $G$ has no source vertex.

-

i.e., $\forall v \in V, \, indeg(v) \ge 1$

-

Consider an arbitrary vertex $v_1$. Then $v_1$ has some neighbor(s) $v_2, \ldots$ with an edge into it. Similarly, $v_2$ has some neighbor(s) $v_i, \ldots$ with edges coming into it. You can continue this logic, and must eventually find a repeating vertex, since there are finitely many vertices.

-

Algorithm

-
def topological_sort(G):
-  order = []
-  count = [0] * len(G)
-  S = { v for v in G if not G[v] }
-  for v in G
-    for u in G[v]:
-      count[u] += 1
-
-  while S:
-    v = S.pop()
-    order.append(v)
-
-    for u in G[v]:
-      count[u] -= 1
-      if count[u] == 0:
-        S.add(u)
-
-  return order
-
-
- -
- - \ No newline at end of file diff --git a/site/algorithms/DFS.html b/site/algorithms/DFS.html deleted file mode 100644 index bf8f1c6..0000000 --- a/site/algorithms/DFS.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - - - Depth First Search Algorithm and Tree Properties - - - - - -
- -

Depth First Search Algorithm and Tree Properties

-
- Last modified: 2025-01-01 - Category: algorithms -
-
-

Depth First Search (DFS)

-

Running DFS on a graph produces a DFS tree (or depth-first spanning-tree). The DFS tree contains all the vertices of the graph and the edges of the DFS tree are a subset of the edges of the original graph.

-

Unlike the BFS tree, DFS trees aren't minimum depth, and its levels don't really tell you much. However, the property holds that sub-trees of a DFS tree must not contain any edges connecting them.

-

Lemma: For a DFS tree of graph $G = (V, E)$ $T = (V_t, E_t)$, $\forall e = (x, y) \in E$, if $e \notin E_t$, then one of $x$ or $y$ is an ancestor of the other in the tree.

-

Proof: Without loss of generality, assume $x$ is discovered first.

-

Call $dfs(x)$. At this time, $y$ is still undiscovered. By observation, it is enough to say $y$ will be discovered before finishing $dfs(x)$. This is true because $y$ is a neighbor of $x$, so DFS will eventually visit $y$. If $y$ is still undiscovered when $x$ we visit $x$'s neighbors, it will at least be discovered then.

-

-def dfs_recursive(G, src, vis = set(), f=print):
-  if src in vis:
-    return
-  vis.add(src)
-  f(src)
-  for v in G[src]:
-    dfs_recursive(G, v, vis)
-
-def dfs_iterative(G, src, vis=set(), f=print):
-  stack = [src]
-  while stack:
-    curr = stack.pop()
-    if curr in vis:
-      continue
-    vis.add(curr)
-    f(curr)
-    for v in G[curr]:
-      stack.append(v)
-
-

Properties of DFS Spanning Trees

-

DFS visits every vertex within the starting vertex's connected component, so you can use it to find all connected components of a graph similar to BFS.

-

However, unlike BFS, the DFS tree has the property that every non-tree edge joins a vertex to one of its ancestors/decedents in the tree. We can thus still use DFS to find cycles in a graph.

-
-
Tags: depth first search, graph theory, graph traversal, spanning trees
-
- - \ No newline at end of file diff --git a/site/algorithms/approximation-algorithms.html b/site/algorithms/approximation-algorithms.html deleted file mode 100644 index 231421d..0000000 --- a/site/algorithms/approximation-algorithms.html +++ /dev/null @@ -1,224 +0,0 @@ - - - - - - Approximation Algorithms - - - - - -
- -

Approximation Algorithms

-
- Last modified: 2025-01-01 - Category: algorithms -
-
-

Approximation Algorithms

-

When faced with a problem that can be reduced to some NP-Complete problem, you (most probably) cannot generally solve it in polynomial time. For example:

- -

Instead of finding an optimal solution in polynomial time, we have two approaches:

- -

Approximation Ratio

-

$$ -\alpha = \frac{\text{computed solution}}{\text{optimum solution}} -$$

-

If we can prove some upper or lower bound on $\alpha$, then we might be able to use and reason about a given approximation algorithm. Finding better approximations is an open problem.

-

A Survey of Approximation Algorithms

-

The following two examples are the best known general approximation algorithms for their respective problems.

-

2-Approximation for Vertex Cover

-

Problem: find the minimal subset $S$ of vertices in a graph such that every edge is connected to some vertex in $S$. -Algorithm: For every edge $(u, v)$, add $u$ and $v$ to $S$

-

By a 2-approximation, it means that $\alpha = 2$. Since we are minimizing the set, we have that that for any graph $G$, it must be the case that $OPT(G) \le ALG(G) \le 2 \cdot OPT(G)$

-

log(n) approximation for Set Cover

-

Problem: Given some number of sets $S_1, S_2, \ldots, S_n$ with $S_i \subseteq U$, choose the minimum number of sets that cover all elements of $U$ -Algorithm: While there are remaining elements, choose the set that maximizes the number of new elements covered.

-

If the optimal solution has $k$ sets, this algorithm always selects at most $k log(n)$ sets. This is because there is at least a set that covers $\frac{1}{k}$ of the remaining elements, so after $t$ steps we have $ \le n(1 - \frac{1}{k})^t \le n \cdot e^{-\frac{t}{k}}$ remaining elements. Therefore, after $t = k\ln(n)$ steps, we have $< 1$ uncovered element remaining.

-
-
Tags: algorithms, approximation, set cover, vertex cover
-
- - \ No newline at end of file diff --git a/site/algorithms/bipartite-graphs.html b/site/algorithms/bipartite-graphs.html deleted file mode 100644 index 7c4d4c3..0000000 --- a/site/algorithms/bipartite-graphs.html +++ /dev/null @@ -1,213 +0,0 @@ - - - - - - Bipartite Graphs Properties, Proofs, and Detection Algorithm - - - - - -
- -

Bipartite Graphs Properties, Proofs, and Detection Algorithm

-
- Last modified: 2025-01-01 - Category: Graph Theory -
-
-

Bipartite Graphs

- -

You can tell if a graph is bipartite if there is a proper coloring of vertices, i.e., you can assign one of two colors to each vertex such that no two adjacent vertices have the same color. Many problems become easier if the underlying graph is bipartite graphs.

-

Odd-Length Cycles

-

Lemma: If $G$ is bipartite, then it does not contain an odd-length cycle. -Proof: You cannot 2-color an odd cycle, let alone $G$.

-

Lemma: Let $G$ be a connected graph, and let $L_0, \ldots, L_k$ be the layers produced by $BFS(s)$. Then exactly one of the following holds:

-
    -
  1. No edges of $G$ joins two nodes of the same layer, and $G$ is bipartite.
  2. -
  3. An edge of $G$ joins two nodes of the same layer, and $G$ contains an odd cycle (and is thus not bipartite).
  4. -
-

Proof: If an edge joins two nodes of the same layer, then the path from the lowest common ancestor of the two nodes to each node forms an odd length cycle. This must be the case, since any edges between two vertices of the same level connects two paths of the same length back to their LCA of the BFS tree. The length of this cycle is thus $2k + 1$, where $k$ is the length of back to the LCA, and the $1$ comes from the edge between the two nodes in the same level.

-

Algorithm

-

Problem: Given a graph $G$, output true if it is bipartite, false otherwise.

-
-
Tags: bipartite graphs, breadth-first search, graph coloring, odd cycles
-
- - \ No newline at end of file diff --git a/site/algorithms/connected-components.html b/site/algorithms/connected-components.html deleted file mode 100644 index 4efd942..0000000 --- a/site/algorithms/connected-components.html +++ /dev/null @@ -1,236 +0,0 @@ - - - - - - Finding Connected Components in Undirected Graphs Using BFS/DFS - - - - - -
- -

Finding Connected Components in Undirected Graphs Using BFS/DFS

-
- Last modified: 2025-01-01 - Category: algorithms -
-
-

Connected Components

-

Given an undirected graph $G = (V, E)$, you can find partition $V$ into sets of connected components $C_1, C_2, \ldots$ in $O(|V| + |E|)$ using breadth-first search (BFS) or depth-first search (DFS).

-

In other words, we can create a data structure from $G$ such that given two vertices $u, v \in V$, we we answer whether there exists a path from $u \to v$ in $O(1)$ time and $O(|V|)$ space.

-

The basic idea is to run a BFS/DFS starting at every vertex, and to assign a label to each vertex we visit during this traversal. We can use an array (if vertices are numbered) or hash map to store the vertex to component set mapping $V \to C_i$

-
import networkx as nx
-import random
-from collections import deque, defaultdict
-
-def connected_components(graph):
-  a = [None] * len(graph)
-
-  def bfs(label, G, src):
-    q = deque()
-    vis = set()
-
-    q.append(src)
-    while q:
-      curr = q.popleft()
-      vis.add(curr)
-      a[curr] = label
-      for v in G[curr]:
-        if v not in vis:
-          q.append(v)
-
-  curr_label = 0
-
-  for v in range(len(graph)):
-    bfs(curr_label, graph, v)
-    curr_label += 1
-  return a
-
-def component_sets(G):
-  n = len(G)
-  comp = connected_components(G)
-  component_dict = defaultdict(lambda: set())
-  for v, c in enumerate(comp):
-    component_dict[c].add(v)
-
-  return list(component_dict.values())
-
-

Strategy for Unconnected Graph

-

In general, if you are solving a graph problem you should first assume your graph is fully connected, and then after you've found a solution for connected graphs, you can run your algorithm on all the connected components of your graph.

-
-
Tags: breadth-first search, connected components, depth-first search, graph theory
-
- - \ No newline at end of file diff --git a/site/algorithms/divide-and-conquer.html b/site/algorithms/divide-and-conquer.html deleted file mode 100644 index ce7c5d1..0000000 --- a/site/algorithms/divide-and-conquer.html +++ /dev/null @@ -1,351 +0,0 @@ - - - - - - Divide and Conquer Algorithm Analysis with Implementation Examples - - - - - -
- -

Divide and Conquer Algorithm Analysis with Implementation Examples

-
- Last modified: 2025-01-01 - Category: algorithms -
-
- -
-

Divide and Conquer

-

Reduce problem to multiple sub-problems. While in induction, you typically only reduce your problem size by 1, with divide and conquer it is more common to reduce to some constant fraction of your original problem size. After recursively solving each sub-problem, merge the solutions.

-

Examples: Merge-sort, Binary Search, Strassen's Algorithm

-

Why Balanced Partitioning?

-

With regular induction, split problem into $n - 1$ and $1$, then solve the $n - 1$ and merge the solution with the $1$.

-

$$ -T(n) = T(n - 1) + T(1) + n -$$

-

Instead, divide into two problems of size $\frac{n}{2}$

-

$$ -2T(\frac{n}{2}) + n = \frac{n^2}{2} + n -$$

-

Divide and Conquer Approach

- -

Finding the Root of a Function

-

Given a continuous function $f$ and two points $a, b$ and $b > a$ such that $f(a) \le 0$ and $f(b) \ge 0$.

-

Find an approximate root of $f$, ie a point $c$ where there is $r$ s.t. $|r - c| \le \epsilon$ and $f(r) = 0$. Note that this must exist by IVT.

-

Naive Approach

-

Divide $[a, b]$ into $n = \frac{b - a}{\epsilon}$ intervals. For each interval, check $f(x) \le 0, f(x + \epsilon) \ge 0$.

-

This runs in $O(n) = O(\frac{b - a}{\epsilon})$.

-

D&C Approach

-
def Bisection(f, a, b, e):
-    if (b - a) < e:
-        return a
-
-    m = (a + b)/2
-    if f(m) < 0:
-        return Bisection(c, b, e)
-    else
-        return Bisection(a, c, e)
-
-

Let $n = \frac{a - b}{\epsilon}$ and $c = \frac{a + b}{2}$. SO in each step we reduce by half.

-

$$ -T(n) = T(\frac{n}{2}) + O(1) = O(\log(n)) = O(\log \frac{a - b}{\epsilon}) -$$

-

Correctness

-

$P(n)$: $\forall a, b$ s.t. $f(a) \le 0$ and $f(b) \ge 0$, and $\frac{a - b}{\epsilon} = n$, that Bisection returns a value, say $c$, s.t. $\exists r$ s.t. $|r - c| \le \epsilon$

-

Base Case. $P(1)$: By IVT, $\exists r \in [a, b]$ s.t. $f(r) = 0$. We output $a$, and $|a - r| \le |a - b| \le \epsilon$.

-

IH: Assume $P(n)$

-

IS: $P(2n)$

-

Given arbitrary $b > a$ s.t. $\frac{|a - b|}{\epsilon} = 2n$ and $f(a) \le 0$, $f(b) \ge 0$, start with $c = \frac{a + b}{2}$.

-

Case 1: $f(c) \ge 0$

-

Then we satisfy $P(n)$ for $a, c$ because $\frac{c - a}{\epsilon} = n$, and $f(a) \le 0$, $f(c) \ge 0$.

-

Case 2: $f(c) \le 0$

-

Then we satisfy $P(n)$ for $c, b$, for the exact same reasoning.

-

Closest Pair of Points (geometrically)

-

Given $n$ points and an arbitrary distance function between them, find the closest pair (not just Euclidean distance).

-

1 Dimensional Version

-

Given $n$ points on the real line, you can find the closest pair by sorting and then comparing each consecutive pair of points.

-

Key point: Don't need to check every pair. Can instead exploit geometry.

-

2 Dimensional Version

-

Given $n$ points in the plane$, find the pair with the smallest Euclidean distance between them.

- -

Suppose $\delta$ is the minimum on each side. Then you only need to consider points within $\delta$ of $L$ as the case where our two points lie on opposite sides of $L$.

-

Partition each side of $L$ into $\frac{\delta}{2} \times \frac{\delta}{2}$ squares. This guarantees that each square has at most one point, since multiple points in the same square would have to have been the minimum on that side (since the maximum distance within a single square is $\frac{\delta}{\sqrt{2}}$).

-

Then, sort the points in the strip $x \in [L - \delta, L + \delta]$ by $y$-coordinate to make $s_1, s_2, \ldots s_i$.

-

Claim: $\forall s_i, s_j$, if $|i - j| > 11$, then $d(s_i, s_j) > \delta$.

-

Proof: There are 4 squares in each row of the strip, each occupying at most on point. We know that any point $p_j$ more than two rows away from $p_i$ must have $d(p_i, p_j) > \delta$.

-

There are at most 3 points in the same row as $p_i$, and 8 in the 2 rows above/below. Thus, any point more than $8 + 3 = 11$ points away from $p_i$ will be more than two rows away, and thus have a distance greater than $\delta$.

-

Implementation

-
# divide and conquer closest points
-
-def bounding_indices(P, low, high, key=lambda x: x[0]):
-    n = len(P)
-    l, r = 0, n - 1
-
-    while l <= r:
-        mid = (l + r) // 2
-        if low <= key(P[mid]):
-            r = mid - 1
-        else:
-            l = mid + 1
-    smallest_index = l
-
-    l, r = 0, n - 1
-    while l <= r:
-        mid = (l + r) // 2
-        if high >= key(P[mid]):
-            l = mid + 1
-        else:
-            r = mid - 1
-    highest_index = r
-
-    return smallest_index, highest_index
-
-def d(p1, p2):
-  if p1 is None or p2 is None:
-    return float('inf')
-  return ((p1[0] - p2[0]) ** 2 + (p1[1] - p2[1]) ** 2) ** .5
-
-def cp_brute_force(P):
-  n = len(P)
-  ans = P[:2]
-  min_d = d(P[0], P[1])
-  for i in range(n):
-    for j in range(i + 1):
-      if i == j:
-        continue
-      curr_d = d(P[i], P[j])
-      if curr_d < min_d:
-        min_d = curr_d
-        ans = [P[i], P[j]]
-  return ans
-
-def cp_recursive(P):
-  n = len(P)
-  if n < 2:
-    return None, None
-
-  if n <= 10:
-    return cp_brute_force(P)
-
-  l1, l2 = cp_recursive(P[:n//2])
-  r1, r2 = cp_recursive(P[n//2:])
-
-  m1, m2 = (l1, l2) if d(l1, l2) < d(r1, r2) else (r1, r2)
-
-  delta = d(m1, m2)
-  L = (P[n//2][0] + P[n//2 + 1][0]) / 2
-
-  l, h = bounding_indices(P, L - delta, L + delta)
-
-  middle = sorted(P[l:h + 1], key=lambda x: x[1])
-  k = len(middle)
-  for i in range(k):
-    low = max(0, i - 11)
-    high = min(k, i + 11)
-    for j in range(low, high):
-      if i == j:
-        continue
-      curr_dist = d(middle[i], middle[j])
-      if curr_dist  < delta:
-        delta = curr_dist
-        m1, m2 = middle[i], middle[j]
-
-  return m1, m2
-
-def closest_points(P):
-  return cp_recursive(sorted(P, key = lambda x: x[0]))
-
-
-
- -
- - \ No newline at end of file diff --git a/site/algorithms/dynamic-programming.html b/site/algorithms/dynamic-programming.html deleted file mode 100644 index 1eafe9f..0000000 --- a/site/algorithms/dynamic-programming.html +++ /dev/null @@ -1,814 +0,0 @@ - - - - - - Dynamic Programming Algorithms and Problem Solutions Guide - - - - - -
- -

Dynamic Programming Algorithms and Problem Solutions Guide

-
- Last modified: 2025-01-01 - Category: algorithms -
-
-

Dynamic Programming

-

Dynamic Programming is an algorithmic paradigm where you break up a problem into a series of overlapping sub-problems, building up solutions to progressively larger subproblems until the original answer is obtained. The key efficiency of DP is to memoize the answers to sub problems, often yielding a polynomial time algorithm.

-

You can design dynamic programming algorithms by induction, with the added construct of somehow memorizing/caching previously solved problems. The key then becomes finding a valid recurrence relation that relates a given instance of the problem to its composite subproblems.

-

Weighted Interval Scheduling

-

Problem: Given a set of jobs $J$ with start times $s_i$, finish times $f_i$, and weights $w_i$, find the maximum weight subset of jobs that are compatible with each other.

-

A naive approach would be to use the following induction:

-

Given jobs $j_1, j_2, \ldots, j_n$, suppose we can compute the optimum job scheduling for $< n$ jobs.

-

Then, for any $n$ jobs we can compute $OPT$ as:

- -

However, this approach is unfortunately still exponential, since there are potentially $2^n$ possible subsets of jobs to consider. Note that this problem is equivalent to maximum independent set, which is NP-complete. To differentiate our solution from the general (supposed) unsolvability of this problem, we can instead rely on an extra property of our inputs, namely that they are partially order-able.

-

So we sort by finishing time of each job, which reduces the number of subproblems from $O(2^n) \to O(n)$, i.e. the $n$ possible prefix subsets of our sorted jobs.

-

Algorithm: Given weighted jobs sorted by finish time, suppose we can compute the $OPT$ for $< n$ jobs.

- -
# J[i] = (s_i, f_i, c_i)
-def max_weighted_interval_subset(J: tuple[int, int, int]) -> int:
-    J = sorted(J, key=lambda x: x[1])
-    n = len(J)
-    memo = [0] * n
-
-    def p(n: int) -> int:
-        for i in range(n - 1, -1, -1):
-            if J[i][1] <= J[n][0]:
-                return i
-        return -1
-
-    def dp(n: int) -> int:
-        if n < 0:
-            return 0
-        if memo[n] != 0:
-            return memo[n]
-        memo[n] = max(J[n][2] + dp(p(n)), dp(n - 1))
-        return memo[n]
-
-    return dp(n - 1)
-
-

Knapsack Problem

-

Problem: Given $n$ items with weights $w_1, \ldots, w_n$ and values $v_1, \ldots, v_n$, and a knapsack of capacity $W$, find the maximum value subset of items that fit in the knapsack.

-

In other words, let $I$ be our set of items, and $S \subseteq I$ be the items we select. We want to find the maximum value of $S$ such that $\sum_{i \in S} w_i \le W$

-

What are we inducting on?

-

Assume $P(i, w)$ holds $\forall w \in [0, W]$, and then show $P(i + 1, w)$ holds $\forall w \in [0, W]$. This is a bottom-up approach, where we start from the base case and build up to the final solution.

-

Algorithm: Define $OPT(i, w)$ to be the solution for items $1, \ldots, i$ with a knapsack of capacity $w$. Then, we have the following induction:

-

$$ -OPT(i, w) = \begin{cases} -OPT(i - 1, w) & \text{if } w_i > w \ -\max(v_i + OPT(i - 1, w - w_i), OPT(i - 1, w)) & \text{otherwise} -\end{cases} -$$

-
# recursive
-def knapsack_rec(W: int, w: list[int], v: list[int]) -> int:
-    n = len(w)
-    M = [[-1] * (W + 1) for _ in range(n + 1)]
-
-    def dp(i: int, w: int) -> int:
-      if M[i][w] != -1:
-        return M[i][w]
-
-      if i < 0 or w == 0:
-        M[i][w] = 0
-      elif w[i] > w:
-        M[i][w] = dp(i - 1, w)
-      else:
-        M[i][w] = max(v[i] + dp(i - 1, w - w[i]), dp(i - 1, w))
-      return M[i][w]
-
-    return dp(n, W)
-
-# iterative
-def knapsack_it(W: int, w: list[int], v: list[int]) -> int:
-    n = len(w)
-    M = [[0] * (W + 1) for _ in range(n + 1)]
-
-    for i in range(1, n + 1):
-        for j in range(1, W + 1):
-            if w[i - 1] > j:
-                M[i][j] = M[i - 1][j]
-            else:
-                M[i][j] = max(v[i - 1] + M[i - 1][j - w[i - 1]], M[i - 1][j])
-
-    return M[n][W]
-
-

String Building

-

Given 3 integers $n \geq 1$ and $k_A,k_B \geq 1$, design an algorithm that runs in time polynomial in $n, k_A, k_B$ and outputs the number of length $n$ strings composed of copies of $A, B$ such that no more than $k_A$ copies of $A$ are placed consecutively and no more than $k_B$ copies of $B$ are placed consecutively.

-

Algorithm:

-
A, B = 0, 1
-
-def num_strings(n, ka, kb):
-  dp = [[None, None]] * (n + 1)
-  dp[1][A], dp[1][B] = None, None, 1, 1
-
-  if ka == 0 or kb == 0:
-    if ka < n and kb < n:
-      return 0
-    else:
-      return 1
-
-  def f(i, c):
-    if i == 1:
-      return 1
-
-    if dp[i][c] != None:
-      return dp[i][c]
-
-    kc = ka if c == A else kb
-    notc = A if c == B else B
-
-    if i <= kc:
-      dp[i][c] = f(i - 1, notc) + f(i - 1, c)
-    else:
-      dp[i][c] = sum(f(i - j, notc) for j in range(1, kc + 1))
-
-    return dp[i][c]
-
-  return f(n, A) + f(n, B)
-
-

Correctness: A string is valid if it has $\le k_A$ consecutive $A$ and $\le k_B$ consecutive $B$.

-

Define $OPT(i, c)$ for $i, k_A, k_B \ge 1$ to be the number of valid strings ending in character $c \in {A, B}$. Our base case is $OPT(1, c) = 1$, since there is only one length-1 string ending in $c$.

-

Assuming we can compute $OPT(i, c)$ for $i < n$, we can compute $OPT(n, c)$ as follows:

-

$$ -OPT(n, c) = \begin{cases} -1 & n = 1 \ -OPT(n - 1, c') + OPT(n - 1, c) & 2 \le n \le k_c \ -\sum_{j = 1}^{k_c} OPT(n - j, c') & n > k_c -\end{cases} -$$

-

where $c'$ is the other character, and $k_c$ is the maximum number of consecutive $c$ allowed.

-

For the case where $n > k_c$, we can sum over all possible lengths of the last run of $c$, and then recurse on the remaining string. For a run of $c$ of length $k$ in a string of length $n$, we must have the first $n - k$ characters be some other valid string ending in $c'$, since otherwise our string would end in a run of $c$ of length $> k$. Therefore, we only need to count the number of strings of left $n - k_c \le i \le n - 1$ ending in $c'$, since this is exactly the number of valid strings ending in runs of $1 \le k \le k_c$ $c$.

-

In the case where $2 \le n \le k_c$, we know that there is no way we'd have a run of $c$ of more than $k_c$, so we can simply count the number of valid strings of size $n - 1$ that either end in a $c$ or a $c'$. Then, we can add a $c$ to the end of all those strings to get our valid strings of length $n$.

-

Therefore, we cover all cases and $OPT(n, c)$ computes the correct value $\forall n \ge 1$.

-

To compute the total number of valid strings of length $n$, we simply return $OPT(n, A) + OPT(n, B)$, since all valid strings either end in $A$ or $B$.

-

Running Time

-

We compute $2n$ total values of $OPT(n, c)$, $n$ corresponding to strings ending in $A$, and $n$ ending in $B$. When $n \le k_c$, we can compute our answer in constant time using the values previously computed. When $n > k_c$, we need to sum over $k_c$ previously computed values, which is $O(k_c)$. Therefore, our total running time is at most $O(nk_Ak_B)$.

-

Post Office

-

Problem: Interstate highway 5 is a straight highway from Washington all the way to California. There are $n$ villages alongside this highway. Think about the highway as an integer axis, and the position of village $i$ is an integer $x_i$ along this axis. Assume that there are no two villages in the same position, i.e., $x_i \neq x_j$ for $i \neq j$. The distance between two villages $x_i,x_j$ is simply $| x_i - x_j |$.

-

USPS is interested in building k post offices in some, but not necessarily all of the villages along highway 5, for some $1 \leq k \leq n$. A village and the post office in it have the same position. We want to choose the positions of these post offices so that the sum of the distances from each village to its nearest post office is minimized. Design an algorithm that runs in time polynomial in $n$ and outputs the minimum possible sum of distances to the optimal location for post offices.

-

Algorithm:

-
def min_dist_td(X, K):
-  N = len(X)
-  if K >= N:
-    return 0
-
-  dp = [[None] * (K + 1) for _ in range(N + 1)]
-  for n in range(N + 1):
-    for k in range(n, K + 1):
-      dp[n][k] = 0
-
-  def f(n, k):
-    if n >= k:
-      return 0
-
-    if dp[n][k] is not None:
-      return dp[n][k]
-
-    dp[n][k] = float('inf')
-
-    if k == 1:
-      dp[n][k] = sum(abs(X[i - 1] - X[n - 1]) for i in range(1, n))
-    else:
-      for i in range(k - 1, n):
-        dp[n][k] = min(
-            dp[n][k],
-            f(i, k - 1) + sum(
-              min(
-                abs(X[j - 1] - X[n - 1]),
-                abs(X[j - 1] - X[i - 1])
-              ) for j in range(i + 1, n))
-        )
-
-    return dp[n][k]
-
-  def c(i):
-    return f(i, k) + sum(abs(X[i - 1] - X[j - 1]) for j in range(i + 1, n + 1))
-
-  return min(c(i) for i in range(k, n + 1))
-
-

Correctness

-

Define $f(n, k)$ to be the minimum sum of distances from the nearest post office of the first $n$ villages to $k$ post offices given the $kth$ post office is placed in village $n$.

-

When $n \le k$, we have a sum of $0$, since every village has a post office. Then, assuming $f(i, k)$ is correct $\forall k > 0, 0 \le i < n$, we can calculate $f(n, k)$ by considering every possible placement of the $k - 1th$ post office, and choose the one that minimizes the sum of distances for villages between the $k - 1th$ and $kth$ post office. However, if $k = 1$, then the only post office we can place is the one in $x_n$, so we only need to find the distances of towns $x_1, \ldots, x_{n - 1}$ from $x_n$.

-

In the case where $k > 1$, we must consider placing a post office in towns $x_{k - 1}, x_k, \ldots, x_{n - 1}$, since it doesn't make sense to place the $k - 1th$ post office before we've seen $k - 1$ villages, and we can only place up to the village before $x_n$, since by construction it is the last of our $k$ post offices places in the $n$ towns. Therefore, we have the following recurrence:

-

$$ -f(n, k) = \begin{cases} -0 & n \le k \ -\sum_{i = 1}^{n - 1} | x_i - x_n | & k = 1 \ -\min_{k - 1 \le i < n} \left( f(i, k - 1) + \sum_{i < j < n} \min \left( | x_j - x_n |, | x_j - x_i | \right) \right) & \text{otherwise}\ -\end{cases} -$$

-

Since $f(i, k - 1)$ holds for $i < n$, and we always choose the placement of the $k - 1th$ post office that minimizes the total sum of distances, $f(n, k)$ is correctly computed.

-

Now define $c(i)$ for $k \le i \le n$ to be the total sum of distances if we place the $kth$ post office in village $x_i$ (i.e. the last post office), and all other post offices optimally. Since $f(i, k)$ correctly computes the sum of distances in this case up to the $ith$ post office, and we are guaranteed that no other post office comes after the one placed in $x_i$, all villages $x_j$ for $j > i$ will be closest to the post office in $x_i$.

-

Therefore, we can compute $c(i)$ as follows:

-

$$ -c(i) = f(i, k) + \sum_{i < j \le n} | x_i - x_j | -$$

-

Finally, in order to compute our final answer, we just need to check every placement of the last post office and return the minimum sum of distances returned. Thus, we have a minimal sum of distances of...

-

$$ -\min_{k \le i \le n} c(i) -$$

-

And so our overall optimum is as follows:

-

$$ -OPT(n, k) = \min_{k \le i \le n} {f(i, k) + \sum_{i + 1 \le j \le n} | x_i - x_j |} -$$

-

Running Time: Computing $f(n, k)$ runs in $O(n^2)$ time, and since we need to run this calculation over at most $nk$ values, to fully memoize all inputs it takes $O(kn^3)$.

-

Then, each $c(i)$ takes constant time to retrieve each $f(i, k)$, and an additional $O(n)$ to compute the sum of the remaining distances, for a total of $O(n)$.

-

Finally, we call $c(i)$ $O(n)$ times, for a total of $O(n^2)$.

-

Thus, our running time is $O(kn^3)$

-

RNA Secondary Structure

-

Problem: Given an RNA molecule $B = b_1 b_2 \ldots b_n$, find a secondary structure $S$ that maximizes the number of base pairs.

-

Note that maximizing the number of base pairs is a practical problem, since RNA molecules fold into a secondary structure that minimizes free energy, and the number of base pairs is a good proxy for free energy.

-

Rules:

- -

Algorithm:

-
# Top Down Recursive
-
-WC = { 'A': 'U', 'U': 'A', 'C': 'G', 'G': 'C' }
-
-def ssr(B):
-  N = len(B)
-  dp = [[None] * N for _ in range(N)]
-
-  for i in range(N) for j in range(N):
-    if abs(j - i) <= 4:
-      dp[i][j] = 0
-
-  def f(i, j):
-    if dp[i][j] is not None:
-      return dp[i][j]
-
-    for t in range(i, j - 5 + 1):
-      if WC[B[t]] == B[j]:
-        dp[i][j] = max(dp[i][j], 1 + f(i, t - 1) + f(t + 1, j - 1))
-
-    return dp[i][j]
-
-  return f(0, N - 1)
-
-def ssi(B):
-  N = len(B)
-  dp = [[0] * N for _ in range(N)]
-
-  for l in range(2, N + 1):
-    for i in range(N - l + 1):
-      j = i + l - 1
-      if j - i <= 4:
-        dp[i][j] = 0
-      else:
-        dp[i][j] = dp[i][j - 1]
-        for t in range(i, j - 5 + 1):
-          if WC[B[t]] == B[j]:
-            dp[i][j] = max(dp[i][j], 1 + dp[i][t - 1] + dp[t + 1][j - 1])
-
-

Correctness:

-

Let $OPT(i, j)$ be the maximum number of base pairs in a secondary structure of the substring $b_i, b_{i + 1}, \ldots, b_j$.

-

Base Case: $\forall i, j$, we have $|j - i| \le 4 \to OPT(i, j) = 0$

-

Suppose that for some $l = j - i \ge 4$, we have computed all $OPT(i, j)$. Then to compute $OPT(i, j)$ for $|j - i| > l$, we do the following:

- -

Sequence Alignment

-

Problem: Given two strings $x_1 x_2 \ldots x_m$ and $y_1 y_2 \ldots y_n$, find an alignment with the minimum number of mismatches and gaps.

-

An alignment is a set of ordered pairs $(x_{i_1}, y_{j_1}), \ldots, (x_{i_k}, y_{j_k})$ such that $i_1 < i_2 < \ldots$ and $j_1 < j_2 < \ldots$.

- -

Our base case is when either $i$ or $j$ are $0$, we just need to do $j$ or $i$ deletes respectively.

- -
# Bottom up, non memory optimized - T = O(mn), S = O(mn)
-def seq_alignment(x, y):
-  m, n = len(x), len(y)
-  dp = [[None] * (n + 1) for _ in range(m + 1)]
-  for i in range(m + 1):
-    dp[i][0] = i
-  for j in range(n + 1):
-    dp[0][j] = j
-
-  for i in range(1, m + 1):
-    for j in range(1, n + 1):
-      dp[i][j] = max(
-        (0 if x[i] == y[j] else 1) + dp[i - 1][j - 1],
-        1 + dp[i - 1][j],
-        1 + d[i][j - 1]
-      )
-
-  return dp[m][n]
-
-

Note that in computational biology, you'll be running this on strings with thousands or even millions of characters. Therefore, the space starts to become a problem. For instance, if $m = n = 100,000$, we do $10,000,000,000$ operations (which isn't terrible), but we end up with a $40$ GB dp matrix.

-

You can optimize the space by only tracking the previous row of dp.

-
# Bottom up DP, optimized for space
-def seq_alignment(x, y):
-  m, n = len(x), len(y)
-  # base cases covered
-  dp_prev = list(range(m + 1))
-  dp_curr = [None] * (m + 1)
-
-  for i in range(1, m + 1):
-    dp_curr[0] = i
-    for j in range(1, n + 1):
-      dp[curr] = min(
-        (0 if x[i] == y[j] else 1) + dp_prev[j - 1],
-        1 + dp_prev[j],
-        1 + dp_curr[j - 1]
-      )
-    for j in range(1, n + 1):
-      dp_old[j] = dp_curr[j]
-
-  return dp_curr[n]
-
-

Longest Path in a DAG

-

Problem Given a DAG $G$, find the longest path.

-

Note: This problem is NP-hard for general directed graphs , as it has the Hamiltonian Path as a special case. However, with a DAG you can solve this in polynomial time.

-

Approach: Since $G$ is a DAG, it has a topological sort. Start by sorting vertices in their topological order.

-

Let $OPT(j)$ be the length of the longest path that ends at vertex $j$ in the topological sort. We just need to guess the last edge of the longest path that ends in $j$.

-

$$ -OPT(j) = \max_{k \to j} { OPT(k) + 1 } -$$

-

Then, to get our answer we can output...

-

$$ -\max_{j \in [1, n]} OPT(j) -$$

-

Longest Increasing Subsequence

-

Given a sequence of numbers $x_1, x_2, \ldots, x_n$, find the longest increasing (not necessarily contiguous) subsequence.

-

Define $OPT(j)$ as LIS that ends at $x_j$. Our approach is to guess the previous element $x_k$. Our base case is $OPT(1) = 1$, and for any $j$ such that $\forall k < j$, $x_j < x_k$, $OPT(j) = 1$.

-

$$ -OPT(j) = \begin{cases} -1 & j = 1 \ -1 + \max_{k < j} { OPT(k) : x_k < x_j } & \text{otherwise} -\end{cases} -$$

-
def LIS(X):
-  n = len(X)
-  dp = [None] * (n + 1)
-  dp[0], dp[1] = 0, 1
-
-  def f(i):
-    if dp[i] is None:
-      dp[i] = 1 + max([0] + [f(k) for k in range(i) if X[k - 1] < X[i - 1]])
-
-    return dp[i]
-
-  return max(f(i) for i in range(1, n + 1))
-
-

Shortest Paths with Negative Edge Weights (Bellman-Ford)

-

Given a weighted directed graph $G = (V, E)$, and a source vertex $s4, where the weight of edges $e = (u, v) \in E$ is $c_e = c_{u, v}$, find the shortest path from $s$ to all other vertices.

-

Note: if $G$ has a negative cycle, there is no solution, so suppose $G$ has no negative cycles.

-

Approach: Define $OPT(v, i)$ as the length of the shortest path from $s \to v$ among all paths that use at most $i$ edges. We induct on $i$.

- -

So we have...

-

$$ -OPT(v, i) = \begin{cases} -0 & v = s, i = 0 \ -\infty & v \ne s, i = 0 \ -\min { OPT(v, i - 1), \min_{u \to v} { OPT(u, i - 1) + c_{u, v} } } & \text{otherwise} -\end{cases} -$$

-

Since any path in any graph has at most $n - 1$ edges, the shortest path to $v$ has $\le n - 1$ edges. Thus, $OPT(v, n - 1)$ is the length of the shortest path from $s \to v$.

-

Running Time: We solve $O(n)$ subproblems, each taking $O(m)$ time to solve, for a total running time of $O(nm)$.

-
def bellman_ford(G, C, s):
-  n = len(G)
-  dp = [[float('inf')] * (n + 1) for _ in range(n)]
-
-  for i in range(n + 1):
-    dp[s][i] = 0
-
-  for i in range(1, n):
-    for v in range(n):
-      dp[v][i] = dp[v][i - 1]
-      for u in range(n):
-        dp[v][i] = min(dp[v][i], dp[u][i - 1] + C[u][v])
-
-  return [dp[v][n] for v in range(n)]
-
-

P1 - Knapsack Approximation

-

Given $n$ items with integer weights $w_1, \ldots , w_n$ and values $v_1, \ldots , v_n$ and knapsack of weight $W$ such that $w_i \le \frac{W}{2}$ for all $i$. Design an algorithm that runs in time polynomial in $n$ and $\log W$ and outputs a 2-approximation for the knapsack problem.

-
def A(weights, values, W):
-  n = len(weights)
-  greed = [(values[i] / weights[i], i) for i in range(n)]
-  greed.sort(reverse=True)
-
-  w, v = 0, 0
-  for i in range(n):
-    if w + weights[greed[i][1]] <= W:
-      w += weights[greed[i][1]]
-      v += values[greed[i][1]]
-    else:
-      break
-
-  return v
-
-

Correctness: Let $A$ be my algorithm above, $G$ be an algorithm that makes the same greedy choice of selecting items in increasing order of $\frac{v_i}{w_i}$, but with fractional selection allowed, and $OPT$ be the optimal algorithm for selecting items of only whole quantities. Let $S$ be the set of items chosen by $A$, $S'$ be the set of items chosen by $G$, and $S^*$ be the set of items chosen by $OPT$. Define $w(X)$ and $v(X)$ as the sum of weights and values respectively of items in the set of items $X$.

-

Lemma 1: With fractional choices allowed, greedy (choosing highest $v_i/w_i$ ratio) upper bounds the optimal for non-fractional choices, i.e. $G \ge OPT$. This is given

-

Lemma 2: $A$ chooses items with a total sum of weights $\ge \frac{W}{2} + 1$

-

Proof: We know that each $w_i \le W/2$, so $A$ is guaranteed to choose at least two items before running out of room in the knapsack. Suppose for the sake of contradiction that the set of items chosen by $A$ had a total weight $< \frac{W}{2} + 1$. Let $x$ be the weight of our knapsack right before $A$ terminates. Since we are guaranteed to be able to choose at least $2$ items, and each item has an integer weight $1 \le w_i \le \frac{W}{2}$, we have that $2 \le x \le W$. In order for our premise to be true, we need to have $x \le \frac{W}{2}$, but since each $w_i \le \frac{W}{2}$, we must have then been able to select the next item, so our algorithm wouldn't have terminated at this point, which is a contradiction.

-

Lemma 3: $G$ will choose the whole item up until the last item it adds to its knapsack

-

Proof: By the design of the algorithm this must be true. Consider an arbitrary round of $G$ in which it considers item $i$ and has a current remaining weight of $x$. We have two cases:

- -

No matter what, if we select a fractional amount of an item then it means we've filled the remaining weight of our knapsack and so the algorithm terminates immediately after.

-

Lemma 4: For any integers $a, b$, if $a \le b$, then $a \le \frac{a + b}{2}$

-

Proof: $a \le b \to 2a \le a + b \to a \le \frac{a + b}{2}$

-

From (1), we have that $\frac{1}{2} OPT \le \frac{1}{2} G$

-

From (2) we have that $w(S) \ge \frac{W}{2} + 1$, and by definition $w(S') \le W$, so we have that $w(S) \ge \frac{1}{2} w(S')$. Both $G$ and $A$ choose values in the same order, and by (3) we have that $G$ picks only full items until the very last item added. Therefore, the items chosen by $A$ are exactly the same as the items chosen by $G$ up until the very last item. Let $x$ be the remaining weight in both $A$ and $G$'s knapsack at this point, and $g_k$ be the last item added by $G$. We have the following cases:

- -

In case 1, we clearly have a 2-approximation of $OPT$, since $A$ is optimal.

-

In case 2 however, the key insight is that the value that the last item contributes to $v(S')$ is no more than half of the total value, which I will show below.

-

We have that $w(g_1, \ldots, g_{k - 1}) > \frac{W}{2}$, since otherwise we would be able to choose a non-fractional amount of $g_k$ (since $w_{g_k} \le \frac{W}{2}$). Considering the sum of values of the first $k - 1$ items chosen by $G$ (as well as $A$), we have...

-

$$ -v(g_1, \ldots, g_{k - 1}) = v_{g_1} + \ldots + v_{g_{k - 1}} = w_{g_1} \cdot \frac{v_{g_1}}{w_{g_1}} + \ldots + w_{g_{k - 1}} \cdot \frac{v_{g_{k - 1}}}{w_{g_{k - 1}}} -$$

-

Since $G$ chooses items in increasing order of $\frac{v_{g_i}}{w_{g_i}}$, we also have that $\frac{v_{g_i}}{w_{g_i}} \ge \frac{v_{g_k}}{w_{g_k}}$ for all $i < k$, so we can put a lower bound on the sum of the first $k - 1$ values chosen by $G$

-

$$ -v(g_1, \ldots, g_{k - 1}) \ge w_{g_1} \cdot \frac{v_{g_k}}{w_{g_k}} + \ldots + w_{g_{k - 1}} \cdot \frac{v_{g_k}}{w_{g_k}} = \frac{v_{g_k}}{w_{g_k}} (w_{g_1} + \ldots + w_{g_{k - 1}}) -$$

-

And since $(w_{g_1} + \ldots + w_{g_{k - 1}}) > \frac{W}{2}$, we have...

-

$$ -v(g_1, \ldots, g_{k - 1}) \ge \frac{v_{g_k}}{w_{g_k}} \frac{W}{2} -$$

-

Similarly, we can lower bound $x \cdot \frac{v_{g_k}}{w_{g_k}}$, since $x \le w_{g_k} \le \frac{W}{2}$

-

$$ -x \cdot \frac{v_{g_k}}{w_{g_k}} \le \frac{W}{2} \cdot \frac{v_{g_k}}{w_{g_k}} -$$

-

So we have...

-

$$ -x \cdot \frac{v_{g_k}}{w_{g_k}} \le v(g_1, \ldots, g_{k - 1}) -$$

-

And since $v(S') = v(g_1, \ldots, g_{k - 1}) + x \cdot \frac{v_{g_k}}{w_{g_k}}$, by (4) we have that $x \cdot \frac{v_{g_k}}{w_{g_k}} \le \frac{1}{2} v(S')$.

-

Therefore, knowing that $G$ is optimal with fractional weights, and that the only way $A$ and $G$ differ is in the last item added in case 2, we have that...

-

$$ -v(a_1, a_2, \ldots, a_{k - 1}) = v(g_1, g_2, \ldots, g_{k - 1}) \ge \frac{1}{2} v(S') = \frac{1}{2}v(S^*) -$$

-

And so $v(S) \ge \frac{1}{2} v(S^)$. Additionally, since $v(S^)$ is optimal for non-fractional weights, by definition we have $v(S) \le v(S^*)$, and $A$ is therefore a 2-approximation of $OPT$.

-

Running Time:

-

Since $w_i \le \frac{W}{2}$, and division by $W$ is $O(\log W)$, calculating greed, which has $n$ elements, takes $O(n \cdot \log W)$, and sorting it takes $O(n \log n)$.

-

Then, I iterate through $O(n)$ items in the loop, and once again since $w_i \le \frac{W}{2}$, and each iteration is only performing addition, we have $O(\log W)$ work in each iteration, for a total of $O(n \log W)$

-

Therefore, the overall running time of the algorithm is $O(n (\log n + \log W))$, which is polynomial in $n$ and $\log W$

-

P2 - Maximum Sub-Rectangle

-

Problem: You are given an $n \times n$ array $A$ where for all $1 \le i, j \le n$, $A_{ij}$ is an integer that may be negative. For a rectangle $(x_1, y_1), (x_2, y_2)$ where $x_1 \le x_2$ and $y_1 \le y_2$, the value is the sum of all numbers in this rectangle, i.e.,

-

$$ -\sum_{i = x_1}^{x_2} \sum_{j = y_1}^{y_2} A_{ij} -$$

-

Design an algorithm that runs in time $O(n^3)$ and outputs the value of the rectangle of largest value. Note that the value of the empty rectangle is zero.

-
def max_rectangle(A):
-  n = len(A)
-
-  dp = [[0] * (n + 1) for _ in range(n + 1)]
-  pf_row = [[0] * (n + 1) for _ in range(n + 1)]
-  for x in range(1, n + 1):
-    for y in range(1, n + 1):
-      pf_row[x][y] = pf_row[x][y - 1] + A[x - 1][y - 1]
-
-  def effsum(x1, y1, x2, y2):
-    return A[x1 - 1][y1 - 1] if x1 == x2 and y1 == y2 else sum(
-        pf_row[i][y2] - pf_row[i][y1 - 1]
-        for i in range(x1, x2 + 1)
-    )
-
-  def g(y1, y2):
-    if y1 > y2:
-      return 0
-
-    dp[y1][y2] = max(g(y1 + 1, y2), g(y1, y2 - 1))
-    curr = 0
-    for x in range(1, n + 1):
-      curr = max(curr + effsum(x, y1, x, y2), 0)
-      dp[y1][y2] = max(dp[y1][y2], curr)
-    return dp[y1][y2]
-
-  return max(0, g(1, n))
-
-

Correctness: Define $g(y_1, y_2)$ as the maximum sum rectangle with the upper left corner being $(x_1, y_1')$, and lower right corner being $(x_2, y_2')$ for all $x_1 \le x_2$ and $y_1' \ge y_1, y_2' \le y_2$. Additionally, define $s(x_1, y_1, x_2, y_2)$ as the sum of the rectangle with the upper left corner being $(x_1, y_1)$ and lower right corner being $(x_2, y_2)$, i.e. $\sum_{i = x_1}^{x_2} \sum_{j = y_1}^{y_2} A_{ij}$

-

Our base case is when $y_2 - y_1 < 0$, in which case we return $0$ because this is an empty/negative size rectangle.

-

Assuming we've calculated $g(y_1', y_2')$ for all $y_2 - y_1 > y_2' - y_1'$, we can calculate $g(y_1, y_2)$ by considering all rectangles that either (1) don't contain any of the row $y_2$, (2) don't contain any of the row $y_1$, or (3) contain both rows $y_1$ and $y_2$.

-

Since $y_2 - (y_1 + 1) < y_2 - y_1$ and $(y_1 - 1) - y_2 < y_2 - y_1$, we can find (1) and (2) with $g(y_1, y_2 - 1)$ and $g(y_1 + 1, y_2)$ respectively. Then, to find (3) we need to consider each possible $x_1, x_2$, choosing the values that maximum our sum for $y_1, y_2$ being fixed. This can be calculated as the largest sum of contiguous subsequences of rows between $y_1$ and $y_2$. To do this, I reduce this problem to the largest contiguous subsequence of a list of numbers $a_1, a_2, \ldots, a_n$,

-

Defining $f(i)$ to be the LCS that can be made with $a_1, \ldots, a_i$ which either includes $a_i$ or is empty, we induct over $i$. My base case is when $i = 0$, in which case $f(i) = 0$. Assuming $f(i - 1)$ is correct, we can calculate $f(i)$ by taking maximum of either choosing to include $a_i$ in our subsequence, or to reset our subsequence to zero. Therefore, we have...

-

$$ -f(i) = \max { f(i - 1) + a_i, 0 } -$$

-

Then, to find the maximum sum of a contiguous subsequence of $a_1, \ldots, a_n$, we just need to take the maximum of $f(i)$ for all $i$. Let $h(i) = \max_{0 \le j \le n} {f(j)}$. My code calculates $h(i)$ over the sum of rows between $y_1$ and $y_2$ in a bottom up fashion, but instead of storing all previous $f(j)$ for $j < i$, I only store the previous in a variable curr. Additionally, I keep track of the maximum $f(i)$ I've seen thus far in another variable dp[y1][y2], but this is equivalent. Let $h(n, y_1, y_2)$ denote the same problem, but ranging over $n$ values of the form $a_i = A_{x_i, y_1} + \ldots + A_{x_i, y_2}$, with the induction being over $n$ (the number of elements) still. Note that this is an equivalently solved problem, and the extra parameters are just to define bounds on the rows in which the instance of the problem exists.

-

Then we have the following recurrence for $g(y_1, y_2)$:

-

$$ -g(y_1, y_2) = \begin{cases} -0 & y_2 - y_1 < 0 \ -\max { g(y_1 + 1, y_2), g(y_1, y_2 - 1), h(n, y_1, y_2) } & \text{otherwise} -\end{cases} -$$

-

Since $g$ checks all possible $y_1, y_2$ and finds the best $x_1, x_2$ for each of them, we must find the largest sum rectangle, and can thus return $g(1, n)$ as the maximum sub-rectangle in $A$.

-

Running Time:

-

I start by finding the prefix sum of all rows in $O(n^2)$.

-

Then, I solve $O(n^2)$ many subproblems, one for each $y_1, y_2$. Each subproblem takes $O(n)$ time to solve, since I need to loop through $n$ many sub-sub problems to find the maximum sum contiguous subsequence of row sums. Note that to calculate row sums, I use my precomputed prefix sum, which allows me to calculate the sum of a row in $O(1)$.

-

Therefore, the overall runtime is $O(n^3)$.

-

P3 - Count connected subsets of size k

-

Given a tree $T$ with $n$ vertices and an integer $k \ge 1$ such that every vertex of $T$ has degree $deg(v) \le 3$, we want to choose a set $S$ of $k$ vertices of the tree which are connected, i.e., for any pair of vertices $u, v \in S$ the unique path between $u, v$ in $T$ is also in $S$. Design a polynomial time algorithm that outputs the number of such sets $S$.

-
def num_sets_deg_3(T, k):
-
-  def levels(v):
-    q = deque()
-    vis = set()
-    L = {}
-    q.append((v, 0))
-
-    while q:
-      size = len(q)
-      curr, l = q.popleft()
-      L[curr] = l
-      vis.add(curr)
-      for _ in range(size):
-        for nxt in T[curr]:
-          if nxt not in vis:
-            q.append((nxt, l + 1))
-    return L
-
-  root = arb_key(T)
-  L = levels(root)
-
-  def children(v):
-    return { u for u in T[v] if L[u] == L[v] + 1 }
-
-  dp = { v: { 1: 1, 0: 1 } for v in T.keys() }
-
-  def r(v, k):
-    if k in dp[v]:
-      return dp[v][k]
-
-    c = list(children(v))
-    ans = 0
-    if len(c) == 0:
-      return 1 if k == 1 else 0
-    elif len(c) == 1:
-      ans = r(c[0], k - 1)
-    elif len(c) == 2:
-      for c1 in range(k):
-        c2 = k - c1 - 1
-        ans += r(c[0], c1) * r(c[1], c2)
-    elif len(c) == 3:
-      for c1 in range(k):
-        for c2 in range(k - c1):
-          c3 = k - c1 - c2 - 1
-          ans += r(c[0], c1) * r(c[1], c2) * r(c[2], c3)
-
-
-    dp[v][k] = ans
-    return dp[v][k]
-
-  return sum(f(u, k) for k in T.keys())
-
-

Correctness:

-

Start by choosing an arbitrary vertex $r$ as root, and run $BFS(r)$, returning the level of each vertex in the BFS-tree (with L[r] = 0).

-

Define $f(v, k)$ as the number of connected subsets of size $k$ that must contain $v$, and are otherwise composed of vertices $u$ with $L[u] > L[v]$, i.e. only containing descendants of $v$ in the BFS tree. We prove correctness of $f$ by inducting on $k$.

-

Our base cases are as follows:

- -

Note that $children(v)$ is defined at ${ u \in neighbors(v) : L[u] = L[v] + 1 }$

-

And so to calculate $f(v, k)$, we just need to sum over all possibilities that involve $v$ in terms of children of $v$. Let $c = |children(v)| \le 3$ (since $deg(v) \le 3$), and $u_1, \ldots, u_c$ be $v$'s children. Any given subproblem contains vertices in any subset of $children(v)$ of size $\le \min(k - 1, c)$, so we need to iterate through all possible ways to choose such a subset of children for a given subproblem of size $k - 1$, accounting for the fact that any subset counted must contain $v$.

- -

Note that these are disjoint cases, but the case where we choose only vertices from a subset of subtrees is handled by letting $c_i = 0$ for the subtrees not chosen.

-

This gives rise to the following recurrence:

-

$$ -f(v, k) = \begin{cases} -1 & k \le 1 \ -0 & k > 1, children(v) = \emptyset \ -f(u_1, k - 1) & \text{if } |children(v)| = 1\ -\sum_{c_1 + c_2 = k - 1, c_i \ge 0} f(u_1, c_1) \cdot f(u_2, c_2) & \text{if } |children(v)| = 2\ -\sum_{c_1 + c_2 + c_3 = k - 1, c_i \ge 0} f(u_1, c_1) \cdot f(u_2, c_2) \cdot f(u_3, c_3) & \text{if } |children(v)| = 3 -\end{cases} -$$

-

And since each of these problems has at most size $k - 1$ in the recursive case, our inductive hypothesis holds and we can calculate $f(v, k)$.

-

To arrive at our final answer, we partition our possible answer space by sets of subsets that contain a given vertex $v$, none of $v$'s ancestors, and some of $v$'s descendants. Our final answer is then the sum of all $f(u, k)$ for $u \in T$. Note that each of these cases is disjoint, since each set of subsets must contain $u$, and no ancestors of $u$, so any previously calculated $f(u', k)$ for $u'$ being an ancestor of $u$ will not be included in answer for $f(u, k)$.

-

Therefore, we have a final solution of $\sum_{u \in T} f(u, k)$

-

Running Time:

-

Let $n$ be the number of vertexes in $T$. We start by calculating the levels of each node in a BFS tree rooted at an arbitrary $v$, which takes $O(n + n - 1) = O(n)$ (since $T$ is a tree and therefore has $n - 1$ edges) time.

-

For each call to $f(n, k)$, we solve $k \cdot n$ subproblems, each of which take a constant time to compute given that subproblems with a $k' < k$ have already been solved. We only have this constant time computation since the number of children is at most $3$, since otherwise we would need to solve a number of subproblems exponential in the $deg(v)$ to check every subset of children to include. Therefore, each call takes $O(n)$ time.

-

Since we call $f(v, k)$ on all vertices in the tree, and each call takes at most $O(n)$ time (but often performs better due to memoized answers from calls on ancestors of $v$), we have an upper bound of $O(n^2)$ on computing the overall answer, which is polynomial

-
-
Tags: algorithm-analysis, dynamic-programming, optimization, problem-solving
-
- - \ No newline at end of file diff --git a/site/algorithms/graphs-intro.html b/site/algorithms/graphs-intro.html deleted file mode 100644 index d3a6ed3..0000000 --- a/site/algorithms/graphs-intro.html +++ /dev/null @@ -1,274 +0,0 @@ - - - - - - Introduction to Undirected Graphs and Their Properties - - - - - -
- -

Introduction to Undirected Graphs and Their Properties

-
- Last modified: 2025-01-01 - Category: Graph Theory -
-
-

Graphs Introduction

-

Undirected Graphs

-

An undirected graph is defined by a set of vertices and a set of edges.

-

$$ -G = (V, E) -$$

-

Terminology

- -

Degree Sum

-

Claim

-

In any undirected graph, the number of edges is half the sum of all vertices degrees.

-

$$ -\text{edges } = \frac{1}{2} \sum_{v \in V} deg(v) -$$

-

Proof:

-

The sum counts each edge twice.

-

Odd Degree Vertices

-

Claim

-

In any undirected graph, the number of odd degree vertices is even.

-

Proof

-

Adding any two odd numbers results in an even number. Adding an odd and even number is odd. With this in mind, knowing that the sum of all vertex degrees is even,there must be even number of odd degree vertices, because sum of odd number of odd numbers is odd.

-

Degree 1 vertices

-

Claim

-

Suppose $G$ is an acyclic graph. Then $G$ must have a vertex of degree less than or equal to 1.

-

$$ -G = (V, E) \text{ is acyclic} \to \exists v \in V, deg(v) \le 1 -$$

-

Proof

-

Proof by contradiction.

-

Assume $\forall v \in V, d(v) \ge 2$.

-

Consider a path from $v_1$ to $v_n$. At $v_i$, we choose the next vertex such that isnt an edge to $v_{i - 1}$, which is possible because $deg(v_i) \ge 2$. The first time we see a repeated vertex $v_j = v_i$, we get a cycle. Since $G$ has finitely many edges, at some point you need to either terminate your traversal, or loop back and repeat a node.

-

Number of edges

-

Let $G = (V, E)$ be a graph with $n = |V|$ vertices and $m = |E|$ edges.

-

Claim: $m \le (n \choose 2) = \frac{n(n - 1)}{2} = O(n^2)$

-

Proof: Each vertex can be connected to at most $n - 1$ other vertices. Thus, the total number of edges is at most $n(n - 1)/2$.

-

Sparsity

-

A graph is called sparse if $|E| << |V|^2$, and dense otherwise. Sparse graphs are common in applications like social networks, the web, planar graphs, etc.

-

Technically, $O(n + m) = O(n^2)$, but in practice, $O(n + m) = O(n)$ for sparse graphs.

-

Storing Graphs

-

Adjacency Matrix

-

A matrix $A$ where $A_{ij} = 1$ if there is an edge between $v_i$ and $v_j$, and $0$ otherwise.

- -

Good for dense graphs.

-

Adjacency List

-

A list of lists, where each vertex has a list of its neighbors.

- -

Good for sparse graphs.

-

-def build_adjacency_list(n: int, edges: List[Tuple[int, int]]) -> List[List[int]]:
-    adj = [[] for _ in range(n)]
-    for u, v in edges:
-        adj[u].append(v)
-        adj[v].append(u)
-    return adj
-
-def build_adjacency_matrix(n: int, edges: List[Tuple[int, int]]) -> List[List[int]]:
-    adj = [[0] * n for _ in range(n)]
-    for u, v in edges:
-        adj[u][v] = 1
-        adj[v][u] = 1
-    return adj
-
-
-
-
Tags: data structures, graph, graph fundamentals, graph properties, graph representation
-
- - \ No newline at end of file diff --git a/site/algorithms/greedy-algorithms.html b/site/algorithms/greedy-algorithms.html deleted file mode 100644 index 45fafc3..0000000 --- a/site/algorithms/greedy-algorithms.html +++ /dev/null @@ -1,253 +0,0 @@ - - - - - - Greedy Algorithms for Interval Scheduling and Partitioning - - - - - -
- -

Greedy Algorithms for Interval Scheduling and Partitioning

-
- Last modified: 2025-01-01 - Category: Algorithm Analysis -
-
-

Greedy Algorithms

-

Choose the most attractive choice at each step, and hope that this will lead to the optimal solution. Proofs of correctness are particularly important for greedy algorithms.

-

Interval Scheduling

-

Job $j$ starts at $s(j)$ and finishes at $f(j)$. Two jobs are compatible if they don't overlap. The goal is to schedule as many jobs as possible without overlapping.

-

Start by sorting the jobs with $f(j)$, and iterate over the jobs in order and choose as many jobs as you can.

-
def interval_scheduling(jobs):
-  jobs.sort(key=lambda x: x[1])
-  last = 0
-  S = []
-  for job in jobs:
-    if job[0] >= last:
-      S.append(job)
-      last = job[1]
-  return S
-
-

Greedy Stays Ahead Proof

-

Suppose the above algorithm has chosen jobs $f(i_1) \le f(i_2) \le \ldots \le f(i_k)$, and suppose $f(j_1) \le f(j_2) \le \ldots \le f(j_m)$.

-

Goal: $m \le k$

-

Lemma: $\forall r$, $f(i_r) \le f(j_r)$

-

Proof: Induction, $P(r) := f(i_r) \le f(j_r)$

-

Base Case: $P(1)$. $i_1$ has the smallest finishing time.

-

IH: Assume $P(r - 1)$

-

IS: Goal $P(r)$

-

Applying $P(r - 1)$, and using the fact that both sets of jobs chosen are non-overlapping within themselves, we have...

-

$$ -f(i_{r - 1}) \le f(j_{r - 1}) \le s(j_r) -$$

-

So $j_r$ is a candidate for $i_r$. However, greedy chose $i_r$, which implies $f(i_r) \le $f(j_r)$. (induction over)

-

Now, we want to show that $m \le k$. Suppose for contradiction that $m > k$.

-

We know $f(i_k) \le f(j_k) \le s(j_{k + 1})$, so greedy could have executed $j_{k + 1}$ after $i_k$, which is a contradiction.

-

Exchange Argument

-

Make the optimal solution similar to greedy without changing its value.

-

This would look like removing $j_1$ from the optimal solution, adding $i_1$ instead. Then this optimal solution still has the same number of jobs, and is thus still optimal, but also shares $i_1$ in common with the greedy solution. Continue this into the general case where the first $k$ jobs are in common, and we can use the previous lemma to show that we can continue this exchange, until the greedy solution becomes the optimal.

-

Interval Partitioning

-

Given a set of intervals $I$, partition them into the minimum number of sets $S_1, S_2, \ldots, S_k$ such that each $S_i$ contains no overlapping intervals.

-
def partition_intervals(I: list[tuple[int, int]]):
-  # sort by start time
-  I.sort(key=lambda x: x[0])
-  d = 0
-  S = []
-  for itvl in I:
-    # if some partition works, add itvl to it
-    for i, S_i in enumerate(S):
-      if itvl[0] >= S_i[-1][1]:
-        S[i].append(itvl)
-        break
-    # otherwise, allocate new partition with itvl
-    else:
-      S[d] = [itvl]
-      d += 1
-  return S
-
-

Proof of Correctness

-

Observation: The algorithm never schedules two incompatible lectures in the same classroom.

-

Lemma: The algorithm is optimal.

-

Proof: using structural property

-

Let $d$ be the number of classrooms the greedy algorithm uses. Classroom $d$ is then allocated because we needed to schedule a job, $j$, that is incompatible with all $d - 1$ previously allocated classrooms.

-

Since we sorted by start time, all these incompatible jobs must have started before $s(j)$, and thus we have $d$ lectures overlapping at time $s(j) + \epsilon$, so our maximum depth is $\ge d$.

-

Since we have that the optimal solution must schedule at least depth number of classrooms, we have that the greedy algorithm is optimal.

-
-
Tags: algorithms, greedy-algorithms, interval, partitioning, scheduling
-
- - \ No newline at end of file diff --git a/site/algorithms/induction.html b/site/algorithms/induction.html deleted file mode 100644 index 80245ce..0000000 --- a/site/algorithms/induction.html +++ /dev/null @@ -1,225 +0,0 @@ - - - - - - Mathematical Induction and Pigeonhole Principle Proofs - - - - - -
- -

Mathematical Induction and Pigeonhole Principle Proofs

-
- Last modified: 2025-01-01 - Category: Mathematics -
-
-

Induction

-

Prove...

-

$$ -\forall n \in \mathbb{N}, \sum^{n}_{i = 1} i = \frac{n(n + 1)}{2} -$$

-

$$ -P(n) = \sum^{n}_{i = 1} i = \frac{n(n + 1)}{2} -$$

-

Base Case

-

$$ -P(1) = 1 = \frac{1(1 + 2)}{2} -$$

-

IH

-

Assume $P(n)$ for some $n \ge 2$

-

IS

-

$$ -1 + \ldots + n - 1 + n = \frac{(n - 1)(n)}{2} + n = \frac{n(n + 1)}{2} -$$

-

Pigeon Hole Principle

-

Suppose that we put $n + 1$ balls into $n$ bins. Prove that there is a bin with at least $2$ balls.

-

$P(n) :=$ For any possible way to put $n + 1$ balls into $n$ bins, there exists a bin with $\ge 2$ balls.

-

Base Case

-

duh...

-

IH

-

Assume $P(n - 1)$ holds for some $n \ge 2$

-

IS

-

Suppose we are given $n + 1$ balls arbitrarily placed into $n$ bins, labeled $b_1, \ldots, b_{n}$.

-

Consider $b_1$. If $b_1$ has 2 balls in it, we are done. If it has 1 ball, then throw away $b_1$ and call $P(n - 1)$ for $b_2, \ldots, b_{n}, so we are done.

-

Finally, if $b_1$ has no balls, throw away an arbitrary ball and call $P(n - 1)$ on $b_2, \ldots, b_n$

-

Generally

-

With this type of induction, start with $P(n)$, and reduce to $P(n - 1)$.

-
-
Tags: induction, pigeonhole principle, proof techniques
-
- - \ No newline at end of file diff --git a/site/algorithms/linear-programming.html b/site/algorithms/linear-programming.html deleted file mode 100644 index d0424bb..0000000 --- a/site/algorithms/linear-programming.html +++ /dev/null @@ -1,342 +0,0 @@ - - - - - - Linear Programming Fundamentals and Applications in Optimization - - - - - -
- -

Linear Programming Fundamentals and Applications in Optimization

-
- Last modified: 2025-01-01 - Category: Operations Research -
-
-

Linear Programming

-

Linear Systems

-

Systems of linear equations (key word being equality) can be solved via gaussian elimination. This is relatively easy, since your solution space ends up being a line, a plane, or a hyperplane in higher dimensions.

-

Let $a$ be a column vector in $\mathbb{R}^d$, and $x$ a column vector of $d$ variables. We can represent a system using the inner product of these two vectors:

-

$$ -\langle a, x \rangle = a^Tx = a_1x_1 + a_2x_2 + \ldots + a_dx_d = \sum_{i=1}^d a_ix_i -$$

-

A hyperplane is the set of points $x$ such that $\langle a, x \rangle = b$ for some $b$. A handspace is the set of points on one side of a hyperplace, ${x : \langle a, x \rangle \geq b}$ or ${x : \langle a, x \rangle \leq b}$.

-

The intersection of a system of half spaces creates a polytope, which is a convex set. A convex set is a set where the line segment between any two points in the set is also in the set.

-

Linear Programs

-

The goal of a linear program is to optimize some objective function subject to a set of constraints, which are also linear functions. For example...

-

$$ -\begin{align} -&max & 3x_1 - 4x_3\ -&s.t. & x_1 + x_2 \le 5\ -& & x_3 + x_1 = 4\ -& & x_3 - x_2 \ge -5\ -& & x_1, x_2, x_3 \ge 0\ -\end{align} -$$

-

Linear Algebra Review

-

$$ -\langle a, x \rangle = a^Tx = a_1x_1 + a_2x_2 + \ldots + a_dx_d -$$

-

$$ -A = \begin{bmatrix} -a_1^T \ -a_2^T \ -\vdots \ -a_m^T -\end{bmatrix} \Rightarrow -Ax = \begin{pmatrix} -\langle a_1, x \rangle \ -\langle a_2, x \rangle \ -\vdots \ -\langle a_m, x \rangle -\end{pmatrix} -$$

-

$$ -Ax \le b \Rightarrow \begin{array}{c} -\langle a_1, x \rangle \le b_1 \ -\langle a_2, x \rangle \le b_2 \ -\vdots \ -\langle a_m, x \rangle \le b_m -\end{array} -$$

-

Linear Program Standard Form

-

We can write any linear program in the standard form below.

-

$$ -\begin{array}{cc} -max & \langle c, x \rangle \ -s.t., & Ax \le b\ -~ & x \ge 0 -\end{array} -$$

-

For example, we can transform the following linear program by turning the minimization problem into a maximization, and turning all equalities into two inequalities.

-

$$ -\begin{array}{cc} -min & y_1 - 2y_2\ -s.t., & y_1 + 2y_2 = 3\ -~ & y_1 - y_2 \ge 1\ -~ & y_1, y_2 \ge 0\ -\end{array} -$$

-

To turn a min into a max, we just negate the objective function. The same holds for reversing the direction of a $\geq$. We can turn the above linear program into the following:

-

$$ -\begin{array}{cc} -max & -y_1 + 2y_2\ -s.t., & y_1 + 2y_2 \le 3\ -~ & -(y_1 + 2y_2) \le -3\ -~ & y_1 + y_2 \le 1\ -~ & y_1, y_2 \ge 0\ -\end{array} -$$

-

In LPs where you don't have non-negativity for all variables, you can replace the variables missing this constraint with the difference of two non-negative variables. For example...

-

$$ -\begin{array}{cc} -max & y_1\ -s.t., & y_1 + y_2 \le 3\ -~ & y_2 \ge 0\ -\end{array} -$$

-

We replace $y_1$ with $z_1 - z_1'$, where $z_1, z_1' \ge 0$. The linear program becomes...

-

$$ -\begin{array}{cc} -max & z_1 - z_1'\ -s.t., & z_1 - z_1' + y_2 \le 3\ -~ & z_1, z_1', y_2 \ge 0\ -\end{array} -$$

-

Applications of Linear Programming.

-

LPs generalize to all sorts of problems, such as 2-person zero-sum games, shortest path, max-flow, matching, multi-commodity flow, MST, min weighted, arborescence, ...

-

We can solve linear programs in polynomial time, and they turn out to be useful for approximation algorithms.

-

Components of a Linear Program

- -

Max-Flow

-

Given a graph $G = (V, E)$ with source $s$ and sink $t$, for every edge $e$ we have a variable $x_e$ as the flow on the edge $e$.

-

The bounding constraints would be...

- -

And our objective function would be...

- -

So we have an overall LP of...

-

$$ -\begin{array}{ccc} -max & \sum_{e \text{ out of } s} x_e & ~\ -s.t., & \sum_{e \text{ out of } v} x_e = \sum_{e \text{ into} v} x_e & \forall c \ne s, t\ -~ & x_e \le c(e) & \forall e\ -~ & x_e \ge 0 & \forall e -\end{array} -$$

-

Note that the flow this gives you is not necessarily an integer max-flow.

-

Min-Cost Max-Flow

-

Expanding on the previous example, we can add a cost to each edge $p(e)$ and try to minimize $p(e)$ while achieving some constant flow $f$.

-

$$ -\begin{array}{ccc} -max & \sum_{e \in E} p(e) \cdot x_e & ~\ -s.t., & \sum_{e \text{ out of } v} x_e = \sum_{e \text{ into} v} x_e & \forall c \ne s, t\ -~ & \sum_{e \text{ out of } s} x_e = f & ~\ -~ & x_e \le c(e) & \forall e\ -~ & x_e \ge 0 & \forall e -\end{array} -$$

-

Weighted Vertex Cover

-

Given a graph $G = (V, E)$, where each vertex has a cost $c_v$, find the minimum cost vertex cover, i.e. $\min \sum_{v \in S} c_v$

-

We have a variable $x_v$ for each $v$, where $x_v = 1$ if $v \in S$ and $x_v = 0$ otherwise. Then we have the constraint that for every edge $(u, v) \in

-

$$ -\begin{array}{ccc} -max & \sum_{e \in E} p(e) \cdot x_e & ~\ -s.t., & \sum_{e \text{ out of } v} x_e = \sum_{e \text{ into} v} x_e & \forall c \ne s, t\ -~ & \sum_{e \text{ out of } s} x_e = f & ~\ -~ & x_e \le c(e) & \forall e\ -~ & x_e \ge 0 & \forall e -\end{array} -$$

-
-
Tags: linear programs, linear systems, optimization
-
- - \ No newline at end of file diff --git a/site/algorithms/network-flows.html b/site/algorithms/network-flows.html deleted file mode 100644 index c259d02..0000000 --- a/site/algorithms/network-flows.html +++ /dev/null @@ -1,417 +0,0 @@ - - - - - - Network Flow Algorithms and Applications in Graph Theory - - - - - -
- -

Network Flow Algorithms and Applications in Graph Theory

-
- Last modified: 2025-01-01 - Category: Graph Theory -
-
-

Network Flow - Max Flow and Min Cut

-

Max Flow/Min Cut

-

Max Flow and Min Cut are two important concepts in graph theory with wide reaching applications. Many problems are reduced to either of them, such as...

- -

Typically, you use min cut when you are looking for some partition of objects into two sets such that some cost is minimized. You use max flow typically in problems involving optimal routing. Note that if there is some natural ordering associated with objects, you should also try dynamic programming. Additionally, problems on trees can sometimes be solved with greedy (using induction on the leaves), or dynamic programming on sub-trees.

-

Given a graph $G$ and vertices $s, t$, the min flow/max cut problem is to find the minimum capacity of a flow from $s$ to $t$ in the graph. This is equivalent to finding the maximum capacity of a cut that separates $s$ from $t$.

-

We have the following definitions:

- -

Ford Fulkerson Algorithm

-

The Ford Fulkerson algorithm is a method for solving the min flow/max cut problem. An augmenting path is a path from $s$ to $t$ in the residual graph $G_f$ with positive capacity. The algorithm works by finding augmenting paths and increasing the flow along them until no more augmenting paths exist. Once no more augmenting paths exist, the flow is at its maximum, and the minimum cut is found by finding the set of vertices reachable from $s$ in the residual graph.

-

A residual graph $G_f$ is a graph that represents the remaining capacity of edges in the original graph. If an edge $(u, v)$ has capacity $c$, and $f(u, v)$ is the flow along that edge, then the residual capacity is $c - f(u, v)$. We represent residual edges as an edge traversable in the opposite direction to the way capacity flows, and keep a forward edge for the remaining capacity.

-

If all edges along an augmenting path are in the forward direction, we can increase the flow by the minimum capacity of the path, updating the residual graph accordingly.

-

On the other hand, if an edge is in the reverse direction, we can decrease the flow along that edge by the minimum capacity of the path. This is equivalent to increasing the flow along the reverse edge.

-
def augment(G, f, c, P):
-  min_cap = min(c[u][v] - f[u][v] for u, v in P)
-  for u, v in P:
-    if v in G[u]:
-      f[u][v] += min_cap
-      c[u][v] -= min_cap
-      c[v][u] += min_cap
-    else:
-      f[v][u] -= min_cap
-      c[v][u] += min_cap
-      c[u][v] -= min_cap
-  return f
-
-def find_augmenting_path(G, s, t):
-  stack = [(s, [s])]
-  while stack:
-    curr, path = stack.pop()
-    if curr == t:
-      return path
-    for v in G[curr]:
-      if v in path:
-        continue
-      stack.append((v, path + [v]))
-  return None
-
-def ford_fulkerson(G, s, t, c):
-  Gf = {u: {v for v in G[u]} for u in G}
-  f = {u: {v: 0 for v in G[u]} for u in G}
-
-  while True:
-    P = find_augmenting_path(Gf, s, t)
-    if not P:
-      break
-    f = augment(G, f, c, P)
-  return f
-
-

Running Time

-

Assume that all capacities are integers between $1$ and $C$. Invariantly, every flow value $f(e)$ and every residual capacity $c_f(e)$ remains an integer throughout the algorithm.

-

Theorem: The algorithm terminates after at most $v(f^) < nC$, if $f^$ is an optimal flow.

-

This is because only one vertex ($S$) actually produces flow. Therefore, we have a tight upper bound on $v(f^*)$ of $(n - 1)C$. Each iteration of the algorithm increases the value of the flow by at least $1$ (since flows are integers).

-

Since at each iteration, we need to find an augmenting path via DFS, we have an overall runtime of $O(mnC)$, or more generally, $O(mv(f^*))$

-

Maximum Matching

-

Given an undirected graph $G = (V, E)$, find the matching $M \subseteq E$ with largest cardinality. Note that in a matching is a set of edges where each vertex touches at most one edge in $M$.

-

In general, you can solve this in polynomial time, but I won't go into detail. Instead, I cover the case where $G$ is bipartite.

-

Bipartite Maximum Matching

-

Given an undirected bipartite graph $G = (X \cup Y, E)$, find the maximum matching $M$.

-

Add vertices $s, t$, with edges $(s, v)$ of capacity $1$ for $v \in X$, and $(u, t)$ of capacity $1$ for $u \in Y$. All other edges have weight $\infty$. Orient all edges from $s \to t$, i.e. all edges between $X$ and $Y$ have direction $v \to u$.

-

Find the maximum flow from $s \to t$. The maximum flow is the maximum matching, and the minimum cut is the minimum vertex cover.

-

Proof: Let $M$ be a matching in $G$, and $f$ be the value of the maximum integer flow some $H$ in the constructed graph. We need to prove $|M| = f$.

-

$|M| \le f$: We need to find some flow that satisfies the preceding constraint. Since all flows are a candidate for max flow, and the max flow must be $\ge$ any other, showing any flow satisfies it is sufficient.

- -

$f \le |M|$: We need to use $v(f) = k$ to construct a matching $|M| = k$.

- -

Foreground/Background Segmentation

-

Label each pixel of an image as foreground or background. $V = $ set of pixels, and $E =$ neighboring pixels.

- -

Find a partition $(A, B)$ that maximizes...

-

$$ -\sum_{i \in A} a_i + \sum_{j \in B} b_j - \sum_{(i, j) \in E, i \in A, j \in B} p_{i, j} -$$

-

We proceed to a solution that uses min-cut. Since this is a maximization problem, but min-cut is a minimization problem, we can just multiply our scoring function by $-1$, so we are now trying to minimize...

-

$$ -\sum_{(i, j) \in E, i \in A, j \in B} p_{i, j} - \sum_{i \in A} a_i - \sum_{j \in B} b_j -$$

-

Additionally, working with negatives sucks, so we can get rid of them by adding a constant value to make our function positive.

-

$$ -\sum_{(i, j) \in E, i \in A, j \in B} p_{i, j} - \sum_{i \in A} a_i - \sum_{j \in B} b_j + \sum_{i \in V} a_i + \sum_{j \in V} b_j -$$

-

Since $V = A \cup B$ and $A \cap B = \emptyset$, it is equivalent to...

-

$$ -\sum_{(i, j) \in E, i \in A, j \in B} p_{i, j} + \sum_{i \in B} a_i + \sum_{j \in A} b_j -$$

-

Add vertices $s$ and $t$, with an edge weight of $a_i$ between all edges $(s, i)$, and an edge $(j, t)$ with a weight of $b_j$. Then, make all edges $(i, j)$ have a weight $p_{i, j}$. Then the min-cut of this graph would be the minimization of our objective function, giving us the optimal sets $A, B$.

-

P1 - Count Disjoint Paths

-

Given an undirected graph $G = (V, E)$ and a set of vertices $S \subseteq V$ and a disjoint set of vertices $T \subseteq V$ , i.e., $S \cap T = \emptyset$. Design a polynomial time algorithm that outputs the maximum number of vertex disjoint paths between vertices of $S$ and $T$ (note that every vertex in $S$ and every vertex in $T$ can be in at most one path).

-

Algorithm:

- -

Correctness:

-

Let $G = (V, E)$ be an undirected graph, and $S$, $T$ be disjoint subsets of $V$.

-

Since we have an algorithm from section to find the number of vertex disjoint paths from $s$ to $t$ in a directed graph containing vertices $s$, $t$, I will just show that I can construct such a directed graph $G'$ from $G$, and that the number of vertex disjoint paths in $G'$ from $s$ to $t$ is equal to the number of vertex disjoint paths in $G$ from vertices in $S$ to vertices in $T$.

-

My algorithm first adds both a forward and backward directed edge for every edge in $E$, since $G$ is undirected and we therefore need to consider paths taking either direction of each directed edge. Note that the vertices of each path considered by the algorithm from section must be disjoint (not counting $s$ and $t$), so only one of these edges will ever be used in a path counted by the algorithm in section, so it is safe to add these additional edges without ever counting additional paths.

-

Furthermore, I add two vertices $s$ and $t$ to $V'$, and add a directed edge from $s$ to every vertex in $S$, and a directed edge from every vertex in $T$ to $t$. This is to form a bijection between $A$, the maximal set of vertex disjoint paths I find in $G'$ from $s \to t$, and $B$, the maximal set of vertex disjoint paths that start at a vertex in $S$ and end at a vertex in $T$. In particular, letting $s' \in S$ and $t' \in T$ be arbitrary vertices from their respective sets, we have paths of the following forms:

- -

All paths in $A$ must have some $s'$ as their second vertex, since the only edges leaving $s$ are those going to vertices in $S$. Similarly, all paths in $A$ must have their second-to-last vertex as some $t'$, since the only edges incident to $t$ are those coming from vertices in $T$.

-

By definition of the algorithm from section, paths in $A$ are vertex disjoint not counting $s$ and $t$, and $A$ contains the maximal number of such existing paths, or in other words, there are no additional vertex disjoint paths from $s \to t$ that can be added, nor other ways to choose vertex disjoint paths to end up with a larger set $A$.

-

Therefore, by creating the set $B$ by taking each $a \in A$ and removing the starting and ending vertices $s$ and $t$, we also have vertex disjoint paths, since removing elements of disjoint sets will never make them not disjoint. Furthermore, since $A$ is of maximal size, there could not exist any other paths from $s \to t$ that would be vertex disjoint. Therefore, for $B$ to be maximal sized, we must have $|B| = |A|$. To show this more concretely, I will show that $|B| \le |A|$ and $|A| \le |B|$.

- -

Running Time: We can construct $G'$ in $O(|V| + |E|)$, since we only need to visit each edge of each vertex in our graph for duplicating/orienting edges, and an additional $O(|S| + |T|) = O(|V|)$ to add $s$, $t$, and edges to vertices in $S$ and $T$. Then, we simply need to run the algorithm from section, which takes $O(|V||E|)$, so our total running time is $O(|V||E|)$, which is polynomial.

-

P2 - Number of Disjoint Paths

-

Given a directed unweighted graph $G = (V, E)$. Suppose that there are $k$ edge disjoint paths from $s$ to $t$ and there are $k$ edge disjoint paths from $t$ to $u$ for vertices $s, t, u \in V$ . Prove that there are $k$ edge disjoint paths from $s$ to $u$.

-

Proof:

-

Let $G = (V, E)$ be a directed unweighted graph, and $s, t, u \in V$ be vertices such that there are $k$ edge disjoint paths from $s \to t$ and $k$ edge disjoint paths from $t \to u$. We wish to show that there are $k$ edge disjoint paths from $s \to u$.

-

Construct a graph $G'$ by adding vertices $w, x, y, z$ with edges $(w, s), (t, x), (y, t), (u, z)$, each having capacity $k$, and the rest of the edges having capacity $1$. We have a maximum flow of $k$ from $w \to x$, since each added edge can support a flow of up to $k$, and we can send $1$ unit of the $k$ flow going into $s$ through the $k$ edge-disjoint paths from $s$ to $t$, and then all $k$ units of flow going into $t$ through the edge $(t, x)$. Similarly, we have a max flow from $y \to z$ of $k$, since we can send $k$ units of flow from $y$ into $t$, and then $1$ unit of flow from $t$ into $u$ through the $k$ edge disjoint paths from $t$ to $u$, and then all $k$ units of flow from $u$ into $z$. Notice that the conservation of flow holds for all of these vertices, since $s$ receives and outputs $k$ units of flow, as does $t$ and $u$, and the remaining vertices each receive and output $1$ unit of flow.

-

Now, in order to show that there are at least $k$ edge disjoint paths from $s \to u$, we need to find a flow that sends $k$ units of flow from $w \to z$, and equivalently, directly from $s \to u$. Since there are at least $k$ edge disjoint paths from $s \to t$, we have at least $k$ edges leaving $s$ and entering $t$, and similarly, we have at least $k$ edges leaving $t$ and entering $u$. Otherwise, we wouldn't have enough outgoing or incoming edges to construct said $k$ edge-disjoint paths, since each path leaving $s$ or $t$ or entering $t$ or $u$ must do so by a unique edge. Therefore, instead of routing $k$ units of flow from $y$ into $t$, and from $t$ into $x$, instead route the $k$ incoming units of flow into $t$ from the $k$ capacity $1$ edges into $t$, and route the $k$ units of flow out of $t$ through the $k$ outgoing edges of capacity $1$ from $t$.

-

We therefore have a flow from $w \to z$ of capacity $k$. Since $w$ is only connected to $s$ (with an edge of capacity $k$), and $z$ is only connected to $u$ (with an edge of capacity $k$), and all edges in the middle of said flow have capacity $1$, we must route the $k$ units of flow through $k$ edge disjoint paths from $s$ to $u$, since each such path can only support $1$ unit of flow. Note that the edges in paths between $s$ and $t$, and $t$ and $u$ are not necessarily disjoint, but the edges between $s$ and $u$ that we routed the aforementioned flow through must be disjoint, since that is the only way to route $k$ units of flow through edges all with capacity $1$.

-

P3 - Minimum Vertex Cover, Maximum Independent Set (Bipartite)

-

In this exercise we give a polynomial time algorithm to find the minimum vertex cover and maximum independent set in a bipartite graph $G = (X, Y, E)$.

-

(a) Construct H

-

Let $H$ be a directed graph with vertices $X \cup Y$ with each edge $e = (u \to v) \in E$, where $u \in X$ and $v \in Y$ with $c_e = \infty$. Add vertices $s, t$, and edges $e_s = (s \to u)$ $\forall u \in X$, and $e_t = (v \to t)$, $\forall v \in Y$ with $c_{e_s} = c_{e_t} = 1$. Let said edges $e_s$ and $e_t$ form the sets $E_s$ and $E_t$ respectively.

-

(b) Construct S

-

Let $(A, B)$ be a min s-t cut in $H$. I will construct a vertex cover $S \subseteq X \cup Y$ such that $cap(A, B) = |S|$. Define the set $A_X, B_X, A_Y, B_Y$ as $A_X = A \cap X$, $B_X = B \cap X$, $A_Y = A \cap Y$, $B_Y = B \cap Y$ respectively.

-

The following lemmas will become useful:

- -

By (3), all edges of $G$ must fall into sets $E_1$ between $A_X, A_Y$, $E_2$ between $A_Y, B_X$, and $E_3$ between $B_X, B_Y$.

-

We can therefore select a vertex cover of $S = B_X \cup A_Y$, since $A_Y$ covers $E_1$ and $E_2$, and $B_X$ covers $E_2$ and $E_3$. By (4), we have $|S| = |B_X| + |A_Y| = cap(A, B)$. Note that this isn't necessarily the minimum vertex cover, but it is a vertex cover of size $cap(A, B)$.

-

(c) Construct (A, B)

-

Let $S$ be a minimal vertex cover of $G$. I will construct a min s-t cut $(A, B)$ in $H$ such that $|S| = cap(A, B)$.

-

Since $S$ is a vertex cover, it must contain at least one vertex from each edge in $E$. Let $S_X = S \cap X$, and $S_Y = S \cap Y$. Since $G$ is bipartite, we have that each edge between $X$ and $Y$ in $G$ contains some vertex in $S$.

-

To construct $(A, B)$ with $cap(A, B) = |S|$, we must choose $|S|$ vertices from $S_X$ to route $1$ unit of flow through. We can then choose $|S|$ distinct arbitrary vertices from $S_Y$ which are neighbors of the previously chosen vertices to route the $|S|$ units of flow through.

-

We know that $|S| \le |X|$ and $|S| \le |Y|$, since if we were to cover all of either $X$ or $Y$ in $S$, we would have a candidate vertex cover. This argument holds because $G$ is bipartite and therefore only has edges between $X$ and $Y$, and we have an upper bound because $|S|$ is minimal.

-

Additionally, without loss of generality (since we could just exchange $X$ for $Y$ in $G$ when constructing $H$), suppose $|S_X| \ge |S_Y|$. We can choose all of $S_X$ to route $|S_X|$ units of flow through, and then additionally for each $u \in S_Y$ we can pick a neighbor of $u$, $v' \notin S_X$ to route an additional $|S_Y|$ units of flow through to each $u$. I claim that such a $v'$ always exists, which I will prove by contradiction.

-

Suppose $\exists u \in S_Y$ such that $N(u) \subseteq S_X$. Then we would have already covered all edges $(v', u)$ for $v' \in N(u)$, and so we could remove $u$ from our set cover to make a smaller set cover $S'$. However, $S$ is a minimal set cover, so this is a contradiction.

-

Therefore, we can create a set of edges $E_{S_X} = {(u, v) \in E: u \in S_X, v \in Y \setminus S_Y}$, and $E_{S_Y} = {(u, v) \in E: u \in S_Y, v \in X \setminus S_X}$, and since these edges are both vertex disjoint and edge disjoint by construction, we have $|E_{S_Y}| + |E_{S_X}| = |S|$.

-

Letting $E_S$ be the set of edges $E_{S_X} \cup E_{S_Y}$ directed from $X \to Y$, we can create an s-t cut by cutting $H$ along all edges in $E_S$, along with edges from $s$ to vertices in $X$ which were not involved with any edge in $E_S$. Note that these two sets of edges we cut disconnect paths which form a partition of paths from $s \to t$, namely those which contain a vertex involved in some $E_S$, and those which don't. Therefore, this is a feasible s-t cut, since all paths from $s$ to $t$ disconnected.

-

Taking $(A, B)$ to be the cut described above, we have exactly $|S|$ units of flow leaving $s$, and by extension $A$, and also entering $t$, and by extension $B$. Therefore, $cap(A, B) = |S|$.

-

### (d) Design the algorithm

- -

Correctness:

-

As mentioned in lecture, adjusting the edge capacities that were originally $\infty$ to $n + 1$ works here, since we still have the desired property that no edges could exist between $A_X \to B_Y$, since this would still respect our upper bound of $|X|$ on the capacity of the min s-t cut.

-

From (c), we know that given a min s-t cut $(A, B)$, we can construct a vertex $S$ with $|S| = cap(A, B)$. Suppose for the sake of contradiction that $S$ wasn't minimal, i.e. there exists some min vertex cover $S'$ with $|S'| < |S|$. From (b), we know we can construct an s-t cut $(A', B')$ with $cap(A', B') = |S'|$. Then we have $|S'| = cap(A', B') < cap(A, B) = |S|$, and so $cap(A', B') < cap(A, B)$. However, $(A, B)$ could not have been a min s-t cut, which is a contradiction.

-

Therefore, we have that for any bipartite graph $G = (X, Y, E)$ and $H$ constructed as described in (a), we can find a min s-t cut $(A, B)$ in $H$, and construct a min vertex cover $S$ of $G$ such that $|S| = cap(A, B)$, simply by taking the elements of $B \cap X$ and $A \cap Y$. - as $S$ as described in (b) and written in my algorithm.

-

Running Time

-

For a bipartite graph $G = (X, Y, E)$ in adjacency list form with $|X| + |Y| = n$ and $|E| = m$, we can create $H$ in $O(n + m)$, since we only need to add two additional vertices, along with $n$ additional edges between them, also taking into account the extra initialization of edge capacities which would only involve $O(m)$ operations.

-

Then, since we have integer capacities, we can run ford-fulkerson on $H$ to find the min s-t cut, which will terminate in at most $nC$ iterations, where $C = n + 1$ here, each iteration taking $O(m)$ time to compute the augmenting path, for a total runtime of $O(mn^2)$.

-

Since ford-fulkerson outputs the min cut $(A, B)$, each of which are sets with size $O(n)$, and additionally $X$ and $Y$ are sets with size $O(n)$, assuming an efficient implementation of intersection using a hash set, we can compute $B_X \cap A_Y$ in $O(n)$, and additionally $S$ in $O(n)$. Even with an inefficient implementation using a linear structure, we could perform the intersections in $O(n^2)$, and the union in $O(n)$.

-

Therefore, the total runtime is $O(mn^2)$, which is polynomial.

-

P4 - Knights

-

Given an $n \times n$ chess board where some cells are removed. Design a polynomial time algorithm to find the maximum number of knights that can be placed on this board such that no two knights attack each other.

-

Algorithm:

- -

Running Time:

-

To create $G$, we first create our set of vertices $V$, which can be done in $O(n^2)$ (assuming we can check $(i, j) \in X$ in $O(1) by using an $n \times n$ grid of boolean values).

-

Next, in order to compute $E$, we once again need to iterate through all $n^2$ cells, and for each cell, we need to check at most $8$ cells that could potentially attack the current cell, only adding an edge between them if we haven't done so before. Checking the $8$ cells is $O(1)$, so it takes a total time of $O(n^2)$ to compute edges.

-

Our graph therefore has a size of $|V| \le n^2 = O(n^2)$, and $|E| \le 8n^2 = O(n^2)$.

-

We can then find the maximum independent set of $G$ by first calculating the minimum vertex cover $S$ in $O(|V|^2|E|) = O(n^6)$, and then the M.I.S as $I = V \setminus S$. However, since $S \subseteq V$, and we only need to compute the maximum number of placements, we can return $|V| - |S|$ which can be done in constant time.

-

Therefore, our total running time is $O(n^6)$, which is polynomial.

-

Proof:

-

Let $n$ be the size of the chess board, and $X$ be the set of pairs $(i, j) \in [n]^2$ that correspond to the cells that are removed. We construct a graph $G = (V, E)$, with $V = (i, j) \in [n]^2 \setminus X$, and $E$ being the set of edges between each pair of vertices that correspond to cells that are $2$ cells away horizontally and $1$ cell away vertically or $2$ cells away vertically and $1$ cell away horizontally.

-

By virtue of its construction, $G$ has a vertex for each cell we could place a knight on, since it contains each $(i, j) \in [n]^2$ which wasn't removed. Furthermore, the set of edges corresponds to each pair of cells that would attack each other if knights were placed on them, since the cells that a knight placed in $(i, j)$ would attack are exactly those that are $2$ cells away horizontally and $1$ cell away vertically or $2$ cells away vertically and $1$ cell away horizontally, but that are still within the board and not removed, i.e. pairs $(i', j') \in [n]^2 \setminus X$, or more simply $(i', j') \in V$, and that satisfy either $|i - i'| = 2, |j - j'| = 1$ or $|i - i'| = 1, |j - j'| = 2$.

-

Let $I \subseteq V$ be the maximum independent set of $G$, and $k$ be the maximum number of knights that can be placed on the board such that no two knights attack each other. I claim that $|I| = k$, which I will prove by showing that $|I| \le k$ and $|I| \ge k$.

- -
-
Tags: bipartite matching, ford-fulkerson algorithm, independent set, max flow min cut, vertex cover
-
- - \ No newline at end of file diff --git a/site/algorithms/patterns/BFS.html b/site/algorithms/patterns/BFS.html deleted file mode 100644 index d7de9d2..0000000 --- a/site/algorithms/patterns/BFS.html +++ /dev/null @@ -1,203 +0,0 @@ - - - - - - Bfs - - - - - -
- -

Bfs

-
- Last modified: 2024-04-04 - -
-
-

Breadth First Search

-

BFS is an extremely versatile algorithm that comes up a lot in graph problems. The key property of BFS is that it explores vertices in order of the length of their shortest path from the starting vertex. Any time you're dealing with an unwieghted graph and need to find the shortest path between two vertices, BFS is a good place to start. Its usefulness doesn't end there though; BFS can be used to find connected components, detect cycles, and more.

-

You can think of BFS as producing level-wise sets of vertices in a graph.

-

$$ -L_0 = {s} -$$

-

$$ -L_i = {neighbors(v_j)\ :\ v_j\ \in L_{i\ -\ 1}\setminus L_0\ \cup L_1\ \cup\ldots\cup L_{i\ -\ 2}} -$$

-
- -
- - \ No newline at end of file diff --git a/site/algorithms/patterns/sliding-window.html b/site/algorithms/patterns/sliding-window.html deleted file mode 100644 index 6ca009f..0000000 --- a/site/algorithms/patterns/sliding-window.html +++ /dev/null @@ -1,229 +0,0 @@ - - - - - - Sliding Window - - - - - -
- -

Sliding Window

-
- Last modified: 2024-03-31 - -
-
-

Sliding Window

-

Sliding window is a useful pattern when you need to maintain a contiguous subarray of elements within a linear data structure. Problems typically fall into one of two categories:

-
    -
  1. Fixed Size Window: The window size is fixed and does not change as you iterate through the data structure. These are typically easier to solve.
  2. -
  3. Variable Size Window: The window size changes as you iterate through the data structure. These are typically harder to solve, since you need to dynamically adjust the window size depending on the state of your algorithm.
  4. -
-

Fixed Size Window

-

Any time you are given a linear data structure and are asked to find some minimal or maximal contiguous subset of elements, you should immediately think of using a fixed size sliding window.

-

A typical algorithm might look like this:

- -

To design such an algorithm, you need to identify the following: -- What state do you need to describe a window of elements? - - For example: the sum of elements: integer, the frequency of elements: dict, the maximum element: monotonic stack, etc. -- How do you update the window state as add and remove elements? - - This can be as simple as adding/subtracting, and as complex as iterating through an auxiliary data structure or solving an entire subproblem based on the window state. -- How and when do you update the result based on the window state? - - For minimum/maximum problems, often your state is Comparable, so you can easily update the result by comparing the current state with the result.

-

Some problems will require you to map (as in apply a function to each element of a collection) over all windows of a fixed size, outputting some linear data structe of results based on each iteration's window state. In this case, you can use the same algorithm, but instead of updating a single result, you can append the result of each iteration to a list. In others, you can update a single result in more of a "streaming" fashion. You should always look for the streaming approach if your final result only depends on a single window state.

-

Dynamic Size Window

-

Dynamic size windows often involve more complicated logic within each iteration, since at any given step you need to decide whether to expand or contract the window, on top of how to update the window state for each case.

-

A typical algorithm might look like this:

-

Practice Problems

- -
- -
- - \ No newline at end of file diff --git a/site/algorithms/practice/4.html b/site/algorithms/practice/4.html deleted file mode 100644 index e80ca2d..0000000 --- a/site/algorithms/practice/4.html +++ /dev/null @@ -1,388 +0,0 @@ - - - - - - 4 - - - - - -
- -

4

-
- Last modified: 2024-04-24 - -
-
-

Problem Set 4

-

Problem 1

-

$I = [(0, 1), (0, 3), (4, 5), (2, 5)]$

-

These intervals are already sorted by finish time.

-
    -
  1. Initially, there are no classrooms allocated, so $I_0$ is placed in $C_0$.
  2. -
  3. Next, $I_1$ is incompatible with $I_0$, so it is placed in a newly allocated $C_1$.
  4. -
  5. $I_2$ is compatible with $C_0$ and $C_1$, so it is placed in $C_0$.
  6. -
  7. $I_3$ isn't compatible with $C_0$ or $C_1$ (because of $I_2$ and $I_1$ respectively), so it is placed in a newly allocated $C_2$
  8. -
-

So we have...

- -

As you can see below, the maximum depth is 2, so this algorithm allocated more than max-depth classrooms.

-

-I_1  +-----------------+     +-----+  I_2
-I_0  +-----+     +-----------------+  I_3
-
-     +-----+-----+-----+-----+-----+
-     0     1     2     3     4     5
-
-

Problem 2

-

Algorithm

- -
# O(nlog(n))
-def min_max_pairs(A):
-  A.sort()
-
-  l, r = 0, len(A) - 1
-  res = -infinity
-  while l < r:
-    res = max(res, A[l] + A[r])
-    l += 1
-    r -= 1
-  return res
-
-

Running Time

-

Sorting takes $O(n\log(n))$. Then, we perform $\frac{n}{2}$ iterations of our while loop, each requiring constant work. This gives us an overall runtime of $O(n\log(n))$

-

Correctness: Greedy Stays Ahead

-

For notational convenience, let $a_1, a_2, \ldots, a_n$ be sorted. Let $G$ be my algorithm, and $X$ be the optimum, and for the sake of contradiction, suppose $X$ chooses pairs differently from $G$, e.g. not of the form $(a_i, a_{n - i})$ as $G$ does.

-

Let $g(a_i)$ be a function that outputs the "choice" of partner for any $a_i$ by $G$, so $g(a_i) = a_{n - i}$, and let $x(a_i)$ mean the same for $X$.

-

Define $P(n)$ as...

-

Given $n$ sorted numbers (with $n$ being even) $a_1, \ldots a_n$, $G_{max} \le X_{max}$, where $G_{max}$ and $X_{max}$ are defined as follows:

-

$$ -G_{max} = max({ g(a_i) + a_i : 1 \le i \le n }) -$$

-

$$ -X_{max} = max({ x(a_i) + a_i : 1 \le i \le n }) -$$

-

Base Case:

-

$P(2)$: we have $a_1, a_2$, with $a_1 < a_2$. There is only one possible pair, so $G_{max} \le X_{max}$, and $P(2)$ holds.

-

IH: Suppose $P(k - 2)$ holds.

-

IS:

-

Let $A = a_1, a_2, \ldots, a_{k - 1}, a_k$ be sorted numbers. Remove $a_1$ and $a_k$ to get $A' = a_2, \ldots, a_{k - 1}$. Since $A'$ is still sorted, and has $|A'| = k - 2$, we know $P(k - 2)$ holds.

-

Therefore, we have $G'{max} \le X'$ for $A'$.

-

Consider an arbitrary pair chosen by $G$ on $A'$, $(a_i, a_j)$, with $a_i \le a_j$. By our sort order, we have $a_1 \le a_i \le a_j \le a_k$. Now, consider the ways we could swap elements between these two pairs

-

Case 1: $a_1 + a_k > a_i + a_j$

- -

Case 2: $a_1 + a_k \le a_i + a_j$

- -

In either of the above cases, we are increasing the max of the two sums by swapping elements, and this holds for an arbitrary $a_i, a_j$ chosen by $G'$, which is at least as good as the optimum. Since we know $X'{max} \ge G'$, and so $P(k)$ holds.}$, and also that any pairing other than the one picked by $G$ ($a_1, a_k$) leads to an increased sum, it must be the case that $G_{max} \le X_{max

-

Problem 3

-

Lemma (1): For any two trees $T_1$ and $T_2$ with $v \in T_1$ and $u \in T_2$, if we add the edge $(u, v)$ between the two trees, we get a new tree $T_3$.

-

Proof:

-

Let $n$ be the number of vertices in $T_1$, and $m$ be the number of vertices in $T_2$. Since they are both trees, we have $n - 1$ edges in $T_1$, and $m - 1$ edges in $T_2$. Connecting $T_1$ and $T_2$ on $(u, v)$ then results in a graph with $m + n$ vertices, and $m - 1 + n - 1 = m + n - 1$ edges.

-

Furthermore, both $T_1$ and $T_2$ are connected. Therefore, after adding an the edge $(u, v)$ between them, this results in a graph that is also connected, since all vertices in $T_1$ can reach all vertices in $T_2$ through this edge, and vice versa.

-

Finally, since the resulting graph is connected and has $m + n - 1$ edges, it must be a tree.

-

Lemma (2): For any tree $T$ removing an edge $e = (u, v)$ results in two trees.

-

Proof:

-

Let $T$ be an arbitrary tree and $e = (u, v)$ be an arbitrary edge in $T$.

-

Removing the edge $e$ from $T$ results in two separate connected components, each containing a subset of the vertices from $T$.

-

The resulting components have no cycles, since $T$ was a tree, and is therefore acyclic, and removing an edge cannot create a cycle.

-

Each component is connected, as every pair of vertices in $T$ was connected by a unique path (since $T$ is a tree), and removing $e$ does not disconnect any other pairs of vertices within each component.

-

Therefore, the two components formed by cutting $T$ on $e$ are both trees.

-

Lemma (3): For any partition of vertices of a tree $T$ into two sets $A$ and $B$, there must be an edge ($u, v$) between some vertex $u \in A$ and $v \in B$.

-

Proof:

-

Let $T$ be a tree and ${ A, B } be a partition of vertices in $T$. Suppose for the sake of contradiction that there was no edge between vertices in $A$ and $B$.

-

Let $x \in A$ and $y \in B$ be arbitrary vertices. Since $T$ is a tree, it is connected. Therefore, there must exist a path $x \to y$, so at some point along this path a vertex in $A$ must be connected to a vertex in $B$, which is a contradiction.

-

Main Proof:

-

Let $T_1$ and $T_2$ be edge disjoint spanning trees over $G$. Consider an arbitrary edge $e = (u, v) \in T_1$.

-

Let $T_1' = T_1 - e$. Since $T_1$ was a tree, this splits $T_1'$ into two connected components $C_1, C_2$, both of which are also trees by (2). We have $u \in C_1$ and $v \in C_2$.

-

Since $T_2$ is also a tree, and is therefore connected, by (3), there must exist an edge $f = (x, y) \in T_2$, such that $x \in C_1$ and $y \in C_2$. For any two vertices in a tree, there is exactly one path between them, since otherwise there would either be a non-connected vertex (no path), or a cycle (more than one path). We know $(u, v) \notin T_2$, since $T_1$ and $T_2$ are edge-disjoint, so letting $u, p_1, p_2, \ldots, v$ be the path between $u$ and $v$ in $T_2$, choose $f = (u, p_1)$ (where $p_1$ may or may not be $v$).

-

Cutting $T_2$ on $f$ to get $T_2'$, we have two connected components $K_1$, $K_2$, both of which are trees. Now, we can add $e$ to $T_2'$, and by (1), we get a tree, since $K_1$ and $K_2$ are both trees, with $u \in K_1$ and $v \in K_2$.

-

Similarly, adding $f = (x, y)$ to $T_1'$, we also get a tree by (1), since $C_1$ is a tree with $x \in C_1$, and $C_2$ is a tree with $y \in C_2$.

-

Therefore, we have both $T_1 - e + f$ and $T_2 - f + e$ are trees.

-

Problem 5

-

Call my algorithm $S(A, l, u)$.

-

Correctness

-

Claim: Given a list of $n$ numbers $A = a_1, a_n, \ldots a_n$, the number of interval sums in $[l, u]$ is correctly calculated by $S(A, l, u) = R + L + C$, where they are defined as...

-

$$L = S([a_1, \ldots, a_{\lceil n/2 \rceil - 1}], l, u)$$

-

i.e. the number of interval sums in the first half of $A$ $[l, u]$.

-

$$R = S([a_{\lceil n/2 \rceil}, \ldots, a_n], l, u)$$

-

i.e. the number of interval sums in the latter half of $A$ in $[l, u]$.

-

$$C = | { (i, j) : i \le \lceil n/2 \rceil - 1 \land j \ge \lceil n/2 \rceil \land I(i, j) \in [l, u] } |$$

-

i.e the number of interval sums that cross over the midpoint of $A$ and are in $[l, u]$

-

Proof: By induction

-

Let $P(n)$ be the above claim, i.e. that given $n$ numbers $A$ and a range $[l, u]$, my algorithm returns the number of interval sums on $A$ that are between $[l, u]$ via the following calculation:

-

$$S(A, l, u) = L + R + C$$

-

Base Case:

-

$P(1)$

-

Given a single number, there is only one interval sum to consider, i.e. $I(1, 1)$. In this case, the left half of $A$ is empty, and the right half consists of a single element. There are no interval sums that cross over the midpoint, so if $l \le I(1, 1) \le u$, then $L = 1$, and otherwise, $L = 0$. This gives us a final answer of $S(A, l, u) = L + R + C = 0 + R + 0 = R$. There is only one possible interval to consider, and $R$ is set depending on whether it is in $[l, u]$, so this is correct.

-

$P(2)$

-

Given two numbers $A = a_1, a_2$, we have $3$ intervals to check corresponding to $I(1, 1)$, $I(2, 2)$, and $I(1, 2)$. We have that $L = 1$ if $I(1, 1) \in [l, u]$, and $0$ otherwise, $R = 1$ if $I(2, 2) \in [l, u]$, and $0$ otherwise, and $C = 1$ if $I(1, 2) \in [l, u]$, and $0$ otherwise. Therefore, we check all possible interval sums, and output the sum of $L, R, C$, which corresponds to the number of these interval sums in $[l, u]$.

-

I.H.: Suppose $P(1) \land \ldots \land P(\lceil n/2 \rceil) \land \ldots \land P(n - 1)$

-

I.S.:

-

Let $A = a_1, a_2, \ldots, a_n$, and $l, u$ be arbitrary numbers such that $l < u$. Let $m = \lceil n/2 \rceil$, and define $A_1 = a_1, \ldots, a_{m - 1}$, and $A_2 = a_m, \ldots, a_n$.

-

Since $|A_1| \le n - 1$, and $|A_2| \le n - 1$, we have that $P(|A_1|)$ and $P(|A_2|)$ holds. Thus, $L = S(A_1, l, u)$ and $R = S(A_2, l, u)$, which are the total number of interval sums in $A_1$ and $A_2$ respectively.

-

Considering the set $U \subset [n]^2$ of every possible interval over $A$, we can partition $U$ into two sets, ones that don't cross over the midpoint, $M^c$, and those that do cross over the midpoint, $M$. Note that $U = M \cup M^c$, and $M \cap M^c = \emptyset$.

-

By our inductive hypothesis, we have...

-

$$L + R = |{ (i, j) : (i, j) \in M^c \land I(i, j) \in [l, u]}|$$

-

since we considered precisely the intervals in each side of $A$, which are exactly the intervals that don't cross the midpoint, and counted the number of such intervals that have $I(i, j) \in [l, u]$.

-

To find $|{ (i, j) : (i, j) \in M \land I(i, j) \in [l, u]}| = C$, we must check each suitable pair $(i, j)$ that has $i < m \le j$. We can first consider each $I(m, j)$ for $j \ge m$, and then find all such $I(i, m - 1)$ for $i \le m - 1$ such that $I(m, j) + I(i, m - 1) \in [l, u]$. Note that we don't need to consider all elements of $M$, since we can skip those that don't have $I(i, j) \in [l, u]$.

-

My algorithm does this by first calculating each $I(i, m - 1)$ and sorting them into a list $H$. Then, it iterates through all $I(m, j)$ for $m \le j \le n$, and searches for the lowest and highest index $i_{min}, i_{max}$ in $H$ such that $H_i + I(m, j) \in [l, u]$. We then calculate $C$ by maintaining a running sum of $i_{max} - i_{min} + 1$, ie the number of suitable $i$'s for a given $j$. This is guaranteed to consider each relevant $i < m \le j$, since $H$ is sorted, and so all indices between and including $[i_{min}, i_{max}]$ would also be valid prefixes to the right half of the intervals we are considering, and those outside of $[i_{min}, i_{max}]$ would correspond to an interval sum that is outside $[l, u]$.

-

Therefore, we have...

-

$$C = |{ (i, j) : (i, j) \in M \land I(i, j) \in [l, u]}|$$

-

And so $S(A, l, u) = L + R + C$ correctly considers all relevant interval sums, counting the number of interval sums between $[l, u]$.

-
def count_interval_sums_bruteforce(A, I, l, u, low, high):
-  ans = 0
-  for i in range(low, high):
-    for j in range(i, high):
-      s = I(i, j)
-      if s <= u and s >= l:
-        ans += 1
-  return ans
-
-def count_interval_sums_recursive(A, I, l, u, low, high):
-  if high - low <= 2:
-    return count_interval_sums_bruteforce(A, I, l, u, low, high)
-
-  mid = low + (high - low)//2
-  left = count_interval_sums_recursive(A, I, l, u, low, mid)
-  right = count_interval_sums_recursive(A, I, l, u, mid, high)
-
-  total_sum = sum(A[low:high])
-  left_sum = sum(A[low:mid])
-  right_sum = total_sum - left_sum
-
-  prefix = prefix_sum(A[low:high])
-  suffix = suffix_sum(A[low:high])
-
-  possible_left_halfs = sorted(
-      [total - prefix[i] - right_sum for i, s in enumerate(suffix[:mid])]
-  )
-
-  cross = 0
-
-  for i in range(len(prefix)//2, len(prefix)):
-    right_half = right_sum - suffix[i]
-
-    min_left_half = l - right_half
-    max_left_half = u - right_half
-
-    j = possible_left_halfs.binary_search(min_left_half)
-    k = possible_left_halfs.binary_search(max_left_half)
-
-    cross += k - j + 1
-
-  return left + right + cross
-
-
-def count_interval_sums(A, l, u):
-  n = len(A)
-  prefix, suffix = [0] * (n+1), [0] * (n+1)
-  pre = 0
-  total = sum(A)
-  for i in range(n):
-    prefix[i] = pre
-    pre += A[i]
-    suffix[i] = total - pre
-
-  def I(i, j):
-    return total - prefix[i] - suffix[j]
-
-  return count_interval_sums_recursive(A, I, l, u, 0, len(A))
-
-
- -
- - \ No newline at end of file diff --git a/site/algorithms/problems/graphs-and-trees.html b/site/algorithms/problems/graphs-and-trees.html deleted file mode 100644 index f4395bb..0000000 --- a/site/algorithms/problems/graphs-and-trees.html +++ /dev/null @@ -1,219 +0,0 @@ - - - - - - Graphs And Trees - - - - - -
- -

Graphs And Trees

-
- Last modified: 2024-04-08 - -
-
-

Graphs and Trees

-

Problem 1

-

Let $G$ be a tree. Use induction to prove that the number of leaves of $G$ is at least the number of vertices of degree at least $3$ in $G$.

-

Solution

-

$P(n)$: The number of leaves of any graph $G$ with $n$ vertices is at least the number of vertices of degree at least $3$ in $G$.

-

Base Case: $P(1)$

-

A tree with one vertex has no leaves or vertices of degree at least $3$, so the base case holds.

-

IH: Assume $P(k)$ holds for some $k \ge 1$.

-

IS: We want to show that $P(k + 1)$ holds.

-

Let $G = (V, E)$ be a tree with $|V| = k + 1$.

-

Choose some arbitrary leaf $l \in V$, and $(l, p) \in E$. Removing $l$ (and its corresponding edge) from $G$ results in a tree $G' = (V', E')$ with $|V'| = k$. By the IH, $G'$ satisfies $P(k)$.

-

Case 1: $p$ has degree $1$ in $G'$.

-

Then $p$ is a leaf in $G'$, and $G'$ has the same number of leaves and vertices with degree 3 as $G$ (since we didn't remove any vertices of degree 3). Thus, $G$ has at least as many leaves as vertices of degree 3.

-

Case 2: $p$ has degree $2$ in $G'$.

-

Then $p$ is not a leaf in $G'$, and $G'$ has one fewer leaf than $G$ (since we removed a leaf). However, we also removed a vertex of degree 3 from $G$, so the number of vertices of degree 3 in $G$ is still at least the number of leaves in $G$ (both increased by 1 from $G' \to G$).

-

Case 3: $p$ has degree at least $3$ in $G'$.

-

Then $p$ is a vertex of degree at least $3$ in $G$, and $G$ has the same number of vertices of degree greater than or equal to $3$ as $G'$. Since we only add a leaf to $G'$ to get $G$, the number of leaves in $G$ is at least the number of leaves in $G'$, which is at least the number of vertices of degree at least $3$ in $G'$.

-

In all cases, $G$ has at least as many leaves as vertices of degree at least $3$, so $P(k + 1)$ holds.

-

Problem 2

-

Let $G$ be a graph with $n$ vertices and at least $n$ edges. Show that $G$ has a cycle.

-

Solution

-

Let $G$ be a graph with $n$ vertices and at least $n$ edges. Suppose for the sake of contradiction that $G$ has no cycles.

-

Since $G$ has no cycles, it is a tree.

-

Claim: The sum of the degrees of all vertices in a graph is equal to twice the number of edges.

-

Proof: Each edge contributes $1$ to the degree of two vertices, so the sum of the degrees of all vertices is twice the number of edges.

-
- -
- - \ No newline at end of file diff --git a/site/algorithms/runtime.html b/site/algorithms/runtime.html deleted file mode 100644 index 93e5602..0000000 --- a/site/algorithms/runtime.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - - - Measuring Algorithm Efficiency with Asymptotic Notation - - - - - -
- -

Measuring Algorithm Efficiency with Asymptotic Notation

-
- Last modified: 2025-01-01 - Category: Computer Science -
-
-

Measuring Efficiency

-

Time is roughly proportional to the number of operations performed. Generally, this holds for simple operations. As a side note, you should avoid hashing.

-

O-Notation Definition

-

Given two functions $f(n)$ and $g(n)$, we say that $f(n)$ is $O(g(n))$ if there exist constants $c$ and $n_0$ such that $0 \leq f(n) \leq c \cdot g(n)$ for all $n \geq n_0$.

-

Omega-Notation Definition

-

Given two functions $f(n)$ and $g(n)$, we say that $f(n)$ is $\Omega(g(n))$ if there exist constants $c$ and $n_0$ such that $0 \leq c \cdot g(n) \leq f(n)$ for all $n \geq n_0$.

-

Theta-Notation Definition

-

Given two functions $f(n)$ and $g(n)$, we say that $f(n)$ is $\Theta(g(n))$ if there exist constants $c_1$, $c_2$, and $n_0$ such that $0 \leq c_1 \cdot g(n) \leq f(n) \leq c_2 \cdot g(n)$ for all $n \geq n_0$.

-

Common Bounds

-

Logarithms always grow slower than polynomial functions.

-

Polynomial

-

$$ -a_0 + a_1n + a_2n^2 + \ldots + a_kn^k \in O(n^k) -$$

-

Logarithmic

-

$$ -\log_a n \in O(\log_b n) \text{ for all } a, b > 1 -$$

-

Exponential

-

$$ -a^n \in O(b^n) \text{ for all } a, b > 1 -$$

-

Factorial

-

$$ -n! \in O(n^n) -$$

-

"Efficient" Algorithms

-

A CPU typically does less than $2^30$ operations per second. For this reason, some things just aren't computable.

-

Polynomial time algorithms are great, since if a problem size grows by at most a constant factor, then so does its run-time.

-
-
Tags: algorithm, asymptotic notation, efficiency, time complexity
-
- - \ No newline at end of file diff --git a/site/algorithms/stable-matching.html b/site/algorithms/stable-matching.html deleted file mode 100644 index 551c906..0000000 --- a/site/algorithms/stable-matching.html +++ /dev/null @@ -1,370 +0,0 @@ - - - - - - Stable Matching Algorithms and Proofs in Computer Science - - - - - -
- -

Stable Matching Algorithms and Proofs in Computer Science

-
- Last modified: 2025-01-01 - Category: Algorithm Analysis -
-
-

Stable Matching

-

Given a list of $n$ companies $c_1, c_2, \ldots, c_n$, and a list of students $s_1, s_2, \ldots, s_n$, each company ranks the students in order of preference, and each student ranks the companies in order of preference.

-

Find a stable matching between the companies and students, where no company and student would prefer each other over their current match.

- -

A stable matching is a perfect and stable matching. In other words, there should be no incentive for any company or student to break up their current match.

-

You can confirm a matching is stable by checking all non-existant matches and seeing if they are preferred. So with $n$ pairs, there are $n(n - 1)$ pairs to check to confirm stability.

-

Stable matches are guaranteed to exist.

-

Propose and Reject Algorithm (Gale-Shapley)

-
Initialize all companies and students to be free
-
-while some company is free and hanst proposed to all students:
-    c = first such company
-    s = some student c has not yet proposed to
-    if s is free:
-        (c, s) become paired
-    else if s prefers c to current match c':
-        c' becomes free
-        (c, s) become paired
-    else:
-        s rejects c
-
-return the set of pairs
-
-
def GS(C, S):
-    # C: list of companies
-    # S: list of students
-    n = len(C)
-    free = set(C)
-    matches = {c: None for c in C}
-    while free:
-        c = free.pop()
-        for s in c.prefs:
-            if matches[s] is None:
-                matches[s] = c
-                break
-            elif s.prefs.index(c) < s.prefs.index(matches[s]):
-                free.add(matches[s])
-                matches[s] = c
-                break
-    return matches
-
-

Properties

- -

Proof of Correctness

-

When designing/analyzing algorithms, need to show the following:

- -

(1) Since $n$ companies propose to at most $n$ students, the algorithm runs in $O(n^2)$ time.

-

(2) Proof by contradiction

-

note: $p \to q$ is equiv to $p \land \neg q \equiv F$

-

Output is always perfect:

-

Suppose there is a company $c_1$ with no match after the algorithm terminates. Therefore, there is also an unmatched applicant.

-

$$ -\exists \text{ unmatched company } \leftrightarrow \exists \text{ unmacted person} -$$

-

By observation, in order for a company to be unmatched, it will need to have proposed to and been rejected by every applicant. On the other hand, for an applicant to remain unmatched, it would need to have never been proposed to.

-

This means there is an applicant that was never proposed to by the unmatched company, which is a contradiction.

-

Output is a stable matching

-

For the sake of contradiction, suppose there exists an unstable pair not matched.

-

Let $S$ be the output of the GS algorithm, and $(c, a)$ to be some unstable pair. Since $S$ is perfect, there is an existing pair $(c, a') \in S$. Furthermore, since $S$ is unstable, we have that $c >{a} c'$ and $a' > a$.

-

But $c$ must have proposed to and been rejected by $a$. Additionally, $a$ must have traded $c$ for a better company. Yet, $a$ ends up with a less preferred company, which is a contradiction.

-

GS Solution Properties

-

Company Optimal Assignments

- -

Claim: If you run GS, every company receives their BVP. Furthermore, the output of GS is unique.

-

Proof

-

Proof by contradiction. Suppose that some company c is not matched with their BVP. Since companies propose in decreasing order of preference, there must exist a company $c$ that was rejected by their best valid partner $BVP(c) = a$.

-

Consider the moment when $a$ rejects $c$. Let $S^$ be the current* state of the algorithm. Therefore, $a$ is matched to some other company $c'$.

-

$$ -a \in valid(c) \to \exists \text{ stable matching } S \text{ such that } (c, a) \in S -$$

-

Say $(c', a') \in S$. If $a >{c'} a'$, then $(a, c')$ is unstable for $S$, which is a contradiction to the preceding claim. Therefore, we must have $a' > a$. This also implies that $c'$ is also rejected by $BVP(c')$, since $c'$ proposed in decreasing order of preference, so it must already be rejected by $a'$.

-

We can continue the same line of reasoning since $c'$ is also rejected by $BVP(c')$, and so on. This is a contradiction because $c$ is the first company rejected by their BVP.

-

Applicant Pessimality

-

Claim: Each applicant receives their worst valid partner (self descriptive).

-

Proof

-

Let $S^$ be the output of $GS$. For cont. suppose $(c, a) \in S^$, but $c \ne WVP(a)$.

-

Say $c' = WVP(a)$. Since $c' \in valid(a)$, $\exists \text{ stable matching } S$ such that $(c', a) \in S$. Further, suppose $(c, a') \in S$.

-

If $a >{c} a'$, then $(c, a)$ is unstable for $S$. Therefore, we must have $a' > a$, then by the above prove, $a = BVP(c)$. That is a contradiction because $a'$ is also valid.} a$. If $a' >_{c

-

Efficient Implementation

-

Can be implemented in $O(n^2)$ time.

-

Companies are named $1, ..., n$, and students are named $n + 1, ..., 2n$. Each company has a list of preferences of students, and each student has a list of preferences of companies.

-

The key idea is to also maintain an inverse array of the preference index for a matching.

-
for i in range(n):
-    for j in range(n):
-        inverse[i][pref[i][j]] = j
-
-

Stable Roommate Problem

-

Given a list of $2n$ people, each person ranks the other $2n - 1$ people in order of preference from $1$ to $2n - 1$. Find a stable matching between the people.

-

Does a stable match always include at least one person's top choice?

-

No! Consider every possible instance of a stable matching problem with $n = 3$. By brute force, you can find plenty (12) unique examples where no person is matched with their top choice.

-
from itertools import permutations, product
-
-def is_stable_matching(company_prefs, applicant_prefs, matching):
-    imatching = { v:k for k, v in matching.items() }
-    for company, applicant in matching.items():
-        company_index = company_prefs[company].index(applicant)
-        for other_applicant in company_prefs[company][:company_index]:
-            if applicant_prefs[other_applicant].index(company) < applicant_prefs[other_applicant].index(imatching[other_applicant]):
-                return False
-    return True
-
-def find_stable_matchings(company_prefs, applicant_prefs):
-    matchings = []
-    for perm in permutations(applicant_prefs.keys()):
-        matching = dict(zip(company_prefs.keys(), perm))
-        if is_stable_matching(company_prefs, applicant_prefs, matching):
-            matchings.append(matching)
-    return matchings
-
-
-A1, A2, A3 = 'A1', 'A2', 'A3'
-C1, C2, C3 = 'C1', 'C2', 'C3'
-
-def generate_preferences():
-
-    company_labels = [C1, C2, C3]
-    applicant_labels = [A1, A2, A3]
-
-    cperms = list(permutations(company_labels))
-    aperms = list(permutations(applicant_labels))
-
-    cprod = product(cperms, cperms, cperms)
-    aprod = product(aperms, aperms, aperms)
-
-    c = [ dict(zip(applicant_labels, c)) for c in cprod ]
-    a = [ dict(zip(company_labels, a)) for a in aprod ]
-
-    return c, a
-
-all_c, all_a = generate_preferences()
-
-data = []
-res = []
-for c in all_c:
-  for a in all_a:
-
-    stable_matchings = find_stable_matchings(c, a)
-
-    for matching in stable_matchings:
-      imatching = { v: k for k, v in matching.items() }
-      match_dict = dict(matching)
-      match_dict.update(imatching)
-
-      data.append((c, a, matching))
-      curr = 0
-      for co, pref in c.items():
-        if pref[0] == match_dict[co]:
-          curr += 1
-
-      for ap, pref in a.items():
-        if pref[0] == match_dict[ap]:
-          curr += 1
-
-      res.append(curr)
-
-
-candidates = []
-
-for i in range(len(res)):
-  if res[i] == 0:
-    candidates.append(data[i])
-    print(data[i])
-
-
-
Tags: complexity analysis, gale-shapley, matching, optimization, proof techniques, stable matching
-
- - \ No newline at end of file diff --git a/site/algorithms/tree-intro.html b/site/algorithms/tree-intro.html deleted file mode 100644 index b31b9bf..0000000 --- a/site/algorithms/tree-intro.html +++ /dev/null @@ -1,210 +0,0 @@ - - - - - - Tree Properties and Proof of Edge Count - - - - - -
- -

Tree Properties and Proof of Edge Count

-
- Last modified: 2025-01-01 - Category: Graph Theory -
-
-

Trees

-

Lemma: acyclic and connected

-

A graph is a tree if and only if it is acyclic and connected.

-

Claim: Every tree with $n$ vertices has $n - 1$ edges.

-

Proof by induction.

-

Base case: $n = 1$. A tree with 1 vertex has 0 edges.

-

IH: Suppose every tree with $n - 1$ vertices has $n - 2$ edges ($P(n - 1)$).

-

IS: Let $T$ be a tree with $n$ vertices.

-

$T$ must have a vertex of degree 1 (since it is by definition acyclic). Remove this vertex and its edge to get a tree $T'$ with $n - 1$ vertices. By IH, $T'$ has $n - 2$ edges. Adding back the vertex and edge, we get $n - 1$ edges.

-

Properties of Trees

-

Any graph $G$ that satisfies two of the following properties must satisfy the third (and thus be a tree):

- -
-
Tags: acyclic graphs, connected graphs, graph properties, induction proofs, trees
-
- - \ No newline at end of file diff --git a/site/categories/algorithm analysis.html b/site/categories/algorithm analysis.html deleted file mode 100644 index b46cde8..0000000 --- a/site/categories/algorithm analysis.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Category: Algorithm Analysis - - - - - -
- -

Category: Algorithm Analysis

-
- Last modified: 2025-01-16 - -
-
-

Category: Algorithm Analysis

- -
- -
- - \ No newline at end of file diff --git a/site/categories/algorithms.html b/site/categories/algorithms.html deleted file mode 100644 index 0fd6eb0..0000000 --- a/site/categories/algorithms.html +++ /dev/null @@ -1,203 +0,0 @@ - - - - - - Category: algorithms - - - - - -
- -

Category: algorithms

-
- Last modified: 2025-01-16 - -
-
-

Category: algorithms

- -
- -
- - \ No newline at end of file diff --git a/site/categories/computer science.html b/site/categories/computer science.html deleted file mode 100644 index 8d9a9af..0000000 --- a/site/categories/computer science.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Computer Science - - - - - -
- -

Category: Computer Science

-
- Last modified: 2025-01-16 - -
-
-

Category: Computer Science

- -
- -
- - \ No newline at end of file diff --git a/site/categories/database design.html b/site/categories/database design.html deleted file mode 100644 index 8a3f8bf..0000000 --- a/site/categories/database design.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Database Design - - - - - -
- -

Category: Database Design

-
- Last modified: 2025-01-16 - -
-
-

Category: Database Design

- -
- -
- - \ No newline at end of file diff --git a/site/categories/database systems.html b/site/categories/database systems.html deleted file mode 100644 index 82fa0df..0000000 --- a/site/categories/database systems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Database Systems - - - - - -
- -

Category: Database Systems

-
- Last modified: 2025-01-16 - -
-
-

Category: Database Systems

- -
- -
- - \ No newline at end of file diff --git a/site/categories/distributed systems.html b/site/categories/distributed systems.html deleted file mode 100644 index e30bd69..0000000 --- a/site/categories/distributed systems.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Category: Distributed Systems - - - - - -
- -

Category: Distributed Systems

-
- Last modified: 2025-01-16 - -
-
-

Category: Distributed Systems

- -
- -
- - \ No newline at end of file diff --git a/site/categories/graph theory.html b/site/categories/graph theory.html deleted file mode 100644 index eafd643..0000000 --- a/site/categories/graph theory.html +++ /dev/null @@ -1,200 +0,0 @@ - - - - - - Category: Graph Theory - - - - - -
- -

Category: Graph Theory

-
- Last modified: 2025-01-16 - -
-
-

Category: Graph Theory

- -
- -
- - \ No newline at end of file diff --git a/site/categories/index.html b/site/categories/index.html deleted file mode 100644 index 4d591aa..0000000 --- a/site/categories/index.html +++ /dev/null @@ -1,211 +0,0 @@ - - - - - - Categories - - - - - -
- -

Categories

-
- Last modified: 2025-01-16 - -
-
- -
- -
- - \ No newline at end of file diff --git a/site/categories/mathematics.html b/site/categories/mathematics.html deleted file mode 100644 index 417905f..0000000 --- a/site/categories/mathematics.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Mathematics - - - - - -
- -

Category: Mathematics

-
- Last modified: 2025-01-16 - -
-
-

Category: Mathematics

- -
- -
- - \ No newline at end of file diff --git a/site/categories/natural language processing.html b/site/categories/natural language processing.html deleted file mode 100644 index 5cb6eef..0000000 --- a/site/categories/natural language processing.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Category: natural language processing - - - - - -
- -

Category: natural language processing

-
- Last modified: 2025-01-16 - -
-
-

Category: natural language processing

- -
- -
- - \ No newline at end of file diff --git a/site/categories/networking.html b/site/categories/networking.html deleted file mode 100644 index 3451a13..0000000 --- a/site/categories/networking.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: networking - - - - - -
- -

Category: networking

-
- Last modified: 2025-01-16 - -
-
-

Category: networking

- -
- -
- - \ No newline at end of file diff --git a/site/categories/operating systems.html b/site/categories/operating systems.html deleted file mode 100644 index 771f0c3..0000000 --- a/site/categories/operating systems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Operating Systems - - - - - -
- -

Category: Operating Systems

-
- Last modified: 2025-01-16 - -
-
-

Category: Operating Systems

- -
- -
- - \ No newline at end of file diff --git a/site/categories/operations research.html b/site/categories/operations research.html deleted file mode 100644 index 26512c7..0000000 --- a/site/categories/operations research.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Operations Research - - - - - -
- -

Category: Operations Research

-
- Last modified: 2025-01-16 - -
-
-

Category: Operations Research

- -
- -
- - \ No newline at end of file diff --git a/site/categories/research.html b/site/categories/research.html deleted file mode 100644 index 677760c..0000000 --- a/site/categories/research.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Category: research - - - - - -
- -

Category: research

-
- Last modified: 2025-01-16 - -
-
-

Category: research

- -
- -
- - \ No newline at end of file diff --git a/site/categories/software engineering.html b/site/categories/software engineering.html deleted file mode 100644 index 8e70154..0000000 --- a/site/categories/software engineering.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: Software Engineering - - - - - -
- -

Category: Software Engineering

-
- Last modified: 2025-01-16 - -
-
-

Category: Software Engineering

- -
- -
- - \ No newline at end of file diff --git a/site/categories/system design.html b/site/categories/system design.html deleted file mode 100644 index 14528a7..0000000 --- a/site/categories/system design.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: System Design - - - - - -
- -

Category: System Design

-
- Last modified: 2025-01-16 - -
-
-

Category: System Design

- -
- -
- - \ No newline at end of file diff --git a/site/categories/system-design.html b/site/categories/system-design.html deleted file mode 100644 index c3d3e5b..0000000 --- a/site/categories/system-design.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: system-design - - - - - -
- -

Category: system-design

-
- Last modified: 2025-01-16 - -
-
-

Category: system-design

- -
- -
- - \ No newline at end of file diff --git a/site/categories/systems.html b/site/categories/systems.html deleted file mode 100644 index 8a1bfc1..0000000 --- a/site/categories/systems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Category: systems - - - - - -
- -

Category: systems

-
- Last modified: 2025-01-16 - -
-
-

Category: systems

- -
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/algorithms/divide-and-conquer.html b/site/cheatsheets/algorithms/divide-and-conquer.html deleted file mode 100644 index 42ddf02..0000000 --- a/site/cheatsheets/algorithms/divide-and-conquer.html +++ /dev/null @@ -1,229 +0,0 @@ - - - - - - Divide And Conquer - - - - - -
- -

Divide And Conquer

-
- Last modified: 2024-04-28 - -
-
-

Divide and Conquer

-

Master Theorem

-

Given any recurrence of the form $T(n) = a T(\frac{n}{b}) + c n^k$ for all $n > b$, we have:

- -

Root Finding

-

Given a continuous function $f$ and two points $a < b$ such that $f(a) \cdot f(b) < 0$, there exists a root of $f$ in the interval $[a, b]$ by the intermediate value theorem. Since said root may be irrational, we aim to approximate it with an arbitrary precision $\epsilon$.

- -

kth Smallest Element

- -
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/algorithms/final.tex b/site/cheatsheets/algorithms/final.tex deleted file mode 100644 index 28f8cd2..0000000 --- a/site/cheatsheets/algorithms/final.tex +++ /dev/null @@ -1,308 +0,0 @@ -\documentclass[10pt,landscape,a4paper]{article} -\usepackage{multicol} -\usepackage{calc} -\usepackage{ifthen} -\usepackage[landscape]{geometry} -\usepackage{graphicx} -\usepackage{amsmath, amssymb, amsthm} -\usepackage{latexsym, marvosym} -\usepackage{pifont} -\usepackage{lscape} -\usepackage{graphicx} -\usepackage{array} -\usepackage{booktabs} -\usepackage[bottom]{footmisc} -\usepackage{tikz} -\usetikzlibrary{shapes} -\usepackage{pdfpages} -\usepackage{wrapfig} -\usepackage{enumitem} -\setlist[description]{leftmargin=0pt} -\usepackage{xfrac} -\usepackage{bbm} -\usepackage[pdftex, - pdfauthor={William Chen}, - pdftitle={Probability Cheatsheet}, - pdfsubject={A cheatsheet pdf and reference guide originally made for Stat 110, Harvard's Introduction to Probability course. Formulas and equations for your statistics class.}, - pdfkeywords={probability} {statistics} {cheatsheet} {pdf} {cheat} {sheet} {formulas} {equations} - ]{hyperref} -\usepackage[ - open, - openlevel=2 - ]{bookmark} -\usepackage{relsize} -\usepackage{rotating} - - \newcommand\independent{\protect\mathpalette{\protect\independenT}{\perp}} - \def\independenT#1#2{\mathrel{\setbox0\hbox{$#1#2$}% - \copy0\kern-\wd0\mkern4mu\box0}} - -\newcommand{\noin}{\noindent} -\newcommand{\logit}{\textrm{logit}} -\newcommand{\var}{\textrm{Var}} -\newcommand{\cov}{\textrm{Cov}} -\newcommand{\corr}{\textrm{Corr}} -\newcommand{\N}{\mathcal{N}} -\newcommand{\Bern}{\textrm{Bern}} -\newcommand{\Bin}{\textrm{Bin}} -\newcommand{\Beta}{\textrm{Beta}} -\newcommand{\Gam}{\textrm{Gamma}} -\newcommand{\Expo}{\textrm{Expo}} -\newcommand{\Pois}{\textrm{Pois}} -\newcommand{\Unif}{\textrm{Unif}} -\newcommand{\Geom}{\textrm{Geom}} -\newcommand{\NBin}{\textrm{NBin}} -\newcommand{\Hypergeometric}{\textrm{HGeom}} -\newcommand{\HGeom}{\textrm{HGeom}} -\newcommand{\Mult}{\textrm{Mult}} -\newcommand{\R}{\mathbb{R}} - - -\geometry{top=.2in,left=.2in,right=.2in,bottom=.2in} - -\pagestyle{empty} -\makeatletter -\renewcommand{\section}{\@startsection{section}{1}{0mm}% - {-1ex plus -.5ex minus -.2ex}% - {0.5ex plus .2ex}%x - {\normalfont\large\bfseries}} -\renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% - {-1explus -.5ex minus -.2ex}% - {0.5ex plus .2ex}% - {\normalfont\normalsize\bfseries}} -\renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% - {-1ex plus -.5ex minus -.2ex}% - {1ex plus .2ex}% - {\normalfont\small\bfseries}} -\makeatother - -\setcounter{secnumdepth}{0} -\graphicspath{ {./utils/images/} } -\setlength{\parindent}{0pt} -\setlength{\parskip}{0pt plus 0.5ex} - -% ----------------------------------------------------------------------- - -\usepackage{titlesec} - -\titleformat{\section} -{\color{blue}\normalfont\large\bfseries} -{\color{red}\thesection}{1em}{} -\titleformat{\subsection} -{\color{violet}\normalfont\normalsize\bfseries} -{\color{cyan}\thesection}{1em}{} -% Comment out the above 5 lines for black and white - -\begin{document} - -\raggedright -\footnotesize -\begin{multicols*}{3} - -% multicol parameters -% These lengths are set only within the two main columns -%\setlength{\columnseprule}{0.25pt} -\setlength{\premulticols}{1pt} -\setlength{\postmulticols}{1pt} -\setlength{\multicolsep}{1pt} -\setlength{\columnsep}{2pt} - -% Redefine bullet symbols for each level of itemize -\renewcommand{\labelitemi}{$\bullet$} % Level 1 -\renewcommand{\labelitemii}{$\bullet$} % Level 2 -\renewcommand{\labelitemiii}{$\bullet$} % Level 3 -\renewcommand{\labelitemiv}{$\bullet$} % Level 4 - -\section{Graphs} - -\subsection{Undirected Graphs} -\begin{itemize} - \item $|E| = \frac{1}{2} \sum_{v \in V} deg(v)$ - \item The number of odd degree vertices is even - \item If a graph is acyclic, there is a vertex of degree $\le 1$ -\end{itemize} - -\subsection{Trees} -For any graph $G = (V, E)$, if two of the following are true, then all three are and $G$ is a tree -\begin{itemize} - \item $|E| = |V| - 1$ - \item $G$ is connected - \item $G$ is acyclic -\end{itemize} - -\subsection{Breadth-First Search} -\begin{itemize} - \item $BFS(s)$ visits $v$ iff there is a path from $s \to v$ - \item Edges into then-undiscovered vertices define the BFS-tree of $G$ - \item Level $i$ of the BFS-tree contains all vertices with shortest path $i$ from the root - \item \textbf{All non-tree edges} join vertices on the same, or adjacent levels of the tree - \item If $G$ contains edges between vertices in the same layer, it is not bipartite, nor a tree -\end{itemize} - -\subsection{Bipartite Graphs} -\begin{itemize} - \item $G$ is bipartite iff you can partition $V$ into $V_1$ and $V_2$ such that all edges are between $V_1$ and $V_2$, i.e. no edges between vertices in different sets - \item $G$ is bipartite iff $G$ has no odd cycles -\end{itemize} - - -\subsection{Depth First Search} -\begin{itemize} - \item $DFS(s)$ visits $x$ iff there is a path from $s \to x$ (so you can find C.C.s) - \item DFS Spanning Tree formed by edges into then-undiscovered vertices - \begin{itemize} - \item DFS tree not minimum depth, nor do its levels reflect the min distance - \item Non-tree edges never join vertices on the same or adjacent level. Always join a vertex with one of its ancestors or descendants - \item All vertices visited during $DFS(s)$ are a descendant of $s$ in the DFS tree - \begin{itemize} - \item For every edge $(u, v) \notin T_{DFS(s)}$, either $x$ is an ancestor of $y$ or $y$ is an ancestor of $x$ - \end{itemize} - \end{itemize} -\end{itemize} - -\subsection{Directed Acyclic Graphs} -\begin{itemize} - \item \textbf{Source}: vertex with no incoming edges - \item \textbf{Sink}: vertex with no outgoing edges - \item Every DAG has a source and a sink - \item Every DAG has a topological order, and every graph with a topological order is a DAG -\end{itemize} - -\subsection{Topological Order} -An ordering of nodes $v_1, v_2, \ldots, v_n$ so that for every edge $(v_i, v_j)$, $i < j$. -\begin{itemize} - \item To find, initialize map of in-degrees for each vertex, and a queue of vertices with in-degree 0. - \item Then, while the queue isn't empty, remove a vertex, adding it to the ordering, and decrement its neighbors in-degree. - \item If any of them become 0, add them to the queue. -\end{itemize} - - -\subsection{Cuts} -\begin{itemize} - \item A cut of $G = (V, E)$ is a bipartition of $V$ into $S, V - S$ for some $S \subseteq V$ - \item $e = (u, v) \in (S, V - S)$ if exactly one of $u, v \in S$ - \item If $G$ is connected, then there is at least one edge in every cut - \item Every cycle crosses a cut an even number of times - \item \textit{Cut property}: Let $(S, V - S)$ be any cut, and let $e$ be the \textbf{min} cost edge with exactly \textbf{one} endpoint in $S$. Then \textbf{every} MST contains $e$. - \item \textit{Cycle property}: Let $C$ be any cycle, and $f$ be the \textbf{max} cost edge belonging to $C$. Then \textbf{no} MST contains $f$. -\end{itemize} - -\subsection{Minimum Spanning Tree} -\begin{itemize} - \item \textbf{Algorithm}: $O(m\log(n))$ - \begin{itemize} - \item Sort edges by increasing weight, initialize an empty tree $T$, and add each vertex to its own set. - \item Then, for each edge $e = (u, v)$, if $u$ and $v$ are currently in different sets, add $e$ to $T$ and merge the sets containing $u$ and $v$. - \end{itemize} - \item \textbf{Proof}: - \begin{itemize} - \item Show that it is a tree - \begin{itemize} - \item Initially start with $|V| = n$ sets, and only add an edge if you are connecting two of them. Therefore, we end with $n - 1$ sets to add an edge between each original set - \item Only add edges between disconnected components, so it must be acyclic, since each additional edge $e$ connecting $C_1$ and $C_2$ (two disconnected components) is the only edge between them. This means $C_1 + e + C_2$ has an odd number of edges in its cut, so there are no cycles formed. - \end{itemize} - \item Must be an MST - \begin{itemize} - \item Considered edges in increasing order of cost. Taking the first edge where the optimal and Kruskal's differ, we can exchange them for an equal or better solution. - \end{itemize} - \end{itemize} -\end{itemize} - -\subsection{Disjoint Sets} -\begin{itemize} - \item \textbf{Implementation}: - \begin{itemize} - \item Maintain a tree of pointers, where every vertex is labeled with the longest path ending at that vertex. To check set membership of $u$ and $v$, traverse to root and check if $root(u) = root(v)$. This is $O(\log(n))$ - \item To merge two sets, point root with smaller label to root with larger label, adjusting labels of the new root if necessary. This is $O(1)$. - \end{itemize} - \item \textbf{Properties}: - \begin{itemize} - \item If the label of a root is $k$, there are at least $2^k$ elements in the set. - \end{itemize} -\end{itemize} - -\section{Intervals} - -\subsection{Scheduling the max number of intervals} -\begin{itemize} - \item \textbf{Algorithm}: sort by finish time and select the next compatible interval - \item \textbf{Proof}: Greedy stays ahead, by induction - \begin{itemize} - \item \textit{Claim}: Greedy algorithm is optimal - \item \textbf{Lemma} - \begin{itemize} - \item $P(r)$: For greedy choices $g_1, \ldots, g_n$ and optimal choices $k_1, \ldots, k_m$, $f(g_r) \le f(k_r)$ - \item $P(1)$: $g_1$ is chosen to have the minimum finish time, so $f(g_1) \le f(k_1)$ - \item Suppose $P(r)$. Since $f(g_r) \le f(k_r) \le s(k_{r + 1})$, $k_{r + 1}$ is among the candidates considered for $g_{r + 1}$. Of those candidates, it picks the minimum finish time, so $f(g_{r + 1}) \le f(k_{r + 1})$. - \end{itemize} - \item By this lemma, we must have $n \ge m$, since since otherwise $k_{n + 1}$ is in the set of candidates for $g_{n + 1}$. - \end{itemize} -\end{itemize} - - -\subsection{Partitioning intervals into the minimum number of sets} -\begin{itemize} - \item \textbf{Algorithm}: sort intervals by start time, adding them to \textbf{any} compatible set. If no set is compatible, create a new one - \item \textbf{Proof}: exploit structural property - \begin{itemize} - \item \textit{Claim}: greedy algorithm is optimal - \item Let $d$ be the number of sets the greedy algorithm allocates. The $d$th set, $S_d$ is allocated because we had to assign some interval, $I_i$, that was not compatible with any of the $d - 1$ previous sets. - \item Since we sorted by start time, all intervals $I_j \in S_1 \cup \ldots \cup S_{d - 1}$ have $s(I_i) \ge s(I_j)$. Thus, we have at least depth $d$ intervals, and so all valid partitions must have $\ge d$ sets. - \end{itemize} -\end{itemize} - -\section{Divide and Conquer} - -\subsection{Master Theorem} -\begin{itemize} - \item Given any recurrence of the form $T(n) = a T(\frac{n}{b}) + c n^k$ for all $n > b$, we have: - \begin{itemize} - \item If $a > b^k$, then $T(n) = \Theta(n^{\log_b a})$ - \item If $a < b^k$, then $T(n) = \Theta(n^k)$ - \item If $a = b^k$, then $T(n) = \Theta(n^k \log n)$ - \end{itemize} -\end{itemize} - -\subsection{Root Finding} - -Given a continuous function $f$ and two points $a < b$ such that $f(a) \cdot f(b) < 0$, there exists a root of $f$ in the interval $[a, b]$ by the \textit{intermediate value theorem}. Since said root may be irrational, we aim to approximate it with an arbitrary precision $\epsilon$. - -\begin{itemize} - \item \textbf{Algorithm}: $Bisect(a, b, \epsilon)$ - \begin{itemize} - \item If $b - a < \epsilon$, $a$ is a suitable approximation - \item Otherwise, calculate the midpoint $m = (a + b)/2$ - \item If $f(m) \le 0$ then return $Bisect(m, b, \epsilon)$ - \item else return $Bisect(a, m, \epsilon)$ - \end{itemize} - \item \textbf{Time}: $T(n) = T(\frac{n}{2}) + O(1) = O(\log(\frac{b - a}{\epsilon}))$ - \item \textbf{Proof}: - \begin{itemize} - \item $P(k) =$ For any $a, b$ such that $k\epsilon \le |a - b| \le (k + 1)\epsilon$, if $f(a)f(b) \le 0$, then we find an $\epsilon$ approx to a root using $\log k$ queries to $f$. - \item $P(1)$: Output $a + \epsilon$, since the whole interval is at most $\epsilon$. This requires $0$ calls to $f$. - \item Suppose $P(k)$ and consider an arbitrary $a$, $b$ s.t. $2k\epsilon \le |a - b| \le (2k + 1)\epsilon$. - \item If $f(a + k\epsilon) = 0$, output $a + k\epsilon$. - \item If $f(a)f(a + k\epsilon) < 0$, solve on the interval $[a, a + k\epsilon]$. By I.H. this takes at most $\log(k)$ queries of $f$. - \item Otherwise, we have $f(b)f(a + k\epsilon) < 0$, since $f(a)f(b) < 0$ and $f(a)f(a + k\epsilon) \ge 0$. Solve the interval $[a + k\epsilon, b]$. - \item In any case, we used at most $\log(k) + 1 = \log(2k)$ queries to $f$. - \end{itemize} -\end{itemize} - -\subsection{kth Smallest Element} - -\begin{itemize} - \item \textbf{Algorithm}: $f(S \in \mathbb{R}^n, k \in \mathbb{R})$ - \begin{itemize} - \item Select an approximate median element $w$ using median of $\frac{n}{5}$ medians with subarrays of size $5$ - \item Partition each element into three sets, $S_{>}, S_{<}, S_{=}$ - \item If $k \le |S_{<}|$, recurse on $f(S_{<}, k)$ - \item Else, if $k \le |S_{<}| + |S_{=}|$, return $w$ - \item Else, recurse on $f(S_{>}, k - |S_{<}| - |S_{=}|)$ - \end{itemize} -\end{itemize} - - - -\end{multicols*} -\end{document} -endsnippet diff --git a/site/cheatsheets/algorithms/graphs.html b/site/cheatsheets/algorithms/graphs.html deleted file mode 100644 index e1c2508..0000000 --- a/site/cheatsheets/algorithms/graphs.html +++ /dev/null @@ -1,279 +0,0 @@ - - - - - - Graphs - - - - - -
- -

Graphs

-
- Last modified: 2024-04-28 - -
-
-

Graphs

-

Undirected Graphs

- -

Trees

-

For any graph $G = (V, E)$, if two of the following are true, then all three are and $G$ is a tree

- - - -

Bipartite Graphs

- - - -

Directed Acyclic Graphs

- -

Topological Order

-

An ordering of nodes $v_1, v_2, \ldots, v_n$ so that for every edge $(v_i, v_j)$, $i < j$.

- -

Cuts

- -

Minimum Spanning Tree

- -

Disjoint Sets

- -
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/algorithms/intervals.html b/site/cheatsheets/algorithms/intervals.html deleted file mode 100644 index d5c93b2..0000000 --- a/site/cheatsheets/algorithms/intervals.html +++ /dev/null @@ -1,216 +0,0 @@ - - - - - - Intervals - - - - - -
- -

Intervals

-
- Last modified: 2024-04-28 - -
-
-

Interval Scheduling/Partitioning

-

Scheduling the max number of intervals

- -

Partitioning intervals into the minimum number of sets

- -
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/circuits/components.html b/site/cheatsheets/circuits/components.html deleted file mode 100644 index a9ef605..0000000 --- a/site/cheatsheets/circuits/components.html +++ /dev/null @@ -1,350 +0,0 @@ - - - - - - Components - - - - - -
- -

Components

-
- Last modified: 2024-03-12 - -
-
-

Component Dictionary

-

DC Sources

-

Direct current sources are sources that provide a constant voltage or current. Batteries are a common example of a DC voltage source. If you graph the voltage over time, it's a flat line.

-

Voltage sources supply the required current to maintain a constant voltage, and current sources supply the required voltage to maintain a constant current.

-

AC Sources

-

Alternating current sources provide an oscillating voltage or current over time. The voltage or current will oscillate between the two peek values, the distance between them being referred to as the peak-to-peak voltage ($V_{pp}) or current ($I_{pp}$). The frequency of oscillation ($f$) is the number of oscillations per second, and is measured in hertz (Hz), but can also be represented in radians per second ($\omega$). The average value, or mid point between the two peek values, is referred to as the DC offset ($V_{dc}). Finally, the phase shift ($\phi$) is the amount of time shift between the voltage and current oscillations.

-

$$ -V(t) = \frac{V_{pp}}{2} \sin(\omega t + \phi) + V_{dc} -$$

-

$$ -I(t) = \frac{I_{pp}}{2} \sin(\omega t + \phi) + I_{dc} -$$

-

$$ -\omega (rad/s) = 2\pi f (Hz) -$$

-

Resistors

-

Resistors are simple components that drop the voltage of a circuit branch as current flows through them. Their resistance measured in ohms ($\Omega$), and can be thought of as a hill in the flow of electricity. The voltage drop across a resistor is the product of the current flowing through it and its resistance AKA Ohm's Law.

-

Resistors in Series

-

$$ -R_{eq} = \sum_{i=1}^{n} R_i = R_1 + R_2 + \ldots + R_n -$$

-

Resistors in Parallel

-

$$ -R_{eq} = (\sum_{i=1}^{n} \frac{1}{R_i})^{-1} = (\frac{1}{R_1} + \frac{1}{R_2} + \ldots + \frac{1}{R_n})^{-1} -$$

-

Capacitors

-

Capacitors store energy in an electric field between two plates. They're measured in capacitance ($C$), which has units of farads (F). A capacitor will charge up to its maximum voltage when connected to a voltage source, and will discharge when the source is removed. Treating capacitors as a black box (even though they are relatively simple), we can describe their behavior in simple AC circuits using impedance:

-

$$ -Z_C = \frac{1}{j\omega C} = \frac{-j}{\omega C} -$$

-

Capacitors in Series

-

$$ -C_{eq} = (\sum_{i=1}^{n} \frac{1}{C_i})^{-1} = (\frac{1}{C_1} + \frac{1}{C_2} + \ldots + \frac{1}{C_n})^{-1} -$$

-

Capacitors in Parallel

-

$$ -C_{eq} = \sum_{i=1}^{n} C_i = C_1 + C_2 + \ldots + C_n -$$

-

Note: combining capacitors behaves opposite to resistors.

-

Inductors

-

Inductors store energy in a magnetic field created by a current flowing through a coil of wire. They're measured in inductance ($L$), which has units of henrys (H). An inductor will resist changes in current, and will generate a voltage to oppose changes in current. Treating inductors as a black box (even though they are also pretty simple), we can once again describe their behavior with impedance:

-

$$ -Z_L = j\omega L -$$

-

Inductors in Series

-

$$ -L_{eq} = \sum_{i=1}^{n} L_i = L_1 + L_2 + \ldots + L_n -$$

-

Inductors in Parallel

-

$$ -L_{eq} = (\sum_{i=1}^{n} \frac{1}{L_i})^{-1} = (\frac{1}{L_1} + \frac{1}{L_2} + \ldots + \frac{1}{L_n})^{-1} -$$

-

Note: just like resistors!

-

Diodes

-

Diodes are one way valves for current, built by sandwiching a p-n (positive-negative) junction between two semiconductors. Diodes have a built in voltage drop ($\approx 0.7$ V), which can be thought of as a threshold voltage that must be overcome before current can flow through the diode. Diodes are often used to rectify AC signals, whereby the negative half of the signal is removed, and the positive half is preserved. There are 2 main regions of operation for a diode:

-
    -
  1. Forward Bias: The diode is conducting, meaning there is sufficient voltage across the diode to overcome the built in voltage drop. The diode will conduct current in the direction of the arrow on the diode symbol, and acts as a voltage source with a drop of $ \approx 0.7$ V.
  2. -
  3. Reverse Bias: The diode is not conducting due to a voltage applied in the opposite direction (can think about it as the current "trying" to flow through it). The diode acts as an open circuit, and no current will flow through it.
  4. -
  5. Zero Bias: The diode is not conducting, as there is no voltage across the diode. This is the same as reverse bias, but the voltage is 0 V.
  6. -
-

Bipolar Junction Transistors (BJTs)

-

BJTs are devices that allow you to control one voltage source with another (often) smaller voltage source. They are used in pretty much every electronic device, and are the building blocks of digital logic and amplifiers. They have 3 terminals: the base, collector, and emitter. They come in two flavors: NPN and PNP.

-

Terminals

- - - - - - - - - - - - - - - - - - - - - -
TerminalDescription
BaseThe control terminal, which allows a small current to control a larger current.
CollectorThe terminal that collects the current from the emitter.
EmitterThe terminal that emits the current to the collector.
-

$$ -I_E = I_C + I_B = \beta I_B + I_B = (\beta + 1)I_B -$$

-

Where $\beta$ is the current gain of the transistor, and is typically around $20$ to $100$.

-

NPN

-

Current flows into the collector, and out of the emitter. The base current controls the collector current. If the base current is zero, the collector current is zero, and if the base current is at its maximum, the collector current is at its maximum.

-
       C
-       |
-      /
-B --|<
-      \
-       |  | Ic
-       E  v
-
-

$$ -I_c = \beta I_b -$$

-

PNP

-

The opposite of NPN. Current flows into the emitter, and out of the collector. The base current controls the collector current, such that if the base current is zero, the

-
       E
-       |
-      /
-B --|<
-      \
-       |  | Ic
-       C  v
-
-

Metal Oxide Semiconductor Field Effect Transistors (MOSFETs)

-

Has 3 terminals: the gate, drain, and source. They are voltage controlled devices, and are used in pretty much every electronic device. They come in two flavors: N-channel and P-channel, or NMOS and PMOS. The high level function is the same as a BJT, but the way they work is different.

-

$$ -I_D = k (V_{GS} - V_{th})^2 -$$

-

Where $k$ is a constant for that specific MOSFET based on its geometry and material, $V_GS$ is the voltage between the gate and source, and $V_{th}$ is the threshold voltage, which is the voltage required to turn the MOSFET on.

-

Terminals

- - - - - - - - - - - - - - - - - - - - - -
TerminalDescription
GateThe control terminal, which allows a voltage to control the current between the drain and source.
SourceThe input terminal for current.
DrainThe output terminal for current.
-

NMOS

-

When the gate voltage is high, the MOSFET is on and current flows from the drain to the source. When the gate voltage is low, the MOSFET is off and no current flows.

-

PMOS

-

When the gate voltage is low, the MOSFET is on and current flows from the drain to the source. When the gate voltage is high, the MOSFET is off and no current flows.

-

PMOS are drawn with a circle on the gate terminal to indicate that the gate voltage is inverted.

-

Operational Amplifiers (Op-Amps)

-

Op-Amps are voltage amplifiers with 2 input pins and 1 output pin. They typically have very high gain, and can be used to increase the voltage or current of a signal. They often use feedback loops to control their gain, and can be used to perform a variety of operations on signals. They are often used in filters, amplifiers, and comparators, but can also be used to perform mathematical operations on signals (like addition, integration, and differentiation).

-

Rules (ideal negative feedback networks)

-

Ideal Op-Amps have 3 terminals: the inverting input (-), the non-inverting input (+), and the output. Negative feedback is when the inverting input is connected to the output, and is an extremely common configuration. The difference between the two input terminal voltages is amplified, and the circuit behaves according to the following properties:

-
    -
  1. The Op-Amp behaves so as to make the voltage difference between the two input terminals zero. You can think of these two terminals as being shorted together (virtual short).
  2. -
  3. No current flows into either of the inputs
  4. -
-

Non-Ideal Op-Amps

-

In practice, real Op-Amps have two additional terminals for a positive supply and ground (or negative supply). The output voltage is limited to the range between the positive and negative supply voltages. The gain is limited by this maximum output voltage.

-
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/circuits/electricity.html b/site/cheatsheets/circuits/electricity.html deleted file mode 100644 index 7b171d0..0000000 --- a/site/cheatsheets/circuits/electricity.html +++ /dev/null @@ -1,251 +0,0 @@ - - - - - - Electricity - - - - - -
- -

Electricity

-
- Last modified: 2024-03-11 - -
-
-

Electricity Basics

-

Flow of Electricity

-

Electricity, being the flow of electrons, is similar to many other types of flow. For prosperity, engineers use conventional current, a bad theory made long ago that positive charges flow from high potential to low potential. In reality, electrons are negatively charged and flow from low potentials to high potentials.

-

Potential? What's that? It's the energy per unit charge at a point in space. It's measured in volts, and is the driving force behind the flow of electricity. From this point on I'll refer to potential as voltage, like a true engineer.

-

If you're smooth brained like me, it helps to think of it as similar to potential energy of balls on a track, but for electrons. A higher voltage means the electron has a long way to fall. This model only takes you so far, but it helps as a mental model for characterizing the behavior of electricity.

-

If potential is the driving factor behind the flow of electricity (like gravity), then current is the magnitude of the flow itself (like throughput). It's measured in amperes (A, Amp), which is a compound unit of charge per unit time.

-

Finally, resistance is the opposition to the flow of electricity. It's measured in ohms ($\Omega$), and can be thought of as anything that drops the voltage (like a hill or friction for potential energy) when current flows through it.

-

Short Circuit

-

A short circuit is just any path with negligible resistance, or ideally zero resistance. Connecting any two points in a circuit with a wire could be call "shorting" them together. Doing this to something like a battery is bad, because a theoretically limitless, but in practice problematic amount of current will flow through a short circuit with a forced voltage drop.

-

Just "shorting" two points together isn't bad, but they need to be points that are supposed to have the same voltage.

-

Open Circuit

-

Open circuits are the opposite of short circuits. They're non-conducting paths, or ideally infinite resistance. No current can flow through an open circuit, and it is like a gap between two wires in a circuit.

-

A simple switch will be an open circuit in the off position, and is literally as simple as a gap in the wire that is closed when the switch is flipped.

-

Ohm's Law

-

$$ -V = IR -$$

-

Where $V$ is voltage, $I$ is current, and $R$ is resistance. The amount of voltage we lose across a resistor is the product of its resistance and the current flowing through it. This simple linear relationship holds, and can be used any time we need to solve for values directly, or the relationship between them.

-

Power

-

$$ -P = IV -$$

-

$$ -P = I^2R -$$

-

$$ -P = \frac{V^2}{R} -$$

-

Power is the rate at which energy is transferred. It's measured in watts (W), and is the product of voltage and current. It can also be expressed in terms of current and resistance, or voltage and resistance.

-

Power can either "sink" into a component, or "source" from it. When power is being dissipated, it's a sink. When power is being generated, it's a source. Typically, when current flows out of a component, it's a source, and when current flows into a component, it's a sink. We represent a source with a positive sign, and a sink with a negative sign.

-

Kirchoff's Laws

-

Kirchoff's Current Law (KCL)

-

The sum of currents entering a node is equal to the sum of currents leaving a node. Current has to go somewhere (it can't just disappear).

-

Kirchoff's Voltage Law (KVL)

-

The sum of voltages around a loop is equal to zero. This is a consequence of conservation of energy. Easiest to start from ground, a reference point with zero voltage, and work your way around the loop right back to any other ground in the circuit to complete the loop.

-

Any other ground? Yes, any other ground. Ground is a reference point, and can be placed anywhere in the circuit. If it helps, you can think of all the grounds as a reference to the same point in space, and the circuit as a series of loops around that point (only for the purpose of KVL analysis).

-

Impedance

-

Impedence ($Z$) is measured in Ohms ($\Omega$), and is basically a generalization of resistance to AC circuits. It is similarly a measure of opposition to flow of electricity, but can be a complex number which results in a phase shift between the voltage and current.

-

It is convinient to transform sinusoidal AC signals into phasors so you don't need to solve a differential equation. Instead of worrying about the real and imaginary parts of the signal seperately as time varies, we instead transform the signal into a vector rotating with the same angular frequency $\omega$, so we only need to worry about the relative offsets and magnitudes for systems oscillating at a single frequency. The following formulas come in handy:

-

Euler's formula:

-

$$e^{j\theta} = \cos(\theta) + j\sin(\theta)$$

-

Phasor representation:

-

$$V(t) = V_m \cos(\omega t + \phi) = V_m \cos(\phi) + jV_m \sin(\phi)$$

-

$$ = V_m \angle \phi = V_m e^{j\phi}$$

-

Series vs. Parallel

-

Series

-

Connected end to end, so that the current flowing out of one flow into the other. Voltage drops accumulate across each component, and the current is the same through each component.

-
+---O---O---O---+
-
-

Parallel

-

Connected side by side, on two branches that originate from the same point. The current is split between the two branches.

-
   +---O---+
-   |       |
----+       +---
-   |       |
-   +---O---+
-
-
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/circuits/index.html b/site/cheatsheets/circuits/index.html deleted file mode 100644 index e686d57..0000000 --- a/site/cheatsheets/circuits/index.html +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - Hello, World! - - - - - - - - - \ No newline at end of file diff --git a/site/cheatsheets/circuits/memory_management_diagram.pdf b/site/cheatsheets/circuits/memory_management_diagram.pdf deleted file mode 100644 index 19966a0..0000000 Binary files a/site/cheatsheets/circuits/memory_management_diagram.pdf and /dev/null differ diff --git a/site/cheatsheets/java-spring-boot/reference.html b/site/cheatsheets/java-spring-boot/reference.html deleted file mode 100644 index 1ca7e3b..0000000 --- a/site/cheatsheets/java-spring-boot/reference.html +++ /dev/null @@ -1,211 +0,0 @@ - - - - - - Reference - - - - - -
- -

Reference

-
- Last modified: 2023-12-21 - -
-
-

Annotations and their meanings

- -
- -
- - \ No newline at end of file diff --git a/site/cheatsheets/java-spring-boot/running.html b/site/cheatsheets/java-spring-boot/running.html deleted file mode 100644 index 36728c0..0000000 --- a/site/cheatsheets/java-spring-boot/running.html +++ /dev/null @@ -1,214 +0,0 @@ - - - - - - Running - - - - - -
- -

Running

-
- Last modified: 2023-12-21 - -
-
-

Quickstart

-

Create a zip file with (start.spring.io)[https://start.spring.io/] including the following dependencies:

- -

Unzip the file

-
unzip helloworld.zip
-
-

Open the project in IntelliJ. Start the server and verify that the application is running by visiting http://localhost:8080/hello

-

Running the application

-
./gradlew bootRun
-
-

Building the application

-
./gradlew build
-
-

Deploying the application

-
./gradlew build
-cp build/libs/<JAR_NAME>.jar <DEPLOYMENT_DIRECTORY>
-
-
- -
- - \ No newline at end of file diff --git a/site/deploy.sh b/site/deploy.sh deleted file mode 100644 index f2cf37a..0000000 --- a/site/deploy.sh +++ /dev/null @@ -1,86 +0,0 @@ -#!/bin/bash - -# Set error handling -set -e - -log_message() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" -} - -# Function to generate commit message based on git status -generate_commit_message() { - local changes=$(git status --short) - echo "sync changes: $(echo "$changes" | head -n 3 | sed 's/^/\n- /')" -} - -# Change to the repository directory -cd "$HOME/repos/notes" || { - log_message "ERROR: Could not change to repository directory" - exit 1 -} - -# Activate virtual environment -if [ -z "$VIRTUAL_ENV" ]; then - if [ -f "venv/bin/activate" ]; then - source venv/bin/activate || { - log_message "ERROR: Failed to activate virtual environment" - exit 1 - } - log_message "Virtual environment activated" - else - log_message "ERROR: Virtual environment not found" - exit 1 - fi -fi - -# Check if git repository exists -if [ ! -d .git ]; then - log_message "ERROR: Not a git repository" - exit 1 -fi - -# Execute deployment steps -log_message "Starting deployment process" - -# Run clean script -if ! python scripts/clean.py; then - log_message "ERROR: Clean script failed" - exit 1 -fi -log_message "Clean script completed successfully" - -# Run build script -if ! python scripts/build.py . site; then - log_message "ERROR: Build script failed" - exit 1 -fi -log_message "Build script completed successfully" - -# Check if there are any changes to commit -if [ -z "$(git status --porcelain)" ]; then - log_message "No changes to commit" - exit 0 -fi - -# Add all changes -if ! git add .; then - log_message "ERROR: Failed to stage changes" - exit 1 -fi - -# Generate commit message and commit -commit_message=$(generate_commit_message) -if ! git commit -m "$commit_message"; then - log_message "ERROR: Failed to commit changes" - exit 1 -fi -log_message "Committed changes with message: $commit_message" - -# Push changes -if ! git push; then - log_message "ERROR: Failed to push changes" - exit 1 -fi -log_message "Successfully pushed changes" - -log_message "Deployment completed successfully" diff --git a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.html b/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.html deleted file mode 100644 index fd1b912..0000000 --- a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.html +++ /dev/null @@ -1,243 +0,0 @@ - - - - - - Fundamentals of Data-Intensive Application Design and Scalability - - - - - -
- -

Fundamentals of Data-Intensive Application Design and Scalability

-
- Last modified: 2025-01-01 - Category: Software Engineering -
-
-

Chapter 1

-

Reliable, Scalable, and Maintainable Applications

-

"Data intensive" vs. "Computer intensive"

-

A data-intensive application is typically built from standard building blocks:

- -

image

-

Reliability

-

A fault is usually defined as one component of the system deviating from its spec, whereas a failure is when the system as a whole stops providing the required service to the user.

-

Hardware faults (hard disk crashes, RAM errors, etc.) -...

-

Software faults (crashes, runaway process, slowdowns, cascading failures, etc.) -Lots of small things can help: carefully thinking about assumptions and interactions in the system; thorough testing; process isolation; allowing processes to crash and restart; measuring, monitoring, and analyzing system behavior in production. If a system is expected to provide some guarantee (for example, in a message queue, that the number of incoming messages equals the number of outgoing messages), it can constantly check itself while it is running and raise an alert if a discrepancy is found

-

Human errors (operator error, configuration error, etc.) -- Design systems in a way that minimizes opportunities for error. -- Decouple the places where people make the most mistakes from the places where they can cause failures. -- Test thoroughly at all levels. -- Allow quick and easy recovery -- Set up detailed and clear monitoring

-

Scalability: the term we use to describe a system's ability to cope with increased -load. Load can be described with a few numbers that are easy to measure:

-

Describing Load -- Requests per second (RPS) -- Reads vs. writes (read-write ratio) -- Cache hit rate -- Concurrent users

-

When you increase a load parameter and keep the system resources (CPU, memory, network bandwidth, etc.) unchanged, how is the performance of your system affected?

-

When you increase a load parameter, how much do you need to increase the resources if you want to keep performance unchanged?

-

Latency and Response Time

-

Latency is the duration that a request is waiting to be handled-during which it is latent, awaiting service. Response time is the delay between a client sending a request and receiving a response.

-

When Measuring Load

-

Use percentiles. Look at the median (p50), the 95th percentile (p95), and the 99th percentile (p99).

-

When making multiple backend calls to external services, take special care to measure the p99 of the overall request latency (not just the p99 of each individual service call). This prevents latency amplification that would potentially impact users.

-

Coping with Load

- -

There is no generic solution. Scale based on access patterns, not on data size.

-

Maintainability

-

Operability: make it easy for operations teams to keep the system running smoothly. Make common tasks easy, and preferably automatic. Good monitoring is also crucial.

-

Simplicity: make it easy for new engineers to understand the system, by removing as much complexity as possible from the system. Manage complexity with abstraction.

-

Evolvability: make it easy for engineers to make changes to the system in the future, adapting it for unanticipated use cases as requirements change. Good abstractions and modularity allow components to be replaced, and the overall system architecture to be modified, without complete reimplementation.

-
-
Tags: data systems, maintainability, performance, reliability, scalability
-
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.html b/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.html deleted file mode 100644 index 49a8eb4..0000000 --- a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.html +++ /dev/null @@ -1,354 +0,0 @@ - - - - - - Data Models and Relationships in Database Systems - - - - - -
- -

Data Models and Relationships in Database Systems

-
- Last modified: 2025-01-01 - Category: Database Design -
-
-

Chapter 2

-

Data Models and Query Languages

-

Relational Model We all know SQL

-

Document Model As well as NoSQL

-

Mismatch between Relational Model and OOP

-

Awkward translation between objects and relations leads to middleware, oftentimes a object-relational mapping (ORM)

-

Relationships

-

It is useful to use ids to refer to related data, rather than embedding it. It never changes, and it is often much smaller.

-

One-to-many relationships are by far the most common type of relationship in databases.

-
CREATE TABLE users (
-    user_id INT PRIMARY KEY,
-    username VARCHAR(200) UNIQUE
-);
-
-CREATE TABLE orders (
-    order_id VARCHAR(255) PRIMARY KEY,
-    user_id INT REFERENCES users (user_id)
-);
-
-
{
-  "user_id": 1,
-  "username": "martin",
-  "order_ids": [1, 2]
-}
-
-

Many-to-many relationships are less common, but they do exist.

-
CREATE TABLE users (
-    user_id INT PRIMARY KEY,
-    username VARCHAR(200) UNIQUE
-);
-
-CREATE TABLE groups (
-    group_id INT PRIMARY KEY,
-    group_name VARCHAR(200) UNIQUE
-);
-
-CREATE TABLE user_groups (
-    user_id INT REFERENCES users (user_id),
-    group_id INT REFERENCES groups (group_id),
-    PRIMARY KEY (user_id, group_id)
-);
-
-
{
-  "user_id": 1,
-  "username": "martin",
-  "group_ids": [1, 2]
-}
-
-

Many-to-many relationships are natural in SQL, but awkward in document databases. Often, the best way to deal with many-to-many relationships in a document database is to denormalize some of the data, since joins are limited.

-
SELECT * FROM users
-JOIN user_groups USING (user_id)
-JOIN groups USING (group_id)
-WHERE group_name = 'devops';
-
-
{
-  "user_id": 1,
-  "username": "martin",
-  "groups": [
-    { "group_id": 1, "group_name": "devops" },
-    { "group_id": 2, "group_name": "dba" }
-  ]
-}
-
-

Network model is a generalization of the hierarchical model, in which a child record may have several parents. It is a graph of records, connected by links.

-

Graph model is a further generalization, in which edges can have properties as well as vertices.

-

Choosing a data model

-

Schema-on-write is the traditional approach of defining a relational schema for your data, and then writing data that conforms to the schema. It is good for ensuring data quality, and it is also good for performance. However, it is inflexible.

-

Schema-on-read is the approach of not enforcing a schema ahead of time, but only reading it when you read from the database. It is good for handling data with rapidly changing requirements, and for cases where you need to load data from many different applications.

-

Choose whichever model simplifies your application code the most, and matches your access patterns.

-

Data locality

-

is the collocation of related data items on the same storage device (e.g. disk block, server node, etc.). It is important for performance.

-

Locality in query execution is the collocation of data items that are accessed together in the same query. It is important for performance.

-

For document databases, you need to load the entire document, even if you only need a small portion of it. Keep documents small.

-

Examples: Google Spanner uses nested data structures, and it allows secondary indexes on nested fields. MongoDB and Elastic search allow you to index nested fields, but they don't allow secondary indexes on nested fields. Column family databases such as BigTable, HBase, and Cassandra.

-

Many relational databases also have XML and JSON data types, which allow you to store documents within a row of a table.

-

MapReduce Querying

-

MapReduce is a programming model for processing large amounts of data in bulk across many machines, popularized by Google. It is a two-step process:

-
    -
  1. -

    The map step takes a document as input and produces an intermediate representation of key-value pairs.

    -
  2. -
  3. -

    The reduce step takes the output of the map step, with the same key appearing once for each value, and produces a result. The result can be a single value per key, or an arbitrarily complex data structure.

    -
  4. -
-

MapReduce is a good model for batch processing of data, but it is not suitable for interactive queries or applications that require low latency. Some NoSQL databases have a limited form of MapReduce for reading batches of documents (e.g. MongoDB, CouchDB).

-
-- traditional SQL
-SELECT date_trunc('month', observation_timestamp) AS observation_month,
- sum(num_animals) AS total_animals
-FROM observations
-WHERE family = 'Sharks'
-GROUP BY observation_month;
-
-
// Map Reduce MongoDB
-db.observations.mapReduce(
-    // collect data
-  function map() {
-    var year = this.observationTimestamp.getFullYear()
-    var month = this.observationTimestamp.getMonth() + 1
-    emit(year + '-' + month, this.numAnimals)
-  }, // aggregate the data
-  function reduce(key, values) {
-    return Array.sum(values)
-  },
-  { // query to select the documents
-    query: { family: 'Sharks' },
-    out: 'monthlySharkReport'
-  }
-)
-
-
// Aggregation pipeline with MongoDB
-db.observations.aggregate([
- { $match: { family: "Sharks" } },
- { $group: {
- _id: {
- year: { $year: "$observationTimestamp" },
- month: { $month: "$observationTimestamp" }
- },
- totalAnimals: { $sum: "$numAnimals" }
- } }
-]);
-
-

Property Graphs

-

Each vertex/node consists of:

- -

Each edge consists of:

- -
-- Conceptual graph model in SQL
-CREATE TABLE vertices (
- vertex_id integer PRIMARY KEY,
- properties json
-);
-
-CREATE TABLE edges (
- edge_id integer PRIMARY KEY,
- tail_vertex integer REFERENCES vertices (vertex_id),
- head_vertex integer REFERENCES vertices (vertex_id),
- label text,
- properties json
-);
-
-CREATE INDEX edges_tails ON edges (tail_vertex);
-
-CREATE INDEX edges_heads ON edges (head_vertex);
-
-

In practice, graph databases are usually implemented differently, because the above model is not very efficient. Specialized query languages are used to traverse the graph in an imperative style.

-

Triple-Stores and SPARQL

-

Consists of three-part statements of the form (subject, predicate, object). The subject and object are vertices, and the predicate is an edge.

-

```sql

-
-
Tags: data modeling, document databases, graph databases, query languages, relational databases
-
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.html b/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.html deleted file mode 100644 index 0de7133..0000000 --- a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.html +++ /dev/null @@ -1,290 +0,0 @@ - - - - - - Storage and Retrieval Techniques for Database Systems - - - - - -
- -

Storage and Retrieval Techniques for Database Systems

-
- Last modified: 2025-01-01 - Category: Database Systems -
-
-

Chapter 3

-

Storage and Retrieval

-
#!/bin/bash
-
-# instant database
-db_set () {
- echo "$1,$2" >> database
-}
-db_get () {
- grep "^$1," database | sed -e "s/^$1,//" | tail -n 1
-}
-
-

The fact that it is a log makes this...sort of performant. Writes are fast O(1), and reads are slow O(n). The log-structured storage engine is a good fit for append-only workloads, such as event sourcing.

-

To make reads faster, we can use an index. An index is an additional structure that is derived from the primary data. This is a trade-off between write performance and read performance, since writes now have to update the index as well as the primary data.

-

Hash Indexes

-

Map keys to offsets in the data file. This is fast for equality queries, but not for range queries. Store log-structured key-value data like above in binary format, and use a hash index to find the offset of the key in the data file. Delete by marking the key as deleted in the data file (sometimes with a "tombstone"), and periodically reindex the data file to remove the deleted keys.

-

For crash recovery:

- -

For concurrency, maintain a single write thread, and multiple read threads. Writes are serialized, but reads can be parallelized.

-

SSTables and LSM-Trees

-

Sorted String Tables (SSTables) are a way to store key-value data in sorted order. Keep data sectioned based on time range, and search from most to least recent, periodically merging previous sections of data. Should store key value pairs, as well as byte offset metadata. Range queries are fast, and merge operations can be optimized by keeping most recent record and merging ranges at a time. However, both reading and writing are slower than hash indexes.

-

Can use B-Trees to store SSTables on disk, or various balanced binary trees (e.g. AVL, red-black, etc.) to store data in memory. These are called Log-Structured Merge Trees (LSM-Trees).

-

Below is a simplified version of the algorithm used by LevelDB, which is similar to the one used by Cassandra and HBase (which were inspired by Bigtable):

- -

Lucene (used by Elasticsearch and Solr) uses a similar algorithm for its term-dictionary indexes that support full-text search. It uses a log-structured merge (LSM) tree to merge segments of the term dictionary. Here, words are the keys, and the values are the list of documents that contain the word (in a "Posting List"). This is kept in SSTables, and merged periodically.

-

Performance Optimizations

-

Bloom filters are a way to avoid reading SSTables that don't contain a key. They are a memory-efficient data structure for approximating the contents of a set. Give false positives, but no false negatives. Can be used to avoid reading SSTables that don't contain a key.

-

Merge strategies such as size-tiered, where segments are merged when they reach a certain size, or leveled, where segments are kept in subranges.

-

Keeping things append-only is a good way to keep throughput for writes high.

-

B-Trees vs LSM-Trees

-

B-trees trade off write speed for read speed. N-ary tree with sorted keys in every node. Increase branch factor for maximal locality and minimal disk seeks.

-

LSM-trees trade off read speed for write speed. Writes are sequential, but reads are not. Writes are batched, and reads are parallelized. Write amplification (multiple writes to disk for a single DB write) can be a problem on both B-trees and LSM-trees. However, LSM-trees often still have a higher write throughput.

-

Compaction can impact LSM-tree read performance, but can be mitigated by keeping a small number of SSTables per level, and by using a merge strategy that does not require reading all SSTables. Still, it can impact high percentiles of read latency.

-

With high enough write throughput, it is also important to monitor disk space usage, since compaction can be slow enough that it doesn't keep up with the incoming writes.

-

Secondary Indexes

-

Secondary indexes are indexes on non-primary keys. They can be implemented with hash indicies (still bad for range-queries), or with B-trees (good for range queries, but slower writes). They can also be implemented with LSM-trees. Similar to the setup for full-text search (Posting Lists), can use a list of matching records as a value in an index (inverted index).

-

Storing Indexes

-

Must choose between storing references, or data directly in the index. Storing references is more space efficient update-friendly, but requires more disk seeks. Storing data directly in the index is more read friendly, but requires more disk space, and makes updates slower.

-

Heap files let you store data referenced in your index and update in place when the new data fits in the space of the old data. Otherwise, you have to move the data to a new location, and update the index (or heap file with redirect pointer). However, this extra indirection can slow down reads.

-

Directly storing data in the index (clustered index) is good for read-heavy workloads, and storing references is good for write-heavy workloads. InnoDB (MySQL storage engine) uses a clustered index for primary keys, and a secondary index point directly to the primary key (instead of a heap file). SQLServer let you specify one clustered index per table.

-

B-trees are good for one-dimensional indexes, but not for multi-dimensional indexes. R-trees are a good alternative for multi-dimensional indexes, but are more complex.

-

Fuzzy indexes are often used in full-text search. They are good for matching similar words, but not for exact matches. Lucene keeps an in-memory finite state automaton over characters in each key that is similar to a trie, allowing for fuzzy matching to a certain "edit distance".

-

Keeping everything in memory

-

As RAM gets cheaper, it makes more sense to keep data in memory. In-memory databases are good for applications that need low latency and high throughput, but make crash recovery more difficult. Write-ahead logs are a good way to make crash recovery easier, but can slow down writes. Some databases (e.g. Redis) let you choose between durability and performance, and have "weak durability" by asyncronously writing to disk. Some in-memory databases can even exceed the amount of RAM available by using an eviction policy similar to a cache.

-

Transaction Processing or Analytics?

-

OLTP (online transaction processing) is good for real-time stateful workloads, where low latency and high throughput are important. OLAP (online analytics processing) is good for async batch processing, where high throughput is important, but latency is not.

-

OTLP:

- -

OLAP:

- -

Most of the previous indexes are more OTLP focused, whereas "data warehouses" are more OLAP focused, and often use a different schema and index model.

-

Stars and Snowflakes: Schemas for Analytics

-

Star schema is a simple schema for OLAP. It has a single fact table containing all the records you want to query, and multiple dimension tables that contain the attributes you want to be queriable. The fact table contains foreign keys to the dimension tables.

-

Snowflake schema is a more complex schema for OLAP. It is similar to a star schema, but the dimension tables are normalized into multiple tables. This can make queries more complex, but can also reduce storage space.

-

Warehouses can get huge (petabytes), since records are often events and are kept long term. Columns are often also very wide (100s of columns), since they are often denormalized.

-

Column-Oriented Storage

-
SELECT
- dim_date.weekday, dim_product.category,
- SUM(fact_sales.quantity) AS quantity_sold
-FROM fact_sales
- JOIN dim_date ON fact_sales.date_key = dim_date.date_key
- JOIN dim_product ON fact_sales.product_sk = dim_product.product_sk
-WHERE
- dim_date.year = 2013 AND
- dim_product.category IN ('Fresh fruit', 'Candy')
-GROUP BY
- dim_date.weekday, dim_product.category;
-
-

In transactional databases, storage is "row-oriented", so an entire row needs to be loaded, including columns that aren't used. column-oriented storage is better for OLAP, since it only needs to load the columns that are used, which can amount to a lot of space over large datasets. It also allows for better compression, since columns are often similar.

-

Can be heavily compressed using bitmap indexes, and further by run-length encoding, delta encoding, etc. can also be optimized with vectorized processing (SIMD instructions), where instructions are executed in parallel on multiple values at once (C++ example).

-

Sort Order in Column Storage

-

Can choose a sort key to order rows in a column, and secondary sort key to break ties for a given value. Allows fast range queries, helps optimize compression (like with run length encoding for duplicates), and gives better locality for sequential reads.

-

Can even maintain several sort orders for a column, and choose the best one for a given query. This is called a compound sort key.

-

Writing to Column-Oriented Storage

-

These optimizations are good for reads and make sense in a data-warehouse, but can make writes slower because it has to update multiple indexes. This is fine for OLAP, and we can rely on LSM-trees to make writes fast initially, and periodically merge and compact the data.

-

Aggregation: Data Cubes and Materialized Views

-

Materialized views are precomputed joins and aggregations. They are good for speeding up queries, but can be expensive to maintain. They are often used in data warehouses, where the data is read-heavy, and the cost of maintaining the materialized views is amortized over many queries.

-

Data cubes are a way to precompute aggregations over multiple dimensions. They are good for speeding up queries, but can be expensive to maintain. Essentially a multi-dimensional array, where each cell is an aggregation over a subset of the dimensions. Very expensive to maintain, and inflexible for queries that aren't covered by the precomputed aggregations.

-

Oftentimes, it makes more sense to store raw data, and then benchmark queries to see which ones are slow, and then precompute aggregations for those queries if they need to be faster.

-
-
Tags: column-oriented storage, data structures, indexing, oltp vs olap
-
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.html b/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.html deleted file mode 100644 index b9ac885..0000000 --- a/site/designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.html +++ /dev/null @@ -1,293 +0,0 @@ - - - - - - Encoding, Evolution, and Data Flow in Distributed Systems - - - - - -
- -

Encoding, Evolution, and Data Flow in Distributed Systems

-
- Last modified: 2025-01-01 - Category: Distributed Systems -
-
-

Chapter 4

-

Encoding and Evolution

-

evolvability: the ability to evolve as requirements change.

-

schema-on-read: schema isn't enforced when data is written, only when it is read. it can change over time.

-

schema-on-write: schema is enforced when data is written. it is usually enforced by a database schema.

-

backward compatibility: new code can read data that was written by old code, meaning new code understands old fields.

-

forward compatibility: old code can read data that was written by new code, meaning old code ignores new fields.

-

Formats for Encoding Data

-

Programs typically work with at least two different representations of data:

- -

serialization: the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or transmitted (for example, across a network connection link) and reconstructed later (possibly in a different computer environment). Also called encoding/marshaling.

-

deserialization: the reverse process of serialization, which is to extract data structures from a series of bytes. Also called decoding/unmarshaling.

-

Many built in serialization formats that are included with languages are not performant or space efficient (Java's built in serialization, Python's pickle, etc.). They are also not compatible with other languages. Generally a bad idea to use these formats for long term storage or communication between services.

-

JSON and XML are popular serialization formats that are language independent and human readable. Both have good support for utf-8 strings, but not for binary strings. Workarounds with base64 encoding work, but increase data size by 33%.

-

CSV is also popular, albeit less powerful. It is not self describing, so it is not good for data with multiple types, and instead relies on application code to interpret the data.

-

Binary encoding is essentially a custom encoding format, although binary implementations for other encodings exist. They are usually more compact than JSON, XML, and CSV, and are also usually faster to encode and decode. Probably a bad idea for inter-org communication, but acceptable for internal communication.

-

Protocol Buffers (protobuf) is a popular binary encoding format. It is a schema language that defines data types and a binary encoding format. It is not self describing, and requires a schema definition to be decoded. Most languages have protobuf libraries that generate classes from a schema definition in order to encode/decode data. Uses field tags to identify fields in the binary encoding, which are assigned to each field in the schema definition. This allows fields to be reordered without breaking compatibility. Doesn't have array/list types, instead uses repeated fields.

-

Thrift similar to protobuf, uses IDL to define data types and a binary encoding format. It is more mature than protobuf, but has a more complex schema language. Also has many code gen tools for different languages. Has 3 different protocols: dense protocol (C++ only), binary protocol (all languages), and compact protocol (all languages). In binary protocol, field names are not included, instead field tags are used, which are assigned to each field in the schema definition. Supports array/list types, and allows nested lists.

-

Avro Another binary encoding that was made as an alternative to protobuf and thrift for Hadoop. Uses a schema to specify encoding. Has two schema langs: IDL, for human editing, and one similar to JSON, for machine reading. Doesn't use tag numbers or any identifying field info; just lengths and data values. Uses variable length integers to encode lengths. Uses writer/reader schema setup.

-

What makes protobuf, thrift and avro good?

- -

Writer Reader Schema Setup

- -

Versioning is tricky. Depending on the use case, it is handled differently. For example, if you are storing gigabytes of data in Hadoop, all encoded with the same schema, then you only need to include the writer schema once in the file.

-

However, in a database where data is written over a long period of time, you can use a version number, and a table of versions for schema version.

-

In an inter-service communication context, schemas can be negotiated between client and server on connection setup. Using a database of schema versions might be a good idea here.

-

Dynamically Generated Schemas

-

Situation: Want to dump database to file on disk using a binary encoding format.

-

Problem: Schema has evolved over time, and there are many different versions of the schema in the database. Would need to manuallty assign field tags for each version of the schema, and then write a custom encoder/decoder for each version.

-

Solution: Avro schema dynamically generated from database schema. Can be rerun whenever schema changes.

-

This kind of dynamic schema generation was a design goal of Avro, and is less so for protobuf and thrift. Static code generation was a design goal of protobuf and thrift, and is less so for Avro. This makes Avro a natural choice in dynamic languages where code gen is frowned upon (and more generally compile steps), although there are also code generation tools for Avro in static languages.

-

Keeping a database of schema versions is a good idea, and can be used to generate Avro schemas for each version of the schema. This can be used to generate a schema for each version of the database, and then used to encode/decode data from each version of the database.

-

Modes of Dataflow

-

Generally, one process encodes, and another decodes.

-

Dataflow Through Databases

-

The process that writes to the database is the encoder, and the process that reads from the database is the decoder. Backwards compatability is nessessary for the database to work. It is likely at some point that multiple processes accessing a given database will run different versions of "the code" at a given time. Thus, you need both forward and backward compatibility. Preservation of unknown fields is tricky, and sometimes needs to be solved at the application level.

-

Generally, can update any value at any time, especially long term; "Data outlives code".

-

Dataflow Dumping to Files to Archival Storage

-

Encoder: process that dumps data to file. Decoder: process that reads data from file. Typically encoded with latest schema, and is a good opportunity to encode in OLAP friendly format (column oriented). Avro is a good choice.

-

Dataflow Through Services: REST and RPC

-

Encoder: client process that sends request to server. Decoder: server process that receives request from client. Generally, client and server are written by same org, so they can be updated at the same time. However, if client and server are written by different orgs, then they may be updated at different times. Thus, you need both forward and backward compatibility.

-

Service oriented architectures are often used in large orgs, where different teams are responsible for different services. Services expose an API, which provides encapsulation that allows the service to be updated without breaking clients. This means that different services may be updated at different times, and thus may be running different versions of the code. Thus, you need both forward and backward compatibility. This is especially true for public APIs, where you have no control over the client. Web services are HTTP based services that exist in different contexts:

- -

REST is a design philosophy that builds on HTTP. Uses simple data formats, URLs for identifying resources, and HTTP features for cache control, authentication, and content type negotiation.

-

I'm not even going to get into SOAP.

-

RPC is a design philosophy that builds on the idea of calling a function on a remote server. Tries to provide location transperency by abstracting network communication when triggering remote method calls. Many are overly complex, and are not compatible with other languages. gRPC is a modern RPC framework that uses protobuf as the interface definition language, and HTTP/2 as the underlying protocol. It is a good choice for internal services, but not for public APIs.

-

Problems with RPC

-

Network requests are fundamentally different from local function calls. They can fail, be delayed, or be delivered multiple times. This makes designing systems that use RPC more complex, since you need to take into account these failure modes. Idempotence is a good way to allow transparent retries, since it means that the same request can be sent multiple times without causing unintended side effects. REST doesn't try to hide the fact that it is a network protocol, and thus is more transparent about failure modes.

-

Furthermore, the platform lock-in of RPC is a problem. This ties you to a specific language, and makes it difficult to use other languages. This is especially true for public APIs, where you have no control over the client.

-

Current Direction of RPC

-

Various RPC frameworks exist on top of all the previously mentioned encodings. New generations of RPC frameworks use futures/promises to represent asynchronous responses, and use streaming to represent long lived connections. Some frameworks also include service discovery, which allows clients to find servers without hardcoding their location.

-

Advantages of REST

-

Vast ecosystem of tools, including servers, load balancers, proxies, caches, and monitoring tools. This makes it easy to build a scalable system. Also, since REST is built on HTTP, it is easy to debug with standard tools. Furthermore, since REST is built on HTTP, it is easy to build a system that works with web browsers. This is especially true for public APIs, where you have no control over the client.

-

Data Encoding and Evolution in RPC

-

Simplifying assumption about services: All servers updated first, then all clients. This requires forward compatibility, but not backward compatibility.

-

Service compatibility is difficult in RPC, since it oftentimes needs to be maintained indefinitely. There is no single standard for versioning in RPC, and thus it is often done ad-hoc.

-

Dataflow Through Asynchronous Message-Passing

-

Somewhere between RPC and Databases. Messages are sent from one process to another, and the sender doesn't wait for a response. This is useful for handling high volumes of messages, and for decoupling the sender from the receiver. Uses a message broker (like Kafka, RabbitMQ, etc.), which is a server that receives messages from producers and delivers them to consumers. The broker is responsible for routing messages to the correct destination.

-

Usually works as follows:

- -

A producer is one way communication, but can be used for request/response. This is useful for decoupling the sender from the receiver, and for handling high volumes of messages. Message brokers often don't enforce any particular data format, and thus are not opinionated about encoding. However, they do need to be able to route messages to the correct destination, and thus need to be able to read the message headers.

-

Several advantages:

- -

Usually one way communication, but can be used for request/response. This is useful for decoupling the sender from the receiver, and for handling high volumes of messages.

-

Distributed Actor Frameworks

-

Actor model: a model of concurrent computation that treats actors as the universal primitives of concurrent computation. In response to a message that it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next message received. Actors are essentially state machines that communicate by sending messages to each other.

-

The same message passing system is used for communication between actors on the same machine and actors on different machines. This means that the same code can be used for both local and remote communication. There is less of a mismatch between local and remote communication, since they use the same primitives. This is in contrast to RPC, where local and remote communication are fundamentally different.

-

Actor frameworks are libraries that provide an implementation of the actor model. They are often used for building distributed systems, since they provide a way to scale up the number of actors. They are also useful for building systems that need to be highly available, since they provide a way to replicate actors. Still need to worry about forward and backward compatibility. Different frameworks have different approaches to compatibility. For example:

- -
-
Tags: compatibility, data serialization, encoding formats, message passing, schema evolution
-
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-2-distributed-data/ch5-replication.html b/site/designing-data-intensive-applications/part-2-distributed-data/ch5-replication.html deleted file mode 100644 index 2d01aad..0000000 --- a/site/designing-data-intensive-applications/part-2-distributed-data/ch5-replication.html +++ /dev/null @@ -1,265 +0,0 @@ - - - - - - Replication Strategies in Distributed Data Systems - - - - - -
- -

Replication Strategies in Distributed Data Systems

-
- Last modified: 2025-01-01 - Category: Distributed Systems -
-
-

Chapter 5

-

Replication

-

Replication is the process of keeping a copy of the same data on multiple machines that are connected via a network. Replication is important for a few reasons:

- -

Most distributed data systems that use replication follow one of single-leader, multi-leader, or leaderless replication. The hard part is keeping replicas consistent with each other in the event of updates.

-

Leaders and Followers

-

Also known as master slave or active passive replication, this is the simplest form of replication. One node is designated as the leader/master/primary and the rest are followers/slaves/secondary. The leader handles all write requests and propagates the changes to the followers. The followers handle all read requests. If the leader goes down, one of the followers is promoted to leader. This is a common setup for relational databases. Followers are read-only, and writes are only sent to the leader.

-

Many relational, and some non-relational databases use this setup, as does non-database systems like distributed message-brokers (Kafka, RabbitMQ, etc.).

-

Synchronous Versus Asynchronous Replication

-

The leader can propagate changes to the followers in two ways: synchronous and asynchronous. In synchronous replication, the leader waits for a response from the follower before acknowledging the write request. In asynchronous replication, the leader does not wait for a response from the follower before acknowledging the write request.

-

With synchronous replication, the leader can guarantee that the followers are up to date. However, if the network is slow or the follower is down, the leader cannot acknowledge the write request. This means that the leader cannot accept any more writes until the follower responds. This can cause a cascading failure if the leader is waiting on multiple followers. This is known as write availability.

-

If there is a substantial delay between the leader and the follower, the leader may not be able to accept any writes at all. This is known as write latency, and is a common problem with synchronous replication. In practice, if you enable synchronous replication in a database, this often means that you have a syngle syncronous follower, and the rest are asynchronous. This is known as semi-synchronous replication. If the synchronous follower goes down, the leader elects a new synchronous follower. This guarantees that there is always at least two up to date copies of the data.

-

Leader-based replication is often fully asynchronous. This means that the leader does not wait for a response from the follower before acknowledging the write request. This means that the leader can accept writes even if the follower is down. However, this means that the followers may not be up to date. This is known as replication lag. If the leader goes down, the follower with the most up to date data is elected as the new leader. This means that the new leader may not have all of the writes that the old leader had. This is known as read-your-writes consistency and is a tradeoff that incurs weakened durability. This is a common setup for non-relational databases.

-

Research on Replication

-

Preventing data loss is a hot topic in distributed systems research. There are a few different approaches to this problem, but one example is Chain Replication, a variant of syncronous replication, where the leader sends the write to the first follower, which sends it to the second follower, and so on. The leader does not acknowledge the write until all followers have acknowledged it. This is a synchronous replication protocol that guarantees that the followers are up to date. However, it is not fault tolerant. If any follower goes down, the leader cannot accept any more writes. This is a common setup for distributed message brokers.

-

Setting Up New Followers

-

A simple "file copy" might read across inconsistent snapshots of the data. To avoid this, maintain a consistent snapshot of the data by using a consistent snapshot protocol. Follower nodes start from snapshot and request all writes that have happened since the snapshot (using the log sequence number is Postgres and the binlog coordinates in MySQL). The leader keeps a log of all writes, and the followers request all writes since the snapshot. This workflow varies depending on the database.

-
    -
  1. take consistent snapshots periodically, and ideally asynchronously
  2. -
  3. copy the snapshot to the new follower node
  4. -
  5. the follower connects to the leader and requests all writes since the snapshot
  6. -
  7. follower processes backlog until it catches up to the leader
  8. -
-

Handling Node Outages

-

Goal is to keep the whole system running despite individual failures. Ideally should be able to reboot individual nodes without affecting the whole system.

-

Follower Failure

-

If a follower fails, it can be restarted from the log located on its disk. Once the node has processed its log, it then requests any new writes from the leader. This is known as catch-up recovery.

-

Leader Failure

-

If the leader fails, one of the followers is elected as the new leader during failover. This is way harder. It usually consists of the following:

-
    -
  1. Determining the leader failed. This can be acieved passively with a timeout or actively with a heartbeat.
  2. -
  3. Assigning a new leader. Can be done through an election process, or a designated controller node. Probably a good idea to look for nodes with most up-to-date data.
  4. -
  5. Reconfigure to use new leader. Set clients to write to new leader, and set followers to read from new leader.
  6. -
-

Common problems with failover:

- -

Implementation of Replication Logs

-

Statement-based replication

-

Leader logs ever write request (SQL statement in the case of a RDB) and sends log to followers. Each follower parses and executes the statement as if it were initiated by the client.

-

Problems:

- -

Although there are workarounds, usually not used today. MySQL <5.1 used it, and still sometimes does for deterministic functions.

-

Write-ahead log (WAL) shipping

-

Append only sequence of bytes that record every write to the database. Leader sends WAL to followers, which apply the writes to their own database to build up the same data structures as the leader. Used by Postgres, Oracle, and more.

-

Problems:

- -

In particular, WAL prevent us from doing downtime-free upgrades. Without them, we could upgrade all the followers and catch them up, then failover and upgrade the leader. WAL make this impossible in many cases because the leader and follower must both read and write the same WAL format.

-

Logical (row-based) log replication

-

Use different formats for storage engine (physical) log and replication (logical) log. For RDB, usually a sequence of records describing writes to rows in database.

- -

With multi-row transactions, log contains entry for all rows, as well as entry for transaction commit (MySQL binlog uses this approach)

-

Logical logs decouple the replication from the storage engine, allowing better compatibility between versions. Also allows integrations with external services (e.g. Kafka, Elasticsearch, etc.)

-
-
Tags: data replication, failover, leader-follower model, replication logs, synchronous vs asynchronous
-
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-2-distributed-data/preface.html b/site/designing-data-intensive-applications/part-2-distributed-data/preface.html deleted file mode 100644 index 29672e6..0000000 --- a/site/designing-data-intensive-applications/part-2-distributed-data/preface.html +++ /dev/null @@ -1,208 +0,0 @@ - - - - - - Preface - - - - - -
- -

Preface

-
- Last modified: 2023-12-26 - -
-
-

Preface

-

Distributed Data

-

Moving up a level to data systems that run on multiple machines, we still care about similar things.

- -

Scaling Up

-

Shared memory architecture is a single computer with multiple CPUs, each with their own cache and memory. The CPUs communicate via a shared memory bus. Prices to double power of machine quickly become prohibitive, and the shared memory bus becomes a bottleneck anyways.

-

Shared Disk Architecture is a multi machine setup with each machine having its own CPU and memory, but all machines share a disk over the network. This is a bit better, but the disk becomes a bottleneck.

-

Scaling Out

-

Shared Nothing Architecture is a multi machine setup with each machine having its own CPU, memory, and disk. This is the most scalable, but it requires a lot of coordination between machines. Each machine (or VM) is a node, and each node is completely independent. Inter-node communication is done over the network through software.

-

For the remainder of this part (Part 2), we will be focusing on shared nothing architectures.

-
- -
- - \ No newline at end of file diff --git a/site/designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.html b/site/designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.html deleted file mode 100644 index 79fb149..0000000 --- a/site/designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.html +++ /dev/null @@ -1,349 +0,0 @@ - - - - - - Batch Processing Systems and MapReduce Fundamentals - - - - - -
- -

Batch Processing Systems and MapReduce Fundamentals

-
- Last modified: 2025-01-01 - Category: Distributed Systems -
-
-

Chapter 10

-

Batch Processing

-

Services (online systems) are design to handle requests from users or other services. Performance is measured in requests per second and response time.

-

Batch processing (offline systems) run scheduled jobs periodically that process accumulated data. Performance is measured in throughput.

-

Stream processing (near-real-time systems) are a hybrid of online and offline systems. Performance is measured in latency, and they usually take in a stream of events and calculate aggregates in real time, as opposed to running calculations on accumulated data.

-

Batch Processing with Unix Tools

-

Log analysis is a common batch processing task. Applications append log entries to a file, and a batch job periodically processes the log file and generates a report.

-
cat /var/log/<application>/<logfile> |  # read the log file
-    awk '{print $<url_idx>}' |          # extract the 7th field (URL)
-    sort |                              # sort the URLs
-    uniq -c |                           # count the number of occurrences of each URL
-    sort -r -n |                        # sort numerically in descending order
-    head -n 5                           # take the top 5
- ```
-
- Equivalently, using Python:
-
- ```python
-from collections import Counter
-
-with open('/var/log/<application>/<logfile>') as f:
-    urls = [line.split()[<url_idx>] for line in f]
-    for url, count in Counter(urls).most_common(5):
-        print(url, count)
- ```
-
-However, these two examples differ in that python uses an in-memory hash table, whereas the Unix pipeline uses a disk-based merge sort that can handle data sets larger than memory, and is thus more scalable.
-
-### MapReduce and Distributed Filesystems
-
-**MapReduce** is a programming model for processing large amounts of data in bulk across many machines. It is a batch processing system that runs a user-defined *map* function in parallel over many input *records*, and then runs a user-defined *reduce* function in parallel over the output of the map function.
-
-**Hadoop** is an open source implementation of MapReduce. It is a distributed system that runs on a cluster of machines, and it includes a distributed filesystem called **HDFS** (*Hadoop Distributed Filesystem*).
-
-**HDFS** is designed for storing large files with streaming access patterns, and is optimized for throughput rather than latency. It is based on the *Google File System* (*GFS*), and is similar to *Amazon S3*, although it is not an object store and allows you to run computations on the data stored in it.
-
-Using the Unix pipeline example from above, a (single node) MapReduce job would look like this:
-
-1. Read all input file logs and break into records (lines)
-2. Map: extract the URL from each record and output a key-value pair of `(URL, _)`
-3. Sort all key-value pairs by key
-4. Reduce: count the number of occurrences of each URL
-
-A multi-node MapReduce job would look like this:
-
-1. Read all input file logs and break into records (lines)
-2. Map: extract the URL from each record and output a key-value pair of `(URL, _)`
-3. Shuffle: group all key-value pairs by hashed key and schedule a reduce task for each group written to disk (sorted by url)
-4. Reduce: count the number of occurrences of each URL in the group
-
-This is a single MapReduce job, but often people run a sequence of MapReduce jobs in a pipeline, where the output of one job is the input of the next job. This is called a **workflow**. You can run jobs in sequence using input and output files, or use a scheduler like **Airflow** to manage the workflow.
-
-**Sort-merge joins** Used to combine two sorted lists of records into one sorted list of records. They are used in MapReduce to join the output of the map function before the reduce function.
-
-```python
-# psuedocode to join event and user data by user_id
-# this is JUST PSEUDOCODE, not actual MapReduce code
-
-# user: { user_id, name, date_of_birth, ... }
-map_user_data(user):
-    emit_intermediate(user.user_id, user.date_of_birth)
-
-# event: { user_id, event_type, ... }
-map_events(event):
-    emit_intermediate(event.user_id, event.event_type)
-
-# join: { user_id, date_of_birth, event_type, ... }
-reduce_join(user_id, values):
-    user = values[0]
-    event = values[1]
-    payload = { dob: user.date_of_birth, event: event.event_type }
-    emit(user_id, payload)
-
-

Group-by Used to group records by a key, and is used in MapReduce to group the output of the map function before the reduce function.

-
# psuedocode to group events by user_id
-# this is JUST PSEUDOCODE, not actual MapReduce code
-
-# event: { user_id, event_type, ... }
-map_events(event):
-    emit_intermediate(event.user_id, event.event_type)
-
-# group: { user_id, [event_type, ...] }
-reduce_group(user_id, values):
-    emit(user_id, values)
-
-

Distributing a join acorss multiple machines is difficult in the the presence of skew. If one user has many events, then the reducer that processes that user's events will be slower than the other reducers. To avoid skew, several algorithms exist and are implemented in tools like Pig and Hive.

-

Above was an example of a reduce-side join, where the join is performed in the reduce function. An alternative is a map-side join, where the join is performed in the map function. This is only possible if the join is between two datasets that are partitioned in the same way. For example, if the user data and event data are partitioned by user_id, then the join can be performed in the map function.

-

Map-side joins are best when joining a large dataset with a small dataset, because the small dataset can be loaded into memory on each machine. This is called a broadcast join. Particularly, a broadcast hash join is when the small dataset is hashed in the memory of each machine. This is a "replicated join" in Pig, and a "MapJoin" in Hive. You can also use a disk index instead of a hash table for small datasets that woudn't fit in memory.

-
# psuedocode to join event and user data by user_id
-# this is JUST PSEUDOCODE, not actual MapReduce code
-
-# user: { user_id, name, date_of_birth, ... }
-users = load_users()
-
-# event: { user_id, event_type, ... }
-map_events(event):
-    user = users[event.user_id]
-    payload = { dob: user.date_of_birth, event: event.event_type }
-    emit(user_id, payload)
-
-

A partitioned hash join is when you partition your map-side join in such a way that you only need to read a small portion of either dataset into memory. For example, if you partition both datasets by the first digit of the user_id, then you only need to read ~10% of each dataset into memory on any given partition. This requires that each join input is partitioned in the same way. These are known as "bucketed map joins" in Hive.

-
# psuedocode to join event and user data by user_id
-# this is JUST PSEUDOCODE, not actual MapReduce code
-
-# user: { user_id, name, date_of_birth, ... }
-users_partition = load_users_with(ENV.partition_key)
-
-# event: { user_id, event_type, ... }
-map_events(event):
-    user = users_partition[event.user_id]
-    payload = { dob: user.date_of_birth, event: event.event_type }
-    emit(user_id, payload)
-
-

Output of Batch Workflows

-

Search indexes are used to make data searchable. Often, you can use batch processing to build indexes from a datasource. For example, building search indexes for a massive collection of documents would look something like this:

-
    -
  1. Extract the text from each document
  2. -
  3. Tokenize the text into words
  4. -
  5. Remove common words (stop words)
  6. -
  7. Build an index from words to documents
  8. -
-

This can be distributed across multiple machines by partitioning the documents by ID, and then building an in-memory index for each partition. Then, you can merge the in-memory indexes into a single index.

-
# psuedocode to build a search index
-# this is JUST PSEUDOCODE, not actual MapReduce code
-
-# document: { id, text, ... }
-map_document(document):
-    for word in tokenize(document.text):
-        if word not in stop_words:
-            emit_intermediate(word, document.id)
-
-# index: { word, [document_id, ...] }
-reduce_index(word, values):
-    emit(word, values)
-
-

Reccomendation systems are used to reccomend items to users based on their past behavior. For example, you can use batch processing to build a reccomendation system for a feed of posts from other users. To design a system like this, we want reccomendations to be queryable in real time with low latency. Furthermore, the reccomendations should be processed in batches to reduce the load on the database.

-

Instead of using a database client to process data in our batch job, we create an immutable store of our data in a distributed filesystem. Then, we can run a batch job to process the data and write the results to a database. This is called extract-transform-load (ETL).

-
# psuedocode to build a reccomendation index
-# Running on a single machine that doesn't handle user requests
-
-# load a partition of the data into memory (without relying on db client)
-inmem_store_partition = load_data_from_db(ENV.partition_key)
-
-# process data of this partition
-index = build_index_with_map_reduce(inmem_store)
-
-# write the patition's index to the filesystem
-write_partition_index_to_fs(ENV.partition_key, index)
-
-
# psuedocode to query a reccomendation index
-
-def query(user_id):
-    result = offload_query_to_partition(user_id)
-    return result
-
-
-
Tags: batch processing, data analysis, distributed filesystems, etl, mapreduce
-
- - \ No newline at end of file diff --git a/site/digital-design/combinational-logic.html b/site/digital-design/combinational-logic.html deleted file mode 100644 index c2a4356..0000000 --- a/site/digital-design/combinational-logic.html +++ /dev/null @@ -1,324 +0,0 @@ - - - - - - Combinational Logic - - - - - -
- -

Combinational Logic

-
- Last modified: 2024-03-26 - -
-
-

Combinational Logic

-

Combinational Logic vs. Sequential Logic

- -

Representation

-

Can represent logic with text, circuits, truth tables, or equations.

-

For example:

-

Door is ajar if driver door is open OR passenger door is open.

-

plaintext -DoorAjar = DriverDoorOpen OR PassengerDoorOpen

-
Seat belt light is on if driver seat belt is not fastened OR passenger seat belt is not fastened AND the passenger is present.
-
-```plaintext```
-SeatBeltLight = (NOT DriverSeatBeltFastened) OR (NOT PassengerSeatBeltFastened AND PassengerPresent)
-
-

Translating Truth Table to Boolean Expressions

- -

Boolean Identities

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IdentityDescription
A + 0 = AIdentity
A + A = AIdempotent
A + 1 = 1Identity
A + A' = 1Complement
A + B = B + ACommutative
A + (B + C) = (A + B) + CAssociative
A(B + C) = AB + ACDistributive
A + AB = AAbsorption
-

Logic Minimalization

-

It is nice to reduce complexity at the gate level. This allows us to build smaller and faster hardware. We care about...

- -

Generally, simpler boolean expressions lead to smaller transistor networks, and smaller circuit delays/faster hardware.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
TypeCMOS required
NOT2
AND6
OR6
NAND4
NOR4
XOR8
XNOR8
-

You can recreate EVERY gate type using just NAND and NOR (universal gates). e.g. AND = NAND(NAND(A, B))

-

DeMorgan's Law

-
NOT(A AND B) = NOT(A) OR NOT(B)
-NOT(A OR B) = NOT(A) AND NOT(B)
-
-

$$ -\overline{A \cdot B} = \overline{A} + \overline{B} -$$

-

$$ -\overline{A + B} = \overline{A} \cdot \overline{B} -$$

-

In a circuit, the more general rule is that if you have an AND or OR gate with some inverted terminals, to apply demorgans law, you just change the type of gate (ie AND to OR or OR to AND) and invert the inputs (ie change all the points that are inverted to not inverted and vice versa).

-
- -
- - \ No newline at end of file diff --git a/site/digital-design/karnaugh-maps.html b/site/digital-design/karnaugh-maps.html deleted file mode 100644 index d95a859..0000000 --- a/site/digital-design/karnaugh-maps.html +++ /dev/null @@ -1,219 +0,0 @@ - - - - - - Karnaugh Maps - - - - - -
- -

Karnaugh Maps

-
- Last modified: 2024-04-09 - -
-
-

Karnaugh Maps

-

Goal: Find neighboring subsets of the On set to eliminate variables and simplify expressions.

-

A K-map is a method of representing a truth table to help visualize adjacencies into $\le$ 4 dimensions.

-
    -
  1. Split inputs into 2 evenly sized groups
  2. -
  3. Draw a grid with the 2 groups as the axes, yielding $2^n$ cells.
  4. -
  5. Number cells based on truth table
  6. -
  7. Group 1s in powers of two (can be in multiple dimensions, and also wraps around the map).
  8. -
  9. Left over 1s are corner cases and should be grouped with any adjacent 1s if possible.
  10. -
-

7 Segment Display in Verilog

-

Procedural Blocks

- -
- -
- - \ No newline at end of file diff --git a/site/digital-design/quartus-workflow.html b/site/digital-design/quartus-workflow.html deleted file mode 100644 index 1f03899..0000000 --- a/site/digital-design/quartus-workflow.html +++ /dev/null @@ -1,205 +0,0 @@ - - - - - - Quartus Workflow - - - - - -
- -

Quartus Workflow

-
- Last modified: 2024-04-25 - -
-
- -
- -
- - \ No newline at end of file diff --git a/site/digital-design/sequential-logic.html b/site/digital-design/sequential-logic.html deleted file mode 100644 index a08d378..0000000 --- a/site/digital-design/sequential-logic.html +++ /dev/null @@ -1,221 +0,0 @@ - - - - - - Sequential Logic - - - - - -
- -

Sequential Logic

-
- Last modified: 2024-05-14 - -
-
-

Sequential Logic (SL)

-

Whereas in combinational logic, you have outputs which are direct functions of their inputs, with sequential logic the presence of feedback gives circuits the ability to store state. This is the basis for memory and computation in digital systems.

-

This helps control the flow of information through blocks of combinational logic, usually synchronizing with a clock signal. One of the major use cases of sequential logic is in Finite State Machines (FSM). Without SL, the output of a combinational circuit would change instantly with every change in input, which can lead to unpredictable behavior within intermediate states, leading to unexpected outputs.

-

Positive Edge-Triggered D-type Flip-Flop

-

On the rising edge of the clock signal, input is sampled and transferred to the output signal. At all other times, changes in the input are ignored and the previously samples value is retained. This essentially has the effect of synchronizing the input signal with the clock signal, or rather quantizing changes in the input signal to only fall on the rising edge of the clock signal.

-

Registers

-

A n bit register is composed of n flip-flops. Registers have the addition of a reset signal, which sets the register to a known state if it is high during a clock trigger.

-

Flip-Flop Timing

- -

Let $t_{input, i}$ be the time it takes for the input of a register to change for the $i$-th time in a single clock cycle, measured from the clock signal. Then we need...

-

$$ -t_{hold} \le t_{input, i} \le t_{period} - t_{setup} -$$

-

Minimum Delay

-

If the shortest path to a register input is too short, then $t_{hold}$ could be violated, meaning the input could change before the state is "locked in". We have...

- -

Maximum Clock Frequency

-

The maximum frequency you can run your clock at is limited by the amount of time needed to get a correct next state to your registers. We must have...

-

max_delay = max(clock_to_q + max_cl_delay, max_cl_delay)

-

Then, min_period = max_delay + t_setup, and max_freq = 1/min_period.

-
- -
- - \ No newline at end of file diff --git a/site/digital-design/system-verilog.html b/site/digital-design/system-verilog.html deleted file mode 100644 index f934a97..0000000 --- a/site/digital-design/system-verilog.html +++ /dev/null @@ -1,285 +0,0 @@ - - - - - - System Verilog - - - - - -
- -

System Verilog

-
- Last modified: 2024-04-02 - -
-
-

SystemVerilog

-

Verilog

-

A programming language for describing hardware. Lets you programmatically describe the behavior you want, giving you the ability to test things beforehand with simulation. Help prevents hurting your hardware.

-

Syntax can be similar to C, but behavior is different. SystemVerilog is a superset of the older Verilog. Will refer to SV as Verilog.

-

Primitives

-

Nets (wire): transmit value of a connected source.

- -

Variables (reg): variable Voltage source.

- -

Values

- -

Modules

- -

Execution

-

You can't turn wires off. They transmit voltages near instantly. Gates and modules are constantly performing computation, which can be hard to keep track of.

-

In pure hardware, there is no notion of initialization. Wires can naturally pick up voltages from their enviornment.

-

Structural Verilog

-
// SystemVerilog code for AND-OR-INVERT circuit
-module AOI (F, A, B, C, D);
-    output logic F;                     // each variable
-    input logic A, B, C, D;             // is 1-bit (logic)
-
-    assign F = ~((A & B) | (C & D));    // continuous assignment
-endmodule
-// end of SystemVerilog code
-
-

Equivalently with wires

-
// SystemVerilog code for AND-OR-INVERT circuit
-module AOI (F, A, B, C, D);
-    output logic F;
-    input logic A, B, C, D;
-    logic AB, CD, O; // now necessary
-
-    assign AB = A & B;
-    assign CD = C & D;
-    assign O = AB | CD;
-    assign F = ~O;
-endmodule
-
-

Equivalently with gates

-
// SystemVerilog code for AND-OR-INVERT circuit
-module AOI (F, A, B, C, D);
-    output logic F;
-    input logic A, B, C, D;
-    logic AB, CD, O; // now necessary
-
-    // and is the module name. a1 is the instance name
-    // AB, A, B are port connections
-    and a1(AB, A, B);
-    and a2(CD, C, D);
-    or o1(O, AB, CD);
-    not n1(F, O);
-endmodule
-
-

2-input MUX

-
// SystemVerilog code for AND-OR-INVERT circuit
-module AOI (F, A, B, C, D);
-    output logic F;
-    input logic A, B, C, D;
-
-    assign F = ~((A & B)|(C & D));
-endmodule
-
-
// 2:1 multiplexer built on top of AOI module
-module MUX2 (V, SEL, I, J);
-    output logic V;
-    input logic SEL, I, J;
-    logic SELN, VN;
-
-    not G1 (SELN, SEL);
-    // order of ports matter. this is explicit
-    // port assignment
-    AOI G2 (.F(VN), .A(I), .B(SEL), .C(SELN), .D(J));
-    not G3 (V, VN);
-endmodule
-
-
- -
- - \ No newline at end of file diff --git a/site/digital-design/waveform-diagram.html b/site/digital-design/waveform-diagram.html deleted file mode 100644 index e3a96b3..0000000 --- a/site/digital-design/waveform-diagram.html +++ /dev/null @@ -1,234 +0,0 @@ - - - - - - Waveform Diagram - - - - - -
- -

Waveform Diagram

-
- Last modified: 2024-04-02 - -
-
-

Waveform Diagrams

-

Group bits of values into a bus or a bit vector. You can view the state of your system as slices of a waveform for each bit, corresponding to a number.

-

Circuit Timing Behavior

-

Every gate has some fixed delay. In reality, you can look them up in their data sheet. However, for simplicity assume delay of all gates is 1 ns ( = 3 ticks).

-

Verilog stuff

-

Verilog bus

-

Defining them: [n-1:0] is an $n$-bit bus. Access with array syntax. Can access sub-bus using bus[start:size].

-

Multi-bit constants

-

n'b#...# is a constant with width $n$.

-

Concat

-

{A, B, C, ...}

-
// SystemVerilog code for AND-OR-INVERT circuit
-module AOI (F, A, B, C, D);
-    output logic F;
-    input logic A, B, C, D;
-    logic [2:0] w; // necessary
-    assign w[0] = A & B;
-    assign w[1] = C & D;
-    assign w[2] = w[0] | w[1];
-    assign F = ~w[2];
-endmodule
-
-

Test Benches

-

Create emulated inputs for all of the FPGA's physical connections.

-
module MUX2_tb ();
-    logic SEL, I, J; // simulated inputs
-    logic V; // net for reading output
-
-    // instance of module we want to test ("device under test")
-    MUX2 dut (.V(V), .SEL(SEL), .I(I), .J(J));
-
-    initial // build stimulus (test vectors)
-    begin // start of "block" of code
-      {SEL, I, J} = 3'b100; #10; // t=0: S=1, I=0, J=0 -> V=0
-      I = 1; #10; // t=10: S=1, I=1, J=0 -> V=1
-      SEL = 0; #10; // t=20: S=0, I=1, J=0 -> V=0
-      J = 1; #10; // t=30: S=0, I=1, J=1 -> V=1
-      end // end of "block" of code
-endmodule // MUX2_tb
-
-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/RPC.html b/site/distributed-systems/RPC.html deleted file mode 100644 index b920e2f..0000000 --- a/site/distributed-systems/RPC.html +++ /dev/null @@ -1,336 +0,0 @@ - - - - - - Rpc - - - - - -
- -

Rpc

-
- Last modified: 2024-03-27 - -
-
-

Look Into

- -

Remote Procedure Call (RPC)

-

A request from a client to execute a function on a server/different machine. -- To the client, looks like a local function call. -- To the server, looks like an implementation of a function call.

-

Google handles $10^{10}$ RPCs per second.

-

Local Procedure Call

- -

The compiler defines the protocol for the call above.

-

Remote Procedure Call

- -

Client/server implementation is usually auto-generated from procedure spec, e.g. Google's Protocol Buffers/Protobuf.

-

RPC vs. Local Procedure Call

-

Binding

- -

Service Discovery Service

- -

Interface Description Language (IDL, e.g. Protobuf)

-

Serialization is important!

- -

Failures

- -

Some of the network issues can be mitigated by TCP, but sockets can fail and messages aren't always transmitted over TCP anyways.

-

Fault Model

- -

Naive RPC

- -

Client timeout and retry

-

If a request or reply message is dropped, the client will wait forever for the response. This can be fixed with client timer/retransmission, where the client sends the request again if it doesn't get a response in a certain amount of time. This leads to duplication and reordering of messages at the server.

-

We can handle this with a unique request ID. Include a message ID in each request/reply. When the client retransmits, it uses the same message ID. The server can then ignore duplicate requests.

-

RPC Semantics

- -

At least once

-

Client should do a finite number of retries, eventually giving up and returning an error to the caller.

-

This only works if the server is idempotent, meaning it has the same effect if it's executed multiple times. All read-only operations are idempotent, but not all write operations are. For example, icrementing a counter is not idempotent, but setting the counter to a value is.

-

Does TCP handle this? Not really despite being reliable. Most RPCs are sent over TCP, and it guarantees in-order delivery with retransmission and duplicate detection. However, it doesn't guarantee exactly-once semantics. If the server crashes after processing the request but before sending the response, the client will retransmit the request, and the server will execute it again.

-

End to end principle: Functionality should be implemented where it can be completely handled, rather than partially handled at each layer. This decreases the chance of partially completed work due to unrelated failures.

-

Examples: -| Example | Explanation | -|---|---| -| DNS lookup | Queries are read-only, so it's idempotent | -| MapReduce | The Map phase is idempotent since it is a pure function | -| NFS | If the client maintains offset, reading/writing a block is idempotent |

-

Importantly, in situations with multiple clients, operations like Put(k, v) are not idempotent, since the value of k can change between the time the client reads the value and the time it writes the value.

-

Two Generals Problem

-

Just a thought experiment to emphasize the difficulty of message passing in a distributed system. Two generals are trying to coordinate an attack on a city. They are separated by a valley, and can only communicate by messenger. The messenger can be captured by the city, and the generals don't know if the message was delivered. The generals need to agree on a time to attack, but they can't be sure the message was delivered. They can only attack if they both agree on the time.

-

The problem boils down to the fact that at any point in time, if we sent a message, we don't know if it was delivered. Regardless of how many round trips you make to confirm, the last message sent could always have been dropped. This is a fundamental and central problem in distributed systems.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/bigtable.html b/site/distributed-systems/bigtable.html deleted file mode 100644 index d232041..0000000 --- a/site/distributed-systems/bigtable.html +++ /dev/null @@ -1,305 +0,0 @@ - - - - - - Bigtable - - - - - -
- -

Bigtable

-
- Last modified: 2024-05-17 - -
-
-

Bigtable: A Distributed Storage System for Structured Data

-

Bigtable Paper

-

Introduction

-

Bigtable is a widely applicable, scalable, highly performant, and highly available database that is used by many of Google's services including Personalized Search, Google Earth, Google Analytics, and more (as of 2006). It is able to handle various workloads, ranging from throughput-oriented batch processing jobs, to latency sensitive end user applications, and it is able to scale up to petabytes of data across thousands of commodity machines.

-

Bigtable supports a simple data model that supports dynamic control over data layout and format, allowing clients to reason about the locality of their data in the underlying storage. It indexes data by row and column names which are arbitrary strings, and data is stored as uninterpreted strings. Clients are able to control the locality and storage location (in memory or from disk) via their schema design.

-

Data Model

-

Under the hood, Bigtable is a sparse, distributed, persistent multi-dimensional sorted map. The map is indexed by a row key, column key, and a timestamp; each cell in the map is indexed by these three keys. The row key is a string, the column key is a pair of strings (column family and column qualifier), and the timestamp is a 64-bit integer. The value of the cell is an uninterpreted array of bytes.

-

(row:string, column:string, time:int64) -> string

-

Rows

-

Row keys are arbitrary strings up to 64KB in size. Every read/write of a single row key is atomic, making it easier to reason about concurrent updates to the same row.

-

Bigtable maintains data in lexicographic order by row key. The row range for a table is dynamically partitioned, each range being called a tablet, which is the unit of distribution/load balancing. This makes reads over short row ranges efficient, usually only requiring communication with a small number of machines to complete. This property can and should be exploited by users to make their regular access patterns more efficient. For instance, storing webpage content indexed via reversed domain groups all shared subdomains closer together, making host and domain analyses more efficient.

-

Column Families

-

Column keys are grouped into sets called column families. All data stored in the same family is usually of the same type, although this isn't constrained by the system. Whereas tables can have an unbounded number of columns, the number of column families should remain relatively small (in the hundreds at most).

-

Column keys are defined as <family>:<qualifier>, both of which are strings, although family names must be printable.

-

Timestamps

-

Each cell maintains multiple versions of your data indexed by timestamp. You can either assign timestamps yourself, or let Bigtable do it at execution time, in which case they represent "real time" in microseconds. Versions are stored in decreasing order, such that you always have the most recent version as the highest locality version.

-

You can also specify garbage collection conditions, like only keeping the last n versions of a cell, or only keeping versions within a certain time range. Furthermore, you can intersect, union, and nest garbage collection rulesets.

-

Extension: it would be nice if you could define rules for keeping progressively sparser "snapshots" as records get older, similar to an LSM tree.

-

API

-

Applications can interact with Bigtable through a wide variety of client libraries, but the base API provides functions for...

- -

Building Blocks

-

Bigtable uses Google's distributed filesystem, GFS, for storing log and data files. The SSTable file format is used for storing data on disk, which provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. You can look up a specific key, as well as iterate over all kv pairs in a specified key range. Internally, each SSTable contains a sequence of blocks (typically 64KB in size, but configurable), with indices stored at the end of the table and loaded into memory when the SSTable is opened. After loading the index, lookups can be done with a single disk seek, or the table can optionally be loaded entirely into memory to avoid needing to go to disk at all. First, the block containing the key is found by binary search, then the block is read and scanned linearly for the key.

-

Additionally, Bigtable uses Chubby, Google's distributed lock service, which is running a 5 active node paxos group under the hood, one of which is leader and serves requests. Chubby provides a simple API for creating and managing locks, and for storing small files. It uses Chubby for a wide variety of tasks, including...

- -

Implementation

-

Made up of three major components:

- -

Most clients rarely ever communicate with the master since their data is accessed through tablet servers, so it is under relatively small load, and doesn't become a bottleneck.

-

A Bigtable cluster stores a number of tables, each consisting of a set of tablets, each tablet containing all the data associated with a row range. Initially, each table is just one tablet, but it grows, automatically splitting and load balancing to each tablet being ~100-200 MB by default.

-

Tablet Location

-

Bigtable uses a three-level hierarchy similar to a B+ tree to store tablet location information. The root tablet is the first level, and its location is stored in a Chubby file. The root tablet stores the location of all tablets in a special METADATA table, and each METADATA tablet contains the location of a set of user tablets. The root tablet is really just the first tablet in the METADATA table, and it is treated specially to never be split so there are only ever three levels of indirection.

-

The next level of indirection in the hierarchy is the rest of the METADATA table, which stores the location of an end tablet under a row key which encodes the tablet's table identifier and end row.

-

This scheme is sufficient to store $2^{34}$ tablets, or $2^{61}$ bytes in 128 MB tablets.

-

The client library caches tablet locations, and on cache miss recursively moves up the tablet location hierarchy searching for said tablet. If the cache is empty, the location algorithm requires three network round trips, including a read to Chubby. Additionally, the client library prefetches entries from the METADATA table to try and reduce the number of cache misses.

-

Finally, the METADATA table contains secondary information, including a log of all events pertaining to each tablet.

-

Tablet Assignment

-

Each tablet is assigned to a single tablet server at a time, and the master manages the assignment of tablets to servers. When there is an unassigned tablet and a tablet server with adequate room, the master assigns the tablet by sending a tablet load request to the server.

-

Bigtable uses Chubby to keep track of tablet servers. On tablet server startup it creates and acquires an exclusive lock on a uniquely named file in Chubby in the servers directory. The master monitors this directory to discover new tablet servers. Tablet servers use Chubby sessions, and continue to function until the file no longer exists, at which point they kill themselves. When tablet servers are manually removed in reconfiguration, the server will attempt to release its lock gracefully so the master reassigns its tablets more quickly.

-

The master periodically polls the tablet servers' lock status, and if it detects any issues it reassigns the offending server's tablets. At this point, the master also checks Chubby to see if there are any issues apart from the tablet server itself, and if not then the master deletes the tablet server's file, effectively dooming it.

-

When the master is started by the cluster management system, it does the following:

-
    -
  1. Acquires the unique master lock in Chubby, which prevents multiple master instantiations
  2. -
  3. Scans the servers directory in Chubby
  4. -
  5. Communicates with every live tablet server to discover the current tablet assignments
  6. -
  7. Scans the METADATA table and adds any not previously learned tablets to the unassigned set
  8. -
-

The set of tablets only ever changes when one is created or deleted, or when they are merged or split. The leader does all but merges, and merges are handled by the tablet server writing directly to the METADATA table.

-

Also, the master kills itself if its session with Chubby ever expires, but this doesn't change the assignment of tablets to tablet servers.

-

Tablet Serving

-

The persistent state of each tablet is stored in GFS. Updates are stored in a commit log, with the most recently committed updates being stored in-memory in a memtable, and older updates being stored in a sequence of SSTables.

-

On a read operation, the authorization is checked via a Chubby file, and then the SSTables for the tablet are scanned and merged to form the result of the read. Write operations are first checked for authorization, and then get added to the commit log using group commit, after which the contents are inserted into the memtable.

-

Compactions

-

The memtable grows on each write operation until a certain threshold, at which point the memtable is frozen and converted to an SSTable and written to GFS, being replaced by a new memtable. This minor compaction process both shrinks memory usage of tablet servers, and reduces the amount of data needed to be read from the commit log during recovery.

-

Periodically, major compaction takes place, where all SSTables are merged into a single SSTable containing no deleted data.

-

Refinements

-

Locality Groups

-

Clients can group multiple column families into a locality group, each of which get their own SSTable. Tuning characteristics like loading SSTables into memory can be specified for locality groups, which is a feature used on the METADATA table.

-

Compression

-

Clients can control whether or not, and if so how SSTables for locality groups are compressed. Compression schemes are applied to each SSTable block. Many clients use a two pass approach, which first uses Bentley and McIlroy's scheme, compressing long common strings across a large window. In the second pass, repetitions in a small window (16 KB) are searched for and compressed. This compression scheme prioritizes speed over size, although it does well at both metrics.

-

Caching for read performance

-

Tablet servers use two levels of caching:

- -

Bloom filters

-

Clients can specify that SSTables be created with a bloom filter in the tablet server's memory, reducing the number of disk accesses required for read operations by preventing many lookups for non-existent rows

-

Commit-log implementation

-

The commit logs for different tablets are all actually a single commit log on the tablet server, preventing us from needing to concurrently write to many different files in GFS.

-

On recovery, instead of reading the entire log to find only the tablets assigned to you, the log is sorted by keys (table, row name, log sequence number). This sorting is parallelized by partitioning the log file into 64 MB segments which are each sorted in parallel on different tablet servers, being coordinated by the master.

-

To mitigate GFS writing latency spikes, two separate threads are maintained, writing to two different files. If one is performing badly, the other one starts, but only one writes at a time.

-

Speeding up tablet recovery

-

Before the master moves a tablet, the source tablet server does an initial minor compaction on the tablet, after which it stops serving the tablet, performing one more minor compaction to eliminate any state in the log that came in after the first compaction, and then the tablet is unloaded on this server and loaded onto the other server.

-

Exploiting immutability

-

Since SSTables are immutable, no synchronization needs to be done when reading from SSTables, and concurrent row accesses can be implemented efficiently. The only mutable concurrently accessed data structure is the memtable, which is optimized with row copy-on-write, allowing parallel reads and writes.

-

To permanently remove deleted data, the table is mark and sweep garbage collected.

-

Additionally, since SSTables are immutable, when splitting tablets the children can continue to rely on the parent's SSTable, not needing to create two new SSTables.

-

Performance Evaluation

-

Although scaling relatively well, it is not perfectly linear with the number of servers in the cluster. In particular, for random reads from disk, increasing the number of servers from 1 to 500 only increased the throughput by a factor of ~100. For random reads in particular, transferring 64KB blocks over the network for every read ends up saturating the network link, becoming a bottleneck. Random reads from memory on the other had saw a ~300 times increase in throughput.

-

Real Applications

-

Google Analytics

-

Google Analytics (analytics.google.com) is a service that helps analyze traffic patterns to websites. To enable the service, a small JavaScript program is embedded in a web page, which is invoked whenever the page is visited. It records various information like a user identifier, information about the page, etc., and the data is made available in Google Analytics to the website owner.

-

Two of the tables stored in Bigtable used by this service are the raw click table, which maintains a row for each end-user session, and the summary table, containing various predefined summaries for each website. The summary table is periodically computed via a Map-Reduce job over the raw click table. The click table's schema is designed so that sessions that visit the same website are contiguous and sorted chronologically, and the table is able to be compressed to 14% its original size (~200 TB). The summary table is able to be compressed to 29% of its original size (~20 TB).

-

Google Earth

-

The data used by both Google Maps (maps.google.com) and Google Earth (earth.google.com) are partially stored in Bigtable. The system uses one table to preprocess data, and another set of tables for serving client data.

-

The preprocessing pipeline uses one table to store raw image data (with compression turned off since it is handled manually). During preprocessing, the images are cleaned and consolidated into the final serving data. Each row in the preprocessing table corresponds to a single geographic segment, and rows are named so that adjacent geographic segments are stored near eachother. This preprocessing pipeline relies heavily on MapReduce over Bigtable.

-

The serving system uses a single table to index data stored in GFS. Although its relatively small (500 GB), it serves tens of thousands of queries per second, so it is hosted on hundreds of tablet servers to load balance, each containing in-memory column families.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/clocks.html b/site/distributed-systems/clocks.html deleted file mode 100644 index ed98c99..0000000 --- a/site/distributed-systems/clocks.html +++ /dev/null @@ -1,292 +0,0 @@ - - - - - - Clocks - - - - - -
- -

Clocks

-
- Last modified: 2024-04-14 - -
-
-

Clocks

-

There are two main approaches to time in a distributed system: physical clocks and virtual (logical) clocks.

-

Physical Clocks

-

Actual clocks running in most computers drift apart by ~30 ppm due to their temperature sensitivity. Although more accurate clocks (atomic, GPS, etc.) are available, they are expensive and are only maybe present in some data centers.

-

The crux of the problem is that physical clocks are not perfectly synchronized, and the sending of messages between processes can introduce unpredictable delays. In general, network latency is unpredictable, but with a lower bound.

-

A practical, albeit naive approach might be to use NTP and have clients query a set of time servers. Then, take the minimum (or some average, subtracting outliers) of the readings received. This can synchronize to ~50 microseconds in a LAN.

-

Such a system was implemented with Google Huygens. Some interesting optimizations they added include:

- -

This enabled them to achieve a 50 ns clock skew 99% of the time. This is okay if time is only used as a hint, but shows that even with all of the above optimizations, it isn't good enough.

-

To drive this point home, due to the massive scale Google operates at (1 billion RPCs/sec = 10 million clock skews above 50ns per sec), even for a minimum sized message that takes 2 ns to send in a high-performance network, thousands of instructions can be executed on a single server's processor.

-

Virtual Clocks

-

We want to design systems such that the ordering of events that can be concurrently executed doesn't matter, and the ordering of events that must be performed sequentially is enforced on all possible executions.

-

Virtual clocks are a framework for reasoning about the order of events using no assumptions about physical clock skew or message delays in way way that both respects causality, and relies only on local information.

-

Happens before

-

We say that event a happens before event b if:

-
    -
  1. a happens earlier than b in the same process
  2. -
  3. a is the sending of a message and b is the receipt of that message
  4. -
  5. a happens before c and c happens before b, aka transitivity
  6. -
-

This is a partial order.

-

Happens concurrently

-

Two events a and b are said to happen concurrently if neither a happens before b nor b happens before a.

-

Logical Clock Implementation

- -

Vector Clocks

-

Note that with the above implementation of a logic clock system, it was not the case that $T(a) < T(b) \to $ $a$ happened before $b$. With vector clocks, we have $T(a) < T(b) \leftrightarrow a$ happened before $b$ by precisely representing transitive causal relationships between events. This is used in practice for eventual and causal consistency (ie Git, Amazon Dynamo, etc.).

-

Algorithm

-

Clock is a vector C, with length = # of nodes in the system

- -
public class VectorClock {
-  public final int[] clock;
-
-  public VectorClock(int n) {
-    clock = new int[n];
-  }
-
-  public void increment(int i) {
-    clock[i]++;
-  }
-
-  public void handleMessage(VectorClock other) {
-    for (int i = 0; i < clock.length; i++)
-      clock[i] = Math.max(clock[i], other.clock[i]);
-  }
-}
-
-
public class Node {
-  private int id;
-  private VectorClock vc;
-
-  public Node(int id, int n) {
-    this.id = id;
-    this.vc = new VectorClock(n);
-  }
-
-  public void event() {
-    vc.increment(id);
-  }
-
-  public void merge(Node other) {
-    vc.handleMessage(other.vc);
-  }
-
-  public void send(Node other) {
-    other.vc.handleMessage(vc.clock);
-  }
-
-  public boolean[] didHappenBefore(Node other) {
-    boolean[] res = new boolean[2];
-    res[0] = true;
-    res[1] = true;
-
-    for (int i = 0; i < vc.clock.length; i++) {
-      if (vc.clock[i] > other.vc.clock[i])
-        res[0] = false;
-       else if (vc.clock[i] < other.vc.clock[i])
-        res[1] = false;
-    }
-
-    return res;
-  }
-}
-
-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/consistency.html b/site/distributed-systems/consistency.html deleted file mode 100644 index e69e5d9..0000000 --- a/site/distributed-systems/consistency.html +++ /dev/null @@ -1,322 +0,0 @@ - - - - - - Consistency - - - - - -
- -

Consistency

-
- Last modified: 2024-05-12 - -
-
-

Consistency

-

Consistency: the allowed semantics of operations that mutate a data store/shared object.

-

Consistency specifies the interface (as opposed to implementation) for behavior of your system. It is essentially the contract between the programmer and implementer. An anomaly is a violation of the consistency semantics of the system

-

Types of Consistency

- - - - - - - - - - - - - - - - - - - - - -
TypeDescription
Strong ConsistencyThe system behaves as if there is a single server. Systems that maintain a single consistent log of operations are often strongly consistent.
Weak ConsistencyDefinitions vary, but basically just not strong consistency.
Eventual ConsistencyWeak consistency with any anomalies guaranteed to be temporary.
-

Coordinating through a KV Store

-
def Produce(key, lock, command):
-  result = application.execute(command)
-  storage.put(key, result)
-  storage.put(lock, True)
-
-def Consume(key, lock):
-  while storage.get(lock) is False:
-    pass
-  return storage.get(key)
-
-

With strong consistency semantics, the above approach works fine. However, with eventual consistency, and particularly for any system without multi-key transactions, we might see the update for storage.get(done) before the update for storage.get(key), leading to unexpected behavior.

-

Formalization

-

Read here for more info/theory.

-

For a given RPC, the initial request starts at time $t$ and the reply returns at time $t + x$. We cannot be sure what happens during $(t, t + x)$, since the request/reply could be lost and retransmitted, and intermediate coordination sometimes has to take place.

-

With only a single server, you don't know precisely when the operation takes place, but we expect it to be some time in $(t, t + x)$. However, weaker consistency models relax this assumption, also sometimes allowing different readers to see different results concurrently.

-

We use different models because of the following tradeoffs:

- -

Lamport's Register Semantics

-

Registers hold a single value, and we define operations $r_i, $w(v)$ as the $i$th read, and a write to the register with value $v$. Each operation has some starting time and ending time.

- - - - - - - - - - - - - - - - - - - - - - -
SemanticsConstraints
safe$r_1 \to v_1$
regular$r_1 \to v_1 \land (r_2 \to v_1 \lor r_2 \to v_2) \land (r_3 \to v_1 \lor r_3 \to v_2)$
atomic$r_1 \to v_1 \land (r_2 \to v_1 \lor r_2 \to v_2) \land (r_3 \to v_1 \lor r_3 \to v_2) \land (r_2 \to v_2 \implies t_3 \to v_2)$
-
            r1           r2     r3
-          |----|       |----| |----|
-   w(v1)                w(v2)
-|------|             |---------|
-
-

Linearizability

-

A linearizable system is one in which actions appear to occur in a single global order that is consistent with real time/causal order. Not all systems enforce linearizability.

-

To do linearizable reads in Paxos, you need to first verify that the leader is still the leader at the time of the read. Otherwise, its possible that some other leader took over and formed a majority without the old leader. This can be done by waiting for the leader to execute some other request, which will only go through if we are indeed still the leader.

-

Linearizable Sharding with Paxos

-

For linearizability with shards, we have the following requirements:

- -

Parallelism/concurrency of batched requests becomes difficult in a sharded system, since breaking up operations of a batched request into a pipeline completely throws out the original order of the request. We can instead think of systems in terms of a weaker consistency model.

-

Sequential Consistency

-

Sequential Consistency is a weaker form of consistency that requires all operations to be executed in some order that is consistent with the order in which they were issued. However, S.C. doesn't always follow real-time order. This is also referred to as serializability in the context of transactions.

-

Simplistically, we can think of sequential consistency as a system where all operations are executed in some order that is consistent with the order in which they were issued, but not necessarily during their window of request/response timing. This allows stale reads, while still maintaining some order that is consistent with a prefix of the global state of the system.

-

Snapshot Reads

-

Gives us a consistent view of our global state across some set of views of the system. This requires all operations being serializable, but it is okay if reads return stale data.

- -

To implement this (without sharding) in conjunction with Paxos, we can do the following:

-
    -
  1. Primary defines update order in log
  2. -
  3. Shadow replicas apply changes in that order
  4. -
  5. Each lag primary from some variable amount
  6. -
  7. Snapshot reads occur at a single replica
  8. -
  9. If a replica crashes during a transaction, restart transaction at another snapshot replica
  10. -
-

Causal Consistency

- -

Processor Consistency

- -

Memory Barrier/Fence

- -

This is how POSIX files work. Also many mutli-cache systems use fences to enforce consistency.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/consistent-global-state.html b/site/distributed-systems/consistent-global-state.html deleted file mode 100644 index 5a70c79..0000000 --- a/site/distributed-systems/consistent-global-state.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Consistent Global State - - - - - -
- -

Consistent Global State

-
- Last modified: 2024-04-13 - -
-
-

Consistent Global State in Distributed Systems

-

reading

-

Introduction

-

Many problems in distributed computing boil down to being able to maintain a consistent global state, and to run predicates on that state in order to trigger events. The true state of a distributed system is the union of all node's states. However, since nodes don't share memory, the actual state must be meaningful when inferred solely based on messages passed among nodes.

-

A global state is said to be inconsistent if it never could have been constructed by an ideal external observer. This paper formalizes this concept into the context of a Global Predicate Evaluation (GPE), which determines if the system satisfies some predicate $\Phi$.

-

Asynchronous Distributed Systems

-

Define a distributed system as a set $P$ of sequential processes $p_1, p_2, \ldots, p_n$, and a network consisting of channels in which unidirectional communication is possible in the space of $P^2$. The network is assumed to be reliable, but may deliver messages out of order, and is taken to be strongly connected, but not necessarily fully connected (i.e. communication may require intermediate message passing).

-

It is useful to reason about distributed systems with the weakest possible assumptions, such that results hold for arbitrary systems.

-

Distributed Computations

-

A distributed computation is the execution of a distributed program over a collection of processes, each of which sequentially process a stream of events. Particularly, for two nodes to communicate, a message $m$ is enqueued on a channel via $send(m)$, and the message is dequeued via $receive(m)$. There is an obvious relationship between the happening of event $send(m)$ at process $p$, and the happening of event $receive(m)$ at process $q$, such that we can be sure $send(m)$ happened before $receive(m)$.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/disconnected-operation.html b/site/distributed-systems/disconnected-operation.html deleted file mode 100644 index a381c33..0000000 --- a/site/distributed-systems/disconnected-operation.html +++ /dev/null @@ -1,229 +0,0 @@ - - - - - - Disconnected Operation - - - - - -
- -

Disconnected Operation

-
- Last modified: 2024-12-08 - -
-
-

Disconnected Operation

-

Always available writes inherently pose a problem in distributed systems. To allow for disconnected operation, we need to be able to write to a local copy of the data, and then synchronize it with the rest of the system when we're back online. Many apps today are built to work with intermittent lack of connectivity, for example, file syncing apps/sourcing control systems. In many of these systems, writes can conflict, and need to be resolved either manually or automatically.

-

Two Models

- -

Coda

-

Coda lets you mount a remote file system as a local directory. The local file system is a partial replica of the global version. It makes extensive use of local caching to reduce latency. While disconnected, it uses a write ahead log to record changes. When reconnected, it replays the log to the server and merges changes atomically.

-

When possible, Coda will merge changes automatically. If there are conflicts, i.e. two users edit the same file, Coda will create a conflict file and let the user resolve it.

-

Gmail/Google Docs

-

Apps like Gmail and Google Docs allow for offline editing by using a local cache and log of changes. When reconnected, the changes are sent to the server and merged. In the case of conflicts, the application specifies a set of rules to resolve them automatically.

-

One common general approach is to use a version vector to track changes. Each client has a unique ID, and each change is tagged with the client's ID. When changes are merged, the version vector is used to determine which changes are newer.

-

Source Code Control

- -

Interesting Application Model

-

Use local storage engines (like SQLite, LevelDB, etc.) for local writes and sync with the server when online. The key to making this strategy work is the synchronization mechanism, and how conflicts are resolved.

-

Conflict Resolution

- -

Merge Strategies

- -
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/distributed-cache-coherence.html b/site/distributed-systems/distributed-cache-coherence.html deleted file mode 100644 index c75c3d7..0000000 --- a/site/distributed-systems/distributed-cache-coherence.html +++ /dev/null @@ -1,281 +0,0 @@ - - - - - - Distributed Cache Coherence - - - - - -
- -

Distributed Cache Coherence

-
- Last modified: 2024-05-17 - -
-
-

Distributed Cache Coherence

-

When linearizability is a concern, any duplication of mutable data across multiple nodes must be kept consistent. This is the problem of cache coherence: ensuring that all nodes in a distributed system have the same view of the data.

-

Distributed Caching with Leases

-

A lease is a time-limited right to do something. In the context of distributed caching, our lease gives us a right to cache some data.

-

If a node holding the lease fails, we just wait for the lease to expire. Leases can be renewed by the holder, so long as the node is still up.

-

Cache Reads

-
    -
  1. Cache obtains a lease containing the data
  2. -
  3. No one can modify the data until the lease either expires or is revoked. Thus, a server/service needs to track who has which data and for how long
  4. -
  5. Once the lease expires, the value can then change. The items is no longer cached by anyone, so it can only be copied at the server. All subsequent caches can refetch the new data.
  6. -
-

This approach is both linearizable and fault tolerant, since the lease will eventually expire if the node holding it fails, allowing another node to take over. However, this approach is not very scalable, since the server needs to maintain state for every cached item.

-

Clients are also able to cache values, and you can do this by forwarding the lease along with the data to the client.

-

Cache Updates

-

Leases allow the server to reclaim a single copy, regardless of whether caches are up or not. A naive approach would be to wait for all copies to timeout any time you want to update.

-

An optimized version would be to use a callback, preventing the need for timeouts if no error occurs. When the server receives an update for a cached value, it forwards an invalidation/revoke to all nodes with a copy of the data, and waits for a response from all (or for the lease to timeout), after which it can proceed with the update.

-

The key insight however is that in order for your system to be linearizable, you must have only one copy of the data while updating.

-

Lease Timeouts

-

If we use the same timeout value for all leases, then we need to track less state at our server, and this also reduces the total time needed to reclaim all leases. On the other hand, if we use different timeouts, then caches will all ask for a new lease at staggered times, preventing an overwhelming number of requests at once.

-

Weaknesses of Linearizable Caches

- -

Caching Widely Shared Data

-

It's often okay to use snapshot read consistency, allowing for reads to return stale data. Much of the web follows this model.

-

Usually this would look like having many read only caches and a single copy that is writable, for which updates are propagated from to the caches.

-

Examples

-

Sun Network File System (NFS)

- -

Domain Name System (DNS)

- -

Caching Terminology

- -

Write Back Cache Coherence

- -

However, with write back caching, durability becomes an issue since a failure might lose writes.

- -
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/dynamo-db.html b/site/distributed-systems/dynamo-db.html deleted file mode 100644 index 672b5c8..0000000 --- a/site/distributed-systems/dynamo-db.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Dynamo Db - - - - - -
- -

Dynamo Db

-
- Last modified: 2024-05-24 - -
-
-

Dynamo: Amazon's Highly Available Key-value Store

-

reading

-

Dynamo is a highly available key-value storage system that sacrifices consistency under certain failure conditions, making extensive use of object versioning and application assisted conflict resolution.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/google-file-system.html b/site/distributed-systems/google-file-system.html deleted file mode 100644 index ec37682..0000000 --- a/site/distributed-systems/google-file-system.html +++ /dev/null @@ -1,289 +0,0 @@ - - - - - - Google File System - - - - - -
- -

Google File System

-
- Last modified: 2024-05-21 - -
-
-

Google File System (GFS)

-

reading

-

Introduction

-

GFS differs from more traditional file systems in a lot of ways. Particularly...

- -

Design Overview

-

Assumptions

- -

Interface

-

Although familiar, GFS is not POSIX compliant. Files are organized hierarchically in directories and identified by path names, supporting create, delete, open, close, read, and write. Additionally, supports snapshot and record append. Snapshot creates a copy of a file or directory tree. Record append allows multiple clients to append data to the same file concurrently while guaranteeing the atomicity of each individual record.

-

Architecture

-

A GFS cluster is composed of a single master, and multiple chunkservers, and is accessed by multiple clients. Each component is typically run on a commodity Linux machine running a user-level server process, and it isn't uncommon to run both a client and chunkserver on the same machine (although this comes at a reliability cost).

-

Files are made of fixed-size chunks, identified by an immutable and globally unique 64 bit chunk handle which is assigned by the master at the time of creation. Chunkservers store chunks as Linux files on local disk, and read/write to specified chunk handle and byte ranges. Each chunk is replicated on multiple chunkservers (by default with three replicas, but is configurable).

-

The master maintains all filesystem metadata, including the namespace, access control information, file to chunk and chunk to server mappings, etc. The master communicates with chunkservers through a HeartBeat, which is a two-way message that allows the master to monitor and instruct the chunkservers and chunkservers to report their status.

-

GFS client libraries are linked in each application. All metadata updates go through the master, but simple data-bearing communication goes directly to the respective chunkserver.There is no need to go through something like the Linux vnode layer, which is a major benefit of not supporting POSIX.

-

Neither clients nor chunkservers cache file data, but clients will typically cache metadata. Technically, chunkservers do cache file data in that they use Linux files, and thus have a buffer cache for frequently accessed data in memory, but this is transparent to GFS.

-

Single Master

-

Using a single master architecture vastly simplifies things, since we can use global knowledge of the filesystem at the master for replication and chunk placement. To help this scale, the master's level of involvement in reads/writes needs to be minimized.

-

Common Case Read

-
    -
  1. Client translates file name and byte offset to chunk index locally
  2. -
  3. Requests chunk handle and replica locations from master
  4. -
  5. Client caches the result with the file name and byte offset
  6. -
  7. Client chooses replica (often the closest) and reads/writes while their cache entry is still valid
  8. -
-

Note that for almost no extra cost, steps 1 and 2 can be batched for many files/chunks.

-

Chunk Size

-

GFS uses 64 MB lazily allocated (to mitigate internal fragmentation) chunks, each of which are stored as a plain Linux file. Using such large chunks has many benefits, including allowing clients to cache plenty of metadata, reducing network overhead/load of metadata acquisition, and allows for in-memory metadata at the master.

-

Large chunk sizes also have their disadvantages, including leading to hotspots for chunkservers with many smaller files. One such example is when you have an executable stored in GFS that you want to concurrently read and execute across many hundreds of machines, this leads to a temporary overload of those chunk servers. This was fixed by using a higher replication factor, and staggering starting times. Extension: peer to peer sharing (between clients) is a possible solution to hotspots.

-

Metadata

-

There are three main types of metadata, all of which is stored in-memory in the master.

- -

The first two are also persisted in an operation log stored on the master's local disk. The last however, is not persisted, and instead on startup/when a new chunkserver joins the cluster, the master requests metadata about the chunks stored locally.

-

In-Memory Data Structures

-

The master stores metadata in memory and goes fast! Additionally, this allows it to quickly scan through its entire state, making things like garbage collection, re-replication on failures, and migration for load balancing possible.

-

"b-b-bbbut that's nOt sCaLabLE" - someone right now probably

-

In reality, if the metadata were a limiting factor for your system, you would have a MASSIVE amount of data, and would need to be using a very under-specced machine as the master. Each 64 MB chunk only requires ~64 bytes of metadata, and namespace metadata is compressed using prefix compression (which is very effective for filesystems). Therefore, adding even an extra few GB of memory to your master would allow you to store a huge amount of additional metadata.

-

Chunk Locations

-

As stated previously, the master doesn't persist the location of chunks, instead opting to poll chunkservers on startup. This design choice simplifies things greatly, and is a general approach to fault tolerance, in that it would have been an uphill battle trying to maintain a globally consistent and persistent view of the system, since chunkservers can partially fail, and they ultimately know best which chunks they have.

-

Operation Log

-

The operation log is not only the sole persistent metadata in the system, but also defines the order in which concurrent operations are executed. The log is replicated remotely, and changes are batched and flushed to disk. The log is replayed on startup, but is also kept small by being checkpointed with a memory-serializable compact B-tree structure, making recovery faster. The master can create a checkpoint in a separate background thread, allowing concurrent operations to be executed. Once created, the checkpoint is written to disk both locally and remotely.

-

Consistency Model

-

Guarantees by GFS

-

File namespace mutations are atomic through being done exclusively at the master with locking. File mutations have looser guarantees, particularly at the level of defined, and undefined regions

- -

After a sequence of successful mutations, the file is guaranteed to be defined and to contain the data written by the last mutation. GFS achieves this by...

- -

Once a chunk becomes stale, it is no longer returned to the client, and is garbage collected ASAP. Since clients cache chunk metadata, there is a window of time in which a client will read from stale chunks, but this typically presents as reading a premature end of chunk (in the case of append workloads).

-

Component failures can lead to corrupted or destroyed data, but this is mitigated by checksumming files. Once a problem is detected, the data is restored from a valid replica if possible. In the (uncommon) case where all replicas are lost before the master can react, the data is unavailable, but corrupted data is never returned.

-

Implications for Applications

-

Long story short, you should always prefer appends over random writes. Note that GFS has "append at least once" semantics, and can also insert arbitrary padding between appends. It is thus important to use techniques like structuring your data and using unique identifiers for non-idempotent log entries.

-

System Interactions

-

Lease and Mutation Order

-

A mutation is any change to the contents of a chunk. Each mutation is performed at all replicas. Mutations are carried out as follows:

-
    -
  1. Master grants a lease (~60 sec) to one of the replicas, making it the primary
  2. -
  3. The primary chooses a serial order for all mutations to the chunk
  4. -
  5. All replicas execute the mutations in the order defined by the primary
  6. -
-

Leases can be extended, and the extensions are piggybacked through HeartBeat messages. The master can also revoke a lease, although if it loses communication with the primary it only needs to wait for the lease to expire.

-

A write can be carried out through the following:

-
    -
  1. The client asks the master which chunkserver holds the current lease, and the locations of other replicas. If no lease is held, the master grants one to a replica of its choice
  2. -
  3. The master replies with the identity of the primary and location of replicas (secondaries), which is cached by the client
  4. -
  5. The client pushes data to all replicas, which is stored by an LRU buffer cache until the data is used or expires
  6. -
  7. Once all replicas have acknowledged receiving the data, the client sends a write request to the primary, which identifies the data that the client just pushed. The primary assigns consecutive sequence numbers to all mutations it receives (possibly from other clients as well), and then applies the mutations locally
  8. -
  9. The primary forwards the write request to all secondary replica, and the replicas apply the mutations in the same order defined by the primary.
  10. -
  11. The secondaries ack that they completed the operation
  12. -
  13. The primary replies to the client, and any errors are reported to the client. Errors leave the region in an inconsistent state, but the failed mutation is usually retried multiple times, until eventually falling back to redoing the entire write.
  14. -
-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/load-balancing.html b/site/distributed-systems/load-balancing.html deleted file mode 100644 index e80eb48..0000000 --- a/site/distributed-systems/load-balancing.html +++ /dev/null @@ -1,227 +0,0 @@ - - - - - - Load Balancing - - - - - -
- -

Load Balancing

-
- Last modified: 2024-05-06 - -
-
-

Load Balancing

-

In a load balancing systems, we want the following:

-
    -
  1. Clients all follow the same assignment
  2. -
  3. Load is evenly distributed
  4. -
  5. Adding/removing only moves a few keys
  6. -
  7. Tail latency is minimized
  8. -
  9. Redistributing keys should not overload a single server
  10. -
  11. Workload should be evenly distributed despite key popularity differences
  12. -
-

Scaling Paxos with Sharding

-

Use Paxos to define the order of a state machine running on a set of servers. For a key value store, we can split the key space into multiple shards, assigning some set of keys to a given shard. The Paxos group that performs this is known as the shard master. Then, each shard is a Paxos group that runs the state machine for its subset of keys.

-

This has the advantage of spreading load across multiple servers, as well as distributing the data.

-

Edge Caching

-

Many things should be cached locally to the users machine. However, for content that is not user specific (like logo.png), we can cache it on a single server and redirect all requests for that content to that specific server, effectively load balancing the requests while condensing the cache to use less memory across all servers.

-

Queueing

-

Assuming completely random (Poisson) arrivals and service times, the average number of requests in the system is given by:

-

$$ -R = \frac{S}{1 - U} -$$

-

Where $R$ is the response time, $S$ is the service time, and $U$ is the utilization of the server. This formula is derived from the M/M/1 queueing model.

-

The variance of the response time is $\propto \frac{S}{1 - U}$, so as the server utilization approaches 1, the variance of the response time approaches infinity.

-

In practice, load is bursty and services need to be overprovisioned to handle the spikes in load. This is why the variance of the response time is so important, since tail latencies can be very high if the server is overloaded.

-

The system can be modeled as a Markov chain with states $0, 1, 2, \ldots, n$. The state $i$ represents the system with $i$ requests in the system. The transition rate from state $i$ to state $i+1$ is $\lambda$ and the transition rate from state $i$ to state $i-1$ is $\mu$.

-

Key Popularity

-

The Zipf distribution says that the $k$th most popular item follows some curve $\frac{1}{k^c}$, where $1 \le c \le 2$. This is said to...sort of apply to many things

- -

We can cope with popular keys using power of two choices. Keys can be hashed to multiple (in this case two, but generalizes to $k$) servers, and requests are forwarded to whichever server is under less load.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/managing-critical-state.html b/site/distributed-systems/managing-critical-state.html deleted file mode 100644 index d1084a9..0000000 --- a/site/distributed-systems/managing-critical-state.html +++ /dev/null @@ -1,258 +0,0 @@ - - - - - - Managing Critical State - - - - - -
- -

Managing Critical State

-
- Last modified: 2024-04-04 - -
-
-

Managing Critical State

-

reading

-

CAP Theorem

-

The CAP theorem states that a distributed system can only guarantee two of the following three properties: -- Consistency: All nodes see the same data at the same time. -- Availability: Every request receives a response, without guarantee that it contains the most recent write. -- Partition tolerance: The system continues to operate despite network partitions.

-

Essentially, if some nodes go down in a distrubuted system, you can either choose to continue serving requests (availability), or to stop and wait for the nodes to come back up (consistency).

-

ACID's Alternative: BASE

-

While ACID (Atomicity, Consistency, Isolation, Durability) lists properties that provide semantics for consistent transactions on a single node, this doesn't translate or scale well to distributed systems.

-

Instead, some datastores use BASE (Basically Available, Soft state, Eventually consistent) semantics, allowing for more flexibility in the face of network partitions. Most systems that support BASE semantics use multi-leader replication, where each leader can accept writes and propagate them to other leaders. This allows for better availability and partition tolerance, but at the cost of complexity to deal with eventual consistency in application code.

-

Motivating the Use of Consensus: Distributed Systems Coordination Failures

-

The Split Brain Problem

-

Split brain is where multiple nodes think they are the leader. A naiive approach to solving this is to use a heartbeat mechanism, where a leader sends out a heartbeat to followers. If multiple nodes think they are the leader, they will both send out heartbeats, and eventually one of the "leaders" will realize that there is another leader and issue a STONITH (Shoot The Other Node In The Head) command to kill the other leader.

-

However, due to the asynchronous and unreliable nature of networks, it is possible for these messages to be delayed to a point where both nodes issue a STONITH command to each other, causing both nodes to go down. Furthermore, the issue of actually detecting and avoiding a split brain is non-trivial, since it is difficult to distinguish between a network partition and a node failure, and nodes can be partitioned from eachother arbitrarily.

-

Faulty Group Membership Algorithms

-

Using gossip protocols to maintain group memberships of clusters can lead to issues. Specifically, partitions within a cluster lead to multiple leaders being elected in the same cluster, often leading to data loss or corruption.

-

How Distributed Consensus Works

-

In distributed software systems, we care about asynchronous distributed consensus, where nodes can fail and messages can be delayed, lost, or duplicated arbitrarily. Technically, this is impossible to solve in bounded time (see Dijkstra's FLP result), but we can solve problems by ensuring the system has sufficient healthy replicas and network connectivity, allowing the system to make progress most of the time. Futher, exponential backoff can be used to prevent cascading failures.

-

Some characteristics of distributed consensus algorithms include: -- crash-fail: nodes that fail never re-enter the system -- crash-recovery: nodes that fail can re-enter the system. This is more realistic, but also more complex. -- Byzantine fault tolerance: nodes can fail arbitrarily, including sending incorrect messages.

-

Paxos Overview

-

Paxos is a distributed consensus algorithm that is used to ensure that a majority of nodes agree on a value. Importantly however, Paxos doesn't guarantee that all nodes agree on a value (this is impossible in an asynchronous network), but it does guarantee that a majority of nodes agree on a value.

-

Paxos operates as a sequence of proposals, which may or may not be accepted by a quorum (majority) of nodes. If a proposal isn't accepted, the proposal has failed/been rejected. Each proposal is given a sequenceNumber, such that there is a strict ordering of proposals. This sequenceNumber must be unique for all proposals, and must also be monotonically increasing.

-

In the first phase, a proposer sends a sequenceNumber to all acceptors. If the acceptors have not seen a proposal with a higher sequenceNumber, they respond with a promise to not accept any proposals with a lower sequenceNumber. Otherwise, they reject the proposal. Once a proposer has received a majority of promises, it can commit the proposal by sending a commit message with a value.

-

The majority of nodes agreeing to a given proposal ensures that any committed proposal has a unique committed value, since two different quorums must have at least one node in common.

-

It is extremely important that any acceptors maintain a crash-recovery log of all proposals they have seen/accepted, so that they continue to honor their promises even after a crash.

-

System Architecture Patterns for Distributed Consensus

-

Distributed consensus algorithms should be used as low-level building blocks for distributed systems, and should be hidden behind higher-level abstractions. This allows for better separation of concerns, and allows for easier testing and debugging. Furthermore, the specific protocol used can then be swapped out without affecting the rest of the system.

-

In fact, it is common to use a consensus service, such as Zookeeper, to provide distributed consensus within a system. Designing applications as clients to a consensus service allows for even better seperation of concerns, and is done at Google with Chubby.

-

Reliable Replicated State Machines

-

A replicated state machine (RSM) is a system that maintains multiple copies of the same process by executing the same commands on all copies. Any deterministic program can be implemented as a highly available service by turning it into an RSM.

-

The order of operations is determined by a consensus algorithm running in a lower layer of the system. However, since there can be nodes part of a consensus group that aren't part of a given consensus quorum, nodes need to synchronize state from peers, which can be done using a sliding-window protocol.

-

Reliable Replicated Datastores and Configuration Stores

-

Many non-distributed consensus-based storage systems use timestamps to determine order of operations, but this approach doesn't work in distributed systems (due to clock drift). While some systems (like Google's Spanner) use a probabilistic approach to determining timestamps (TrueTime), this gets complicated and expensive. There is inherent uncertainty in the time at any given nodes, and Spanner tries to account for this uncertainty, which also minimizing it through periodic slow-downs to resynchronize clocks.

-

Instead, distributed consensus protocols can be used when replicating data across multiple nodes. However, these protocols can be slow, especially since operations on a storage system are often small and frequent, yet consensus protocols require multiple round trips to complete.

-

Highly Available Processing Using Leader Election

-

Leader election is an equivalent problem to distributed consensus, and is used in distributed systems to ensure that only one node is responsible for processing requests at a time. This might be used in cases where a single leader node is able to process requests, but it is often the case that a single leader needs to delegate work to a pool of worker nodes (like GFS or BigTable).

-

With this pattern, the leader election service is off of the critical path of the system, and so it has a smaller impact on the system's throughput.

-

Distributed Coordination and Locking Services

-

A barrier is a synchronization primitive that allows a group of nodes to wait until all nodes have reached a certain point before continuing. This lets you split a distributed computation into multiple stages that must be completed in order. For instance, in MapReduce, a barrier is used to ensure that all mappers have finished before reducers start.

-

While barriers can be implemented as a single coordinator node, this introduces a single point of failure. Instead, once can use an RSM to implement a barrier, which is done by Zookeeper's implementation of the barrier pattern.

-

Distributed locking is a more general distributed synchronization primitive that allows for mutual exclusion of shared resources among nodes. In practice, it is essential to use renewable leases with timeouts to prevent deadlocks. Distributed locks are another fairly low-level primitive, and it is often a good idea to use a higher-level abstraction that provides distributed transactions.

-

Reliable Distributed Queuing and Messaging

-

It is common to use a lease mechanism to ensure that only one node processes a message from a queue at a time, while also allowing for failover in case the node processing the message fails.

-

Queuing is also a powerful abstraction that can be used to implement other patterns like atomic broadcast and publish-subscribe messaging systems, where messages need to be reliably delivered to multiple nodes. This is useful for things like sending notifications to multiple clients, but can be used in other applications like distributed cache coherence. Furthermore, queuing as workload distribution can be used to distribute work among a pool of worker nodes

-

Distributed Consensus Performance

-

People are apparently pretty pessimistic about the performance of distributed consensus algorithms, but they can actually be quite fast. According to Google SREs, this is not the case.

-

There are many factors that can affect the performance of distributed consensus algorithms, including:

- -

One common performance pitfall with single-leader replication is that a client's perceived latency is proportional to the round-trip time between the client and the leader.

-

Multi-Paxos: Detailed Message Flow

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/mutual-exclusion.html b/site/distributed-systems/mutual-exclusion.html deleted file mode 100644 index b97e01f..0000000 --- a/site/distributed-systems/mutual-exclusion.html +++ /dev/null @@ -1,231 +0,0 @@ - - - - - - Mutual Exclusion - - - - - -
- -

Mutual Exclusion

-
- Last modified: 2024-04-15 - -
-
-

Distributed Mutual Exclusion

-

We want the same old mutual exclusion via locking, but in a distributed system. The trick is to keep a consistent ordering of locking events on every node in the system.

-

Implementation

-

Each message carries a timestamp $T_m$, and a sequence number $s$.

-

There are three message types:

- -

Each node maintains:

- -

On request receive:

- -

On receiving release:

- -

On acknowledge receive:

- -

To acquire the lock:

- -
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/non-blocking-two-phase-commit.html b/site/distributed-systems/non-blocking-two-phase-commit.html deleted file mode 100644 index 16d2cc0..0000000 --- a/site/distributed-systems/non-blocking-two-phase-commit.html +++ /dev/null @@ -1,238 +0,0 @@ - - - - - - Non Blocking Two Phase Commit - - - - - -
- -

Non Blocking Two Phase Commit

-
- Last modified: 2024-05-11 - -
-
-

Non-Blocking Two Phase Commit

-

Regular 2PC is blocking because we need to wait for all nodes to agree that an operation is commit-able. There are massive performance implications to read-only transactions (could fix with snapshot reads) and lock contention. We can fix this by relying on Paxos.

-

We make the coordinator as well as each participant a Paxos group with its own shard, and then we can just wait until all groups agree to an operation.

-

2PC on Paxos

-
    -
  1. Client requests multi-key operation at coordinator
  2. -
  3. Coordinator logs request
  4. -
  5. Coordinator sends prepare
  6. -
  7. Participants decide to commit/abort, and log result
  8. -
  9. Coordinator sends a commit/abort
  10. -
  11. Participants record results
  12. -
-

Multi-Key Transactions in a KV Store

-

Assuming reader-writer locking scheme, and that the application code runs on the client:

- -

The basic approach is to:

- -

Caution: Deadlocks

-

Deadlocks are very easy to trigger when performing operations across shards. A general solution is to always kill things that need to wait. For instance, with a checking/savings account at a bank:

- -

Deadlocks can be an issue in sharded systems when moving shards across groups.

-

Google's Bigtable in Retrospect

-

Jeff Dean of Google said that not supporting distributed transactions was the biggest mistake in the the design of Bigtable. Incremental updates make it a very important feature, so users really wanted them and often tried to implement it themselves on top of Bigtable

-

Spanner, Google's multi-datacenter KV store uses 2PC over Paxos, and is one of the backbones of Google's ad service.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/ordering-events-in-distributed-systems.html b/site/distributed-systems/ordering-events-in-distributed-systems.html deleted file mode 100644 index 66bc018..0000000 --- a/site/distributed-systems/ordering-events-in-distributed-systems.html +++ /dev/null @@ -1,293 +0,0 @@ - - - - - - Ordering Events In Distributed Systems - - - - - -
- -

Ordering Events In Distributed Systems

-
- Last modified: 2024-04-07 - -
-
-

Time, Clocks, and the Ordering of Events in a Distributed System

-

reading

-

Introduction

-

It would be convenient if we could order events in a distributed system, but it is impossible to do so in a way that is consistent with the order in which they actually occurred. However, we can define a partial ordering of events that is consistent with the order in which they occurred, and can extend this to a total ordering of events that is consistent with the partial ordering.

-

A partial ordering of objects is a relation that is reflexive, antisymmetric, and transitive. Conversely, a total ordering is a partial ordering that is also connexive, i.e. for any two objects $a$ and $b$, either $a \leq b$ or $b \leq a$.

-

Notation

- -

Clock Condition

-

If $a$ and $b$ are events in process $P_i$ and $a \to b$, then $C_i\langle a \rangle < C_i\langle b \rangle$.

-

If $a$ is the sending of a message and $b$ is the receipt of that message, then $a \to b$ and $C_i\langle a \rangle < C_j\langle b \rangle$.

-

To meet the clock condition, we must always increment the clock value of the process when an event occurs. Furthermore, if event $a$ is the sending of a message $m$ containing the clock value $T_m = C_i\langle a \rangle$, then upon receipt of $m$, the receiving process must set its clock value $C_j\langle b \rangle$ to a value greater than $T_m$.

-

Synchronized Access to a Shared Resource

-

We wish to find an algorithm for granting the resource to a process which satisfies the following three conditions:

- -

Assumptions

-

For any two processes $P_i$ and $P_j$, messages sent by $P_i$ are received in the order they were sent by $P_i$. This can be achieved using stop-and-wait or sliding window protocols.

-

Further, we assume that all messages are received within a bounded time, i.e. all messages will eventually be received.

-

Also note that this protocol requires active participation from all processes. If any one process fails, then the entire system stops, because the algorithm requires all processes to respond to a request before the resource can be granted.

-

Algorithm

-
    -
  1. To request the resource, a process $P_i$ sends a message $T_m:P_i$ request to all other processes, and puts that message on its request queue, where $T_m$ is the current clock value of $P_i$.
  2. -
  3. When process $P_j$ receives $T_m:P_i$ request, it places it on its request queue and sends a reply message to $P_i$ with its current clock value.
  4. -
  5. To release the resource, $P_i$ removes $T_m:P_i$ request from its request queue and sends a release message with its current clock value to all other processes.
  6. -
  7. When process $P_j$ receives a $P_i$ release message, it removes $T_m:P_i$ request from its request queue.
  8. -
  9. $P_i$ is granted the resource when the following conditions are met:
  10. -
  11. There is a $T_m:P_i$ request message in its queue ordered by $\Rightarrow$ before any other request messages.
  12. -
  13. $P_i$ has received a reply message from all other processes with a clock value greater than $T_m$.
  14. -
-

State Machine Perspective

-

The algorithm can be viewed as a state machine consisting of $C$, the set of commands and $S$, the set of states. The state machine is defined by the following function:

-

$$ -e: C \times S \to S -$$

-

Where $e(c, s) = s'$ means that executing the command $c$ in state $s$ results in a transition to state $s'$. In this algorithm, we have state $S$ corresponding to the queue of requests, and commands $C$ corresponding to the request and release messages for any given process.

-

$$ -C = { P_i \text{ request}, P_i \text{ release} } -$$

-

Executing $P_i \text{ request}$ in state $s$ results in a transition to state $s'$ where $s'$ is the state with the request message added to the queue. Executing $P_i \text{ release}$ in state $s$ results in a transition to state $s'$ where $s'$ is the state with the request message removed from the queue. Each process $P_i$ is its own state machine, and a process can execute a command time-stamped $T$ only if it has received all messages with time-stamps less than or equal to $T$.

-

Anomalous Behavior of Total Ordering

-

Consider a nationwide system of nodes. A person issues a request $a$ at node $A$, and after doing so, calls their friend in a different city at node $B$ to issue a request $b$. It is possible with the total ordering of events that $b \to a$, even though $a$ was issued before $b$. This is because the total ordering of events is not consistent with the order in which they actually occurred, but rather with the order in which they were observed. The message that would be able to establish the the actual ordering of events (the call) is external to the system.

-

More concretely, let $\mathscr{L}$ be the set of all events, and $L$ be the set of all events in our system.

-

$$L \subseteq \mathscr{L}$$

-

In the above scenario, we had $a \to_{\mathscr{L}} b$, but $b \nrightarrow_{L} a$. No algorithm based soley on the events in $L$, without knowledge more generally of $\mathscr{L}$ can order $a$ before $b$.

-

There are two approaches to fixing this issue:

-
    -
  1. Users must manually specify restraints on timestamps, ie the person at $A$ must tell the person at $B$ that $T_a < T_b$. This is not a great solution.
  2. -
  3. Construct a system of clocks that satisfy the following condition: if $a \to_{\mathscr{L}} b$, then $C\langle a \rangle < C\langle b \rangle$.
  4. -
-

Physical Clocks

-

Let $C_i(t)$ be the value clock $C_i$ at time $t$. Assume that $C_i(t)$ is a continuous and differentiable function of $t$, except for isolated points where it jumps when the clock is reset. Note that a discrete clock can be modeled as a continuous clock with an error of up to $\epsilon = \frac{1}{2} \text{ tick}$.

-

Then $\frac{dC_i(t)}{dt}$ is the rate at which the clock is running at time $t$. If $\frac{dC_i(t)}{dt} = 1$, then the clock is running at the correct rate. We assume that the following holds:

-

$$ -\exists \kappa \ll 1 \text{ such that } \forall i, t: |\frac{dC_i(t)}{dt} - 1| < \kappa -$$

-

And in fact, for typical quartz oscillator clocks, $\kappa \le 10^{-6}$.

-

This however isn't enough. For an effective clock system, we also want our clocks to be synchronized such that...

-

$$ -\forall i, j, t: |C_i(t) - C_j(t)| < \epsilon -$$

-

...where $\epsilon$ is the maximum error in the clocks. Since physical clocks will never run at exactly the same rate, they will tend to drift apart over time. To correct for this, we can use a synchronization algorithm to periodically reset the clocks to a common time.

-

Letting $\mu$ be a number such that if event $a$ occurs at physical time $t$, and event $b$ is an event in another process that satisfies $a \to b$, then $b$ occurs later than physical time $t + \mu$. So $\mu$ is the maximum time it takes for a message to be sent from one process to another.

-

To avoid anomalies, we must ensure...

-

$$ -\forall i, j, t: C_i(t) < C_j(t + \mu), \text{ or, equivalently, } C_i(t + \mu) - C_j(t) > 0 -$$

-

And relating this to $\kappa$...

-

$$ -\forall i, j, t: C_i(t + \mu) - C_j(t) > (1 - \kappa)\mu -$$

-

And calculating the maximum error in the clocks...

-

$$ -\frac{\epsilon}{1 - \kappa} \le \mu -$$

-

Clock Synchronization Algorithm

-

Importantly, one must always synchronize clocks in the forward direction, i.e. $C_i(t) < C_j(t + \mu)$. If $C_i(t) > C_j(t + \mu)$, then $C_i(t + \mu) - C_j(t) < 0$, which is not allowed.

-

Let $m$ be a message sent at time $t$, and received at time $t'$. Let $v_m = t' - t$ be the total delay of the message. Although the delay of a message is not known by any given process, we assume the receiver has some lower bound on the delay, $\mu_m$. Define $\zeta_m = v_m - \mu_m$ to be the unpredictable delay of the message.

-

Now, define the following rules for our physical clocks:

-
    -
  1. $\forall i$ if $P_i$ does not receive a message at physical time $t$, then $C_i$ is differentiable at $t$ and $\frac{dC_i(t)}{dt} > 0$.
  2. -
  3. If $P_i$ sends a message $,$ at physical time $t$, then $m$ contains a timestamp $T_m = C_i(t)$. Upon receiving a message $m$ at physical time $t'$, $P_j$ sets $C_j(t') = \max(\lim_{\delta \to 0} C_j(t' - \delta), T_m + \mu_m)$.
  4. -
-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/paxos-architecture.html b/site/distributed-systems/paxos-architecture.html deleted file mode 100644 index 61a625c..0000000 --- a/site/distributed-systems/paxos-architecture.html +++ /dev/null @@ -1,209 +0,0 @@ - - - - - - Paxos Architecture - - - - - -
- -

Paxos Architecture

-
- Last modified: 2024-05-16 - -
-
-

Distributed Architectures with Paxos

-

Overhead of Simple Architectures

-

Paxos can make progress so long as a majority of nodes are up. For a Paxos group of size $k$, requires a general overhead of $3(k-1) + 2$ messages.

-

Primary-backup (with a single backup) using a view server has an overhead of $4$ messages to forward requests. It can handle any one failure.

-

Using a single server has an overhead of $2$ messages to service a request, and cannot handle any failures.

-

The above demonstrates a clear tradeoff between the overhead of the system and the number of failures that can be tolerated.

-

Paxos as a Lease Server

-

A lease is a time-limited right to do something. They rely on loosely synchronized clocks, and are used to avoid the overhead of Paxos for every operation. A typical lease term is around a few seconds, plus or minus some epsilon to account for clock drift. If a lease holder fails, the system just waits for the lease to expire.

-

The following workflow is used to implement leases:

-
    -
  1. The lease is granted to the primary
  2. -
  3. Primary serves requests until the lease expires, forwarding to the backup
  4. -
  5. If the primary doesn't renew the lease (i.e. fails), a lease is granted to the next primary
  6. -
-

This design pattern is used in BigTable, Chubby, and ZooKeeper. It prevents split brain if the clock drift is within epsilon. We also only need to service reads on the primary, including logic for cache invalidation. Additionally, we can use write ahead logging, and instead of explicitly maintaining a backup, just replace the primary by executing the log on a new primary.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/paxos-intro.html b/site/distributed-systems/paxos-intro.html deleted file mode 100644 index 73f2303..0000000 --- a/site/distributed-systems/paxos-intro.html +++ /dev/null @@ -1,224 +0,0 @@ - - - - - - Paxos Intro - - - - - -
- -

Paxos Intro

-
- Last modified: 2024-04-24 - -
-
-

Paxos Introduction

-

FLP Impossibility Result

-

It's impossible for a deterministic protocol to guarantee consensus in bounded time in an asynchronous distributed system. The progress and safety of a system are at odds with each other.

-

Paxos makes the decision to always be safe, and is able to make progress and avoid blocking as long as the majority of nodes are up and there aren't further failures.

-

State Machine Replication

-

Order events/operations into an append-only log. Consensus is easy if only one client request is handled at a time.

-

Select a leader for clients to send requests to, and define the ordering at that leader. If any leader fails or is slow, elect a new leader (can keep doing this repeatedly). Then, each leader proposes a value that all nodes should agree on.

-

Leader election is where Paxos comes in.

-

Paxos, the algorithm

-
Proposer:
-  Prepare(n) -> Promise(n, n', v')
-  Accept(n, v) -> Accepted(n, v)
-
-Acceptor:
-  Promise(n, n', v') -> Prepare(n)
-  Accepted(n, v) -> Accept(n, v)
-
-

Phase 1: Prepare

- -

Phase 2: Accept

- -
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/paxos-made-simple.html b/site/distributed-systems/paxos-made-simple.html deleted file mode 100644 index d98be80..0000000 --- a/site/distributed-systems/paxos-made-simple.html +++ /dev/null @@ -1,266 +0,0 @@ - - - - - - Paxos Made Simple - - - - - -
- -

Paxos Made Simple

-
- Last modified: 2024-04-24 - -
-
-

Paxos Made Simple

-

reading

-

The Consensus Algorithm

-

The Problem

-

Consider a set of processes that can propose values. A consensus algorithm ensures that a single value is chosen and agreed upon. For safety, we must have...

- -

And it assumes an asynchronous, non-byzantine network in which nodes are fail-stop.

-

Choosing a Value

-

In paxos there are 3 types of agents

- -

P1: An acceptor must accept the first proposal it receives

-

This guarantees that some value is accepted by each node that is proposed to, but it doesn't prevent situations where no proposal is accepted by a majority of acceptors.

-

P2: If a proposal with value $v$ is chosen, then every higher-numbered proposal accepted by any acceptor has value $v$

-

This guarantees that only a single value is chosen, since unique proposal numbers can be used to decide between accepted values.

-
P2a: If a proposal with value $v$ is chosen, then every higher-numbered proposal accepted by any acceptor has value $v$
-

This is a stronger version of P2 that ensures previous values are not forgotten/overridden.

-

However, P2a contradicts P1, since if a proposer "wakes up" after having been out of commission, it must accept whatever value is proposed first. We thus strengthen even further to...

-
P2b: If a proposal with value $v$ is chosen, then every higher-numbered proposal issues by any proposer has value $v$
-

This shifts the burden of remembering and staying consistent with the chosen value to the proposers instead of the acceptors. We then have...

-

$$ -\text{P2b} \to \text{P2a} \to \text{P2} -$$

-

In order to implement P2b, we must further constrain our algorithm's behavior to...

-
P2c: For any $v$ and $n$, if a proposal with value $v$ and number $n$ is issued, there is a set $S$ consisting of a majority of acceptors such that either (a) no acceptor in $S$ has accepted any proposal numbered less than $n$, or (b) $v$ is the value of the highest-numbered proposal among all proposals numbered less than $n$ accepted by the acceptors in $S$
-

To satisfy P2b, we must maintain P2c as an invariant. To make sure this invariant holds, proposers proposing a proposal numbered $n$ must learn the highest-numbered proposal with a number less than $n$ that has been accepted by a majority of acceptors, and propose that value if it exists.

-

Proposition Algorithm

-
    -
  1. A proposer chooses a new proposal number $n$ and sends a prepare request to each member of some set of acceptors, awaiting a response containing:
      -
    • A guarantee that this acceptor will never accept a proposal numbered less than $n$
    • -
    • The proposal with the highest number less than $n$ that it has accepted, if any.
    • -
    -
  2. -
  3. If the proposer receives the requested responses from a majority of acceptors, it issues an accept request, which is a proposal with number $n$ and value $v$, where $v$ is the value of the highest-numbered proposal among the responses, or a value of the proposer's choice if no proposals in the responses were received.
  4. -
-

Acceptor Behavior

-

Acceptors can only receive prepare and accept requests, and can ignore any request without compromising safety (but certainly still sacrificing liveness).

-
P1a: An acceptor can accept a proposal numbered $n$ iff it has not responded to a prepare request numbered greater than $n$
-

which implies P1

-

Phases

-

Phase 1

- -

Phase 2

- -

Note that to increase performance, if an acceptor ignores a prepare or accept request because it has already received a prepare request with a higher number, it should notify the proposer with a reject message. This however doesn't change the correctness, and is thus optional.

-

Learning a Chosen Value

-

One option would be for each acceptor to send a message upon accepting a value to all learners, but this requires a whole lot of message passing. Another option is to maintain a set of distinguished learners, which after hearing of a majority acceptance, notify all other learners of the accepted value. The larger this set of distinguished learners, the more fault-tolerant the system, but also the more communication required.

-

Since messages can be dropped, a value can be chosen without any learner finding out. In this case, learners will find out the chosen value only after a new proposal is chosen. Learners can thus determine whether a value was chosen by following the same protocol to issue a new proposal as above.

-

Progress

-

It is entirely possible with the above protocol that multiple proposers indefinitely one-up each other between sending their propose and accept requests, such that all proposals are ignored. To prevent this, choose and maintain a single distinguished proposer, which is the only proposer allowed to issue proposals. If the distinguished proposer fails, a new one can be elected by the acceptors.

-

By FLP, any such leader election system must rely either on randomness, or real-time (i.e. timeouts).

-

Implementing a State Machine

-

Consider a system of clients that issue requests to execute commands on a cluster of single-threaded application servers. Each application server can be thought of as a deterministic state machine, where the ordering of requests to each server must be consistent for them to end up in the same state.

-

To guarantee consistent ordering of commands executed within our cluster, we implement a separate instance of Paxos, where the $i$th instance's chosen value determines the $i$th command executed on all application servers.

-

During normal operation, a single server is elected to be leader, which acts as the distinguished proposer, and is the only server allowed to issue proposals. Clients then send their requests to this leader, which decides the sequence of commands globally. Any given instance of the protocol might fail, but regardless only one command can ever be chosen as the $i$th command to be executed.

-

For cases where some subsequence of commands are not yet determined, i.e. not chosen yet when a new leader takes over, the new leader issues phase 1 for all such instances (including the infinitely many commands greater than the largest command in our current sequence). Any values received in response are then proposed, but if an instance remains unconstrained (i.e. no value has been accepted), the leader can propose no-ops for the gaps in the sequence of commands before the last accepted command. It must do this before ever executing commands that come after these unconstrained slots.

-

After doing so, the leader can continue proposing any further commands requested by clients. The leader is allowed to propose command $i + \alpha$ before knowing the chosen command for $i$, meaning it can get up to $\alpha - 1$ commands ahead of itself (in the case where all commands less than $i + \alpha$ were dropped).

-

Once a leader has finished phase 1 for all commands thus far and afterwards, it only needs to complete phase 2 for each subsequent command requested, which is known to be the minimal algorithm for reaching consensus after phase 1.

-

To reiterate what was stated previously, in the case where a single leader is not elected, progress is not guaranteed, but safety is.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/primary-backup.html b/site/distributed-systems/primary-backup.html deleted file mode 100644 index 9280930..0000000 --- a/site/distributed-systems/primary-backup.html +++ /dev/null @@ -1,269 +0,0 @@ - - - - - - Primary Backup - - - - - -
- -

Primary Backup

-
- Last modified: 2024-04-01 - -
-
-

Primary Backup

-

Consider a highly available stateful service. It is easy to maintain consistency within one node, simply by performing operations in some well-defined (serializable) order. However, providing both availability and consistency is more of a challenge. One must provide a notion of having a single system, even if a server within the system fails.

-

Single Node KV Store

-

Consider an instance of redis with multiple clients reading and writing to it. You can think of this system more abstractly as a state machine, where each client applies an operation that changes the state of the system.

-

State Machine Replication

-

Replicate state machines across multiple servers. If you apply the same set of operations to each server in the same order, their ending states must be in the same state. This holds so long as the effect of each operation is deterministic.

-

Example: Virtual Machine Replication

-

Take a single VM running a single application. Create $n$ copies of this VM, and feed each instance the exact same inputs (packets, interrupts, instructions). Then, all $n$ VMs will have the same behavior.

-

Any time you introduce randomness into a system, you need to ensure that the randomness is deterministic. This mechanism for VM replication assumes you are only using a single core, and all operations are deterministic.

-

Two Servers (Primary-Backup)

-

At any given time, clients speak to only one server (the primary). Data is replicated on primary and backup servers, and if the primary fails, the backup is elected as the new primary.

-

The goals of doing this is to increase the availability and reliability of the system in the face of failures.

-

Basic Operations

- -

Key Assumptions

- -

Key Challenges

- -

The View Service

-

The view service is a server that provides a consistent view of the system. Clients ask the view service for the primary server's address in order to find out where to send operations. Even if the view server incorrectly identifies failure, the system will still be consistent.

-

The view server is the only authority on who the primary is. This makes it a single point of failure. The hard part is that we need to be able to guarantee only one primary at a time, while not needing to ping the view server on every operation.

-

This system needs to be able to tolerate any individual server failing, while still serving client requests.

-

Detecting Server Failures

- -

When the view server detects a failure, a new view (state of the system sent in ping responses) is created.

-

Primary Failures

- -

If primary dies with no idle servers available, then the backup becomes the primary and there is no backup.

-

Managing Servers

-

Keep a pool of idle servers that can be promoted to backup. If primary dies, create new view with old backup as primary and idle server as backup. If the backup dies, create a new view with idle server as new backup.:

-

Split Brain

-

In the case where a primary appears to be offline, but is really just partitioned from the view server, the view server may elect a new primary. This leads to a split brain scenario, where two primaries are elected.

-

The important part is that two servers can think they are the primary, but it can never be the case that two servers act as the primary.

-

Rules

-
    -
  1. Primary in view $i + 1$ must have been the backup, or the primary in view $i$ (besides the first view).
  2. -
  3. Primary must wait for backup to accept/execute each operation before replying to client (if there is one).
  4. -
  5. Backup must accept forwarded requests only if view is correct.
  6. -
  7. Non-primary must reject client requests.
  8. -
  9. Every operation must be before or after state transfers (not during).
  10. -
-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/scaling-web-services.html b/site/distributed-systems/scaling-web-services.html deleted file mode 100644 index 3069575..0000000 --- a/site/distributed-systems/scaling-web-services.html +++ /dev/null @@ -1,217 +0,0 @@ - - - - - - Scaling Web Services - - - - - -
- -

Scaling Web Services

-
- Last modified: 2024-04-25 - -
-
-

Scaling Web Services with Distributed Architectures

-

Two Tier Architecture

-

Many companies adopted the idea of a two tier architecture for web services. The idea was to have a scalable number of frontend servers, mapping clients to one of those servers. Then, you could also have a scaled out backend with multiple storage severs, mapping frontend servers to the data they needed in the storage servers.

-

Two-Tier RESTful Web Architecture

-

Keep a scalable number stateless servers hosting the client and running application code. Importantly, it doesn't matter if a client server crashes, since the user can just connect to some other client server. At the same time, run a scalable number of backend servers running in a carefully designed distributed system, often using primary/backup or paxos for high availability and fault tolerance. Anything that needs to be persistent across crashes should be handled on the backend.

-

Load Balancing

-

Typically, the layer between the tiers of such architectures are composed of load balancers, which need to map any given client to a desirable front-end server. This needs to be consistent per connection, which can be done with hash(clientIP, port) -> clientServerIP. Additionally, you need to map each client server to a storage server, which can be done by hash(key) -> storageServerIP, where the key is some identifier for the location of the data in a given query to our storage system.

-

Importantly, the system should automatically adapt to the addition of any type of server.

-

Three-Tier Web Architecture: Look-aside Caching

-

Also maintain a set of cache servers to offload queries to the storage server. Client servers first send their query to the cache server, and if there is a cache miss they then fall back to the storage servers, and then write the retrieved data to the cache (this is look-aside caching).

-

There are other ways to do caching. For instance, the cache could directly retrieve values transparently to the client server, such that cache misses don't need to be handled. However, this tightly couples the cache and the storage server, often requiring that all queries pass through the caching layer, making it harder to design the two services independently.

-

Of course, caching needs to be scalable as well. Cache servers don't necessarily need to be 1:1 with client/storage servers, but they should be able to handle the load of the client servers they are caching for adaptively. They should also ideally have lower latency than actual queries executed in the storage layer.

-

Newer Architectures

-

Edge Computing

-

Moving data processing closer to the client.

-

Users are often globally distributed, leading to higher latency and thus worse user experience.

-

To mitigate this, large applications will often be globally distributed in edge data centers, ideally with one reasonably near every user. Often, only the web and cache (RESTful) layer are present on the edge, and content can be distributed by "pushing" it to the edge before it is ever requested.

-

In tandem, core data centers host the web, cache, and storage layer, replicating all of this across data centers for disaster tolerance.

-

Service Oriented Architecture

-

Services define external interfaces, and often requires distributed systems that work in a hostile environment. All teams expose the data/functionality through this interface, and all communication happens through network calls. Each service runs as a standalone product with its own service level agreement to its clients.

-

Microservices

-

Organize complex distributed applications as a large number of independent services communicating through RPC, each using primary/backup or paxos for high availability and fault tolerance.

-

This allows for independent development of components of a larger system, where each component can scale independently.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/sharding.html b/site/distributed-systems/sharding.html deleted file mode 100644 index e2db100..0000000 --- a/site/distributed-systems/sharding.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Sharding - - - - - -
- -

Sharding

-
- Last modified: 2024-05-06 - -
-
-

Sharding

-

Consistent Hashing

-

A classic approach is to maintain a modular space of hashed keys, and use the regions between shards to assign keys. This is known as consistent hashing. It works, but has some drawbacks:

- -

Indirection Tables

-

A cooler approach in my opinion. Just put a table of hash(key) -> server address on every client, and assign fewer table entries to buckets with more keys. This way, the load is more evenly distributed. You can then broadcast any changes to the table to every client server.

-
- -
- - \ No newline at end of file diff --git a/site/distributed-systems/two-phase-commit.html b/site/distributed-systems/two-phase-commit.html deleted file mode 100644 index ff98690..0000000 --- a/site/distributed-systems/two-phase-commit.html +++ /dev/null @@ -1,275 +0,0 @@ - - - - - - Two Phase Commit - - - - - -
- -

Two Phase Commit

-
- Last modified: 2024-05-11 - -
-
-

Two Phase Commit

-

ACID

-

For updates that span multiple keys, or even multiple updates across different storage systems, we need all-or-nothing semantics so errors can be properly handled. Two-phase commit (2PC) is a protocol that ensures distributed transactions, i.e. groups of operations, are atomic, consistent, isolated, and durable (ACID).

- - - - - - - - - - - - - - - - - - - - - - - - - -
TermDescription
Atomicoperations appear to either happen as a group, or not at all
DurableOperations that complete stay completed
IsolationOther transactions don't see the results until of earlier transactions unless they were already committed
Consistencylinearizability (or some other consistency model)
-

Two Phase Locking (2PL) - Consistency and Isolation

-

In 2PL, locks are acquired on all structures touched during the transaction, and are only released upon commit or abort. This provides isolation and consistency for multi-key transactions.

-
- start transaction -
-Phase 1: acquire locks
-- commit or abort -
-Phase 2: release locks
-
-

Redo Logging - Atomicity and Durability

-

Log all changes to disk, followed by a log commit. If there is a crash before the log commit, abandon the transaction. If it was committed in the log, we can just redo the changes.

-

Deadlock

-

Deadlock is when two or more transactions are waiting for locks held by each other in a cycle. To solve this you can stop one of the transactions to break the cycle.

-

Deadlock prevention is generally a better idea, and you can achieve it by always ordering lock acquisition consistently.

-

Distributed Transactions

-

From the two generals problem, it is theoretically impossible to agree on performing some action at the same time. Instead, we agree in virtual time when an operation happens.

-

Atomic Commit Protocol (ACP)

- -

2PC in Detail

-

2PC is a blocking protocol, meaning that it makes no progress if some participants are unavailable. It has fault tolerance, but is not highly available, which is a fundamental limit of the protocol.

- -

Handling Failures

-

Participant Fails Before Sending Response

-

You can maintain a timer on the coordinator to retry prepares. If some threshold is reached, just log a no and abort

-

If the participant then comes back online, they will need to ask the coordinator for the decision, at which point the coordinator sends an abort to the participant

-

Participant Fails Before After Sending Vote

-

If the participant crashes immediately after sending their response. Then either they come back online before the commit is sent, at which point the protocol continues, or they will need to check their log and request the decision from the coordinator, which will resent the commit and the protocol continues.

-

Coordinator Fails Before Sending Prepare

-

They would have logged the prepare request, so when they come back online and execute the transaction.

-

Coordinator Fails After Sending Prepare

-

If the coordinator fails after sending prepares, but before receiving responses, they must have logged the prepared already, and they need to be resent.

-

Roles

- -

Messages

- -
- -
- - \ No newline at end of file diff --git a/site/ds-backup/RPC.html b/site/ds-backup/RPC.html deleted file mode 100644 index 5952f0f..0000000 --- a/site/ds-backup/RPC.html +++ /dev/null @@ -1,336 +0,0 @@ - - - - - - Rpc - - - - - -
- -

Rpc

-
- Last modified: 2024-04-27 - -
-
-

Look Into

- -

Remote Procedure Call (RPC)

-

A request from a client to execute a function on a server/different machine. -- To the client, looks like a local function call. -- To the server, looks like an implementation of a function call.

-

Google handles $10^{10}$ RPCs per second.

-

Local Procedure Call

- -

The compiler defines the protocol for the call above.

-

Remote Procedure Call

- -

Client/server implementation is usually auto-generated from procedure spec, e.g. Google's Protocol Buffers/Protobuf.

-

RPC vs. Local Procedure Call

-

Binding

- -

Service Discovery Service

- -

Interface Description Language (IDL, e.g. Protobuf)

-

Serialization is important!

- -

Failures

- -

Some of the network issues can be mitigated by TCP, but sockets can fail and messages aren't always transmitted over TCP anyways.

-

Fault Model

- -

Naive RPC

- -

Client timeout and retry

-

If a request or reply message is dropped, the client will wait forever for the response. This can be fixed with client timer/retransmission, where the client sends the request again if it doesn't get a response in a certain amount of time. This leads to duplication and reordering of messages at the server.

-

We can handle this with a unique request ID. Include a message ID in each request/reply. When the client retransmits, it uses the same message ID. The server can then ignore duplicate requests.

-

RPC Semantics

- -

At least once

-

Client should do a finite number of retries, eventually giving up and returning an error to the caller.

-

This only works if the server is idempotent, meaning it has the same effect if it's executed multiple times. All read-only operations are idempotent, but not all write operations are. For example, icrementing a counter is not idempotent, but setting the counter to a value is.

-

Does TCP handle this? Not really despite being reliable. Most RPCs are sent over TCP, and it guarantees in-order delivery with retransmission and duplicate detection. However, it doesn't guarantee exactly-once semantics. If the server crashes after processing the request but before sending the response, the client will retransmit the request, and the server will execute it again.

-

End to end principle: Functionality should be implemented where it can be completely handled, rather than partially handled at each layer. This decreases the chance of partially completed work due to unrelated failures.

-

Examples: -| Example | Explanation | -|---|---| -| DNS lookup | Queries are read-only, so it's idempotent | -| MapReduce | The Map phase is idempotent since it is a pure function | -| NFS | If the client maintains offset, reading/writing a block is idempotent |

-

Importantly, in situations with multiple clients, operations like Put(k, v) are not idempotent, since the value of k can change between the time the client reads the value and the time it writes the value.

-

Two Generals Problem

-

Just a thought experiment to emphasize the difficulty of message passing in a distributed system. Two generals are trying to coordinate an attack on a city. They are separated by a valley, and can only communicate by messenger. The messenger can be captured by the city, and the generals don't know if the message was delivered. The generals need to agree on a time to attack, but they can't be sure the message was delivered. They can only attack if they both agree on the time.

-

The problem boils down to the fact that at any point in time, if we sent a message, we don't know if it was delivered. Regardless of how many round trips you make to confirm, the last message sent could always have been dropped. This is a fundamental and central problem in distributed systems.

-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/clocks.html b/site/ds-backup/clocks.html deleted file mode 100644 index 07ef65d..0000000 --- a/site/ds-backup/clocks.html +++ /dev/null @@ -1,292 +0,0 @@ - - - - - - Clocks - - - - - -
- -

Clocks

-
- Last modified: 2024-04-27 - -
-
-

Clocks

-

There are two main approaches to time in a distributed system: physical clocks and virtual (logical) clocks.

-

Physical Clocks

-

Actual clocks running in most computers drift apart by ~30 ppm due to their temperature sensitivity. Although more accurate clocks (atomic, GPS, etc.) are available, they are expensive and are only maybe present in some data centers.

-

The crux of the problem is that physical clocks are not perfectly synchronized, and the sending of messages between processes can introduce unpredictable delays. In general, network latency is unpredictable, but with a lower bound.

-

A practical, albeit naive approach might be to use NTP and have clients query a set of time servers. Then, take the minimum (or some average, subtracting outliers) of the readings received. This can synchronize to ~50 microseconds in a LAN.

-

Such a system was implemented with Google Huygens. Some interesting optimizations they added include:

- -

This enabled them to achieve a 50 ns clock skew 99% of the time. This is okay if time is only used as a hint, but shows that even with all of the above optimizations, it isn't good enough.

-

To drive this point home, due to the massive scale Google operates at (1 billion RPCs/sec = 10 million clock skews above 50ns per sec), even for a minimum sized message that takes 2 ns to send in a high-performance network, thousands of instructions can be executed on a single server's processor.

-

Virtual Clocks

-

We want to design systems such that the ordering of events that can be concurrently executed doesn't matter, and the ordering of events that must be performed sequentially is enforced on all possible executions.

-

Virtual clocks are a framework for reasoning about the order of events using no assumptions about physical clock skew or message delays in way way that both respects causality, and relies only on local information.

-

Happens before

-

We say that event a happens before event b if:

-
    -
  1. a happens earlier than b in the same process
  2. -
  3. a is the sending of a message and b is the receipt of that message
  4. -
  5. a happens before c and c happens before b, aka transitivity
  6. -
-

This is a partial order.

-

Happens concurrently

-

Two events a and b are said to happen concurrently if neither a happens before b nor b happens before a.

-

Logical Clock Implementation

- -

Vector Clocks

-

Note that with the above implementation of a logic clock system, it was not the case that $T(a) < T(b) \to $ $a$ happened before $b$. With vector clocks, we have $T(a) < T(b) \leftrightarrow a$ happened before $b$ by precisely representing transitive causal relationships between events. This is used in practice for eventual and causal consistency (ie Git, Amazon Dynamo, etc.).

-

Algorithm

-

Clock is a vector C, with length = # of nodes in the system

- -
public class VectorClock {
-  public final int[] clock;
-
-  public VectorClock(int n) {
-    clock = new int[n];
-  }
-
-  public void increment(int i) {
-    clock[i]++;
-  }
-
-  public void handleMessage(VectorClock other) {
-    for (int i = 0; i < clock.length; i++)
-      clock[i] = Math.max(clock[i], other.clock[i]);
-  }
-}
-
-
public class Node {
-  private int id;
-  private VectorClock vc;
-
-  public Node(int id, int n) {
-    this.id = id;
-    this.vc = new VectorClock(n);
-  }
-
-  public void event() {
-    vc.increment(id);
-  }
-
-  public void merge(Node other) {
-    vc.handleMessage(other.vc);
-  }
-
-  public void send(Node other) {
-    other.vc.handleMessage(vc.clock);
-  }
-
-  public boolean[] didHappenBefore(Node other) {
-    boolean[] res = new boolean[2];
-    res[0] = true;
-    res[1] = true;
-
-    for (int i = 0; i < vc.clock.length; i++) {
-      if (vc.clock[i] > other.vc.clock[i])
-        res[0] = false;
-       else if (vc.clock[i] < other.vc.clock[i])
-        res[1] = false;
-    }
-
-    return res;
-  }
-}
-
-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/consistent-global-state.html b/site/ds-backup/consistent-global-state.html deleted file mode 100644 index db20676..0000000 --- a/site/ds-backup/consistent-global-state.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Consistent Global State - - - - - -
- -

Consistent Global State

-
- Last modified: 2024-04-27 - -
-
-

Consistent Global State in Distributed Systems

-

reading

-

Introduction

-

Many problems in distributed computing boil down to being able to maintain a consistent global state, and to run predicates on that state in order to trigger events. The true state of a distributed system is the union of all node's states. However, since nodes don't share memory, the actual state must be meaningful when inferred solely based on messages passed among nodes.

-

A global state is said to be inconsistent if it never could have been constructed by an ideal external observer. This paper formalizes this concept into the context of a Global Predicate Evaluation (GPE), which determines if the system satisfies some predicate $\Phi$.

-

Asynchronous Distributed Systems

-

Define a distributed system as a set $P$ of sequential processes $p_1, p_2, \ldots, p_n$, and a network consisting of channels in which unidirectional communication is possible in the space of $P^2$. The network is assumed to be reliable, but may deliver messages out of order, and is taken to be strongly connected, but not necessarily fully connected (i.e. communication may require intermediate message passing).

-

It is useful to reason about distributed systems with the weakest possible assumptions, such that results hold for arbitrary systems.

-

Distributed Computations

-

A distributed computation is the execution of a distributed program over a collection of processes, each of which sequentially process a stream of events. Particularly, for two nodes to communicate, a message $m$ is enqueued on a channel via $send(m)$, and the message is dequeued via $receive(m)$. There is an obvious relationship between the happening of event $send(m)$ at process $p$, and the happening of event $receive(m)$ at process $q$, such that we can be sure $send(m)$ happened before $receive(m)$.

-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/managing-critical-state.html b/site/ds-backup/managing-critical-state.html deleted file mode 100644 index c4e3906..0000000 --- a/site/ds-backup/managing-critical-state.html +++ /dev/null @@ -1,258 +0,0 @@ - - - - - - Managing Critical State - - - - - -
- -

Managing Critical State

-
- Last modified: 2024-04-27 - -
-
-

Managing Critical State

-

reading

-

CAP Theorem

-

The CAP theorem states that a distributed system can only guarantee two of the following three properties: -- Consistency: All nodes see the same data at the same time. -- Availability: Every request receives a response, without guarantee that it contains the most recent write. -- Partition tolerance: The system continues to operate despite network partitions.

-

Essentially, if some nodes go down in a distrubuted system, you can either choose to continue serving requests (availability), or to stop and wait for the nodes to come back up (consistency).

-

ACID's Alternative: BASE

-

While ACID (Atomicity, Consistency, Isolation, Durability) lists properties that provide semantics for consistent transactions on a single node, this doesn't translate or scale well to distributed systems.

-

Instead, some datastores use BASE (Basically Available, Soft state, Eventually consistent) semantics, allowing for more flexibility in the face of network partitions. Most systems that support BASE semantics use multi-leader replication, where each leader can accept writes and propagate them to other leaders. This allows for better availability and partition tolerance, but at the cost of complexity to deal with eventual consistency in application code.

-

Motivating the Use of Consensus: Distributed Systems Coordination Failures

-

The Split Brain Problem

-

Split brain is where multiple nodes think they are the leader. A naiive approach to solving this is to use a heartbeat mechanism, where a leader sends out a heartbeat to followers. If multiple nodes think they are the leader, they will both send out heartbeats, and eventually one of the "leaders" will realize that there is another leader and issue a STONITH (Shoot The Other Node In The Head) command to kill the other leader.

-

However, due to the asynchronous and unreliable nature of networks, it is possible for these messages to be delayed to a point where both nodes issue a STONITH command to each other, causing both nodes to go down. Furthermore, the issue of actually detecting and avoiding a split brain is non-trivial, since it is difficult to distinguish between a network partition and a node failure, and nodes can be partitioned from eachother arbitrarily.

-

Faulty Group Membership Algorithms

-

Using gossip protocols to maintain group memberships of clusters can lead to issues. Specifically, partitions within a cluster lead to multiple leaders being elected in the same cluster, often leading to data loss or corruption.

-

How Distributed Consensus Works

-

In distributed software systems, we care about asynchronous distributed consensus, where nodes can fail and messages can be delayed, lost, or duplicated arbitrarily. Technically, this is impossible to solve in bounded time (see Dijkstra's FLP result), but we can solve problems by ensuring the system has sufficient healthy replicas and network connectivity, allowing the system to make progress most of the time. Futher, exponential backoff can be used to prevent cascading failures.

-

Some characteristics of distributed consensus algorithms include: -- crash-fail: nodes that fail never re-enter the system -- crash-recovery: nodes that fail can re-enter the system. This is more realistic, but also more complex. -- Byzantine fault tolerance: nodes can fail arbitrarily, including sending incorrect messages.

-

Paxos Overview

-

Paxos is a distributed consensus algorithm that is used to ensure that a majority of nodes agree on a value. Importantly however, Paxos doesn't guarantee that all nodes agree on a value (this is impossible in an asynchronous network), but it does guarantee that a majority of nodes agree on a value.

-

Paxos operates as a sequence of proposals, which may or may not be accepted by a quorum (majority) of nodes. If a proposal isn't accepted, the proposal has failed/been rejected. Each proposal is given a sequenceNumber, such that there is a strict ordering of proposals. This sequenceNumber must be unique for all proposals, and must also be monotonically increasing.

-

In the first phase, a proposer sends a sequenceNumber to all acceptors. If the acceptors have not seen a proposal with a higher sequenceNumber, they respond with a promise to not accept any proposals with a lower sequenceNumber. Otherwise, they reject the proposal. Once a proposer has received a majority of promises, it can commit the proposal by sending a commit message with a value.

-

The majority of nodes agreeing to a given proposal ensures that any committed proposal has a unique committed value, since two different quorums must have at least one node in common.

-

It is extremely important that any acceptors maintain a crash-recovery log of all proposals they have seen/accepted, so that they continue to honor their promises even after a crash.

-

System Architecture Patterns for Distributed Consensus

-

Distributed consensus algorithms should be used as low-level building blocks for distributed systems, and should be hidden behind higher-level abstractions. This allows for better separation of concerns, and allows for easier testing and debugging. Furthermore, the specific protocol used can then be swapped out without affecting the rest of the system.

-

In fact, it is common to use a consensus service, such as Zookeeper, to provide distributed consensus within a system. Designing applications as clients to a consensus service allows for even better seperation of concerns, and is done at Google with Chubby.

-

Reliable Replicated State Machines

-

A replicated state machine (RSM) is a system that maintains multiple copies of the same process by executing the same commands on all copies. Any deterministic program can be implemented as a highly available service by turning it into an RSM.

-

The order of operations is determined by a consensus algorithm running in a lower layer of the system. However, since there can be nodes part of a consensus group that aren't part of a given consensus quorum, nodes need to synchronize state from peers, which can be done using a sliding-window protocol.

-

Reliable Replicated Datastores and Configuration Stores

-

Many non-distributed consensus-based storage systems use timestamps to determine order of operations, but this approach doesn't work in distributed systems (due to clock drift). While some systems (like Google's Spanner) use a probabilistic approach to determining timestamps (TrueTime), this gets complicated and expensive. There is inherent uncertainty in the time at any given nodes, and Spanner tries to account for this uncertainty, which also minimizing it through periodic slow-downs to resynchronize clocks.

-

Instead, distributed consensus protocols can be used when replicating data across multiple nodes. However, these protocols can be slow, especially since operations on a storage system are often small and frequent, yet consensus protocols require multiple round trips to complete.

-

Highly Available Processing Using Leader Election

-

Leader election is an equivalent problem to distributed consensus, and is used in distributed systems to ensure that only one node is responsible for processing requests at a time. This might be used in cases where a single leader node is able to process requests, but it is often the case that a single leader needs to delegate work to a pool of worker nodes (like GFS or BigTable).

-

With this pattern, the leader election service is off of the critical path of the system, and so it has a smaller impact on the system's throughput.

-

Distributed Coordination and Locking Services

-

A barrier is a synchronization primitive that allows a group of nodes to wait until all nodes have reached a certain point before continuing. This lets you split a distributed computation into multiple stages that must be completed in order. For instance, in MapReduce, a barrier is used to ensure that all mappers have finished before reducers start.

-

While barriers can be implemented as a single coordinator node, this introduces a single point of failure. Instead, once can use an RSM to implement a barrier, which is done by Zookeeper's implementation of the barrier pattern.

-

Distributed locking is a more general distributed synchronization primitive that allows for mutual exclusion of shared resources among nodes. In practice, it is essential to use renewable leases with timeouts to prevent deadlocks. Distributed locks are another fairly low-level primitive, and it is often a good idea to use a higher-level abstraction that provides distributed transactions.

-

Reliable Distributed Queuing and Messaging

-

It is common to use a lease mechanism to ensure that only one node processes a message from a queue at a time, while also allowing for failover in case the node processing the message fails.

-

Queuing is also a powerful abstraction that can be used to implement other patterns like atomic broadcast and publish-subscribe messaging systems, where messages need to be reliably delivered to multiple nodes. This is useful for things like sending notifications to multiple clients, but can be used in other applications like distributed cache coherence. Furthermore, queuing as workload distribution can be used to distribute work among a pool of worker nodes

-

Distributed Consensus Performance

-

People are apparently pretty pessimistic about the performance of distributed consensus algorithms, but they can actually be quite fast. According to Google SREs, this is not the case.

-

There are many factors that can affect the performance of distributed consensus algorithms, including:

- -

One common performance pitfall with single-leader replication is that a client's perceived latency is proportional to the round-trip time between the client and the leader.

-

Multi-Paxos: Detailed Message Flow

-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/mutual-exclusion.html b/site/ds-backup/mutual-exclusion.html deleted file mode 100644 index ac5ae98..0000000 --- a/site/ds-backup/mutual-exclusion.html +++ /dev/null @@ -1,231 +0,0 @@ - - - - - - Mutual Exclusion - - - - - -
- -

Mutual Exclusion

-
- Last modified: 2024-04-27 - -
-
-

Distributed Mutual Exclusion

-

We want the same old mutual exclusion via locking, but in a distributed system. The trick is to keep a consistent ordering of locking events on every node in the system.

-

Implementation

-

Each message carries a timestamp $T_m$, and a sequence number $s$.

-

There are three message types:

- -

Each node maintains:

- -

On request receive:

- -

On receiving release:

- -

On acknowledge receive:

- -

To acquire the lock:

- -
- -
- - \ No newline at end of file diff --git a/site/ds-backup/ordering-events-in-distributed-systems.html b/site/ds-backup/ordering-events-in-distributed-systems.html deleted file mode 100644 index 6d5a0df..0000000 --- a/site/ds-backup/ordering-events-in-distributed-systems.html +++ /dev/null @@ -1,293 +0,0 @@ - - - - - - Ordering Events In Distributed Systems - - - - - -
- -

Ordering Events In Distributed Systems

-
- Last modified: 2024-04-27 - -
-
-

Time, Clocks, and the Ordering of Events in a Distributed System

-

reading

-

Introduction

-

It would be convenient if we could order events in a distributed system, but it is impossible to do so in a way that is consistent with the order in which they actually occurred. However, we can define a partial ordering of events that is consistent with the order in which they occurred, and can extend this to a total ordering of events that is consistent with the partial ordering.

-

A partial ordering of objects is a relation that is reflexive, antisymmetric, and transitive. Conversely, a total ordering is a partial ordering that is also connexive, i.e. for any two objects $a$ and $b$, either $a \leq b$ or $b \leq a$.

-

Notation

- -

Clock Condition

-

If $a$ and $b$ are events in process $P_i$ and $a \to b$, then $C_i\langle a \rangle < C_i\langle b \rangle$.

-

If $a$ is the sending of a message and $b$ is the receipt of that message, then $a \to b$ and $C_i\langle a \rangle < C_j\langle b \rangle$.

-

To meet the clock condition, we must always increment the clock value of the process when an event occurs. Furthermore, if event $a$ is the sending of a message $m$ containing the clock value $T_m = C_i\langle a \rangle$, then upon receipt of $m$, the receiving process must set its clock value $C_j\langle b \rangle$ to a value greater than $T_m$.

-

Synchronized Access to a Shared Resource

-

We wish to find an algorithm for granting the resource to a process which satisfies the following three conditions:

- -

Assumptions

-

For any two processes $P_i$ and $P_j$, messages sent by $P_i$ are received in the order they were sent by $P_i$. This can be achieved using stop-and-wait or sliding window protocols.

-

Further, we assume that all messages are received within a bounded time, i.e. all messages will eventually be received.

-

Also note that this protocol requires active participation from all processes. If any one process fails, then the entire system stops, because the algorithm requires all processes to respond to a request before the resource can be granted.

-

Algorithm

-
    -
  1. To request the resource, a process $P_i$ sends a message $T_m:P_i$ request to all other processes, and puts that message on its request queue, where $T_m$ is the current clock value of $P_i$.
  2. -
  3. When process $P_j$ receives $T_m:P_i$ request, it places it on its request queue and sends a reply message to $P_i$ with its current clock value.
  4. -
  5. To release the resource, $P_i$ removes $T_m:P_i$ request from its request queue and sends a release message with its current clock value to all other processes.
  6. -
  7. When process $P_j$ receives a $P_i$ release message, it removes $T_m:P_i$ request from its request queue.
  8. -
  9. $P_i$ is granted the resource when the following conditions are met:
  10. -
  11. There is a $T_m:P_i$ request message in its queue ordered by $\Rightarrow$ before any other request messages.
  12. -
  13. $P_i$ has received a reply message from all other processes with a clock value greater than $T_m$.
  14. -
-

State Machine Perspective

-

The algorithm can be viewed as a state machine consisting of $C$, the set of commands and $S$, the set of states. The state machine is defined by the following function:

-

$$ -e: C \times S \to S -$$

-

Where $e(c, s) = s'$ means that executing the command $c$ in state $s$ results in a transition to state $s'$. In this algorithm, we have state $S$ corresponding to the queue of requests, and commands $C$ corresponding to the request and release messages for any given process.

-

$$ -C = { P_i \text{ request}, P_i \text{ release} } -$$

-

Executing $P_i \text{ request}$ in state $s$ results in a transition to state $s'$ where $s'$ is the state with the request message added to the queue. Executing $P_i \text{ release}$ in state $s$ results in a transition to state $s'$ where $s'$ is the state with the request message removed from the queue. Each process $P_i$ is its own state machine, and a process can execute a command time-stamped $T$ only if it has received all messages with time-stamps less than or equal to $T$.

-

Anomalous Behavior of Total Ordering

-

Consider a nationwide system of nodes. A person issues a request $a$ at node $A$, and after doing so, calls their friend in a different city at node $B$ to issue a request $b$. It is possible with the total ordering of events that $b \to a$, even though $a$ was issued before $b$. This is because the total ordering of events is not consistent with the order in which they actually occurred, but rather with the order in which they were observed. The message that would be able to establish the the actual ordering of events (the call) is external to the system.

-

More concretely, let $\mathscr{L}$ be the set of all events, and $L$ be the set of all events in our system.

-

$$L \subseteq \mathscr{L}$$

-

In the above scenario, we had $a \to_{\mathscr{L}} b$, but $b \nrightarrow_{L} a$. No algorithm based soley on the events in $L$, without knowledge more generally of $\mathscr{L}$ can order $a$ before $b$.

-

There are two approaches to fixing this issue:

-
    -
  1. Users must manually specify restraints on timestamps, ie the person at $A$ must tell the person at $B$ that $T_a < T_b$. This is not a great solution.
  2. -
  3. Construct a system of clocks that satisfy the following condition: if $a \to_{\mathscr{L}} b$, then $C\langle a \rangle < C\langle b \rangle$.
  4. -
-

Physical Clocks

-

Let $C_i(t)$ be the value clock $C_i$ at time $t$. Assume that $C_i(t)$ is a continuous and differentiable function of $t$, except for isolated points where it jumps when the clock is reset. Note that a discrete clock can be modeled as a continuous clock with an error of up to $\epsilon = \frac{1}{2} \text{ tick}$.

-

Then $\frac{dC_i(t)}{dt}$ is the rate at which the clock is running at time $t$. If $\frac{dC_i(t)}{dt} = 1$, then the clock is running at the correct rate. We assume that the following holds:

-

$$ -\exists \kappa \ll 1 \text{ such that } \forall i, t: |\frac{dC_i(t)}{dt} - 1| < \kappa -$$

-

And in fact, for typical quartz oscillator clocks, $\kappa \le 10^{-6}$.

-

This however isn't enough. For an effective clock system, we also want our clocks to be synchronized such that...

-

$$ -\forall i, j, t: |C_i(t) - C_j(t)| < \epsilon -$$

-

...where $\epsilon$ is the maximum error in the clocks. Since physical clocks will never run at exactly the same rate, they will tend to drift apart over time. To correct for this, we can use a synchronization algorithm to periodically reset the clocks to a common time.

-

Letting $\mu$ be a number such that if event $a$ occurs at physical time $t$, and event $b$ is an event in another process that satisfies $a \to b$, then $b$ occurs later than physical time $t + \mu$. So $\mu$ is the maximum time it takes for a message to be sent from one process to another.

-

To avoid anomalies, we must ensure...

-

$$ -\forall i, j, t: C_i(t) < C_j(t + \mu), \text{ or, equivalently, } C_i(t + \mu) - C_j(t) > 0 -$$

-

And relating this to $\kappa$...

-

$$ -\forall i, j, t: C_i(t + \mu) - C_j(t) > (1 - \kappa)\mu -$$

-

And calculating the maximum error in the clocks...

-

$$ -\frac{\epsilon}{1 - \kappa} \le \mu -$$

-

Clock Synchronization Algorithm

-

Importantly, one must always synchronize clocks in the forward direction, i.e. $C_i(t) < C_j(t + \mu)$. If $C_i(t) > C_j(t + \mu)$, then $C_i(t + \mu) - C_j(t) < 0$, which is not allowed.

-

Let $m$ be a message sent at time $t$, and received at time $t'$. Let $v_m = t' - t$ be the total delay of the message. Although the delay of a message is not known by any given process, we assume the receiver has some lower bound on the delay, $\mu_m$. Define $\zeta_m = v_m - \mu_m$ to be the unpredictable delay of the message.

-

Now, define the following rules for our physical clocks:

-
    -
  1. $\forall i$ if $P_i$ does not receive a message at physical time $t$, then $C_i$ is differentiable at $t$ and $\frac{dC_i(t)}{dt} > 0$.
  2. -
  3. If $P_i$ sends a message $,$ at physical time $t$, then $m$ contains a timestamp $T_m = C_i(t)$. Upon receiving a message $m$ at physical time $t'$, $P_j$ sets $C_j(t') = \max(\lim_{\delta \to 0} C_j(t' - \delta), T_m + \mu_m)$.
  4. -
-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/paxos-intro.html b/site/ds-backup/paxos-intro.html deleted file mode 100644 index f6903e0..0000000 --- a/site/ds-backup/paxos-intro.html +++ /dev/null @@ -1,224 +0,0 @@ - - - - - - Paxos Intro - - - - - -
- -

Paxos Intro

-
- Last modified: 2024-04-27 - -
-
-

Paxos Introduction

-

FLP Impossibility Result

-

It's impossible for a deterministic protocol to guarantee consensus in bounded time in an asynchronous distributed system. The progress and safety of a system are at odds with each other.

-

Paxos makes the decision to always be safe, and is able to make progress and avoid blocking as long as the majority of nodes are up and there aren't further failures.

-

State Machine Replication

-

Order events/operations into an append-only log. Consensus is easy if only one client request is handled at a time.

-

Select a leader for clients to send requests to, and define the ordering at that leader. If any leader fails or is slow, elect a new leader (can keep doing this repeatedly). Then, each leader proposes a value that all nodes should agree on.

-

Leader election is where Paxos comes in.

-

Paxos, the algorithm

-
Proposer:
-  Prepare(n) -> Promise(n, n', v')
-  Accept(n, v) -> Accepted(n, v)
-
-Acceptor:
-  Promise(n, n', v') -> Prepare(n)
-  Accepted(n, v) -> Accept(n, v)
-
-

Phase 1: Prepare

- -

Phase 2: Accept

- -
- -
- - \ No newline at end of file diff --git a/site/ds-backup/paxos-made-simple.html b/site/ds-backup/paxos-made-simple.html deleted file mode 100644 index 534d35a..0000000 --- a/site/ds-backup/paxos-made-simple.html +++ /dev/null @@ -1,266 +0,0 @@ - - - - - - Paxos Made Simple - - - - - -
- -

Paxos Made Simple

-
- Last modified: 2024-04-27 - -
-
-

Paxos Made Simple

-

reading

-

The Consensus Algorithm

-

The Problem

-

Consider a set of processes that can propose values. A consensus algorithm ensures that a single value is chosen and agreed upon. For safety, we must have...

- -

And it assumes an asynchronous, non-byzantine network in which nodes are fail-stop.

-

Choosing a Value

-

In paxos there are 3 types of agents

- -

P1: An acceptor must accept the first proposal it receives

-

This guarantees that some value is accepted by each node that is proposed to, but it doesn't prevent situations where no proposal is accepted by a majority of acceptors.

-

P2: If a proposal with value $v$ is chosen, then every higher-numbered proposal accepted by any acceptor has value $v$

-

This guarantees that only a single value is chosen, since unique proposal numbers can be used to decide between accepted values.

-
P2a: If a proposal with value $v$ is chosen, then every higher-numbered proposal accepted by any acceptor has value $v$
-

This is a stronger version of P2 that ensures previous values are not forgotten/overridden.

-

However, P2a contradicts P1, since if a proposer "wakes up" after having been out of commission, it must accept whatever value is proposed first. We thus strengthen even further to...

-
P2b: If a proposal with value $v$ is chosen, then every higher-numbered proposal issues by any proposer has value $v$
-

This shifts the burden of remembering and staying consistent with the chosen value to the proposers instead of the acceptors. We then have...

-

$$ -\text{P2b} \to \text{P2a} \to \text{P2} -$$

-

In order to implement P2b, we must further constrain our algorithm's behavior to...

-
P2c: For any $v$ and $n$, if a proposal with value $v$ and number $n$ is issued, there is a set $S$ consisting of a majority of acceptors such that either (a) no acceptor in $S$ has accepted any proposal numbered less than $n$, or (b) $v$ is the value of the highest-numbered proposal among all proposals numbered less than $n$ accepted by the acceptors in $S$
-

To satisfy P2b, we must maintain P2c as an invariant. To make sure this invariant holds, proposers proposing a proposal numbered $n$ must learn the highest-numbered proposal with a number less than $n$ that has been accepted by a majority of acceptors, and propose that value if it exists.

-

Proposition Algorithm

-
    -
  1. A proposer chooses a new proposal number $n$ and sends a prepare request to each member of some set of acceptors, awaiting a response containing:
      -
    • A guarantee that this acceptor will never accept a proposal numbered less than $n$
    • -
    • The proposal with the highest number less than $n$ that it has accepted, if any.
    • -
    -
  2. -
  3. If the proposer receives the requested responses from a majority of acceptors, it issues an accept request, which is a proposal with number $n$ and value $v$, where $v$ is the value of the highest-numbered proposal among the responses, or a value of the proposer's choice if no proposals in the responses were received.
  4. -
-

Acceptor Behavior

-

Acceptors can only receive prepare and accept requests, and can ignore any request without compromising safety (but certainly still sacrificing liveness).

-
P1a: An acceptor can accept a proposal numbered $n$ iff it has not responded to a prepare request numbered greater than $n$
-

which implies P1

-

Phases

-

Phase 1

- -

Phase 2

- -

Note that to increase performance, if an acceptor ignores a prepare or accept request because it has already received a prepare request with a higher number, it should notify the proposer with a reject message. This however doesn't change the correctness, and is thus optional.

-

Learning a Chosen Value

-

One option would be for each acceptor to send a message upon accepting a value to all learners, but this requires a whole lot of message passing. Another option is to maintain a set of distinguished learners, which after hearing of a majority acceptance, notify all other learners of the accepted value. The larger this set of distinguished learners, the more fault-tolerant the system, but also the more communication required.

-

Since messages can be dropped, a value can be chosen without any learner finding out. In this case, learners will find out the chosen value only after a new proposal is chosen. Learners can thus determine whether a value was chosen by following the same protocol to issue a new proposal as above.

-

Progress

-

It is entirely possible with the above protocol that multiple proposers indefinitely one-up each other between sending their propose and accept requests, such that all proposals are ignored. To prevent this, choose and maintain a single distinguished proposer, which is the only proposer allowed to issue proposals. If the distinguished proposer fails, a new one can be elected by the acceptors.

-

By FLP, any such leader election system must rely either on randomness, or real-time (i.e. timeouts).

-

Implementing a State Machine

-

Consider a system of clients that issue requests to execute commands on a cluster of single-threaded application servers. Each application server can be thought of as a deterministic state machine, where the ordering of requests to each server must be consistent for them to end up in the same state.

-

To guarantee consistent ordering of commands executed within our cluster, we implement a separate instance of Paxos, where the $i$th instance's chosen value determines the $i$th command executed on all application servers.

-

During normal operation, a single server is elected to be leader, which acts as the distinguished proposer, and is the only server allowed to issue proposals. Clients then send their requests to this leader, which decides the sequence of commands globally. Any given instance of the protocol might fail, but regardless only one command can ever be chosen as the $i$th command to be executed.

-

For cases where some subsequence of commands are not yet determined, i.e. not chosen yet when a new leader takes over, the new leader issues phase 1 for all such instances (including the infinitely many commands greater than the largest command in our current sequence). Any values received in response are then proposed, but if an instance remains unconstrained (i.e. no value has been accepted), the leader can propose no-ops for the gaps in the sequence of commands before the last accepted command. It must do this before ever executing commands that come after these unconstrained slots.

-

After doing so, the leader can continue proposing any further commands requested by clients. The leader is allowed to propose command $i + \alpha$ before knowing the chosen command for $i$, meaning it can get up to $\alpha - 1$ commands ahead of itself (in the case where all commands less than $i + \alpha$ were dropped).

-

Once a leader has finished phase 1 for all commands thus far and afterwards, it only needs to complete phase 2 for each subsequent command requested, which is known to be the minimal algorithm for reaching consensus after phase 1.

-

To reiterate what was stated previously, in the case where a single leader is not elected, progress is not guaranteed, but safety is.

-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/primary-backup.html b/site/ds-backup/primary-backup.html deleted file mode 100644 index e3f3df7..0000000 --- a/site/ds-backup/primary-backup.html +++ /dev/null @@ -1,269 +0,0 @@ - - - - - - Primary Backup - - - - - -
- -

Primary Backup

-
- Last modified: 2024-04-27 - -
-
-

Primary Backup

-

Consider a highly available stateful service. It is easy to maintain consistency within one node, simply by performing operations in some well-defined (serializable) order. However, providing both availability and consistency is more of a challenge. One must provide a notion of having a single system, even if a server within the system fails.

-

Single Node KV Store

-

Consider an instance of redis with multiple clients reading and writing to it. You can think of this system more abstractly as a state machine, where each client applies an operation that changes the state of the system.

-

State Machine Replication

-

Replicate state machines across multiple servers. If you apply the same set of operations to each server in the same order, their ending states must be in the same state. This holds so long as the effect of each operation is deterministic.

-

Example: Virtual Machine Replication

-

Take a single VM running a single application. Create $n$ copies of this VM, and feed each instance the exact same inputs (packets, interrupts, instructions). Then, all $n$ VMs will have the same behavior.

-

Any time you introduce randomness into a system, you need to ensure that the randomness is deterministic. This mechanism for VM replication assumes you are only using a single core, and all operations are deterministic.

-

Two Servers (Primary-Backup)

-

At any given time, clients speak to only one server (the primary). Data is replicated on primary and backup servers, and if the primary fails, the backup is elected as the new primary.

-

The goals of doing this is to increase the availability and reliability of the system in the face of failures.

-

Basic Operations

- -

Key Assumptions

- -

Key Challenges

- -

The View Service

-

The view service is a server that provides a consistent view of the system. Clients ask the view service for the primary server's address in order to find out where to send operations. Even if the view server incorrectly identifies failure, the system will still be consistent.

-

The view server is the only authority on who the primary is. This makes it a single point of failure. The hard part is that we need to be able to guarantee only one primary at a time, while not needing to ping the view server on every operation.

-

This system needs to be able to tolerate any individual server failing, while still serving client requests.

-

Detecting Server Failures

- -

When the view server detects a failure, a new view (state of the system sent in ping responses) is created.

-

Primary Failures

- -

If primary dies with no idle servers available, then the backup becomes the primary and there is no backup.

-

Managing Servers

-

Keep a pool of idle servers that can be promoted to backup. If primary dies, create new view with old backup as primary and idle server as backup. If the backup dies, create a new view with idle server as new backup.:

-

Split Brain

-

In the case where a primary appears to be offline, but is really just partitioned from the view server, the view server may elect a new primary. This leads to a split brain scenario, where two primaries are elected.

-

The important part is that two servers can think they are the primary, but it can never be the case that two servers act as the primary.

-

Rules

-
    -
  1. Primary in view $i + 1$ must have been the backup, or the primary in view $i$ (besides the first view).
  2. -
  3. Primary must wait for backup to accept/execute each operation before replying to client (if there is one).
  4. -
  5. Backup must accept forwarded requests only if view is correct.
  6. -
  7. Non-primary must reject client requests.
  8. -
  9. Every operation must be before or after state transfers (not during).
  10. -
-
- -
- - \ No newline at end of file diff --git a/site/ds-backup/scaling-web-services.html b/site/ds-backup/scaling-web-services.html deleted file mode 100644 index 7fe0653..0000000 --- a/site/ds-backup/scaling-web-services.html +++ /dev/null @@ -1,217 +0,0 @@ - - - - - - Scaling Web Services - - - - - -
- -

Scaling Web Services

-
- Last modified: 2024-04-27 - -
-
-

Scaling Web Services with Distributed Architectures

-

Two Tier Architecture

-

Many companies adopted the idea of a two tier architecture for web services. The idea was to have a scalable number of frontend servers, mapping clients to one of those servers. Then, you could also have a scaled out backend with multiple storage severs, mapping frontend servers to the data they needed in the storage servers.

-

Two-Tier RESTful Web Architecture

-

Keep a scalable number stateless servers hosting the client and running application code. Importantly, it doesn't matter if a client server crashes, since the user can just connect to some other client server. At the same time, run a scalable number of backend servers running in a carefully designed distributed system, often using primary/backup or paxos for high availability and fault tolerance. Anything that needs to be persistent across crashes should be handled on the backend.

-

Load Balancing

-

Typically, the layer between the tiers of such architectures are composed of load balancers, which need to map any given client to a desirable front-end server. This needs to be consistent per connection, which can be done with hash(clientIP, port) -> clientServerIP. Additionally, you need to map each client server to a storage server, which can be done by hash(key) -> storageServerIP, where the key is some identifier for the location of the data in a given query to our storage system.

-

Importantly, the system should automatically adapt to the addition of any type of server.

-

Three-Tier Web Architecture: Look-aside Caching

-

Also maintain a set of cache servers to offload queries to the storage server. Client servers first send their query to the cache server, and if there is a cache miss they then fall back to the storage servers, and then write the retrieved data to the cache (this is look-aside caching).

-

There are other ways to do caching. For instance, the cache could directly retrieve values transparently to the client server, such that cache misses don't need to be handled. However, this tightly couples the cache and the storage server, often requiring that all queries pass through the caching layer, making it harder to design the two services independently.

-

Of course, caching needs to be scalable as well. Cache servers don't necessarily need to be 1:1 with client/storage servers, but they should be able to handle the load of the client servers they are caching for adaptively. They should also ideally have lower latency than actual queries executed in the storage layer.

-

Newer Architectures

-

Edge Computing

-

Moving data processing closer to the client.

-

Users are often globally distributed, leading to higher latency and thus worse user experience.

-

To mitigate this, large applications will often be globally distributed in edge data centers, ideally with one reasonably near every user. Often, only the web and cache (RESTful) layer are present on the edge, and content can be distributed by "pushing" it to the edge before it is ever requested.

-

In tandem, core data centers host the web, cache, and storage layer, replicating all of this across data centers for disaster tolerance.

-

Service Oriented Architecture

-

Services define external interfaces, and often requires distributed systems that work in a hostile environment. All teams expose the data/functionality through this interface, and all communication happens through network calls. Each service runs as a standalone product with its own service level agreement to its clients.

-

Microservices

-

Organize complex distributed applications as a large number of independent services communicating through RPC, each using primary/backup or paxos for high availability and fault tolerance.

-

This allows for independent development of components of a larger system, where each component can scale independently.

-
- -
- - \ No newline at end of file diff --git a/site/idx/file_to_keywords.json b/site/idx/file_to_keywords.json deleted file mode 100644 index 0420684..0000000 --- a/site/idx/file_to_keywords.json +++ /dev/null @@ -1 +0,0 @@ -{"requirements.txt": [["markdown", 1.0]], "test.md": [["icloud", 0.4263], ["ai", 0.4165], ["apple", 0.3955], ["opting", 0.3357], ["privacy", 0.2953]], "distributed-systems/dynamo-db.md": [["dynamo", 0.4902], ["storage", 0.4294], ["amazon", 0.3672], ["versioning", 0.3414], ["consistency", 0.2888]], "distributed-systems/disconnected-operation.md": [["synchronization", 0.4542], ["cloud", 0.3663], ["reconnection", 0.3533], ["versioning", 0.3343], ["writes", 0.3293]], "distributed-systems/bigtable.md": [["bigtable", 0.5412], ["mapreduce", 0.4804], ["storage", 0.4401], ["database", 0.3543], ["locality", 0.2955]], "distributed-systems/RPC.md": [["compiler", 0.3408], ["executes", 0.3339], ["rpc", 0.3243], ["tcp", 0.3176], ["nfs", 0.3108]], "distributed-systems/paxos-intro.md": [["paxos", 0.535], ["replication", 0.4036], ["protocol", 0.3413], ["proposer", 0.3117], ["algorithm", 0.3044]], "distributed-systems/clocks.md": [["clock", 0.5469], ["latency", 0.49], ["ntp", 0.4015], ["synchronized", 0.3572], ["scheduling", 0.3458]], "distributed-systems/consistent-global-state.md": [["distributed", 0.477], ["process", 0.3502], ["communicate", 0.3423], ["observer", 0.3402], ["asynchronous", 0.3233]], "distributed-systems/ordering-events-in-distributed-systems.md": [["ordering", 0.438], ["synchronization", 0.419], ["protocol", 0.3675], ["event", 0.3661], ["transitive", 0.3556]], "distributed-systems/mutual-exclusion.md": [["queue", 0.466], ["locking", 0.3798], ["distributed", 0.361], ["exclusion", 0.3354], ["ordering", 0.2852]], "distributed-systems/managing-critical-state.md": [["availability", 0.4744], ["consistency", 0.4555], ["replication", 0.4459], ["mapreduce", 0.3795], ["acid", 0.2994]], "distributed-systems/non-blocking-two-phase-commit.md": [["locking", 0.4176], ["concurrently", 0.4075], ["rpcs", 0.3862], ["paxos", 0.3762], ["commit", 0.3353]], "distributed-systems/scaling-web-services.md": [["server", 0.4431], ["scalable", 0.4266], ["tier", 0.424], ["microservices", 0.3577], ["architecture", 0.3545]], "distributed-systems/sharding.md": [["shard", 0.494], ["hashing", 0.4583], ["sharding", 0.4205], ["bucket", 0.3689], ["hotspot", 0.2619]], "distributed-systems/primary-backup.md": [["replication", 0.5389], ["availability", 0.4884], ["consistency", 0.3964], ["vms", 0.3926], ["redis", 0.355]], "distributed-systems/two-phase-commit.md": [["locking", 0.4], ["commit", 0.3831], ["consistency", 0.3553], ["atomicity", 0.3455], ["phase", 0.3169]], "distributed-systems/load-balancing.md": [["paxos", 0.4446], ["queueing", 0.4206], ["utilization", 0.361], ["shard", 0.351], ["scaling", 0.3257]], "distributed-systems/paxos-made-simple.md": [["paxos", 0.4603], ["consensus", 0.3711], ["p1a", 0.3458], ["proposer", 0.3263], ["node", 0.3086]], "distributed-systems/google-file-system.md": [["filesystems", 0.5367], ["gfs", 0.5088], ["concurrency", 0.3567], ["workload", 0.3519], ["writes", 0.3345]], "distributed-systems/distributed-cache-coherence.md": [["caching", 0.5308], ["distributed", 0.3876], ["maintaining", 0.3788], ["lease", 0.367], ["linearizable", 0.2866]], "distributed-systems/paxos-architecture.md": [["paxos", 0.5411], ["lease", 0.3247], ["overhead", 0.2923], ["architecture", 0.2855], ["synchronized", 0.229]], "distributed-systems/consistency.md": [["consistency", 0.5472], ["concurrency", 0.4489], ["serializability", 0.293], ["storage", 0.2776], ["kv", 0.2253]], "ds-backup/RPC.md": [["compiler", 0.3408], ["executes", 0.3339], ["rpc", 0.3243], ["tcp", 0.3176], ["nfs", 0.3108]], "ds-backup/paxos-intro.md": [["paxos", 0.535], ["replication", 0.4036], ["protocol", 0.3413], ["proposer", 0.3117], ["algorithm", 0.3044]], "ds-backup/clocks.md": [["clock", 0.5469], ["latency", 0.49], ["ntp", 0.4015], ["synchronized", 0.3572], ["scheduling", 0.3458]], "ds-backup/consistent-global-state.md": [["distributed", 0.477], ["process", 0.3502], ["communicate", 0.3423], ["observer", 0.3402], ["asynchronous", 0.3233]], "ds-backup/ordering-events-in-distributed-systems.md": [["ordering", 0.438], ["synchronization", 0.419], ["protocol", 0.3675], ["event", 0.3661], ["transitive", 0.3556]], "ds-backup/mutual-exclusion.md": [["queue", 0.466], ["locking", 0.3798], ["distributed", 0.361], ["exclusion", 0.3354], ["ordering", 0.2852]], "ds-backup/managing-critical-state.md": [["availability", 0.4744], ["consistency", 0.4555], ["replication", 0.4459], ["mapreduce", 0.3795], ["acid", 0.2994]], "ds-backup/scaling-web-services.md": [["server", 0.4431], ["scalable", 0.4266], ["tier", 0.424], ["microservices", 0.3577], ["architecture", 0.3545]], "ds-backup/primary-backup.md": [["replication", 0.5389], ["availability", 0.4884], ["consistency", 0.3964], ["vms", 0.3926], ["redis", 0.355]], "ds-backup/paxos-made-simple.md": [["paxos", 0.4603], ["consensus", 0.3711], ["p1a", 0.3458], ["proposer", 0.3263], ["node", 0.3086]], "digital-design/karnaugh-maps.md": [["verilog", 0.4887], ["karnaugh", 0.3782], ["adjacency", 0.3006], ["trigger", 0.2939], ["map", 0.2869]], "digital-design/waveform-diagram.md": [["verilog", 0.4661], ["bit", 0.2895], ["delay", 0.2775], ["waveform", 0.2696], ["bus", 0.2197]], "digital-design/sequential-logic.md": [["register", 0.4246], ["sequential", 0.3561], ["flop", 0.3489], ["sl", 0.295], ["synchronizing", 0.2687]], "digital-design/combinational-logic.md": [["combinational", 0.4503], ["boolean", 0.4151], ["logic", 0.3916], ["passenger", 0.3791], ["table", 0.3037]], "digital-design/quartus-workflow.md": [["modelsim", 0.3923], ["runlab", 0.3784], ["testbench", 0.3522], ["verilog", 0.3363], ["waveform", 0.3]], "digital-design/system-verilog.md": [["verilog", 0.6229], ["systemverilog", 0.4332], ["module", 0.277], ["wire", 0.2574], ["programmatically", 0.2337]], "natural-language-processing/multinomial-logistic-regression.md": [["classify", 0.4559], ["logistic", 0.4155], ["label", 0.4098], ["multinomial", 0.4039], ["vocabulary", 0.2549]], "systems-research/hints-for-computer-system-design.md": [["cache", 0.4524], ["hardware", 0.2596], ["function", 0.2528], ["delta", 0.1998], ["adaptive", 0.1857]], "systems-research/how-to-read-a-paper.md": [["reading", 0.4828], ["paper", 0.4279], ["pas", 0.2984], ["annotate", 0.2918], ["keshav", 0.2454]], "systems-research/strong-inference.md": [["science64_strong_inference", 0.6038], ["hypothesis", 0.4319], ["effective", 0.3194], ["scientist", 0.2934], ["analyze", 0.2859]], "systems-research/paper-review-template.md": [["template", 0.464], ["tag", 0.3679], ["paper", 0.3226], ["reviewing", 0.3111], ["summary", 0.2804]], "algorithms/BFS.md": [["algorithm", 0.4728], ["graph", 0.4419], ["breadth", 0.4355], ["level_order_traversal", 0.3579], ["discover", 0.2853]], "algorithms/DAGs.md": [["topological_sort", 0.4968], ["ordering", 0.3782], ["graph", 0.3637], ["acyclic", 0.3044], ["dag", 0.2995]], "algorithms/DFS.md": [["dfs_recursive", 0.445], ["tree", 0.3899], ["depth", 0.3583], ["traversal", 0.3479], ["edge", 0.2846]], "algorithms/graphs-intro.md": [["adjacency", 0.5552], ["build_adjacency_matrix", 0.5426], ["graph", 0.5334], ["vertex", 0.5032], ["build_adjacency_list", 0.4955]], "algorithms/divide-and-conquer.md": [["algorithm", 0.4644], ["recursively", 0.3732], ["conquering", 0.3468], ["dividing", 0.3384], ["bisection", 0.2942]], "algorithms/stable-matching.md": [["stable_matchings", 0.5675], ["find_stable_matchings", 0.5646], ["matchings", 0.5556], ["algorithm", 0.3434], ["student", 0.3151]], "algorithms/bipartite-graphs.md": [["bipartite", 0.4921], ["graph", 0.3732], ["coloring", 0.3309], ["scheduling", 0.3048], ["v_2", 0.2096]], "algorithms/induction.md": [["induction", 0.4424], ["inductive", 0.3609], ["proof", 0.3013], ["pigeonhole", 0.2988], ["mathematics", 0.2012]], "algorithms/approximation-algorithms.md": [["algorithm", 0.4483], ["approximation", 0.395], ["cover", 0.3137], ["vertex", 0.2774], ["satisfiability", 0.2773]], "algorithms/linear-programming.md": [["hyperplanes", 0.3816], ["optimization", 0.3593], ["polytopes", 0.3324], ["bmatrix", 0.2313], ["lp", 0.2131]], "algorithms/dynamic-programming.md": [["scheduling", 0.5168], ["knapsack_rec", 0.4175], ["algorithm", 0.408], ["max_weighted_interval_subset", 0.363], ["runtime", 0.3328]], "algorithms/connected-components.md": [["breadth", 0.4321], ["adjacency", 0.4223], ["traversal", 0.364], ["algorithm", 0.359], ["connected_components", 0.3547]], "algorithms/runtime.md": [["efficiency", 0.3949], ["asymptotic", 0.381], ["complexity", 0.3608], ["notation", 0.358], ["n_0", 0.2598]], "algorithms/greedy-algorithms.md": [["interval_scheduling", 0.6847], ["scheduling", 0.5847], ["partition_intervals", 0.5152], ["greedy", 0.3418], ["algorithm", 0.3018]], "algorithms/network-flows.md": [["algorithm", 0.4258], ["edge", 0.3879], ["flow", 0.3425], ["routing", 0.3341], ["partitioning", 0.3139]], "algorithms/tree-intro.md": [["tree", 0.4087], ["induction", 0.3125], ["vertex", 0.2939], ["edge", 0.2337], ["acyclic", 0.2218]], "performance-engineering/efficiently-implementing-state-pattern-JVM.md": [["concurrent", 0.4253], ["implementation", 0.3786], ["jvm", 0.284], ["transition", 0.2832], ["documentstate", 0.227]], "networks/reference.md": [["network", 0.4549], ["system", 0.4386], ["textbook", 0.4211], ["systemsapproach", 0.3592], ["html", 0.1629]], "networks/sockets.md": [["sock_stream", 0.5397], ["sock_dgram", 0.5343], ["socket", 0.532], ["ipproto_tcp", 0.5097], ["server_port", 0.4879]], "machine-learning-for-big-data/intro-mapreduce-spark.md": [["mapreduce", 0.5108], ["dataflow", 0.42], ["cluster", 0.412], ["hdfs", 0.387], ["computing", 0.3716]], "linear-algebra/cheatsheet.md": [["vector", 0.5098], ["geometric", 0.3263], ["v_3", 0.307], ["interpretation", 0.2628], ["projection", 0.2618]], "linear-algebra/python-cheatsheet.md": [["cross_product", 0.4152], ["numpy", 0.4074], ["v_sum", 0.3619], ["is_orthonormal_matrix", 0.3488], ["l2_norm", 0.3211]], "linear-algebra/elementry-linear-algebra.md": [["bmatrix", 0.3734], ["ax", 0.2736], ["system", 0.2671], ["b_m", 0.243], ["equation", 0.2267]], "operating-systems/reference.md": [["process", 0.4715], ["concurrency", 0.4636], ["kernel", 0.4384], ["storage", 0.4218], ["osppv2", 0.2646]], "tmp/pairing-algorithm.md": [["pairing", 0.3901], ["calculate_pairs", 0.3447], ["algorithm", 0.297], ["student_i", 0.2927], ["bmatrix", 0.2884]], "natural-language-processing/reading/information-retrieval.md": [["retrieval", 0.5093], ["indexing", 0.4538], ["search", 0.4492], ["document", 0.3901], ["ir", 0.3764]], "signal-conditioning/lecture-notes/lecture-1.md": [["swap", 0.4059], ["device", 0.3304], ["reusable", 0.3164], ["energy", 0.3123], ["cost", 0.2405]], "signal-conditioning/lecture-notes/lecture-5.md": [["circuit", 0.4448], ["resistance", 0.3097], ["algorithm", 0.2522], ["theorem", 0.2447], ["thevenin", 0.2351]], "signal-conditioning/lecture-notes/lecture-4.md": [["dissipation", 0.3455], ["circuit", 0.3226], ["r_2", 0.3053], ["resistance", 0.2859], ["node", 0.2203]], "signal-conditioning/lecture-notes/lecture-3.md": [["resistance", 0.5463], ["resistance", 0.5411], ["resistor", 0.4816], ["dissipated", 0.3126], ["watt", 0.2838]], "signal-conditioning/lecture-notes/lecture-2.md": [["electricity", 0.5323], ["electron", 0.4643], ["current", 0.4361], ["ampere", 0.4282], ["circuit", 0.3952]], "signal-conditioning/lecture-notes/lecture-6.md": [["inductor", 0.4783], ["capacitance", 0.362], ["ac", 0.2987], ["jq", 0.212], ["imaginary", 0.1793]], "algorithms/patterns/BFS.md": [["breadth", 0.5243], ["algorithm", 0.4236], ["bfs", 0.3999], ["graph", 0.371], ["search", 0.2912]], "algorithms/patterns/sliding-window.md": [["algorithm", 0.3782], ["contiguous", 0.3383], ["subarray", 0.3372], ["sliding", 0.3177], ["window", 0.2965]], "algorithms/problems/graphs-and-trees.md": [["tree", 0.4067], ["induction", 0.3292], ["graph", 0.3185], ["leaf", 0.2984], ["edge", 0.228]], "algorithms/practice/4.md": [["algorithm", 0.381], ["interval", 0.3741], ["classroom", 0.3504], ["allocated", 0.3347], ["disjoint", 0.3143]], "networks/0-foundation/information-theory.md": [["bandwidth", 0.5855], ["transmit", 0.4057], ["wireless", 0.402], ["frequency", 0.3279], ["shannon", 0.2864]], "networks/0-foundation/3-performance.md": [["bandwidth", 0.6557], ["throughput", 0.5869], ["latency", 0.5295], ["performance", 0.4724], ["bit", 0.3718]], "networks/0-foundation/1-network-components-and-protocols.md": [["network", 0.5043], ["802", 0.4945], ["protocol", 0.434], ["node", 0.428], ["interface", 0.3331]], "networks/0-foundation/2-physical-layer.md": [["modulator", 0.4846], ["coding", 0.4081], ["decode", 0.4004], ["analog", 0.3573], ["signal", 0.3484]], "networks/3-network/ARP.md": [["arp", 0.5963], ["protocol", 0.3984], ["ip", 0.3206], ["mac", 0.2397], ["bonjour", 0.2246]], "networks/3-network/routing.md": [["routing", 0.4634], ["forwarding", 0.424], ["router", 0.3861], ["packet", 0.3355], ["updateroutingtable", 0.3045]], "networks/3-network/internetworking.md": [["routing", 0.4111], ["internetworking", 0.3868], ["packet", 0.3646], ["ipv4", 0.3417], ["qos", 0.3031]], "networks/3-network/networking-services.md": [["networking", 0.4879], ["packet", 0.4438], ["bandwidth", 0.3681], ["multiplexing", 0.3483], ["qos", 0.2803]], "networks/3-network/DHCP.md": [["dhcp", 0.6157], ["protocol", 0.4097], ["ip", 0.3915], ["udp", 0.3537], ["configuration", 0.2797]], "networks/3-network/ICMP.md": [["traceroute", 0.6086], ["ping", 0.565], ["icmp", 0.4716], ["routing", 0.4485], ["packet", 0.4112]], "networks/3-network/motivation.md": [["network", 0.3695], ["routing", 0.3683], ["protocol", 0.3584], ["qos", 0.2973], ["scaling", 0.2794]], "networks/3-network/global-internet.md": [["router", 0.4611], ["bgp", 0.4375], ["ospf", 0.4293], ["infrastructure", 0.4168], ["protocol", 0.3737]], "networks/3-network/BGP.md": [["bgp", 0.6095], ["route", 0.5347], ["traffic", 0.4734], ["protocol", 0.3916], ["interconnected", 0.3689]], "networks/2-direct-links/framing.md": [["protocol", 0.5077], ["byte", 0.4491], ["ddcmp", 0.3752], ["sdlc", 0.3633], ["frame", 0.3449]], "networks/2-direct-links/retransmission.md": [["delay", 0.4484], ["retransmission", 0.4009], ["arq", 0.4006], ["acknowledged_frames", 0.3765], ["rtt", 0.3685]], "networks/2-direct-links/errors.md": [["hamming_encode", 0.4146], ["bit", 0.3571], ["detect", 0.301], ["redundancy", 0.3005], ["internet_checksum", 0.2947]], "networks/2-direct-links/wireless.md": [["802", 0.5291], ["protocol", 0.3144], ["collision", 0.2815], ["node", 0.2589], ["maca", 0.2535]], "networks/2-direct-links/multiple-access.md": [["multiplexing", 0.5238], ["gsm", 0.4066], ["bandwidth", 0.3953], ["fdm", 0.3668], ["centralized", 0.3613]], "networks/2-direct-links/switching.md": [["bridgemap", 0.4838], ["ethernet", 0.4562], ["bridge_tab_size", 0.4099], ["switch", 0.3524], ["macaddr", 0.268]], "networks/5-application/HTTP.md": [["http", 0.5373], ["server", 0.3515], ["fetching", 0.3419], ["tcp", 0.3418], ["request", 0.328]], "networks/5-application/overview.md": [["udp", 0.5663], ["tcp", 0.5575], ["application", 0.4433], ["skype", 0.3614], ["layer", 0.3105]], "networks/5-application/CDNs.md": [["cdns", 0.5281], ["cache", 0.4743], ["proxy", 0.4266], ["dns", 0.4249], ["server", 0.4195]], "networks/5-application/DNS.md": [["dns", 0.5449], ["nameservers", 0.4568], ["resolver", 0.4033], ["hierarchy", 0.3758], ["namespaces", 0.3511]], "networks/4-transport/TCP.md": [["tcp", 0.5387], ["protocol", 0.4391], ["retransmits", 0.3451], ["timeout", 0.3398], ["rtt_", 0.2979]], "networks/4-transport/UDP.md": [], "networks/4-transport/ACK-clocking.md": [["tcp", 0.5277], ["congestion", 0.4405], ["buffer", 0.3912], ["ack", 0.3731], ["pipelining", 0.3679]], "networks/4-transport/transport-overview.md": [["packet", 0.4972], ["transport", 0.4917], ["bytestreams", 0.4896], ["layer", 0.3937], ["segment", 0.3834]], "networks/4-transport/flow-control.md": [["throughput", 0.4293], ["packet", 0.3929], ["rtt", 0.3429], ["buffer", 0.3187], ["recall", 0.2538]], "networks/1-physical/coding-and-modulation.md": [["coding", 0.4514], ["modulation", 0.3841], ["receiver", 0.3716], ["nrz", 0.3263], ["01001", 0.2614]], "networks/1-physical/media.md": [["interference", 0.4884], ["wireless", 0.4451], ["shielding", 0.4205], ["microwave", 0.3436], ["fiber", 0.2675]], "teaching/modern-java/lambdas-and-streams.md": [["java", 0.3967], ["arraylist", 0.3834], ["lambda", 0.3518], ["admins", 0.3407], ["boolean", 0.2882]], "teaching/modern-java/collections-and-records.md": [["java", 0.4542], ["collection", 0.4182], ["array", 0.383], ["implement", 0.3622], ["immutable", 0.3408]], "cheatsheets/circuits/components.md": [["oscillation", 0.313], ["dc", 0.2524], ["phase", 0.2245], ["inductance", 0.2118], ["radian", 0.2077]], "cheatsheets/circuits/electricity.md": [["electricity", 0.5212], ["circuit", 0.4328], ["potential", 0.3581], ["resistance", 0.3482], ["flow", 0.2897]], "cheatsheets/java-spring-boot/reference.md": [["springbootapplication", 0.5599], ["bean", 0.4083], ["enableautoconfiguration", 0.3669], ["restcontroller", 0.363], ["annotation", 0.3298]], "cheatsheets/java-spring-boot/running.md": [["gradlew", 0.4169], ["intellij", 0.3445], ["spring", 0.3265], ["deploying", 0.2303], ["quickstart", 0.2287]], "cheatsheets/algorithms/graphs.md": [["graph", 0.435], ["breadth", 0.3974], ["tree", 0.3897], ["bipartite", 0.3084], ["v_1", 0.2712]], "cheatsheets/algorithms/divide-and-conquer.md": [["approximation", 0.2907], ["log_b", 0.2738], ["root", 0.2453], ["recurrence", 0.2307], ["bisect", 0.202]], "cheatsheets/algorithms/intervals.md": [["scheduling", 0.485], ["greedy", 0.4102], ["optimal", 0.3426], ["interval", 0.2924], ["lemma", 0.2222]], "operating-systems/lecture-notes/processes.md": [["process", 0.4879], ["execution", 0.4321], ["runtime", 0.4099], ["stack", 0.4073], ["memory", 0.385]], "operating-systems/lecture-notes/kernel-abstraction.md": [["privilege", 0.4612], ["hardware", 0.366], ["mode", 0.3648], ["interrupt", 0.3087], ["mips", 0.2672]], "operating-systems/lecture-notes/paging.md": [["fragmentation", 0.4367], ["paging", 0.4291], ["allocated", 0.3976], ["process", 0.297], ["virtual", 0.2803]], "operating-systems/lecture-notes/windows-objects-handles-refcounts.md": [["handle", 0.3813], ["process", 0.3698], ["object", 0.3397], ["reference", 0.3281], ["kernel", 0.2762]], "operating-systems/lecture-notes/components.md": [["process", 0.5255], ["memory", 0.4231], ["processor", 0.4137], ["o", 0.3881], ["multiprogramming", 0.3395]], "operating-systems/lecture-notes/io-systems-secondary-storage.md": [["hardware", 0.4162], ["storage", 0.377], ["ethernet", 0.3738], ["bus", 0.3696], ["hierarchy", 0.2643]], "operating-systems/lecture-notes/page-faults.md": [["page", 0.4183], ["process", 0.3845], ["fault", 0.354], ["interrupt", 0.283], ["pte", 0.2688]], "operating-systems/lecture-notes/tlb.md": [["tlb", 0.3583], ["buffer", 0.3449], ["x86", 0.3316], ["table", 0.3052], ["mapped", 0.2703]], "operating-systems/lecture-notes/file-systems.md": [["writefile", 0.4241], ["flushfilebuffers", 0.4046], ["unix", 0.3887], ["storage", 0.3797], ["getfilesize", 0.3171]], "operating-systems/lecture-notes/windows-rtz.md": [["rtlcopymemory", 0.3911], ["rtlzeromemory", 0.3787], ["bug", 0.3727], ["register", 0.3434], ["interrupt", 0.2872]], "operating-systems/lecture-notes/handle-tables.md": [["process", 0.4576], ["scheduling", 0.4434], ["task_struct", 0.4104], ["cpu", 0.3998], ["queue", 0.3427]], "operating-systems/lecture-notes/windows-memory-management.md": [["page", 0.4494], ["memory", 0.43], ["paging", 0.4147], ["process", 0.2811], ["maintained", 0.2578]], "operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md": [["process", 0.5279], ["_processor", 0.4375], ["executable", 0.4148], ["o", 0.4074], ["kernel_open", 0.3745]], "operating-systems/v1-kernels-and-processes/1-introductions.md": [["virtualization", 0.6254], ["o", 0.4491], ["process", 0.4253], ["processor", 0.4226], ["interface", 0.3097]], "operating-systems/v1-kernels-and-processes/3-the-programming-interface.md": [["syscalls", 0.5249], ["createprocess", 0.485], ["process", 0.4723], ["fork", 0.4294], ["_interpreter_", 0.3485]], "operating-systems/section-notes/lab-3-questions.md": [["malloc", 0.4515], ["heap", 0.4212], ["process", 0.4072], ["allocation", 0.3956], ["shell", 0.3644]], "operating-systems/section-notes/section-1.md": [["gdb", 0.4961], ["debugging", 0.4614], ["pointer", 0.3976], ["printf", 0.3762], ["uninitialized", 0.3354]], "operating-systems/v4-persistent-storage/11-file-systems-overview.md": [["filesystem", 0.5186], ["ntfs", 0.5058], ["file", 0.4059], ["byte", 0.3387], ["dram", 0.2802]], "operating-systems/v4-persistent-storage/13-files-and-directories.md": [["directory", 0.4725], ["storage", 0.4539], ["ntfs", 0.4176], ["indexing", 0.3202], ["defragmentation", 0.3169]], "operating-systems/v2-concurrency/7-queueing-theory.md": [["queueing", 0.5586], ["scheduling", 0.4168], ["process", 0.4011], ["utilization", 0.3903], ["arrival", 0.3719]], "operating-systems/v2-concurrency/7-multiprocessor-scheduling.md": [["multiprocessor", 0.5418], ["hyperthreading", 0.4885], ["mfqs", 0.4449], ["processor", 0.4318], ["queue", 0.376]], "operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md": [["concurrentqueue", 0.5281], ["wait_lock", 0.4809], ["lock", 0.477], ["synchronization", 0.4292], ["atomic_t", 0.3829]], "operating-systems/v2-concurrency/4-concurrency-and-threads.md": [["multithreading", 0.6239], ["concurrency", 0.6021], ["processor", 0.5024], ["thread_yield", 0.4748], ["process", 0.4329]], "operating-systems/v2-concurrency/7-uniprocessor-scheduling.md": [["scheduling", 0.5887], ["workload", 0.444], ["uniprocessor", 0.4126], ["multithreading", 0.3868], ["fifo", 0.3196]], "tmp/bench/spec.md": [["dimension_in", 0.3554], ["lisp", 0.3417], ["sweccathon", 0.3331], ["argv_1", 0.3076], ["benchmarking", 0.2516]], "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md": [["scalable", 0.4239], ["performance", 0.4007], ["database", 0.3986], ["maintainability", 0.3932], ["process", 0.3273]], "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md": [["serialized", 0.4834], ["encoding", 0.445], ["schema", 0.4345], ["implementation", 0.4005], ["dataflow", 0.3446]], "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md": [["relational", 0.5655], ["nosql", 0.551], ["database", 0.5108], ["mongodb", 0.4897], ["vertex_id", 0.3281]], "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md": [["database", 0.4782], ["storage", 0.4478], ["index", 0.4049], ["log", 0.3333], ["olap", 0.3125]], "designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md": [["replication", 0.6379], ["distributed", 0.4633], ["availability", 0.444], ["syncronous", 0.4195], ["consistency", 0.3675]], "designing-data-intensive-applications/part-2-distributed-data/preface.md": [["scalable", 0.4581], ["distributed", 0.4189], ["cpu", 0.3523], ["architecture", 0.3172], ["vm", 0.3086]], "designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md": [["mapreduce", 0.5768], ["batch", 0.4316], ["process", 0.4216], ["pipeline", 0.3603], ["unix", 0.3575]]} \ No newline at end of file diff --git a/site/idx/keyword_to_files.json b/site/idx/keyword_to_files.json deleted file mode 100644 index b53dfd0..0000000 --- a/site/idx/keyword_to_files.json +++ /dev/null @@ -1 +0,0 @@ -{"markdown": ["requirements.txt"], "icloud": ["test.md"], "ai": ["test.md"], "apple": ["test.md"], "opting": ["test.md"], "privacy": ["test.md"], "dynamo": ["distributed-systems/dynamo-db.md"], "storage": ["distributed-systems/dynamo-db.md", "distributed-systems/bigtable.md", "distributed-systems/consistency.md", "operating-systems/reference.md", "operating-systems/lecture-notes/io-systems-secondary-storage.md", "operating-systems/lecture-notes/file-systems.md", "operating-systems/v4-persistent-storage/13-files-and-directories.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md"], "amazon": ["distributed-systems/dynamo-db.md"], "versioning": ["distributed-systems/dynamo-db.md", "distributed-systems/disconnected-operation.md"], "consistency": ["distributed-systems/dynamo-db.md", "distributed-systems/managing-critical-state.md", "distributed-systems/primary-backup.md", "distributed-systems/two-phase-commit.md", "distributed-systems/consistency.md", "ds-backup/managing-critical-state.md", "ds-backup/primary-backup.md", "designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md"], "synchronization": ["distributed-systems/disconnected-operation.md", "distributed-systems/ordering-events-in-distributed-systems.md", "ds-backup/ordering-events-in-distributed-systems.md", "operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md"], "cloud": ["distributed-systems/disconnected-operation.md"], "reconnection": ["distributed-systems/disconnected-operation.md"], "writes": ["distributed-systems/disconnected-operation.md", "distributed-systems/google-file-system.md"], "bigtable": ["distributed-systems/bigtable.md"], "mapreduce": ["distributed-systems/bigtable.md", "distributed-systems/managing-critical-state.md", "ds-backup/managing-critical-state.md", "machine-learning-for-big-data/intro-mapreduce-spark.md", "designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md"], "database": ["distributed-systems/bigtable.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md"], "locality": ["distributed-systems/bigtable.md"], "compiler": ["distributed-systems/RPC.md", "ds-backup/RPC.md"], "executes": ["distributed-systems/RPC.md", "ds-backup/RPC.md"], "rpc": ["distributed-systems/RPC.md", "ds-backup/RPC.md"], "tcp": ["distributed-systems/RPC.md", "ds-backup/RPC.md", "networks/5-application/HTTP.md", "networks/5-application/overview.md", "networks/4-transport/TCP.md", "networks/4-transport/ACK-clocking.md"], "nfs": ["distributed-systems/RPC.md", "ds-backup/RPC.md"], "paxos": ["distributed-systems/paxos-intro.md", "distributed-systems/non-blocking-two-phase-commit.md", "distributed-systems/load-balancing.md", "distributed-systems/paxos-made-simple.md", "distributed-systems/paxos-architecture.md", "ds-backup/paxos-intro.md", "ds-backup/paxos-made-simple.md"], "replication": ["distributed-systems/paxos-intro.md", "distributed-systems/managing-critical-state.md", "distributed-systems/primary-backup.md", "ds-backup/paxos-intro.md", "ds-backup/managing-critical-state.md", "ds-backup/primary-backup.md", "designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md"], "protocol": ["distributed-systems/paxos-intro.md", "distributed-systems/ordering-events-in-distributed-systems.md", "ds-backup/paxos-intro.md", "ds-backup/ordering-events-in-distributed-systems.md", "networks/0-foundation/1-network-components-and-protocols.md", "networks/3-network/ARP.md", "networks/3-network/DHCP.md", "networks/3-network/motivation.md", "networks/3-network/global-internet.md", "networks/3-network/BGP.md", "networks/2-direct-links/framing.md", "networks/2-direct-links/wireless.md", "networks/4-transport/TCP.md"], "proposer": ["distributed-systems/paxos-intro.md", "distributed-systems/paxos-made-simple.md", "ds-backup/paxos-intro.md", "ds-backup/paxos-made-simple.md"], "algorithm": ["distributed-systems/paxos-intro.md", "ds-backup/paxos-intro.md", "algorithms/BFS.md", "algorithms/divide-and-conquer.md", "algorithms/stable-matching.md", "algorithms/approximation-algorithms.md", "algorithms/dynamic-programming.md", "algorithms/connected-components.md", "algorithms/greedy-algorithms.md", "algorithms/network-flows.md", "tmp/pairing-algorithm.md", "signal-conditioning/lecture-notes/lecture-5.md", "algorithms/patterns/BFS.md", "algorithms/patterns/sliding-window.md", "algorithms/practice/4.md"], "clock": ["distributed-systems/clocks.md", "ds-backup/clocks.md"], "latency": ["distributed-systems/clocks.md", "ds-backup/clocks.md", "networks/0-foundation/3-performance.md"], "ntp": ["distributed-systems/clocks.md", "ds-backup/clocks.md"], "synchronized": ["distributed-systems/clocks.md", "distributed-systems/paxos-architecture.md", "ds-backup/clocks.md"], "scheduling": ["distributed-systems/clocks.md", "ds-backup/clocks.md", "algorithms/bipartite-graphs.md", "algorithms/dynamic-programming.md", "algorithms/greedy-algorithms.md", "cheatsheets/algorithms/intervals.md", "operating-systems/lecture-notes/handle-tables.md", "operating-systems/v2-concurrency/7-queueing-theory.md", "operating-systems/v2-concurrency/7-uniprocessor-scheduling.md"], "distributed": ["distributed-systems/consistent-global-state.md", "distributed-systems/mutual-exclusion.md", "distributed-systems/distributed-cache-coherence.md", "ds-backup/consistent-global-state.md", "ds-backup/mutual-exclusion.md", "designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md", "designing-data-intensive-applications/part-2-distributed-data/preface.md"], "process": ["distributed-systems/consistent-global-state.md", "ds-backup/consistent-global-state.md", "operating-systems/reference.md", "operating-systems/lecture-notes/processes.md", "operating-systems/lecture-notes/paging.md", "operating-systems/lecture-notes/windows-objects-handles-refcounts.md", "operating-systems/lecture-notes/components.md", "operating-systems/lecture-notes/page-faults.md", "operating-systems/lecture-notes/handle-tables.md", "operating-systems/lecture-notes/windows-memory-management.md", "operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md", "operating-systems/v1-kernels-and-processes/1-introductions.md", "operating-systems/v1-kernels-and-processes/3-the-programming-interface.md", "operating-systems/section-notes/lab-3-questions.md", "operating-systems/v2-concurrency/7-queueing-theory.md", "operating-systems/v2-concurrency/4-concurrency-and-threads.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md", "designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md"], "communicate": ["distributed-systems/consistent-global-state.md", "ds-backup/consistent-global-state.md"], "observer": ["distributed-systems/consistent-global-state.md", "ds-backup/consistent-global-state.md"], "asynchronous": ["distributed-systems/consistent-global-state.md", "ds-backup/consistent-global-state.md"], "ordering": ["distributed-systems/ordering-events-in-distributed-systems.md", "distributed-systems/mutual-exclusion.md", "ds-backup/ordering-events-in-distributed-systems.md", "ds-backup/mutual-exclusion.md", "algorithms/DAGs.md"], "event": ["distributed-systems/ordering-events-in-distributed-systems.md", "ds-backup/ordering-events-in-distributed-systems.md"], "transitive": ["distributed-systems/ordering-events-in-distributed-systems.md", "ds-backup/ordering-events-in-distributed-systems.md"], "queue": ["distributed-systems/mutual-exclusion.md", "ds-backup/mutual-exclusion.md", "operating-systems/lecture-notes/handle-tables.md", "operating-systems/v2-concurrency/7-multiprocessor-scheduling.md"], "locking": ["distributed-systems/mutual-exclusion.md", "distributed-systems/non-blocking-two-phase-commit.md", "distributed-systems/two-phase-commit.md", "ds-backup/mutual-exclusion.md"], "exclusion": ["distributed-systems/mutual-exclusion.md", "ds-backup/mutual-exclusion.md"], "availability": ["distributed-systems/managing-critical-state.md", "distributed-systems/primary-backup.md", "ds-backup/managing-critical-state.md", "ds-backup/primary-backup.md", "designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md"], "acid": ["distributed-systems/managing-critical-state.md", "ds-backup/managing-critical-state.md"], "concurrently": ["distributed-systems/non-blocking-two-phase-commit.md"], "rpcs": ["distributed-systems/non-blocking-two-phase-commit.md"], "commit": ["distributed-systems/non-blocking-two-phase-commit.md", "distributed-systems/two-phase-commit.md"], "server": ["distributed-systems/scaling-web-services.md", "ds-backup/scaling-web-services.md", "networks/5-application/HTTP.md", "networks/5-application/CDNs.md"], "scalable": ["distributed-systems/scaling-web-services.md", "ds-backup/scaling-web-services.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md", "designing-data-intensive-applications/part-2-distributed-data/preface.md"], "tier": ["distributed-systems/scaling-web-services.md", "ds-backup/scaling-web-services.md"], "microservices": ["distributed-systems/scaling-web-services.md", "ds-backup/scaling-web-services.md"], "architecture": ["distributed-systems/scaling-web-services.md", "distributed-systems/paxos-architecture.md", "ds-backup/scaling-web-services.md", "designing-data-intensive-applications/part-2-distributed-data/preface.md"], "shard": ["distributed-systems/sharding.md", "distributed-systems/load-balancing.md"], "hashing": ["distributed-systems/sharding.md"], "sharding": ["distributed-systems/sharding.md"], "bucket": ["distributed-systems/sharding.md"], "hotspot": ["distributed-systems/sharding.md"], "vms": ["distributed-systems/primary-backup.md", "ds-backup/primary-backup.md"], "redis": ["distributed-systems/primary-backup.md", "ds-backup/primary-backup.md"], "atomicity": ["distributed-systems/two-phase-commit.md"], "phase": ["distributed-systems/two-phase-commit.md", "cheatsheets/circuits/components.md"], "queueing": ["distributed-systems/load-balancing.md", "operating-systems/v2-concurrency/7-queueing-theory.md"], "utilization": ["distributed-systems/load-balancing.md", "operating-systems/v2-concurrency/7-queueing-theory.md"], "scaling": ["distributed-systems/load-balancing.md", "networks/3-network/motivation.md"], "consensus": ["distributed-systems/paxos-made-simple.md", "ds-backup/paxos-made-simple.md"], "p1a": ["distributed-systems/paxos-made-simple.md", "ds-backup/paxos-made-simple.md"], "node": ["distributed-systems/paxos-made-simple.md", "ds-backup/paxos-made-simple.md", "signal-conditioning/lecture-notes/lecture-4.md", "networks/0-foundation/1-network-components-and-protocols.md", "networks/2-direct-links/wireless.md"], "filesystems": ["distributed-systems/google-file-system.md"], "gfs": ["distributed-systems/google-file-system.md"], "concurrency": ["distributed-systems/google-file-system.md", "distributed-systems/consistency.md", "operating-systems/reference.md", "operating-systems/v2-concurrency/4-concurrency-and-threads.md"], "workload": ["distributed-systems/google-file-system.md", "operating-systems/v2-concurrency/7-uniprocessor-scheduling.md"], "caching": ["distributed-systems/distributed-cache-coherence.md"], "maintaining": ["distributed-systems/distributed-cache-coherence.md"], "lease": ["distributed-systems/distributed-cache-coherence.md", "distributed-systems/paxos-architecture.md"], "linearizable": ["distributed-systems/distributed-cache-coherence.md"], "overhead": ["distributed-systems/paxos-architecture.md"], "serializability": ["distributed-systems/consistency.md"], "kv": ["distributed-systems/consistency.md"], "verilog": ["digital-design/karnaugh-maps.md", "digital-design/waveform-diagram.md", "digital-design/quartus-workflow.md", "digital-design/system-verilog.md"], "karnaugh": ["digital-design/karnaugh-maps.md"], "adjacency": ["digital-design/karnaugh-maps.md", "algorithms/graphs-intro.md", "algorithms/connected-components.md"], "trigger": ["digital-design/karnaugh-maps.md"], "map": ["digital-design/karnaugh-maps.md"], "bit": ["digital-design/waveform-diagram.md", "networks/0-foundation/3-performance.md", "networks/2-direct-links/errors.md"], "delay": ["digital-design/waveform-diagram.md", "networks/2-direct-links/retransmission.md"], "waveform": ["digital-design/waveform-diagram.md", "digital-design/quartus-workflow.md"], "bus": ["digital-design/waveform-diagram.md", "operating-systems/lecture-notes/io-systems-secondary-storage.md"], "register": ["digital-design/sequential-logic.md", "operating-systems/lecture-notes/windows-rtz.md"], "sequential": ["digital-design/sequential-logic.md"], "flop": ["digital-design/sequential-logic.md"], "sl": ["digital-design/sequential-logic.md"], "synchronizing": ["digital-design/sequential-logic.md"], "combinational": ["digital-design/combinational-logic.md"], "boolean": ["digital-design/combinational-logic.md", "teaching/modern-java/lambdas-and-streams.md"], "logic": ["digital-design/combinational-logic.md"], "passenger": ["digital-design/combinational-logic.md"], "table": ["digital-design/combinational-logic.md", "operating-systems/lecture-notes/tlb.md"], "modelsim": ["digital-design/quartus-workflow.md"], "runlab": ["digital-design/quartus-workflow.md"], "testbench": ["digital-design/quartus-workflow.md"], "systemverilog": ["digital-design/system-verilog.md"], "module": ["digital-design/system-verilog.md"], "wire": ["digital-design/system-verilog.md"], "programmatically": ["digital-design/system-verilog.md"], "classify": ["natural-language-processing/multinomial-logistic-regression.md"], "logistic": ["natural-language-processing/multinomial-logistic-regression.md"], "label": ["natural-language-processing/multinomial-logistic-regression.md"], "multinomial": ["natural-language-processing/multinomial-logistic-regression.md"], "vocabulary": ["natural-language-processing/multinomial-logistic-regression.md"], "cache": ["systems-research/hints-for-computer-system-design.md", "networks/5-application/CDNs.md"], "hardware": ["systems-research/hints-for-computer-system-design.md", "operating-systems/lecture-notes/kernel-abstraction.md", "operating-systems/lecture-notes/io-systems-secondary-storage.md"], "function": ["systems-research/hints-for-computer-system-design.md"], "delta": ["systems-research/hints-for-computer-system-design.md"], "adaptive": ["systems-research/hints-for-computer-system-design.md"], "reading": ["systems-research/how-to-read-a-paper.md"], "paper": ["systems-research/how-to-read-a-paper.md", "systems-research/paper-review-template.md"], "pas": ["systems-research/how-to-read-a-paper.md"], "annotate": ["systems-research/how-to-read-a-paper.md"], "keshav": ["systems-research/how-to-read-a-paper.md"], "science64_strong_inference": ["systems-research/strong-inference.md"], "hypothesis": ["systems-research/strong-inference.md"], "effective": ["systems-research/strong-inference.md"], "scientist": ["systems-research/strong-inference.md"], "analyze": ["systems-research/strong-inference.md"], "template": ["systems-research/paper-review-template.md"], "tag": ["systems-research/paper-review-template.md"], "reviewing": ["systems-research/paper-review-template.md"], "summary": ["systems-research/paper-review-template.md"], "graph": ["algorithms/BFS.md", "algorithms/DAGs.md", "algorithms/graphs-intro.md", "algorithms/bipartite-graphs.md", "algorithms/patterns/BFS.md", "algorithms/problems/graphs-and-trees.md", "cheatsheets/algorithms/graphs.md"], "breadth": ["algorithms/BFS.md", "algorithms/connected-components.md", "algorithms/patterns/BFS.md", "cheatsheets/algorithms/graphs.md"], "level_order_traversal": ["algorithms/BFS.md"], "discover": ["algorithms/BFS.md"], "topological_sort": ["algorithms/DAGs.md"], "acyclic": ["algorithms/DAGs.md", "algorithms/tree-intro.md"], "dag": ["algorithms/DAGs.md"], "dfs_recursive": ["algorithms/DFS.md"], "tree": ["algorithms/DFS.md", "algorithms/tree-intro.md", "algorithms/problems/graphs-and-trees.md", "cheatsheets/algorithms/graphs.md"], "depth": ["algorithms/DFS.md"], "traversal": ["algorithms/DFS.md", "algorithms/connected-components.md"], "edge": ["algorithms/DFS.md", "algorithms/network-flows.md", "algorithms/tree-intro.md", "algorithms/problems/graphs-and-trees.md"], "build_adjacency_matrix": ["algorithms/graphs-intro.md"], "vertex": ["algorithms/graphs-intro.md", "algorithms/approximation-algorithms.md", "algorithms/tree-intro.md"], "build_adjacency_list": ["algorithms/graphs-intro.md"], "recursively": ["algorithms/divide-and-conquer.md"], "conquering": ["algorithms/divide-and-conquer.md"], "dividing": ["algorithms/divide-and-conquer.md"], "bisection": ["algorithms/divide-and-conquer.md"], "stable_matchings": ["algorithms/stable-matching.md"], "find_stable_matchings": ["algorithms/stable-matching.md"], "matchings": ["algorithms/stable-matching.md"], "student": ["algorithms/stable-matching.md"], "bipartite": ["algorithms/bipartite-graphs.md", "cheatsheets/algorithms/graphs.md"], "coloring": ["algorithms/bipartite-graphs.md"], "v_2": ["algorithms/bipartite-graphs.md"], "induction": ["algorithms/induction.md", "algorithms/tree-intro.md", "algorithms/problems/graphs-and-trees.md"], "inductive": ["algorithms/induction.md"], "proof": ["algorithms/induction.md"], "pigeonhole": ["algorithms/induction.md"], "mathematics": ["algorithms/induction.md"], "approximation": ["algorithms/approximation-algorithms.md", "cheatsheets/algorithms/divide-and-conquer.md"], "cover": ["algorithms/approximation-algorithms.md"], "satisfiability": ["algorithms/approximation-algorithms.md"], "hyperplanes": ["algorithms/linear-programming.md"], "optimization": ["algorithms/linear-programming.md"], "polytopes": ["algorithms/linear-programming.md"], "bmatrix": ["algorithms/linear-programming.md", "linear-algebra/elementry-linear-algebra.md", "tmp/pairing-algorithm.md"], "lp": ["algorithms/linear-programming.md"], "knapsack_rec": ["algorithms/dynamic-programming.md"], "max_weighted_interval_subset": ["algorithms/dynamic-programming.md"], "runtime": ["algorithms/dynamic-programming.md", "operating-systems/lecture-notes/processes.md"], "connected_components": ["algorithms/connected-components.md"], "efficiency": ["algorithms/runtime.md"], "asymptotic": ["algorithms/runtime.md"], "complexity": ["algorithms/runtime.md"], "notation": ["algorithms/runtime.md"], "n_0": ["algorithms/runtime.md"], "interval_scheduling": ["algorithms/greedy-algorithms.md"], "partition_intervals": ["algorithms/greedy-algorithms.md"], "greedy": ["algorithms/greedy-algorithms.md", "cheatsheets/algorithms/intervals.md"], "flow": ["algorithms/network-flows.md", "cheatsheets/circuits/electricity.md"], "routing": ["algorithms/network-flows.md", "networks/3-network/routing.md", "networks/3-network/internetworking.md", "networks/3-network/ICMP.md", "networks/3-network/motivation.md"], "partitioning": ["algorithms/network-flows.md"], "concurrent": ["performance-engineering/efficiently-implementing-state-pattern-JVM.md"], "implementation": ["performance-engineering/efficiently-implementing-state-pattern-JVM.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md"], "jvm": ["performance-engineering/efficiently-implementing-state-pattern-JVM.md"], "transition": ["performance-engineering/efficiently-implementing-state-pattern-JVM.md"], "documentstate": ["performance-engineering/efficiently-implementing-state-pattern-JVM.md"], "network": ["networks/reference.md", "networks/0-foundation/1-network-components-and-protocols.md", "networks/3-network/motivation.md"], "system": ["networks/reference.md", "linear-algebra/elementry-linear-algebra.md"], "textbook": ["networks/reference.md"], "systemsapproach": ["networks/reference.md"], "html": ["networks/reference.md"], "sock_stream": ["networks/sockets.md"], "sock_dgram": ["networks/sockets.md"], "socket": ["networks/sockets.md"], "ipproto_tcp": ["networks/sockets.md"], "server_port": ["networks/sockets.md"], "dataflow": ["machine-learning-for-big-data/intro-mapreduce-spark.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md"], "cluster": ["machine-learning-for-big-data/intro-mapreduce-spark.md"], "hdfs": ["machine-learning-for-big-data/intro-mapreduce-spark.md"], "computing": ["machine-learning-for-big-data/intro-mapreduce-spark.md"], "vector": ["linear-algebra/cheatsheet.md"], "geometric": ["linear-algebra/cheatsheet.md"], "v_3": ["linear-algebra/cheatsheet.md"], "interpretation": ["linear-algebra/cheatsheet.md"], "projection": ["linear-algebra/cheatsheet.md"], "cross_product": ["linear-algebra/python-cheatsheet.md"], "numpy": ["linear-algebra/python-cheatsheet.md"], "v_sum": ["linear-algebra/python-cheatsheet.md"], "is_orthonormal_matrix": ["linear-algebra/python-cheatsheet.md"], "l2_norm": ["linear-algebra/python-cheatsheet.md"], "ax": ["linear-algebra/elementry-linear-algebra.md"], "b_m": ["linear-algebra/elementry-linear-algebra.md"], "equation": ["linear-algebra/elementry-linear-algebra.md"], "kernel": ["operating-systems/reference.md", "operating-systems/lecture-notes/windows-objects-handles-refcounts.md"], "osppv2": ["operating-systems/reference.md"], "pairing": ["tmp/pairing-algorithm.md"], "calculate_pairs": ["tmp/pairing-algorithm.md"], "student_i": ["tmp/pairing-algorithm.md"], "retrieval": ["natural-language-processing/reading/information-retrieval.md"], "indexing": ["natural-language-processing/reading/information-retrieval.md", "operating-systems/v4-persistent-storage/13-files-and-directories.md"], "search": ["natural-language-processing/reading/information-retrieval.md", "algorithms/patterns/BFS.md"], "document": ["natural-language-processing/reading/information-retrieval.md"], "ir": ["natural-language-processing/reading/information-retrieval.md"], "swap": ["signal-conditioning/lecture-notes/lecture-1.md"], "device": ["signal-conditioning/lecture-notes/lecture-1.md"], "reusable": ["signal-conditioning/lecture-notes/lecture-1.md"], "energy": ["signal-conditioning/lecture-notes/lecture-1.md"], "cost": ["signal-conditioning/lecture-notes/lecture-1.md"], "circuit": ["signal-conditioning/lecture-notes/lecture-5.md", "signal-conditioning/lecture-notes/lecture-4.md", "signal-conditioning/lecture-notes/lecture-2.md", "cheatsheets/circuits/electricity.md"], "resistance": ["signal-conditioning/lecture-notes/lecture-5.md", "signal-conditioning/lecture-notes/lecture-4.md", "signal-conditioning/lecture-notes/lecture-3.md", "signal-conditioning/lecture-notes/lecture-3.md", "cheatsheets/circuits/electricity.md"], "theorem": ["signal-conditioning/lecture-notes/lecture-5.md"], "thevenin": ["signal-conditioning/lecture-notes/lecture-5.md"], "dissipation": ["signal-conditioning/lecture-notes/lecture-4.md"], "r_2": ["signal-conditioning/lecture-notes/lecture-4.md"], "resistor": ["signal-conditioning/lecture-notes/lecture-3.md"], "dissipated": ["signal-conditioning/lecture-notes/lecture-3.md"], "watt": ["signal-conditioning/lecture-notes/lecture-3.md"], "electricity": ["signal-conditioning/lecture-notes/lecture-2.md", "cheatsheets/circuits/electricity.md"], "electron": ["signal-conditioning/lecture-notes/lecture-2.md"], "current": ["signal-conditioning/lecture-notes/lecture-2.md"], "ampere": ["signal-conditioning/lecture-notes/lecture-2.md"], "inductor": ["signal-conditioning/lecture-notes/lecture-6.md"], "capacitance": ["signal-conditioning/lecture-notes/lecture-6.md"], "ac": ["signal-conditioning/lecture-notes/lecture-6.md"], "jq": ["signal-conditioning/lecture-notes/lecture-6.md"], "imaginary": ["signal-conditioning/lecture-notes/lecture-6.md"], "bfs": ["algorithms/patterns/BFS.md"], "contiguous": ["algorithms/patterns/sliding-window.md"], "subarray": ["algorithms/patterns/sliding-window.md"], "sliding": ["algorithms/patterns/sliding-window.md"], "window": ["algorithms/patterns/sliding-window.md"], "leaf": ["algorithms/problems/graphs-and-trees.md"], "interval": ["algorithms/practice/4.md", "cheatsheets/algorithms/intervals.md"], "classroom": ["algorithms/practice/4.md"], "allocated": ["algorithms/practice/4.md", "operating-systems/lecture-notes/paging.md"], "disjoint": ["algorithms/practice/4.md"], "bandwidth": ["networks/0-foundation/information-theory.md", "networks/0-foundation/3-performance.md", "networks/3-network/networking-services.md", "networks/2-direct-links/multiple-access.md"], "transmit": ["networks/0-foundation/information-theory.md"], "wireless": ["networks/0-foundation/information-theory.md", "networks/1-physical/media.md"], "frequency": ["networks/0-foundation/information-theory.md"], "shannon": ["networks/0-foundation/information-theory.md"], "throughput": ["networks/0-foundation/3-performance.md", "networks/4-transport/flow-control.md"], "performance": ["networks/0-foundation/3-performance.md", "designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md"], "802": ["networks/0-foundation/1-network-components-and-protocols.md", "networks/2-direct-links/wireless.md"], "interface": ["networks/0-foundation/1-network-components-and-protocols.md", "operating-systems/v1-kernels-and-processes/1-introductions.md"], "modulator": ["networks/0-foundation/2-physical-layer.md"], "coding": ["networks/0-foundation/2-physical-layer.md", "networks/1-physical/coding-and-modulation.md"], "decode": ["networks/0-foundation/2-physical-layer.md"], "analog": ["networks/0-foundation/2-physical-layer.md"], "signal": ["networks/0-foundation/2-physical-layer.md"], "arp": ["networks/3-network/ARP.md"], "ip": ["networks/3-network/ARP.md", "networks/3-network/DHCP.md"], "mac": ["networks/3-network/ARP.md"], "bonjour": ["networks/3-network/ARP.md"], "forwarding": ["networks/3-network/routing.md"], "router": ["networks/3-network/routing.md", "networks/3-network/global-internet.md"], "packet": ["networks/3-network/routing.md", "networks/3-network/internetworking.md", "networks/3-network/networking-services.md", "networks/3-network/ICMP.md", "networks/4-transport/transport-overview.md", "networks/4-transport/flow-control.md"], "updateroutingtable": ["networks/3-network/routing.md"], "internetworking": ["networks/3-network/internetworking.md"], "ipv4": ["networks/3-network/internetworking.md"], "qos": ["networks/3-network/internetworking.md", "networks/3-network/networking-services.md", "networks/3-network/motivation.md"], "networking": ["networks/3-network/networking-services.md"], "multiplexing": ["networks/3-network/networking-services.md", "networks/2-direct-links/multiple-access.md"], "dhcp": ["networks/3-network/DHCP.md"], "udp": ["networks/3-network/DHCP.md", "networks/5-application/overview.md"], "configuration": ["networks/3-network/DHCP.md"], "traceroute": ["networks/3-network/ICMP.md"], "ping": ["networks/3-network/ICMP.md"], "icmp": ["networks/3-network/ICMP.md"], "bgp": ["networks/3-network/global-internet.md", "networks/3-network/BGP.md"], "ospf": ["networks/3-network/global-internet.md"], "infrastructure": ["networks/3-network/global-internet.md"], "route": ["networks/3-network/BGP.md"], "traffic": ["networks/3-network/BGP.md"], "interconnected": ["networks/3-network/BGP.md"], "byte": ["networks/2-direct-links/framing.md", "operating-systems/v4-persistent-storage/11-file-systems-overview.md"], "ddcmp": ["networks/2-direct-links/framing.md"], "sdlc": ["networks/2-direct-links/framing.md"], "frame": ["networks/2-direct-links/framing.md"], "retransmission": ["networks/2-direct-links/retransmission.md"], "arq": ["networks/2-direct-links/retransmission.md"], "acknowledged_frames": ["networks/2-direct-links/retransmission.md"], "rtt": ["networks/2-direct-links/retransmission.md", "networks/4-transport/flow-control.md"], "hamming_encode": ["networks/2-direct-links/errors.md"], "detect": ["networks/2-direct-links/errors.md"], "redundancy": ["networks/2-direct-links/errors.md"], "internet_checksum": ["networks/2-direct-links/errors.md"], "collision": ["networks/2-direct-links/wireless.md"], "maca": ["networks/2-direct-links/wireless.md"], "gsm": ["networks/2-direct-links/multiple-access.md"], "fdm": ["networks/2-direct-links/multiple-access.md"], "centralized": ["networks/2-direct-links/multiple-access.md"], "bridgemap": ["networks/2-direct-links/switching.md"], "ethernet": ["networks/2-direct-links/switching.md", "operating-systems/lecture-notes/io-systems-secondary-storage.md"], "bridge_tab_size": ["networks/2-direct-links/switching.md"], "switch": ["networks/2-direct-links/switching.md"], "macaddr": ["networks/2-direct-links/switching.md"], "http": ["networks/5-application/HTTP.md"], "fetching": ["networks/5-application/HTTP.md"], "request": ["networks/5-application/HTTP.md"], "application": ["networks/5-application/overview.md"], "skype": ["networks/5-application/overview.md"], "layer": ["networks/5-application/overview.md", "networks/4-transport/transport-overview.md"], "cdns": ["networks/5-application/CDNs.md"], "proxy": ["networks/5-application/CDNs.md"], "dns": ["networks/5-application/CDNs.md", "networks/5-application/DNS.md"], "nameservers": ["networks/5-application/DNS.md"], "resolver": ["networks/5-application/DNS.md"], "hierarchy": ["networks/5-application/DNS.md", "operating-systems/lecture-notes/io-systems-secondary-storage.md"], "namespaces": ["networks/5-application/DNS.md"], "retransmits": ["networks/4-transport/TCP.md"], "timeout": ["networks/4-transport/TCP.md"], "rtt_": ["networks/4-transport/TCP.md"], "congestion": ["networks/4-transport/ACK-clocking.md"], "buffer": ["networks/4-transport/ACK-clocking.md", "networks/4-transport/flow-control.md", "operating-systems/lecture-notes/tlb.md"], "ack": ["networks/4-transport/ACK-clocking.md"], "pipelining": ["networks/4-transport/ACK-clocking.md"], "transport": ["networks/4-transport/transport-overview.md"], "bytestreams": ["networks/4-transport/transport-overview.md"], "segment": ["networks/4-transport/transport-overview.md"], "recall": ["networks/4-transport/flow-control.md"], "modulation": ["networks/1-physical/coding-and-modulation.md"], "receiver": ["networks/1-physical/coding-and-modulation.md"], "nrz": ["networks/1-physical/coding-and-modulation.md"], "01001": ["networks/1-physical/coding-and-modulation.md"], "interference": ["networks/1-physical/media.md"], "shielding": ["networks/1-physical/media.md"], "microwave": ["networks/1-physical/media.md"], "fiber": ["networks/1-physical/media.md"], "java": ["teaching/modern-java/lambdas-and-streams.md", "teaching/modern-java/collections-and-records.md"], "arraylist": ["teaching/modern-java/lambdas-and-streams.md"], "lambda": ["teaching/modern-java/lambdas-and-streams.md"], "admins": ["teaching/modern-java/lambdas-and-streams.md"], "collection": ["teaching/modern-java/collections-and-records.md"], "array": ["teaching/modern-java/collections-and-records.md"], "implement": ["teaching/modern-java/collections-and-records.md"], "immutable": ["teaching/modern-java/collections-and-records.md"], "oscillation": ["cheatsheets/circuits/components.md"], "dc": ["cheatsheets/circuits/components.md"], "inductance": ["cheatsheets/circuits/components.md"], "radian": ["cheatsheets/circuits/components.md"], "potential": ["cheatsheets/circuits/electricity.md"], "springbootapplication": ["cheatsheets/java-spring-boot/reference.md"], "bean": ["cheatsheets/java-spring-boot/reference.md"], "enableautoconfiguration": ["cheatsheets/java-spring-boot/reference.md"], "restcontroller": ["cheatsheets/java-spring-boot/reference.md"], "annotation": ["cheatsheets/java-spring-boot/reference.md"], "gradlew": ["cheatsheets/java-spring-boot/running.md"], "intellij": ["cheatsheets/java-spring-boot/running.md"], "spring": ["cheatsheets/java-spring-boot/running.md"], "deploying": ["cheatsheets/java-spring-boot/running.md"], "quickstart": ["cheatsheets/java-spring-boot/running.md"], "v_1": ["cheatsheets/algorithms/graphs.md"], "log_b": ["cheatsheets/algorithms/divide-and-conquer.md"], "root": ["cheatsheets/algorithms/divide-and-conquer.md"], "recurrence": ["cheatsheets/algorithms/divide-and-conquer.md"], "bisect": ["cheatsheets/algorithms/divide-and-conquer.md"], "optimal": ["cheatsheets/algorithms/intervals.md"], "lemma": ["cheatsheets/algorithms/intervals.md"], "execution": ["operating-systems/lecture-notes/processes.md"], "stack": ["operating-systems/lecture-notes/processes.md"], "memory": ["operating-systems/lecture-notes/processes.md", "operating-systems/lecture-notes/components.md", "operating-systems/lecture-notes/windows-memory-management.md"], "privilege": ["operating-systems/lecture-notes/kernel-abstraction.md"], "mode": ["operating-systems/lecture-notes/kernel-abstraction.md"], "interrupt": ["operating-systems/lecture-notes/kernel-abstraction.md", "operating-systems/lecture-notes/page-faults.md", "operating-systems/lecture-notes/windows-rtz.md"], "mips": ["operating-systems/lecture-notes/kernel-abstraction.md"], "fragmentation": ["operating-systems/lecture-notes/paging.md"], "paging": ["operating-systems/lecture-notes/paging.md", "operating-systems/lecture-notes/windows-memory-management.md"], "virtual": ["operating-systems/lecture-notes/paging.md"], "handle": ["operating-systems/lecture-notes/windows-objects-handles-refcounts.md"], "object": ["operating-systems/lecture-notes/windows-objects-handles-refcounts.md"], "reference": ["operating-systems/lecture-notes/windows-objects-handles-refcounts.md"], "processor": ["operating-systems/lecture-notes/components.md", "operating-systems/v1-kernels-and-processes/1-introductions.md", "operating-systems/v2-concurrency/7-multiprocessor-scheduling.md", "operating-systems/v2-concurrency/4-concurrency-and-threads.md"], "o": ["operating-systems/lecture-notes/components.md", "operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md", "operating-systems/v1-kernels-and-processes/1-introductions.md"], "multiprogramming": ["operating-systems/lecture-notes/components.md"], "page": ["operating-systems/lecture-notes/page-faults.md", "operating-systems/lecture-notes/windows-memory-management.md"], "fault": ["operating-systems/lecture-notes/page-faults.md"], "pte": ["operating-systems/lecture-notes/page-faults.md"], "tlb": ["operating-systems/lecture-notes/tlb.md"], "x86": ["operating-systems/lecture-notes/tlb.md"], "mapped": ["operating-systems/lecture-notes/tlb.md"], "writefile": ["operating-systems/lecture-notes/file-systems.md"], "flushfilebuffers": ["operating-systems/lecture-notes/file-systems.md"], "unix": ["operating-systems/lecture-notes/file-systems.md", "designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md"], "getfilesize": ["operating-systems/lecture-notes/file-systems.md"], "rtlcopymemory": ["operating-systems/lecture-notes/windows-rtz.md"], "rtlzeromemory": ["operating-systems/lecture-notes/windows-rtz.md"], "bug": ["operating-systems/lecture-notes/windows-rtz.md"], "task_struct": ["operating-systems/lecture-notes/handle-tables.md"], "cpu": ["operating-systems/lecture-notes/handle-tables.md", "designing-data-intensive-applications/part-2-distributed-data/preface.md"], "maintained": ["operating-systems/lecture-notes/windows-memory-management.md"], "_processor": ["operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md"], "executable": ["operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md"], "kernel_open": ["operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.md"], "virtualization": ["operating-systems/v1-kernels-and-processes/1-introductions.md"], "syscalls": ["operating-systems/v1-kernels-and-processes/3-the-programming-interface.md"], "createprocess": ["operating-systems/v1-kernels-and-processes/3-the-programming-interface.md"], "fork": ["operating-systems/v1-kernels-and-processes/3-the-programming-interface.md"], "_interpreter_": ["operating-systems/v1-kernels-and-processes/3-the-programming-interface.md"], "malloc": ["operating-systems/section-notes/lab-3-questions.md"], "heap": ["operating-systems/section-notes/lab-3-questions.md"], "allocation": ["operating-systems/section-notes/lab-3-questions.md"], "shell": ["operating-systems/section-notes/lab-3-questions.md"], "gdb": ["operating-systems/section-notes/section-1.md"], "debugging": ["operating-systems/section-notes/section-1.md"], "pointer": ["operating-systems/section-notes/section-1.md"], "printf": ["operating-systems/section-notes/section-1.md"], "uninitialized": ["operating-systems/section-notes/section-1.md"], "filesystem": ["operating-systems/v4-persistent-storage/11-file-systems-overview.md"], "ntfs": ["operating-systems/v4-persistent-storage/11-file-systems-overview.md", "operating-systems/v4-persistent-storage/13-files-and-directories.md"], "file": ["operating-systems/v4-persistent-storage/11-file-systems-overview.md"], "dram": ["operating-systems/v4-persistent-storage/11-file-systems-overview.md"], "directory": ["operating-systems/v4-persistent-storage/13-files-and-directories.md"], "defragmentation": ["operating-systems/v4-persistent-storage/13-files-and-directories.md"], "arrival": ["operating-systems/v2-concurrency/7-queueing-theory.md"], "multiprocessor": ["operating-systems/v2-concurrency/7-multiprocessor-scheduling.md"], "hyperthreading": ["operating-systems/v2-concurrency/7-multiprocessor-scheduling.md"], "mfqs": ["operating-systems/v2-concurrency/7-multiprocessor-scheduling.md"], "concurrentqueue": ["operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md"], "wait_lock": ["operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md"], "lock": ["operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md"], "atomic_t": ["operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.md"], "multithreading": ["operating-systems/v2-concurrency/4-concurrency-and-threads.md", "operating-systems/v2-concurrency/7-uniprocessor-scheduling.md"], "thread_yield": ["operating-systems/v2-concurrency/4-concurrency-and-threads.md"], "uniprocessor": ["operating-systems/v2-concurrency/7-uniprocessor-scheduling.md"], "fifo": ["operating-systems/v2-concurrency/7-uniprocessor-scheduling.md"], "dimension_in": ["tmp/bench/spec.md"], "lisp": ["tmp/bench/spec.md"], "sweccathon": ["tmp/bench/spec.md"], "argv_1": ["tmp/bench/spec.md"], "benchmarking": ["tmp/bench/spec.md"], "maintainability": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch1-reliable-scalable-and-maintainable-applications.md"], "serialized": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md"], "encoding": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md"], "schema": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch4-encoding-and-evolution.md"], "relational": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md"], "nosql": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md"], "mongodb": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md"], "vertex_id": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch2-data-models-and-query-languages.md"], "index": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md"], "log": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md"], "olap": ["designing-data-intensive-applications/part-1-foundations-of-data-systems/ch3-storage-and-retrieval.md"], "syncronous": ["designing-data-intensive-applications/part-2-distributed-data/ch5-replication.md"], "vm": ["designing-data-intensive-applications/part-2-distributed-data/preface.md"], "batch": ["designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md"], "pipeline": ["designing-data-intensive-applications/part-3-derived-data/ch10-batch-processing.md"]} \ No newline at end of file diff --git a/site/index.html b/site/index.html deleted file mode 100644 index 68b8ef2..0000000 --- a/site/index.html +++ /dev/null @@ -1,341 +0,0 @@ - - - - - - - - - - - -
- -

-
- Last modified: 2025-01-16 - -
-
- -
-
- 157 - Notes -
-
- 17 - Categories -
-
- 104 - Tags -
-
-
-
-

Recent

- -
-
-

Categories

- -
- -
- -
- -
- - \ No newline at end of file diff --git a/site/linear-algebra/cheatsheet.html b/site/linear-algebra/cheatsheet.html deleted file mode 100644 index f66fde0..0000000 --- a/site/linear-algebra/cheatsheet.html +++ /dev/null @@ -1,1612 +0,0 @@ - - - - - - Cheatsheet - - - - - -
- -

Cheatsheet

-
- Last modified: 2024-11-09 - -
-
-

Fundamentals of Vectors

-

Geometric Basics

-

Definition and Representation

- -

Geometric Interpretation

- -

Position Vectors

- -

Direction Vectors

- -

Unit Vectors

- -

Standard Basis Vectors

- - - - - - - - - - - - - - - - - - - - - - - - - -
VectorComponentsDescription
$\mathbf{i}$$\langle 1,0,0 \rangle$Unit vector in x-direction
$\mathbf{j}$$\langle 0,1,0 \rangle$Unit vector in y-direction
$\mathbf{k}$$\langle 0,0,1 \rangle$Unit vector in z-direction
-

Basic Vector Operations

-

Addition and Subtraction

- -

Parallelogram Law

- -

Triangle Inequality

- -

Scalar Multiplication

- -

Systems of Linear Equations

-

Matrix Form (Ax = b)

-

$$ -\begin{bmatrix} -a_{11} & a_{12} & \cdots & a_{1n} \ -a_{21} & a_{22} & \cdots & a_{2n} \ -\vdots & \vdots & \ddots & \vdots \ -a_{m1} & a_{m2} & \cdots & a_{mn} -\end{bmatrix} -\begin{bmatrix} -x_1 \ x_2 \ \vdots \ x_n -\end{bmatrix} = -\begin{bmatrix} -b_1 \ b_2 \ \vdots \ b_m -\end{bmatrix} -$$

-

Solution Types

- - - - - - - - - - - - - - - - - - - - - - - - - -
TypeConditionGeometric Interpretation
Unique Solution$\text{rank}(A) = \text{rank}([A|b]) = n$Lines/planes intersect at one point
Infinite Solutions$\text{rank}(A) = \text{rank}([A|b]) < n$Lines/planes overlap
No Solution$\text{rank}(A) < \text{rank}([A|b])$Lines/planes are parallel
-

Geometric Interpretation

- -

Solution Methods

-

Gaussian Elimination

-
    -
  1. -

    Forward Elimination

    -
  2. -
  3. -

    Convert matrix to row echelon form (REF)

    -
  4. -
  5. Create zeros below diagonal
  6. -
-

[1 * * *] - [0 1 * *] - [0 0 1 *] - [0 0 0 1]

-
    -
  1. Back Substitution
  2. -
  3. Solve for variables from bottom up
  4. -
  5. $x_n \rightarrow x_{n-1} \rightarrow \cdots \rightarrow x_1$
  6. -
-

Gauss-Jordan Elimination

- -
[1 0 0 *]
-[0 1 0 *]
-[0 0 1 *]
-
-

Key Operations

- - - - - - - - - - - - - - - - - - - - - - - - - -
OperationDescriptionNotation
Row SwapSwap rows $i$ and $j$$R_i \leftrightarrow R_j$
Scalar MultiplicationMultiply row $i$ by $c$$cR_i$
Row AdditionAdd multiple of row $i$ to row $j$$R_j + cR_i$
-

Matrices

-

Types and Properties

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
TypeDefinitionProperties
Square$n \times n$ matrix- Same number of rows and columns
- Can have determinant
- May be invertible
Rectangular$m \times n$ matrix- Different number of rows and columns
- No determinant
Identity ($I_n$)$a_{ij} = \begin{cases} 1 & \text{if } i=j \ 0 & \text{if } i\neq j \end{cases}$- Square matrix
- 1's on diagonal, 0's elsewhere
- $AI = IA = A$
Zero ($0$)All entries are 0- Can be any dimension
- $A + 0 = A$
Diagonal$a_{ij} = 0$ for $i \neq j$- Non-zero elements only on main diagonal
TriangularUpper: $a_{ij} = 0$ for $i > j$
Lower: $a_{ij} = 0$ for $i < j$
- Square matrix
- Determinant = product of diagonal entries
-

Basic Matrix Operations

-

Addition and Subtraction

- -

Scalar Multiplication

- -

Transpose

- -

Row Echelon Form

-

A matrix is in row echelon form if:

-
    -
  1. All zero rows are at the bottom
  2. -
  3. Leading coefficient (pivot) of each nonzero row is to the right of pivots above
  4. -
  5. All entries below pivots are zero
  6. -
-

Reduced Row Echelon Form

-

Additional conditions for RREF:

-
    -
  1. Leading coefficient of each nonzero row is 1
  2. -
  3. Each leading 1 is the only nonzero entry in its column
  4. -
-

Pivot Positions

- -

Leading Entries

-

Properties:

- -

Vector Spaces

-

Axioms of Vector Spaces

-

Let $V$ be a vector space over field $F$ with vectors $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and scalars $c, d \in F$:

-
    -
  1. Closure under addition: $\mathbf{u} + \mathbf{v} \in V$
  2. -
  3. Commutativity: $\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}$
  4. -
  5. Associativity: $(\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})$
  6. -
  7. Additive identity: $\exists \mathbf{0} \in V$ such that $\mathbf{v} + \mathbf{0} = \mathbf{v}$
  8. -
  9. Additive inverse: $\exists -\mathbf{v} \in V$ such that $\mathbf{v} + (-\mathbf{v}) = \mathbf{0}$
  10. -
  11. Scalar multiplication closure: $c\mathbf{v} \in V$
  12. -
  13. Scalar multiplication distributivity: $c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}$
  14. -
  15. Vector distributivity: $(c + d)\mathbf{v} = c\mathbf{v} + d\mathbf{v}$
  16. -
  17. Scalar multiplication associativity: $c(d\mathbf{v}) = (cd)\mathbf{v}$
  18. -
  19. Scalar multiplication identity: $1\mathbf{v} = \mathbf{v}$
  20. -
-

Linear Independence

-

Definition

-

Vectors ${\mathbf{v}_1, \mathbf{v}_2, ..., \mathbf{v}_n}$ are linearly independent if: -$$c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + ... + c_n\mathbf{v}_n = \mathbf{0}$$ -implies $c_1 = c_2 = ... = c_n = 0$

-

Tests and Algorithms

- - - - - - - - - - - - - - - - - - - - - - - - - -
TestDescriptionResult
Matrix TestForm matrix $A$ with vectors as columns and solve $A\mathbf{x}=\mathbf{0}$Independent if only solution is trivial
DeterminantFor square matrix of vectorsIndependent if det$(A) \neq 0$
RankCompute rank of matrix $A$Independent if rank = number of vectors
-

Span

-

Definition

-

The span of vectors ${\mathbf{v}_1, ..., \mathbf{v}_n}$ is: -$$\text{span}{\mathbf{v}_1, ..., \mathbf{v}_n} = {c_1\mathbf{v}_1 + ... + c_n\mathbf{v}_n : c_i \in F}$$

-

Geometric Interpretation

- -

Basis

-

A basis is a linearly independent set of vectors that spans the vector space.

-

Standard Basis

-

For $\mathbb{R}^n$: ${\mathbf{e}_1, \mathbf{e}_2, ..., \mathbf{e}_n}$ where: -$\mathbf{e}_i = [0, ..., 1, ..., 0]^T$ (1 in $i$th position)

-

Dimension

-

The dimension of a vector space is the number of vectors in any basis.

-

Basis Theorem

-

Every basis of a vector space has the same number of vectors.

-

Dimension Theorem

-

For finite-dimensional vector space $V$:

- -

Subspaces

-

Tests for Subspaces

-

A subset $W$ of vector space $V$ is a subspace if:

-
    -
  1. $\mathbf{0} \in W$
  2. -
  3. Closed under addition: $\mathbf{u}, \mathbf{v} \in W \implies \mathbf{u} + \mathbf{v} \in W$
  4. -
  5. Closed under scalar multiplication: $c \in F, \mathbf{v} \in W \implies c\mathbf{v} \in W$
  6. -
-

Common Subspaces

- - - - - - - - - - - - - - - - - - - - - - - - - -
SubspaceDefinitionDimension
Null Space$N(A) = {\mathbf{x}: A\mathbf{x}=\mathbf{0}}$$n - \text{rank}(A)$
Column Space$C(A) = {\mathbf{y}: \mathbf{y}=A\mathbf{x}}$$\text{rank}(A)$
Row Space$R(A) = C(A^T)$$\text{rank}(A)$
-

Advanced Vector Operations (cont. 1)

-

Dot Product

-

The scalar product of two vectors.

-

Geometric Definition

-

$$\vec{a} \cdot \vec{b} = |\vec{a}||\vec{b}|\cos(\theta)$$ -where $\theta$ is the angle between vectors

-

Algebraic Definition

-

For vectors in $\mathbb{R}^n$: -$$\vec{a} \cdot \vec{b} = \sum_{i=1}^n a_ib_i = a_1b_1 + a_2b_2 + ... + a_nb_n$$

-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyFormulaDescription
Commutative$\vec{a} \cdot \vec{b} = \vec{b} \cdot \vec{a}$Order doesn't matter
Distributive$\vec{a} \cdot (\vec{b} + \vec{c}) = \vec{a} \cdot \vec{b} + \vec{a} \cdot \vec{c}$Distributes over addition
Scalar Multiplication$(k\vec{a}) \cdot \vec{b} = k(\vec{a} \cdot \vec{b})$Scalars can be factored out
Self-Dot Product$\vec{a} \cdot \vec{a} = |\vec{a}|^2$Dot product with itself equals magnitude squared
-

Angle Formula

-

$$\cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|}$$

-

Projection Formula

-

Vector projection of $\vec{a}$ onto $\vec{b}$: -$$\text{proj}_{\vec{b}}\vec{a} = \frac{\vec{a} \cdot \vec{b}}{|\vec{b}|^2}\vec{b}$$

-

Cross Product

-

Vector product resulting in a vector perpendicular to both input vectors (3D only).

-

Right-Hand Rule

-
    -
  1. Point index finger in direction of first vector
  2. -
  3. Point middle finger in direction of second vector
  4. -
  5. Thumb points in direction of cross product
  6. -
-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyFormulaDescription
Anti-commutative$\vec{a} \times \vec{b} = -(\vec{b} \times \vec{a})$Order matters
Distributive$\vec{a} \times (\vec{b} + \vec{c}) = \vec{a} \times \vec{b} + \vec{a} \times \vec{c}$Distributes over addition
Magnitude$|\vec{a} \times \vec{b}| = |\vec{a}||\vec{b}|\sin(\theta)$Area of parallelogram
Perpendicular$\vec{a} \times \vec{b} \perp \vec{a}$ and $\vec{a} \times \vec{b} \perp \vec{b}$Result is perpendicular to both vectors
-

Triple Product

-

Scalar triple product: -$$\vec{a} \cdot (\vec{b} \times \vec{c}) = \det[\vec{a} \; \vec{b} \; \vec{c}]$$ -Represents volume of parallelepiped

-

Linear Combinations

-

Sum of vectors with scalar coefficients: -$$c_1\vec{v_1} + c_2\vec{v_2} + ... + c_n\vec{v_n}$$

-

Geometric Interpretation

- -

Matrix Properties (cont. 1)

-

Rank

-

The rank of a matrix is the dimension of the vector space spanned by its columns (or rows).

-

Full Rank

-

A matrix has full rank when:

- -

Rank Theorems

- - - - - - - - - - - - - - - - - - - - - -
TheoremStatement
Rank-NullityFor m×n matrix A: rank(A) + nullity(A) = n
Product Rankrank(AB) ≤ min(rank(A), rank(B))
Addition Rankrank(A + B) ≤ rank(A) + rank(B)
-

Trace

-

The trace of a square matrix is the sum of elements on the main diagonal. -$$tr(A) = \sum_{i=1}^n a_{ii}$$

-

Properties

- -

Determinant

-

For square matrix A, det(A) or |A| measures the scaling factor of the linear transformation.

-

Properties

-
    -
  1. det(AB) = det(A)⋅det(B)
  2. -
  3. det(A^T) = det(A)
  4. -
  5. det(A^{-1}) = \frac{1}{det(A)}
  6. -
  7. For triangular matrices: det = product of diagonal entries
  8. -
  9. det(cA) = c^n det(A) for n×n matrix
  10. -
-

Calculation Methods

-

Cofactor Expansion

-

For n×n matrix: -$$det(A) = \sum_{j=1}^n a_{ij}C_{ij}$$ -where Cᵢⱼ is the (i,j) cofactor

-

Row/Column Expansion

-

Choose any row/column i: -$$det(A) = \sum_{j=1}^n (-1)^{i+j} a_{ij}M_{ij}$$ -where Mᵢⱼ is the minor

-

Triangular Method

-
    -
  1. Convert to upper triangular using row operations
  2. -
  3. Multiply diagonal elements
  4. -
  5. Account for row operation signs
  6. -
-

Cramer's Rule

-

For system Ax = b with det(A) ≠ 0: -$$x_i = \frac{det(A_i)}{det(A)}$$ -where Aᵢ is A with column i replaced by b

-

Matrix Operations (cont. 1)

-

Matrix Multiplication

-

For matrices $A_{m×n}$ and $B_{n×p}$: -$$(AB){ij} = \sum$$}^n a_{ik}b_{kj

-

Properties

- -

Non-commutativity

- -

Associativity

-

$(AB)C = A(BC)$ always holds for conformable matrices

-

Inverse

-

For square matrix $A$, if $AA^{-1} = A^{-1}A = I$, then $A^{-1}$ is the inverse of $A$

-

Existence Conditions

- -

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyFormula
Double Inverse$(A^{-1})^{-1} = A$
Product Inverse$(AB)^{-1} = B^{-1}A^{-1}$
Scalar Inverse$(cA)^{-1} = \frac{1}{c}A^{-1}$
Transpose Inverse$(A^{-1})^T = (A^T)^{-1}$
-

Calculation Methods

-

Adjugate Method

-

For an n×n matrix: -$$A^{-1} = \frac{1}{\det(A)}\text{adj}(A)$$ -where adj(A) is the adjugate matrix

-

Gaussian Elimination

-
    -
  1. Form augmented matrix $[A|I]$
  2. -
  3. Convert left side to identity matrix
  4. -
  5. Right side becomes $A^{-1}$
  6. -
-

Elementary Matrices

- -

Conjugate Transpose

-

For matrix $A$:

- -

Systems of Linear Equations (cont. 1)

-

Homogeneous Systems ($A\mathbf{x} = \mathbf{0}$)

-

A system where all constants are zero: $A\mathbf{x} = \mathbf{0}$

-

Trivial Solution

- -

Nontrivial Solutions

- -

Consistency Theorems

-

Consistent System Conditions

- - - - - - - - - - - - - - - - - - - - - -
ConditionSystem is Consistent When
Rank Test$\text{rank}(A) = \text{rank}([A|\mathbf{b}])$
Square Matrix$\text{det}(A) \neq 0$
GeneralSolutions exist if $\mathbf{b}$ is in column space of $A$
-

Fredholm Alternative

-

For system $A\mathbf{x} = \mathbf{b}$, exactly one of these is true:

-
    -
  1. System has a solution
  2. -
  3. $\mathbf{y}^T A = \mathbf{0}$ has a solution with $\mathbf{y}^T\mathbf{b} \neq 0$
  4. -
-

Cramer's Rule

-

For system $A\mathbf{x} = \mathbf{b}$ where $A$ is $n \times n$ with $\text{det}(A) \neq 0$: -$$x_i = \frac{\text{det}(A_i)}{\text{det}(A)}$$ -Where $A_i$ is matrix $A$ with column $i$ replaced by $\mathbf{b}$

-

Matrix Inverse Method

-

For square system $A\mathbf{x} = \mathbf{b}$ where $A$ is invertible: -$$\mathbf{x} = A^{-1}\mathbf{b}$$

-

Requirements:

- -

Linear Transformations

-

Definition

-

A linear transformation $T: V \to W$ is a function between vector spaces that preserves:

-
    -
  1. Addition: $T(u + v) = T(u) + T(v)$
  2. -
  3. Scalar multiplication: $T(cv) = cT(v)$
  4. -
-

Matrix Representation

-

Every linear transformation $T: \mathbb{R}^n \to \mathbb{R}^m$ can be represented by a unique $m \times n$ matrix $A$

-

Standard Matrix

-

For transformation $T$, the standard matrix $A$ is formed by: -$$A = [T(e_1) \; T(e_2) \; \cdots \; T(e_n)]$$ -where $e_i$ are standard basis vectors

-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyDefinitionTest
Kernel (Null Space)$\text{ker}(T) = {v \in V : T(v) = 0}$Solve $Ax = 0$
Range (Image)$\text{range}(T) = {T(v) : v \in V}$Span of columns of $A$
One-to-One (Injective)$T(v_1) = T(v_2) \implies v_1 = v_2$$\text{ker}(T) = {0}$
Onto (Surjective)$\text{range}(T) = W$Columns span $W$
IsomorphismBijective linear transformationOne-to-one and onto
-

Special Transformations

-

Rotation

- -

Reflection

- -

Projection

- -

Scaling

- -

Shearing

- -

Inner Product Spaces

-

Definition

-

An inner product on a vector space $V$ is a function $\langle \cdot,\cdot \rangle: V \times V \to \mathbb{R}$ (or $\mathbb{C}$)

-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyMathematical FormDescription
Positive Definiteness$\langle x,x \rangle \geq 0$ and $\langle x,x \rangle = 0 \iff x = 0$Always non-negative, zero only for zero vector
Symmetry$\langle x,y \rangle = \overline{\langle y,x \rangle}$Complex conjugate for complex spaces
Linearity$\langle ax+by,z \rangle = a\langle x,z \rangle + b\langle y,z \rangle$Linear in first argument
-

Norm

-

The norm induced by inner product: $|x| = \sqrt{\langle x,x \rangle}$

-

Properties

- -

Distance Function

-

$$d(x,y) = |x-y| = \sqrt{\langle x-y,x-y \rangle}$$

-

Orthogonality

-

Orthogonal Vectors

-

Two vectors $x,y$ are orthogonal if $\langle x,y \rangle = 0$

-

Orthogonal Sets

- -

Orthogonal Matrices

-

Matrix $Q$ is orthogonal if $Q^TQ = QQ^T = I$

-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyFormulaNote
Inverse$Q^{-1} = Q^T$Transpose equals inverse
Determinant$\det(Q) = \pm 1$Always unit magnitude
Column/Rows$\langle q_i,q_j \rangle = \delta_{ij}$Form orthonormal set
Length Preservation$|Qx| = |x|$Preserves distances
-

Orthogonal Complements

-

For subspace $W$, orthogonal complement $W^⊥$: -$$W^⊥ = {x \in V : \langle x,w \rangle = 0 \text{ for all } w \in W}$$

-

Orthogonal Projections

-

Projection onto subspace $W$: -$$\text{proj}W(x) = \sumw_i$$ -where ${w_1,\ldots,w_k}$ is basis for $W$}^k \frac{\langle x,w_i \rangle}{|w_i|^2

-

For orthonormal basis: -$$\text{proj}W(x) = \sum^k \langle x,w_i \rangle w_i$$

-

Special Matrices (cont. 1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
TypeDefinitionPropertiesExample
Symmetric$A = A^T$- Diagonal elements can be any real number
- Elements symmetric across main diagonal
- All eigenvalues are real
$$\begin{bmatrix} 1 & 2 & 3\ 2 & 4 & 5\ 3 & 5 & 6 \end{bmatrix}$$
Skew-symmetric$A = -A^T$- Diagonal elements must be zero
- $a_{ij} = -a_{ji}$
- All eigenvalues are imaginary or zero
$$\begin{bmatrix} 0 & 2 & -1\ -2 & 0 & 3\ 1 & -3 & 0 \end{bmatrix}$$
Orthogonal$AA^T = A^TA = I$- $A^{-1} = A^T$
- Columns/rows form orthonormal basis
- $\det(A) = \pm 1$
- Preserves lengths and angles
$$\begin{bmatrix} \cos\theta & -\sin\theta\ \sin\theta & \cos\theta \end{bmatrix}$$
Idempotent$A^2 = A$- Eigenvalues are only 0 or 1
- Trace = rank
- Used in projection matrices
$$\begin{bmatrix} 1 & 0\ 0 & 0 \end{bmatrix}$$
Nilpotent$A^k = 0$ for some $k$- All eigenvalues = 0
- Trace = 0
- $k \leq n$ where $n$ is matrix size
$$\begin{bmatrix} 0 & 1\ 0 & 0 \end{bmatrix}$$
-
-

Additional Properties:

-
    -
  1. -

    Symmetric Matrices:

    -
  2. -
  3. -

    All real symmetric matrices are diagonalizable

    -
  4. -
  5. -

    $x^TAx$ is a quadratic form

    -
  6. -
  7. -

    Orthogonal Matrices:

    -
  8. -
  9. -

    Every column/row has unit length

    -
  10. -
  11. Any two columns/rows are perpendicular
  12. -
  13. -

    Preserves inner products: $(Ax)^T(Ay) = x^Ty$

    -
  14. -
  15. -

    Idempotent Matrices:

    -
  16. -
  17. -

    $I - A$ is also idempotent if $A$ is idempotent

    -
  18. -
  19. -

    Rank = Trace for idempotent matrices

    -
  20. -
  21. -

    Nilpotent Matrices:

    -
  22. -
  23. The minimal $k$ for which $A^k = 0$ is called the index of nilpotency
  24. -
  25. Characteristic polynomial is $\lambda^n$
  26. -
-

Change of Basis

-

Transition Matrices

-

A transition matrix $P$ transforms coordinates from one basis to another: -$$P_{B←C} = [v_1 \; v_2 \; \cdots \; v_n]$$ -where $v_i$ are the basis vectors of B expressed in basis C

- - - - - - - - - - - - - - - - - - - - - - - - - -
OperationFormulaMeaning
Change from C to B$[v]B = P[v]_C$Vector v in basis B
Change from B to C$[v]C = P[v]_B$Vector v in basis C
Inverse relation$P_{C←B} = P_{B←C}^{-1}$Matrices are inverses
-

Similar Matrices

-

Two matrices A and B are similar if: -$$B = P^{-1}AP$$ -where P is an invertible matrix

-

Properties:

- -

Coordinate Vectors

-

For a vector $v$ and basis $B = {b_1, b_2, ..., b_n}$: -$$[v]_B = \begin{bmatrix} c_1 \ c_2 \ \vdots \ c_n \end{bmatrix}$$ -where $v = c_1b_1 + c_2b_2 + ... + c_nb_n$

-

Orthogonalization

-

Gram-Schmidt Process

-

Converting basis ${v_1, v_2, ..., v_n}$ to orthogonal basis ${u_1, u_2, ..., u_n}$:

-
    -
  1. $u_1 = v_1$
  2. -
  3. $u_2 = v_2 - \text{proj}_{u_1}(v_2)$
  4. -
  5. $u_3 = v_3 - \text{proj}{u_1}(v_3) - \text{proj}(v_3)$
  6. -
-

General formula: -$$u_k = v_k - \sum_{i=1}^{k-1} \text{proj}_{u_i}(v_k)$$ -where $\text{proj}_u(v) = \frac{\langle v,u \rangle}{\langle u,u \rangle}u$

-

QR Decomposition

-

Matrix A can be decomposed as: -$$A = QR$$ -where:

- -

Orthonormal Basis

-

Construction

-
    -
  1. Start with any basis
  2. -
  3. Apply Gram-Schmidt process
  4. -
  5. Normalize each vector: $e_i = \frac{u_i}{|u_i|}$
  6. -
-

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyDescription
Orthogonality$\langle e_i,e_j \rangle = 0$ for $i \neq j$
Normality$|e_i| = 1$ for all i
Transition MatrixP is orthogonal ($P^T = P^{-1}$)
Coordinates$[v]_B = [\langle v,e_1 \rangle \; \langle v,e_2 \rangle \; \cdots \; \langle v,e_n \rangle]^T$
-

Eigenvalues and Eigenvectors

-

Definitions

- -

Characteristic Equation

-

$$\det(A - \lambda I) = 0$$ -where:

- -

Characteristic Polynomial

-

$$p(\lambda) = \det(A - \lambda I)$$

- -

Properties

- - - - - - - - - - - - - - - - - - - - - -
PropertyDescription
Sum$\sum \lambda_i = \text{trace}(A)$
Product$\prod \lambda_i = \det(A)$
Number≤ dimension of matrix
-

Geometric Multiplicity

- -

Algebraic Multiplicity

- -

Eigenspace

-

$$E_\lambda = \text{null}(A - \lambda I)$$

- -

Diagonalization

-

$A = PDP^{-1}$ where:

- -

Diagonalizability Conditions

-

Matrix $A$ is diagonalizable if and only if:

-
    -
  1. Geometric multiplicity = algebraic multiplicity for all eigenvalues
  2. -
  3. Sum of dimensions of eigenspaces = matrix dimension
  4. -
-

Diagonalization Process

-
    -
  1. Find eigenvalues (solve characteristic equation)
  2. -
  3. Find eigenvectors for each eigenvalue
  4. -
  5. Form $P$ from eigenvectors as columns
  6. -
  7. Form $D$ with eigenvalues on diagonal
  8. -
  9. Verify $A = PDP^{-1}$
  10. -
-

Similar Matrices

- -

Special Cases

-

Repeated Eigenvalues

- -

Complex Eigenvalues

- -

Applications

-

Powers of Matrices

-

$$A^n = PD^nP^{-1}$$

- -

Difference Equations

-

For system $\vec{x}_{k+1} = A\vec{x}_k$:

- -

Differential Equations

-

For system $\frac{d\vec{x}}{dt} = A\vec{x}$:

- -
-

Important Theorems

-

Fundamental Theorem of Linear Algebra

-

For a linear transformation $T: V \rightarrow W$ and its matrix $A$:

- -

Cayley-Hamilton Theorem

-

Every square matrix $A$ satisfies its own characteristic equation: -$$p(A) = 0 \text{ where } p(\lambda) = \det(A - \lambda I)$$

-

Spectral Theorem

-

For symmetric matrices $A \in \mathbb{R}^{n \times n}$:

- -

Triangle Inequality

-

For vectors $x, y$: -$$|x + y| \leq |x| + |y|$$

-

Cauchy-Schwarz Inequality

-

For vectors $x, y$: -$$|x^Ty| \leq |x||y|$$

- -

Invertible Matrix Theorem

-

The following statements are equivalent for a square matrix $A$:

-
    -
  1. $A$ is invertible
  2. -
  3. $\det(A) \neq 0$
  4. -
  5. $\text{rank}(A) = n$
  6. -
  7. $\text{Null}(A) = {0}$
  8. -
  9. Columns are linearly independent
  10. -
  11. $Ax = b$ has unique solution for all $b$
  12. -
-

Matrix Factorization Theorems

-

LU Decomposition

-

For matrix $A$: -$$A = LU$$ -where:

- -

QR Decomposition

-

For matrix $A$: -$$A = QR$$ -where:

- -

Cholesky Decomposition

-

For symmetric positive definite matrix $A$: -$$A = LL^T$$ -where:

- -

Advanced Topics

-

Singular Value Decomposition (SVD)

-

Any matrix $A \in \mathbb{R}^{m \times n}$ can be decomposed as $A = U\Sigma V^T$

-

Singular Values

- -

U and V Matrices

- -

Applications

- - - - - - - - - - - - - - - - - - - - - - - - - -
ApplicationDescription
Low-rank approximationTruncate to $k$ largest singular values
Image compressionReduce dimensionality while preserving structure
Principal Component AnalysisUse right singular vectors as principal components
Pseudoinverse$A^+ = V\Sigma^+U^T$
-

Jordan Canonical Form

-

For matrix $A$, exists invertible $P$ such that $P^{-1}AP = J$

-

Jordan Blocks

-

$$ -J_k(\lambda) = \begin{bmatrix} -\lambda & 1 & 0 & \cdots & 0 \ -0 & \lambda & 1 & \cdots & 0 \ -\vdots & \vdots & \ddots & \ddots & \vdots \ -0 & 0 & \cdots & \lambda & 1 \ -0 & 0 & \cdots & 0 & \lambda -\end{bmatrix} -$$

-

Generalized Eigenvectors

- -

Positive Definite Matrices

-

Symmetric matrix $A$ where $x^TAx > 0$ for all nonzero $x$

-

Tests for Positive Definiteness

- - - - - - - - - - - - - - - - - - - - - - - - - -
TestCondition
EigenvalueAll eigenvalues > 0
Leading PrincipalsAll leading principal minors > 0
CholeskyExists unique lower triangular $L$ with $A = LL^T$
Quadratic Form$x^TAx > 0$ for all nonzero $x$
-

Applications

- -

Linear Operators

-

Maps between vector spaces preserving linear structure

-

Adjoint Operators

- -

Self-Adjoint Operators

- -

Dual Spaces

-

Vector space $V^*$ of linear functionals on $V$

-

Dual Basis

- -

Dual Maps

-

For linear map $T: V \to W$, dual map $T^: W^ \to V^$ -$$\langle T^f,v \rangle = \langle f,Tv \rangle$$

-

Tensor Products

-

$V \otimes W$ is universal space for bilinear maps

-

Definition

- -

Properties

- - - - - - - - - - - - - - - - - - - - - - - - - -
PropertyDescription
Dimension$\dim(V \otimes W) = \dim(V)\dim(W)$
Associativity$(U \otimes V) \otimes W \cong U \otimes (V \otimes W)$
Distributivity$(U \oplus V) \otimes W \cong (U \otimes W) \oplus (V \otimes W)$
Field multiplication$(\alpha v) \otimes w = v \otimes (\alpha w) = \alpha(v \otimes w)$
-

Multilinear Algebra

-

Tensors

- -

Exterior Algebra

- -
- -
- - \ No newline at end of file diff --git a/site/linear-algebra/elementry-linear-algebra.html b/site/linear-algebra/elementry-linear-algebra.html deleted file mode 100644 index 778bf67..0000000 --- a/site/linear-algebra/elementry-linear-algebra.html +++ /dev/null @@ -1,337 +0,0 @@ - - - - - - Elementry Linear Algebra - - - - - -
- -

Elementry Linear Algebra

-
- Last modified: 2024-11-14 - -
-
-

Elementary Linear Algebra

-

Systems of Equations

-

Systems of equations are both fundamental and important to actually understanding linear algebra. With that being said, the two primary introductory courses at the University of Washington, Math 208 and (to a much lesser extent) Applied Math 352, spend a significant amount of time on methods for solving systems of equations, something that I have almost no interest in. For the sake of completeness, I will briefly touch on notation and algorithms, but will try to both confine it to this document, and to focus on takeaways that become more useful later on.

-

Notation

-

$$ -\begin{align} -a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= b_1 \ -a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= b_2 \ -&\vdots \ -a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n &= b_m -\end{align} -$$

-

Or equivalently, in matrix form $Ax = b$.

-

$$ -\begin{align} -\begin{bmatrix} -a_{11} & a_{12} & \cdots & a_{1n} \ -a_{21} & a_{22} & \cdots & a_{2n} \ -\vdots & \vdots & \ddots & \vdots \ -a_{m1} & a_{m2} & \cdots & a_{mn} -\end{bmatrix} -\begin{bmatrix} -x_1 \ -x_2 \ -\vdots \ -x_n -\end{bmatrix} -&= -\begin{bmatrix} -b_1 \ -b_2 \ -\vdots \ -b_m -\end{bmatrix} -\end{align} -$$

-

Gaussian Elimination

-

Perform any of the following elementary row operations to the augmented matrix $[A|b]$:

- -

The aim of the algorithm is to get the matrix into either row echelon form or reduced row echelon form. The former is a matrix where the first nonzero element in each row is 1, and the first nonzero element in each row is to the right of the first nonzero element in the row above it. The latter is a matrix where the first nonzero element in each row is 1, and the first nonzero element in each row is the only nonzero element in its column. For example, below $A$ is in row echelon form, and $B$ is in reduced row echelon form.

-

$$ -\begin{align} -A &= -\begin{bmatrix} -a_{11} & a_{12} & a_{13} & a_{14} \ -0 & a_{22} & a_{23} & a_{24} \ -0 & 0 & a_{33} & a_{34} \ -0 & 0 & 0 & a_{44} -\end{bmatrix} -\ -B &= -\begin{bmatrix} -1 & 0 & 0 & 0 \ -0 & 1 & 0 & 0 \ -0 & 0 & 1 & 0 \ -0 & 0 & 0 & 1 -\end{bmatrix} -\end{align} -$$

-

As you perform row operations, you also act on $b$ to keep the system equivalent. Once you have the matrix in row echelon form, you can solve the system by back substitution, or by continuing to row reduce to reduced row echelon form, where the solution is immediately apparent.

-

Takeaways

-

Row Operations

-

Solving systems of equations is pretty boring, but the emergent structure of a system on the verge of being solved cements a few important ideas:

- -

Span

-

The span of a set of vectors is the set of all possible linear combinations of those vectors. The span of a set of vectors is a subspace of the vector space. The span of a set of vectors is the null space of the matrix whose columns are those vectors.

-

$$ -\begin{align} -\text{span}\left{ \begin{bmatrix} 1 \ 0 \end{bmatrix}, \begin{bmatrix} 0 \ 1 \end{bmatrix} \right} &= \mathbb{R}^2 \ -\text{span}\left{ \begin{bmatrix} 1 \ 0 \end{bmatrix}, \begin{bmatrix} 0 \ 1 \end{bmatrix}, \begin{bmatrix} 1 \ 1 \end{bmatrix} \right} &= \mathbb{R}^2 \ -\text{span}\left{ \begin{bmatrix} 1 \ 0 \end{bmatrix}, \begin{bmatrix} 0 \ 1 \end{bmatrix}, \begin{bmatrix} 1 \ 1 \end{bmatrix} \right} &= \text{span}\left{ \begin{bmatrix} 1 \ 0 \end{bmatrix}, \begin{bmatrix} 0 \ 1 \end{bmatrix} \right} -\end{align} -$$

-

For a set of $n$ vectors in $\mathbb{R}^n$ to span $\mathbb{R}^n$, the vectors must be linearly independent. This is a necessary and sufficient condition for a set of vectors to be a basis for $\mathbb{R}^n$. You can perform additional reasoning to determine the rules for spanning sets, and implications on linear independence. For instance, a set of less than $n$ vectors in $\mathbb{R}^n$ cannot span $\mathbb{R}^n$, and a set of more than $n$ vectors in $\mathbb{R}^n$ must be linearly dependent.

-

Linear Transformations

-

For a function to be linear, it must satisfy two properties:

- -

A linear transformation is a function $T: \mathbb{R}^n \to \mathbb{R}^m$ that satisfies these properties. The kernel of a linear transformation is the set of vectors that are mapped to the zero vector, e.g. $T(x) = 0$, or $Ax = 0$ for a matrix $A$ that represents the transformation. The kernel is a subspace of the domain. The range of a linear transformation is the set of all possible outputs, and is a subspace of the codomain.

-

$$ -\begin{align} -T: \mathbb{R}^2 &\to \mathbb{R}^2 \ -T\left( \begin{bmatrix} x \ y \end{bmatrix} \right) &= \begin{bmatrix} x \ 0 \end{bmatrix} -\end{align} -$$

-

$$ -\begin{align} -\text{ker}(T) &= \text{span}\left{ \begin{bmatrix} 0 \ 1 \end{bmatrix} \right} \ -\text{range}(T) &= \text{span}\left{ \begin{bmatrix} 1 \ 0 \end{bmatrix} \right} -\end{align} -$$

-

Matrix-Vector Multiplication

-

$$ -\begin{align} -A\begin{bmatrix} x \ y \end{bmatrix} &= x\begin{bmatrix} a_{11} \ a_{21} \end{bmatrix} + y\begin{bmatrix} a_{12} \ a_{22} \end{bmatrix} \ -&= \begin{bmatrix} a_{11}x + a_{12}y \ a_{21}x + a_{22}y \end{bmatrix} -\end{align} -$$

-

Matrix-vector multiplication is a linear transformation. The columns of the matrix are the images of the basis vectors, and the result is the image of the input vector. The kernel of the transformation is the null space of the matrix, and the range is the column space of the matrix.

-

Visually, you can picture transforming the basis vectors/unit square of the domain into the basis vectors/unit square of the codomain. The matrix is the transformation matrix, and the columns are the images of the basis vectors. The result is the image of the input vector.

-

Matrix-Matrix Multiplication

-

$$ -\begin{align} -AB &= A\begin{bmatrix} b_1 & b_2 & \cdots & b_n \end{bmatrix} \ -&= \begin{bmatrix} Ab_1 & Ab_2 & \cdots & Ab_n \end{bmatrix} -\end{align} -$$

-

The algorithm here is to multiply the matrix on the right by each column of the matrix on the left. The result is a matrix whose columns are the images of the columns of the matrix on the right. This is a linear transformation, and the kernel of the transformation is the null space of the matrix on the right, and the range is the column space of the matrix on the left.q

-
import numpy as np
-
-# inefficient
-def multiply_bad(A, B):
-    C = np.zeros((A.shape[0], B.shape[1]))
-    for i in range(A.shape[0]):
-        for j in range(B.shape[1]):
-            for k in range(A.shape[1]):
-                C[i, j] += A[i, k] * B[k, j]
-    return C
-
-# efficient
-def multiply_good(A, B):
-    return np.dot(A, B)
-
-M, N = 10000, 10000
-A = np.random.rand(M, N)
-B = np.random.rand(N, M)
-
-%timeit multiply_bad(A, B)
-%timeit multiply_good(A, B)
-
-

-
-

Matrix multiplication is fundamentally a costly operation, taking $O(n^3)$ time. That being said, libraries like numpy are heavily optimized and can perform matrix multiplication orders of magnitude faster than naive implementations using vectorized operations. In practice you should never write your own matrix multiplication.

-
- -
- - \ No newline at end of file diff --git a/site/linear-algebra/midterm2-cheatsheet.pdf b/site/linear-algebra/midterm2-cheatsheet.pdf deleted file mode 100644 index 53ca950..0000000 Binary files a/site/linear-algebra/midterm2-cheatsheet.pdf and /dev/null differ diff --git a/site/linear-algebra/pca_image_compression.py b/site/linear-algebra/pca_image_compression.py deleted file mode 100644 index 3683268..0000000 --- a/site/linear-algebra/pca_image_compression.py +++ /dev/null @@ -1,116 +0,0 @@ -import numpy as np -import matplotlib.pyplot as plt -from sklearn.datasets import load_digits -from time import time - -def compress_image(image, n_components): - """ - Compress image using PCA by keeping only the top n_components. - Returns compressed image and compression ratio. - """ - mean = np.mean(image, axis=0) - centered = image - mean - - # get covariance matrix and eigenvectors - cov_matrix = np.cov(centered.T) - eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix) - - # sort eigenvectors by eigenvalues in descending order - idx = np.argsort(eigenvalues)[::-1] - eigenvectors = eigenvectors[:, idx] - - # keep only top n_components - reduced_vecs = eigenvectors[:, :n_components] - - # project data onto reduced eigenvectors - projected = centered @ reduced_vecs - - # reconstruct the image - reconstructed = projected @ reduced_vecs.T + mean - - # compression ratio - original_size = image.size * 8 # assuming 8 bits per number - compressed_size = (projected.size + reduced_vecs.size + mean.size) * 8 - compression_ratio = original_size / compressed_size - - return reconstructed, compression_ratio, projected, reduced_vecs - -def plot_compression_results(original, reconstructed, n_components, compression_ratio): - plt.figure(figsize=(12, 4)) - - plt.subplot(121) - plt.imshow(original, cmap='gray') - plt.title('Original Image') - plt.axis('off') - - plt.subplot(122) - plt.imshow(reconstructed, cmap='gray') - plt.title(f'Reconstructed with {n_components} components\n' - f'Compression ratio: {compression_ratio:.1f}x') - plt.axis('off') - - plt.tight_layout() - -def main(): - digits = load_digits() - - digit_8_idx = np.where(digits.target == 8)[0][0] - image = digits.data[digit_8_idx].reshape(8, 8) - - n_components_list = [1, 2, 3, 4, 5, 6, 7, 8] - plt.figure(figsize=(15, 8)) - - for i, n in enumerate(n_components_list, 1): - reconstructed, ratio, _, _ = compress_image(image, n) - - plt.subplot(2, 4, i) - plt.imshow(reconstructed, cmap='gray') - plt.title(f'{n} components\nRatio: {ratio:.1f}x') - plt.axis('off') - - plt.suptitle('Image Reconstruction with Different Numbers of Components', y=1.02) - plt.tight_layout() - - # Now let's analyze compression performance - n_samples = 100 - test_images = digits.data[:n_samples] - - compression_results = [] - timing_results = [] - - for n in n_components_list: - total_ratio = 0 - start_time = time() - - for img in test_images: - img_2d = img.reshape(8, 8) - _, ratio, _, _ = compress_image(img_2d, n) - total_ratio += ratio - - avg_ratio = total_ratio / n_samples - total_time = time() - start_time - - compression_results.append(avg_ratio) - timing_results.append(total_time) - - plt.figure(figsize=(12, 4)) - - plt.subplot(121) - plt.plot(n_components_list, compression_results, 'bo-') - plt.xlabel('Number of Components') - plt.ylabel('Average Compression Ratio') - plt.title('Compression Performance') - plt.grid(True) - - plt.subplot(122) - plt.plot(n_components_list, timing_results, 'ro-') - plt.xlabel('Number of Components') - plt.ylabel('Processing Time (seconds)') - plt.title(f'Processing Time for {n_samples} Images') - plt.grid(True) - - plt.tight_layout() - plt.show() - -if __name__ == "__main__": - main() \ No newline at end of file diff --git a/site/linear-algebra/python-cheatsheet.html b/site/linear-algebra/python-cheatsheet.html deleted file mode 100644 index 8ade19a..0000000 --- a/site/linear-algebra/python-cheatsheet.html +++ /dev/null @@ -1,496 +0,0 @@ - - - - - - Python Cheatsheet - - - - - -
- -

Python Cheatsheet

-
- Last modified: 2024-12-08 - -
-
-

Fundamentals of Vectors

-

Working with Vectors in NumPy

-
import numpy as np
-
-

Creating Vectors

-
# Row vector
-v_row = np.array([1, 2, 3])
-
-# Column vector
-v_col = np.array([[1], [2], [3]])
-assert v_col = np.transpose(v_row)
-
-

Vector Operations

-
# Addition
-v1, v2 = np.array([1, 2, 3]), np.array([4, 5, 6])
-v_sum = v1 + v2
-
-# Subtraction
-v_diff = v1 - v2
-
-# Scalar multiplication
-c = 2
-v_scaled = c * v1
-
-# Dot product
-dot_product = np.dot(v1, v2)
-
-# Cross product
-cross_product = np.cross(v1, v2)
-
-

Vector Norms

-
# L1 norm
-l1_norm = np.linalg.norm(v, ord=1)
-
-# L2 norm
-l2_norm = np.linalg.norm(v, ord=2)
-
-# Infinity norm
-inf_norm = np.linalg.norm(v, ord=np.inf)
-
-

Useful Functions

-
# Normalize vector
-normalize = lambda v: v / np.linalg.norm(v)
-
-# Angle between vectors (radians)
-angle_rad = lambda v1, v2: np.arccos(np.clip(np.dot(normalize(v1), normalize(v2)), -1.0, 1.0))
-
-# Angle between vectors (degrees)
-angle_deg = lambda v1, v2: np.degrees(angle_rad(v1, v2))
-
-# Projection of v1 onto v2
-projection = lambda v1, v2: np.dot(v1, normalize(v2)) * normalize(v2)
-
-# Rejection of v1 from v2
-rejection = lambda v1, v2: v1 - projection(v1, v2)
-
-# Distance between vectors
-distance = lambda v1, v2: np.linalg.norm(v1 - v2)
-
-

Visualization with Matplotlib

-
import matplotlib.pyplot as plt
-
-

2D Vectors

-
import numpy as np
-import matplotlib.pyplot as plt
-
-def plot_2d(*vectors, figsize=(10, 10), arrows=True, grid=True, equal_aspect=True):
-    """
-    Plot multiple 2D vectors on a Cartesian plane.
-
-    Parameters:
-    -----------
-    *vectors : array-like
-        Variable number of 2D vectors to plot. Each vector should be a 2D point
-        or array-like object with x and y coordinates.
-    figsize : tuple, optional (default=(10, 10))
-        Figure size as (width, height)
-    arrows : bool, optional (default=True)
-        If True, draws vectors as arrows from origin. If False, plots as points/lines.
-    grid : bool, optional (default=True)
-        Whether to show the grid
-    equal_aspect : bool, optional (default=True)
-        Whether to maintain equal aspect ratio for x and y axes
-
-    Returns:
-    --------
-    fig, ax : tuple
-        Matplotlib figure and axis objects
-    """
-    # convert all inputs to numpy arrays
-    vectors = [np.array(v, dtype=float) if not isinstance(v, np.ndarray) else v for v in vectors]
-
-    # validate inputs
-    for i, v in enumerate(vectors):
-        if v.shape != (2,):
-            raise ValueError(f"Vector {i} has shape {v.shape}, expected (2,)")
-
-    fig, ax = plt.subplots(figsize=figsize)
-
-    colors = plt.cm.tab10(np.linspace(0, 1, len(vectors)))
-
-    for v, color in zip(vectors, colors):
-        if arrows:
-            ax.quiver(0, 0, v[0], v[1], angles='xy', scale_units='xy', scale=1,
-                     color=color, width=0.005)
-        else:
-            ax.plot([0, v[0]], [0, v[1]], '-o', color=color)
-
-    all_coords = np.vstack(vectors)
-    max_abs_coord = np.max(np.abs(all_coords)) * 1.2  # Add 20% padding
-    ax.set_xlim(-max_abs_coord, max_abs_coord)
-    ax.set_ylim(-max_abs_coord, max_abs_coord)
-    ax.axhline(y=0, color='k', linestyle='-', linewidth=0.5, alpha=0.3)  # x-axis
-    ax.axvline(x=0, color='k', linestyle='-', linewidth=0.5, alpha=0.3)  # y-axis
-
-    if grid:
-        ax.grid(True, linestyle='--', alpha=0.3)
-
-    if equal_aspect:
-        ax.set_aspect('equal')
-
-    ax.set_xlabel('x')
-    ax.set_ylabel('y')
-
-    return fig, ax
-
-

3D Vectors

-
import numpy as np
-import matplotlib.pyplot as plt
-
-def plot_3d(*vectors):
-    """Plot multiple 3D vectors from origin as lines with dots at endpoints."""
-    # Convert inputs to numpy arrays
-    vectors = [np.array(v) for v in vectors]
-
-    fig = plt.figure()
-    ax = fig.add_subplot(111, projection='3d')
-
-    for v in vectors:
-        # Plot line from origin to vector endpoint with dot
-        ax.plot([0, v[0]], [0, v[1]], [0, v[2]], '-o')
-
-    # Set equal aspect ratio and add labels
-    ax.set_box_aspect([1,1,1])
-    ax.set_xlabel('x')
-    ax.set_ylabel('y')
-    ax.set_zlabel('z')
-
-    return fig, ax
-
-

Matrices

-

Working with Matrices in NumPy

-
import numpy as np
-
-

Creating Matrices

-
# 2D array
-A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
-
-# Identity matrix
-I = np.eye(3)
-
-# Zero matrix
-Z = np.zeros((3, 3))
-
-# Ones matrix
-O = np.ones((3, 3))
-
-# Random matrix
-R = np.random.rand(3, 3)
-
-

Matrix Operations

-
# Addition
-A1, A2 = np.array([[1, 2], [3, 4]]), np.array([[5, 6], [7, 8]])
-A_sum = A1 + A2
-
-# Subtraction
-A_diff = A1 - A2
-
-# Scalar multiplication
-c = 2
-A_scaled = c * A1
-
-# Matrix multiplication
-A_prod = np.dot(A1, A2)
-
-# Transpose
-A_transpose = np.transpose(A)
-
-# Inverse
-A_inv = np.linalg.inv(A)
-
-# Determinant
-det_A = np.linalg.det(A)
-
-# Trace
-trace_A = np.trace(A)
-
-# Rank
-rank_A = np.linalg.matrix_rank(A)
-
-# Eigenvalues and eigenvectors
-eigenvalues, eigenvectors = np.linalg.eig(A)
-
-# Singular value decomposition
-U, S, V = np.linalg.svd(A)
-
-# Matrix norms
-l1_norm = np.linalg.norm(A, ord=1)
-
-l2_norm = np.linalg.norm(A, ord=2)
-
-inf_norm = np.linalg.norm(A, ord=np.inf)
-
-fro_norm = np.linalg.norm(A, ord='fro')
-
-# Matrix power
-A_squared = np.linalg.matrix_power(A, 2)
-
-

Systems of Linear Equations

-

Solving Linear Systems

-

Matrix Inversion Method

-
A = np.array([[2, 1], [1, 1]])
-b = np.array([3, 2])
-
-x = np.linalg.inv(A).dot(b)
-
-# or, much faster
-
-x = np.linalg.solve(A, b)
-
-print(x)
-
-

Vector Spaces

-

Linear Independence, Span, Basis

-

-# check if set of vectors is linearly independent
-is_linearly_independent = lambda *vectors: np.linalg.matrix_rank(np.array(vectors)) == len(vectors[0])
-
-# check if vector is in span of other vectors
-is_in_span = lambda v, *S: np.linalg.matrix_rank(np.array([*S, v])) == np.linalg.matrix_rank(S)
-
-# find basis of a set of vectors
-basis = lambda *vectors: np.array(vectors).T[:, np.linalg.matrix_rank(np.array(vectors)):]
-
-

Matrix Properties (cont. 1)

-

Dimension, Rank, Nullity, Row Space

-
# dimension of a vector space
-dim = lambda *vectors: np.linalg.matrix_rank(np.array(vectors))
-rank = lambda A: np.linalg.matrix_rank(A)
-nullity = lambda A: A.shape[1] - rank(A)
-
-

Special Transformations

-
# reflection matrix
-reflect = lambda n: np.eye(len(n)) - 2 * np.outer(n, n) / np.dot(n, n)
-
-# projection matrix
-project = lambda n: np.outer(n, n) / np.dot(n, n)
-
-# rotation matrix
-rotate = lambda theta: np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
-
-# scaling matrix
-scale = lambda s: np.diag(s)
-
-# shearing matrix
-shear = lambda s: np.array([[1, s], [0, 1]])
-
-# translation matrix
-translate = lambda t: np.array([[1, 0, t[0]], [0, 1, t[1]], [0, 0, 1]])
-
-# homogenous coordinates
-homogenize = lambda v: np.append(v, 1)
-
-# dehomogenize
-dehomogenize = lambda v: v[:-1] / v[-1]
-
-# affine transformation
-affine = lambda A, t: np.block([[A, t], [0, 0, 1]])
-
-# perspective transformation
-perspective = lambda d: np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1/d]])
-
-

Orthogonality

-
# check if two vectors are orthogonal
-is_orthogonal = lambda u, v: np.dot(u, v) == 0
-
-# check if set of vectors is orthogonal
-is_orthogonal_set = lambda *vectors: all(np.dot(v1, v2) == 0 for v1, v2 in itertools.combinations(vectors, 2))
-
-# check if set of vectors is orthonormal
-is_orthonormal_set = lambda *vectors: is_orthogonal_set(*vectors) and all(np.linalg.norm(v) == 1 for v in vectors)
-
-# check if matrix is orthogonal
-is_orthogonal_matrix = lambda A: np.allclose(np.dot(A, A.T), np.eye(A.shape[0]))
-
-# check if matrix is orthonormal
-is_orthonormal_matrix = lambda A: is_orthogonal_matrix(A) and np.allclose(np.linalg.det(A), 1)
-
-

Change of Basis

-
# change of basis matrix
-change_of_basis = lambda B, C: np.linalg.inv(C) @ B
-
-
- -
- - \ No newline at end of file diff --git a/site/machine-learning-for-big-data/intro-mapreduce-spark.html b/site/machine-learning-for-big-data/intro-mapreduce-spark.html deleted file mode 100644 index 9f91464..0000000 --- a/site/machine-learning-for-big-data/intro-mapreduce-spark.html +++ /dev/null @@ -1,272 +0,0 @@ - - - - - - Intro Mapreduce Spark - - - - - -
- -

Intro Mapreduce Spark

-
- Last modified: 2024-03-26 - -
-
-

Distributed Computing for Data Mining

-

How can we extract knowledge from large data sets?

- -

Typically, data is stored on networks of commodity hardware (cheap, off-the-shelf hardware) within data centers. A major challenge with this computing model is the failure of individual machines. One server may survive for ~3 years, but with 10,000 servers, you can expect one to fail every day. With 1M servers, you can expect 1000 failures per day.

-

One approach is to replicate data across multiple machines. However, transferring data is expensive and time intensive. A core idea of distributed computing is to move computation to the data, rather than moving data to the computation. Spark/Hadoop address these problems.

- -

Distributed file system give you a global namespace. Typical usage patterns include huge files (100s of GBs to TBs), no updates in place (append only logs), and large streaming reads. HDFS is optimized for these patterns.

- -

MapReduce

-

MapReduce is a style of programming that is designed for -- Easy parallelization -- Invisible management of hardware/software failures -- Easy management of very large datasets -- Very little required memory (since data is read and written to disk)

-

There are several implementations of MapReduce, including Hadoop and Spark.

- -

It is important that your distribution of keys outputted by the map function is semi-uniform. Skew in keys leads to skew in the workload of reducers associated with those keys.

-

Example: Word Count

-

You have a huge text document and you want to count the number of times each word appears (ie analyzing a log file).

-

Map: For each word in the document, output a key-value pair where the key is the word and the value is 1.

-
def map(doc):
-    for word in doc.split():
-        yield (word, 1)
-
-

Group by key: Sort and shuffle the output of the mappers so that all values for a given key are grouped together.

-
def group_by_key(pairs):
-    pairs.sort()
-    for key, group in itertools.groupby(pairs, key=lambda x: x[0]):
-        yield (key, [x[1] for x in group])
-
-

Reduce: For each key and its associated list of values, sum the values.

-
def reduce(key, values):
-    yield (key, sum(values))
-
-

Spark

-

The two major limitations of MapReduce are -- Rigid programming model -- Performance bottleneck due to disk I/O

-

Spark is a general-purpose cluster computing system that addresses these limitations. It is instead dataflow based, where you define a series of transformations on data, and Spark figures out how to execute them in parallel. It is meant to be a more expressive and efficient than MapReduce. There are higher-level APIs like dataframes and SQL that make it easier to work with data.

-

Resilient Distributed Datasets (RDDs)

-

The core data structure in Spark. They are immutable, distributed collections of objects. You can perform transformations on RDDs to create new RDDs, and Spark will optimize the execution of these transformations.

-

They are essentially a partitioned collection of records that can be cached in memory across machines. They are fault-tolerant, meaning if a partition is lost, it can be recomputed from the original source.

- -

Task Scheduling

-

Spark supports general DAGs of tasks, where each task is a unit of work that is sent to a worker. The DAG scheduler breaks the computation into stages, where each stage is a set of tasks that can be executed in parallel. The task scheduler then schedules tasks within each stage. Functions are pipelined together when possible, and tasks are scheduled in both a cache aware and partition aware manner.

-

Libraries

- -

Spark vs. Hadoop + MapReduce

- -
- -
- - \ No newline at end of file diff --git a/site/natural-language-processing/multinomial-logistic-regression.html b/site/natural-language-processing/multinomial-logistic-regression.html deleted file mode 100644 index 1a44220..0000000 --- a/site/natural-language-processing/multinomial-logistic-regression.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Multinomial Logistic Regression - - - - - -
- -

Multinomial Logistic Regression

-
- Last modified: 2025-01-14 - Category: natural language processing -
-
-

Multinomial Logistic Regression

-

Classification

-

Input can be anything (document, image, etc.) and output is a class label from the finite set $\mathcal{L}$.

-

$$ -classify : \mathcal{V}* \rightarrow \mathcal{L} -$$

-

$\mathcal{V}$ is the set of words in our vocabulary.

-

$X$ is a random variable representing the input, in a given instance taking values from $\mathcal{V}*$.

-

$Y$ is a random variable representing the output, taking values from $\mathcal{L}$.

-

$p(X, Y)$ is the "true" distribution of labeled texts. $p(Y)$ is the distribution of labels. We don't normally know this without looking at the data.

-
-
Tags: classification, machine learning, multinomial logistic regression
-
- - \ No newline at end of file diff --git a/site/natural-language-processing/neural-networks.html b/site/natural-language-processing/neural-networks.html deleted file mode 100644 index 8e55b0c..0000000 --- a/site/natural-language-processing/neural-networks.html +++ /dev/null @@ -1,345 +0,0 @@ - - - - - - Feedforward Neural Networks - - - - - -
- -

Feedforward Neural Networks

-
- Last modified: 2025-01-14 - Category: natural language processing -
-
-

Neural Networks

-

Contrasting with MLR, neural networks are a more flexible model that can learn complex patterns in the data, even without hand-crafted features.

-

Activation Functions

-

A single computational unit $z = w \cdot x + b$ is a linear function of the input $x$ with weights $w$ and bias $b$. The output $y$ is a non-linear function of $f(z)$, where $f$ is the activation function (typically one of $\tanh$, $\text{ReLU}$, or $\sigma$).

-

$$ -y = \sigma(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}} -$$

-

In practice, $\sigma$ is rarely the best choice, and $\tanh$ is similar yet almost always better. $\tanh$ is a scaled version of $\sigma$ that ranges from $-1$ to $1$.

-

$$ -y = \tanh(w \cdot x + b) = \frac{e^{w \cdot x + b} - e^{-(w \cdot x + b)}}{e^{w \cdot x + b} + e^{-(w \cdot x + b)}} -$$

-

The simplest activation function is the Rectified Linear Unit (ReLU), which is $0$ for negative inputs and linear for positive inputs.

-

$$ -y = \text{ReLU}(w \cdot x + b) = \max(0, w \cdot x + b) -$$

-

A potential upside with ReLU is that it is computationally efficient, and also prevents the vanishing gradient problem, e.g. when the gradient is $\approx 0$, and the network stops learning.

-

The XOR Problem

-

It can be shown that a single computational unit cannot solve XOR, as it is a non-linear problem. However, a two-layer network can solve XOR, as it can learn to represent the input in a higher-dimensional space where the problem is linearly separable.

-

$$ -y = \begin{cases} -1 & \text{if } w \cdot x + b > 0 \ -0 & \text{otherwise} -\end{cases} -$$

-

XOR turns out to be a simple example of a problem that is not linearly separable in the input space, since the inputs $(x_1, x_2) = (0, 0)$ and $(1, 1)$ are in the same class, while $(0, 1)$ and $(1, 0)$ are in the other class. It is not possible to draw a straight line that separates the two classes.

-

Feedforward Neural Networks

-

A feedforward NN is a multi-layer network where the output of each layer is the input to the next layer, all with no cycles. They are sometimes called multilayer perceptrons (MLPs), although this term is technically only applicable to networks with a single step function as their activation function.

-

The network has three different types of nodes:

-

Input units

-

vector of input units is $x$. One node for each feature in the input.

-

Hidden layers

-

one or more layers of hidden units, each with a non-linear activation function. In the standard architecture, each node is connected with all nodes in the previous layer. Thus, each hidden unit sums over all input values.

-

For a given hidden layer $h$, we combine the weights $w$ and bias $b$ for each computational unit into a weight matrix $W$ and bias vector $b$. Each element $W_{ij}$ of the weight matrix is the weight from the $i$th input unit $x_i$ to the $j$th hidden unit $h_j$.

-

Thus, the output for a given hidden layer with activation function $f$ is:

-

$$ -h = f(W \cdot x + b) -$$

-

Dimensionality

-

Referring to the input layer as layer $0$, and $n_0$ as the number of input units, we have an input $x \in \mathbb{R}^{n_0}$, e.g. a column vector with dimension $n_0 \times 1$.

-

The first hidden layer $h^{(1)}$ has $n_1$ hidden units, so $W \in \mathbb{R}^{n_1 \times n_0}$, and $b \in \mathbb{R}^{n_1}$.

-

$$ -h_j = f\left(\sum_{i=1}^{n_0} W_{ji} x_i + b_j\right) -$$

-

Output units

-

one or more output units, each with a non-linear activation function. The output layer is the final layer of the network, and the output $y$ with $dim(y) = n_{\text{output}}$ is an estimate for the probability distribution of the correct class/output.

-

Normalization

-

In order to get that probability distribution, we normalize the output of the network using the softmax function.

-

$$ -y = \text{softmax}(W \cdot h + b) -$$

-

$$ -\text{softmax}(z) = \frac{e^z}{\sum_{i=1}^n e^{z_i}} -$$

-

Comparison with MLR

-

A NN is like MLR but with with a few differences: -- many layers, since a deep NN is like layer after layer of MLR classifiers -- intermediate layers have non-linear activation functions. In fact, without these, the network would just be a linear classifier since the composition of linear functions is still linear -- instead of feature selection, previous layers build up a representation of the input that is useful for the final layer

-

Details/Notation

- -

Example: 2-layer NN

-

$$ -\begin{align} -z^{[1]} &= W^{[1]} \cdot a^{[0]} + b^{[1]} \ -a^{[1]} &= g^{[1]}(z^{[1]}) \ -z^{[2]} &= W^{[2]} \cdot a^{[1]} + b^{[2]} \ -a^{[2]} &= g^{[2]}(z^{[2]}) \ -\hat{y} &= a^{[2]} -\end{align} -$$

-

Feedforward Computation

-

$$ -\begin{align*} -\text{for } l = 1, \ldots, L: \ -z^{[l]} &= W^{[l]} \cdot a^{[l-1]} + b^{[l]} \ -a^{[l]} &= g^{[l]}(z^{[l]})\

-

\text{return } \hat{y} = a^{[L]} -\end{align*} -$$

-
def feedforward(x):
-  a = x
-  for l in range(1, L):
-    z = W[l] @ a + b[l]
-    a = g[l](z)
-  return a
-
-

Replacing the Bias

-

Often, the bias term is included in the weight matrix, by adding a column of $1$s to the input vector $x$.

-

With $a^{[0]}_0 = 1$, we can write $z^{[l]} = W^{[l]} \cdot a^{[l-1]}$.

-

$$ -h_j = f\left(\sum_{i=1}^{n_0} W_{ji} x_i\right) -$$

-

FF networks for NLP: Classification

-

Instead of manually designed features, use words as embeddings (e.g. word2vec, GloVe). This constitutes "pre-training", i.e. relying on already computed values/embeddings. One simple method of representing a sentence is to sum the embeddings of the words in the sentence, or to average them.

-

To classify many examples at once, pack inputs into a single matrix $X$ where each row $i$ is an input vector $x^{(i)}$. If our input has $d$ features, then $X \in \mathbb{R}^{m \times d}$ where $m$ is the number of examples.

-

$W \in \mathbb{R}^{d_h \times d}$ is the weight matrix for the hidden layer, and $b \in \mathbb{R}^{d_h}$ is the bias vector. $Y \in \mathbb{R}^{m \times n_{\text{output}}}$ is the output matrix.

-

$$ -\begin{align} -H &= f(X W^T + b) \ -Z &= H U^T\ -\hat{Y} &= \text{softmax}(Z) -\end{align} -$$

-

Training Neural Nets

-

We want to learn the parameters $W^{[i]}$ and $b^{[i]}$ for each layer $i$ that make $\hat{y}$ as close as possible to the true $y$.

-

Loss Function

-

Same as the one used for MLR, the cross-entropy loss function.

-

For binary classification, the loss function is: -$$ -L_{\text{CE}}(\hat{y}, y) = - \log p(y | x) = - \left [ y \log \hat{y} + (1 - y) \log (1 - \hat{y}) \right ] -$$

-

For multi-class classification, the loss function is:

-

$$ -L_{\text{CE}}(\hat{y}, y) = - \sum_{i=1}^n y_i \log \hat{y}_i = - \log \hat{y}_i \text{ where } y_i = 1 -$$

-

$$ -L_{\text{CE}}(\hat{y}, y) = -\log \frac{exp(z_{c})}{\sum_{i=1}^K exp(z_i)} -$$

-

Backpropagation

-

One must pass gradients back through the network to update the weights. This is done using the chain rule. Each node in a computation graph takes an upstream gradient and computes its local gradient, multiplying the two to get the downstream gradient. A node may have multiple local gradients, one for each incoming edge.

-

A very simple example

-

Consider the function $L(a, b, c) = c(a + 2b)$. Create a computation graph with nodes $a, b, c$ for the inputs, and $d = 2b, e = a + d, L = ce$ for the intermediate computations.

-
(a) ---------------- \
-                      (e) ------------ (L)
-                     /                /
-(b) --------(d)-----    /-------------
-                       /
-(c) -------------------
-
-

$$ -\begin{align} -\frac{\partial L}{\partial c} &= e = a + 2b \ -\frac{\partial L}{\partial a} &= \frac{\partial L}{\partial e} \cdot \frac{\partial e}{\partial a} = c \ -\frac{\partial L}{\partial b} &= \frac{\partial L}{\partial e} \cdot \frac{\partial e}{\partial d} \cdot \frac{\partial d}{\partial b} = 2c -\end{align} -$$

-

Learning details

-

NN optimization is a non-convex optimization problem, so it requires a few techniques to work well:

- -
-
Tags: deep learning, feedforward, machine learning, natural language processing, neural networks
-
- - \ No newline at end of file diff --git a/site/natural-language-processing/reading/information-retrieval.html b/site/natural-language-processing/reading/information-retrieval.html deleted file mode 100644 index db3d63b..0000000 --- a/site/natural-language-processing/reading/information-retrieval.html +++ /dev/null @@ -1,301 +0,0 @@ - - - - - - Information Retrieval - - - - - -
- -

Information Retrieval

-
- Last modified: 2025-01-07 - -
-
-

Information Retrieval

-

IR in general is the process of obtaining information based on user queries, and can be applied to pretty much any form of media. Probably the most prevalent form of IR that we use every day is through search engines.

-

Ad Hoc Retrieval

-

A user poses a query to a retrieval system, which then returns an ordered set of documents from some collection. A document refers to whatever unit of text the system indexes and retrieves (e.g. a webpage, a book, a tweet, etc.). The collection is the set of all documents that the system has indexed. A term can correspond to either a word, phrase, or some other unit of text which documents are indexed by. A query is therefore a set of terms.

-

A simple architecture for an IR system is as follows:

- -
persistent storage
- +-----------+++
- | Documents ||| ----> Indexing/Preprocessing ----> Inverted Index
- +-----------+++                                         |
-                                                         |
-                                                         v
-
- User Query ---> Query Processing ---(query vector)--> Search
-    ^                                                    |
-    |                                                    |
-    +---------------(ranked docs)------------------------+
-
-

Usually, we'll want to also persist the inverted index to disk, so that we don't have to recompute it every time we want to search, but online queries will at least usually be served by using an in-memory index.

-

We can map queries and documents both to vectors based on unigram word counts, and then use cosine similarity between vectors to rank documents. This is an example of the bag-of-words model, since words are considered independently of their positions.

-

Term weighting (tf-idf)

-

Using raw word counts isn't very effective. We instead compute a term weight for each document word (e.g. tf-idf or BM25). For tf-idf (term frequency-inverse document frequency), we compute the term frequency (tf) and inverse document frequency (idf) for each term in each document. The tf is the number of times a term appears in a document, and the idf is the log of the total number of documents divided by the number of documents containing the term. The tf-idf score is then the product of these two values.

-

$$ -\text{tf}{t, d} = \begin{cases} - 1 + \log(t, d) > 0 \ - 0 & \text{otherwise} -\end{cases} -$$} \text{count}(t, d) & \text{if count

-

For intuition behind using $log$, if $w_1$ appears $100$ times in a document, and $w_2$ only once, it doesn't mean that $w_1$ is $100$ times more important. Note that alternative definitions of tf exist, e.g. $\log_{10}(1 + \text{count}(t, d))$.

-

On the other hand, the document frequency is the number of documents containing a term. The idf is then defined as:

-

$$ -\text{idf}t = \log \right) -$$} \left( \frac{N}{\text{df}_t

-

where $N$ is the total number of documents in the collection. Therefore, for a word that is contained in every document, we'd have an $idf$ of 0. The tf-idf score is then:

-

$$ -\text{tf-idf}{t, d} = \text{tf}} \times \text{idft = \begin{cases} - (1 + \log(t, d) > 0 \ - 0 & \text{otherwise} -\end{cases} -$$} \text{count}(t, d)) \times \log_{10} \left( \frac{N}{\text{df}_t} \right) & \text{if count

-

Document scoring

-

We can then score a document $d$ by the cosine of its vector $v_d$ with the query vector $v_q$:

-

$$ -\text{score}(q, d) = cos(v_q, v_d) = \frac{v_q \cdot v_d}{|v_q| |v_d|} -$$

-

Alternatively, you can think of the cosine as the dot product of the document and query unit vectors, e.g.:

-

$$ -\text{score}(q, d) = cos(v_q, v_d) = \frac{v_t}{|v_q|} \cdot \frac{v_d}{|v_d|} -$$

-

Then, plugging in the tf-idf scores:

-

$$ -\text{score}(q, d) = \sum_{t \in q} \frac{\text{tf-idf}{t, q}}{\sqrt{\sum} \text{tf-idf}^2(q_i, q)}} \times \frac{\text{tf-idf{t, d}}{\sqrt{\sum -$$} \text{tf-idf}^2(d_i, d)}

-

Many variations exist, particularly ones that drop terms in order to reduce computation required. A notable variant is BM25, which introduces parameters $k$ to adjust balance between $tf$ and $idf$, and $b$ which controls the importance of document length normalization.

-

$$ -\text{score}(q, d) = \sum_{t \in q} \log \left( \frac{N}{\text{df}t} \right) \cdot \frac{tf -$$}}{k(1 - b + b \cdot \frac{|d|}{|d_{avg}|}) + tf_{t, d}

-

Where $d_{avg}$ is the average document length in the collection. When $k = 0$, BM25 reverts to no use of term frequency, just like a binary selection of terms in the query (plus idf). A large $k$ results in raw term frequency (plus idf). $b$ ranges from $1$ (scaling by document length) to $0$ (no scaling). Reasonable defaults for these parameters are $k = [1.2, 2.0]$ and $b = 0.75$.

-

Quick aside: stop words

-

Stop words are common words that would traditionally be removed from the text before indexing, since they don't add much information. However, tf-idf already does a good job of downweighting common words, so stop words are less important in modern systems, an are often included in the index to make search for phrases easier.

-

Inverted Index

-

Using an inverted index, want to be able to find all documents $d \in C$ that contain a term $q \in Q$. The index is composed of two parts: a dictionary and a postings list. The dictionary is a collection of terms (designed to be efficiently accessed) which map to a postings list for the term. A posting list is the list of document IDs associated with each term, which can also contain additional metadata (e.g. term frequency, positions, etc.).

-

This gives us an efficient access pattern for computing tf-idf scores for documents, since we can look up the postings list for each term in the query. However, alternatives, especially for question answering, exist (e.g. Chen et al. 2017).

-

Evaluation

-

Use precision, the fraction of returned docs that are relevant, and recall, the fraction of all relevant docs that are returned.

-

Assume that each document in our IR system is either relevant or not relevant to a query. Further, let $U$ be the set of all relevant documents, $T$ be the set of ranked documents returned, and $R$ be the set of relevant documents in $T$. Then, we can define precision and recall as:

-

$$ -\text{precision} = \frac{|R|}{|T|} \quad \text{recall} = \frac{|R|}{|U|} -$$

-

Note that recall always increases, e.g. it isn't penalized by returning an irrelevant document. Precision, on the other hand, can decrease if we return irrelevant documents. It is useful to plot precision-recall curves, which show the tradeoff between precision and recall as we vary the number of documents returned.

-

$$ -\text{InterpolatedPrecision} = \text{maxPrecision}(i) \text{ for } i \ge r -$$

-
def interpolate_PR_curve(precision, recall):
-    """
-    plot averaged precision values at 11 fixed levels of recall (0 to 100 by 10)
-    """
-    recall_levels = np.linspace(0, 1, 11)
-    interpolated_precision = np.zeros_like(recall_levels)
-    for i, r in enumerate(recall_levels):
-        interpolated_precision[i] = np.max(precision[recall >= r])
-    return interpolated_precision, recall_levels
-
-

Mean Average Precision (MAP)

-

Assume $R_r$ is the set of relevant documents at or above $r$ in the ranked list. Then, the average precision at $r$ is:

-

$$ -\text{AP} = \frac{1}{|R_r|} \sum_{d \in R_r} \text{Precision}_{r}(d) -$$

-

Where $\text{Precision}_{r}(d)$ is the precision measured at the rank $r$ where document $d$ was retrieved. For an ensemble of queries $Q$, we average the AP over all queries to get the MAP:

-

$$ -\text{MAP} = \frac{1}{|Q|} \sum_{q \in Q} \text{AP}(q) -$$

-

IR with Dense Vectors

-

tf-idf and BM25 both kind of suck in a way (read vocabulary mismatch problem). Instead, we need to handle synonyms by using dense vectors (as opposed to sparse ones like word counts). This is implemented today via encoders like BERT.

-

The general approach is to present both the query and the document to a single encoder, allowing the transformer self-attention to see all tokens of both the query and the document, thus also building a representation that is sensitive to the meanings in both. Then, a linear layer can be put on top of the [CLS] token to predict the similarity score for the query and document.

-

$$ -z = BERT(q;[SEP];d)[CLS] -$$

-

$$ -\text{score}(q, d) = \text{softmax}(U(z)) -$$

-

Note: BERT was trained using [CLS] sen A [SEP] sen B [SEP]. [SEP] is used to help the model distinguish between the two sentences. [CLS] is used to represent the entire sentence.

-
- -
- - \ No newline at end of file diff --git a/site/natural-language-processing/tf-idf.py b/site/natural-language-processing/tf-idf.py deleted file mode 100644 index b1574be..0000000 --- a/site/natural-language-processing/tf-idf.py +++ /dev/null @@ -1,128 +0,0 @@ -import numpy as np -import os -from typing import Dict, List, Tuple - - -def read_markdown_files(root_dir: str, max_depth: int = 8) -> Dict[str, str]: - - corpus = {} - - def process_dir(path: str, depth: int = 0): - if depth > max_depth: - return - - try: - for entry in os.listdir(path): - full_path = os.path.join(path, entry) - if os.path.isdir(full_path): - process_dir(full_path, depth + 1) - elif entry.endswith(".md"): - try: - with open(full_path, "r") as f: - corpus[full_path] = f.read() - except Exception: - continue - except Exception: - return - - process_dir(root_dir) - return corpus - - -def create_index(corpus: Dict[str, str]) -> Tuple[Dict[str, Dict[str, int]], List[str]]: - - inv_idx = {} - for doc_name, content in corpus.items(): - words = content.split() - for word in words: - if word not in inv_idx: - inv_idx[word] = {} - inv_idx[word][doc_name] = inv_idx[word].get(doc_name, 0) + 1 - - return inv_idx, list(inv_idx.keys()) - - -def calculate_tfidf( - corpus: Dict[str, str], inv_idx: Dict[str, Dict[str, int]], word_list: List[str] -) -> Dict[str, np.ndarray]: - - N = len(corpus) - tfidf = {} - - for doc_name, content in corpus.items(): - doc_words = content.split() - doc_len = len(doc_words) - tfidf[doc_name] = np.zeros(len(word_list)) - - for i, word in enumerate(word_list): - if word in inv_idx and doc_name in inv_idx[word]: - tf = inv_idx[word][doc_name] / doc_len - idf = np.log(N / len(inv_idx[word])) - tfidf[doc_name][i] = tf * idf - - return tfidf - - -def search( - query: str, - tfidf: Dict[str, np.ndarray], - word_list: List[str], - inv_idx: Dict[str, Dict[str, int]], - num_results: int = 5, -) -> List[Tuple[str, float]]: - - N = len(tfidf) - query_vec = np.zeros(len(word_list)) - - for i, word in enumerate(word_list): - if word in query.split() and word in inv_idx: - query_vec[i] = np.log(N / len(inv_idx[word])) - - similarities = [ - (doc_name, float(np.dot(doc_vec, query_vec))) - for doc_name, doc_vec in tfidf.items() - ] - - return sorted(similarities, key=lambda x: x[1], reverse=True)[:num_results] - - -def main(): - - try: - root_dir = ( - input("Enter root directory to search (default '.'): ").strip() or "." - ) - print(f"Reading markdown files from {root_dir}...") - - corpus = read_markdown_files(root_dir) - if not corpus: - print("No markdown files found!") - return - - print(f"Found {len(corpus)} markdown files") - print("Building search index...") - - inv_idx, word_list = create_index(corpus) - tfidf = calculate_tfidf(corpus, inv_idx, word_list) - - print("Search system ready!") - - while True: - query = input('\nEnter search query (or "exit" to quit): ').strip() - if query.lower() == "exit": - break - - results = search(query, tfidf, word_list, inv_idx) - - print("\nSearch results:") - for doc_name, similarity in results: - print(f"{doc_name}: {similarity:.4f}") - - except KeyboardInterrupt: - print("\nExiting...") - except Exception as e: - print(f"\nError: {str(e)}") - - -if __name__ == "__main__": - main() diff --git a/site/networks/0-foundation/1-network-components-and-protocols.html b/site/networks/0-foundation/1-network-components-and-protocols.html deleted file mode 100644 index 9ba1789..0000000 --- a/site/networks/0-foundation/1-network-components-and-protocols.html +++ /dev/null @@ -1,355 +0,0 @@ - - - - - - 1 Network Components And Protocols - - - - - -
- -

1 Network Components And Protocols

-
- Last modified: 2024-01-11 - -
-
-

Network Components

-

Parts of a network

-

Application (app, user) - The application is the program that is running on the computer. It is the program that is using the network to communicate with other computers. Examples of applications are web browsers, email clients etc.

-

Host (end system, edge device, node) - The host is the computer that is running the application. It is the computer that is using the network to communicate with other computers. Examples of hosts are desktop computers, laptops, mobile phones etc.

-

Router (switch, node, hub) - Device used to relay messages between links. Connects networks together. Examples of routers are home routers/access points, cabel/DSL modems etc.

-

Link (channel) - A connection between nodes. Examples of links are Ethernet cables, fiber optic cables, wireless connections etc.

- - - -

Messages are broadcast. All nodes in range recieve the message. Often, in graph depictions of a network, only the logical (but not all possible) links are shown.

-

Network Names by Scale

-

Personal Area Network (PAN) - A network that is available in a single person's vicinity.

-

Examples: Bluetooth, USB, FireWire etc.

-

Local Area Network (LAN) - A network that is available in a single building.

-

Examples: Ethernet, WiFi etc.

-

Metropolitan Area Network (MAN) - A network that is available in a city.

-

Examples: cable TV, DSL etc.

-

Wide Area Network (WAN) - A network that is available in a country or geographic location.

-

Examples: a large ISP, 3G/4G wireless networks etc.

-

Internet - A network that is available globally.

-

Examples: the Internet.

-

When you connect multiple networks, you get an internetwork, or internet. The Internet (capital I) is the internet we all know and love.

-

Switched Network

-

Switched networks forward messages from node-to-node, until they reach their destination. The two most common switched networks are circuit-switched (phones) and packet-switched (most computer networks) networks.

-
    +-- (Host)      --+
-    |                 |
-(Link)                |
-    |                 |  logical
-    +-- (Host)        |    link
-    |                 |
-(Link)                |
-    |                 |
-    +-- (Host)      --+
-
-

Packet switched networks (PSN) send data in discrete chunks, called packets, or messages. PSNs typically use store-and-forward switching, where the entire packet is received and loaded into memory, then forwarded to the next node. This is opposed to a circuit switched network, where a stream of data is sent over a maintained connection.

-

Networks use an address to identify the destination of a packet. Packets can be sent from node to node (unicast), but also to all other nodes (broadcast), or to a subset of nodes (multicast).

-

Network Boundaries

-
(Router) --- (Host) --- client
-   |
-(Link)
-   |
-(Router) --- (Host) --- server
-
-

What part is the network?

-

Everything that isn't the application level. Some people do and don't include the host, but in this course we do.

-

Can think of "the cloud" as a generic network...

-
   +-- (Host) --- client
-   |
-(Cloud)
-   |
-   +-- (Host) --- server
-
-

Key Interfaces

-

The network is designed to be modular, and there are clearly defined interfaces betweem (1) apps and the network, and (2) the network components themselves.

-

This is achieved through protocols and layering.

- -

"Protocols are horrizontal, and layers are vertical."

-
# define protocols X, Y,
-# where Y is a lower below X
-
-
-   (comm using X)
-X <---------------> X  <- (peers)
-^                   ^
-| <- (Y service) -> |
-|                   |
-Y <---------------> Y  <- (peers)
-    (comm using Y)
-
-

Examples of protocols:

-

TCP, UDP, HTTP, FTP, SMTP, POP3, IMAP, DNS, DHCP, ARP, ICMP, IP, Ethernet, WiFi, Bluetooth, USB, FireWire, DSL, cable TV, 3G/4G, etc.

-

Example of a stack

-
 (browser)
-    ||
-+--------+
-| HTTP   |
-+--------+
-| TCP    |
-+--------+
-| IP     |
-+--------+
-| 802.11 |
-+--------+
-    ||
-    ++==>
-
-

Encapsulation

-

Protocol layering is built upon literal encapsulation of data. Each lower level protocol wraps the higher level protocol's data in its own format with extra information. Similar to putting a letter in an envelope, and then sending it to someone in the mail.

-

The message "on the wire" for the above stack might look like...

-
                    +------+
-                    | HTTP |
-                    +------+
-              +-----+------+
-              | TCP | HTTP |
-              +-----+------+
-         +----+-----+------+
-         | IP | TCP | HTTP |
-         +----+-----+------+
-+--------+----+-----+------+
-| 802.11 | IP | TCP | HTTP |
-+--------+----+-----+------+
-
-

When two nodes communicate, the sender builds up these layers until the data is ready to be transported over the physical medium. Then, once the data is recieved, the reciever peels back the layers until it reaches the application layer.

-

It is more involved that this diagram in practice. Trailers and headers of each request segment are needed, and the content is often encrypted or compressed. Furthermore, segmentation and reassembly happens when nodes communicate as well.

-

Demultiplexing

-

When a message is recieved, it needs to be passed through exactly the protocols that use it. This is done using demultiplexing keys found in the headers of each protocol. Ex: IP protocol field, TCP port number, etc.

-

Advantages of Layering

- -

For example, when a person submits a request on their home wifi, the router strips the WiFi header and adds an ethernet header to send it to the server.

-

Disadvantages of Layering

- -

OSI Layers

-

Application Layer

-

Services that are used with end user applications. Examples: HTTP, FTP, SMTP, POP3, IMAP, DNS, DHCP, etc.

-

Presentation Layer

-

Formats the data so it can be understood by the application layer. Also handles encryption and compression. Examples: JPEG, MPEG, ASCII, etc.

-

Session Layer

-

Manages the connection between two nodes. Examples: NetBIOS, PPTP, etc.

-

Transport Layer

-

Responsible for the transport protocol and error handling. Examples: TCP, UDP, etc.

-

Network Layer

-

Responsible for routing and addressing. Reads the IP address from a packet. Examples: Routers, Layer 3 switches, etc.

- -

Responsible for the physical addressing. Reads the MAC address from a data packet/frame. Examples: Switches, bridges, etc.

-

Physical Layer

-

Transfer data on a physical medium. Examples: Hubs, NICS, Cables, etc.

-

The actual Internet Protocol Stack

-
+-------------+---------------+
-| Application | SMTP, HTTP,   |
-|             | RTP, DNS      |
-+-------------+---------------+
-| Transport   | TCP, UDP      |
-+-------------+---------------+
-| Internet    | IP            |
-+-------------+---------------+
-| Link        | Ethernet, DSL,|
-|             | 3G/4G, WiFi,  |
-+-------------+---------------+
-
-

Course Reference Model

- -
- -
- - \ No newline at end of file diff --git a/site/networks/0-foundation/2-physical-layer.html b/site/networks/0-foundation/2-physical-layer.html deleted file mode 100644 index 687b276..0000000 --- a/site/networks/0-foundation/2-physical-layer.html +++ /dev/null @@ -1,267 +0,0 @@ - - - - - - 2 Physical Layer - - - - - -
- -

2 Physical Layer

-
- Last modified: 2024-01-18 - -
-
-

The Physical Layer

-

Scope: How signals are used to transfer bits over a link. i.e, how analog signals are converted to digital signals, and vise versa.

-

Coding and Modulation

-

A modem (modulator-demodulator) converts digital signals to analog signals, and vise versa.

-

A simple coding

-

A high positive voltage for 1, and a low negative voltage for 0. This is called NRZ(Non-Return-to-Zero). Each time interval (symbol) is like a sample point.

-

Problems?

-

Only 1 bit/symbol. Can use more than just 2 voltage levels to get more bits/symbol. To get N bits/symbol, need 2^n voltage levels. There is a tradeoff between encoding efficiency and the sensitivity to noise.

-

There are many other practical coding schemes, all of which are driven by engineering considerations.

-

Clock Recovery

-

Reciever needs requent signal transitions to decode bits. Several possible designs, including Manchester Coding and Scrambling.

-

A simple solution is to alternate between positive/negative, and zero voltages. This is return to zero (RZ) coding.

-
    0       1        1      1       0
-+V |        ___     ___     ___
-   |   |   |   |   |   |   |   |   |   |
-   |   |   |   |   |   |   |   |   |   |
-0  |   |___|   |___|   |___|   |___|   |
-   |   |   |   |   |   |   |   |   |   |
-   |   |   |   |   |   |   |   |   |   |
--V |___|   |   |   |   |   |   |   |___|
-
-

Better Solution

- -

Modulation vs. Coding

-

In order to agree on the timing of data streams, AKA the start and end of a symbol being transmitted, you need to have a common clock between the two systems that are communicating.

-

With coding, signal is sent directly on a wire. This doesn't work well for wireless, so we use modulation. Modulation carries a signal by varying the frequency, amplitude, or phase of a carrier wave. Baseband is the original signal, and passband is the modulated signal. We can modulate a signal by varying the amplitude, frequency, or phase of a carrier wave.

-

Some examples:

- -

WiFi for example goes all in and listens on an entire band of frequencies instead of just the binary 2 frequencies.Modern WiFi uses 256 frequency levels.

-

Key Points

- - -

Two main parameters:

- -

Additional info:

- -

Message Latency

-

Latency is the time it takes for a message to travel from one end of a link to the other. It is the sum of the transmission delay (time to put bits on wire) and the propagation delay (time for bits to travel from one end of the link to the other).

-
Transimission Delay:
-T (delay) = L (message length) / R (rate) = L/R seconds
-
-Propagation Delay:
-P (delay) = D (distance) / S (speed) = D/(2/3 * C) = 3D/2C seconds
-
-Total Latency:
-L_t = T + P = L/R + 3D/2C
-
-

Example

-
Broadband cross-country link:
-P = 50ms, R = 10Mbps, L = 1MB
-
-L_t = 1MB/10MBps + 50ms = .1s + .05s = .15s
-
-

Cut Through Routing

-
- -
- - \ No newline at end of file diff --git a/site/networks/0-foundation/3-performance.html b/site/networks/0-foundation/3-performance.html deleted file mode 100644 index d2cbc1e..0000000 --- a/site/networks/0-foundation/3-performance.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - 3 Performance - - - - - -
- -

3 Performance

-
- Last modified: 2024-01-12 - -
-
-

Performance

-

Measured in bandwidth (or throughput) and latency (or delay).

-

Bandwidth: the number of bits per second

-
- -
- - \ No newline at end of file diff --git a/site/networks/0-foundation/information-theory.html b/site/networks/0-foundation/information-theory.html deleted file mode 100644 index 20e0ed1..0000000 --- a/site/networks/0-foundation/information-theory.html +++ /dev/null @@ -1,214 +0,0 @@ - - - - - - Information Theory - - - - - -
- -

Information Theory

-
- Last modified: 2024-01-12 - -
-
-

Key Channel Properties

- -

Nyquist Limit

-

Maximum symbol rate is 2B symbols/sec.

-

If there are V signal levels, max bit rate is:

-

R = 2B log_2(V) bits/sec

-

Shannon Capacity

-

Capacity (C) limit is the maximum lossless information carrying rate of a channel.

-

C = B log_2(1 + S/N) bits/sec

- -

Can't beat the Shannon limit

-

Wired/Wireless Perspecitive

-
- -
- - \ No newline at end of file diff --git a/site/networks/1-physical/coding-and-modulation.html b/site/networks/1-physical/coding-and-modulation.html deleted file mode 100644 index 9acb349..0000000 --- a/site/networks/1-physical/coding-and-modulation.html +++ /dev/null @@ -1,260 +0,0 @@ - - - - - - Coding And Modulation - - - - - -
- -

Coding And Modulation

-
- Last modified: 2024-02-03 - -
-
-

Coding and Modulation

-

How information sent over a link?

-

Coding

-

None-return-to-zero (NRZ)

-

1 is represented by a high voltage and a 0 is represented by a low voltage. This can however lead to problems with long runs of 1s or 0s.

- -

Clock Recovery

-

It is difficult to recover the clock from a NRZ signal that doesn't transition often. This is because the receiver doesn't know when to sample the signal. Several techniques are used to solve this problem, including manchester and scrambling.

-

Return-to-Zero (RZ)

-

After each bit, the signal returns to zero. This makes it easier to recover the clock, since the receiver can sample the signal after each return to zero.

-

Designing Codes

-

One can design codes that are more robust to noise, or that have better clock recovery properties. In general, you map one set of bits to another set of bits, and then send the new bits over the link.

-

For example, in 4b/5b, we let $S$ be the set of all possible 4-bit sequences, and let $C$ be the set of all possible 5-bit sequences. Then we can define a code $f: S \to C$:

-

$$ -f(0000) = 11110 \ -f(0001) = 01001 \ -f(0010) = 10100 \ -\vdots -$$

-

One can then choose a set $C$ such that there aren't long runs of 1s or 0s, or such that the clock can be recovered easily. Since there are left over symbols in $C$, we can use them to represent control information, such as the start of a frame.

-

Modulation

-

Modulation transmits a digital signal over an analog channel by modulating a carrier. Baseband is the original signal, and passband is the modulated signal. Keying is the digital form of modulation (equivalent to coding, but using modulation instead).

-

Amplitude Shift Keying (ASK)

-

In ASK, the amplitude of the carrier is changed to represent the digital signal. For example, a 1 might be represented by a high amplitude, and a 0 by a low amplitude. This faces the same problem as NRZ, in that it is difficult to recover the clock.

-

Frequency Shift Keying (FSK)

-

In FSK, the frequency of the carrier is changed to represent the digital signal. For example, a 1 might be represented by a high frequency, and a 0 by a low frequency. This is more robust to noise than ASK.

-

Phase Shift Keying (PSK)

-

In PSK, the phase of the carrier is changed to represent the digital signal. For example, a 1 might be represented by a phase of 0 degrees, and a 0 by a phase of 180 degrees. PSK can support more bits per symbol than ASK or FSK, leading to higher data rates.

- - -

Properties of the Channel

- -

Latency

-

The delay in sending a signal from the transmitter to the receiver. This is the sum of the propagation delay and the transmission delay.

- -

$$ -D_{\text{transmission}} = \frac{\text{message size}}{\text{transmission rate}} = \frac{M}{R} -$$

-

where $R$ is the rate of the link in bits/sec and $M$ is the size of the frame in bits.

-

$$ -D_{\text{propagation}} = \frac{\text{distance}}{\text{speed of the medium}} = \frac{d}{.66c} -$$

-

where $d$ is the distance between the transmitter and the receiver, and $c$ is the speed of light.

-

Overall, we can calulate the latency as:

-

$$ -L = \frac{M}{R} + D_{\text{propagation}} -$$

-

Bandwidth-Delay Product

-

Messages take up space on the wire, and the wire can only hold so many bits at a time. The bandwidth-delay product is the maximum number of bits that can be in transit at any time.

-

$$ -\text{Bandwidth-Delay Product} = BD = R \cdot D_{\text{propagation}} -$$

-

Usually, either the bandwidth or the delay is the bottleneck. The BD gives us a sense of the overall capacity of the link

-
- -
- - \ No newline at end of file diff --git a/site/networks/1-physical/media.html b/site/networks/1-physical/media.html deleted file mode 100644 index 8ba05c3..0000000 --- a/site/networks/1-physical/media.html +++ /dev/null @@ -1,243 +0,0 @@ - - - - - - Media - - - - - -
- -

Media

-
- Last modified: 2024-02-03 - -
-
-

Media

-

Media propagates signals that carry information.

-

Wires

-

Twisted Pair

- -

EX: Ethernet, DSL

-

Coaxial Cable

- -

EX: Cable TV, Internet

-

Fiber

- -

EX: Internet backbone

-

Wireless

-

Sends signals in all directions through a region of space. Nearby signals can interfere with each other, especially if they are on the same frequency; must coordinate use over time and frequency.

-

Wifi largely uses unlicensed spectrum, which is free to use but can be crowded. Interference can be a big problem. For example, turning on your microwave can interfere with your wifi.

-

Signals can take multiple paths (multipath), and are affected by physical barriers. The higher the frequency, the more easily it is absorbed by walls and other obstacles (this is why 5G sucks).

-

Channel Properties

- -

Nyquist Limit

-

The max symbol-rate (rate at which symbols are sent) is twice the bandwidth. This would mean maintaining a maximum frequency, and sending a signal for each peak or trough of the wave.

-

If there are $V$ signal levels, ignoring noise, the max data rate is $2B \log_2(V)$ bits/sec.

-

Shannon Capacity

-

The number of levels we can distinguish is limited by the ratio of signal power to noise power. The signal-to-noise ratio (SNR) is the ratio of the signal power to the noise power, and the higher the SNR, the more levels we can distinguish. Usually measured in decibels (dB).

-

$$ -SNR_{dB} = 10 \log_{10} \left( \frac{S}{N} \right) -$$

-

Capacity (C) is the max lossless data rate over a channel. Don't ask me how to derive this...

-

$$ -C = B \log_2 \left( 1 + \frac{S}{N} \right) -$$

-

Note that increasing bandwidth increases the capacity linearly, but increasing SNR increases the capacity logarithmically.

-
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/errors.html b/site/networks/2-direct-links/errors.html deleted file mode 100644 index 8cb458a..0000000 --- a/site/networks/2-direct-links/errors.html +++ /dev/null @@ -1,321 +0,0 @@ - - - - - - Errors - - - - - -
- -

Errors

-
- Last modified: 2024-02-08 - -
-
-

Error Detection and Correction

-

Some bits will inevitably be recieved in error. Noise may flip the bits recieved over the network. We need to be able to...

- -

Approach: Add Redundancy

- -

Generally, a codeword is a $D$-bit message with $R$ check bits added to it. The sender computes the check bits and appends them to the message. The reciever then verifies the check bits by recomputing them and comparing them to the recieved check bits.

-

Example Code:

-

$$ -1 \to 11\ -0 \to 00\ -x \to xx\ -$$

-

For example: -$$ -101110 \to 101110101110 -$$

-

Can detect errors with this code up to 1 bit. However, no guarantee if more bits are flipped. Also cannot correct errors. This sucks.

-

Intuition

-

Let $S$ be the set of all possible $n$-bit sequences, and let $C$ be the set of all $n$-bit code words with $D$ data bits and $R$ check bits. We have $C \subset S$, and we want to choose $C$ such that the probability of a random $D$-bit sequence being in $C$ is low.

-

Consider a random $n$-bit sequence $x \in S$...

-

$$ -\mathbb{P}(x \in C) = \frac{|C|}{|S|} = \frac{2^R}{2^n} = 2^{-D} -$$

-

Error correction/detection is hard because even the check bits can be corrupted. Given $d$ errors, we can detect and correct $d$ errors if the distance between any two code words is $2d + 1$.

-

Hamming Distance

-

Distance: The number of bit flips needed to change from one valid code word to another. The distance of a code is the minimum distance between any two code words. Letting $C$ be the set of all code words, the distance is:

-

$$ -\min_{x, y \in C, x \neq y} d(x, y) -$$

- -

Internet Checksum

-

Sum up chunks of data and append the sum to the end of the data. If the sum is 0 on the recieving end, then the data is valid.

-

This code has a distance of 2, so it can detect 1 bit errors, and can correct 0 bit errors (since it can't correct any errors)

-

Internet Checksum Algorithm

-

sender

-
    -
  1. Split up data into 4 byte chunks.
  2. -
  3. Sum up all the 4 byte chunks, wrapping around for carry beyond 4 bytes.
  4. -
  5. netgate append the checksum to the end of the data.
  6. -
-

reciever

-
    -
  1. Split up data into 4 byte chunks.
  2. -
  3. Sum up all the 4 byte chunks, wrapping around for carry beyond 4 bytes.
  4. -
  5. netgate append the checksum to the end of the data.
  6. -
  7. If the sum is 0, then the data is valid.
  8. -
-
def internet_checksum(data):
-    checksum = 0
-    for i in range(0, len(data), 4):
-        chunk = data[i:i+4]
-        checksum += int.from_bytes(chunk, 'big')
-    return checksum.to_bytes(4, 'big')
-
-

Cyclical Redundancy Check (CRC)

-

Given a generator polynomial $C$ and a message of $n$ bits, generate $k$ bits such that the $n + k$ bit message is divisible by $C$. Works with binary values that operate over the field $\mathbb{Z}_2$ (mod 2 arithmetic).

-

CRC Algorithm

-

sender

-
    -
  1. create binary representation of polynomial as divisor
  2. -
  3. append 0s to the end of the data to be sent, equal to the degree of the polynomial
  4. -
  5. divide the data by the polynomial, using XOR
  6. -
  7. change redundant bits to remainder
  8. -
-

reciever

-
    -
  1. create binary representation of polynomial as divisor
  2. -
  3. divide the data by the polynomial, using XOR
  4. -
  5. if the remainder is 0, then the data is valid
  6. -
-

Notes:

-

x + 1 as a generating polynomial is just a parity bit!

-

Hamming Code

-

A code with distance 3. Can detect 2 bit errors and correct 1 bit errors.

-

Uses $n = 2^k - k - 1$. Put check bits in positions that are powers of 2, and fill in the rest with data bits. Check bits are calculated by checking the bits in the ith position for all data bits whose ith bit is 1.

-
def hamming_encode(data):
-    n = len(data)
-    k, acc = 0, 1
-    while acc < n + k + 1:
-        k += 1
-        acc *= 2
-
-    code = [0] * (n + k)
-    j = 0
-    for i in range(1, n + k + 1):
-        if i & (i - 1) == 0:
-            code[i - 1] = 0
-        else:
-            code[i - 1] = data[j]
-            j += 1
-    for i in range(k):
-        j = 2 ** i
-        for l in range(j, n + k + 1):
-            if l & j:
-                code[j - 1] ^= code[l - 1]
-    return code
-
-

Algorithm

-

Send

-
    -
  1. Put all parity bits in positions that are powers of 2 (1, 2, 4, 8, 16, etc.)
  2. -
  3. Fill in all other positions with data bits.
  4. -
  5. calculate parity bits for each of the ith check bits. (check the bits in ith position for all data bits whose ith bit is 1)
  6. -
-

Recieve

-
    -
  1. calculate parity bits for each of the ith check bits. (check the bits in ith position for all data bits whose ith bit is 1 of its index is a power of 2)
  2. -
  3. If the parity bits are all 0, then the data is valid (remove check bits from data bits)
  4. -
  5. If the parity bits are not all 0, then the data is invalid. The syndrome (concat and reverse the parity bits) is the index of the bit that is wrong. Flip that bit.
  6. -
-

Detection vs. Correction

- -

Error correction is heavily used in the physical layer. Low Density Parity Check (LDPC) codes are used in 802.11, DVB, WiMAX etc, and convolutional codes are used a lot in practice. On the other hand, detection combined with retranmission is used in the data link layer and above for residual errors.

-
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/framing.html b/site/networks/2-direct-links/framing.html deleted file mode 100644 index 014db0a..0000000 --- a/site/networks/2-direct-links/framing.html +++ /dev/null @@ -1,275 +0,0 @@ - - - - - - Framing - - - - - -
- -

Framing

-
- Last modified: 2024-02-04 - -
-
-

Byte Oriented Protocols, Point-to-point protocol (PPP)

-

Byte-Oriented Framing

- -

Length Field Approach

- -

Sentinel-based Approach/Byte Stuffing

- -

PPP Frame Format

- -

LCP Protocol and Negotiation

- -

Bit Oriented Protocols (HDLC)

-

Bit-Oriented Framing

- -

HDLC Frame Format

- -

Bit Stuffing in HDLC

- -

Frame Size Dependency

- -

Clock-Based Protocols (SONET)

-

Clock-Based Framing in SONET

- -

SONET Frame Structure

- -

Overhead and Payload

- -

Multiplexing in SONET

- -

Concatenation in SONET

- -
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/multiple-access.html b/site/networks/2-direct-links/multiple-access.html deleted file mode 100644 index 5b98f97..0000000 --- a/site/networks/2-direct-links/multiple-access.html +++ /dev/null @@ -1,282 +0,0 @@ - - - - - - Multiple Access - - - - - -
- -

Multiple Access

-
- Last modified: 2024-02-04 - -
-
-

Multiple Access

-

Multiplexing

-

Multiplexing is the networking concept of sharing a resource among multiple clients. Network traffic is generally bursty, to the point that two concurrent users each using 1 Mbps of bandwidth might only need 1.5 Mbps of bandwidth to share. Multiplexing allows for this sharing.

-

Time Division Multiplexing (TDM)

-

Users take turns on a fixed schedule, often being scheduled in a round-robin fashion.

-

Frequency Division Multiplexing (FDM)

-

Users are assigned different frequency bands, and can transmit at the same time, interfering minimally with each other.

-

TDM vs. FDM

-

In TDM, users send at a high rate for a short time, while in FDM, users send at a low rate contantly. For a fixed number of users, TDM might be better if the users are bursty, while FDM might be better if the users are constant.

-

Ex: TV and radio use FDM, while GSM (2G cellular) uses TDM within FDM.

-

Controlling Access

-

Two classes: centralized and distributed.

-

Centralized

-

Uses a privileged "Scheduler" to allocate resources/coordinate access. This usually scales well vertically and has low overhead, but is a single point of failure and doesn't hold up with the demands of the modern internet.

- -

Distributed

-

All of the participants "figure it out" themselves. Scaling this is really hard, but it operates well under low load, is easy to set up, and is very fault-tolerant (not to mention the benefits of being decentralized).

- -

Distributed (Random) Access

-

Assumes noone is in charge, and that everyone is trying to access the channel at the same time.

- -

Carrier Sense Multiple Access (CSMA)

-

Another distributed access protocol. The sender listens to the channel (only really works for wired) before sending, and if it's busy, waits. This is better than ALOHA, but still has a high collision rate due to delay. Only a good idea for small BD-product links.

-

CSMA/CD (with Collision Detection):

-

Used in Ethernet. If a collision is detected, the sender stops sending and waits a random amount of time before trying again. Complicated because everyone involved in collision must be able to detect it.

-

Nodes have $2 \cdot D_{\text{propagation}}$ time to detect a collision. One can thus impose a minimum frame length of $2D# secones so that nodes can't finish sending a frame before the collision is detected. This is why Ethernet has a minimum frame size of $64$ bytes, and a maximum network length of $500$ m for Coaxial Ethernet, and $100$ m for Twisted Pair Ethernet.

-

CSMA "Persistence"

-

Cannot simply wait until the channel is free, as nodes might just continue to queue up until the channel becomes free and they collide. Instead, design such that heuristically, given $N$ queued senders, the probability of any given node sending is $1/N$.

-

Binary Exponential Backoff

-

Given some discrete base quantity of time $q$, after collision $i$, sender waits $t_{i} = q \cdot \text{rand}(0, 2^i - 1)$.

-
Q = 1 # seconds
-def send(frame):
-    t = 1
-    while not send_frame(frame):
-        time.sleep(Q * randint(0, t))
-        t *= 2
-
-

Math behind Binary Exponential Backoff

- -

$W_k = q \cdot \text{randchoice}([0, 2^k - 1])$

-

Now, calculate the expected value $E[W_k]$:

-

$E[W_k] = \sum_{i=0}^{2^k - 1} i \cdot q \cdot P(W_k = i \cdot q)$

-

Since $P(W_k = i \cdot q)$ is uniform $\forall i \in [0, 2^k - 1]$, it is equal to $2^{-k}$.

-

$E[W_k] = q \cdot 2^{-k} \cdot \sum_{i=0}^{2^k - 1} i$

-

$ \sum_{i=0}^{2^k - 1} i = (2^k - 1) \cdot \frac{2^k}{2}$ (sum of first $n$ natural numbers)

-

$E[W_k] = q \cdot 2^{-k} \cdot ( (2^k - 1) \cdot 2^{k - 1} )$

-

Simplify further:

-

$E[W_k] = q \cdot ( (2^k - 1) / 4 )$

-

Therefore, the expected value of the binary exponential backoff for a fixed discrete time interval $q$ is $E[W_k] = \frac{q}{4} \cdot (2^k - 1)$.

-

Ethernet

-

Classis Ethernet (IEEE 802.3), popular in the 80s and 90s. 10 Mbpd over shared coaxial cable. Multiple Access with 1-peristence CSMA/CD with BEB". Modern ethernet is based on switches, avoiding the need for CSMA/CD.

-

Ethernet Frames

- -
+----------------+
-| Preamble (8B)  |
-|                |
-+----------------+
-| Destination    |
-| Address (6B)   |
-+----------------+
-| Source Address |
-| (6B)           |
-+----------------+
-| Type (2B)      |
-+----------------+
-| Data (0-1500B) | --->
-|                | ---> network layer (IP packet)
-|      ...       | --->
-+----------------+
-| Padding (0-46B)|
-+----------------+
-| Checksum (4B)  |
-+----------------+
-
-
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/retransmission.html b/site/networks/2-direct-links/retransmission.html deleted file mode 100644 index ed5e3b3..0000000 --- a/site/networks/2-direct-links/retransmission.html +++ /dev/null @@ -1,277 +0,0 @@ - - - - - - Retransmission - - - - - -
- -

Retransmission

-
- Last modified: 2024-02-07 - -
-
-

Retransmission

-

Reliability is a key feature of a network, and there are measures in place accross the entire stack to ensure it.

-

Automatic Repeat reQuest (ARQ)

-

Often used when errors are common or must be corrected (e.g. wireless links). Receiver automatically acknowledges correct frames, and sender retransmits frames that are not acknowledged by a certain timeout.

- -

Timeouts

-

They need to be not too long (link is idle), but also not too short (link is busy). Timeouts are easy to set in a LAN, but harder over the internet where latency can vary greatly.

-

Sequence Numbers

-

Both frames and ACKs are numbered, so that the sender knows which frames are acknowledged. In stop and wait ARQ, the sequence number is 0 or 1. In go-back-N ARQ and sliding-window ARQ, the sequence number is a number modulo $2^k$.

-

Limitations of Stop and Wait ARQ

- -

Examples

-

(These are just pseudocode examples)

-
# Stop and Wait ARQ
-def sender():
-    while True:
-        frame = create_frame()
-        send_frame(frame)
-
-        ack_received = wait_for_ack()
-
-        if ack_received:
-            break
-
-def receiver():
-    while True:
-        frame = receive_frame()
-        process_frame(frame)
-
-        send_ack()
-
-
-
# Sliding Window ARQ
-def sender():
-    window_size = 3
-    frames = [create_frame() for _ in range(window_size)]
-    send_frames(frames)
-
-    acknowledged_frames = wait_for_acknowledgment()
-
-    # Move window forward
-    frames = frames[len(acknowledged_frames):] + [create_frame()]
-    send_frames(frames)
-
-def receiver():
-    while True:
-        frames = receive_frames()
-        process_frames(frames)
-
-        send_acknowledgment()
-
-
# Go-Back-N ARQ
-def sender():
-    window_size = 3
-    frames = [create_frame() for _ in range(window_size)]
-    send_frames(frames)
-
-    while True:
-        acknowledged_frames = wait_for_acknowledgment()
-
-        if not acknowledged_frames:
-            resend_frames(frames)
-
-def receiver():
-    expected_frame = 0
-
-    while True:
-        frames = receive_frames()
-
-        for frame in frames:
-            if frame.sequence_number == expected_frame:
-                process_frame(frame)
-                expected_frame += 1
-
-        send_acknowledgment()
-
-
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/switching.html b/site/networks/2-direct-links/switching.html deleted file mode 100644 index 558deb9..0000000 --- a/site/networks/2-direct-links/switching.html +++ /dev/null @@ -1,312 +0,0 @@ - - - - - - Switching - - - - - -
- -

Switching

-
- Last modified: 2024-02-25 - -
-
-

Switching

-

Switched Ethernet

-

L2 Switches are used within enterprise and university networks to connect multiple devices on the same network. Historically, they were used as a "bridge" to connect multiple Ethernet segments, but are now typically used in a "point-to-point" configuration.

-

Learning Switches are a type of L2 switch that learns the MAC addresses of devices connected to it. When a device sends a frame to the switch, the switch records the source MAC address and the port into its forwarding table. When a device sends a frame to the switch, the switch looks up the destination MAC address in its forwarding table and forwards the frame to the appropriate port. If the destination MAC address is not in the table, the switch floods the frame to all ports except the one it was received on.

-
#define BRIDGE_TAB_SIZE   1024  /* max size of bridging table */
-#define MAX_TTL           120   /* time (in seconds) before an entry is flushed */
-
-typedef struct {
-    MacAddr     destination;    /* MAC address of a node */
-    int         ifnumber;       /* interface to reach it */
-    u_short     TTL;            /* time to live */
-    Binding     binding;        /* binding in the Map */
-} BridgeEntry;
-
-int     numEntries = 0;
-Map     bridgeMap = mapCreate(BRIDGE_TAB_SIZE, sizeof(BridgeEntry));
-
-void
-updateTable (MacAddr src, int inif)
-{
-    BridgeEntry       *b;
-
-    if (mapResolve(bridgeMap, &src, (void **)&b) == FALSE )
-    {
-        /* this address is not in the table, so try to add it */
-        if (numEntries < BRIDGE_TAB_SIZE)
-        {
-            b = NEW(BridgeEntry);
-            b->binding = mapBind( bridgeMap, &src, b);
-            /* use source address of packet as dest. address in table */
-            b->destination = src;
-            numEntries++;
-        }
-        else
-        {
-            /* can't fit this address in the table now, so give up */
-            return;
-        }
-    }
-    /* reset TTL and use most recent input interface */
-    b->TTL = MAX_TTL;
-    b->ifnumber = inif;
-}
-
-

Distributed Spanning Tree Algorithm

-

Initially, all switches assume they are the root of the tree. Switches broadcast configuration messages that contain the `following:

- -

Each swtich listens to configuration messages being broadcasted, and remebers the "best" root it has seen. This is defined as:

- -

Once it identifies the best root, it adds one to the disance and broadcasts a configuration message with the new root and distance. If a switch realizes it isn't the root anymore, it stops broadcasting configuration messages and starts forwarding frames (still adding 1). Similarly, once a switch recieves a message that indicates a better path, it stops sending messages over that port.

-

Once the system stabilizes, the only the root will be sending configuration messages, and all other switches will be forwarding.

-
// Psuedo code for the distributed spanning tree algorithm
-
-#define MAX_BRIDGES 1024
-
-struct bridge {
-    macaddr     root;       // ID of the root
-    int     distance;       // distance to the root
-    int     port;           // port to reach the root
-    int     TTL;            // time to live
-};
-
-struct message {
-    macaddr root;           // ID of the root (mac address)
-    int     distance;       // distance to the root
-    macaddr     sender;     // ID of the sender (mac address)
-};
-
-struct bridge bridges[MAX_BRIDGES];
-struct message state;
-
-void
-update(struct message* m, int port)
-{
-    if (m->root < state.root || (m->root == state.root && m->distance < state.distance))
-    {
-        state.root = m->root;
-        state.distance = m->distance + 1;
-        state.sender = m->sender;
-        state.port = port;
-    }
-    else if (m->root == state.root && m->distance == state.distance && m->sender < state.sender)
-    {
-        state.sender = m->sender;
-        state.port = port;
-    }
-}
-
-void
-broadcast(struct message* m)
-{
-    for (int i = 0; i < MAX_BRIDGES; i++)
-        if (i != state.port)
-            send(i, m);
-}
-
-

Implementation

-

To create a switch, all you need to do is buy a general-purpose processor and equip it with multiple network interfaces. However, switches that deliver high-end performance typically take advantage of additional hardware acceleration. These are referred to as hardware switches, although both approaches obviously include a combination of hardware and software.

-

The implementation of switches and routers have so much in common that a network administrator typically buys a single forwarding box and then configures it to be an L2 switch, an L3 router, or some combination of the two.

-

Software Switches

-

The performance of switches is limited by the need to pass packets through main memory. This can be a bottleneck, especially for short packets. The cost of processing each packet, such as parsing its header and making forwarding decisions, can also be a limiting factor. Non-trivial algorithms used by switches, like the spanning tree algorithm, are not directly part of the per-packet forwarding decision. Table lookups are often the most costly routine executed on a per-packet basis.

-

Bare-Metal Switches

-

The distinction between the control plane (background processing) and the data plane (per-packet processing) is important. Recent advances in domain-specific processors have made it possible to build high-performance switches that combine the performance of ASICs with the agility of software. These switches can be programmed to be L2 switches, L3 routers, or a hybrid.

-

Bare-metal switches using Network Processing Units (NPUs) have become popular. NPUs are optimized for processing packet headers and making forwarding decisions. They can process packets at rates measured in Terabits per second (Tbps). The exact programming of an NPU depends on the chip vendor.

-

Internally, an NPU utilizes fast SRAM-based memory, TCAM-based memory for matching bit patterns, and a multi-stage forwarding pipeline. The use of a multi-stage pipeline allows for efficient packet processing and high throughput.

-

The networking industry is entering a commoditized world, similar to the computing industry, where standardized components and open source software are available for building high-performance switches.

-

Software Defined Networks

-

SDN (Software Defined Networks) decouples the network control plane from the data plane, allowing routing algorithms to run on software and packet forwarding decisions to be made by bare-metal switches. The standard interface between the control and data planes, known as OpenFlow, enables interoperability between different implementations. This concept of disaggregation has led to the trend of using commodity servers and bare-metal switches in network infrastructure.

-

Another important aspect of disaggregation is that a logically centralized control plane can be used to control a distributed network data plane. This allows for scalability and availability, as the two planes can be configured and scaled independently. Cloud providers have embraced this approach, using SDN-based solutions within their datacenters and across their backbone networks.

-

The Network Operating System (NOS) is a key enabler for SDN's success. It provides high-level abstractions and a Network Map to simplify network control functionality. The NOS detects changes in the network and the control application implements desired behavior on an abstract graph. By centralizing this logic, a globally optimized solution can be achieved.

-
- -
- - \ No newline at end of file diff --git a/site/networks/2-direct-links/wireless.html b/site/networks/2-direct-links/wireless.html deleted file mode 100644 index 878d617..0000000 --- a/site/networks/2-direct-links/wireless.html +++ /dev/null @@ -1,230 +0,0 @@ - - - - - - Wireless - - - - - -
- -

Wireless

-
- Last modified: 2024-02-07 - -
-
-

Wireless

-

Wireless media is infinite, and therefore cannot carrier sense. Futhermore, nodes cannot hear the network while sending

-

Hidden Terminal Problem

-

Two nodes are out of range of each other, but in range of a third node. The third node can hear both nodes, but the two nodes cannot hear each other. This can cause collisions at the node in the middle.

-

Exposed Terminal Problem

-

Two nodes are in range of each other, but are sending to different nodes out of each other's range. The two nodes can hear each other, but are not in each other's way. The two nodes should be able to send at the same time, but this might be prevented by some protocols.

-

Multiple Access with Collision Avoidance (MACA)

-

Uses short handshake instead of CSMA. Collisions are still possible, but less likely.

-
    -
  1. Request to Send (RTS): Sender sends a request to send to the receiver.
  2. -
  3. Clear to Send (CTS): Receiver sends a clear to send to the sender, including the frame size.
  4. -
  5. Data: Sender sends the frame while nodes that heard the CTS stay silent.
  6. -
-

802.11 (WiFi)

-

Clients connect to the network through an access point (AP).

-

Physical Layer

- - - -

Centralized MAC: Cellular

-

Usually on a very limited spectrum because there are more regulations on non-ISM bands. The base station coordinates the transmissions of the mobiles, and is able to provide more strict control over the network to provide things like QoS and robustness.

-

GSM MAC uses FDMA/TDMA, and BEB for random access. One channel for coordination, and other channels for traffic. There is also a dedicated channel for QoS.

-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/ARP-packet.png b/site/networks/3-network/ARP-packet.png deleted file mode 100644 index ebe4911..0000000 Binary files a/site/networks/3-network/ARP-packet.png and /dev/null differ diff --git a/site/networks/3-network/ARP.html b/site/networks/3-network/ARP.html deleted file mode 100644 index e0e4f07..0000000 --- a/site/networks/3-network/ARP.html +++ /dev/null @@ -1,212 +0,0 @@ - - - - - - Arp - - - - - -
- -

Arp

-
- Last modified: 2024-02-25 - -
-
-

Address Resolution Protocol (ARP)

-

ARP is used to map IP addresses to MAC addresses. It sits directly on top of the link layer, without using any servers or routers. ARP is just one example of a discovery protocol, which are used to find devices on a network. Other examples include Zeroconf and Bonjour. Discovery protocols more often than not use broadcast messages to find devices on the network.

-

ARP Table

-

ARP maintains a table of IP addresses and their corresponding MAC addresses. This is typically stored in a given host's memory. You can see your ARP table with the arp command.

-
arp -a
-
-

ARP Request

-

When a device wants to send a packet to another device on the same network, it first checks its ARP table to see if it knows the MAC address of the destination IP address. If it doesn't, it sends an ARP request to the broadcast MAC address FF:FF:FF:FF:FF:FF asking for the MAC address of the destination IP address.

-
ARP Request Packet:
-
-  ....Link layer.....
-+----------+----------+----------+----------+----------+
-| SRC MAC  | DST MAC  | SRC IP   | DST IP   | Payload  |
-+----------+----------+----------+----------+----------+
- from NIC    From ARP   From DHCP
-
-

ARP Reply

-

Although the ARP request is broadcast, only nodes that already have an entry, or the destination IP address itself, will act on the request (ie refresh tables/learn MAC address). Other nodes will ignore the request. The destination IP address will then send an ARP reply to the source IP address with its MAC address.

-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/BGP.html b/site/networks/3-network/BGP.html deleted file mode 100644 index 0d14aa7..0000000 --- a/site/networks/3-network/BGP.html +++ /dev/null @@ -1,230 +0,0 @@ - - - - - - Bgp - - - - - -
- -

Bgp

-
- Last modified: 2024-03-01 - -
-
-

Border Gateway Protocol (BGP)

-

You can view the internet as a graph of interconnected Autonomous Systems (ASes). Each AS can act on its own, and is responsible for routing traffic within itself. The routers within an AS are typically connected to eachother through high-speed links, and are connected to other ASes through border routers.

-

ASes are able to assume the role of a customer, provider, or peer to other ASes.

- -

The protocol used to route traffic between ASes is called BGP (Border Gateway Protocol). Unlike OSPF and RIP, BGP doesn't find the "best" path to a destination, but rather finds the "best" path according to the policies of the ASes involved. In fact, BGP only advertises based on reachability, and not on cost, since cost is not a well-defined concept across ASes.

-

BGP Overview

-

Each AS has one or more border routers that handle ingress and egress traffic to and from other ASes. ASes also need to have at least one BGP speaker that is responsible for exchanging routing information with other ASes. Border routers and BGP speakers are often the same device, but they don't have to be.

-

BGP is a path vector protocol, which means that it advertises the entire path to a destination (as a sequence of ASes), not just the next hop. This prevents routing loops and allows for policy-based routing through the entire path a packet takes.

-

BGP speakers aren't obligated to advertise any given path, but they are obligated to advertise the best path according to their policies. They can also cancel advertisements using a withdraw route message.

-

BGP works over TCP, and uses a keep-alive mechanism. If a BGP speaker doesn't receive a keep-alive message from a neighbor within a certain time frame, it will assume that the neighbor is down and will stop advertising the routes it learned from that neighbor.

-

Relationships

- -

BGP Algorithm

-

Let $AS_i \to AS_j$ be the best path from $AS_i$ to $AS_j$. Then, the BGP routing protocol is defined as follows:

- -

Prefer Customer > Peer > Provider, and select the ONE path with the minimum number of AS hops.

-

Policy-Based Routing Summary

- -
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/DHCP.html b/site/networks/3-network/DHCP.html deleted file mode 100644 index 283ac0c..0000000 --- a/site/networks/3-network/DHCP.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Dhcp - - - - - -
- -

Dhcp

-
- Last modified: 2024-02-25 - -
-
-

Dynamic Host Configuration Protocol (DHCP)

-

DHCP is the protocol used to assign IP addresses within a given network. Unlike MAC addresses, IPs are not hard-coded or unique to a device, and can be reassigned to different devices over time. This gives the network much needed flexibility, but also requires a way to manage the assignment of IPs. This is where DHCP comes in.

-

DHCP sits on top of the UDP protocol, and uses port 67 for the server, and port 68 for the client.

-

DHCP Server

-

The DHCP Server is a device that assigns IP addresses to devices on the network. It is typically a router or a server, and is responsible for managing a pool of IP addresses. When a device connects to the network, it broadcasts a DHCP Discover message to the network (sending to addr of all 1s). The DHCP server listens for these messages, and responds with a DHCP Offer message. This message contains an IP address that the server is willing to assign to the device. The device then sends a DHCP Request message to the server, and the server responds with a DHCP Ack message, confirming the assignment of the IP address.

-

The protocol also supports replicated DHCP servers, which can be used to provide fault tolerance. If a DHCP server fails, the client can simply request an IP from another server.

-

DHCP Client

-

DHCP clients are pretty ubiquitous, and are built into most devices that connect to a network. To renew an IP address, the client sends a DHCP Request message to the server, and the server responds with a DHCP Ack message, confirming the renewal.

-

DHCP Relay

-

Not all networks have a running DHCP server. In this case, a DHCP Relay can be used to forward DHCP messages to a server on another network. The relay listens for DHCP Discover messages, and forwards them to the server. The server then responds with a DHCP Offer message, and the relay forwards it to the client.

-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/ICMP.html b/site/networks/3-network/ICMP.html deleted file mode 100644 index bfe9675..0000000 --- a/site/networks/3-network/ICMP.html +++ /dev/null @@ -1,202 +0,0 @@ - - - - - - Icmp - - - - - -
- -

Icmp

-
- Last modified: 2024-02-25 - -
-
-

Internet Control Message Protocol (ICMP)

-

ICMP is a network layer protocol used to report errors and exchange control messages. It is also the basis for tools like ping and traceroute.

-

One particularly useful ICMP message is the ICMP Redirect, which is used to inform a host that a better route exists for a given destination. When a host receives an ICMP Redirect, it updates its routing table to use the new route.

-

Traceroute

-

traceroute works by continuously sending packets with increasing TTLs, and listening for ICMP Time Exceeded messages. When a packet reaches a router, the router decrements the TTL, and if it reaches 0, it sends an ICMP Time Exceeded message back to the source. This message contains the IP address of the router, and the time it took for the packet to reach it.

-

After each packet, traceroute prints the IP address of the router and the time it took for the packet to reach it. This is repeated until the destination is reached.

-

Ping

-

ping works by sending ICMP Echo Request messages to the destination, and listening for ICMP Echo Reply messages. When the destination receives an Echo Request, it responds with an Echo Reply. This is repeated until the user stops the command.

-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/global-internet.html b/site/networks/3-network/global-internet.html deleted file mode 100644 index 5706337..0000000 --- a/site/networks/3-network/global-internet.html +++ /dev/null @@ -1,202 +0,0 @@ - - - - - - Global Internet - - - - - -
- -

Global Internet

-
- Last modified: 2024-02-25 - -
-
-

The Global Internet

-

It isn't possible to scale to the billions of devices with routing protocols like RIP and OSPF. Both of these would require routers to maintain a complete list of all the networks in the internet, which isn't feasible.

-

The modern internet is mainly composed of end-user sites and service providers. The end-user sites are typically collections of devices that are connected to the internet through either a single IP address (NAT in home networks), or switched ethernet in enterprise LAN settings. The service providers are the companies that provide the infrastructure for the internet, and are responsible for routing traffic between end-user sites. Service providers have many high performance routers in metro areas, connected to eachother and other service providers through high-speed links.

-

Routing Areas

-

As described with OSPF, a network can be divided into areas to reduce the size of the routing table. This is also done in the global internet, but on a much larger scale. The internet is divided into routing areas called Autonomous Systems (AS). An AS is a collection of routers and networks that are under the control of a single organization. Each AS has its own routing protocol, and is responsible for routing traffic within the AS. The routers within an AS are typically connected to eachother through high-speed links, and are connected to other ASes through border routers.

-

Inter-Domain Routing

-

The internet is a collection of interconnected ASes, which are independently managed and operated organizations. The routers in an AS are responsible for routing traffic within the AS, and the border routers are responsible for routing traffic between ASes. The protocol used to route traffic between ASes is called BGP (Border Gateway Protocol).

-

See BGP.md

-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/internetworking.html b/site/networks/3-network/internetworking.html deleted file mode 100644 index 88bbcea..0000000 --- a/site/networks/3-network/internetworking.html +++ /dev/null @@ -1,300 +0,0 @@ - - - - - - Internetworking - - - - - -
- -

Internetworking

-
- Last modified: 2024-05-15 - -
-
-

Internetworking

-

How netorks may differ

- -

Internetworking hides the difference with a common prorocol (IP) and a common addressing scheme (IP addresses).

-

Connecting Datagram and Virtual Circuit Networks

-

Need to map a destination address to a VC and visa versa. In order to accomplish this, might have to set up a VC between two routers, and then use datagrams to send packets between the routers.

-
+--------+   802.11   +--------+ <---- MPLS ----> +--------+  Ethernet  +--------+
-|  host  |------------| Router |                  | Router |------------|  host  |
-+--------+            +--------+ <--------------> +--------+            +--------+
- source                            VC network                           destination
-
-

IP Addressing

-

IP is the lowest common denominator of the internet. It allows networks that support entirely different services to communicate. Asks very little of the underlying network, and provides very little in return.

- -

IPv4

-

-<--------------------------------------- 32 bits --------------------------------------->
-
-+---------------------+---------------------+---------------------+---------------------+
-| Version |    IHL    |   Dif. Services     |         Total Length (bytes)              |
-+---------------------+---------------------+---------------------+---------------------+
-|              Identification               |    | DF | MF |  Fragment Offset (13 bits) |
-+---------------------+---------------------+---------------------+---------------------+
-|  Time to Live (TTL) | Protocol (TCP, UDP) |            Header Checksum                |
-+---------------------+---------------------+---------------------+---------------------+
-|                               Source IP Address (32 bits)                             |
-+---------------------+---------------------+---------------------+---------------------+
-|                            Destination IP Address (32 bits)                           |
-+---------------------+---------------------+---------------------+---------------------+
-|                                  Options (0 or more words)                            |
-|                                                                                       |
-|                                        ....                                           |
-+---------------------+---------------------+---------------------+---------------------+
-|                                       Payload                                         |
-|                                                                                       |
-|                                        ....                                           |
-+---------------------+---------------------+---------------------+---------------------+
-
-
-

IP Prefixes

- -

Example:

-

128.13.0.0/16

-
        Network                Host
-+----------+----------+----------+----------+
-| 10000000 | 00001101 |   ....   |   ....   |
-+----------+----------+----------+----------+
-
-

IP Datagram Forwarding

-

When a host wants to send a packet to another host, it first checks if the destination IP address is on the same network (matching subnet). If it is, it can send the packet directly over the link layer (using ARP to get resolve IP address to MAC address). Otherwise, it sends the packet to the default gateway, or router.

-

The router then forwards the packet to its next hop based on the destination IP and the router's routing table.

-
if (NetworkNum of destination = NetworkNum of one of my interfaces):
-    deliver packet to destination over that interface (using ARP)
-else:
-    if (NetworkNum of destination is in my forwarding table):
-        deliver packet to NextHop router
-    else:
-        deliver packet to default router
-
-

Longest Prefix Match

- -
for each entry in the forwarding table:
-    if (NetworkNum of destination & Mask) == (NetworkNum of entry & Mask):
-        deliver packet to NextHop router
-
- -
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/motivation.html b/site/networks/3-network/motivation.html deleted file mode 100644 index 6d176ec..0000000 --- a/site/networks/3-network/motivation.html +++ /dev/null @@ -1,233 +0,0 @@ - - - - - - Motivation - - - - - -
- -

Motivation

-
- Last modified: 2024-02-10 - -
-
-

Motivation behind the Network Layer

-

We can already build networks using the link layer using packet forwarding, but this doesn't scale well. The network layer allows you to connect multiple networks together, allowing for transparent communication between different link layer technologies.

- - -

Network Layer Approach

- -

Routing vs. Forwarding

- -

Topics

- -
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/networking-services.html b/site/networks/3-network/networking-services.html deleted file mode 100644 index 275c825..0000000 --- a/site/networks/3-network/networking-services.html +++ /dev/null @@ -1,245 +0,0 @@ - - - - - - Networking Services - - - - - -
- -

Networking Services

-
- Last modified: 2024-02-23 - -
-
-

Networking Services

-

Store-and-Forward Packet Switching

- -

Datagrams vs. Virtual Circuits

-

Datagrams are a connectionless service that sends packets from one host to another. Like postal letters. Each packet is independent and can take a different route to the destination. This is the most common service on the internet. Each packet contains destination address. Routers use this to forward packets, maybe on different paths for each subsequent message. Each router has a forwarding table keyed by destination address. The table contains the next hop for each destination, but the entries change over time as the network topology changes.

-

Virtual Circuits are a connection-oriented service that sets up a path between the source and destination before sending packets. Like a phone call. Each packet follows the same path and is identified by a connection ID. This is less common on the internet, but is used in technologies like ATM and Frame Relay.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IssueDatagramVirtual Circuit
Setup phaseNot neededRequired
Router statePer destinationPer connection
AddressesPacket carries full destination addressShort connection ID to label
RoutingPer packetPer circuit
FailuresEasy to maskHard to mask
QoSHard to provideEasier to provide
-
- -
- - \ No newline at end of file diff --git a/site/networks/3-network/routing.html b/site/networks/3-network/routing.html deleted file mode 100644 index b13b867..0000000 --- a/site/networks/3-network/routing.html +++ /dev/null @@ -1,366 +0,0 @@ - - - - - - Routing - - - - - -
- -

Routing

-
- Last modified: 2024-03-07 - -
-
-

-

Routers are devices within a network that handle forwarding packets out of the network. They use a routing table to determine the best path for a packet to reach its destination. The routing table contains information about known networks, including the next hop router to reach them.

-

Forwarding Table vs. Routing Table

-

1. Scope:

- -

2. Information:

- -

3. Dynamic Updates:

- -

4. Size:

- -

5. Function:

- -

Analogy: Think of the routing table as a city map showing different neighborhoods and highways. The forwarding table is like a detailed building floor plan within a specific neighborhood, guiding you to specific rooms.

-

The Network as a Graph

-

Model the internet as a graph of routers (nodes) and links (edges). To simplify, we can model the graph as an undirected weighted graph, where the weight of an edge represents the cost of sending a packet over that link. In practice, the internet is more accurately modeled as a directed graph, but the undirected model is simpler.

-

The basic problem of routing is to find the shortest path between nodes in the graph. One could use Dijkstra's algorithm to find paths, and then save them to disk/memory as the routing table. However:

- -

Therefore, routing protocols are typcially distributed and adaptive, running idependently on each router.

-

Distance Vector Routing (RIP)

-

The Routing Information Protocol (RIP), is a fundamental algorithm for navigating paths of the internet. It operates by exchanging routing information between neighboring routers, building a distributed understanding of the network topology. It is in the same class of algorithms as Bellman-Ford.

-

Core Concepts:

- -

Strengths:

- -

Weaknesses:

- -

Implementation:

-

Here is a simple implementation of Distance Vector Routing in C:

-
#define MAX_ROUTES      128     /* maximum size of routing table */
-#define MAX_TTL         120     /* time (in seconds) until route expires */
-
-typedef struct {
-    NodeAddr  Destination;    /* address of destination */
-    NodeAddr  NextHop;        /* address of next hop */
-    int        Cost;          /* distance metric */
-    u_short   TTL;            /* time to live */
-} Route;
-
-int      numRoutes = 0;
-Route    routingTable[MAX_ROUTES];
-
-
-void mergeRoute (Route *new) {
-    int i;
-    for (i = 0; i < numRoutes; ++i) {
-        if (new->Destination == routingTable[i].Destination) {
-            if (new->Cost + 1 < routingTable[i].Cost)
-                break; /* found a better route */
-            else if (new->NextHop == routingTable[i].NextHop)
-                break; /* next hop may have changed */
-            else
-                return; 
-        }
-    }
-    if (i == numRoutes) {
-        if (numRoutes < MAXROUTES)
-            numRoutes++;
-        else
-            return; /* can`t fit this route in table so give up */
-    }
-    routingTable[i] = *new;
-    routingTable[i].TTL = MAX_TTL; /* reset TTL */
-    routingTable[i].Cost++;
-}
-
-void updateRoutingTable (Route *newRoute, int numNewRoutes) {
-    for (int i=0; i < numNewRoutes; ++i)
-        mergeRoute(&newRoute[i]);
-}
-
-
-

However, actual RIP operates on UDP port 520 and might look something like this:

-
RIP Message:
-    Version
-    Command (request, response, update)
-    Number of entries 
-    RIP Entry (variable size)
-    Address Family Identifier
-    Route Tag
-    IP Address
-    Next Hop IP Address
-    Metric
-
-

RIP is a fairly limited and simple implementation of distance vector routing. It assigns a cost of 1 to each hop (effectively calculating the fewest hop path), and it only allows distances of up to 16 hops. Not used in most modern networks.

- -

Each node maintains state locally, and then creates a Link State Packet (LSP) containing the following information:

- -

These packets are flooded throughout the network, meaning that each node sends the packet to all of its neighbors.

-

Reliable Flooding

-

The sequence number and TTL of an LSP are used to ensure reliability and prevent loops. Adjacent nodes use ACKs and retransmissions. For non-adajcent nodes, to ensure that the most recent LSP is used, the LSP with the higher sequence number is always picked (if there is an existing LSP stored already). Once an LSP is received, the node stores it and forwards it to all of its neighbors, except the one it received it from.

-

New LSPs are generated both at regular intervals, and when an adjacent node goes down. Nodes send their neighbors "hello" packets to demonstrate that they are still alive. If a node doesn't receive a "hello" packet from a neighbor for a certain period, it assumes the neighbor is down and generates a new LSP.

-

In practice, the "regular interval" LSPs should be generated very infrequently to reduce overhead on the network. This way, the network can still converge quickly in the event of a failure, but not waste resources when nothing is changing.

-

When a node receives an LSP with TTL of 0, it deletes the record from its own database, and then floods the network with this packet so that all the other nodes can delete the LSP from their databases.

-

Route Calculation

-

Based on Dijkstra's algorithm, but in practice uses Forward Search. Each node calculates the shortest path to all other nodes in the network, and then stores this information locally.

-
M = {s}
-for each n in N - {s}:
-    C(n) = l(s,n)
-while (N != M):
-    M = M + {w} such that C(w) is the minimum for all w in (N-M)
-    for each n in (N-M):
-        C(n) = MIN(C(n), C(w)+l(w,n))
-
- -

Uses a Confirmed and a Tentative list, both of which contain entries (Destination, Cost, NextHop). The algorithm proceeds as follows:

-
    -
  1. Initialize the Confirmed list with an entry for myself; this entry has a cost of 0.
  2. -
  3. For the node just added to the Confirmed list in the previous step, call it node Next and select its LSP.
  4. -
  5. For each neighbor (Neighbor) of Next, calculate the cost (Cost) to reach this Neighbor as the sum of the cost from myself to Next and from Next to Neighbor. If Neighbor is currently on neither the Confirmed nor the Tentative list, then add (Neighbor, Cost, NextHop) to the Tentative list, where NextHop is the direction I go to reach Next.
  6. -
  7. If Neighbor is currently on the Tentative list, and the Cost is less than the currently listed cost for Neighbor, then replace the current entry with (Neighbor, Cost, NextHop), where NextHop is the direction I go to reach Next.
  8. -
  9. If the Tentative list is empty, stop. Otherwise, pick the entry from the Tentative list with the lowest cost, move it to the Confirmed list, and retu
  10. -
-

Open Shortest Path First (OSPF)

-

OSPF is a more modern and feature-rich link state routing protocol. It adds the following features to basic link state routing:

- -

Of the five OSPF message types, type 1 is the "hello" message, which a router sends to its peers to notify them that it is still alive and connected as described above. The remaining types are used to request, send, and acknowledge the receipt of link-state messages.

-

OSPF sends Link State Advertisements (LSAs) instead of LSPs. LSAs contain information about links, including the Link ID, Link Data, and metric. The Link ID is typically the router ID of the router at the far end of the link. The metric represents the cost of the link. TOS (type of service) information allows OSPF to choose different routes based on the TOS field of IP packets.

-

Metrics

-

The ARPANET tested different approaches to link-cost calculation. The original metric measured queued packets on each link, but it didn't consider bandwidth or latency. A later version used delay as a measure of load, taking into account link bandwidth and latency. However, it suffered from instability under heavy load and had a large range of link values. A third approach compressed the metric range, accounted for link type, and smoothed the variation over time.

-

In real-world network deployments, metrics change rarely, if at all, and only under the control of a network administrator. Static metrics are the norm, with a common approach being to use a constant multiplied by (1/link_bandwidth).

-
- -
- - \ No newline at end of file diff --git a/site/networks/4-transport/ACK-clocking.html b/site/networks/4-transport/ACK-clocking.html deleted file mode 100644 index a2d906b..0000000 --- a/site/networks/4-transport/ACK-clocking.html +++ /dev/null @@ -1,207 +0,0 @@ - - - - - - Ack Clocking - - - - - -
- -

Ack Clocking

-
- Last modified: 2024-02-24 - -
-
-

ACK Clocking

-

Sliding Window Ack CLock

-

Each in-order ACK advances the sliding window and lets a new segment enter the network. This prevents queuing buildups at slow links, and thus congestion. Segments are buffered and spread out on slow links.

- -

TCP uses sliding window, controlling the number of segments in the network. It ensures that only small bursts of segments are sent to keep traffic smooth.

-

Problem at the Receiver

-

Sliding window has pipelining to keep network busy, but what if the receiver is overloaded?

-

Consider receiver with $w$ buffers. Application should recv to accept packets, but if it didn't, then

-
- -
- - \ No newline at end of file diff --git a/site/networks/4-transport/AIMD.png b/site/networks/4-transport/AIMD.png deleted file mode 100644 index 2c8fb99..0000000 Binary files a/site/networks/4-transport/AIMD.png and /dev/null differ diff --git a/site/networks/4-transport/TCP-sawtooth.png b/site/networks/4-transport/TCP-sawtooth.png deleted file mode 100644 index 08374cd..0000000 Binary files a/site/networks/4-transport/TCP-sawtooth.png and /dev/null differ diff --git a/site/networks/4-transport/TCP.html b/site/networks/4-transport/TCP.html deleted file mode 100644 index ce98544..0000000 --- a/site/networks/4-transport/TCP.html +++ /dev/null @@ -1,343 +0,0 @@ - - - - - - Tcp - - - - - -
- -

Tcp

-
- Last modified: 2024-03-04 - -
-
-

Transmission Control Protocol (TCP)

-

Connection Establishment

-

Both sender and receiver must be ready to transfer data, and they need to agree on parameters like max segment size.

-

Three-Way Handshake

-

Opens up connection between client and server. Each side probes the other with a fresh Initial Sequence Number (ISN). Sends on a SYNchronize segment, and echos on ACKnowledge segment. This gives us robustness, but requires extra overhead.

- -

Connection Release

-

TCP requires a two-way close. Client and server both finish sending all their data and send a FIN segment. Each FIN closes one direction of the connection.

- -

TIME_WAIT State

-

Wait a long time afer sending all segments before actually closing (2 x max segment lifetime). This is because the final ACK may be lost, and we need to make sure the other side has received it. Otherwise it might interfere with a new connection.

-

Timeout Problem

-

If you set it too small, you will retransmit too often. If you set it too large, you will wait too long to detect a failure. Setting the timeout of TCP is a difficult problem because RTT varies widely (due to queueing, routing, etc).

-

TCP uses adaptive timeout. It smooths the estimates and variance of the RTT using a moving average.

-

$$SRTT_{N + 1} = (1 - \alpha) \cdot SRTT_N + \alpha \cdot RTT_{N + 1}$$

-

$$Svar_{N + 1} = (1 - \beta) \cdot Svar_N + \beta \cdot |RTT_{N + 1} - SRTT_{N + 1}|$$

-

Set the timeout to be a multiple of the smoothed RTT, plus a margin for error. Adaptive timeout works well in practice, but it's not perfect. It can be thrown off by a sudden change in RTT, and it can be exploited by an attacker to cause a DoS attack.

-

Network Congestion

-

Think about a "traffic jam" in the network. The network is temporarily overloaded and can't handle all the traffic. This can lead to packet loss, increased latency, and decreased throughput. Want to push as close to being congested as possible without actually being congested, which is a high level function of TCP.

-

Switches and routers have internal buffers to store packets, which essentially act like a queue for each port. When working normally, queues are able to absorb bursts of traffic. However, if the input rate is persistently higher than the output rate, the queue will eventually fill up and packets will be dropped.

-

Congestion is a function of traffic patterns, and it can even occur even when the network isn't at full capacity. We would like it if the throughput increased linearly, or even logarithmically towards the capacity of the network as you increase the load on a system. In practice, there is a sharp dropoff in throughput past a certain point of load where congestion collapse occurs.

-

Connection collapse is when packets are dropped/time out, and so the sender retransmits the packets, which only makes the problem worse. This is a positive feedback loop that can lead to a complete breakdown of the network.

-

TCP Tahoe/Reno

-

Has the following extensions/features:

- -

Bandwidth Allocation

-

Fairness

-

Kind of a weird problem. Just think about it like scheduling threads and processes. You want to give each process a fair share of the CPU, but simply allocating bandwidth equally to each thread in the process is not fair. You want to allocate bandwidth to each connection fairly, but you also want to allocate bandwidth to each user fairly, and you also want to allocate bandwidth to each application fairly.

-

Equal per Flow Fairness

-

The bottleneck for a flow of traffic is the link that limits its bandwidth. TCP tries to allocate bandwidth fairly to each flow by sharing the bottleneck bandwidth equally among the flows that traverse it.

-

Max-Min Fairness

-

Intuitively, flows bottlenecked on a link get an equal share of that link. A max-min fair allocation is one that maximizes the minimum bandwidth allocated to any flow. To find it given a network, you can imagine "pouring water" into the network and seeing how much each flow gets. You can then adjust the flow rates to maximize the minimum flow rate.

-

When flows start and stop, need to rerun the algorithm to find new allocation.

-

TCP needs to allocate bandwidth of a network fairly and efficiently (which are conflicting goals). The network witnesses congestion and provides direct feedback to the transport layer. Then, the transport layer can decrease its sending rate to avoid congestion.

-

This is a hard problem. The number of senders is almost never constant, and each customer's load is constantly changing. Also, senders often lack capacity in certain parts of the network, and no one entity has a complete view of the network.

-

This is solved by having the senders continuously probe the network and adapt their sending rate based on feedback from the network.

- -

TCP uses closed-loop, host-driven, window-based allocation. The network sends packet-dropped messages as feedback. Note that there is nothing stopping a sender from implementing a more "aggressive" or "greedy" algorithm, but it would be against the spirit of the protocol. There are also different implementations of TCP that have different congestion control algorithms.

- - - - - - - - - - - - - - - - - - - - - - - - - -
SignalProtocolPros/Cons
Packet LossTCP NewReno/Cubic (linux)Simple, but there is some latency
Packet delayTCP BBR (YouTube)Early notifacation, but need to infer congestion
Explicit congestion notification via routerTCPs with explicity congestion notificationFast, but requires router support
-

Additive Increase, Multiplicative Decrease (AIMD)

-

A control law hosts use to reach a good allocation. Hosts additively increase rate while network isn't congested, and multiplicitively decrease rate when congestion is detected.

-

Let $x$ be the allocated bandwidth to H1, and $y$ be the allocated bandwidth to H2. Assumming a capacity $1$, a fair allocation would be $x = y$, but an efficient allocation would be $x + y = 1$.

-

On the plot of x and y, the algorithn is as follows:

- -

This guarantees that the allocation converges to the intersection of the "fair" line and the "efficient" line. Also creates a "sawtooth" pattern of sending rate over time.

-

Slow Start with TCP (Additive Increase)

-

Sender uses congestion window (cwnd) to set its rate (cwnd/RTT), and packet loss as a signal.

-

You want to quickly converge to the ideal window size, but you also don't want to cause congestion. Solution is to increase exponentially until you hit a packet loss, then set the window size to half of the current window size. Then, switch to additive increase.

- -

TCP Tahoe

- -

Need to go back to cwnd = 1 on packet loss because we lost the ack clock. This is conservative but not very efficient.

-

Fast Recovery

-

Infering Packet Loss

-

Reciever sends duplicate ACKs when it receives out-of-order packets. The sender can infer that the packet was lost and retransmit it. If the reciever is stuck on an out of order packet, the subsequently sent packets will still be buffered, but the duplicate ack indicates where to retransmit on the client side.

-

Need to be able to infer loss of packets before timeout actually occurs. TCP uses a cumulative ACK, which carries the highest in-order sequence number received. If you receive three duplicate ACKs, you can infer that the packet was lost and retransmit it.

-

Sender decreases cwnd by half on packet loss, but it can also retransmit the lost packet immediately. This is called fast retransmit.

-

Inferring Non-Loss from ACKs

-

Each duplicate ACK indicates that a packet was received, but not in order. This means that the network is still capable of transmitting packets, and so the sender can increase its sending rate. This is called fast recovery.

-

If multiple packets are lost in a row, this will not work and the sender will go back to slow start cwnd = 1.

-

TCP Reno, NewReno, SACK

- -

Explicit Congestion Notification (ECN)

- -

Random Early Detection (RED)

-

Instead of marking pakets, just randomly drop packets to throttle senders. The probablity of dropping a packet increases as the queue fills up.

-
- -
- - \ No newline at end of file diff --git a/site/networks/4-transport/UDP.html b/site/networks/4-transport/UDP.html deleted file mode 100644 index 25d3c0d..0000000 --- a/site/networks/4-transport/UDP.html +++ /dev/null @@ -1,195 +0,0 @@ - - - - - - Udp - - - - - -
- -

Udp

-
- Last modified: 2024-02-23 - -
-
- -
- -
- - \ No newline at end of file diff --git a/site/networks/4-transport/flow-control.html b/site/networks/4-transport/flow-control.html deleted file mode 100644 index 777caab..0000000 --- a/site/networks/4-transport/flow-control.html +++ /dev/null @@ -1,218 +0,0 @@ - - - - - - Flow Control - - - - - -
- -

Flow Control

-
- Last modified: 2024-02-24 - -
-
-

Flow Control

-

Recall stop and wait ARQ. This only allows a single packet to be transmitted at a time. This is inefficient, as the sender is often waiting for the receiver to acknowledge the packet.

-

Next recall sliding window ARQ. This allows multiple packets to be transmitted at a time. The sender can keep sending packets until the window is full. The receiver can acknowledge packets in any order, and the sender can keep sending packets until the window is full. With a window size of $w$, you can send $w$ packets before waiting for an acknowledgement (ie $w$ packets per RTT).

-

Sliding Window Example

-

Assuming $10$ kb packets, $R = 1$ Mbps, $d = 50$ ms, what window size $w$ do you need to use the network capacity?

-

First, knowing RTT = $2d = 100$ ms, we can calculate the maximum throughput using the bandwidth-delay product:

-

BD $= R \times d = 1 \times 10^6 \times 0.05 = 50,000$ bits

-

so $100000$ bits per RTT can be stored in the network. Therefore, the window size $w$ is:

-

$w = \frac{100000}{10000} = 10$

-

Sliding Window Sender

-

Sender buffers up to $w$ segments until they are acked. LFS (Last Frame Sent) is the sequence number of the last frame sent. LAR (Last Ack Received) is the sequence number of the last frame acked. The sender can send up to $w$ frames, and the receiver can receive up to $w$ frames, where

-

$w > LFS - LAR$

-

Sliding Window Receiver

-

Go-Back-N ARQ

-

In Go-Back-N ARQ, the sender can send multiple packets before waiting for an acknowledgement. The receiver can acknowledge packets in any order, and the sender can keep sending packets until the window is full. If a packet is lost, the sender will have to resend all packets from the lost packet onwards.

-

The reciever keeps only a single packet buffer for the next segment. (ie keeps LAS = LAST ACK SENT). On receive, if seq number is LAS + 1, then accept, update LAS, and send ACK. Else, discard.

-

Uses a single timer to detect lost packets. On timeout, resends buffered packets starting from LAS + 1.

-

Selective Repeat ARQ

-

In Selective Repeat ARQ, the sender can send multiple packets before waiting for an acknowledgement. The receiver can acknowledge packets in any order, and the sender can keep sending packets until the window is full. If a packet is lost, the sender will only have to resend the lost packet.

-

Receiver buffers $w$ segments, and keeps LAS = LAST ACK SENT. On reeive, buffer segments $[LAS + 1, LAS + w]$. If seq number is LAS + 1, then accept, update LAS, and send ACK.

-

If it receives something out of order, it will buffer it and send an ACK for the last in-order segment. If it receives a duplicate, it will send an ACK for the last in-order segment.

-

Uses a timer for each segment. On timeout, resends the segment.

-

Sequence Numbers

-

For stop and wait, only need 0/1. For selective repeat, need $w$ numbers for packets, and $w$ numbers for acks of earlier packets. ($2w$ in total). For go-back-n, need $w$ numbers for packets, and 1 number for the ack of the last packet. ($w + 1$ in total).

-
- -
- - \ No newline at end of file diff --git a/site/networks/4-transport/transport-overview.html b/site/networks/4-transport/transport-overview.html deleted file mode 100644 index 63fc5ad..0000000 --- a/site/networks/4-transport/transport-overview.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Transport Overview - - - - - -
- -

Transport Overview

-
- Last modified: 2024-02-24 - -
-
-

Transport Layer Overview

-

The transport layer provides end-to-end connectivity accross the network. Segments carry application data, which are carried within packets, which are carried within frames.

-

The two main services are messages (datagram, UDP) and bytestreams (streams, TCP). Messages are discrete units of data, while bytestreams are continuous streams of data.

-
- -
- - \ No newline at end of file diff --git a/site/networks/5-application/CDNs.html b/site/networks/5-application/CDNs.html deleted file mode 100644 index 71f1fdb..0000000 --- a/site/networks/5-application/CDNs.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - - - Cdns - - - - - -
- -

Cdns

-
- Last modified: 2024-03-05 - -
-
-

Content Delivery Networks (CDNs)

-

How do you place content near clients?

-

Browser Cache

-

Web browsers can cache static content such as stylesheets, scripts, images, and some AJAX requests. This caching can speed up page load time on subsequent page views.

-

When a web server returns a response, it sends a Cache-Control header. This header specifies the amount of time that a file should be cached. Once a file is cached in the browser, the browser will not request the file from the server until the file has expired.

-

Proxy Caches

-

A proxy server is a server that sits between a client application, such as a web browser, and a real server. It intercepts all requests to the real server to see if it can fulfill the requests itself. If not, it forwards the request to the real server.

-

Content Delivery Networks (CDNs)

-

A CDN is a system of distributed servers that deliver web content to a user based on the geographic locations of the user. Each region will have a number of edge locations, which are data centers that cache the content of the main server.

-

The DNS resolution of a CDN URL will direct the user to the nearest edge location, which will then deliver the cached content. This process reduces the load on the original server and speeds up the delivery of the content to the user.

-
- -
- - \ No newline at end of file diff --git a/site/networks/5-application/DNS.html b/site/networks/5-application/DNS.html deleted file mode 100644 index fa54faa..0000000 --- a/site/networks/5-application/DNS.html +++ /dev/null @@ -1,242 +0,0 @@ - - - - - - Dns - - - - - -
- -

Dns

-
- Last modified: 2024-03-05 - -
-
-

Domain Name System (DNS)

-

Namespaces

- -

Before DNS

-

Machines retrieved a file called hosts.txt from a centralized server to resolve names to addresses.

-

This approach was not scalable, as the hosts.txt file would have to be updated on every machine on the network.

-

DNS

-

DNS is a naming service that maps names to IP addresses and vice versa. It is a distributed database implemented in a hierarchy of name servers.

- -

DNS Zones

-

Zones are a way to divide the domain name space into manageable sections. A zone is a contiguous portion of the global DNS namespace.

-

Example

- -

Each zone has one or more name server that is authoritative for the zone. The name server is responsible for maintaining the zone's records. For instance, the name server for washingon.edu is responsible for maintaining the records for cs.washington.edu.

-

DNS Resolution

-

When a client wants to resolve an unknown name to an IP address, it sends a query to a DNS resolver. The resolver then sends a query to a root name server, which responds with the address of a TLD name server. The resolver then sends a query to the TLD name server, which responds with the address of the authoritative name server for the domain. The resolver then sends a query to the authoritative name server, which responds with the IP address of the domain.

-

Iterative vs. Recursive Resolution

- -

Recursive queries are nice because it offloads the client burden for address resolution. Also lets the server cache over a pool of clients.

-

Iterative queries are nice because it minimizes the complexity of the server, and is easier to build high load servers. Also gives the client more control over the resolution process.

-

DNS Caching

-

Latency needs to be fairly low for DNS resolution, so caching is important. Nameservers cache query results (including partial/iterative answers) for the duration of their TTL (time to live).

-

The caching is hierarchical, with the root servers at the top, and the client at the bottom. This means that the root servers are the most heavily loaded, and the client is the least heavily loaded.

-

Local Nameservers

-

Local nameservers are the first point of contact for a client's DNS resolution. They are usually provided by the ISP, and are responsible for caching DNS records for a short period of time.

-

They can also run on a host, or at an access point. Alternatively, you can use a public DNS resolver like Google's. Local name servers are typically configured via DHCP.

-

Root Name Servers

-

Root (.) is served by 13 root servers (labeled a.root-servers.net through m.root-servers.net). These servers are distributed around the world and are operated by 12 independent organizations.

-

There are more than 250 distributed server instances to increase the fault tolerance and availability of the root server system. Most servers are reached by IP anycast, which means that the same IP address is advertised from multiple locations.

-

DNS Protocol

-

DNS uses the User Datagram Protocol (UDP) on port 53 to serve requests. Uses ARQ for reliability. Messages are linked by a 16-bit ID field.

-

Servers can be replicated to handle load and reliability. Queries can return multiple records, and the client can choose which one to use.

-

Security is a major concern for DNS. DNSSEC is a suite of extensions that add security to the DNS protocol by signing DNS data, but it is not widely adopted.

-
- -
- - \ No newline at end of file diff --git a/site/networks/5-application/HTTP.html b/site/networks/5-application/HTTP.html deleted file mode 100644 index 7f0e445..0000000 --- a/site/networks/5-application/HTTP.html +++ /dev/null @@ -1,323 +0,0 @@ - - - - - - Http - - - - - -
- -

Http

-
- Last modified: 2024-03-05 - -
-
-

Hyper Text Transfer Protocol (HTTP)

-

You can think of a web page as a set of related HTTP transactions. Each transaction consists of a request and a response, which runs on TCP (typically on port 80).

-

Fetching a Web Page

-
protocol://host:port/path
-
-
    -
  1. Resolve server to IP address using DNS.
  2. -
  3. Establish a TCP connection to the server.
  4. -
  5. Send an HTTP request for the page.
  6. -
  7. Await HTTP response
  8. -
  9. Execute/fetch embedded resources and render the page.
  10. -
  11. Close the TCP connection.
  12. -
-

Static vs. Dynamic Web-pages

-

Static web pages are pre-built and served as-is to the client. Dynamic web pages are built on the server and served to the client to be run/interpreted (usually with JavaScript).

-

Methods

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MethodDescription
GETRead a Web page
HEADRead a Web page's header
POSTAppend to a Web page
PUTStore a Web page
DELETERemove the Web page
TRACEEcho the incoming request
CONNECTConnect through a proxy
OPTIONSQuery options for a page
-

Status Codes

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CodeDescriptionExample
1xxInformational100 Continue - server agrees to handle client's request
2xxSuccess201 Created - resource created - posted data
3xxRedirection304 - Not Modified - client should use cached version
4xxClient Error404 Not Found - resource not found
5xxServer Error503 Service Unavailable - server overloaded
-

Performance

-

Page Load Time (PLT)

-

The time it takes to download and display the entire content of a web page in the browser. Small increases in PLT can have a significant impact on user satisfaction. Depends on many factors including the page's content, the network RTT and bandwidth, and HTTP caching/TCP optimizations.

-

HTTP/1.0

-

Uses one TCP connection per request. This can be slow due to the overhead of setting up and tearing down connections. Also used sequential requests to all resources, requiring multiple TCP connections to the same server.

-

Decreasing PLT

- -

In practice, this might look like:

- -

HTTP Caching and Proxies

-

Users often revisit web pages, so caching is important. Some strategies include:

- -

Proxies can cache resources on behalf of clients, reducing the load on the server and speeding up the response time for clients. However, they can also introduce security and privacy concerns.

-

The general workflow for a proxy is:

-
    -
  1. Client sends a request to the proxy.
  2. -
  3. Proxy checks the expiry for the resource
  4. -
  5. If the resource is still valid, the proxy returns it to the client. Otherwise, the proxy fetches the resource from the server.
  6. -
  7. The proxy fetches the resource, and maybe some metadata through headers like Not-Modified and updates its cache.
  8. -
  9. The proxy returns the resource to the client.
  10. -
-

This places an intermediary between the pool of clients and the server, which can be useful for load balancing, security, and privacy. Has the added benefit of being able to improve physical locality of data to be closer to clients while in the cache. Benefits are limited by secure/dynamic content, and the "long tail" of resources that are rarely accessed.

-
- -
- - \ No newline at end of file diff --git a/site/networks/5-application/overview.html b/site/networks/5-application/overview.html deleted file mode 100644 index c1575b0..0000000 --- a/site/networks/5-application/overview.html +++ /dev/null @@ -1,196 +0,0 @@ - - - - - - Overview - - - - - -
- -

Overview

-
- Last modified: 2024-03-01 - -
-
-

Application Layer Overview

-

Applications built upon TCP have the benefit of being able to transfer arbitrary length data. Also provides reliability and flow control. However, some applications may not require these features and may be better suited to use UDP. Some applications even need to use UDP because they can't handle the overhead of TCP (like skype or online gaming).

-
- -
- - \ No newline at end of file diff --git a/site/networks/networkx_tutorial.py b/site/networks/networkx_tutorial.py deleted file mode 100644 index fb0ad6f..0000000 --- a/site/networks/networkx_tutorial.py +++ /dev/null @@ -1,17 +0,0 @@ -import networkx as nx -import matplotlib.pyplot as plt - -def create_graph_from_adjacency_list(adj_list) -> nx.Graph: - G = nx.Graph() - for node, neighbors in adj_list.items(): - for neighbor in neighbors: - G.add_edge(node, neighbor) - return G - -def create_graph_from_adjacency_matrix(adj_matrix) -> nx.Graph: - G = nx.Graph() - for i in range(len(adj_matrix)): - for j in range(len(adj_matrix[i])): - if adj_matrix[i][j] == 1: - G.add_edge(i, j) - return G \ No newline at end of file diff --git a/site/networks/reference.html b/site/networks/reference.html deleted file mode 100644 index ac6d245..0000000 --- a/site/networks/reference.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Reference - - - - - -
- -

Reference

-
- Last modified: 2024-03-25 - -
-
-

Textbook

- -
- -
- - \ No newline at end of file diff --git a/site/networks/sockets.html b/site/networks/sockets.html deleted file mode 100644 index 0ee836d..0000000 --- a/site/networks/sockets.html +++ /dev/null @@ -1,519 +0,0 @@ - - - - - - Sockets - - - - - -
- -

Sockets

-
- Last modified: 2024-01-11 - -
-
-

Socket Reference

-

Creating a Socket

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-
-/**
- * @brief Creates a socket.
- *
- * @param domain specifies the protocol family to use.
- *   ex: AF_INET, AF_INET6, AF_UNIX, etc.
- * @param type The type of the socket.
- *   ex: SOCK_STREAM, SOCK_DGRAM, SOCK_RAW, etc.
- * @param protocol The protocol of the socket.
- *   ex: (IPPROTO_TCP, IPPROTO_UDP, UNSPEC, etc.)
- *
- * @return The file descriptor of the socket.
-*/
-int socket(int domain, int type, int protocol);
-
-
import socket
-# TCP socket
-sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-# UDP socket
-sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
-
-

Server Side

-

Binding a Socket

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- *
- * @brief Assigns the address specified by @p address to the socket
- *   @p socket.
- *
- * @param socket The socket fd to bind.
- * @param address The sockaddr struct containing the address and
- *   port to bind to. Usually retrieved from a call to getaddrinfo().
- * @param addr_len The length of the sockaddr struct.
- *
- * @return 0 on success, -1 on failure.
-*/
-int bind(int socket, struct sockaddr *address, int addr_len);
-
-
# ...
-port = 80
-
-# Bind the socket to an address
-serversock.bind((socket.gethostname(), port))
-
-

Listening on a Socket

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- * @brief Marks the socket @p socket as a passive socket, that is,
- *   as a socket that will be used to accept incoming connection
- *   requests using accept().
- *
- * @param socket The socket to listen on.
- * @param backlog The maximum length of the queue of pending
- *   connections.
- *
- * @return 0 on success, -1 on failure.
-*/
-int listen(int socket, int backlog);
-
-
# ...
-
-backlog = 5
-
-# Listen for connections
-serversock.listen(backlog)
-
-

Accepting a Connection

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- * @brief Accepts a connection on the socket @p socket.
- *
- * @param socket The socket to accept a connection on.
- * @param address A pointer to a sockaddr struct that will be
- *   filled with the address of the peer socket.
- * @param addr_len A pointer to an int that will be filled with
- *   the length of the sockaddr struct.
- *
- * @return The file descriptor of the new socket, or -1 on failure.
-*/
-int accept(int socket, struct sockaddr *address, int *addr_len);
-
-
# ...
-
-# Accept a connection
-clientsock, addr = serversock.accept()
-
-

Client Side

-

Connecting to a Server

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- * @brief Connects the socket @p socket to the address specified
- *   by @p address.
- *
- * @param socket The socket to connect.
- * @param address The sockaddr struct containing the address and
- *   port to connect to. Usually retrieved from a call to getaddrinfo().
- * @param addr_len The length of the sockaddr struct.
- *
- * @return 0 on success, -1 on failure.
-*/
-int connect(int socket, struct sockaddr *address, int addr_len);
-
-
import socket
-
-# create a socket
-clientsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-
-# connect to a server
-clientsock.connect(('http://elimelt.com', port))
-
-

Sending and Receiving Data

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- * @brief Sends @p len bytes from @p buf on the socket @p socket.
- *
- * @param socket The socket to send data on.
- * @param buf The buffer containing the data to send.
- * @param len The length of the data to send.
- * @param flags Flags to modify the behavior of the send.
- *
- * @return The number of bytes sent, or -1 on failure.
-*/
-ssize_t send(int socket, const void *buf, size_t len, int flags);
-
-/**
- * @brief Receives data from the socket @p socket and stores it
- *   in @p buf.
- *
- * @param socket The socket to receive data on.
- * @param buf The buffer to store the received data in.
- * @param len The length of the buffer.
- * @param flags Flags to modify the behavior of the receive.
- *
- * @return The number of bytes received, or -1 on failure.
-*/
-ssize_t recv(int socket, void *buf, size_t len, int flags);
-
-
# ...
-
-# Send data
-clientsock.send('Hello, world!')
-
-# Receive data
-data = clientsock.recv(1024)
-
-

Closing a Socket

-
#include <sys/types.h>
-#include <sys/socket.h>
-
-/**
- * @brief Closes the socket @p socket.
- *
- * @param socket The socket to close.
- *
- * @return 0 on success, -1 on failure.
-*/
-int close(int socket);
-
-
# ...
-
-# Close the socket
-clientsock.close()
-
-

Simple Client

-
// from book.systemsapproach.org
-
-#include <stdio.h>
-#include <sys/types.h>
-#include <sys/socket.h>
-#include <netinet/in.h>
-#include <netdb.h>
-#include <stdlib.h>
-#include <unistd.h>
-#include <string.h>
-
-#define SERVER_PORT 5432
-#define MAX_LINE 256
-
-int
-main(int argc, char * argv[])
-{
-  FILE *fp;
-  struct hostent *hp;
-  struct sockaddr_in sin;
-  char *host;
-  char buf[MAX_LINE];
-  int s;
-  int len;
-
-  if (argc==2) {
-    host = argv[1];
-  }
-  else {
-    fprintf(stderr, "usage: simplex-talk host\n");
-    exit(1);
-  }
-
-  /* translate host name into peer's IP address */
-  hp = gethostbyname(host);
-  if (!hp) {
-    fprintf(stderr, "simplex-talk: unknown host: %s\n", host);
-    exit(1);
-  }
-
-  /* build address data structure */
-  bzero((char *)&sin, sizeof(sin));
-  sin.sin_family = AF_INET;
-  bcopy(hp->h_addr, (char *)&sin.sin_addr, hp->h_length);
-  sin.sin_port = htons(SERVER_PORT);
-
-  /* active open */
-  if ((s = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
-    perror("simplex-talk: socket");
-    exit(1);
-  }
-  if (connect(s, (struct sockaddr *)&sin, sizeof(sin)) < 0)
-  {
-    perror("simplex-talk: connect");
-    close(s);
-    exit(1);
-  }
-  /* main loop: get and send lines of text */
-  while (fgets(buf, sizeof(buf), stdin)) {
-    buf[MAX_LINE-1] = '\0';
-    len = strlen(buf) + 1;
-    send(s, buf, len, 0);
-  }
-}
-
-
import socket
-import sys
-
-s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-s.connect((sys.argv[1], 5432))
-
-while 1:
-    line = sys.stdin.readline()
-    if not line:
-        break
-    s.send(line)
-s.close()
-
-

Simple Server

-
// from book.systemsapproach.org
-#include <stdio.h>
-#include <sys/types.h>
-#include <sys/socket.h>
-#include <netinet/in.h>
-#include <netdb.h>
-#include <stdlib.h>
-#include <unistd.h>
-#include <string.h>
-
-#define SERVER_PORT  5432
-#define MAX_PENDING  5
-#define MAX_LINE     256
-
-int
-main()
-{
-  struct sockaddr_in sin;
-  char buf[MAX_LINE];
-  int buf_len, addr_len;
-  int s, new_s;
-
-  /* build address data structure */
-  bzero((char *)&sin, sizeof(sin));
-  sin.sin_family = AF_INET;
-  sin.sin_addr.s_addr = INADDR_ANY;
-  sin.sin_port = htons(SERVER_PORT);
-
-  /* setup passive open */
-  if ((s = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
-    perror("simplex-talk: socket");
-    exit(1);
-  }
-  if ((bind(s, (struct sockaddr *)&sin, sizeof(sin))) < 0) {
-    perror("simplex-talk: bind");
-    exit(1);
-  }
-  listen(s, MAX_PENDING);
-
- /* wait for connection, then receive and print text */
-  while(1) {
-    if ((new_s = accept(s, (struct sockaddr *)&sin, &addr_len)) < 0) {
-      perror("simplex-talk: accept");
-      exit(1);
-    }
-    while (buf_len = recv(new_s, buf, sizeof(buf), 0))
-      fputs(buf, stdout);
-    close(new_s);
-  }
-}
-
-
import socket
-import sys
-
-s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-s.bind((socket.gethostname(), 5432))
-s.listen(5)
-
-while True:
-    clientsock, addr = s.accept()
-    while True:
-        data = clientsock.recv(1024)
-        if not data:
-            break
-        print(data)
-    clientsock.close()
-
-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/address-translation-page-fault.png b/site/operating-systems/lecture-notes/address-translation-page-fault.png deleted file mode 100644 index e9717ee..0000000 Binary files a/site/operating-systems/lecture-notes/address-translation-page-fault.png and /dev/null differ diff --git a/site/operating-systems/lecture-notes/components.html b/site/operating-systems/lecture-notes/components.html deleted file mode 100644 index b505a2d..0000000 --- a/site/operating-systems/lecture-notes/components.html +++ /dev/null @@ -1,288 +0,0 @@ - - - - - - Components - - - - - -
- -

Components

-
- Last modified: 2024-01-12 - -
-
-

Components of an OS

-

Process Operations

-

OS provides the following kinds of operations:

- -

Memory Management

-

primary memory is directly accessible to the CPU.

- -

OS must:

- -

I/O

- -

Device Drivers

-

Routines that interact with specific device types. Encapsulate the details of the device.

- -

Note: Windows has ~35,000 device drivers.

-

File Systems

-

A nice abstraction on top of physica storage device drivers. Provides the usual list of operations (open, close, read, write, seek, etc.), but also higher level operations on the fs itself:

- -

Structure

-
Monolith:
-
-+----------------------+
-|    User Programs     |
-+----------------------+
-|      Everything      |
-|      Else in OS      |
-+----------------------+
-|     Device Driver/   |
-|     hardware stuff   |
-+----------------------+
-
-
-

advantages: -- cost of module interaction is low (procedure call)

-

disadvantages: -- hard to maintain -- hard to extend

-

Layering

-

Dikjstra's "THE" multiprogramming system:

-

Each layer presents a "virtual machine" to the layer above it. Each layer only uses the services of the layer below it.

- -

Each layer can be tested and verified independently.

-

Hardware Abstraction Layer

-

Goal: seperate hardwarte-specific routines from the "core" OS. Provides protability and improves readability.

-

Microkernels

-

Introduced in the late 80s, early 90s. Goal: minimize the kernel, move as much as possible into user space.

-

Results in: -- better reliability (isolation between components) -- better portability (less code to port) -- better extensibility (easier to add new features)

-

OS is split into two parts:

- -

Probably slower than monolithic kernel because of all the expensive context switches.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/file-systems.html b/site/operating-systems/lecture-notes/file-systems.html deleted file mode 100644 index 51b6bc2..0000000 --- a/site/operating-systems/lecture-notes/file-systems.html +++ /dev/null @@ -1,279 +0,0 @@ - - - - - - File Systems - - - - - -
- -

File Systems

-
- Last modified: 2024-02-28 - -
-
-

File Systems

-

A file system interacts with storage by reading/writing blocks (sectors) on a per-volume basis. It is basically a thick layer of abstraction over the raw storage device.

-

Programming Interface

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
WindowsUnix
CreateFile(name, CREATE)open(name, O_CREAT)
CreateFile(name, OPEN)open(name, O_RDONLY)
ReadFile(handle, buffer, size)read(fd, buffer, size)
WriteFile(handle, buffer, size)write(fd, buffer, size)
CloseHandle(handle)close(fd)
FlushFileBuffers(handle)fsync(fd)
SetFilePointer(handle, offset, whence)lseek(fd, offset, whence)
GetFileSize(handle)fstat(fd, &buf)
DeleteFile(name)unlink(name)
MoveFile(old, new)rename(old, new)
-

We all know about this and I'm not going to get into it.

-

Windows vs. Unix

-

Moving Files

-

Windows and Unix have different ways of moving files. In Unix, you always use the rename system call. In Windows, moving a file within the same volume is a rename operation, whereas moving a file between volumes is a copy operation followed by a delete operation.

-

Deleting Files

-

In Unix, a file is not actually deleted until all references to it are removed. In Windows, a file is deleted as soon as the last reference to it is removed. This is why you can't delete a file that is open in Windows.

-

Files

-

A file is logically just a sequence of bytes, typically consisting of data and properties/metadata. Some file systems also have types (ie. regular file, directory, symbolic link, device, etc.).

-

Some files are also understood to be executable by the OS. In Windows, the file type is determined by the file extension. In Unix, the file type is determined by the file's metadata.

-

Shared file handles will also share the file's offset. This can be an issue if you have multiple threads or processes reading/writing to the same file handle.

-

Directories

-

A directory is typically just a file that contains metadata about the files it contains in the form of directory entries. It is a mapping from file names to file metadata.

-

Design Constriants

- -

On-Disk Structure

-

There is typically a layer of storage that actually gets persisted on disk, as well as the in-memory representation of the file system that is manipulated by the OS.

-

FAT File System

-

The File Allocation Table file system is a simple file system that was used in DOS and Windows. It has a simple on-disk structure that consists of a reserved area, a number of FAT areas, and a data area. The reserved area contains the boot sector and metadata about the file system/disk layout. The FAT areas contain the file allocation table, which is a table that maps file names to disk blocks. The data area contains the actual file data.

-
+-----------------+-----------------+-----------------+-----------------+
-| Reserved Area   | FAT Area 1      | FAT Area 2      | Data Area       |
-+-----------------+-----------------+-----------------+-----------------+
-
-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/handle-tables.html b/site/operating-systems/lecture-notes/handle-tables.html deleted file mode 100644 index 1ea19d3..0000000 --- a/site/operating-systems/lecture-notes/handle-tables.html +++ /dev/null @@ -1,224 +0,0 @@ - - - - - - Handle Tables - - - - - -
- -

Handle Tables

-
- Last modified: 2024-02-25 - -
-
-

Handle Tables

-

There is a local handle table for each process, and a global handle table within the kernel.

-

Process Control Block (PCB)

- -

proc has many many fields. (pid, pointer to paprent proc, execution state, etc.)

-

In Linux, defined within task_struct(include/linux/sched.h). Has over 95 fields!

-

proc and CPU state

-

context switch: take the currently running processes state from the CPU and save it in the PCB, then grab the next scheduled proccess' proc off of the PCB and load it into the CPU to run.

-

Choosing the next process to run is called scheduling.

-

The OS kernel is NOT a process, just a block of code. Remember: the CPI is always executing code in the context of a process. That code may be in either kernel or user mode.

-

State Queues

-

Typically there is a queue of procs that correspond to each of the states a process can take (ie WAITING, READY). There might even be many wait queues, one for each type of wait (particular device, timer, message, etc.).

- -

Creation

- -

Note that exec doesn't create a new process.

-

To create a new process (ie from a shell), fork the shell process, and then exec the program

-

Making Creation Faster

-

method 1: vfork:

-

the older (now uncommon) way to do it. Instead of making a new child address space being a copy, just point to the parent address space from the child. This was an unenforced "promise" that the child wouldn't modify the address space. The child has the same page table and everything.

-

method 2: copy on write.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/intel-5-level-paging.png b/site/operating-systems/lecture-notes/intel-5-level-paging.png deleted file mode 100644 index 008e37b..0000000 Binary files a/site/operating-systems/lecture-notes/intel-5-level-paging.png and /dev/null differ diff --git a/site/operating-systems/lecture-notes/io-systems-secondary-storage.html b/site/operating-systems/lecture-notes/io-systems-secondary-storage.html deleted file mode 100644 index 0e91a98..0000000 --- a/site/operating-systems/lecture-notes/io-systems-secondary-storage.html +++ /dev/null @@ -1,294 +0,0 @@ - - - - - - Io Systems Secondary Storage - - - - - -
- -

Io Systems Secondary Storage

-
- Last modified: 2024-02-26 - -
-
-

I/O Systems and Secondary Storage

-

I/O System Hardware Environment

-

I/O devices are typically either block devices, which transfer data in fixed-size blocks, or character devices, which transfer data one character at a time in a stream.

-

Device controllers are the hardware (a mini-computer) that connects the CPU to the I/O devices. They are responsible for sending commands to the device, and for receiving and sending data to and from the device.

-

The I/O devices communicate via controller registers/memory-mapped I/O, or direct memory access (DMA).

-

Old computers typically had a single bus (the system bus) that connects the CPU, memory, and I/O devices. The topology of the computer was similar to old ethernet networks with a single broadcast domain.

-

More modern and performant systems have multiple buses. The PCI bus is a high-speed backbone that all the other busses branch off of (memory, SCSI, USB, etc.).

-

The I/O system needs to be able to handle a wide variety of devices with different data transfer rates, data formats, and control mechanisms.

-

I/O System's Role and Structure

-

Needs to provide:

- -

Organization

- -

Secondary Storage

-

Anything outside of primary memory (RAM) is considered secondary storage. This includes hard drives, SSDs, and other storage devices. SS doesn't allow direct execution of instructions or data access via load/store instructions, and is instead accessed via I/O operations.

-

Secondary storage is... -- Non-volatile: Data is retained even when the power is off. -- Very slow compared to primary storage. -- Failure-prone. -- Giant and relatively cheap compared to primary storage. 2024 numbers: - - 2 TB HDD for $73 ($0.04/GB) - - 30 TB HDD for $700 - - 500 GB SSD for $50 ($10/GB) - - 100 TB SSD for $40,000

-

Memory Hierarchy

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
LevelSpeedCostSizeVolatility
RegistersFastestMost ExpensiveSmallestVolatile
L1 CacheFastExpensiveSmallVolatile
L2 CacheFastLess ExpensiveStill not a lotVolatile
Main MemorySlowerLess ExpensiveLargerVolatile
Secondary StorageSlowCheapLargestNon-Volatile
Tertiary StorageSlowestLeast ExpensiveLargestNon-Volatile
-

HDDs and SDDs

-

HDDs are mechanical devices that store data on spinning disks. They have a read/write head that moves across the disk to read and write data. They are slow, but have a large capacity and are relatively cheap.

-

SSDs are solid-state devices that store data on flash memory. They are much faster than HDDs, but are more expensive. They are also more reliable and consume less power.

-

Disks and the OS

-

Disks are messy, slow, error-prone, horrible devices, and its the OS's job to make them look like a nice, clean, fast, reliable, and easy-to-use device.

-

The OS typically provides different levels of disk access to different clients, including: -- physical block access: the ability to read and write blocks of data to the disk. -- disk logical block access: the ability to read and write to a disk block number, without needing to know the physical location of the block. -- file system: the ability to read and write files at a specified offset, block, or byte.

-

With old disks, only physical block access was available. With modern disks, the controller provides a higher-level interface that maps physical regions on disk (cylinders, sectors, ect.) to logical block numbers from $[0, n]$, (ie a contiguous range of blocks to the OS).

-

Performance Issues

-

The HDD's performance is affected by its mechanically moving parts. Limiting the amount of seeking and defragmenting help, but only to an extent.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/kernel-abstraction.html b/site/operating-systems/lecture-notes/kernel-abstraction.html deleted file mode 100644 index f6324db..0000000 --- a/site/operating-systems/lecture-notes/kernel-abstraction.html +++ /dev/null @@ -1,207 +0,0 @@ - - - - - - Kernel Abstraction - - - - - -
- -

Kernel Abstraction

-
- Last modified: 2024-01-18 - -
-
-

Lecture 2 -

-

Hardware Modes

-

Who actually gets to control the hardware?

-

The application? It would be simple and quick, but isn't safe at all.

-

The OS? Can act on behalf of the application, which gives us protection.

-

Challenge: Protection with Restrictions

-

How do we execute code with restricted privileges?

-

Hardware Support: Dual Mode Operation

- -

On x86, mode stored in EFLAGS register. On MIPS, mode stored in status register.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/page-faults.html b/site/operating-systems/lecture-notes/page-faults.html deleted file mode 100644 index f959912..0000000 --- a/site/operating-systems/lecture-notes/page-faults.html +++ /dev/null @@ -1,275 +0,0 @@ - - - - - - Page Faults - - - - - -
- -

Page Faults

-
- Last modified: 2024-02-16 - -
-
-

Page Fautls

-

How does the OS handle a page fault?

-

On fault, an interrupt causes the CPU to jump to the page fault handler:

- -

Finding the page on disk

- -

Find or create a page frame

- -

Issues with Page Faults

-

Memory reference overhead

-

There are 2 references to memory for every memory access: one to the page table, and one to the actual memory. This can be mitigated by using a TLB (Translation Lookaside Buffer), which is essentially a cache for the page table.

-

Memory required to hold a page table can be large

- -

Old Solution: Paging the page table

-

Can be solved by paging the page table! (ie. have a page table for the page table). Keep the "system" page table in physical memory, and the "user" page table in virtual memory. This is no longer done in practice.

-

New Solution: Multi-level page tables

-

Simply add another level of indirection. Instead of having a single page table, have a page table of page tables. This works well because the page table is sparsely populated in practice, so it is a waste to have a PTE mapped for every page in the address space.

-

This is the solution used in modern operating systems.

-
Two-level page table
- -

Other Alternatives

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/paging-diagram.png b/site/operating-systems/lecture-notes/paging-diagram.png deleted file mode 100644 index 4099367..0000000 Binary files a/site/operating-systems/lecture-notes/paging-diagram.png and /dev/null differ diff --git a/site/operating-systems/lecture-notes/paging.html b/site/operating-systems/lecture-notes/paging.html deleted file mode 100644 index 3514446..0000000 --- a/site/operating-systems/lecture-notes/paging.html +++ /dev/null @@ -1,369 +0,0 @@ - - - - - - Paging - - - - - -
- -

Paging

-
- Last modified: 2024-02-14 - -
-
-

Virtual Memory and Paging

-

Use pages to map virtual memory to physical memory. This prevents external fragmentation by dividing memory into fixed-size pages, and internal fragmentation by making the units of allocation smaller.

-

Fragmentation

- -

Virtual address space is divided into pages, and physical address space is divided into frames. The page table maps pages to frames. The page table is stored in memory, and the page table base register (PTBR) points to the page table. The page table is indexed by the page number, and the value at that index is the frame number.

-

From the programmers perspective, memory is a giant contiguous block, completely independent of the physical memory and hardware.

-

Protection

-

One processes cannot "name" or address the memory of another process. This provides protection between processes.

-

Set the first page to be invalid so that if a process tries to access it (NULL pointer), it will cause an exception.

-

Address Translation

- -

Page tables are managed by the operating system, and are stored in memory. There is one PTE for each page, ie one PTE per VPN. The page table maps VPNs to PFNs. Each process has its own page table, and the PTBR points to the page table.

-

Shared Frames

-

Multiple processes can share the same frame. This is useful for shared libraries, and for shared memory between processes. Can also be used when implementing copy-on-write (COW) to optimize things like read-only fork, or exec.

-

Page Table Entries

-

More functionality to the PTEs:

- -

More out there.

-

Advantages of Paging

- -

Disadvantages of Paging

- -

Paged Virtual Memory

- -

Page Faults

- -

Demand Paging

- -

Page Replacement

- -

Page Replacement Algorithms

- -
Belady's Optimal Algorithm
- -
FIFO
- -
LRU
- -
Approximate LRU
- -
LRU Clock
- -

How do you load a program?

- -

Locality

- -

Locality means paging can be infrequent, and the OS can bring in multiple pages at once. This assumes that:

- -

Local vs Global Page Replacement

-

Local page replacement means that each process has its own set of pages that it is replacing. Global page replacement means that the OS can choose any page to replace, regardless of which process it belongs to. Linux uses global page replacement.

-

This is typically implemented by keeping a pool of free pages, and when a page is needed, the OS can choose any page to evict. This is useful because it allows the OS to make better decisions about which pages to evict, and can reduce the number of page faults.

-

Working Set Model

- -

$$ -WS(t,w) = {\text{pages P such that P was referenced in the time -interval } (t, t-w)} -$$

-

$|WT(t, w)|$ is the number of pages in the working set at time $t$, and varies with time. During a time interval with particularly bad locality, the working set can be very large.

- -

The goal is to reduce page faults by keeping the working set in memory. Thrashing is when a process is spending more time paging than executing, and keeping the working set in memory can help prevent thrashing.

-

Hard vs Soft Page Faults

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/processes.html b/site/operating-systems/lecture-notes/processes.html deleted file mode 100644 index aa516ec..0000000 --- a/site/operating-systems/lecture-notes/processes.html +++ /dev/null @@ -1,239 +0,0 @@ - - - - - - Processes - - - - - -
- -

Processes

-
- Last modified: 2024-01-12 - -
-
-

Processes

-

What is a process?

-

The OS's abstraction of a running program. A process is a program in execution.

-

Simplest case:

- -

What's "in" a process?

-

Consists of (at least)

- -

Address Space

-

Remember stack grows down (ie push decrements ESP, pop increments ESP). Stack contains all runtime frames.

-

A processes address space (idealized):

-
+----------------------+ <- 0x7FFFFFFF
-|        Stack         |
-|   (dynamic memory)   |
-+----------------------+ <- ESP
-|          |           |
-|          v           |
-|                      |
-|          ^           |
-|          |           |
-+----------------------+
-|        Heap          |
-|   (dynamic memory)   |
-+----------------------+
-|        Data          |
-|   (data segment)     |
-+----------------------+
-|        Code          | <- EIP
-|   (text segment)     |
-+----------------------+ 0x00000000
-
-

OS Process Namespace

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/tlb.html b/site/operating-systems/lecture-notes/tlb.html deleted file mode 100644 index ec14895..0000000 --- a/site/operating-systems/lecture-notes/tlb.html +++ /dev/null @@ -1,238 +0,0 @@ - - - - - - Tlb - - - - - -
- -

Tlb

-
- Last modified: 2024-02-16 - -
-
-

Translation Lookaside Buffer (TLB)

-

Translates virtual page numbers to physical page numbers. It is a small, fully associative cache of page table entries implemented in hardware to improve the speed of address translation and decrease the number of memory accesses.

-

Associative and Direct Mapping

- -

Managing TLBs

- -

Context Switching

-

The OS needs to ensure TLB and page table are consistent. This is done by invalidating the TLB when the page table changes. When a process is switched, the OS must invalidate/flush the entire TLB, which is a big part of the overhead of context switching (since there will be many TLB misses subsequently). You can also use the PID as part of the TLB lookup to make the TLB "global" (ie. shared between processes).

-

Functionality Enhanced by Page Tables

- -

Loading Shared Libraries

- -

Memory Mapped Files

-

Forget about doing reads/writes. Instead, map the file into the address space. Any time you write to the address space, it writes to the file. Depending on the OS and cache type (write-through vs write-back), the file may be written to immediately or later. This is a very efficient way to read/write files, and is used in many applications.

-

Soft Page Faults

-

Fault on a page that are actually in memory, but the PTE was marked as invalid. Resolving soft faults is relatively cheap. This can be used whenever you need to wake up the OS to do something on reference to a page (for instance, a debugger watch point). Windows uses soft faults in its page replacement strategy.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/windows-memory-management.html b/site/operating-systems/lecture-notes/windows-memory-management.html deleted file mode 100644 index aae23a2..0000000 --- a/site/operating-systems/lecture-notes/windows-memory-management.html +++ /dev/null @@ -1,214 +0,0 @@ - - - - - - Windows Memory Management - - - - - -
- -

Windows Memory Management

-
- Last modified: 2024-02-21 - -
-
-

Windows Memory Management

-

Pages have one of the following states defined in the page frame number (PFN) database:

- -

Windows uses a mixture of local and global page replacement, and maintains an LRU list of pages on top of a FIFO list. The LRU list is used to select pages for replacement (working set), and the FIFO list (standby list) is used to select pages for trimming. The LRU list is maintained by the memory manager, and the FIFO list is maintained by the cache manager.

-

A Major Problem with Page Replacement in Early Windows

-

Using a working set model for page replacement, when a user's process goes idle for long periods of time, all of the pages used by the process are removed from memory. Background processes like antivirus and indexing services exacerbate this problem. When the user returns to the process, it must wait for the pages to be read back into memory from disk.

-

An Interesting Exam Question from 2013

-

Examine how long it takes a user mode program writing to an array of integers using various access patterns. Assume the entire array fits into memory, and that the system is idle besides this program. The Stride is the number of elements between each access. The array is a constant size of PGSIZE (4096 bytes).

-
int access(int* arr, int size, int stride);
-
-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/windows-objects-handles-refcounts.html b/site/operating-systems/lecture-notes/windows-objects-handles-refcounts.html deleted file mode 100644 index 04c2a03..0000000 --- a/site/operating-systems/lecture-notes/windows-objects-handles-refcounts.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Windows Objects Handles Refcounts - - - - - -
- -

Windows Objects Handles Refcounts

-
- Last modified: 2024-01-19 - -
-
-

Obects Handles and Reference Counts

-

Object Manager

-

Windows was written in C, which doesn't have a notion of "objects". Early Windows had its own Object Manager, which was responsible for defining object types, naming objects, handling handles for users etc.

-

For example, there were objects for threads, processes, files, semaphores, etc.

-

Each object had a handle count, the number of handles the user has to the object, and the ref count, the number of references to the object within the kernel (which also includes handles).

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/lecture-notes/windows-page-frame-diagram.png b/site/operating-systems/lecture-notes/windows-page-frame-diagram.png deleted file mode 100644 index f3012b0..0000000 Binary files a/site/operating-systems/lecture-notes/windows-page-frame-diagram.png and /dev/null differ diff --git a/site/operating-systems/lecture-notes/windows-rtz.html b/site/operating-systems/lecture-notes/windows-rtz.html deleted file mode 100644 index 11fa5b7..0000000 --- a/site/operating-systems/lecture-notes/windows-rtz.html +++ /dev/null @@ -1,217 +0,0 @@ - - - - - - Windows Rtz - - - - - -
- -

Windows Rtz

-
- Last modified: 2024-02-09 - -
-
-

Hard Lessons Learned: Windows RtlZeroMemory

-

Zero Memory

-

Make it faster by picking larger register (same optimization can be done to copy memory).

-

In early Windows, they picked the BIGGEST register (floating point)

-

Speed Up Interrupt Handling

- -

Gary's Sad Story

- -

It turns out Dave Cutler tried to optimize interrupt handling to not save fp registers. Both RtlZeroMemory and RtlCopyMemory didnt copy the fp registers. When the filesystem calls RtlZeroMemory, if an interrupt occurs and the device calls CopyMemory, this has side effects when control is returned to the process.

-

Moral of the Story

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/reference.html b/site/operating-systems/reference.html deleted file mode 100644 index 9a76ed3..0000000 --- a/site/operating-systems/reference.html +++ /dev/null @@ -1,201 +0,0 @@ - - - - - - Reference - - - - - -
- -

Reference

-
- Last modified: 2024-03-25 - -
-
-

Textbook

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/section-notes/lab-3-questions.html b/site/operating-systems/section-notes/lab-3-questions.html deleted file mode 100644 index 12a7425..0000000 --- a/site/operating-systems/section-notes/lab-3-questions.html +++ /dev/null @@ -1,251 +0,0 @@ - - - - - - Lab 3 Questions - - - - - -
- -

Lab 3 Questions

-
- Last modified: 2024-02-21 - -
-
-

Lab 3 Questions

-

Question #1

-

Q: Why might an application prefer using malloc and free instead of using sbrk directly?

-

A: Malloc and free are higher-level and don't require the user to think about the details of the heap. Instead of needing to consider the current top of the heap, and how the memory you request is laid out contiguously, malloc and free handle all of that for you. Thus, user programs are able to apply more complex optimizations/strategies for memory allocation like first and best fit, as well as coalescing free blocks to reduce fragmentation, without having to manage the added complexity.

-

Question #2

-

Q: What is the relationship between malloc/free and sbrk?

-

A: When malloc lacks the space nessesary to fulfill an allocation request, it calls sbrk to expand its heap region and increase the memory available to the process. With the current implementation of malloc in umalloc.c, sbrk only ever expands the heap (through morecore), but we never shrink the heap. In a real system, we would probably also want to shrink the heap once we have enough free space that it would be unlikely to be used in the near future.

-

Question #3:

-

Q: How many child processes are created by the shell program in order to run the command ls | wc? (This mirrors real OS'es).

-
-

Hint: the shell will go into the case PIPE case in user/sh.c:runcmd when it receives a cmd with the pipe operator |.

-

Fun fact: the | operator behaves this way in most UNIX shells, and is why the infamous forkbomb command :(){ :|:& };: forks. More on the forkbomb LINK.

-
-

A: The shell will create two child processes, one for ls and one for wc. The ls process will write to the pipe, and the wc process will read from the pipe.

-

Question #4:

-

Q: The shell will call pipe() when the command ls | wc is run. What does the shell do with the read end? What does the shell do with the write end? (~1-2 sentences).

-
-

Hint: once again look at case PIPE in user/sh.c:runcmd.

-
-

A: The shell's process (ie the parent) closes both ends of the pipe. However, the child process created for ls closes its own stdout and then immediately duplicates the write end of the pipe, replacing its stdout with the write end of the pipe. The child process created for wc similarly closes its own stdin and then duplicates the read end of the pipe, replacing its stdin with the read end of the pipe. Both child processes then close both file descriptors of the pipe, and execute their commands using the newly duplicated file descriptors for the pipe (which are stdin/out from their perspectives).

-

Question #5:

-

Q: When a syscall completes, user-level execution resumes with the instruction immediately after the syscall. When a page fault exception completes, where does user-level execution resume? (~1 sentence).

-

It resumes at the instruction that caused the page fault. The processor exception saves the state of the program, and then the kernel handles the page fault. Once the kernel has resolved the page fault, it restores the state of the program and resumes execution at the instruction that caused the page fault (not the one after it).

-

Question #6:

-

Q: How should your xk implementation decide whether an unmapped reference is a normal stack operation versus a stray pointer dereference that should cause the application to halt?

-
-

Hint: keep in mind that your stack grower in xk should allow the user stack to grow to 10 pages, and no more than 10 pages.

-
-

The faulting address needs to be within stack->va_base >= addr >= stack->va_base - (PGSIZE * 10). If the faulting address is within the stack region, then it is a normal stack operation. If the faulting address is outside the stack region, then it is a stray pointer dereference and the application should halt.

-

Question #7:

-

Q: Is it possible to reduce the user stack size at run-time (i.e., to deallocate the user stack when a procedure with a large number of local variables goes out of scope)? If not, explain why. If so, sketch how that might work. (~1-3 sentences).

-

Question #8:

-

Q: The TLB caches the page table entries of recently referenced pages. When you modify the page table entry to allow write access, which function in vspace.c should you use to ensure that the TLB does not have a stale version of the page table entry you modified?

-
-

Hint: in x86-64 modifying the CR3 register changes what page table the CPU uses and flushes the TLB. Look for a function that modifies the CR3 register.

-
-

Question #9

-

Q: For each member of the project team, how many hours did you spend on this lab?

-

Question #10

-

Q: What did you like or dislike about this lab? Is there anything you wish you knew earlier?

-

Debugging

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/section-notes/section-1.html b/site/operating-systems/section-notes/section-1.html deleted file mode 100644 index b9a184a..0000000 --- a/site/operating-systems/section-notes/section-1.html +++ /dev/null @@ -1,226 +0,0 @@ - - - - - - Section 1 - - - - - -
- -

Section 1

-
- Last modified: 2024-01-18 - -
-
-

Section 1 - C and GDB review

-

C Review

-

static: has different meanings

- -

extern: declares variable without allocating any memory for it

- -

-void change(char** s) { *c = "class"; }
-
-int main() {
-   char* s = "hello";
-   char* w = s;
-
-   change(&w);
-}
-
-

When you use an uninitialized pointer, the address that the pointer stores is the uninitialized part, and will probably lead to errors when it is interpreted as an address.

-

GDB Review

-

printf debugging are useful, but limited when it comes to debugging concurrent code

-

Enter GDB

-

run <...args>: start execution

-

n: next instruction

-

bt: backtrace

-

watch <variable>: breakpoint when it changes

-

p <opt> <arg>: print arg

-

`x : dereference and print arg

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v1-kernels-and-processes/1-introductions.html b/site/operating-systems/v1-kernels-and-processes/1-introductions.html deleted file mode 100644 index 5bd19cc..0000000 --- a/site/operating-systems/v1-kernels-and-processes/1-introductions.html +++ /dev/null @@ -1,368 +0,0 @@ - - - - - - 1 Introductions - - - - - -
- -

1 Introductions

-
- Last modified: 2024-01-12 - -
-
-

Introduction

-

What is an Operating System?

-

Referee. Manages the resources of a computer shared between applications.

-

Illusionist. Makes the computer's resources appear to be dedicated to each application through abstraction.

-

Glue. Binds the hardware, software, and users through services that facilitate communication/sharing.

-

Virtualization

- -

Services

-

Services are provided by the operating system so applications can adapt to a single interface (like POSIX) instead of the hardware. Some examples of services are:

- -

Evolution of Operating Systems

-

Reliability and Availability. Does what it is expected without crashing. At scale, an OS must be invulnerable to failure, even in edge cases. Availability, the percentage of time a system is available, is influecned by Mean Time To Failure (MTTF) and Mean Time To Repair (MTTR). Increasing MTTF and decreasing MTTR increases availability.

-

Security. Prevents compromise from malicious users. Privacy is an aspect of security; data is only accessible to authorized users. Strong fault isolation is necessary for security.

-

Portability. Code can be run independently of the hardware. This is achieved through abstraction. Abstract Virtual Machine (AVM) is the interface between the OS and applicatons. Hardware Abstract Layer (HAL) is the interface between the OS and hardware.

-

Performance. Must be fast and efficient for users' sake. Performance is measured by... -- overhead. the cost of abstraction. -- efficiency. the ability to minimize overhead. -- fairness. the ability to allocate resources fairly. -- response time. the time between a request and a response. -- throughput. the number of requests per unit time. -- predictability. the ability to provide consistent performance.

-

Adoption. Also important

-

Design Tradeoffs

-

Sometimes, it is better to sacrifice one aspect for another. For example, losing some performance to fit an interface. The tradeoff of performance and complexity is common.

-

Early (single user) operating systems would let the CPU wait idle for I/O. Later, when multi-user systems were introduced, the CPU would switch to another process while waiting for I/O. This is called multiprogramming. In a bath operating system, the CPU reads from a queue of jobs, loading, running, and unloading each job.

-

While one job is running, the OS may set up IO devices for another in the background through direct memory access (DMA). Through interrupts, the OS can then switch to the new job when it is ready. From the point of view of the original job, there was just a short delay. This is called time sharing.

-

When the OS directly controls multiple concurrent processes, (multiprocessing), debugging becomes difficult, and developers need to essentially stop the system to debug. Virtualization solves this problem by creating the illusion of multiple processors on one processor. This allows developers to debug one virtual processor while the others continue to run. Instead of debugging directly on the hardware, developers can debug on a virtual machine that is being run as an application.

-

Ch. 1 Exercises

-

Introduction

-

1. What is an example of an operating system as:

- -

The OS manages the running processes and their access to resources.

- -

The OS creates the illusion of dedicated resources through abstraction, like with malloc.

- -

The OS provides services like the file system and network.

-

2. What is the difference, if any, between the following terms:

- -

Reliability is the ability to do what is expected without crashing. Availability is the percentage of time a system is available. More reliable systems have higher availability.

- -

Security is the ability to prevent compromise from malicious users. Privacy is the ability to prevent unauthorized users from accessing data.

- -

Security enforcement is the mechanism that prevents compromise. Security policy is the rules that define what is allowed and what is not.

- -

Throughput is the number of requests per unit time. Response time is the time between a request and a response.

- -

Efficiency is the ability to minimize overhead. Overhead is the cost of abstraction.

- -

API is the interface between client applications and the software they're consuming. AVM is the interface between the OS and the applications running on it.

- -

AVM is the interface between the OS and the applications running on it. HAL is the interface between the OS and the hardware.

- -

Proprietary operating systems are owned by a company and are not open source. Open operating systems are open source and free to use.

- -

Batch operating systems read from a queue of jobs, loading, running, and unloading each job. Interactive operating systems allow users to interact with the system while it is running.

- -

Host operating systems run on the hardware and typically manage guest operating systems. Guest operating systems run on top of the host operating system.

- -

Multiprogramming allows the CPU to switch to another process while waiting for I/O. Multiprocessing allows multiple processes to run concurrently.

-

3. Define the term, direct memory access (DMA).

-

DMA allows the OS to set up IO devices for another process in the background while the CPU is running another process. Through interrupts, the OS can then switch to the new process when it is ready.

-

4. Before there were operating systems, someone needed to develop solutions without being able to look them up! How would you have designed the first operating system?

-

Albeit relying on what I just read...

-

Users need to be able to execute arbitrary code on hardware. Although security is important, I won't focus on it for now.

-

First, I would need a way to load and run code. Assuming I can use a preexisting compiler (GCC?), I would first create a set of "kernel" modules that interact directly with the hardware. These modules would be written in C and assembly, and would cover the very basics of interacting with IO devices and any other hardware I need to control. In particular:

- -

Next, I would write a few services that users can interact with, as an interface to the above hardware drivers. In fact, users should NOT be able to interact with the preceding drivers directly, instead using the services I provide. These services would be written in C and assembly, and would cover the following:

- -

My file system would be a simple tree structure, with directories and files. Similar to the UNIX filesystem, I would use file descriptors to refer to all IO devices, including files, directories, and network sockets. Currently open file descriptors would be stored in a table, and would be passed to the kernel modules to perform IO operations. Disk IO would be buffered in memory for writes to make logging efficient. Reading can be done directly from disk with our storage device drivers.

-

My memory allocator would be a basic implementation of malloc using a linked list. Simple compaction running periodically would be used, with heap sweeping on process termination. No virtual memory yet...

-

Network communication would be done entirely through reads/writes over sockets. TPC/IP would be used for communication, assuming a modern network stack is available. If not, then machines would need to be connected directly to each other and would read/write directly to each other's open sockets using whatever protocol is available (or a new one I would design).

-

Process management would be done through fork/exec/wait. I would need to implement a way to handle interrupts, so that processes can be interrupted and switched to when IO is ready. Callbacks would be used to handle this, where processes register a callback with the OS when blocked on IO. When the IO is ready, the OS would call the callback and the process would resume (or be added to the ready queue if it is not the highest priority process). An event driven model would be used for all IO, so that processes can be interrupted when IO is ready. Each event should contain the nessessary information to stop and resume work, and then I can just use a producer/consumer model to handle events off of an event queue. I would use a simple round robin scheduler to switch between processes, and would keep both a log of events, and streaming statistics to try and schedule events fairly. Although limited, this would allow for some concurrency since processes can be interrupted while waiting for IO.

-

System Design

-

5. Suppose a computer system and all of its applications were completely bug-free and everyone in the world were completely honest and trustworthy. In other words, we need not consider fault isolation.

- -

Round robin

- -

Ideally, each application would have its own memory space. This requires implementing some sort of virtual memory abstraction.

- -

Applications should access disk through the file system, which has an interface for creating, deleting, reading, and writing files. The OS keeps a table of open files, and uses this table to perform IO operations on files in a consistent/safe way. Simply blocking access to files while they are being written to would be sufficient, although more complex buffering and locking could be used to improve performance.

-

6. Now suppose the computer system needs to support fault isolation. What hardware and/or operating support do you think would be needed to do the following?

- -

Permissions ^^^ and virtualization. Each process has its own memory space, and each user has its own file system. The OS can then use permissions to control access to resources for individual users. Network access should be mutually consentual, and the OS should prevent unauthorized access to the network.

-

7. How should an operating system support communication between applications?

- -

Each application can independently request access to a file, getting an entry on the currently open file table. If two processes want to read from the same file this is fine, but writing to the same file should be prevented from being overleaved. There should be multiple levels of locking, so that processes can lock a file to keep it for themselves, to prevent writes but allow reads, to prevent reads but allow writes, to only allow atomic reads/writes, but allow concurrent ownership, etc.

- -

Similar to network communication and file io, should be able to read/write to sockets. Pipes could be used to pass messages between processes, and io streams could be used to compose programs together.

- -

Reentrant locking service that thinly wraps a piece of shared memory owned in the kernel.

-

Essentially all of the above, but programmers need to understand the tradeoffs and correcness of each approach.

-

8. How would you design combined hardware and software support to provide the illusion of a nearly infinite virtual memory on a limited amount of physical memory?

-

Analogy of cache to memory, we would use main memory as a temporary space to hold the data of running processes. Then, when another process needs time on the CPU, the context of main memory is switched to another process, and the previously occupying memory is loaded to disk.

-

The process management service would handle this "paging" of memory by keeping a table of processes and the location and metadata for their memory. As tasks are pulled from the job queue, they are loaded into memory and added to the ready queue. When a process is blocked on IO, the OS can switch to another process and load the blocked process's memory to disk. When the process is ready again, the OS can load the process's memory from disk and resume execution.

-

9. How would you design a system to run an entire operating system as an application on top of another operating system?

-

The hardware modules that the host OS run on all follow an interface, so by mocking the interface of these hardware systems, we can simulate the hardware. Then, simply running the guest OS on top of this simulated hardware would allow us to run the guest OS as an application on top of the host OS. This should all be done in an isolated environment, so that the guest OS cannot access the host resources directly.

-

10. How would you design a system to update complex data structures on disk in a consistent fashion despite machine crashes?

-

I would use a write ahead log (buffered in memory for high throughput events, but also flushed to disk in the background as often as possible to aid in recovery) that keeps track of all disk writes. Each record in the log would correspond to an event that happend regarding our data structure, and each event would have an id. We keep track of an atomic counter that keeps track of the last event executed on our structure. In the event of a crash, we can simply replay the log from the last event executed to the end of the log, and our data structure will be in a consistent state. We can also use this log to recover from a crash during a write, by simply replaying the log from the beginning.

-

11. Society itself must grapple with managing resources. What ways do governments use to allocate resources, isolate misuse, and foster sharing in real life?

-

Taxes, laws, and regulations. Taxes are used to allocate resources, and laws are used to prevent misuse and foster sharing. For example, public services (firefighters) are funded by taxes, and laws are supposed to help ensure people can faily access these services.

-

12. Suppose you were tasked with designing and implementing an ultra-reliable and ultra-available operating system. What techniques would you use? What tests, if any, might be sufficient to convince you of the system's reliability, short of handing your operating system to millions of users to serve as beta testers?

-

Extensive testing like in other forms of software (unit, integration, fuzzing, fault injection), as well as monitoring and telemetry of real devices. Formal verification would be nice, but is probably not feasible. A microkernel would be a good start, since it would allow me to isolate "trusted" code.

-

13. For the computer you are currently using, how should the operating system designers prioritize among reliability, security, portability, performance, and adoption? Explain why.

-

As a user of a macbook pro, security, performance, and adoption are extremely important. However, as a developer it would be nice to also have a portable and (less importantly) a reliable system. Something that I think the OS designers were less concerned about is portability (Apple hardware only), but this is also a question of adoption of the Sillicon based chips in Apple's hardware.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.html b/site/operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.html deleted file mode 100644 index c67e75e..0000000 --- a/site/operating-systems/v1-kernels-and-processes/2-the-kernel-abstraction.html +++ /dev/null @@ -1,614 +0,0 @@ - - - - - - 2 The Kernel Abstraction - - - - - -
- -

2 The Kernel Abstraction

-
- Last modified: 2024-01-14 - -
-
-

Chapter 2 - The Kernel Abstraction

-

"A central role of operating systems is protection - the isolation of potentially misbehaving applications and users so that they do not corrupt other applications or the operating system itself."

-

Kernel: The lowest level of software running on the operating system. It is the first program loaded on boot, and it remains in memory until the system is shut down. It is responsible for all interactions with the hardware, and it is the only program that runs in privileged mode.

-

Process: The execution of an application program with restricted access to the operating system and hardware. Processes are managed by the kernel, and they need to request access to resources from the kernel.

-

The Process Abstraction

-

In the same way an object instantiates a class in OOP, a process instantiates a program. In most OS's, a programs instructions are stored once in a file on disk. The user edits the file, then compiles the file into a binary executable. When a user runs the executable, the OS creates a new process including program data, heap, and stack, loads the executable into memory, and starts the process.

-

The OS keeps track of all processes in a process control block (PCB). The PCB contains the process ID, the process state, the program counter, the stack pointer, the memory management information, and the accounting information. The OS uses the PCB to keep track of all processes, and it uses the program counter to keep track of the next instruction to execute.

-

Digging Deeper: in Linux, the PCB is called the task struct, and is a doubly linked list of processes, starting off with the init task which has a pid of 0. Users can view processes using the ps command, or could view the data directly in /proc for the system, or /proc/<pid> for a specific process.

-

Dual-Mode Operation

-

There is a single bit in the CPU that determines whether the CPU is in user mode or kernel mode. When the CPU is in user mode, checks for privileged instructions are enabled that stop the program from executing anything that could harm the system. When in kernel mode, these checks are turned off.

-

The principal of least privilege states that a process should only have access to the resources it needs to do its job. This is enforced by the CPU by only allowing processes to access resources in user mode. Some operating system code runs in their own user-level processes, for example the window manager. Code that runs in Kernel mode needs to be trusted, because it has access to all of the hardware.

-

The hardware must be able to switch between user and kernel mode safely, and must support the following restrictions for user mode:

- - - - - - - - - - - - - - - - - - - - - -
FeatureDescription
Privileged InstructionsProhibited instructions that only the kernel can execute.
Memory ProtectionProcesses are restricted from accessing memory that does not belong to them.
Timer InterruptsThe CPU can interrupt and stop executing a process at any time using timer interrupts.
-

This "kernel bit" is just one of many flags set in the CPU. These flags aren't directly accessible to application code, but the process status register (PSR) is has corresponding flags that are set when the processor switches between user and kernel mode. This register works similarly to other registers (like arithmetic condition codes in assembly).

-

Intel x86 processors actually support 4 privilege levels, but none of MacOS, Windows or Linux make use of the extra 2 levels.

-

User and Kernel Memory

-

In physical memory, the kernel is stored at the top of memory, and the user program is stored at the bottom. The kernel is stored at the top because it needs to be able to access all of memory, and the user program is stored at the bottom because it needs to be able to access all of its own memory. The kernel is stored in a fixed location in memory, and the user program is loaded into memory at runtime.

-

Virtual Memory

-

In a naiive implementation (like MS-DOS), the OS directly loads the program into memory and gives it access to the entire memory space. This is problematic for a few reasons, including security, fragmentation of memory, difficulty in growing the stack/heap. Virtual memory solves these problems by giving each process its own virtual address space in which their memory starts at 0. The OS maps the virtual address space to physical memory, usually in fixed size chunks called pages. The OS can then swap pages in and out of memory as needed for each process. Modern OS's often also use address randomization to prevent attacks.

-

Timer Interrupts

-

The CPU's hardware timer is a timer that runs periodically (either by time or number of instructions) sends an interrupt to the CPU to switch back to kernel mode. Each core has its own timer. Usually, processes will only be stopped by a timer if the user signals the OS to stop the process.

-

Older versions of MacOS lacked the ability to forcibly stop a process, so if a process was stuck, the user would have to reboot the system. This was due to their preemtive scheduling algorithm, that instead of using an interrupt at the hardware level, would instead rely on processes to voluntarily poll the OS to see if they should stop. When a runaway process was stuck in a loop, it would never poll the OS, and the OS would never stop it, leading to those forever spinning beach balls.

-

Types of Mode Transfer

-

In a high performance server, the CPU may switch from kernel to user mode thousands of times every second. It is thus important for this switch to be fast and secure.

-

User to Kernel

-

There are three main ways: system calls, exceptions, and interrupts. Trapping is the process of syncronously switching from user to kernel mode.

-

Interrupts are the most common way to switch from user to kernel mode. They are triggered by an external event, like a hardware device or another process signal. The CPU finished any user instructions, saves the state, and then switches to kernel mode and calls the corresponding interrupt handler. On a multicore system, only one core will handle the interrupt. An alternative to interrupts would be the kernel polling IO devices, but this is inefficient.

-

To enable high performance IO, the OS uses direct memory access (DMA), and a circular queue of requests for each IO device to handle. Each entry in the queue is called a buffer descriptor.

-

Exceptions are triggered by internal events, like a page fault or a divide by zero error. The CPU finishes any user instructions, saves the state, and then switches to kernel mode and calls the corresponding exception handler. Another common use case is in debuggers. A breakpoint is just a modified instruction that traps into kernel mode when it is executed. The kernel then reloads the instruction and transfers control to the debugger.

-

Exceptions are extremely useful for virtualization, and are often used as a signal for the host OS to emulate some hardware functionality, or to execute an operation of behalf of the guest OS. In this way, a VM can run in user mode, and the host OS can handle all of the privileged instructions.

-

System Calls allow the user program to voluntarily request the OS to execute something on its behalf. Most processors have a special syscall or trap instruction to do this. Each call invokes some code at a pre-defined address in the kernel. To see a table of system calls on Linux, run man syscalls.

-

Kernel to User

-

There are several reasons why the kernel would want to switch to user mode.

- - - - - - - - - - - - - - - - - - - - - - - - - -
ActionDescription
New ProcessWhen the kernel creates a new process, it needs to switch to user mode to start executing the process.
Resume ProcessAfter interrupting a process, the kernel needs to switch back to user mode to resume the process.
Switching to a Different ProcessThe kernel may want to switch to a different process, for example if the current process is waiting for IO.
User-Level UpcallSimilar to interrupts, for the user program to handle asynchronous events, the kernel needs to be able to switch to user mode to call the user-level handler.
-

Implementing Safe Mode Transfer

-

Limited Entry Into Kernel

-

The kernel needs to be able to switch to user mode at any time, but the user program should not be able to switch to kernel mode. User programs should only be able to request privileged operations through paths defined by the OS/kernel. If the user program violates any built in rules, the kernel should be able to stop the program and switch back to user mode by returning an error code.

-

Atomic Changes to Processor State

-

When switching between modes, there needs to be distinction between the state of memory; kernel code has access to its local memory and registers as well as the program's memory and registers, whereas user code should be isolated. The switch between these two contexts needs to be atomic.

-

Transparent, Restartable Execution

-

The kernel needs to be able to stop a user program in the middle of execution at any point, and to resume execution with exactly the same state later. User processes shouldn't need to know that they are being interrupted. On interrupt, the processor saves the current state to memory, defers further events, changes to kernel mode, then jumps to the correct handler. After execution of the handler, the same is done in reverse to resume the user program.

-

Interrupt Vector Table

- - - - - - - - - - - - - - - - - - - - - - - - - -
EntryTypeExample
0-31Processor ExceptionsDivide by zero, page fault, etc.
32-255InterruptsHardware interrupts, system calls, etc.
64System Call/TrapSyscall instruction
-

On modern, multicore systems, interrup routing has become increasingly programmable at the kernel level. This is especially important for IO-heavy systems like web servers. The kernel can route interrupts to the core that is handling the IO, helping to avoid cache misses.

-

Interrupt Stack

-

Each core has its own interrupt stack. When an interrupt occurs, the processor pushes the current state onto the interrupt stack, and then switches to kernel mode. The interrupt handler then runs on the interrupt stack. When the handler is finished, the processor pops the state off the interrupt stack and resumes execution of the user program.

-

Most OS's allocate a kernel interrupt stack (KIS) for every user process. When a process is running, the hardware interrupt stack points to the processes' kernel interrupt stack (which should be empty if being run in User Mode). This makes it easy for the kernel to switch between processes inside an interrupt or syscall handler.

- -

Interrupt Masking

-

Hardware provides a privileged instruction to disable interrupts. This is useful for critical sections of code that should not be interrupted. On X86, disable interrupts defers interrupts until an enable interrupt instruction is executed. While they are deferred, interrupts are queued, but will be lost if this queue fills up. Generally, hardware will buffer one interrupt of each type, and will drop any additional interrupts of that type. Interrupts are disabled with %CLI and enabled with %STI, which only applies to the current CPU (for multi-core systems).

-

Different devices all have their own interrupt buffer, and also assign priotiries to each type of interrupt that they handle.

-

Hardware Support for Saving and Restoring Registers

-

For x86...

- -

More on Interrup Handlers

- -

Putting It All Together: x86 Mode Transfers

-

"First, we provide some background on the x86 architecture. The x86 is segmented, so -pointers come in two parts: (i) a segment, a region of memory such as code, data, or stack, -and (ii) an offset within that segment. The current user-level instruction is a combination of -the code segment (cs register) plus the instruction pointer (eip register). Likewise, the -current stack position is the combination of the stack segment (ss) and the stack pointer -within the stack segment (esp). The current privilege level is stored as the low-order bits of -the cs register rather than in the processor status word (eflags register). The eflags register -has condition codes that are modified as a by-product of executing instructions; the eflags -register also has other flags that control the processor's behavior, such as whether -interrupts are masked or not"

-
    -
  1. Mask interrupts to prevent the processor from being interrupted while switching modes
  2. -
  3. Save the current state of the user program on the interrupt stack, including the stack pointer, execution flags, and instruction pointer, all to temp hardware registers
  4. -
  5. Switch onto kernel interrupt stack by changing the stack pointer to point to its base address
  6. -
  7. Push the current state of the user program onto the kernel interrupt stack
  8. -
  9. Optionally save an error code if this was an exception with one.
  10. -
  11. Invoke interrupt handler by jumping to the correct entry in the interrupt vector table
  12. -
-

The interrupt handle also saves some registers to its own stack before executing any code that might overwrite the processor state (callee-saved registers).

-

Once finished executing, the callee then pops the registers off the stack, restoring the interrupted process state, excluding the program counter, execution flags, and stack pointer. The interrupt handler then executes a iret instruction, which restores those aforementioned elements of state, fully restoring the interrupted process.

-

Note... for exceptions that signal instructions in the kernel, the handler will modify the program counter to point to the instruction after the exception occurred, so as to prevent an infinite loop.

-

Implementing Secure System Calls

-

OS kernel constructs restricted enviornment for process execution. Any time a process needs to execute something outside of its protection domain, it requests the OS do so on its behalf using a system call.

-

System calls are the primary interface between user programs and the OS. When the user program makes a call, the library code will trap into kernel mode, and the kernel will execute the system call handler. System calls generally try to appear as a normal function call thorughout a library and often use stubs to mediate between the user program and the kernel. This allows the following flow:

-
    -
  1. User calls function in library (user stub)
  2. -
  3. Stub fills in code for system call, and traps into kernel mode
  4. -
  5. Hardware transfers control to kernel and finds correct system call handler (kernel stub), which checks args and then executes the correct system call
  6. -
  7. Syscall returns to kernel stub, which returns to user stub's next instruction, which returns to the user program
  8. -
-

For example, the user-level library stub for open in x86 looks like:

-
 // We assume that the caller put the filename onto the stack,
- // using the standard calling convention for the x86.
-
- open:
- // Put the code for the system call we want into %eax.
-    movl #SysCallOpen, %eax
-
- // Trap into the kernel.
-    int #TrapCode
-
- // Return to the caller; the kernel puts the return value in %eax.
-    ret
-
-

Where SysCallOpen is the code for the specific system call to run, and TrapCode is the index into the x86 interrupt vector table for the system call handler. Note that %eax is overwritten by the return of the syscall, and that int saves the user state (program counter, stack pointer, eflags) onto the kernel interrupt stack, beforte jumping to the syscall handler on the IVT. This example is fairly simple stub, so some of the details are implicit, but in general a kernel stub has four main tasks:

-

Locate syscall arguments. Unlike regular kernel procedures, the args of a syscall are in user memory, typically on a user's stack. Stubs need to locate arguments and ensure that any pointers reside in the user's address space. These will be virtual addresses, so the stub needs to translate them to physical addresses.

-

Validate parameters. The stub needs to check that all the arguments are valid, and that there aren't any invalid states. For example, if a syscall is trying to open a file, the kernel stub needs to check that it exists, that the user has permission to open it, etc.

-

Copy before check. To prevent time of check vs. time of use (TOCTOU) attacks, the kernel stub copies the arguments to kernel memory before checking them. This prevents the user from modifying the arguments after they have been checked in order to bypass the checks and supply illegal arguments.

-

Copy results back. The kernel stub copies the results of the syscall back to user memory before returning to the user program. This is necessary because the kernel stub executes in kernel mode, and thus has access to kernel memory, but the user program does not.

-
int KernelStub_Open() {
-    char *localCopy[MaxFileNameSize + 1];
-    // Check that the stack pointer is valid and that the arguments are stored at
-    // valid addresses.
-    if (!validUserAddressRange(userStackPointer, userStackPointer + size of arguments))
-        return error_code;
-    // Fetch pointer to file name from user stack and convert it to a kernel pointer.
-    filename = VirtualToKernel(userStackPointer);
-    // Make a local copy of the filename. This prevents the application
-    // from changing the name surreptitiously.
-    // The string copy needs to check each address in the string before use to make sure
-    // it is valid.
-    // The string copy terminates after it copies MaxFileNameSize to ensure we
-    // do not overwrite our internal buffer.
-    if (!VirtualToKernelStringCopy(filename, localCopy, MaxFileNameSize))
-        return error_code;
-    // Make sure the local copy of the file name is null terminated.
-    localCopy[MaxFileNameSize] = 0;
-    // Check if the user is permitted to access this file.
-    if (!UserFileAccessPermitted(localCopy, current_process)
-        return error_code;
-    // Finally, call the actual routine to open the file. This returns a file
-    // handle on success, or an error code on failure.
-    return Kernel_Open(localCopy);
-}
-
-

Starting a New Process

-

At a high level, before running a program the kernel must...

- -

Then, to actually run it, the kernel must...

- -

Finally, there is a level of indirection between program execution in the form of a stub. This ensures that the program calls exit (ie terminates), and is done by the compiler. Conceptually, this looks like:

-
start(int argc, char** argv) {
-    main(argc, argv);
-    exit();
-}
-
-

Implementing Upcalls

-

Upcalls are a way for the kernel to notify the user program of an event. They are similar to interrupts, but are initiated by the kernel rather than hardware. These are called signals in Unix, and asynchronous events in Windows, and are crucial for virtualization.

-

Applications:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Upcall PurposeDescription
Preemptive user-level threadsEnables threading libraries to use a periodic timer upcall to switch and/or terminate threads.
Asynchronous I/O NotificationsApplications can make syscalls asynchronously. When the kernel finishes the operation, it notifies the application with an upcall.
Interprocess communicationAny time two processes need to communicate in real time, they can use upcalls to notify each other of events. Another example might be when the user logs out, the kernel can send an upcall to all processes to terminate.
User-level exception handlingIf application runtimes have their own exception system, they can be notified of processor exceptions with upcalls.
User-level resource allocationIf an application needs to be resource adaptive, it can use upcalls to monitor resource usage and adjust accordingly. The JVM uses upcalls to monitor memory usage and garbage collect.
-

Upcalls aren't always needed, and many programs get by with an event loop model. In fact, Windows didn't support immediate delivery of upcalls to user-level programs until recently.

-

Unix Signals

- -

Booting an Operating System Kernel

-

Systems typically use a special read only memory (Boot ROM) to store boot instructions. On most x86 computers, this is stored in the BIOS (Basic Input/Output System).

-

ROM memory is relatively slow and expensive, and typically never changes, whereas kernel code needs to be updated often, so it is preferable to keep to BIOS minimal.

-

BIOS provides indirection between hardware and OS. It reads a fixed-size block of data from a fixed location on disk (or flash RAM) called the bootloader into memory. Some newer bootloaders also store a cryptographic signature to verify the bootloader's integrity.

-

Then, the bootloader loads the kernel into memory and transfers control to it. The kernel then initializes the hardware, loads the rest of the OS, and starts the first process.

-

Virtual Machines

-

To Boot

-
    -
  1. Host OS loads guest bootloader from virtual disk into memory and starts running it.
  2. -
  3. Guest bootloader loads guest kernel into memory and starts running it.
  4. -
  5. Guest kernel initializes IVT to point to its own handlers.
  6. -
  7. Guest kernel loads process from virtual disk into its own emulated memory.
  8. -
  9. When the guest kernel starts the process, it issues instructions to resume execution at user level (ie iret on x86). This traps into the host kernel.
  10. -
  11. Host kernel validates the request and then simulates the requested mode transfer exactly as if the processor had directly executed it.
  12. -
-

User Level System Call

-
    -
  1. Host kernel saves registers onto interrupt stack of guest operating system.
  2. -
  3. Host kernel transfers control to the guest kernel at start of interrupt handler with guest kernel running in user-mode.
  4. -
  5. Guest kernel performs system call, saving user state and checking arguments.
  6. -
  7. When guest kernel returns from syscall, triggers processor exception, dropping into host kernel.
  8. -
  9. Host kernel restores user state as if guest OS had returned directly.
  10. -
-

Processor Exceptions

-

Handles them similarly to user level system calls, but the host kernel tracks what privilege level the guest kernel is running at, and delegates exception handling to the guest kernel if it is running in kernel mode.

-

Timer Interrupts

-

Host kernel returns from the interrupt to the interrupt handler for the guest kernel. The guest kernel may in turn switch guest processes; its iret will cause a processor exception, returning to the host kernel, which can then resume the correct guest process.

-

I/O Interrupts

-

Simulation of virtual devices doesn't need to be anything like a real device. For example, when the guest OS writes to a virtual disk (ie writing to the buffer descriptor ring for the device), the host OS can read these and perform the actual write to the real disk however it wants. The guest kernel then receives an interrupt when the write is complete, which is handled similarly to a timer interrupt, but executes the guest disk interrupt handler instead of the guest timer interrupt handler.

-

Exercises

-
    -
  1. -

    Kernel Stack and User Process Interruption

    -
  2. -
  3. -

    When a user process is interrupted or causes a processor exception, the x86 hardware switches the stack pointer to a kernel stack. This is done before saving the current process state to ensure that sensitive kernel data is not overwritten or corrupted by user-level processes.

    -
  4. -
  5. -

    Screen Buffer Memory Protection

    -
  6. -
  7. -

    The screen's buffer memory must be protected because if a malicious application could alter any pixel on the screen, it could potentially display misleading information, capture sensitive data, or cause system instability. Protecting the screen buffer ensures the integrity and security of the displayed content.

    -
  8. -
  9. -

    Dual-mode Operation Mechanisms

    -
  10. -
  11. -

    Without privileged instructions: User processes might be able to execute sensitive operations.

    -
  12. -
  13. Without memory protection: User processes could access and modify kernel memory directly.
  14. -
  15. -

    Without timer interrupts: User processes could monopolize the CPU, leading to a denial-of-service situation.

    -
  16. -
  17. -

    Security Checks for Web Browser

    -
  18. -
  19. -

    To ensure executing buggy or malicious scripts cannot corrupt or crash the browser, checks like script sandboxing, code validation, resource limitations, and privilege separation should be implemented.

    -
  20. -
  21. -

    Types of User-Mode to Kernel-Mode Transfers

    -
  22. -
  23. -

    Software Interrupts

    -
  24. -
  25. System Calls
  26. -
  27. -

    Exceptions

    -
  28. -
  29. -

    Types of Kernel-Mode to User-Mode Transfers

    -
  30. -
  31. -

    Returning from an interrupt

    -
  32. -
  33. Returning from a system call
  34. -
  35. Resuming execution after an exception
  36. -
  37. -

    Context switching

    -
  38. -
  39. -

    IRET Instruction in OS -a. The iret instruction is used in the interrupt service routine (ISR) to return from an interrupt, transitioning the mode from kernel-mode back to user-mode. -b. If an application program executes iret, it could potentially disrupt the operating system's internal state, leading to unpredictable behavior or crashes.

    -
  40. -
  41. -

    Large Number of Registers Design -a. Having a large number of registers can reduce the need for frequent memory accesses, improving performance. -b. Hardware features like register renaming, out-of-order execution, and branch prediction can be beneficial. -c. Adding a 16-stage pipeline and precise exceptions might increase the overhead of user-kernel switching due to increased complexity and potential stalls.

    -
  42. -
  43. -

    Virtualization and x86 Architecture -a. Instructions like popf prevent transparent virtualization because they have different behaviors in privileged and unprivileged modes, making it challenging to virtualize without affecting guest OS behavior. -b. Modifying the hardware to provide consistent behavior for instructions like popf in both modes can resolve the virtualization issue.

    -
  44. -
  45. -

    Initial Value in Program Counter

    -
  46. -
  47. -

    The boot ROM is responsible for loading the initial value in the program counter for an application program before it starts running.

    -
  48. -
  49. -

    Safe Access to Virtualized I/O Devices

    -
  50. -
  51. -

    To enable safe access to virtualized I/O devices, hardware support like I/O MMU, device assignment, and software support like device emulation, device drivers, and hypervisor integration are essential.

    -
  52. -
  53. -

    System Calls vs. Procedure Calls

    -
  54. -
  55. -

    System calls are generally more expensive than procedure calls due to the overhead of mode switching, trap handling, and potential context switches. A test program can be designed to measure and compare the time required for each type of call.

    -
  56. -
  57. -

    Substitute for Traps

    -
  58. -
  59. -

    Without a trap instruction, a combination of interrupts and exceptions can be used as a substitute for traps, allowing the operating system to handle events and transitions effectively.

    -
  60. -
  61. -

    Substitute for Interrupts

    -
  62. -
  63. -

    Without interrupts, a combination of exceptions and traps can be utilized to handle events and transitions within the operating system, although it may introduce additional complexity and overhead.

    -
  64. -
  65. -

    Steps for CPU Interrupt Handling

    -
  66. -
  67. -

    The operating system typically follows steps like saving the current state, identifying the interrupt source, invoking the appropriate interrupt handler, performing the required processing, and restoring the saved state.

    -
  68. -
  69. -

    Operating System Stack for System Calls

    -
  70. -
  71. -

    The operating system stack for handling system calls should be separate from the application stack to ensure isolation and prevent potential stack-related vulnerabilities or conflicts.

    -
  72. -
  73. -

    Verifying Rogue System Calls

    -
  74. -
  75. Writing a program to test the operating system's protection against rogue system calls involves trying various illegal calls and observing the system's response to ensure it handles them correctly.
  76. -
-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v1-kernels-and-processes/3-the-programming-interface.html b/site/operating-systems/v1-kernels-and-processes/3-the-programming-interface.html deleted file mode 100644 index d1fee40..0000000 --- a/site/operating-systems/v1-kernels-and-processes/3-the-programming-interface.html +++ /dev/null @@ -1,688 +0,0 @@ - - - - - - 3 The Programming Interface - - - - - -
- -

3 The Programming Interface

-
- Last modified: 2024-01-14 - -
-
-

Chapter 3 - The Programming Interface

-

Syscall API Reference

-

Creating and managing processes

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FunctionDescription
fork()Create a child process as a clone of the current process. The fork call returns to both the parent and child.
exec(prog, args)Run the application prog in the current process.
exit()Tell the kernel the current process is complete, and its data structures should be garbage collected.
wait(processID)Pause until the child process has exited.
signal(processID, type)Send an interrupt of a specified type to a process.
-

I/O operations

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FunctionDescription
fileDesc open(name)Open a file, channel, or hardware device, specified by name; returns a file descriptor that can be used by other calls.
pipe(fileDesc[2])Create a one-directional pipe for communication between two processes. pipe returns two file descriptors, one for reading and one for writing.
dup2(fromFileDesc, toFileDesc)Replace the toFileDesc file descriptor with a copy of fromFileDesc. Used for replacing stdin or stdout or both in a child process before calling exec.
int read(fileDesc, buffer, size)Read up to size bytes into buffer, from the file, channel, or device. read returns the number of bytes actually read.
int write(fileDesc, buffer, size)Analogous to read, write up to size bytes into kernel output buffer for a file, channel, or device. write normally returns immediately but may stall if there is no space in the kernel buffer.
fileDesc select(fileDesc[], arraySize)Return when any of the file descriptors in the array fileDesc[] have data available to be read. Returns the file descriptor that has data pending.
close(fileDescriptor)Tell the kernel the process is done with this file, channel, or device.
-

Overview

-

What features/functions do we need an OS to provide for applications?

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FunctionDescription
Process managementCreate, destroy, and manage processes. This includes the ability to create new processes, terminate existing processes, wait for processes to complete, and send asynchronous notifications to processes.
Input/OutputCommunicate with devices and files, as well as other processes.
Thread managementCreate, manage and destroy threads, aka tasks that share memory and other resources within a process.
Memory managementAllocate and deallocate memory for processes.
File managementCreate, delete, and manipulate files and directories. Users should be able to persist named data on disk.
Networking and Distributed SystemsProcesses should be able to communicate with other processes on different machines over the network. Processes should also be able to coordinate their actions, despite faults and delays.
Graphics/Window ManagementProcesses control pixels on their portion of the screen. Should utilize hardware acceleration to draw graphics quickly.
Authentication and SecurityPermissions system to control access to resources. Processes should be able to authenticate themselves to other processes and to the OS.
-

This chapter will focus on the first two.

-

Design Choices

-

For any bit of functionality, there are several possible places it could be implemented in an OS.

- - - - - - - - - - - - - - - - - - - - - - - - - -
ComponentDescription
User-level programsPrograms for logging in and managing processes in both UNIX and Windows.
User-level libraryUser interface widgets in MacOS and Windows.
Kernel, accessed via system callsFile system and network stack in UNIX and Windows.
Standalone server process invoked by kernelWindow manager in MacOS and Windows.
-

However, UNIX philosophy is to implement as much as possible in user-level programs and hardware, maintaining a "thin waist" in the system architecture. Same design principle behind network stack, the key interface between the highest and lowest levels of the system follows a very simple but powerful design. So long as a program complies to the system call interface, it can run on most UNIX systems.

-

However, some things really do need to go in particular places. We want the following:

- -

Process Management

-

Windows Process Management

-

Has a syscall to create a process, and others for various process management operations. Turns out to be simple in theory, but complicated in practice. In an ideal world, it would be as simple as:

-
boolean CreateProcess(char* prog, char* args);
-
- -

However, the parent process may want to control various aspects of the child process runtime, so instead we have:

-
 if (!CreateProcess(NULL, // No module name (use command line)
-     argv[1], // Command line
-     NULL, // Process handle not inheritable
-     NULL, // Thread handle not inheritable
-     FALSE, // Set handle inheritance to FALSE
-     0, // No creation flags
-     NULL, // Use parent's environment block
-     NULL, // Use parent's starting directory
-     &si, // Pointer to STARTUPINFO structure
-     &pi ) // Pointer to PROCESS_INFORMATION structure
- )
-
-

UNIX Process Management

-

UNIX takes an approach that is complex in theory, but simple in practice. CreateProcess is replaced by two system calls:

-
pid_t fork(void);
-int exec(char* prog, char* args);
-
-

fork creates a copy of the calling process, called the child process. It is almost identical to the parent, except for the return value of fork and the pid. The child process has a new pid, and the return value of fork is 0. The parent process gets the pid of the child process as the return value of fork. The child process sets itself up the the same as the parent, and once the context is set, the child process calls exec to replace itself with a new program. exec loads the program into the address space, and starts it running.

-

fork

- -
Browsers and fork
-

Browsers use new processes to create a new tab. When you click on a link, Chrome forks a process to fetch and render the web page at the link in a new tab. This also gives the browser seperation between tabs, so that if one tab crashes or contains anything harmful, the others are unaffected. Interestingly, Chrome on Windows doesn't even use CreateProcess for new tabs, and relies on a pool of pre-created processes.

-

exec

- -

Note that exec does not create a new process!

-

wait

-

Parent process can wait for child process to complete with wait(pid). This is a blocking call, and returns the exit status of the child process. If the child process has not yet exited, the parent process is blocked until it does. If the child process has already exited, the parent process is not blocked.

-

wait is optional in UNIX, but a little ambiguous in terminology and use. Windows arguably did it better with WaitForSingleObject.

-

Kernel Handles and Garbage Collection

-

UNIX processes call exit to terminate themselves. This releasing various resources, including user stack, heap, code segments. Has to be careful with PCB though. Even if the child exits, the parent still has access to the PCB and can still call wait. Thus, PCB can't be GC's until both parent and child have exited.

-

In general, both the Windows and UNIX kernels have various syscalls that return a handle to a kernel object (e.g. pid of a process, fd of an I/O device, etc.). Thense are not pointers, and are instead specific to a given process, and should be validated as such. The kernel maintains a reference count (important) for each object, and when the reference count reaches 0, the kernel can GC the object.

-

Signals

-

Sending async notifications to processes done with signal.

-
typedef void (*sig_t) (int);
-sig_t signal(int sig, sig_t func);
-
-

You can register a handler function for a given signal, and when the signal is sent to the process, the handler function is called. This is how ctrl-c works to stop a shell. signal returns the previous handler function, so you can chain them together. sigaction is a more modern version of signal that is more flexible.

-
int sigaction(int sig,
-              const struct sigaction *restrict act,
-              struct sigaction *restrict oact);
-
-

Input/Output

-

The key ideas behind UNIX I/O are:

- -

Inter-Process Communication

-

Pipes

-

A UNIX pipe is a kernel buffer with two file descriptors, one for reading and the other for writing. Data is read from the pipe in the same order as it is written, but the buffer allows the decoupling of the consumer and producer processes. The pipe terminates when the last process closes the file descriptors or exits. TCP sockets are similar to pipes.

-

Replace file descriptors

-

dup2(from, to) replaces the file descriptor to with from in a child process. This is useful for redirecting stdin/stdout/stderr to files or pipes.

-

Wait for multiple reads

-

In a client/server context, a server might have a pipe open to multiple clients, and want to read from whichever client has data available. select allows the server to wait for data to be available on any of the pipes, and then read from the pipe that has data available. This is a blocking call, and returns when data is available on one of the pipes.

-
int select(int nfds, fd_set *readfds, fd_set *writefds,
-           fd_set *exceptfds, struct timeval *timeout);
-
-

select is a bit of a pain to use, so poll was introduced as a more convenient alternative for synchronous I/O multiplexing.

-
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
-
-

Case Study: The UNIX Shell

-
//                    [!!!] pseudocode [!!!]
-main() {
-    char *prog = NULL;
-    char **args = NULL;
-    // Read the input a line at a time, and parse each line into the program
-    // name and its arguments. End loop if we've reached the end of the input.
-    while (readAndParseCmdLine(&prog, &args)) {
-        // Create a child process to run the command.
-        int child_pid = fork();
-        if (child_pid == 0) {
-            // I'm the child process.
-            // Run program with the parent's input and output.
-            exec(prog, args);
-            // NOT REACHED
-        } else {
-            // I'm the parent; wait for the child to complete.
-            wait(child_pid);
-            return 0;
-        }
-    }
-}
-
-

Since commands read and write to file descriptors, the programs are decoupled from their input and output. This allows for...

- -

Case Study: Inter-Process Communication

-

Producer/Consumer

-

-    Producer                            Consumer
-    --------                            --------
-       |                                   | ^
-       | write                        read | |
-       |                                   | |
-       |                                   v |
-       |
-       |                                    ^
-  _____|____________________________________|_________________
-KERNEL |                                    |
-       |                                    |
-       |       _____________                |
-       +-------|||||||||||||----------------+
-               _____________
-             Pipe/Kernel Buffer
-
-

When the producer writes to the pipe (kernel buffer), they can do so entirely decoupled from the consumer. If the buffer is ever full, then writes are blocked by the producer until the consumer reads some data from the buffer. Similarly, if the buffer is ever empty, then reads are blocked by the consumer until the producer writes some data to the buffer.

-

WIn UNIX, when the producer is done it usually closes its side of the buffer. The consumer can then read until the buffer is empty, at which point the read syscall will hit EOF and return 0. The consumer can then close its side of the buffer, and the kernel will GC the buffer.

-

Client/Server

-
         Client                              Server
-       ----------                          ----------
-       |      ^ |                             |     ^ |
-     write    | |                             |     | |
-       |      | |                       write |     | |
-       |      | v read                        |     | v read
-User   |       ^                              |      ^
-_______|_______|______________________________|______|__________
-Kernel |       |                              |      |
-       |       |                              |      |
-       |       |       _______________        |      |
-       |       +-------||||||||||||||| <------+      |
-       |               ---------------               |
-       |                                             |
-       |               _______________               |
-       +-------------->|||||||||||||||---------------+
-                       ---------------
-                       Pipe/Kernel Buffer
-
-

In client/server, there are two pipes, one for each direction of communication. To make a request, the client writes data into one pipe, and then reads data from the other. The server does the opposite, reading from the first pipe, validating and handling the request, and then writing to the second pipe the response.

-
//                   [!!!] pseudocode [!!!]
-
-Client:
-    char request[RequestSize];
-    char reply[ReplySize];
-
-    // ..compute..
-
-    // Put the request into the buffer.
-    // Send the buffer to the server.
-    write(output, request, RequestSize);
-
-    // Wait for response.
-    read(input, reply, ReplySize);
-
-    // ..compute..
-
-Server:
-    char request[RequestSize];
-    char reply[ReplySize];
-
-    // Loop waiting for requests.
-    while (1) {
-        // Read incoming command.
-        read(input, request, RequestSize);
-
-        // Do operation.
-
-        // Send result.
-        write(output, reply, ReplySize);
-    }
-
-

Streamlining Client/Server Communication

-

Since both issue a write followed by a read, one could combine these into a single system call (at the expense of adding a system call...) in order to eliminate a context switch.

-

Furthermore, the client always needs to wait for the server, so an even further optimization that was done in microkernel Windows in the early 1990s is to donate the client processor to run server code, reducing latency. However, this requires that code and data for both client and server are in cache simultaneously.

-

Even further, on a multi-core system where the client and server have their own processors, the kernel can set up a shared memory region between them so that they can (safely) communicate directly without involving the kernel at all.

-

Often, the server process needs to select one of many processes to accept a request from (ie in a print queue). This can be done with the select system call.

-
Server:
-    char request[RequestSize];
-    char reply[ReplySize];
-    FileDescriptor clientInput[NumClients];
-    FileDescriptor clientOutput[NumClients];
-    // Loop waiting for a request from any client.
-    while (fd = select(clientInput, NumClients)) {
-        // Read incoming command from a specific client.
-        read(clientInput[fd], request, RequestSize);
-        // Do operation.
-        // Send result.
-        write(clientOutput[fd], reply, ReplySize);
-    }
-
-

Operating System Structures

- -

There is a fundamental tradeoff between maintainability and performance when it comes to designing kernels. Keeping functionality tightly coupled and integrated with kernel code makes it more performant, but also messy.

-

Monolothic Kernels

-

Monolith kernels usually have everything tightly coupled within the kernel itself. Modules often have dependencies that span multiple other modules within the kernel. Not ALL functionality is built directly into the kernel, but much of it is.

-

Since OS designers are free to structure their code however they want with a monolith, there is a lot of variation between systems. However, two emerging patterns are usually present:

-

Hardware Abstraction Layer (HAL)

-

Portable interface to machine configuration and processor specific operations.

-

For an OS to be portable between processor families (ex ARM -> Intel or 32 -> 64 bit), there needs to be a layer over the processor specific code that handles things like context switches, interrupts, exceptions, and traps.

-

All of these hardware specific instructions need to be mapped to the platform independent "virtual" procedure. Porting an OS to a new platform is really just a matter of implementing this layer for a new architecture/hardwarep

-
Windows HAL
-

Windows uses two-pronged strategy for portability. Kernel is dynamically linked at boot time with a set of libaray routines that are specific to the hardware configuration. Also runs a different kernel binary accross different processor architectures, each of which contains conditional execution for closely related processor designs.

-

Dynamically Installed Device Drivers

-

Similar considerations for supporting wide variety of IO devices. Dynamically loadable device drivers are provided as the kernel is already running in order to handle new devices. Device manufacturers write drivers that follow a stardard interface for the OS, and these routines are called by the kernel when the device needs to be used.

-

When the OS boots, there are a small number of drivers that are already loaded, e.g. disk drivers. All devices physically attached to the computer have corresponding drivers usually bundled into a file that is stored along with the boot loader. When the OS starts up, it queries the I/O bus to find out what devices are attached, and then loads the corresponding drivers from disk. Any network-attached devices (like network printers) are loaded from over the internet.

-

Drivers have been found to be responsible for ~90% of OS crashes, and are a potential source of corruption, as well as a security risk. Mitigated with...

- -

Microkernels

-

Run as much of the OS as possible in user mode. The window manager on most modern OS's is a good example of this.

-

The difference between micro and monolithic kernels is often transparent to application programs. User level libraries can either directly make requests to the server process, or make system calls that are then redirected by the kernel.

-

Generally, microkernels provide little benefit beyond the ease of development. The performance cost of context switching between user and kernel mode is high enough that most modern systems take on a hybrid approach.

-

Exercises

-
    -
  1. -

    Can UNIX fork return an error? Why or why not?

    -
  2. -
  3. -

    Can UNIX exec return an error? Why or why not?

    -
  4. -
  5. -

    What happens if we run the following program on UNIX?

    -
  6. -
-

c - main() { - while (fork() >= 0) - ; - }

-
    -
  1. -

    Explain what must happen for UNIX wait to return immediately (and successfully).

    -
  2. -
  3. -

    Suppose you were the instructor of a very large introductory programming class. Explain (in English) how you would use UNIX system calls to automate testing of submitted homework assignments.

    -
  4. -
  5. -

    What happens if you run "exec csh" in a UNIX shell? Why?

    -
  6. -
  7. -

    What happens if you run "exec ls" in a UNIX shell? Why?

    -
  8. -
  9. -

    How many processes are created if the following program is run?

    -
  10. -
-

c - main(int argc, char ** argv) { - forkthem(5); - } - void forkthem(int n) { - if (n > 0) { - fork(); - forkthem(n-1); - } - } -9. Consider the following program:

-

```c - main (int argc, char ** argv) { - int child = fork(); - int x = 5;

-
   if (child == 0) {
-       x += 5;
-   } else {
-       child = fork();
-       x += 10;
-       if(child) {
-           x += 5;
-       }
-   }
-
-

} - ```

-

How many different copies of the variable x are there? What are their values when their process finishes? -10. What is the output of the following programs? (Please try to solve the problem without compiling and running the programs.)

-
- Program 1:
-
-  ```c
-  main() {
-      int val = 5;
-      int pid;
-
-      if (pid = fork())
-          wait(pid);
-      val++;
-      printf("%d\n", val);
-      return val;
-  }
-  ```
-
-- Program 2:
-  ```c
-  main() {
-      int val = 5;
-      int pid;
-      if (pid = fork())
-          wait(pid);
-      else
-          exit(val);
-      val++;
-      printf("%d\n", val);
-      return val;
-  }
-  ```
-
-
    -
  1. Implement a simple Linux shell in C capable of executing a sequence of programs that communicate through a pipe. For example, if the user types ls | wc, your program should fork off the two programs, which together will calculate the number of files in the directory. For this, you will need to use several of the Linux system calls described in this chapter: fork, exec, open, close, pipe, dup2, and wait. Note: You will to replace stdin and stdout in the child process with the pipe file descriptors; that is the role of dup2.
  2. -
  3. Extend the shell implemented above to support foreground and background tasks, as well as job control: suspend, resume, and kill.
  4. -
-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/4-concurrency-and-threads.html b/site/operating-systems/v2-concurrency/4-concurrency-and-threads.html deleted file mode 100644 index 40974f1..0000000 --- a/site/operating-systems/v2-concurrency/4-concurrency-and-threads.html +++ /dev/null @@ -1,586 +0,0 @@ - - - - - - 4 Concurrency And Threads - - - - - -
- -

4 Concurrency And Threads

-
- Last modified: 2024-02-18 - -
-
-

Chapter 4: Concurrency and Threads

-

4.1 Thread Use Cases

- -

Processors are much faster than I/O devices. Latency for disk I/O is measured in milliseconds, which is enough for a CPU to execute millions of instructions. Furthermore, unpredictable latency with I/O like user input or network requests make it nessary to have non-blocking I/O.

-

A common pattern in I/O bound applications is to have multiple threads fetching different quality or levels of resources simultaneously. For example, a media player might have one thread fetching the highest quality video, another fetching the highest quality audio, and a third fetching the lowest quality video for previewing.

-

Threads vs. Processes

-
Some scenarios:
- -

4.2 Thread Abstraction

- -

4.2.1 Running, Suspending, and Resuming Threads

-

Threads provide the illusion of infinite processors. OS uses a thread scheduler to switch between threads to run. How threads are interleaved and scheduled should be transparent to the application.

-

Cooperative vs. preemptive multithreading

-

Early versions of MacOS used cooperative multithreading, where the kernel would only switch threads when the running thread made a system call relinquishing control. This is a bad idea because a thread can hog the CPU and starve other threads.

-

Modern operating systems use preemptive multithreading, where the kernel can preempt a thread at any time, even in the middle of a single instruction.

-

4.2.2 Why "Unpredictable Speed"?

-

Always avoid reasoning about the relative speed of threads when trying to evaluate the correctness of your code. The speed of threads is unpredictable for many reasons.

-

4.3 POSIX Thread API

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Function SignatureDescription
pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg)Create a new thread.
pthread_join(pthread_t thread, void **retval)Wait for a thread to finish.
pthread_detach(pthread_t thread)Detach a thread.
pthread_self()Get the current thread.
pthread_exit(void *retval)Terminate the current thread.
pthread_cancel(pthread_t thread)Cancel a thread.
-

Threads enable asynchronous procedure calls, which means the function called runs in the background

-
 #include <stdio.h>
- #include "thread.h"
- static void go(int n);
- #define NTHREADS 10
- static thread_t threads[NTHREADS];
-
- int main(int argc, char **argv) {
-    int i;
-    long exitValue;
-    for (i = 0; i < NTHREADS; i++){
-       pthread_create(&(threads[i]), &go, i);
-    }
-
-    for (i = 0; i < NTHREADS; i++){
-        exitValue = pthread_join(threads[i]);
-        printf("Thread %d returned with %ld\n",
-        i, exitValue);
-    }
-
-    printf("Main thread done.\n");
-    return 0;
- }
-
- void go(int n) {
-    printf("Hello from thread %d\n", n);
-    pthread_exit(100 + n);
-    // Not reached
- }
-
-

Fork-Join Parallelism

-

Example program to zero a block of memory using multiple threads. In operating systems, often times need to zero a block of memory (like after a process exits to prevent leaking information). This is a good example of a task that can be parallelized.

-

For reference, zeroing 1 GB of memory takes about 50 ms on modern hardware. The overhead of creating a thread however is on the order of 10 microseconds, so it can be worth it to parallelize.

-
 // To pass two arguments, we need a struct to hold them.
- typedef struct params {
-    unsigned char *buffer;
-    int length;
- };
-
- #define NTHREADS 10
- void go (struct params *p) {
-    memset(p->buffer, 0, p->length);
- }
- // Zero a block of memory using multiple threads.
- void blockzero (unsigned char *p, int length) {
-    int i;
-    thread_t threads[NTHREADS];
-    struct params params[NTHREADS];
-
-    // For simplicity, assumes length is divisible by NTHREADS.
-    assert((length % NTHREADS) == 0);
-
-    for (i = 0; i < NTHREADS; i++) {
-        params[i].buffer = p + i * length/NTHREADS;
-        params[i].length = length/NTHREADS;
-        thread_create_p(&(threads[i]), &go, &params[i]);
-    }
-
-    for (i = 0; i < NTHREADS; i++)
-        thread_join(threads[i]);
- }
-
-

You can also lazily zero out blocks of memory with a background thread that zeros out memory while another process runs. Then, if you need to reuse the memory, you can just call join on the background thread.

-

Thread Data Structes and Life Cycle

-

Important to differentiate between shared and individual state of a thread. Shared state consists of code, global variables, and heap-allocated memory. Individual state consists of the thread's stack, registers, and metadata, all of which is stored in the thread control block (TCB) in each thread.

-

Thread Control Block

-

For every thread the OS creates, it creates a TCB to store the thread's individual state. It must store both the state of the computation, and the metadata needed to manage the thread.

-

Per-thread Computation State

-

The thread needs a pointer to the top of its stack, which works the same as a single threaded program's stack (ie one frame per function call). Each frame contains the local variables, parameters, and the return address to jump back to when the function returns. When a new thread is created, the OS allocates a new stack for it.

-

Additionally, needs to store the processor registers. Some systems just put them on the top of the thread's stack, but others have dedicated space in the TCB.

-

How big of a stack?

-

Kernel stacks allocated in physical memory, so good to keep small. The max procedure call nesting in kernel code is usually small, so the kernel stacks are usually small. This works solely because of the convention to allocate all large data structures on the heap. Small stacks can cause problems if you allocate large structures locally.

-

User level stacks typically allocated in virtual memory, so less constrained. However, multithreaded programs can't grow their stacks indefinitely (except in languages like Go where stacks are automatically grown). Very easy to overflow the stack in multithreaded programs, but POSIX let's you configure stack size. Most implementations try to detect stack overflow with known values at the top and bottom of the stack, but this is not foolproof.

-

Per-thread Metadata

-

Things like thread id, scheduling priority, status, etc.

-

Also includes thread local variables, which are similar to global variables in that they span multiple function calls. However, they are private to each thread, and are stored in the TCB. For example, errorno is a macro that expands to a thread local variable holding the error code of the last system call. Additionally, many details of the heap allocator are stored in thread local variables to make parallel allocation in the heap easier (ie subdivide the heap into regions, and each thread has a region).

-

Shared State

-

Program code, statically allocated global variables, dynamicallty allocated heap variables. Note that the kernel doesn't enforce any protection between threads for per-thread state, so it is important know which variables are designed to be shared across threads (global variables, objects on the heap) and which are designed to be private (local/automatic variables).

-

Thread Life Cycle

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StateLocation of TCBLocation of registers
INITBeing created (stack)TCB
READYReady listTCB
RUNNINGRunning listCPU
WAITINGSynchronization variable's waiting listTCB
FINISHEDFinished list then deletedTCB or deleted
-

INIT

- -

READY

- -

RUNNING

- -

WAITING

- -

FINISHED

- -

The idle thread

-

If a system has k processors, it will ensure that there are exactly k threads in the RUNNING state at all times. If there is nothing to run for a given CPU, the idle thread is run instead. On modern systems, the idle thread is just a loop that calls hlt to put the CPU into a low power state that doesn't execute instructions until an interrupt occurs and the processor is woken up.

-

The low-power mode is especially nice for virtualization, because the host operating system can allocate those resources to a different VM while that VM is idle.

-

Where is the TCB stored?

-

On a multiprocessor system, this is a non-trivial implementation issue. x86 has hardware support for fetching the ID of the current processor. In this case, the TCB could be stored in a global array, where the ith entry is the TCB for the thread running on the ith processor.

-

For systems without hardware support, you can use the stack pointer (which is always unique to each thread). You can store a pointer to the TCP at the bottom of the thread's stack under the procedure frames, and then use the stack pointer to find the TCB.

-

Implementing Kernel Threads

-
 // func is a pointer to a procedure the thread will run.
- // arg is the argument to be passed to that procedure.
- void thread_create(thread_t *thread, void (*func)(int), int arg) {
-   // Allocate TCB and stack
-   TCB *tcb = new TCB();
-   thread->tcb = tcb;
-   tcb->stack_size = INITIAL_STACK_SIZE;
-   tcb->stack = new Stack(INITIAL_STACK_SIZE);
-   // Initialize registers so that when thread is resumed, it will start running at
-   // stub. The stack starts at the top of the allocated region and grows down.
-   tcb->sp = tcb->stack + INITIAL_STACK_SIZE;
-   tcb->pc = stub;
-   // Create a stack frame by pushing stub's arguments and start address
-   // onto the stack: func, arg
-   *(tcb->sp) = arg;
-   tcb->sp--;
-   *(tcb->sp) = func;
-   tcb->sp--;
-   // Create another stack frame so that thread_switch works correctly.
-   // This routine is explained later in the chapter.
-   thread_dummySwitchFrame(tcb);
-   tcb->state = READY;
-   readyList.add(tcb); // Put tcb on ready list
- }
-
- void stub(void (*func)(int), int arg) {
-   (*func)(arg); // Execute the function func()
-   thread_exit(0); // If func() does not call exit, call it here.
- }
-
-

Creating a thread should run the code within func asynchronously with the calling thread. To create a thread, you must:

-
    -
  1. Allocate per-thread state. Allocate space for the new thread's TCB and stack.
  2. -
  3. Initialize per-thread state. Initialize TCB by setting machine specific registers to what is needed for RUNNING state. Must set up func to return to a stub that also calls thread_exit.
  4. -
  5. Put TCB on running list. Set state to READY and put on running list, enabling it to be scheduled.
  6. -
-

Deleting a thread

-

When a thread calls thread_exit, must remove it from ready lists so it stops being scheduled, and then free the per-thread state.

-

NOTE: a thread cannot free its own resources because if interrupted, it would be a memory leak (since it will never be scheduled again to finish cleanup). Instead, thread changes its state to FINISHED, and then puts itself on the finished list for some other thread to clean it up,

-

Thread Context Switch

-

When a thread is moved from RUNNING to READY, the OS must save the thread's register values to its TCB, and then load the register values of the next thread to run from its TCB.

-

Note that interrupts must be disabled during a context switch (OSSP 47). This is because the if a low priority thread voluntarily yields to a high priority thread, but then gets stuck in the ready list, the high priority thread will be stuck waiting.

-
// We enter as oldThread, but we return as newThread.
- // Returns with newThread's registers and stack.
- void thread_switch(oldThreadTCB, newThreadTCB) {
-   pushad; // Push general register values onto the old stack.
-   oldThreadTCB->sp = %esp; // Save the old thread's stack pointer.
-   %esp = newThreadTCB->sp; // Switch to the new stack.
-   popad; // Pop register values from the new stack.
-   return;
- }
- void thread_yield() {
-   TCB *chosenTCB, *finishedTCB;
-   // Prevent an interrupt from stopping us in the middle of a switch.
-   disableInterrupts();
-   // Choose another TCB from the ready list.
-   chosenTCB = readyList.getNextThread();
-   if (chosenTCB == NULL) {
-      // Nothing else to run, so go back to running the original thread.
-   } else {
-      // Move running thread onto the ready list.
-      runningThread->state = ready;
-      readyList.add(runningThread);
-      thread_switch(runningThread, chosenTCB); // Switch to the new thread.
-      runningThread->state = running;
-   }
-   // Delete any threads on the finished list.
-   while ((finishedTCB = finishedList->getNextThread()) != NULL) {
-      delete finishedTCB->stack;
-      delete finishedTCB;
-   }
-   enableInterrupts();
- }
- // thread_create must put a dummy frame at the top of its stack:
- // the return PC and space for pushad to have stored a copy of the registers.
- // This way, when someone switches to a newly created thread,
- // the last two lines of thread_switch work correctly.
- void thread_dummySwitchFrame(newThread) {
-   *(tcb->sp) = stub; // Return to the beginning of stub.
-   tcb->sp--;
-   tcb->sp -= SizeOfPopad;
- }
-
-

Seperating mechanism from policy

-

It is useful to seperate the mechanics of performing an action from the rules for deciding when to perform that action. For example, the thread_switch function is a mechanism for switching threads, but the policy for deciding when to switch threads is in the scheduler. This allows different systems to take their own approach to scheduling.

-

Another example of this is with virtual memory. The mechanism for translating virtual addresses to physical addresses is in the MMU, but the policy for deciding which pages to keep in memory is in the page replacement algorithm.

- -

Combining Kernel Threads and Single-Threaded User Processes

-

Switching between kernel threads and kernel handlers

- -

Implementing Multi-threaded Processes

-

Multithreaded Processes with Kernel Threads

-

A thread in a process has a user level stack, a kernel interrupt stack, and a kernel TCB. To create a new thread from the user's perspective, the process calls a library function that allocates a user-level stack and then makes a system call to create a new kernel thread and then returns a thread id. The kernel allocates a TCB and kernel stack, and then places the thread on the ready list.

-

Thread join, exit, and yield are all system calls that the kernel handles in a similar way, by manipulating the TCBs and the ready list.

-

User-Level Threads without Kernel Threads

-

Added to OS's to support concurrency without any kernel support. Early versions of the JVM used this approach with green threads. The running process essentially implements all of the kernel data structures and scheduling policies in user space, allowing threading operations to be simple procedure calls instead of system calls.

-

A limitation is the lack of awareness of the OS's scheduling policies, and the inability to take advantage of multiple processors. For instance, if the process is blocked on I/O, all threads are blocked.

-

Preemptive User-Level Threads

-

Similar to upcalls or signals, but for user-level threads. To preempt some process P:

-
    -
  1. The user-level thread library makes a system call to register a timer signal handler and signal stack with the kernel.
  2. -
  3. When a hardware timer interrupt occurs, the hardware saves P's register state and runs the kernel's handler.
  4. -
  5. Instead of restoring P's register state and resuming P where it was interrupted, the kernel's handler copies P's saved registers onto P 's signal stack.
  6. -
  7. The kernel resumes execution in P at the registered signal handler on the signal stack.
  8. -
  9. The signal handler copies the processor state of the preempted user-level thread from the signal stack to that thread's TCB.
  10. -
  11. The signal handler chooses the next thread to run, re-enables the signal handler (the equivalent of re-enabling interrupts), and restores the new thread's state from its TCB into the processor. execution with the state (newly) stored on the signal stack.
  12. -
-

User-Level Threads with Kernel Support

-

Process using $M$ kernel thrreads, each with their own user-level scheduler that schedules $N$ user-level threads. The kernel threads are scheduled by the OS, and the user-level threads are scheduled by the user-level scheduler.

-

However, there is still an issue of one of your kernel threads being blocked for the entirety of an I/O operation. Solution:

-

Scheduler Activations

-

Let the user and kernel level schedulers coordinate with eachother. Involves system call and upcalls (2-way communication), and thus overhead, but doesnt require the kernel to be aware of the user-level threads, and is able to optimize the uncommon case.

- -

Scheduler Activations replace kernel threads. It has a seperate stack and CPU context, and it can be scheduled the same as a kernel thread. However, if the kernel interrups it, processor restarts execution in the user-level scheduler. Then the user-level scheduler can decide which user-level thread to run next.

-

Alternatives to Threads

-

Asynchronous I/O and Event-Driven Programming

-

Instead of using threads to manage I/O, you can use asynchronous I/O. This is especially useful for I/O bound applications, where the CPU is idle while waiting for I/O to complete. Instead of blocking on I/O, the program can continue to run and be notified when the I/O is complete.

- -

A common design pattern is to have a single thread interleave different I/O bound tasks by waiting for multiple different I/O events. For example, a web server that has 10 active clients might have a single thread that issues a select call to wait for any of the 10 clients to have data ready to read. Then, when the select call returns, the thread can read from the client and it will immediately (non-blocking).

-

Event-Driven Programming

-

A natural extension of this pattern is to be able to handle requests that involve a sequence of I/O operations. For instance, handling a web request can involve (1) accepting a connection, (2) reading the request, (3) processing the request, (4) locating and reading the requested data on disk or in a database, (5) writing the response back to the client.

-

Event-driven programming using the notion of a continuation, a data structure that keeps track of a task's current state and next step.

-

Event-Driven Programming vs. Threads

-

Very similar in practice. In either case, program blocks until next task can proceed, restores state of the task, executes the next step, and then blocks again. The main difference is whether the state is stored in a continuation and managed by the program, or in a thread and managed by the OS.

-

For example, a web server that reads from multiple clients and stores the data in a table of buffers

-

- Hashtable<Buffer*> *hash;
-
- while(1) {
-   connection = use select() to find a
-                  readable connection ID
-   buffer = hash.remove(connection);
-   got = read(connection, tmpBuf, TMP_SIZE);
-   buffer->append(tmpBuf, got);
-   buffer = hash.put(connection, buffer);
- }
-
- // Thread-per-client
- Buffer *b;
- while(1) {
-   got = read(connection, tmpBuf, TMP_SIZE);
-   buffer->append(tmpBuf, got);
- }
-
-
Performance
-

The common argument for event-driven programming is that it is faster for two reasons:

-
    -
  1. No context switch overhead. Context switches are expensive, and the more threads you have, the more context switches you have. Context switches are also unpredictable, and can cause cache misses and TLB misses.
  2. -
  3. No memory overhead. The thread system itself comes with a non-negligible memory overhead. This is less of a problem with modern systems, but it is still a consideration. For example, allocating 1000 threads with an 8 KB stack size on a machine with 1 GB of memory would require less than 1% of the memory to be allocated to the thread stacks.
  4. -
-

On the other hand, event driven programs by themselves don't take advantage of multiple processors, and in practice are combined with threads. A process with $N$ threads can multiplex tasks seperately using an event-driven model, and then use threads to run those tasks in parallel.

-

Furhtermore, the event-driven model can be more complex and harder to reason about, and if there is a reasonable amount of work to be done in the background, it is often easier to use threads.

-

Data Parallel Programming

-

Can use SIMD instructions to perform the same operation on multiple pieces of data at the same time. For example, the addps instruction in x86 can add 4 single precision floating point numbers at the same time. This is useful for things like image processing, where you can apply the same operation to every pixel in an image.

-

In general, with data parallel programming you specify the operation to be performed on a single piece of data, and then the hardware takes care of applying that operation to multiple pieces of data at the same time.

-

This is useful in a wide variety of areas, and is often a source of major optimizations within programs. For instance, SQL databases can take in a query and then identify which parts of the query can be parallelized, leading to a significant speedup. This is also often used in combination with specialized hardware, like GPUs. Multimedia streaming, for example, uses SIMD instructions to decode and encode video.

-

A large scale example of this is the MapReduce programming model, which is used by Google and Hadoop. The idea is to split a large dataset into smaller pieces, and then apply a function to each piece in parallel. The results are then combined together.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.html b/site/operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.html deleted file mode 100644 index cd3b708..0000000 --- a/site/operating-systems/v2-concurrency/5-synchronizing-access-to-shared-objects.html +++ /dev/null @@ -1,456 +0,0 @@ - - - - - - 5 Synchronizing Access To Shared Objects - - - - - -
- -

5 Synchronizing Access To Shared Objects

-
- Last modified: 2024-02-25 - -
-
-

Synchronizing Access to Shared Objects

-

Locks: Mutual Exclusion

-

Locks enable mutual exclusion with two operations: acquire and release.

- -

Formal Properties

- - - - - - - - - - - - - - - - - - - - - -
PropertyDescription
Mutual ExclusionAt most one process can hold the lock at a time.
ProgressIf no process holds the lock and some process wants to acquire it, then some process will eventually acquire the lock.
Bounded WaitingThere is a bound on the number of times that other processes can acquire the lock after a process has requested to acquire the lock.
-

Case Study: Thread-Safe Bounded Queue

-
// Thread-safe queue interface
-const int MAX = 10;
-class ConcurrentQueue {
-  // Synchronization variables
-  Lock lock;
-  // State variables
-  int items[MAX];
-  int front;
-  int nextEmpty;
-
-  public:
-    ConcurrentQueue();
-    ~ConcurrentQueue(){};
-    bool tryInsert(int item);
-    bool tryRemove(int *item);
-};
-
-// Initialize the queue to empty
-// and the lock to free.
-ConcurrentQueue::ConcurrentQueue() {
-  front = nextEmpty = 0;
-}
-
-// Try to insert an item. If the queue is
-// full, return false; otherwise return true.
-bool ConcurrentQueue::tryInsert(int item) {
-  bool success = false;
-  lock.acquire();
-
-  if ((nextEmpty - front) < MAX) {
-    items[nextEmpty % MAX] = item;
-    nextEmpty++;
-    success = true;
-  }
-
-  lock.release();
-  return success;
-}
-// Try to remove an item. If the queue is
-// empty, return false; otherwise return true.
-bool ConcurrentQueue::tryRemove(int *item) {
-  bool success = false;
-  lock.acquire();
-
-  if (front < nextEmpty) {
-    *item = items[front % MAX];
-    front++;
-    success = true;
-  }
-
-  lock.release();
-  return success;
-}
-
-

And here is a simple test program that uses the queue:

-
int main(int argc, char **argv) {
-  ConcurrentQueue *queues[3];
-  sthread_t workers[3];
-  int i, j;
-  // Start worker threads to insert.
-  for (i = 0; i < 3; i++) {
-    queues[i] = new ConcurrentQueue();
-    thread_create_p(&workers[i],
-    putSome, queues[i]);
-  }
-
-  // Wait for some items to be put.
-  thread_join(workers[0]);
-
-  // Remove 20 items from each queue.
-  for (i = 0; i < 3; i++) {
-    printf("Queue %d:\n", i);
-    testRemoval(&queues[i]);
-  }
-}
-
-// Insert 50 items into a queue.
-void *putSome(void *p) {
-  ConcurrentQueue *queue = (ConcurrentQueue *)p;
-  for (int i = 0; i < 50; i++)
-    queue->tryInsert(i);
-
-  return NULL;
-}
-// Remove 20 items from a queue.
-void testRemoval(ConcurrentQueue *queue) {
-  int item;
-  for (int i = 0; i < 20; j++) {
-    if (queue->tryRemove(&item))
-      printf("Removed %d\n", item);
-    else
-      printf("Nothing there.\n");
-  }
-}
- ```
-
-## Spinlocks
-
-A spinlock is a lock that causes a process trying to acquire it to simply wait in a loop while repeatedly checking if the lock is available. They should be used only when the lock is expected to be held for a short period of time, ie. when the time to acquire the lock is less than the time to context switch. While waiting for the lock, the process continues to run and waste CPU cycles.
-
-### Implementing Multiprocessor Spinlocks
-
-Most processor architectures provide atomic `read-modify-write` instructions that acquires an exclusive copy of a given physical memory location, similar to underlying mechanisms for cache coherence among multiple processors. These instructions are used to implement spinlocks. For example, the `test-and-set` instruction atomically sets a memory location to 1 and returns its previous value. This can be used to implement a lock as follows:
-
-```cpp
-struct spinlock {
-  int held = 0;
-}
-void acquire(lock) {
-  while(test_and_set(&lock->held));
-}
-
-void release(lock) {
-  lock->held = 0;
-}
-
-

Queuing Locks

-

Sometimes want to accommodate both short and long critical sections while still minimizing overhead. It is not possible to completely avoid busy waiting on a multiprocessor, but you can minimize it by using a queuing lock.

-

With a queuing lock, access to the underlying data structure is controlled by a spinlock. On acquire, if the lock is FREE, the process can proceed. If the lock is BUSY, the process is added to a queue and suspended. When the lock is released, the next process in the queue is resumed.

-

To suspend a thread on a multiprocessor using this type of lock, first need to disable interrupts so it isn't preempted while holding the ready lists spinlock. Then, acquire the ready list spinlock, and then release the queuing lock's spinlock. Finally, switch to the next thread in the ready list and release the ready list spinlock.

-
class Lock {
- private:
-  int value = FREE;
-  SpinLock spinLock;
-  Queue waiting;
- public:
-  void acquire();
-  void release();
-};
-
-Lock::acquire() {
-  spinLock.acquire();
-  if (value != FREE) {
-    waiting.add(runningThread);
-    scheduler.suspend(&spinLock);
-    // scheduler releases spinLock
-  } else {
-    value = BUSY;
-    spinLock.release();
-  }
-}
-
-void Lock::release() {
-  TCB *next;
-  spinLock.acquire();
-  if (waiting.notEmpty()) {
-    next = waiting.remove();
-    scheduler.makeReady(next);
-  } else {
-    value = FREE;
-  }
-  spinLock.release();
-}
-
-class Scheduler {
- private:
-  Queue readyList;
-  SpinLock schedulerSpinLock;
- public:
-  void suspend(SpinLock *lock);
-  void makeReady(Thread *thread);
-}
-
-void Scheduler::suspend(SpinLock *lock) {
-  TCB *chosenTCB;
-  disableInterrupts();
-  schedulerSpinLock.acquire();
-  lock->release();
-  runningThread->state = WAITING;
-  chosenTCB = readyList.getNextThread();
-  thread_switch(runningThread, chosenTCB);
-  runningThread->state = RUNNING;
-  schedulerSpinLock.release();
-  enableInterrupts();
-}
-
-void Scheduler::makeReady(TCB *thread) {
-  disableInterrupts();
-  schedulerSpinLock.acquire();
-  readyList.add(thread);
-  thread->state = READY;
-  schedulerSpinLock.release();
-  enableInterrupts();
-}
- ```
-
- ### Case Study: Linux 2.6 Kernel Mutex Lock
-
- In Linux, most locks are `FREE` most of the time. Additionally, if a lock is `BUSY`, it is still likely that no other locks are waiting for it. THe Linux implementation of a mutex lock optimizes for this commone case by providing a fast path for the case where threads don't need to wait.
-
- Linux utilizes x86 specific instructions that allow the lock to be acquired and released on the fast path *without* first acquiring the spinlock or disabling interrupts. Mutexes have 3 possible states:
-
- ```c
- struct mutex {
-  /* 1: unlocked, 0: locked, negative: locked, possible waiters */
-  atomic_t count;
-  spinlock_t wait_lock;
-  struct list_head wait_list;
- };
- ```
-
- The Linux lock `acquire` code is a macro to avoid the overhead of a function call on the fast path.
-
-```asm
-      lock decl (%eax)      // atomic decrement of a memory location
-                            // address in %eax is pointer to lock->count
-      jns 1f                // jump if not signed (if value is now 0)
-      call slowpath_acquire
-
-

```c - for (;;) { - / - * Lets try to take the lock again - this is needed even if - * we get here for the first time (shortly after failing to - * acquire the lock), to make sure that we get a wakeup once - * it's unlocked. Later on, if we sleep, this is the - * operation that gives us the lock. We xchg it to -1, so - * that when we release the lock, we properly wake up the - * other waiters: - / - if (atomic_xchg(&lock->count, -1) == 1) - break;

-

/ didn't get the lock, go to sleep: / - ... - } - ```

-

Condition Variables

-

Condition variables are used to wait for a particular condition to become true. They are used in conjunction with locks to provide a way for a thread to be woken up when a condition becomes true. The condition variable is associated with a lock, and the lock must be held when waiting on the condition variable.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/7-multiprocessor-scheduling.html b/site/operating-systems/v2-concurrency/7-multiprocessor-scheduling.html deleted file mode 100644 index 0b742a7..0000000 --- a/site/operating-systems/v2-concurrency/7-multiprocessor-scheduling.html +++ /dev/null @@ -1,228 +0,0 @@ - - - - - - 7 Multiprocessor Scheduling - - - - - -
- -

7 Multiprocessor Scheduling

-
- Last modified: 2024-03-05 - -
-
-

Multiprocessor Scheduling

-

Modern systems usually have multiple processors, each with multiple cores that all have hyperthreading. Scheduling algorithms for multiprocessor systems need to make use of the parallelism that these systems offer.

-

Scheduling Sequential Applications on Multiprocessors

-

Consider a server application that needs to process a large number of requests. A simple approach would be to maintain a single MFQ, as well as a lock on it so only one processor can access it at a time. When a request needs to do something like I/O, it can reenter the MFQ and let another request run. However, this approach has a few problems:

- -

For these reasons, common practice is to have a separate MFQ for each processor. Each processor uses affinity scheduling, where once a thread is scheduled on a processor, it will stick to only being scheduled there. Rebalancing load between processors is done by moving threads between MFQs.

-

Scheduling Parallel Applications on Multiprocessors

-

Although there often exists a logical mapping between work and processors, it's often not possible to know this mapping at compile time. The number of threads and processors available can change at runtime, and the work may not be evenly divisible among the processors.

-

Oblivious Scheduling is when the scheduler operates without knowledge of the intent of the program. Each thread schedules completely independently of the others. This can be simple and effiient, but also has some problems:

- -

Gang Scheduling

-

The application picks some decomposition of the work into some set of threads, and those threads are always either running togther or not at all. This can be especially useful for specialized servers that need fine-grained control over the scheduling of their threads (like a DBMS). Windows, Linux and MacOS all support mechanisms for gang scheduling.

-

It is usually more efficient to run two parallel programs, each with half the number of processors, than to time slice two programs, each gang scheduled onto all processors.

-

Allocating different processors to different tasks is called space sharing, and is useful for minimizing context switches and cache invalidations. Space sharing is straightforward if tasks start and stop at the same tie. However, with a dynamic number of available processors, it's less trivial.

-

Scheduler Activations

-

Applications are given an execution context, or scheduler activation, on each processor assigned to the application. Via upcalls, the application is informed whenever the number of processors changes. Blocking on I/O operations also triggers an upcall to allow the application to repurpose the processor.

-

Scheduler activations only define the mechanism for informing an application of its processor allocation, but not the policy for scheduling.

-

Real Time Scheduling

-

If responsiveness is more important than throughput, we can use real-time scheduling. For instance, in control systems, or with a user interface, we want to ensure tasks are completed by a deadline.

- -
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/7-queueing-theory.html b/site/operating-systems/v2-concurrency/7-queueing-theory.html deleted file mode 100644 index 8e5e65d..0000000 --- a/site/operating-systems/v2-concurrency/7-queueing-theory.html +++ /dev/null @@ -1,278 +0,0 @@ - - - - - - 7 Queueing Theory - - - - - -
- -

7 Queueing Theory

-
- Last modified: 2024-03-06 - -
-
-

Queueing Theory

-

Simplifying assumptions:

- -

Definitions

- -

Little's Law

-

Defines a very general relationship for any system, and is useful for understanding the relationship between throughput, response time, and the number of tasks in the system.

-

$$N = X \cdot R$$

-

eg. If the throughput is 10 tasks per second, and the average response time is 5 seconds, then there are 50 tasks in the system.

-

Examples

-

Server than processes requests sequentually. The average arrival and departure rate is 100 requests/sec. The average request completes after 5 ms. What is the average utilization $U$ of the system?

-

$$U = X \cdot R = 100 \cdot 0.005 = 0.5$$

-

Meaning the server is busy 50% of the time.

-

Web service that takes an average of 100 ms to process a request, and handles 10000 queries per second. What is the average number of requests in the system?

-

$$N = X \cdot R = 10000 \cdot 0.1 = 1000 \text{ queries}$$

-

Response Time vs. Utilization

-

Operating a system with high utilization increases risk of overload. If $\lambda > \mu$, the queue will grow indefinitely, and so will the response time.

-

Higher arrival rate $\lambda$ and burstier arrival process will tend to yield longer queue lengths.

-

Best Case: Uniform Arrival

- -

Worst Case: Bursty Arrival

-

When requests arrive in groups (which they often do), the queue will grow and shrink in response to the bursts. This can lead to a higher average queue length and response time. Even if the average arrival rate is less than the service rate, the queue can still grow if the arrival process is bursty.

-

For example, consider the following two systems:

- -

System 1 processes requests as they come in, with a queue length of 0. The average response time over a 10 second period is 1 second.

-

$$ -R = \frac{1}{10} \sum_{i=1}^{10} 1 = 1 \text{ seconds} -$$

-

System 2 processes requests in bursts, with a queue length that grows and shrinks. The average response time is...

-

$$R = \frac{1}{10} \sum_{i=1}^{10} i = 5.5 \text{ seconds}$$

-

Generally, for a system with arrival rate $\lambda = \frac{n}{t}$ which is a bursty arrival process of $n$ requests every $t$ seconds, and a service time of $S$ seconds, the average response time for each request in the burst is...

-

$$ -R = \frac{1}{n} \sum_{i=1}^{n} i \cdot S -$$

-

Exponential Arrivals

-

An exponential distribution of a continuous random variable has a mean $\lambda^{-1}$, and a variance $\lambda^{-2}$. Its probability density function is...

-

$$ -f(x) = \lambda e^{-\lambda x} -$$

-

It is a memoryless distribution, meaning that the probability of an event occurring in the next $t$ seconds is the same as the probability of an event occurring in the next $t + s$ seconds.

-

A memoryless distribution is useful because it allows us to model a queue as a finite state machine with states corresponding to the number of tasks in the queue, and transition probablilities $\lambda$ and $\mu$ for transitioning up a state (arrival) and down a state (departure).

-

Assuming $\lambda < \mu$, the system is stable and we can calculate the response time $R$ as a function of utilization $U$ and service time $S$:

-

$$ -R = \frac{S}{1 - U} -$$

-

This shows the exponential relationship between response time and utilization. As utilization increases, response time increases very slowly at first, but then increases rapidly as the system approaches overload.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/7-uniprocessor-scheduling.html b/site/operating-systems/v2-concurrency/7-uniprocessor-scheduling.html deleted file mode 100644 index cadf0e7..0000000 --- a/site/operating-systems/v2-concurrency/7-uniprocessor-scheduling.html +++ /dev/null @@ -1,319 +0,0 @@ - - - - - - 7 Uniprocessor Scheduling - - - - - -
- -

7 Uniprocessor Scheduling

-
- Last modified: 2024-03-04 - -
-
-

Uniprocessor Scheduling

-

Preface: Performance Terminology

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Key WordDescription
Task/JobA unit of work that can be scheduled.
Response time/delayThe user-perceived time to do some task.
ThroughputThe number of tasks completed per unit time.
PredictabilityInversely related to variance in response time for repeated tasks.
Scheduling OverheadThe time to switch between tasks.
FairnessEquality in the number and timelines of resource allocations.
StarvationLack of progress for one task due to resources being allocated to higher-priority tasks.
WorkloadThe set of tasks to be scheduled.
-

Uniprocessor Scheduling

-

First in, First Out (FIFO)

-

Complete tasks as they arrive, finishing one before starting the next. On a uni-processor system with a completely CPU bound workload, this is one of the best scheduling algorithm in terms of throughput since overhead is minimized. However, it is not good for interactive systems, since short tasks are delayed by long ones. Concretely, the average response time for a FIFO scheduler is in many cases much worse than for other scheduling algorithms.

-

Memcached from Facebook uses a FIFO scheduler to handle requests to the caches in front of their databases. This works because the requests are all in memory (CPU bound) and of roughly equal length. The simplicity of FIFO makes it a good choice for extremely high-throughput systems that can work under this consistent request pattern.

-

Shortest Job First (SJF)

-

Complete the task with the shortest remaining time first. This is optimal in terms of average response time for a given set of tasks, but it is not implementable in practice because the run time of a task is not known in advance.

-

Bias Towards Short Tasks

-

SJF biases the scheduler towards short tasks, which can lead to starvation of long tasks. There is a fundamental tradeoff between average response time and the variance of response times. This is a problem in interactive systems, where long tasks are often user-initiated. Furthermore, this property can be taken advantage of by breaking a long task into many short tasks, and can even become a security vulnerability whereby malicious users can starve other users by submitting many short tasks.

-

Sample Bias

-

When evaluating the effectiveness of SJF compared to other scheduling algorithms, it is important to avoid sample bias. Starvation can occur for long tasks if there are always short tasks in the queue, and if you only sample response time for completed tasks, you will not see the starvation.

-

Bandwidth Constrained Web Services

-

SJF is in some ways ideal for a web services that is limited by its network egress bandwidth. In this case, the shortest jobs are the ones that can be sent out the door the fastest, and the system can be optimized for throughput by minimizing the time it takes to send out the most data. Since response times would be limited by the network for larger requests anyways, optimizing for the shortest jobs is a good strategy.

-

If a server becomes overloaded, it can start dropping larger requests. This is a natural form of load shedding, and requires additional logic to handle the dropped requests.

-

Round Robin (RR)

-

Each task in a queue is run for a fixed time quantum. If it is not finished, the process is preempted by a timer interrupt and put at the end of the queue. This makes it impossible for a task to starve indefinitely. The time quantum should be chosen to be long enough to minimize the overhead of context switching, but short enough to ensure good response time.

-

Overhead of RR Context Switching

-

There is a tradeoff between responsiveness and overhead, mainly due to the invalidation of cache entries between context switches. It is not nessesarily possible to maintain a good cache hit rate with a very short time quantum, since the cache will be invalidated every time a new task is run. Increasing the time quantum doesn't reduce the overhead of context switching, but it does reduce the frequency of context switches, thus improving the cache hit rate.

-

You can view RR as a compromise between FIFO and SJF. RR with an infinite time quantum is equivalent to FIFO. A special case of SJF can be thought of as having a "time quantum" of a single instruction with 0 overhead. Then, tasks would finish in the order of their lengths, albeit with alonger response times for shorter tasks than SJF without a time quantum.

-

Simultaneous Multithreading (SMT) is a technology that allows multiple threads to issue instructions to a superscalar processor in the same cycle. This can be thought of as a form of hardware-level round-robin scheduling without the overhead of context switching.

-

Maximizing Response Delay

-

Consider a workload of $n$ tasks, each of which take $t \cdot q$ seconds to complete ($q$ being the time quantum).

-

Assuming no scheduling overhead, the overall throughput of RR, SJF, and FIFO are all equal ($n \cdot t \cdot q$). However, the average response time for RR is much worse than SJF and FIFO (derivation below).

-

With RR, all tasks will complete during the last round of scheduling. With our parameters, this round begins at time $(n \cdot q) \cdot (t - 1)$, so our average response time is:

-

$$ -T_{\text{RR}} = \frac{1}{n} \sum_{i=1}^{n} (nq)(t - 1) + iq -$$

-

$$ -= q\frac{(n + 1) (2t - 1)}{2} -$$

-

SJF and FIFO behave identically in this case, and have an average repsonse time as follows:

-

$$ -T_{\text{SJF}} = T_{\text{FIFO}} = \frac{1}{n} \sum_{i=1}^{n} iqt = q\frac{(n + 1)t}{2} -$$

-

This illustrates that RR has a much worse average response time than SJF and FIFO, but the same throughput. Generally, if response time is the most important metric, round-robin is not the best choice.

-

Silver Lining: Stream Processing

-

Round-robin scheduling is ideal for applications where the tasks are not discrete workload on the client side, but rather a continuous stream of data. For instance, when streaming video from a server, the server can send out a small chunk of video to each client in a round-robin fashion. This is a good way to ensure that all clients are served equally, and that no client is starved.

-

Mixed Workloads Being Bad for RR

-

Systems that need to schedule a mixture of I/O and CPU-bound tasks can have a hard time with RR. For instance, a text editor program needs to minimize the latency of user input being displayed on screen. If the text editor is running in a round-robin system, it would likely need to wait for the next round of scheduling to display the user's input.

-

Consider another case where a naive browser is connected to a slow link and needs to download a large file in the background, all while the user continues to surf the web. If network I/O is scheduled in a round-robin fashion, the user's web browsing experience will be degraded.

-

Max Min Fairness

-

Max Min Fairness is a scheduling algorithm that tries to minimize the maximum response time of any task, effectively minimizing the variance.

-

If all processes are compute-bound, then the algorithm is equivalent to RR. However, I/O bound processes that don't use their full time quantum will be allowed to execute fully, and then their remaining allocation will be equally distributed to the other tasks. This pattern continues until all CPU time is used up.

-

A theoretical implementation would be to always schedule the task that has used the processor for the least amount of time. This doesn't work in practice because two equally short tasks will starve each other.

-

Instead, the algorithm can be approximated by tracking CPU usage time only by the quantum, and allowing tasks to get no more than 1 extra quantum of CPU time than their ideal max min allocation. This however still requires a priority queue to be maintained, and is not practical for commercial operating systems.

-

Multi-level Feedback Queue (MLFQ)

-

Grocery store lines with "express lanes" of multiple priorities

-

MLFQ attempts to find a compromise between the following goals:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
GoalDescription
ResponsivenessShort tasks should be completed quickly.
Low OverheadMinimize the number of preemptions, as well as the time spent scheduling.
Starvation AvoidanceAll tasks should be able to make progress.
Background TasksDeferrable tasks like system maintenance should not interfere with foreground tasks.
FairnessAssign non-background tasks an approximately max-min fair share of the CPU.
-

MLFQ Algorithm

-

Maintain multiple RR queues, each with a different priority and time quantum. Tasks with higher priority have smaller time quanta and preempt lower priority tasks. Tasks at the same level are scheduled RR.

-

Tasks are initially placed in the highest priority queue. If a task uses its entire time quantum, it is demoted to the next lower priority queue. If a task uses less than its entire time quantum, it is either kept in the same queue or moved up. This is a form of SJF within each queue.

-

To prevent starvation and achieve max-min fairness, the scheduler monitors process execution time on the CPU. The scheduler then only schedules processes that have yet to recieve their fair share. If a process already got its fair share of the CPU, it is demoted to a lower priority, and if they've yet to get their fair share, they are promoted to a higher priority.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v2-concurrency/async-linux-server-select.png b/site/operating-systems/v2-concurrency/async-linux-server-select.png deleted file mode 100644 index 36d888f..0000000 Binary files a/site/operating-systems/v2-concurrency/async-linux-server-select.png and /dev/null differ diff --git a/site/operating-systems/v2-concurrency/asyncio-read-linux.png b/site/operating-systems/v2-concurrency/asyncio-read-linux.png deleted file mode 100644 index b5977aa..0000000 Binary files a/site/operating-systems/v2-concurrency/asyncio-read-linux.png and /dev/null differ diff --git a/site/operating-systems/v2-concurrency/thread-lifecycle-diagram.png b/site/operating-systems/v2-concurrency/thread-lifecycle-diagram.png deleted file mode 100644 index 6c89e01..0000000 Binary files a/site/operating-systems/v2-concurrency/thread-lifecycle-diagram.png and /dev/null differ diff --git a/site/operating-systems/v2-concurrency/threads.png b/site/operating-systems/v2-concurrency/threads.png deleted file mode 100644 index d42de7e..0000000 Binary files a/site/operating-systems/v2-concurrency/threads.png and /dev/null differ diff --git a/site/operating-systems/v4-persistent-storage/11-file-systems-overview.html b/site/operating-systems/v4-persistent-storage/11-file-systems-overview.html deleted file mode 100644 index 0863d7e..0000000 --- a/site/operating-systems/v4-persistent-storage/11-file-systems-overview.html +++ /dev/null @@ -1,222 +0,0 @@ - - - - - - 11 File Systems Overview - - - - - -
- -

11 File Systems Overview

-
- Last modified: 2024-02-25 - -
-
-

File Systems: Introduction and Overview

-

What do we need in a filesystem?

- -

Nonvolatile Storage

-

As opposed to DRAM, nonvolatile storage is persistent. It is also generally cheaper and can have higher capacity. However, it is also orders of magnitude (~5 in the case of magnetic disk accesses) slower than DRAM. Current nonvolatile storage technologies don't allow random access to words of data, but instead require that data be read and written in blocks of a fixed size (e.g. 512 bytes).

-

The File System Abstraction

-

A File is a named collection of data in a file system. Files are made of metadata and data. I won't go into what those are. Files can be "executable", and executable files on Linux begin with a magic number that tells the OS how to run the file. Scripts can also be executable, and they begin with a "shebang" (#!), following by the interpreter which tells the OS how to run the script.

-

Traditional files can be thought of as a signle logical stream of bytes. However, MacOS's Extended File System (HFS+) and Windows NTFS support multiple streams (forks) for a signle file. In these contexts, you need to specify which stream you want to read from/write to in the corresponding system calls.

-

I won't define them, but you should also know and understand the following terms:

- - -

The mapping between a name and file is called a hard link. File systems that allow multiple hard links aren't a tree, and are instead usually a directed acyclic graph (DAG). A symbolic link is a mapping from a name to another file name. These are useful since they allow you to reference files that are stored on other systems/volumes. Some OS's support features managed outside of the file system. Windows has shortcuts, which are really just files that Windows recognizes and redirects from. MacOS has aliases, which are similar to symbolic links, but also refactor themselves when the target file is moved.

-

Volumes

-

A volume is a collection of physical storage resources that form a logical storage device. In the simplest case, a volume is a single disk. However, a disk can be partitioned into multiple volumes, and a single volume can be made of multiple disks.

-
- -
- - \ No newline at end of file diff --git a/site/operating-systems/v4-persistent-storage/13-files-and-directories.html b/site/operating-systems/v4-persistent-storage/13-files-and-directories.html deleted file mode 100644 index 3245e72..0000000 --- a/site/operating-systems/v4-persistent-storage/13-files-and-directories.html +++ /dev/null @@ -1,267 +0,0 @@ - - - - - - 13 Files And Directories - - - - - -
- -

13 Files And Directories

-
- Last modified: 2024-01-07 - -
-
-

Chapter 13 - Files and Directories

-

Need to achieve the following:

- -

Implementation Overview

-

Most implementations are based on four key ideas:

- -

Directories and Index Structures

-

File names and offsets are mapped to storage blocks in two steps:

-
    -
  1. directories map names to file numbers
  2. -
  3. index structures map file numbers and offsets to storage blocks
  4. -
-

Often, the index is some form of tree.

-

Free Space Maps

-

Free space maps are used to keep track of which blocks are free and which are in use. At a minimum, it needs to work, but it is also nice if files are allocated in a way that gives better spacial locality. For example, many file systems implement free space maps as bitmaps in persistent storage.

-

Locality Heuristics

-

Locality heuristics are used to improve performance. Operating systems implement their own policies on where to store data in order to increase the spacing locality of files. For instance, storing files in the same directory in the same area of the disk. Some also periodically defragment the disk by rewriting files to be contiguous.

-

Directories: Naming Data

-

Can simply store name -> number mappings in a file that represents the directory. As a base case, you can just have a predefined number for the root directory. Linux's Fast File System (FFS) uses 2 as the root directory's number. Files in the same directory are often accessed together, so it is nice to store them in the same area of the disk so that caching can work its magic.

-

Although they are files, directories need their own API to prevent users from accidentally corrupting the directory structure. However, processes can still read directories to get a list of files in a directory, but it can be convinent to have a syscall to get the list of files in a directory (getdents in Linux).

-

Internals

-

Although simple lists of file name number pairs works (and were used in early versions of Linux), modern file systems use more complex data structures to acccomodate large directories. Linux XFS, Microsoft NTFS, and Oracle ZFS all use trees to store directories.

-

XFS uses a B+ tree, and directory entries are stored in the first part of the directory file. The B+ tree's root node is stored in a known offset (BTREE_ROOT_PTR). The fixed-size internal and leaf nodes are stored after the root node, and the variable-size directory entries are stored at the start of the file. Starting from the root, each tree node includes pointers to where in the file its children are stored.

- -

Hard links are multiple names (and directory entries) for the same file on disk. OS uses reference counting to garbage collect files that are unlinked.

-

Soft (symbolic) links are files that directly map to the name of another file.

-

A consequence of hard links is that you can't keep file metadata in the directory entry

-

Files: Finding Data

-

Files systems usually try to:

- -

Some properties of file systems are:

- -

Although storage arranges data in sectors (magnectic disk) or pages (flash), file systems usually use blocks as the unit of allocation. Blocks are usually a power of two multiple of the sector or page size. For example, Linux uses 4KB blocks on 512 byte sectors. FAT and NTFS call blocks clusters. Similarly, file systems store data in variable length arrays of contiguous tracks called extents. NTFs calls them runs.

-

Case Studies

-

FAT

-

Very simple file system that uses a linked list to store the index structure. Still used in places like flash drives and SD cards, and most recent version (FAT32) supports volumes with up to 2^28 blocks, and files with up to 2^32 - 1 byes.

-

The FAT is an array of 32-bit entries that resides in a reserved area of the volume. Each file has a linked list of FAT entries that point to the blocks that make up the file. Directories map file names to the index of the first FAT entry for the file.

-

System can also use the FAT for free space tracking. OS scans for unused entries (0x00000000), and takes it out of the free list.

-

FAT usually uses simple allocation strategies like first-fit or next-fit. To deal with the resulting fragmentation, some implementations have a defragmentation tool that rewrites files to be contiguous. For example, the FAT degrafmenter in Windows XP tries to rewrite files to be within the same extent.

-

It is widely used because it is simple. In addition to simple storage technologies, even some applications use FAT to store data. For example, there is a FAT-like file system embedded in .doc files made in Microsoft Word 1997-2007.

-

Drawbacks:

- -

Unix Fast File System (FFS): Uses a tree-based multi-level index, and many locality heuristics to achieve good spacial locality. Linux's ext2 and ext3 are based on FFS.

-

NTFS: Uses a tree-based structure that is more flexible than FFS's indexing scheme. Indexes variable sized extents* indead of individual blocks. NTFS is still used in Windows, and its techniques are used in many other modern file systems (ext4, XFS, HFS, HFS+).

-

ZFS: Uses copy-on-write, writing new versions of files to free disk space instead of overwriting old versions. This optimizes for reliability and write performance

-
- -
- - \ No newline at end of file diff --git a/site/papers/.mailing b/site/papers/.mailing deleted file mode 100644 index 817b090..0000000 --- a/site/papers/.mailing +++ /dev/null @@ -1,6 +0,0 @@ - -https://calendar.google.com/calendar/u/1/r?cid=cs.washington.edu_kaujd7jilnokn62oukhdms0n8c%40group.calendar.google.com&authuser=3 - -https://mailman.cs.washington.edu/mailman/listinfo/uw-networks - -https://mailman.cs.washington.edu/mailman/listinfo/uw-systems \ No newline at end of file diff --git a/site/papers/cse-454-reading.html b/site/papers/cse-454-reading.html deleted file mode 100644 index b2e8547..0000000 --- a/site/papers/cse-454-reading.html +++ /dev/null @@ -1,603 +0,0 @@ -
- - - -
  • INFORMATION EXTRACTION - - - - -
  • -
  • LEARNING, DATA MINING, PERSONALIZATION - - - - - -
  • -
  • USABILITY & INTERFACES - - - - - -
  • -
  • ADVERTISING - - - -
  • -
  • SOCIAL COMPUTING - - - -
  • -
  • SECURITYu & E-COMMERCE - - - -
  • -
  • HAZARDS: SPAM, VIRUSES, SPYWARE and the like - - -
  • -
    \ No newline at end of file diff --git a/site/performance-engineering/efficiently-implementing-state-pattern-JVM.html b/site/performance-engineering/efficiently-implementing-state-pattern-JVM.html deleted file mode 100644 index 3cb4832..0000000 --- a/site/performance-engineering/efficiently-implementing-state-pattern-JVM.html +++ /dev/null @@ -1,337 +0,0 @@ - - - - - - Efficiently Implementing State Pattern Jvm - - - - - -
    - -

    Efficiently Implementing State Pattern Jvm

    -
    - Last modified: 2024-12-08 - -
    -
    -

    JVM Performance with State Pattern Optimizations

    -

    I was reading some interesting code at AWS and came across an implementation of the state pattern written by one of the senior engineers on our team. If you aren't already familiar, read this. While the problem I would usually reach to the state pattern to solve is one more concerned with logical complexity and structure, (without revealing anything in particular about the code I was reading) I was intrigued by the performance implications of the underlying implementations, especially in a multi-threaded environment.

    -

    The Implementations

    -
      -
    1. Optimized State Pattern Using Enums: In InlineStatePattern, states are represented as enums with direct in-line transitions. State changes are handled by a simple check and assignment operation.
    2. -
    3. Generic State Pattern with Lazy Transitions: In PolymorphicStatePattern, state changes are managed with a generic context using an AtomicReference for thread-safe transitions. Lazy evaluation allows the state to change only if necessary, reducing the overhead in certain cases (emphasis on certain).
    4. -
    -

    When might I reach for either?

    -
      -
    1. Inline State Pattern:
    2. -
    3. Use when state transitions are simple and predictable.
    4. -
    5. -

      Ideal for low contention scenarios where synchronization overhead is a concern. For example, in a high-throughput system with little locking overhead, like a cache or a message queue.

      -
    6. -
    7. -

      Generic State Pattern:

      -
    8. -
    9. Use when state transitions are complex or require additional logic.
    10. -
    11. Ideal for high contention scenarios where synchronization is necessary. For example, in a shared resource or a asynchronous/distributed system. Note that while I'm saying high contention, this is not a hard and fast rule. It's more about the nature of the contention and the nature of the state transitions, e.g. if the state transitions are complex and the contention is low, you might still want to use the generic state patter for the sake of readability maintainability.
    12. -
    -

    JVM Optimizations and Overheads

    -
      -
    1. Enum-Based State Handling (Inlined Transitions):
    2. -
    3. Enums offer low-level performance wins by making state transitions in-line without additional method calls or object creation. OpenJDK initial
    4. -
    5. -

      In this setup, each DocumentState enum implements its own handle and nextState methods. Transitioning states here is a simple assignment with little locking overhead, so the JVM can optimize by inlining these transitions at runtime.

      -
    6. -
    7. -

      Generic Interface with AtomicReference:

      -
    8. -
    9. AtomicReference ensures thread safety for transitions, but adds some overhead due to its CAS (Compare-and-Swap) operations, which are costly under high contention.
    10. -
    11. With lazy state transition, this implementation could theoretically avoid unnecessary state updates, which may be beneficial under lower contention.
    12. -
    -

    Predictions and Testing

    -

    Based on JVM behaviors, here are our predictions:

    -
      -
    1. Single-threaded Performance:
    2. -
    3. -

      InlineStatePattern should outperform PolymorphicStatePattern in single-threaded scenarios due to reduced overhead from AtomicReference.

      -
    4. -
    5. -

      Multi-threaded Performance:

      -
    6. -
    7. At lower levels of contention, InlineStatePattern should still perform better due to fewer synchronization requirements.
    8. -
    9. Under high contention, the AtomicReference CAS operations in PolymorphicStatePattern may actually limit performance due to frequent retries.
    10. -
    -

    To verify these assumptions, let’s test using the following code blocks. You can add these into the main method to see the results.

    -

    Code Blocks to Test Performance

    -
      -
    1. Single-Threaded Performance Test:
    2. -
    -
       Document doc1 = new InlineStatePattern.Document("Single-threaded Test");
    -   benchmark("Optimized State Pattern - Single Thread", 10_000_000, doc1::handleState);
    -
    -   Document doc2 = new PolymorphicStatePattern.Document("Single-threaded Test");
    -   benchmark("Generic State Pattern - Single Thread", 10_000_000, doc2::handleState);
    -
    -

    Here, I predict InlineStatePattern will complete faster, as it avoids the overhead of AtomicReference and CAS operations.

    -
    Inline State Pattern - Single Thread: 201363 µs for 10000000 iterations (0.02 µs/op) - Last result: Published: Single-threaded Test
    -Polymorphic State Pattern - Single Thread: 133314 µs for 10000000 iterations (0.01 µs/op) - Last result: Published: Single-threaded Test
    -
    -

    As expected, InlineStatePattern outperforms PolymorphicStatePattern in a single-threaded scenario by roughly 50%.

    -
      -
    1. Multi-Threaded Performance Test with Low Contention::
    2. -
    -
    Document sharedDoc1 = new InlineStatePattern.Document("Multi-thread Test");
    -Thread[] threads1 = new Thread[4];
    -for (int i = 0; i < threads1.length; i++) {
    -    threads1[i] = new Thread(() -> {
    -        benchmark("Optimized Pattern - Multi-thread Low Contention", 2_500_000, sharedDoc1::handleState);
    -    });
    -    threads1[i].start();
    -}
    -
    -Document sharedDoc2 = new PolymorphicStatePattern.Document("Multi-thread Test");
    -Thread[] threads2 = new Thread[4];
    -for (int i = 0; i < threads2.length; i++) {
    -    threads2[i] = new Thread(() -> {
    -        benchmark("Generic Pattern - Multi-thread Low Contention", 2_500_000, sharedDoc2::handleState);
    -    });
    -    threads2[i].start();
    -}
    -
    -

    Here, InlineStatePattern should still outperform PolymorphicStatePattern, as the contention is low.

    -
    Polymorphic Pattern - Multi-thread Low Contention: 339159 µs for 2500000 iterations (0.14 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 360859 µs for 2500000 iterations (0.14 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 348844 µs for 2500000 iterations (0.14 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 375313 µs for 2500000 iterations (0.15 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 342431 µs for 2500000 iterations (0.14 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 368308 µs for 2500000 iterations (0.15 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 368001 µs for 2500000 iterations (0.15 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 385776 µs for 2500000 iterations (0.15 µs/op) - Last result: Published: Multi-thread Test
    -idata = [339159, 360859, 348844, 375313]
    -pdata = [342431, 368308, 368001, 385776]
    -
    -

    As you can see however, despite being roughly neck and neck with Poly and Inline having respective averages of 356043.75 (0.1424175 µs/op) and 366129.0 (0.1464516 µs/op), the Inline pattern is actually slightly slower in this case. Since I hate being proven wrong, I also wonder if the results would be different if we swapped the order of the tests.

    -
    Polymorphic Pattern - Multi-thread Low Contention: 494816 µs for 2500000 iterations (0.20 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 481862 µs for 2500000 iterations (0.19 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 491883 µs for 2500000 iterations (0.20 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 492075 µs for 2500000 iterations (0.20 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 455870 µs for 2500000 iterations (0.18 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 491346 µs for 2500000 iterations (0.20 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 483745 µs for 2500000 iterations (0.19 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 475941 µs for 2500000 iterations (0.19 µs/op) - Last result: Published: Multi-thread Test
    -idata = [494816, 481862, 491883, 492075]
    -pdata = [455870, 491346, 483745, 475941]
    -
    -

    This time, we have a Poly and Inline average of 490159.0 (0.1960636 µs/op) and 476225.5 (0.1904902 µs/op) respectively. Well well well...

    -

    Lets try this one more time, running the tests twice, while also isolating them to their own JVM instances.

    -
    Polymorphic Pattern - Multi-thread Low Contention: 372321 µs for 2500000 iterations (0.15 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 387558 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 402434 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -Polymorphic Pattern - Multi-thread Low Contention: 407174 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -pdata = [372321, 387558, 402434, 407174]
    -
    -
    Inline Pattern - Multi-thread Low Contention: 397600 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 398064 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 392855 µs for 2500000 iterations (0.16 µs/op) - Last result: Published: Multi-thread Test
    -Inline Pattern - Multi-thread Low Contention: 419996 µs for 2500000 iterations (0.17 µs/op) - Last result: Published: Multi-thread Test
    -idata = [397600, 398064, 392855, 419996]
    -
    -

    So we have a Inline average of 402128.75 (0.1608515 µs/op) and a Poly average of 392371.75 (0.1569487 µs/op). So it seems that the Inline pattern is actually slower in this case. AtomicReferences are preetttty pretty good.

    -
      -
    1. High Contention Test with Increased Threads:
    2. -
    -
    Document highContDoc1 = new InlineStatePattern.Document("High Contention Test");
    -Document highContDoc2 = new PolymorphicStatePattern.Document("High Contention Test");
    -
    -for (int i = 0; i < 8; i++) {
    -    new Thread(() -> {
    -        benchmark("Optimized Pattern - High Contention", 1_250_000, highContDoc1::handleState);
    -    }).start();
    -}
    -
    -for (int i = 0; i < 8; i++) {
    -    new Thread(() -> {
    -        benchmark("Generic Pattern - High Contention", 1_250_000, highContDoc2::handleState);
    -    }).start();
    -}
    -
    -

    With more threads, we expect contention to impact PolymorphicStatePattern more due to frequent CAS retries in AtomicReference.

    -

    Conclusion

    -

    The low-level optimizations in InlineStatePattern using enums make it ideal for performance-critical, low-contention use cases. PolymorphicStatePattern, with its AtomicReference, offers better safety for concurrent environments but incurs a trade-off in performance due to CAS operations. Testing under these scenarios confirms that the right implementation depends on your application’s specific threading needs.

    -
    - -
    - - \ No newline at end of file diff --git a/site/requirements.txt b/site/requirements.txt deleted file mode 100644 index 3cda9eb..0000000 --- a/site/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -markdown \ No newline at end of file diff --git a/site/scripts/build.py b/site/scripts/build.py deleted file mode 100644 index 31009ab..0000000 --- a/site/scripts/build.py +++ /dev/null @@ -1,693 +0,0 @@ -from dataclasses import dataclass -from pathlib import Path -from typing import List, Dict, Optional -import shutil -import markdown -import logging -from datetime import datetime -from urllib.parse import quote -import re -from collections import defaultdict - -# Set up logging -logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") -logger = logging.getLogger(__name__) - - -@dataclass -class Page: - """Represents a page in the static site""" - - title: str - path: Path - content: str - modified_date: datetime - category: Optional[str] - tags: List[str] - description: Optional[str] - is_index: bool = False - - -class SiteGenerator: - """Generates a static site from a directory of mixed content""" - - MARKDOWN_EXTENSIONS = [ - "meta", - "toc", - "fenced_code", - "tables", - "attr_list", - "footnotes", - "def_list", - "admonition", - ] - SUPPORTED_CONTENT = {".md", ".markdown"} - IGNORED_DIRECTORIES = { - ".git", - "__pycache__", - "node_modules", - ".github", - "nlp.venv", - "site", - "venv", - ".venv", - } - - def __init__(self, input_dir: str, output_dir: str): - self.input_dir = Path(input_dir) - self.output_dir = Path(output_dir) - self.markdown_converter = markdown.Markdown(extensions=self.MARKDOWN_EXTENSIONS) - self.pages: Dict[Path, Page] = {} - self.categories: Dict[str, List[Page]] = defaultdict(list) - self.tags: Dict[str, List[Page]] = defaultdict(list) - - def generate_site(self) -> None: - """Main method to generate the static site""" - try: - self._prepare_output_directory() - self._process_content() - self._organize_content() - self._copy_assets() - self._generate_special_pages() - self._generate_html_pages() - logger.info(f"Site generated successfully in {self.output_dir}") - except Exception as e: - logger.error(f"Failed to generate site: {str(e)}") - raise - - def _prepare_output_directory(self) -> None: - """Prepare the output directory with proper permissions""" - if self.output_dir.exists(): - shutil.rmtree(self.output_dir) - self.output_dir.mkdir(parents=True) - self.output_dir.chmod(0o755) - - # Create assets directory - assets_dir = self.output_dir / "assets" - assets_dir.mkdir(parents=True) - assets_dir.chmod(0o755) - - def _process_content(self) -> None: - """Process all content in the input directory""" - for file_path in self._walk_directory(self.input_dir): - if file_path.suffix in self.SUPPORTED_CONTENT: - self._process_markdown(file_path) - - def _walk_directory(self, directory: Path) -> List[Path]: - """Walk through directory while respecting ignored paths""" - files = [] - for item in directory.rglob("*"): - if not any(ignored in item.parts for ignored in self.IGNORED_DIRECTORIES): - if item.is_file(): - files.append(item) - return files - - def _extract_metadata(self, file_path: Path) -> dict: - """Extract metadata from markdown file""" - content = file_path.read_text(encoding="utf-8") - self.markdown_converter.reset() - self.markdown_converter.convert(content) - - metadata = {} - if hasattr(self.markdown_converter, "Meta"): - metadata = { - "title": self.markdown_converter.Meta.get( - "title", [file_path.stem.replace("-", " ").title()] - )[0], - "category": self.markdown_converter.Meta.get("category", [None])[0], - "tags": ( - self.markdown_converter.Meta.get("tags", [""])[0].split(",") - if "tags" in self.markdown_converter.Meta - else [] - ), - "description": self.markdown_converter.Meta.get("description", [None])[ - 0 - ], - } - else: - metadata = { - "title": file_path.stem.replace("-", " ").title(), - "category": None, - "tags": [], - "description": None, - } - - return metadata - - def _process_markdown(self, file_path: Path) -> None: - """Process a markdown file into a Page object""" - try: - content = file_path.read_text(encoding="utf-8") - metadata = self._extract_metadata(file_path) - - # Convert content after metadata extraction - self.markdown_converter.reset() - html_content = self.markdown_converter.convert(content) - - relative_path = file_path.relative_to(self.input_dir) - is_index = file_path.stem.lower() == "index" - - # Clean and normalize tags - tags = [tag.strip().lower() for tag in metadata["tags"] if tag.strip()] - - page = Page( - title=metadata["title"], - path=relative_path, - content=html_content, - modified_date=datetime.fromtimestamp(file_path.stat().st_mtime), - category=metadata["category"], - tags=tags, - description=metadata["description"], - is_index=is_index, - ) - - self.pages[relative_path] = page - - # Organize by category and tags - if page.category: - self.categories[page.category].append(page) - for tag in page.tags: - self.tags[tag].append(page) - - except Exception as e: - logger.error(f"Failed to process {file_path}: {str(e)}") - - def _organize_content(self) -> None: - """Organize pages by category and tags""" - # Sort pages within categories and tags - for category in self.categories: - self.categories[category].sort(key=lambda p: p.title) - for tag in self.tags: - self.tags[tag].sort(key=lambda p: p.title) - - def _copy_assets(self) -> None: - """Copy non-markdown files to output directory""" - for file_path in self._walk_directory(self.input_dir): - if file_path.suffix not in self.SUPPORTED_CONTENT: - relative_path = file_path.relative_to(self.input_dir) - output_path = self.output_dir / relative_path - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.parent.chmod(0o755) - shutil.copy2(file_path, output_path) - output_path.chmod(0o644) - - def _generate_special_pages(self) -> None: - """Generate special pages like main index, category index and tag index""" - # Generate main index - self._generate_main_index() - - # Generate categories and tags pages - # Generate categories index - if self.categories: - categories_content = self._render_categories_index() - categories_page = Page( - title="Categories", - path=Path("categories/index.md"), - content=categories_content, - modified_date=datetime.now(), - category=None, - tags=[], - description="Index of all categories", - is_index=True, - ) - self.pages[categories_page.path] = categories_page - - # Generate tags index - if self.tags: - tags_content = self._render_tags_index() - tags_page = Page( - title="Tags", - path=Path("tags/index.md"), - content=tags_content, - modified_date=datetime.now(), - category=None, - tags=[], - description="Index of all tags", - is_index=True, - ) - self.pages[tags_page.path] = tags_page - - def _render_categories_index(self) -> str: - """Render the categories index page""" - content = "" - return content - - def _render_tags_index(self) -> str: - """Render the tags index page""" - content = "" - return content - - def _generate_html_pages(self) -> None: - """Generate HTML pages for all processed content""" - # Generate regular pages - for page in self.pages.values(): - output_path = self.output_dir / page.path.with_suffix(".html") - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.parent.chmod(0o755) - - html_content = self._render_template(page) - output_path.write_text(html_content, encoding="utf-8") - output_path.chmod(0o644) - - # Generate category pages - for category, pages in self.categories.items(): - self._generate_category_page(category, pages) - - # Generate tag pages - for tag, pages in self.tags.items(): - self._generate_tag_page(tag, pages) - - def _generate_category_page(self, category: str, pages: List[Page]) -> None: - """Generate a page for a specific category""" - output_path = self.output_dir / "categories" / f"{category.lower()}.html" - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.parent.chmod(0o755) - - content = f"

    Category: {category}

    \n" - - page = Page( - title=f"Category: {category}", - path=Path(f"categories/{category.lower()}.md"), - content=content, - modified_date=datetime.now(), - category=None, - tags=[], - description=f"Pages in category {category}", - is_index=False, - ) - - html_content = self._render_template(page) - output_path.write_text(html_content, encoding="utf-8") - output_path.chmod(0o644) - - def _generate_tag_page(self, tag: str, pages: List[Page]) -> None: - """Generate a page for a specific tag""" - output_path = self.output_dir / "tags" / f"{tag.lower()}.html" - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.parent.chmod(0o755) - - content = f"

    Tag: {tag}

    \n" - - page = Page( - title=f"Tag: {tag}", - path=Path(f"tags/{tag.lower()}.md"), - content=content, - modified_date=datetime.now(), - category=None, - tags=[], - description=f"Pages tagged with {tag}", - is_index=False, - ) - - html_content = self._render_template(page) - output_path.write_text(html_content, encoding="utf-8") - output_path.chmod(0o644) - - def _generate_breadcrumbs(self, page: Page) -> str: - """Generate breadcrumb navigation""" - parts = [] - parts.append('Home') - - if page.category: - parts.append( - f'{page.category}' - ) - - if not page.is_index: - parts.append(page.title) - - return " » ".join(parts) - - def _generate_navigation(self, current_page: Page) -> str: - """Generate navigation links""" - nav_items = [] - - # Add home link if not on index - if not current_page.is_index: - nav_items.append('Home') - - # Add categories link - if self.categories: - nav_items.append('Categories') - - # Add tags link - if self.tags: - nav_items.append('Tags') - - return "\n".join(nav_items) - - def _generate_main_index(self) -> None: - """Generate the main index.html page""" - # Get recent pages (excluding special pages) - regular_pages = [ - p - for p in self.pages.values() - if not (p.is_index or str(p.path).startswith(("categories/", "tags/"))) - ] - recent_pages = sorted( - regular_pages, key=lambda p: p.modified_date, reverse=True - )[ - :10 - ] # Show 10 most recent - - # Generate recent posts section - recent_content = "

    Recent

    \n" - - # Generate categories section - categories_content = "

    Categories

    \n" - - # Generate popular tags section (show top 20 most used tags) - tag_counts = {tag: len(pages) for tag, pages in self.tags.items()} - popular_tags = sorted(tag_counts.items(), key=lambda x: (-x[1], x[0]))[:20] - - tags_content = "

    Featured Tags

    \n
    " - for tag, count in popular_tags: - tag_url = f"/tags/{quote(tag.lower())}.html" - tags_content += f'{tag} ({count})' - tags_content += "
    " - - # Calculate some statistics - total_notes = len(regular_pages) - total_categories = len(self.categories) - total_tags = len(self.tags) - - # Create the landing page content - content = f""" -
    -
    - {total_notes} - Notes -
    -
    - {total_categories} - Categories -
    -
    - {total_tags} - Tags -
    -
    -
    -
    - {recent_content} -
    -
    - {categories_content} -
    -
    - {tags_content} -
    -
    - """ - - # Create the index page - index_page = Page( - title="", - path=Path("index.md"), - content=content, - modified_date=datetime.now(), - category=None, - tags=[], - description="So so many notes", - is_index=True, - ) - - # Generate the HTML - output_path = self.output_dir / "index.html" - html_content = self._render_template(index_page) - output_path.write_text(html_content, encoding="utf-8") - output_path.chmod(0o644) - - def _render_template(self, page: Page) -> str: - """Render HTML template for a page""" - navigation = self._generate_navigation(page) - breadcrumbs = self._generate_breadcrumbs(page) - - # Generate tags section if page has tags - tags_section = "" - if page.tags: - tags_section = ( - "
    Tags: " - + ", ".join( - f'{tag}' - for tag in sorted(page.tags) - ) - + "
    " - ) - - return f""" - - - - - {page.title} - {f'' if page.description else ''} - - - - -
    - -

    {page.title}

    -
    - Last modified: {page.modified_date.strftime('%Y-%m-%d')} - {f'Category: {page.category}' if page.category else ''} -
    -
    - {page.content} -
    - {tags_section} -
    - -""" - - -def main(): - """CLI entry point""" - import argparse - - parser = argparse.ArgumentParser( - description="Generate a static site from markdown files" - ) - parser.add_argument("input_dir", help="Input directory containing content") - parser.add_argument("output_dir", help="Output directory for generated site") - parser.add_argument( - "--verbose", "-v", action="store_true", help="Enable verbose logging" - ) - args = parser.parse_args() - - if args.verbose: - logging.getLogger().setLevel(logging.DEBUG) - - try: - generator = SiteGenerator(args.input_dir, args.output_dir) - generator.generate_site() - except Exception as e: - logger.error(f"Failed to generate site: {str(e)}") - exit(1) - - -if __name__ == "__main__": - main() diff --git a/site/scripts/clean.py b/site/scripts/clean.py deleted file mode 100644 index 0f84a7b..0000000 --- a/site/scripts/clean.py +++ /dev/null @@ -1,4 +0,0 @@ -import shutil - -if __name__ == "__main__": - shutil.rmtree('./site', ignore_errors=True) \ No newline at end of file diff --git a/site/scripts/keywords.py b/site/scripts/keywords.py deleted file mode 100644 index 15570cc..0000000 --- a/site/scripts/keywords.py +++ /dev/null @@ -1,132 +0,0 @@ -import json -import string -import nltk - -from nltk.stem import WordNetLemmatizer -from typing import List -from pathlib import Path -from keybert import KeyBERT - -nltk.download("stopwords") -nltk.download('wordnet') - -class DataReader: - def __init__(self, root_dir: str = "."): - self.root_dir = Path(root_dir) - self.ROOT = root_dir - self.IGNORED_PATHS = { - ".git", - "__pycache__", - "node_modules", - ".github", - "venv", - ".venv", - "nlp.venv", - "site", - "README.md", - } - - self.SUPPORTED_EXTENSIONS = {".md", ".txt"} - - def read_files(self) -> List[Path]: - return self._walk_directory(self.root_dir) - - def _walk_directory(self, directory: Path): - """Walk through directory while respecting ignored paths""" - for item in directory.rglob("*"): - if not any(ignored in item.parts for ignored in self.IGNORED_PATHS): - if item.is_file() and item.suffix in self.SUPPORTED_EXTENSIONS: - yield item - - -class Normalizer: - def __init__(self): - self.lower_case = True - self.remove_punctuation = False - self.remove_stopwords = True - - self.stop_words = set(nltk.corpus.stopwords.words("english")) - - def normalize_doc(self, doc: str) -> str: - if self.lower_case: - doc = doc.lower() - - if self.remove_punctuation: - doc = str.translate(doc, str.maketrans("", "", string.punctuation)) - - if self.remove_stopwords: - doc = " ".join( - [word for word in doc.split() if word not in self.stop_words] - ) - - return doc - - -def get_kw_path_map(files: List[Path], model: KeyBERT) -> dict: - reader = DataReader() - normalizer = Normalizer() - - keyword_map = {} - - for i, file in enumerate(files): - with open(file) as f: - doc = normalizer.normalize_doc(f.read()) - - keywords = model.extract_keywords( - doc, - keyphrase_ngram_range=(1, 1), - stop_words="english", - use_mmr=True, - diversity=0.3, - ) - - keyword_map["/".join(file.parts)] = keywords - - return keyword_map - -def deduplicate(file_to_keywords: dict) -> dict: - wnl = WordNetLemmatizer() - deduped = {} - dedup_count = 0 - for f, kws in file_to_keywords.items(): - deduped[f] = [] - for kw, acc in kws: - stem = wnl.lemmatize(kw) - if stem != kw: - print('stem', stem, 'kw', kw) - dedup_count += 1 - if stem not in deduped[f]: - deduped[f].append((stem, acc)) - print('dedup count', dedup_count) - return deduped - -def aggregate(file_to_keywords: dict) -> dict: - keyword_to_files = {} - for file, keywords in file_to_keywords.items(): - for keyword in keywords: - if keyword[0] not in keyword_to_files: - keyword_to_files[keyword[0]] = [] - keyword_to_files[keyword[0]].append(file) - return keyword_to_files - - -def write_idx_to_json_file(idx: dict, output_path: Path): - if output_path.exists(): - output_path.unlink() - - if not output_path.parent.exists(): - output_path.parent.mkdir(parents=True) - - with open(output_path, "w") as f: - json.dump(idx, f) - - -if __name__ == "__main__": - reader = DataReader() - files = reader.read_files() - model = KeyBERT() - - file_to_kw = deduplicate(get_kw_path_map(files, model)) - kw_to_file = aggregate(file_to_kw) - write_idx_to_json_file(file_to_kw, Path("idx/file_to_keywords.json")) - write_idx_to_json_file(kw_to_file, Path("idx/keyword_to_files.json")) diff --git a/site/scripts/lint.py b/site/scripts/lint.py deleted file mode 100644 index 4802367..0000000 --- a/site/scripts/lint.py +++ /dev/null @@ -1,73 +0,0 @@ -import os -import argparse - -colors = { - 'red': '\033[91m', - 'green': '\033[92m', - 'yellow': '\033[93m', - 'blue': '\033[94m', - 'purple': '\033[95m', - 'end': '\033[0m' -} - -def print_c(text, color): - print(f"{colors[color]}{text}{colors['end']}") - -def lint_file(filename): - - if not filename.endswith('.md'): - print_c(f"Skipping {filename}", 'yellow') - return - warncount = 0 - try: - with open(filename, 'r', encoding='utf-8') as f: - line_number = 0 - for line in f: - line_number += 1 - for char in line: - if ord(char) > 127: # ASCII range is 0-127 - warncount += 1 - # print line number and character in green - print_c(f"{filename}:{line_number}:{char}", 'green') - except UnicodeDecodeError: - print_c(f"Skipping {filename}", 'yellow') - - return warncount - -def main(filenames): - total_warncount = 0 - for filename in filenames: - if not filename.endswith('.md'): - print_c(f"Skipping {filename}", 'yellow') - continue - - print("Linting", end=' ') - print_c(filename, 'red') - total_warncount += lint_file(filename) - - if total_warncount > 0: - print_c(f"Total warnings: {total_warncount}", 'red') - else: - print_c("No warnings found!", 'green') - -def get_files_recursively(directory): - file_paths = [] - for root, dirs, files in os.walk(directory): - for file in files: - file_path = os.path.join(root, file) - file_paths.append(file_path) - return file_paths - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description='Lint files.') - parser.add_argument('--recursive', '-r', action='store_true', help='lint files recursively') - parser.add_argument('filenames', nargs='+', help='file(s) to lint') - args = parser.parse_args() - - if args.recursive: - filenames = [] - for directory in args.filenames: - filenames.extend(get_files_recursively(directory)) - args.filenames = filenames - - main(args.filenames) diff --git a/site/scripts/replace.py b/site/scripts/replace.py deleted file mode 100644 index bc9d46d..0000000 --- a/site/scripts/replace.py +++ /dev/null @@ -1,53 +0,0 @@ -import argparse -import subprocess - -def replace_chars(input_files, backup): - # characters to replace - replacements = { - '—': '-', - '–': '-', - "’": "'", - '”': '"', - '“': '"', - '…': '...', - ',': ',' - } - - # generate sed script - sed_script = [] - for old_char, new_char in replacements.items(): - sed_script.append(f"s/{old_char}/{new_char}/g") - - # write to tpm file - sed_file_path = 'sed_commands.tpm' - with open(sed_file_path, 'w') as f: - f.write('\n'.join(sed_script)) - - # Perform replacements on each input file - for input_file in input_files: - output_file = f"{input_file}.tmp" - if backup: - # backup original file - subprocess.run(['cp', input_file, f"{input_file}.bak"]) - - # replacements with sed - subprocess.run(['sed', '-f', sed_file_path, input_file], stdout=open(output_file, 'w')) - - # replace orginal file - subprocess.run(['mv', output_file, input_file]) - - # cleanup - subprocess.run(['rm', sed_file_path]) - - print("Replacement done.") - - -# run with -b to create backup files -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Character replacement utility") - parser.add_argument('input_files', nargs='+', help='List of input files') - parser.add_argument('-b', '--backup', action='store_true', help='Create backup of original files') - - args = parser.parse_args() - - replace_chars(args.input_files, args.backup) diff --git a/site/scripts/tree.py b/site/scripts/tree.py deleted file mode 100644 index 594c724..0000000 --- a/site/scripts/tree.py +++ /dev/null @@ -1,87 +0,0 @@ -import argparse -import re - - -class Section: - def __init__(self, title, content, level, subsections=None): - self.subsections = subsections or [] - self.title = title - self.content = content - self.level = level - - def __str__(self): - return f"{'#' * self.level} {self.title}\n{self.content}\n" + "\n".join(str(sub) for sub in self.subsections) - - def __repr__(self): - return self.__str__() - - def append_subsection(self, subsection): - self.subsections.append(subsection) - - def remove_subsection(self, subsection): - self.subsections.remove(subsection) - - def remove_subsection_by_title(self, title): - for sub in self.subsections: - if sub.title == title: - self.subsections.remove(sub) - break - - def get_subsection(self, title): - for sub in self.subsections: - if sub.title == title: - return sub - return None - - def debug(self): - print(f"Title: {self.title}") - print(f"Content: {self.content}") - print(f"Level: {self.level}") - - - -class MarkdownTree: - def __init__(self, md): - self.md = md - self.root = Section("Root", "", 0) - self.sections = self.parse_sections(md, self.root) - - def __str__(self): - return "\n".join(str(section) for section in self.sections) - - def __repr__(self): - return self.__str__() - - def debug(self, section): - section.debug() - for sub in section.subsections: - self.debug(sub) - - - def parse_sections(self, md: str = None, parent: Section = None): - pass - -def main(args): - for path in args.input_paths: - with open(path) as f: - md = f.read() - tree = MarkdownTree(md) - print(tree) - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Markdown Tree Utility") - parser.add_argument('input_paths', nargs='+', help='List of input files/paths') - parser.add_argument('-r', '--recursive', action='store_true', help='Traverse directories recursively') - - args = parser.parse_args() - - main(args) - - # path = 'simple.md' - # with open (path) as f: - # md = f.read() - # tree = MarkdownTree(md) - # print(tree) - # tree.debug(tree.root) - diff --git a/site/signal-conditioning/lecture-notes/lecture-1.html b/site/signal-conditioning/lecture-notes/lecture-1.html deleted file mode 100644 index 7a2c218..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-1.html +++ /dev/null @@ -1,203 +0,0 @@ - - - - - - Lecture 1 - - - - - -
    - -

    Lecture 1

    -
    - Last modified: 2024-01-04 - -
    -
    -

    C-SWAP

    -

    Cost, Size, Weight and Power

    -

    Cost

    -

    Tradeoff between consumable vs reusable.

    -

    Size + Weight

    -

    Driven by power in many cases because of mass/volume to energy constraints

    -

    Power

    -

    How much time do you use between charges. Device dissipates heat (ROT 10 mW/cm^3 -> 2 deg C)

    -

    3 key sources: sensing, computing, communication.

    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/lecture-notes/lecture-2.html b/site/signal-conditioning/lecture-notes/lecture-2.html deleted file mode 100644 index da1ada6..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-2.html +++ /dev/null @@ -1,263 +0,0 @@ - - - - - - Lecture 2 - - - - - -
    - -

    Lecture 2

    -
    - Last modified: 2024-02-10 - -
    -
    -

    Lecture 2

    -

    Reminders:

    - -

    Electricity

    -
    +-------------+------+----------+
    -|1.5 V Battery|      |Light Bulb|
    -+-------------+------+----------+
    -        current: I ->
    -
    -

    Current is the amount of water flowing through the pipe, and Voltage is like the water pressure through the pipe.

    -

    If the "pressure" was 0, then the "flow" would be 0, and the same for the electrons. Battery is essentially using its chemical potential energy to push the electrons through the circuit to create a current.

    -

    Current (I)

    - -

    Coulomb

    - -

    Energy

    - -

    Energy = Q * V -> Joules (J)

    -

    Power

    -

    The rate of change of energy, measured in Joules/sec -> Watts (W)

    -

    Or the work done by the circuit

    -
    P = V * I
    -P = I^2 * R
    -P = V^2 / R
    -
    -

    Units

    -

    10^3 -> kilo K -10^6 -> mega M -10^9 -> giga G

    -

    10^-3 -> milli m -10^-6 -> micro \mu -10^-9 -> nano n -10^-12 -> pico p

    -

    Constant vs. Time Varying Circuits

    -

    Direct Current (DC): Voltages & Currents are constant, e.g. 1.5 V

    -

    Alternating Current (AC): Voltages and Currents are changing with time. e.g. Home power

    -
    General formula:
    -v(t) - 5 * cos(w * t)
    -w = 2 * pi * f
    -
    -AC Line Voltage (wall power):
    -V(t) = 120 * cos(2 * pi * 60 Hz * t)
    -
    -

    Can use nodal analysis to break down and black box circuits. Mathematically represent each node in a circuit, and then use abstraction to model complicated circuits.

    -

    Problems

    -

    (1.a) An iPhone contains a Lithium Battery with a voltage of 3.8 V, and a capacity of 2900 mAH (milli-Amp-hours). How much energy is stored in the battery? In Joules, and Watt-Hours.

    -

    2900 mAH * 3.8 V = 11,020 mWH = 11.02 WH

    -

    11.02 WH * 3600 sec/H = 39.67 kJ

    -

    (1.b) Apple claims that an iPhone lasts 16 days on standby mode. What is the average power consumed by the device in standby?

    -

    2900 mAH * 3600 sec/H = 10.4 * 10^6 mAsec

    -

    16 days = 1.4 * 10^6 sec

    -

    I = (10.4 * 10^6 mAsec)/(1.4 * 10^6 sec) = 7.552 mA

    -

    Power = 3.8 V * 7.552 mA = 28.7 mW

    -

    (2.a) A Tesla Model S has a battery capacity of 100 kWH. How many Joules of energy are stored in the battery?

    -

    100 kWH * 3600 sec/H = 360,000,000 J = 360 MJ

    -

    (2.b) Bright sunlight has a power density of roughly 1 kW/m^2. A solar panel has 20% efficiency in converting sunlight to energy. How long would it take to fully recharge the car if it were covered in solar panels with a surface area of 10 m^2?

    -

    1 kW/m^2 * 10 m^2 * .2 = 2 kW = 2 kJ/sec = 2000 J/sec

    -

    360,000,000 / 2000 J/sec = 180,000 sec = 50 hours

    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/lecture-notes/lecture-3.html b/site/signal-conditioning/lecture-notes/lecture-3.html deleted file mode 100644 index 4f743d5..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-3.html +++ /dev/null @@ -1,220 +0,0 @@ - - - - - - Lecture 3 - - - - - -
    - -

    Lecture 3

    -
    - Last modified: 2024-01-23 - -
    -
    -

    Lecture 3

    -

    Resistance

    -

    Resistance is the opposition to the flow of current. It is measured in Ohms (\Omega), and is the ratio of voltage to current. Is usually an intrinsic property of the material.

    -

    Ohm's Law

    -

    $$V = IR$$

    -

    Power

    -

    The rate of change of energy, measured in Joules/sec -> Watts (W)

    -

    $$1$$

    -

    Power and Resistance

    -

    The power dissipated by a resistor is the product of the voltage across it and the current through it. Intuitively, this makes sense because it takes energy to push electrons through a resistor, and the more electrons you push through, the more energy you use. Similarly, holding I constant, the power is directly proportional to the resistance.

    -

    $ P = IV = I^2R = V^2/R $

    -

    Resistors in Series and Parallel

    -

    Series

    -

    Resistors in series are connected end-to-end. The total resistance is the sum of the individual resistances.

    -

    $$R_t = R_1 + R_2 + \ldots + R_n$$

    -

    The current through each resistor is the same because the current is the flow of electrons, and the electrons can't go anywhere else.

    -

    The voltage across each resistor can be different, but the total voltage is the sum of all the voltages across each resistor. This can be derived from the above two properties and Ohm's law.

    -

    $$V_t = V_1 + V_2 + \ldots + V_n$$

    -

    $$V_t = IR_t = I(R_1 + R_2 + \ldots + R_n) = IR_1 + IR_2 + \ldots + IR_n = V_1 + V_2 + \ldots + V_n$$

    -

    Parallel

    -

    Connected such that both ends of each resistor are connected to the same two points of the circuit. The resistance of the circuit is the reciprocal of the sum of the reciprocals of the individual resistances.

    -

    $$R_t = 1/(1/R_1 + 1/R_2 + \ldots + 1/R_n)$$

    -

    The voltage across each resistor is the same because the voltage is the difference in potential between two points, and the two points are the same for each resistor (equal to the source voltage).

    -

    The current through each resistor can be different, but the total current from the two endpoints is the sum of all the currents through each resistor. This can be derived from the above two properties and Ohm's law, but also from the fact that the current is the flow of electrons, and the electrons can't go anywhere else.

    -

    $$I_t = I_1 + I_2 + \ldots + I_n$$

    -

    $$I_t = V_t/R_t = V_t/(1/(1/R_1 + 1/R_2 + \ldots + 1/R_n)) = V_t/(R_1 + R_2 + \ldots + R_n) = V_t/R_1 + V_t/R_2 + \ldots + V_t/R_n = I_1 + I_2 + \ldots + I_n$$

    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/lecture-notes/lecture-4.html b/site/signal-conditioning/lecture-notes/lecture-4.html deleted file mode 100644 index 90fe34b..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-4.html +++ /dev/null @@ -1,243 +0,0 @@ - - - - - - Lecture 4 - - - - - -
    - -

    Lecture 4

    -
    - Last modified: 2024-01-23 - -
    -
    -

    Lecture 4

    -

    Power Dissipation In a Resistor

    -

    When a resistor starts getting warm, it is leaking power.

    -
    P = V * I = I^2 * R = V^2/R
    -
    -

    KVL

    -

    Voltage: The sum of all the voltages on any path around a circuit are equal to zero.

    -

    Current: The current arriving at any node is the same as the current leaving that node.

    -

    Example

    -
          R_1 = 1 Ohm
    -  +--/\/\/|--+
    -  |          |
    -  _          |
    -  + 120 V    |
    -  -          |
    -  |          |
    -  +--\/\/\|--+
    -      R_2 = 10 Ohm
    -
    -

    $$V_1 = I * R_1$$ -$$V_2 = I * R_2$$

    -

    Need $P_2$ using $V/R$

    -

    $$P_1 = I^2 * R$$

    -

    $$V_L = V * R_2 / (R_ 1 + R_2) = 120 * 10/11 = 109 V$$

    -

    $$P_2 = V^2/R = (109)^2/10 = 1188 W$$

    -

    $$I_1 = V_1 / R_1 = (V - V_L) / R_1$$

    -

    $$= (120 - 109) / 1 = 11 A$$

    -

    $$P_1 = I_1^2 * R_1 = 11^2 * 1 = 121 W$$

    -

    Sources

    -

    A voltage source produces a constant voltage, regardless of current (an idealistic assumption).

    -

    A current source delivers constant current to a circuit. Current by load resistance (impedence)

    -

    Situation

    -

    We previously estimated an iPhone in standby draws a current of 7.55 mA from its 3.8 V Lithium battery.

    -

    Circuit

    -
    
    -ground
    -  ^
    -  |     battery
    -  |       |
    -  |       |
    -  +----- |-  +|-----+ 
    -  |                 | 
    -  |                 | 
    -  +-----|/\/\/|-----+
    -           |
    -           | 
    - phone power consumption
    -
    -
    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/lecture-notes/lecture-5.html b/site/signal-conditioning/lecture-notes/lecture-5.html deleted file mode 100644 index 8fd84c0..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-5.html +++ /dev/null @@ -1,210 +0,0 @@ - - - - - - Lecture 5 - - - - - -
    - -

    Lecture 5

    -
    - Last modified: 2024-01-23 - -
    -
    -

    Thevenin's Theorem

    -

    Any two-terminal circuit with only linear elements (resistors) and sources can be replaced by an equivalent circuit consisting of a voltage source in series with a resistor.

    -

    Voltage source represented as $V_{th}$, and resistor as $R_{th}$.

    -

    Algorithm

    -
      -
    1. Find $V_{th}$, the open circuit voltage from A to B using KCL and KVL.
    2. -
    3. Find $R_{th}$, the equiv resistance between A and B by replacing voltage sources with short circuits and current sources with open circuits.
    4. -
    -

    Norton's Theorem

    -

    Any two-terminal circuit with only linear elements (resistors) and sources can be replaced by an equivalent circuit consisting of a current source in parallel with a resistor.

    -

    Current source represented as $I_{no}$, and resistor as $R_{no}$.

    -

    Algorithm

    -
      -
    1. find $I_{no}$, the short circuit current from A to B using KCL and KVL.
    2. -
    3. find $R_{no}$, the equiv shunt (parallel with source) resistance between A and B by replacing voltage sources with short circuits and current sources with open circuits.
    4. -
    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/lecture-notes/lecture-6.html b/site/signal-conditioning/lecture-notes/lecture-6.html deleted file mode 100644 index 04b2335..0000000 --- a/site/signal-conditioning/lecture-notes/lecture-6.html +++ /dev/null @@ -1,253 +0,0 @@ - - - - - - Lecture 6 - - - - - -
    - -

    Lecture 6

    -
    - Last modified: 2024-02-27 - -
    -
    -

    Capacitors

    -

    Stores energy in electric field between two plates.

    -

    Passes AC, blocks DC. "High pass filter"

    -

    Capacitance is the ratio of charge to voltage, measured in Farads. -$$ -C = \frac{Q}{V} -$$

    -

    The energy stored in a capacitor is -$$ -E = \frac{1}{2}CV^2 -$$

    -

    $$ -\frac{d}{dt}Q = \frac{d}{dt}CV -$$

    -

    $$ -I_C = C\frac{dV}{dt} -$$

    -

    Inductors

    -

    Stores energy in magnetic field around a coil of wire due to current flowing through it. Units are Henrys (H).

    -

    Passes DC, blocks AC. "Low pass filter"

    -

    Energy stored in an inductor is -$$ -E = \frac{1}{2}LI^2 -$$

    -

    $$ -V_L = L\frac{dI}{dt} -$$

    -

    Complex Numbers Review

    -

    $$e = 2.71828...$$

    -

    $$ -j \cdot j = -1 \to j = \sqrt{-1} -$$

    -

    $$ -e^{jq} = \cos(q) + j\sin(q) -$$

    -

    Complex Numbers for AC Signals

    - -

    EX:

    -

    $$ -cos(2\pi ft + \phi) = e^{j\phi}e^{j2\pi ft} -$$

    -

    Impedance

    -

    The AC version of resistance.

    -

    $$ -Z_{cap} = \frac{1}{j\omega C} -$$

    -

    $$ -Z_{ind} = j\omega L -$$

    -

    $$ -Z_{res} = R -$$

    -

    Phasors

    -

    Complex numbers that represent the amplitude and phase of a sinusoidal signal. Can be used to represent AC signals and greatly simplify analysis by

    -
    - -
    - - \ No newline at end of file diff --git a/site/signal-conditioning/syllabus.html b/site/signal-conditioning/syllabus.html deleted file mode 100644 index 0a64972..0000000 --- a/site/signal-conditioning/syllabus.html +++ /dev/null @@ -1,1391 +0,0 @@ - - - - - - -EE205 Winter 2024-syllabus-R1.docx - - - - - - -
    - -

    UW EE 205 Winter 2024 - Introduction to Signal -Conditioning

    - -

    Lecture: M/W/F 9:30am - 10:20am, -HRC 145 Lab: Tu 3:30pm - 6:20pm, ECE 137

    - -

    Instructor: Prof. Matt Reynolds, M 226 ECE,

    - -

    Instructor Office Hours: Mon 10:30 - 11:30am, or Zoom via -appointment

    - -

    TA Office Hour #1: (Wed 2:30-3:30pm) (Lab TA) Ms. Sara -Reyes,

    - -

    TA Office Hour #2: (Thurs 10:00-11:00am) (Problem Session -TA) Mr. Kevin Ho,

    - -

    Course Summary

    - -

    This course will provide a hands-on introduction to analog -sensors and analog circuits, and how they are interfaced with digital systems -to create "mixed signal" systems. Topics include Kirchhoff's Laws, -independent and dependent voltage and current sources, resistors, capacitors, -inductors, bipolar and MOS transistors, op amps, and how they can be used to -implement amplification, attenuation, filtering, sampling, etc. We will also -discuss sources of noise and how noise can be mitigated in analog and digital -processing. Students will design and simulate analog circuits in SPICE and use -these circuits to interface analog sensors with a microcontroller to perform -digital processing of sensor signals.

    - -

    Prerequisites

    - -

    The course is intended for non-EE majors, and enrollment is -open to computer engineering and computer science students only. Prerequisite: -either MATH 126 or MATH 136; and either PHYS 122 or PHYS 142.

    - -

    Students will need to provide a Mac or Windows laptop -capable of running LTspice, the Arduino IDE, and Python (Jupyter Notebook).

    - -

    LTspice:

    - -

    https://www.analog.com/en/design-center/design-tools-and-calculators/ltspice-simulator.html -Arduino IDE:

    - -

    https://www.arduino.cc/en/software -Jupyter Notebook:

    - -

    https://jupyter.org/install

    - -

    Textbook

    - -

    Practical Electronics for Inventors, 4th ed.; Scherz & -Monk, 2016. ISBN 978-1259587542

    - -

    Grading

    - -

    25% Homework Assignments (5 assignments)

    - -

    30% Lab Assignments (5 assignments)

    - -

    15% Midterm Exam In-Class Monday Feb 5, -9:30-10:20am, Room HRC 145 30% Final Exam - Wednesday March 13, -8:30-10:20am, Room HRC 145

    - -

    Homework and Lab Report Policy

    - -

    Homework and Lab Reports are due via PDF upload to -Canvas by 9:00am on the Friday it is due. To be fair to everyone, the -acceptance window closes at 9:00am, as we will be discussing homework/lab -solutions in class. There will therefore be no credit for late homework/labs, -however your lowest homework score and your lowest lab score will be dropped.

    - -

    Course Delivery Policy

    - -

    The course will be delivered in-person. For maximum -flexibility, and to accommodate student travel, illness, or other absences, -lectures will be recorded and made available as video recordings via Canvas on -a best-efforts basis. PDF lecture notes will be posted to Canvas after each -lecture.

    - -

    Policy on Group Work and Academic Integrity

    - -

    Preparation and delivery of Homework shall be individual -effort. You are encouraged to study and consult with others, but all -homework solutions must be your own. Any use of outside resources (e.g. -assistance given by others, Web searches, other online resources) must be -identified and annotated alongside your solution. The use of ChatGPT or -similar AI tools to assist with the completion of Homework or Exam problems is -not permitted.

    - -

    Preparation and delivery of Lab Assignments shall be a -team effort. A summary of the contributions of each team member to the Lab -Assignments shall be supplied as an addendum to each lab report. The use of -ChatGPT or similar AI tools to assist with the completion of Lab problems is -not permitted.

    - -

    At all times, students are expected to adhere to the -University of Washington Student Code of Conduct, Washington Administrative -Code (WAC) 478-121, and are expected to properly credit the work of others in -all assignments and interactions with the instructor and other members of the -class. Any suspected instances of academic misconduct will be reported in -accordance with these policies. https://www.engr.washington.edu/current/policies/academic-integrity-misconduct

    - -

    Policy on Religious Accommodation

    - -

    "Washington state law requires that UW develop a policy for -accommodation of student absences or significant hardship due to reasons of -faith or conscience, or for organized religious activities. The UW's policy, -including more information about how to request an accommodation, is available -at Religious Accommodations Policy

    - -

    (https://registrar.washington.edu/staffandfaculty/religious-accommodations-policy/).

    - -

    Accommodations must be requested within the first two weeks -of this course using the Religious

    - -

    Accommodations Request form

    - -

    (https://registrar.washington.edu/students/religious-accommodations-request/)."

    - -
    - -
    -
    - -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Lec.

    -
    -

    Date

    -
    -

    Topic

    -
    -

    Reading

    -
    -

    Assignments

    -
    -

    1

    -
    -

    Wed 1/3

    -
    -

    Course Overview, design tools, Units and symbols

    -
    -

    Syllabus

    -

    Scherz Ch 1

    -
    -

     

    -
    -

     

    -
    -

    Fri 1/5

    -
    -

    Problem Session #1

    -
    -

     

    -
    -

     

    -
    -

    2

    -
    -

    Mon 1/8

    -
    -

    Resistance, Ohm's Law, power, resistor combinations

    -
    -

    Scherz 2.8 - 2.12, 3.5

    -
    -

    Lab 1 assigned (Intro. Instruments)

    -

    HW 1 assigned

    -
    -

    3

    -
    -

    Wed 1/10

    -
    -

    [Sara] DC sources, Voltage and current dividers, Thevenin - and Norton theorems

    -
    -

    Scherz 2.12 - 2.19

    -
    -

     

    -
    -

     

    -
    -

    Fri 1/12

    -
    -

    [Kevin] Problem Session #2

    -
    -

     

    -
    -

    HW 1 due, 9:00am SHARP

    -
    -

     

    -
    -

    Mon 1/15

    -
    -

    *** University Holiday - No Lecture - Dr. Martin Luther - King Day ***

    -
    -

    4

    -
    -

    Wed 1/17

    -
    -

    [Kevin] Intro to AC circuits and time varying signals

    -
    -

    Scherz 2.20 - 2.24

    -
    -

     

    -
    -

     

    -
    -

    Fri 1/19

    -
    -

    Problem Session #3

    -
    -

     

    -
    -

    Lab 1 due (Intro. Instruments)

    -
    -

    5

    -
    -

    Mon 1/22

    -
    -

    Capacitors and Inductors

    -
    -

    Scherz 2.20 - 2.24

    -
    -

    HW 2 assigned

    -
    -

    6

    -
    -

    Wed 1/24

    -
    -

    AC circuits w/ R L C

    -
    -

    Scherz 2.25 - 2.29

    -
    -

     

    -
    -

     

    -
    -

    Fri 1/26

    -
    -

    Problem Session #4

    -
    -

     

    -
    -

    HW 2 due, 9:00am SHARP

    -
    -

    7

    -
    -

    Mon 1/29

    -
    -

    Semiconductor Devices: Diodes

    -
    -

    Scherz 4.1 - 4.2

    -
    -

    Lab 2 assigned (RLC circuits) HW 3 assigned

    -
    -

    8

    -
    -

    Wed 1/31

    -
    -

    Bipolar junction transistors switches and amplifiers

    -
    -

    Scherz 4.3

    -
    -

     

    -
    -

     

    -
    -

    Fri 2/2

    -
    -

    Problem Session #5

    -
    -

    Midterm Exam Review

    -
    -

    HW 3 due, 9:00am SHARP

    -
    -

     

    -
    -

    Mon 2/5

    -
    -

    *** MIDTERM EXAM - In Class (50 min) ***

    -
    -

    9

    -
    -

    Wed 2/7

    -
    -

    FETs - switches and amplifiers

    -
    -

    Scherz 4.3

    -
    -

     

    -
    -

     

    -
    -

    Fri 2/9

    -
    -

    Problem Session #6

    -
    -

     

    -
    -

    Lab 2 due (RLC circuits)

    -
    - -

     

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Lec.

    -
    -

    Date

    -
    -

    Topic

    -
    -

    Reading

    -
    -

    Assignments

    -
    -

    10

    -
    -

    Mon 2/12

    -
    -

    Op amps - amplifiers and analog filters

    -
    -

    Scherz 8.1 - 8.18

    -
    -

    HW 4 assigned

    -

    Lab 3 assigned (opamps)

    -
    -

    11

    -
    -

    Wed 2/14

    -
    -

    Op amps - amplifiers and analog filters

    -
    -

    Scherz 8.1 - 8.18

    -
    -

     

    -
    -

     

    -
    -

    Fri 2/16

    -
    -

    Problem Session #7

    -
    -

     

    -
    -

    HW 4 due, 9:00am SHARP

    -
    -

     

    -
    -

    Mon 2/19

    -
    -

    *** University Holiday - No Lecture - Presidents' Day - ***

    -
    -

    12

    -
    -

    Wed 2/21

    -
    -

    Oscillators

    -
    -

    Scherz 10.1 - 10.7

    -
    -

    Lab 4 assigned (oscillators)

    -
    -

     

    -
    -

    Fri 2/23

    -
    -

    Problem Session #8

    -
    -

     

    -
    -

    Lab 3 due (opamps)

    -
    -

    13

    -
    -

    Mon 2/26

    -
    -

    Voltage regulators and power supplies

    -
    -

    Scherz 11.1 - 11.11

    -
    -

    HW 5 assigned

    -
    -

    14

    -
    -

    Wed 2/28

    -
    -

    Analog-to-digital conversion

    -
    -

    Scherz 12.9

    -
    -

     

    -
    -

     

    -
    -

    Fri 3/1

    -
    -

    Problem Session #9

    -
    -

     

    -
    -

    HW 5 due, 9:00am SHARP

    -

    Lab 4 due (oscillators)

    -

    Lab 5 assigned (Sig Proc w/

    -

    Arduino)

    -
    -

    15

    -
    -

    Mon 3/4

    -
    -

    Digital signal processing

    -
    -

     

    -
    -

     

    -
    -

    16

    -
    -

    Wed 3/6

    -
    -

    Digital signal processing

    -
    -

     

    -
    -

     

    -
    -

     

    -
    -

    Fri 3/8

    -
    -

    Problem Session #10

    -
    -

    Final Exam Review

    -
    -

    Lab 5 due (Sig Proc w/ Arduino)

    -
    -

     

    -
    -

     

    -
    -

     

    -
    -

     

    -
    -

     

    -
    -

     

    -
    -

    *** FINAL EXAM *** - Wednesday March 13, 8:30am-10:20am, - Room HRC 145 ***

    -
    - -

    EE 205 A - Winter 2024 - Intro. to Signal -Conditioning - Syllabus v1.0

    - -
    - - - - diff --git a/site/solutions/clean.py b/site/solutions/clean.py deleted file mode 100644 index 91d13b5..0000000 --- a/site/solutions/clean.py +++ /dev/null @@ -1,23 +0,0 @@ -import os -import re - -def remove_comments_from_java(file_path): - """Remove Java comments from the given file.""" - with open(file_path, 'r') as f: - content = f.read() - - pattern = r'(//.*?$|/\*.*?\*/|/\*.*?$)' - cleaned_content = re.sub(pattern, '', content, flags=re.MULTILINE|re.DOTALL) - with open(file_path, 'w') as f: - f.write(cleaned_content) - -def main(): - for root, _, files in os.walk('.'): - for file_name in files: - if file_name.endswith('.java'): - file_path = os.path.join(root, file_name) - remove_comments_from_java(file_path) - print(f"Removed comments from {file_path}") - -if __name__ == "__main__": - main() diff --git a/site/solutions/java/LeetCode75/AsteroidCollisions.java b/site/solutions/java/LeetCode75/AsteroidCollisions.java deleted file mode 100644 index 9066191..0000000 --- a/site/solutions/java/LeetCode75/AsteroidCollisions.java +++ /dev/null @@ -1,31 +0,0 @@ -package LeetCode75; - -import java.util.Stack; - -public class AsteroidCollisions { - public int[] asteroidCollision(int[] asteroids) { - Stack r = new Stack<>(); - - for (int curr : asteroids) { - if (curr > 0 || (curr < 0 && (r.isEmpty() || r.peek() < 0))) - r.push(curr); - else { - while (Math.abs(r.peek()) < Math.abs(curr)) { - r.pop(); - if (r.isEmpty() || r.peek() < 0) { - r.push(curr); - break; - } - } - - if (Math.abs(r.peek()) == Math.abs(curr) && r.peek() > curr) - r.pop(); - } - - } - int[] res = new int[r.size()]; - for (int i = res.length - 1; i >= 0; i--) - res[i] = r.pop(); - return res; - } -} diff --git a/site/solutions/java/LeetCode75/CanPlaceFlowers.java b/site/solutions/java/LeetCode75/CanPlaceFlowers.java deleted file mode 100644 index ebdc3fc..0000000 --- a/site/solutions/java/LeetCode75/CanPlaceFlowers.java +++ /dev/null @@ -1,27 +0,0 @@ -package LeetCode75; - -public class CanPlaceFlowers { - public boolean canPlaceFlowers(int[] flowerbed, int n) { - if (n <= 0) - return true; - if (n > flowerbed.length) - return false; - if (n == 1 && flowerbed.length == 1) - return flowerbed[0] == 0; - - int count = 0; - if (flowerbed[0] == 0 && flowerbed[1] == 0) - flowerbed[count++] = 1; - - for (int curr = 2; curr < flowerbed.length; curr++) { - if (flowerbed[curr] == 0 && flowerbed[curr - 1] == 0 && flowerbed[curr - 2] == 0) { - flowerbed[curr - 1] = 1; - count++; - } - } - - if (flowerbed[flowerbed.length - 1] == 0 && flowerbed[flowerbed.length - 2] == 0) - count++; - return count >= n; - } -} diff --git a/site/solutions/java/LeetCode75/Compress.java b/site/solutions/java/LeetCode75/Compress.java deleted file mode 100644 index 1431712..0000000 --- a/site/solutions/java/LeetCode75/Compress.java +++ /dev/null @@ -1,22 +0,0 @@ -package LeetCode75; - -public class Compress { - public int compress(char[] chars) { - if (chars.length == 1) - return 1; - int curr = 0, length = 1; - for (int i = 0; i < chars.length; i++, length = 1) { - chars[curr++] = chars[i]; - while (i < chars.length - 1 && chars[i] == chars[i + 1]) { - length++; - i++; - } - - if (length > 1) - for (char c : ("" + length).toCharArray()) - chars[curr++] = c; - } - - return curr; - } -} diff --git a/site/solutions/java/LeetCode75/CostToHireKWorkers.java b/site/solutions/java/LeetCode75/CostToHireKWorkers.java deleted file mode 100644 index 254bc9b..0000000 --- a/site/solutions/java/LeetCode75/CostToHireKWorkers.java +++ /dev/null @@ -1,74 +0,0 @@ -package LeetCode75; - -import java.util.Arrays; -import java.util.PriorityQueue; - -public class CostToHireKWorkers { - public long totalCost(int[] costs, int k, int candidates) { - PriorityQueue pq = new PriorityQueue<>((a, b) -> a.compare(b)); - boolean[] visited = new boolean[costs.length]; - Arrays.fill(visited, false); - - int lo = 0, hi = costs.length - 1, count = 0; - long total = 0; - - while (lo < candidates) { - if (pq.size() == costs.length) - break; - if (hi == lo) { - visited[hi] = true; - pq.add(new Pair(hi, costs[hi])); - break; - } - visited[lo] = !visited[lo]; - visited[hi] = !visited[hi]; - pq.add(new Pair(lo, costs[lo++])); - pq.add(new Pair(hi, costs[hi--])); - } - - if (visited[lo]) { - for (; count < k; count++) - total += pq.remove().c; - return total; - } - - while (count++ < k) { - Pair next = pq.remove(); - total += next.c; - if (visited[lo] || visited[hi]) { - continue; - } - - if (next.i <= lo) { - visited[lo] = true; - pq.add(new Pair(lo, costs[lo])); - lo++; - } else if (next.i > hi) { - visited[hi] = true; - pq.add(new Pair(hi, costs[hi])); - hi--; - } - } - return total; - - } - - public class Pair { - int i, c; - - public Pair(int index, int cost) { - i = index; - c = cost; - } - - public int compare(Pair other) { - return this.c != other.c - ? this.c - other.c - : this.i - other.i; - } - - public String toString() { - return "(" + this.i + ", " + this.c + ")"; - } - } -} diff --git a/site/solutions/java/LeetCode75/EatAllBananas.java b/site/solutions/java/LeetCode75/EatAllBananas.java deleted file mode 100644 index f15d125..0000000 --- a/site/solutions/java/LeetCode75/EatAllBananas.java +++ /dev/null @@ -1,33 +0,0 @@ -package LeetCode75; - -public class EatAllBananas { - public int minEatingSpeed(int[] piles, int h) { - int maxPileSize = piles[0]; - for (int n : piles) - maxPileSize = Math.max(n, maxPileSize); - if (piles.length == h) - return maxPileSize; - return findOptimalSpeed(0, maxPileSize, piles, h); - - } - - private int findOptimalSpeed(int lowSpeed, int highSpeed, int[] piles, int h) { - int midSpeed = lowSpeed + (highSpeed - lowSpeed) / 2; - if (highSpeed == midSpeed) - return highSpeed; - - int time = 0; - for (int pile : piles) { - - time += Math.ceil((double) pile / midSpeed); - - if (time > h) - break; - - } - - return time > h - ? findOptimalSpeed(midSpeed + 1, highSpeed, piles, h) - : findOptimalSpeed(lowSpeed, midSpeed, piles, h); - } -} diff --git a/site/solutions/java/LeetCode75/FindPeakElement.java b/site/solutions/java/LeetCode75/FindPeakElement.java deleted file mode 100644 index 6e4fc1f..0000000 --- a/site/solutions/java/LeetCode75/FindPeakElement.java +++ /dev/null @@ -1,16 +0,0 @@ -package LeetCode75; - -public class FindPeakElement { - public int findPeakElement(int[] nums) { - int start = 0, end = nums.length - 1, mid; - - while (start < end) { - mid = start + (end - start - 1) / 2; - if (nums[mid] < nums[mid + 1]) - start = mid + 1; - else - end = mid; - } - return start; - } -} diff --git a/site/solutions/java/LeetCode75/GCDOfStrings.java b/site/solutions/java/LeetCode75/GCDOfStrings.java deleted file mode 100644 index 554ea6f..0000000 --- a/site/solutions/java/LeetCode75/GCDOfStrings.java +++ /dev/null @@ -1,12 +0,0 @@ -package LeetCode75; - -public class GCDOfStrings { - - public String gcdOfStrings(String str1, String str2) { - return (!(str1 + str2).equals(str2 + str1)) ? "" : str1.substring(0, gcd(str1.length(), str2.length())); - } - - private int gcd(int a, int b) { - return (b == 0) ? a : gcd(b, a % b); - } -} diff --git a/site/solutions/java/LeetCode75/HouseRobber.java b/site/solutions/java/LeetCode75/HouseRobber.java deleted file mode 100644 index 11b4ff0..0000000 --- a/site/solutions/java/LeetCode75/HouseRobber.java +++ /dev/null @@ -1,16 +0,0 @@ -package LeetCode75; - -public class HouseRobber { - public int rob(int[] nums) { - if (nums.length == 1) - return nums[0]; - - nums[1] = Math.max(nums[0], nums[1]); - for (byte i = 2; i < nums.length; i++) { - nums[i] = Math.max(nums[i - 2] + nums[i], nums[i - 1]); - } - - return Math.max(nums[nums.length - 1], nums[nums.length - 2]); - } - -} diff --git a/site/solutions/java/LeetCode75/KidWithMostCandies.java b/site/solutions/java/LeetCode75/KidWithMostCandies.java deleted file mode 100644 index 0b7c1a5..0000000 --- a/site/solutions/java/LeetCode75/KidWithMostCandies.java +++ /dev/null @@ -1,16 +0,0 @@ -package LeetCode75; - -import java.util.ArrayList; -import java.util.List; - -public class KidWithMostCandies { - public List kidsWithCandies(int[] candies, int extraCandies) { - List l = new ArrayList<>(); - int max = Integer.MIN_VALUE; - for (int n : candies) - max = Math.max(max, n); - for (int n : candies) - l.add(n + extraCandies >= max); - return l; - } -} diff --git a/site/solutions/java/LeetCode75/KthLargestElement.java b/site/solutions/java/LeetCode75/KthLargestElement.java deleted file mode 100644 index a2cfd2c..0000000 --- a/site/solutions/java/LeetCode75/KthLargestElement.java +++ /dev/null @@ -1,17 +0,0 @@ -package LeetCode75; - -public class KthLargestElement { - public int findKthLargest(int[] nums, int k) { - int[] buckets = new int[20001]; - for (int n : nums) - buckets[n + 10000]++; - - for (int i = buckets.length - 1; i >= 0; i--) { - if (buckets[i] > 0) - k -= buckets[i]; - if (k <= 0) - return i - 10000; - } - return -1; - } -} diff --git a/site/solutions/java/LeetCode75/LengthOfLongestSubstring.java b/site/solutions/java/LeetCode75/LengthOfLongestSubstring.java deleted file mode 100644 index 325e46d..0000000 --- a/site/solutions/java/LeetCode75/LengthOfLongestSubstring.java +++ /dev/null @@ -1,24 +0,0 @@ -package LeetCode75; - -import java.util.HashSet; -import java.util.Set; - -public class LengthOfLongestSubstring { - public int lengthOfLongestSubstring(String s) { - int lo = 0, hi = 0, max = 0; - Set set = new HashSet<>(); - - while (hi < s.length()) { - char curr = s.charAt(hi); - - if (!set.contains(curr)) - set.add(s.charAt(hi++)); - else - set.remove(s.charAt(lo++)); - - max = Math.max(max, hi - lo); - } - - return max; - } -} diff --git a/site/solutions/java/LeetCode75/LetterCombinationsOfPhoneNumber.java b/site/solutions/java/LeetCode75/LetterCombinationsOfPhoneNumber.java deleted file mode 100644 index 637a697..0000000 --- a/site/solutions/java/LeetCode75/LetterCombinationsOfPhoneNumber.java +++ /dev/null @@ -1,35 +0,0 @@ -package LeetCode75; - -import java.util.*; - -public class LetterCombinationsOfPhoneNumber { - List list = new ArrayList<>(); - Map map = new HashMap<>(); - - public List letterCombinations(String digits) { - if (digits.length() == 0) - return this.list; - for (int i = 2; i < 7; i++) { - map.put(i, new char[3]); - for (int j = 0; j < 3; j++) - map.get(i)[j] = ((char) ((int) 'a' + (i - 2) * 3 + j)); - } - map.put(7, new char[] { 'p', 'q', 'r', 's' }); - map.put(8, new char[] { 't', 'u', 'v' }); - map.put(9, new char[] { 'w', 'x', 'y', 'z' }); - StringBuilder sb = new StringBuilder(digits); - fillList(digits, sb, 0); - return this.list; - } - - private void fillList(String digits, StringBuilder curr, int i) { - if (i == digits.length()) - this.list.add(curr.toString()); - else { - for (char c : this.map.get(Integer.parseInt("" + digits.charAt(i)))) { - curr.setCharAt(i, c); - fillList(digits, curr, i + 1); - } - } - } -} diff --git a/site/solutions/java/LeetCode75/MergeStringsAlternately.java b/site/solutions/java/LeetCode75/MergeStringsAlternately.java deleted file mode 100644 index 60fbf37..0000000 --- a/site/solutions/java/LeetCode75/MergeStringsAlternately.java +++ /dev/null @@ -1,18 +0,0 @@ -package LeetCode75; - -class MergeStringsAlternately { - public String mergeAlternately(String word1, String word2) { - StringBuilder sb = new StringBuilder(); - int minLength = Math.min(word1.length(), word2.length()); - - for (int i = 0; i < minLength; i++) - sb.append(word1.charAt(i) + "" + word2.charAt(i)); - - if (minLength == word1.length()) - sb.append(word2.substring(minLength, word2.length())); - else - sb.append(word1.substring(minLength, word1.length())); - - return sb.toString(); - } -} \ No newline at end of file diff --git a/site/solutions/java/LeetCode75/NearestExit.java b/site/solutions/java/LeetCode75/NearestExit.java deleted file mode 100644 index 034c34e..0000000 --- a/site/solutions/java/LeetCode75/NearestExit.java +++ /dev/null @@ -1,78 +0,0 @@ -package LeetCode75; - -import java.util.LinkedList; -import java.util.Queue; - -public class NearestExit { - public int nearestExit(char[][] maze, int[] entrance) { - int m = maze.length, n = maze[0].length; - - boolean[][] visited = new boolean[m][n]; - for (int r = 0; r < m; r++) - for (int c = 0; c < n; c++) - visited[r][c] = (maze[r][c] == '+') ? true : false; - - Queue q = new LinkedList<>(); - q.add(new Pair(entrance[0], entrance[1])); - - int moves = 0; - - while (!q.isEmpty()) { - int size = q.size(); - for (int i = 0; i < size; i++) { - Pair curr = q.remove(); - if (visited[curr.row][curr.col]) - continue; - - visited[curr.row][curr.col] = true; - - if (isExit(m, n, curr) && - (curr.row != entrance[0] || curr.col != entrance[1])) - return moves; - - for (Pair adj : curr.getSurrounding()) - if (isInMaze(m, n, adj) && !visited[adj.row][adj.col]) - q.add(adj); - } - moves++; - } - - return -1; - } - - private boolean isInMaze(int m, int n, Pair p) { - return p.row < m && - p.col < n && - p.row >= 0 && - p.col >= 0; - } - - private boolean isExit(int m, int n, Pair p) { - return p.row == m - 1 || - p.col == n - 1 || - p.row == 0 || - p.col == 0; - } - - private class Pair { - public int row, col; - - public Pair(int r, int c) { - this.row = r; - this.col = c; - } - - public String toString() { - return "row: " + this.row + ", col: " + this.col; - } - - public Pair[] getSurrounding() { - return new Pair[] { - new Pair(this.row + 1, this.col), - new Pair(this.row - 1, this.col), - new Pair(this.row, this.col + 1), - new Pair(this.row, this.col - 1) - }; - } - } -} diff --git a/site/solutions/java/LeetCode75/ReverseVowels.java b/site/solutions/java/LeetCode75/ReverseVowels.java deleted file mode 100644 index e35b097..0000000 --- a/site/solutions/java/LeetCode75/ReverseVowels.java +++ /dev/null @@ -1,50 +0,0 @@ -package LeetCode75; - -import java.util.HashSet; -import java.util.Set; - -public class ReverseVowels { - public String reverseVowels(String s) { - if (s.length() == 1) - return s; - - Set v = new HashSet<>(); - v.add('a'); - v.add('e'); - v.add('i'); - v.add('o'); - v.add('u'); - v.add('A'); - v.add('E'); - v.add('I'); - v.add('O'); - v.add('U'); - - StringBuilder sb = new StringBuilder(); - int left = 0, right = s.length() - 1; - char[] arr = s.toCharArray(); - - while (left < right) { - if (!v.contains(arr[left])) - left++; - else if (!v.contains(arr[right])) - right--; - else { - arr = swap(arr, left, right); - left++; - right--; - } - } - - for (char c : arr) - sb.append(c); - return sb.toString(); - } - - private char[] swap(char[] s, int a, int b) { - char temp = s[a]; - s[a] = s[b]; - s[b] = temp; - return s; - } -} diff --git a/site/solutions/java/LeetCode75/RottingOranges.java b/site/solutions/java/LeetCode75/RottingOranges.java deleted file mode 100644 index 1d23eea..0000000 --- a/site/solutions/java/LeetCode75/RottingOranges.java +++ /dev/null @@ -1,63 +0,0 @@ -package LeetCode75; - -import java.util.LinkedList; -import java.util.Queue; - -public class RottingOranges { - public int orangesRotting(int[][] grid) { - int m = grid.length, n = grid[0].length, count = 0; - Queue q = new LinkedList<>(); - - for (int i = 0; i < m; i++) { - for (int j = 0; j < n; j++) { - if (grid[i][j] == 1) - count++; - if (grid[i][j] == 2) - q.add(new Pair(i, j, 0)); - } - } - - int time = 0; - while (!q.isEmpty()) { - Pair curr = q.remove(); - time = curr.t; - - if (curr.i > 0 && grid[curr.i - 1][curr.j] == 1) { - grid[curr.i - 1][curr.j] = 2; - q.add(new Pair(curr.i - 1, curr.j, curr.t + 1)); - count--; - } - - if (curr.i < m - 1 && grid[curr.i + 1][curr.j] == 1) { - grid[curr.i + 1][curr.j] = 2; - q.add(new Pair(curr.i + 1, curr.j, curr.t + 1)); - count--; - } - - if (curr.j > 0 && grid[curr.i][curr.j - 1] == 1) { - grid[curr.i][curr.j - 1] = 2; - q.add(new Pair(curr.i, curr.j - 1, curr.t + 1)); - count--; - } - - if (curr.j < n - 1 && grid[curr.i][curr.j + 1] == 1) { - grid[curr.i][curr.j + 1] = 2; - q.add(new Pair(curr.i, curr.j + 1, curr.t + 1)); - count--; - } - } - - return count == 0 ? time : -1; - - } - - private class Pair { - public int i, j, t; - - public Pair(int i, int j, int t) { - this.i = i; - this.j = j; - this.t = t; - } - } -} diff --git a/site/solutions/java/LeetCode75/Tribonacci.java b/site/solutions/java/LeetCode75/Tribonacci.java deleted file mode 100644 index 593e078..0000000 --- a/site/solutions/java/LeetCode75/Tribonacci.java +++ /dev/null @@ -1,20 +0,0 @@ -package LeetCode75; - -public class Tribonacci { - public int tribonacci(int n) { - if (n == 0 || n == 1) - return n; - if (n == 2) - return 1; - - int a = 0, b = 1, c = 1, temp; - for (int i = 0; i < n - 2; i++) { - temp = c; - c += a + b; - a = b; - b = temp; - } - - return c; - } -} diff --git a/site/solutions/java/LeetCode75/Trie.java b/site/solutions/java/LeetCode75/Trie.java deleted file mode 100644 index 5523f11..0000000 --- a/site/solutions/java/LeetCode75/Trie.java +++ /dev/null @@ -1,77 +0,0 @@ -package LeetCode75; - -import java.util.HashMap; -import java.util.Map; - -class Trie { - private TrieNode root; - - public Trie() { - this.root = new TrieNode(null, false); - } - - public void insert(String word) { - if (word == "") { - this.root.contained = true; - return; - } - - TrieNode currNode = root; - char[] chars = word.toCharArray(); - int i = 0; - - while (i < chars.length && currNode.pointers.containsKey(chars[i])) - currNode = currNode.pointers.get(chars[i++]); - - while (i < chars.length) { - currNode.pointers.put(chars[i], new TrieNode(chars[i], false)); - currNode = currNode.pointers.get(chars[i++]); - } - - currNode.contained = true; - - } - - public boolean search(String word) { - char[] chars = word.toCharArray(); - TrieNode currNode = root; - int i = 0; - - while (i < chars.length && currNode.pointers.containsKey(chars[i])) - currNode = currNode.pointers.get(chars[i++]); - - return i == chars.length && currNode.contained; - } - - public boolean startsWith(String prefix) { - char[] chars = prefix.toCharArray(); - TrieNode currNode = root; - int i = 0; - while (i < chars.length && currNode.pointers.containsKey(chars[i])) - currNode = currNode.pointers.get(chars[i++]); - - return i == chars.length; - } - - @Override - public String toString() { - return this.root.pointers.toString(); - } - - private class TrieNode { - public Map pointers; - public Character data; - public boolean contained; - - public TrieNode(Character data, boolean contained) { - this.data = data; - this.contained = contained; - this.pointers = new HashMap<>(); - } - - @Override - public String toString() { - return this.data + ": " + contained; - } - } -} diff --git a/site/solutions/java/Misc/AddTwoNumbers.java b/site/solutions/java/Misc/AddTwoNumbers.java deleted file mode 100644 index d06eb0d..0000000 --- a/site/solutions/java/Misc/AddTwoNumbers.java +++ /dev/null @@ -1,50 +0,0 @@ -package Misc; - -import Misc.ListNode; - -public class AddTwoNumbers { - public ListNode addTwoNumbers(ListNode l1, ListNode l2) { - if (l1 == null && l2 == null) - return null; - - if (l1 == null) { - if (l2.val >= 10) { - l2.val -= 10; - if (l2.next == null) { - l2.next = new ListNode(1); - } else { - l2.next.val++; - } - } - l2.next = addTwoNumbers(null, l2.next); - return l2; - } - - if (l2 == null) { - return addTwoNumbers(null, l1); - } - - int combined = l1.val + l2.val; - - if (l1.next == null && l2.next == null) { - if (combined >= 10) { - l1.next = new ListNode(1); - l1.val = combined - 10; - } else - l1.val = combined; - return l1; - } - - if (combined >= 10) { - if (l1.next == null) - l1.next = new ListNode(1); - else - l1.next.val++; - l1.val = combined - 10; - } else - l1.val = combined; - - l1.next = addTwoNumbers(l1.next, l2.next); - return l1; - } -} diff --git a/site/solutions/java/Misc/ContainsDuplicate.java b/site/solutions/java/Misc/ContainsDuplicate.java deleted file mode 100644 index 3d9acca..0000000 --- a/site/solutions/java/Misc/ContainsDuplicate.java +++ /dev/null @@ -1,72 +0,0 @@ -package Misc; - -import java.util.Arrays; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Map; -import java.util.Set; - -public class ContainsDuplicate { - public boolean containsDuplicate(int[] nums) { - Set set = new HashSet<>(); - for (int n : nums) - if (!set.add(n)) - return true; - return false; - } - - public boolean containsNearbyDuplicate(int[] nums, int k) { - Map map = new HashMap<>(); - - for (int i = 0; i < nums.length; i++) { - if (map.containsKey(nums[i]) && Math.abs(i - map.get(nums[i])) <= k) - return true; - map.put(nums[i], i); - } - - return false; - } - - public boolean containsNearbyAlmostDuplicate(int[] nums, int indexDiff, int valueDiff) { - Pair[] pairs = new Pair[nums.length]; - for (int i = 0; i < nums.length; i++) - pairs[i] = new Pair(i, nums[i]); - - Arrays.sort(pairs, (a, b) -> (int) a.getValue() - (int) b.getValue()); - Pair prev, curr; - - for (int i = 0; i < nums.length; i++) { - int hi = i + 1; - - while (hi < nums.length) { - if (Math.abs((int) pairs[i].getValue() - (int) pairs[hi].getValue()) > valueDiff) - break; - if (Math.abs((int) pairs[i].getKey() - (int) pairs[hi++].getKey()) > indexDiff) - continue; - - return true; - } - - } - - return false; - - } - - class Pair { - private int k, v; - - public Pair(int key, int val) { - k = key; - v = val; - } - - public int getKey() { - return k; - } - - public int getValue() { - return v; - } - } -} diff --git a/site/solutions/java/Misc/IntersectionNode.java b/site/solutions/java/Misc/IntersectionNode.java deleted file mode 100644 index fdc60d8..0000000 --- a/site/solutions/java/Misc/IntersectionNode.java +++ /dev/null @@ -1,29 +0,0 @@ -package Misc; - -import java.util.HashSet; -import java.util.Set; -import Misc.ReorderList.ListNode; - -public class IntersectionNode { - public ListNode getIntersectionNode(ListNode headA, ListNode headB) { - if (headA == null || headB == null) - return null; - - Set set = new HashSet<>(); - - ListNode curr = headA; - - while (curr != null) { - set.add(curr); - curr = curr.next; - } - - curr = headB; - - while (curr != null && !set.contains(curr)) - curr = curr.next; - - return curr; - - } -} diff --git a/site/solutions/java/Misc/IsValidAnagram.java b/site/solutions/java/Misc/IsValidAnagram.java deleted file mode 100644 index eecb5e6..0000000 --- a/site/solutions/java/Misc/IsValidAnagram.java +++ /dev/null @@ -1,28 +0,0 @@ -package Misc; - -import java.util.HashMap; -import java.util.Map; - -public class IsValidAnagram { - public boolean isAnagram(String s, String t) { - if (s.length() != t.length()) - return false; - Map map = new HashMap<>(); - - for (int i = 0; i < s.length(); i++) - map.put(s.charAt(i), map.getOrDefault(s.charAt(i), 0) + 1); - - for (int i = 0; i < s.length(); i++) - map.put(t.charAt(i), map.getOrDefault(t.charAt(i), 0) - 1); - - for (int i = 0; i < s.length(); i++) - if (map.get(s.charAt(i)) != 0) - return false; - - for (int i = 0; i < s.length(); i++) - if (map.get(t.charAt(i)) != 0) - return false; - - return true; - } -} diff --git a/site/solutions/java/Misc/ListNode.java b/site/solutions/java/Misc/ListNode.java deleted file mode 100644 index d8da417..0000000 --- a/site/solutions/java/Misc/ListNode.java +++ /dev/null @@ -1,15 +0,0 @@ -package Misc; - -public class ListNode { - int val; - ListNode next; - - ListNode(int val) { - this.val = val; - } - - ListNode(int val, ListNode next) { - this.val = val; - this.next = next; - } -} diff --git a/site/solutions/java/Misc/LongestConsecutiveSequence.java b/site/solutions/java/Misc/LongestConsecutiveSequence.java deleted file mode 100644 index dbf7471..0000000 --- a/site/solutions/java/Misc/LongestConsecutiveSequence.java +++ /dev/null @@ -1,44 +0,0 @@ -package Misc; - -import java.util.HashSet; -import java.util.Set; - -public class LongestConsecutiveSequence { - public int longestConsecutive(int[] nums) { - if (nums.length <= 1) - return nums.length; - Set set = new HashSet<>(); - Set visited = new HashSet<>(); - - for (int n : nums) - set.add(n); - - int maxStreak = 0, currStreak = 0; - for (int n : nums) { - - if (visited.contains(n)) - continue; - - currStreak = 1; - int curr = n; - - visited.add(curr); - - while (set.contains(++curr)) { - currStreak++; - visited.add(curr); - } - - curr = n; - - while (set.contains(--curr)) { - currStreak++; - visited.add(curr); - } - - maxStreak = Math.max(maxStreak, currStreak); - } - - return maxStreak; - } -} diff --git a/site/solutions/java/Misc/MergeKLists.java b/site/solutions/java/Misc/MergeKLists.java deleted file mode 100644 index 1b46242..0000000 --- a/site/solutions/java/Misc/MergeKLists.java +++ /dev/null @@ -1,42 +0,0 @@ -package Misc; - -import java.util.PriorityQueue; -import Misc.ReorderList.ListNode; - -public class MergeKLists { - public ListNode mergeKLists(ListNode[] lists) { - if (lists.length == 0) - return null; - - PriorityQueue pq = new PriorityQueue<>((a, b) -> a.val - b.val); - - for (ListNode l : lists) - if (l != null) - pq.add(l); - - ListNode head = null, curr = null; - - while (!pq.isEmpty()) { - if (head == null) { - head = pq.remove(); - curr = head; - if (head.next != null) - pq.add(head.next); - continue; - } - - curr.next = pq.remove(); - - curr = curr.next; - - if (curr.next != null) - pq.add(curr.next); - } - - if (curr != null) - curr.next = null; - - return head; - - } -} diff --git a/site/solutions/java/Misc/MergeLinkedLists.java b/site/solutions/java/Misc/MergeLinkedLists.java deleted file mode 100644 index 57ba84f..0000000 --- a/site/solutions/java/Misc/MergeLinkedLists.java +++ /dev/null @@ -1,27 +0,0 @@ -package Misc; - -import Misc.ReorderList.ListNode; - -public class MergeLinkedLists { - public ListNode mergeInBetween(ListNode list1, int a, int b, ListNode list2) { - ListNode lo = list1, hi = list1; - - for (int i = 0; i < b - a; i++) - hi = hi.next; - - for (int i = 0; i < a - 1; i++) { - hi = hi.next; - lo = lo.next; - } - - hi = hi.next; - lo.next = list2; - - while (lo.next != null) - lo = lo.next; - - lo.next = hi.next; - - return list1; - } -} diff --git a/site/solutions/java/Misc/MinIndexSum.java b/site/solutions/java/Misc/MinIndexSum.java deleted file mode 100644 index 32dee27..0000000 --- a/site/solutions/java/Misc/MinIndexSum.java +++ /dev/null @@ -1,38 +0,0 @@ -package Misc; - -import java.util.HashMap; -import java.util.Map; - -public class MinIndexSum { - public String[] findRestaurant(String[] list1, String[] list2) { - Map indexMap = new HashMap<>(); - Map sumMap = new HashMap<>(); - - for (int i = 0; i < list1.length; i++) - indexMap.put(list1[i], i); - - for (int i = 0; i < list2.length; i++) - if (indexMap.containsKey(list2[i])) - sumMap.put(list2[i], i + indexMap.get(list2[i])); - - int min = Integer.MAX_VALUE, len = 1; - for (String s : sumMap.keySet()) { - int c = sumMap.get(s); - if (min > c) { - min = c; - len = 1; - } else if (min == c) - len++; - } - - String[] ans = new String[len]; - if (len == 0) - return ans; - int i = 0; - for (String s : sumMap.keySet()) - if (min == sumMap.get(s)) - ans[i++] = s; - return ans; - - } -} diff --git a/site/solutions/java/Misc/PartitionList.java b/site/solutions/java/Misc/PartitionList.java deleted file mode 100644 index 7b98cc5..0000000 --- a/site/solutions/java/Misc/PartitionList.java +++ /dev/null @@ -1,47 +0,0 @@ -package Misc; - -import Misc.ReorderList.ListNode; - -public class PartitionList { - public ListNode partition(ListNode head, int x) { - if (head == null || head.next == null) - return head; - - ListNode curr = head, - lessHead = null, - geqHead = null, - lessTail = null, - geqTail = null; - - while (curr != null) { - if (curr.val < x) { - if (lessHead == null) { - lessHead = curr; - lessTail = curr; - } else { - lessTail.next = curr; - lessTail = lessTail.next; - } - } else { - if (geqHead == null) { - geqHead = curr; - geqTail = curr; - } else { - geqTail.next = curr; - geqTail = geqTail.next; - } - } - curr = curr.next; - } - - if (lessHead == null) - return geqHead; - - lessTail.next = geqHead; - - if (geqHead != null) - geqTail.next = null; - - return lessHead; - } -} diff --git a/site/solutions/java/Misc/ReorderList.java b/site/solutions/java/Misc/ReorderList.java deleted file mode 100644 index b1b4ff4..0000000 --- a/site/solutions/java/Misc/ReorderList.java +++ /dev/null @@ -1,54 +0,0 @@ -package Misc; - -import java.util.Stack; - -public class ReorderList { - public void reorderList(ListNode head) { - if (head == null || head.next == null) - return; - - Stack s = new Stack<>(); - - ListNode curr = head; - - while (curr != null) { - s.push(curr); - curr = curr.next; - } - - int size = s.size(); - - ListNode newHead = head, tail = newHead; - curr = head.next; - - int currSize = 1; - while (!s.isEmpty() && currSize < size) { - tail.next = s.pop(); - tail = tail.next; - currSize++; - tail.next = curr; - tail = tail.next; - currSize++; - curr = curr.next; - } - - tail.next = null; - head = newHead; - - } - - public class ListNode { - int val; - ListNode next; - - ListNode(int val) { - this.val = val; - } - - ListNode(int val, ListNode next) { - this.val = val; - this.next = next; - } - } - -} diff --git a/site/solutions/java/Misc/ReorderLog.java b/site/solutions/java/Misc/ReorderLog.java deleted file mode 100644 index 64660fc..0000000 --- a/site/solutions/java/Misc/ReorderLog.java +++ /dev/null @@ -1,32 +0,0 @@ -package Misc; - -import java.util.ArrayList; -import java.util.List; - -public class ReorderLog { - public String[] reorderLogFiles(String[] logs) { - List let = new ArrayList<>(); - List dig = new ArrayList<>(); - - for (String s : logs) { - if (Character.isLetter(s.charAt(s.indexOf(" ") + 1))) - let.add(s); - else - dig.add(s); - } - - let.sort( - (s1, s2) -> { - String s1Sub = s1.substring(s1.indexOf(" ")); - String s2Sub = s2.substring(s2.indexOf(" ")); - - return (s1Sub.equals(s2Sub)) - ? s1.compareTo(s2) - : s1Sub.compareTo(s2Sub); - }); - - let.addAll(dig); - - return let.toArray(new String[let.size()]); - } -} diff --git a/site/solutions/java/Misc/RotateList.java b/site/solutions/java/Misc/RotateList.java deleted file mode 100644 index 4369b13..0000000 --- a/site/solutions/java/Misc/RotateList.java +++ /dev/null @@ -1,37 +0,0 @@ -package Misc; - -import Misc.ReorderList.ListNode; - -public class RotateList { - public ListNode rotateRight(ListNode head, int k) { - if (k == 0 || head == null || head.next == null) - return head; - - ListNode curr = head, newTail = head; - int size = 0; - while (curr != null) { - curr = curr.next; - size++; - } - - k %= size; - - if (k == 0) - return head; - - curr = head; - - for (int i = 0; i < k; i++) - curr = curr.next; - - while (curr.next != null) { - curr = curr.next; - newTail = newTail.next; - } - - ListNode newHead = newTail.next; - newTail.next = null; - curr.next = head; - return newHead; - } -} diff --git a/site/solutions/java/Misc/SortList.java b/site/solutions/java/Misc/SortList.java deleted file mode 100644 index 57c44b1..0000000 --- a/site/solutions/java/Misc/SortList.java +++ /dev/null @@ -1,42 +0,0 @@ -package Misc; - -import java.util.Arrays; -import Misc.ReorderList.ListNode; - -public class SortList { - public ListNode sortList(ListNode head) { - if (head == null || head.next == null) - return head; - - ListNode curr = head; - int size = 0; - while (curr != null) { - curr = curr.next; - size++; - } - - ListNode[] nodes = new ListNode[size]; - - curr = head; - - for (int i = 0; i < size; i++) { - nodes[i] = curr; - curr = curr.next; - } - - Arrays.sort(nodes, (n1, n2) -> n1.val - n2.val); - - ListNode newHead = nodes[0]; - - curr = newHead; - - for (int i = 1; i < size; i++) { - curr.next = nodes[i]; - curr = curr.next; - } - - curr.next = null; - return newHead; - } - -} diff --git a/site/solutions/java/Misc/TrappingRainWater.java b/site/solutions/java/Misc/TrappingRainWater.java deleted file mode 100644 index f47b6e8..0000000 --- a/site/solutions/java/Misc/TrappingRainWater.java +++ /dev/null @@ -1,27 +0,0 @@ -package Misc; - -public class TrappingRainWater { - public int trap(int[] height) { - int[] rightMax = new int[height.length]; - int[] leftMax = new int[height.length]; - int rMax = 0, lMax = 0; - - for (int i = 0; i < height.length; i++) { - rightMax[rightMax.length - i - 1] = rMax; - leftMax[i] = lMax; - rMax = Math.max(rMax, height[rightMax.length - i - 1]); - lMax = Math.max(lMax, height[i]); - } - - int water = 0; - - for (int i = 0; i < height.length; i++) { - int waterHeight = Math.min(rightMax[i], leftMax[i]); - if (waterHeight - height[i] > 0) - water += waterHeight - height[i]; - } - - return water; - - } -} diff --git a/site/solutions/java/Misc/ZigZag.java b/site/solutions/java/Misc/ZigZag.java deleted file mode 100644 index 52617ee..0000000 --- a/site/solutions/java/Misc/ZigZag.java +++ /dev/null @@ -1,32 +0,0 @@ -package Misc; - -public class ZigZag { - public String convert(String s, int numRows) { - - if (numRows == 1) - return s; - - char[][] grid = new char[numRows][s.length()]; - - int col = 0; - int numPlaced = 0; - while (numPlaced < s.length()) { - for (int row = 0; row < numRows && numPlaced < s.length(); row++) - grid[row][col] = s.charAt(numPlaced++); - - col++; - - for (int i = 0, row = numRows - 2; i < numRows - 2 && numPlaced < s.length(); i++) - grid[row--][col++] = s.charAt(numPlaced++); - } - - StringBuilder sb = new StringBuilder(); - - for (char[] r : grid) - for (char c : r) - if (c != 0) - sb.append(c); - - return sb.toString(); - } -} diff --git a/site/solutions/python/Misc/LongestRepeatingCharacterReplacement.py b/site/solutions/python/Misc/LongestRepeatingCharacterReplacement.py deleted file mode 100644 index 136907f..0000000 --- a/site/solutions/python/Misc/LongestRepeatingCharacterReplacement.py +++ /dev/null @@ -1,17 +0,0 @@ -class Solution: - def characterReplacement(self, s: str, k: int) -> int: - lo, hi, res = 0, 0, 0 - cMap = {} - - while (hi < len(s)): - c = s[hi] - cMap[c] = 1 if c not in cMap else cMap[c] + 1 - - while (hi - lo + 1) - max(cMap.values()) > k: - cMap[s[lo]] -= 1 - lo += 1 - - hi += 1 - res = max(res, hi - lo) - - return res \ No newline at end of file diff --git a/site/systems-research/end-to-end-arguments-in-sys-design.html b/site/systems-research/end-to-end-arguments-in-sys-design.html deleted file mode 100644 index d66cf82..0000000 --- a/site/systems-research/end-to-end-arguments-in-sys-design.html +++ /dev/null @@ -1,224 +0,0 @@ - - - - - - End-to-End Arguments in System Design - - - - - -
    - -

    End-to-End Arguments in System Design

    -
    - Last modified: 2025-01-13 - Category: System Design -
    -
    -

    source

    -
    End-to-End Arguments in System Design
    -
    -

    What is the Problem?

    -
      -
    • Placing functions at lower levels of a system may not be beneficial
        -
      • Functions generally know best, and error checking can therefore be redundant
      • -
      • Low level function placement may be costly
      • -
      -
    • -
    • A correct comms system can only be built with endpoints
        -
      • Ex: detecting crashes, delivering/sequencing messages, etc.
      • -
      -
    • -
    -

    Summary

    -

    Low Level Functionality

    -
      -
    • Paper argues that low-level functionality is mainly a performance optimization
    • -
    • If the probability of an error is low, doesn't make sense to add error checking in the middle of the system. Instead, let the endpoints handle it
    • -
    -

    Weakness

    -
      -
    • Maintainability (checks missing in the middle)
    • -
    • Certain systems ought to have intermediate checks (e.g. comms over lossy media)
    • -
    • Catching errors can take longer, needs to make it all the way to an endpoint to detect
    • -
    -

    Open Questions

    -

    -

    -

    Further Reading

    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/exokernel.html b/site/systems-research/exokernel.html deleted file mode 100644 index 901a76c..0000000 --- a/site/systems-research/exokernel.html +++ /dev/null @@ -1,236 +0,0 @@ - - - - - - Exokernel: An Operating System Architecture for Application-Level Resource Management - - - - - -
    - -

    Exokernel: An Operating System Architecture for Application-Level Resource Management

    -
    - Last modified: 2025-01-15 - Category: systems -
    -
    -

    What is the Problem?

    -

    Operating systems with monolithic kernels prescribe interfaces of key OS abstractions like virtual memory, filesystem, but with these prescriptions come side-effects, particularly in the realm of performance. Applications cannot modify or optimize these abstractions for their specific needs, forcing them to work within the limitations of a "one-size fits all" implementation, which typically prioritizes generality over performance for any specific application.

    -

    Summary

    -

    The paper covers the exokernel architecture, which aims to minimize the "mechanism" role as much as possible, opting to leave implementations to the client, e.g. to the user's choice of library operating system. The key design choice here is to separate resource protection from management, e.g. to provide secure bindings to access a device, without necessarily understanding the use case.

    -

    The authors were able to realize significant (orders of magnitude) speedups on most primitive tasks compared to a more mature OS (Ultraix) by focusing almost solely on efficiently multiplexing hardware, and by minimizing the number of required system calls during regular operation.

    -

    Download Code

    -

    Download code into exokernel from lib os (similar to eBPF today)

    -

    Key Insights

    -
      -
    • General-purpose abstractions in monolithic kernels can lead to performance overhead
    • -
    • Resource management is next-to-impossible since most resources are completely abstracted. Instead the kernel interface should be as close to hardware as possible, opting to use physical addresses, etc.
    • -
    • Applications will oftentimes be working against built in "features" of the OS, e.g. databases slowed down by filesystems, non-zero-copy networking
    • -
    • Applications must frequently defer to the kernel for operations that could be done in user space, incurring context switching overhead
    • -
    -

    Notable Design Details/Strengths

    -
      -
    • The interface provided by the kernel should be as close to hardware as possible, so as to directly expose hardware resources to applications in a safe manner
    • -
    • Library operating systems can be tailored to specific use cases, avoiding the overhead of general-purpose abstractions usually found in monolithic kernels
    • -
    • kernel interface with limited scope leads to better overall design
    • -
    • completeness: optimizing the hell out of what little it does is a lot easier
    • -
    • simplicity: less code to maintain, less code to break
    • -
    • extensibility: easier to add new features via library OSes than to modify the kernel
    • -
    -

    Limitations/Weaknesses

    -
      -
    • Compatibility between system-level software and dependencies has been kicked into user space, likely leading to less stability/reliability, or at the very least more work for the application developer and end user
    • -
    • Third party library OSes are very suspect
    • -
    • Unless standards are not only developed but widely adopted for all OS components in user space, this could be a major issue
    • -
    • Cross-platform compatibility is not a priority, and additional work would be needed to port all upstream library OSes to a new exokernel to port an application
    • -
    -

    Summary of Key Results

    -
      -
    • Many primitives are 1-2 orders of magnitude faster than Ultraix
    • -
    • Exception handling, virtual memory, IPC, etc. faster than Ultraix, and in some cases faster than SOTA implementations
    • -
    • Primarily due to reduced context switching, low-overhead multiplexing of hardware, and specialized implementations of aforementioned systems in user space
    • -
    -

    Open Questions

    -
      -
    • Why didn't this work out? I was fully bought in by the end of the paper, but they were clearly never adopted.
    • -
    • Can this approach be applied to modern, and particularly datacenter, workloads?
    • -
    • How can malicious/destructive library OSes be prevented?
    • -
    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/hints-for-computer-system-design.html b/site/systems-research/hints-for-computer-system-design.html deleted file mode 100644 index 6a4476f..0000000 --- a/site/systems-research/hints-for-computer-system-design.html +++ /dev/null @@ -1,302 +0,0 @@ - - - - - - Hints for Computer System Design - - - - - -
    - -

    Hints for Computer System Design

    -
    - Last modified: 2025-01-13 - Category: system-design -
    -
    -

    source

    -
    Hints for Computer System Design
    -
    -

    Key Insights

    -

    Caching

    -

    Store $[f, x, f(x)]$ tuples in a cache.

    -

    If $f$ isn't a pure function, invalidate with the following:

    -

    $$ -f(x + \Delta) = g(x, \Delta, f(x)) -$$

    -

    For example, $x$ is an int[], $\Delta$ is a write $(i, v)$, and $f$ is a function int sum(int[] x). Then $g(x, \Delta, f(x))$ is f(x) + v - x[i].

    -

    Caches should ideally have adaptive sizes.

    -

    A classic example is the caching in hardware that uses $[Fetch, \text{address}, \text{content of address}]$ tuples. Similarly, virtual memory uses $[Page, \text{address}, \text{content of address}]$ tuples.

    -

    However, more complicated applications of caching exist. In real-time systems, you're often trying to cache the state of a system given small changes corresponding to events. The key here is to try and invalidate as few entries as possible in response to events.

    -

    Lecture Review Notes

    -

    Why is system design hard?

    -
      -
    • external interface isn't well defined
        -
      • requirements aren't clear
      • -
      • Things are often not well-designed
      • -
      -
    • -
    • The measure of success is not very clear
        -
      • Many different ways to interact with a system
      • -
      • Many systems, even in production, have bugs
      • -
      -
    • -
    -

    Throw one away

    -
      -
    • Always be prepared to discard your prototype
    • -
    • Throw ideas at the wall and go with what sticks
    • -
    -

    Interface Design

    -

    Conflicting requirements: -- Simple -- Complete -- Efficient

    -

    In a way it's a lot like PL design; exposing new abstractions, objects and operations, manipulating them, etc.

    -

    KISS; Do one thing at a time and do it well.

    -
      -
    • Don't over-promise
    • -
    • Get it right, but beware the dangers of abstractions (especially performance)
    • -
    • Make it fast rather than general and complete. You should keep scope small so that it's easy to optimize, and also to compose with other systems/components
    • -
    • Procedure args let you keep it general but extendable
        -
      • C function pointers, C++ functions
      • -
      • LD_PRELOAD trick: override calls by providing a wrapper that calls the original function, but with some extra functionality
      • -
      -
    • -
    • Leave it to the client (check Exokernel paper)
        -
      • Unix pipes
      • -
      -
    • -
    • Keep interfaces stable
        -
      • Counterexample LLVM
      • -
      -
    • -
    • Keep a place to stand
        -
      • Virtualization!!!
      • -
      -
    • -
    -

    Implementation

    -

    Plan to throw one away - learn from prototyping

    -

    Keep secrets - impl details hidden from clients. Can be tradeoff for performance optimizations

    -

    Handle normal and worst cases separately

    -
      -
    • Might be OK to crash a few processes if it means the system can recover
    • -
    • Caches in processors are optimized for common case (principle of locality)
    • -
    • Paging in virtual memory is optimized for common case (principle of locality)
    • -
    -

    Efficiency

    -
      -
    • Split resources
        -
      • Faster to allocate a new resource than to wait for one to be freed
      • -
      • Heterogeneous systems
          -
        • Specialized hardware like FPGA or GPU to run specialized tasks
        • -
        • E.G. Google's TPU, Microsoft Azure FPGAs
        • -
        -
      • -
      -
    • -
    • Use static analysis
    • -
    -

    Reliability

    -
      -
    • Log updates
        -
      • Can recover
      • -
      • Append only is efficient
      • -
      • Can be used for replication
      • -
      -
    • -
    • Atomic transactions
        -
      • E.G. ACID
      • -
      -
    • -
    -

    Takeaways

    -
      -
    • Most successful systems are built with particular themes, many of which are discussed in this paper
    • -
    • When reading papers, look for what you can apply, and ignore irrelevant details.
    • -
    • Hints can be added, e.g. approximation vs precision
    • -
    -

    Further Reading

    -
      -
    • MicroLog
    • -
    • https://github.com/DPDK/dpdk
    • -
    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/how-to-read-a-paper.html b/site/systems-research/how-to-read-a-paper.html deleted file mode 100644 index 8b9076c..0000000 --- a/site/systems-research/how-to-read-a-paper.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - - - How to Read a Paper - - - - - -
    - -

    How to Read a Paper

    -
    - Last modified: 2025-01-12 - Category: research -
    -
    -

    source

    -
    How to Read a Paper
    -
    -

    What is the Problem?

    -

    Researchers spend 100s of hours reading papers every year, so they ought to know how to do it effectively. Keshav argues that reading a paper is a skill that can be learned, and that it is a skill that is not taught in school.

    -

    Summary

    -

    Keshav recommends a three-pass approach to reading a paper:

    -

    First Pass

    -

    Gives you a general idea of what the paper is about. Read the title, abstract, introduction, and conclusion. Only read section and sub-section headers. Also glance over the references and note which you've read. This should only take 5-10 minutes.

    -

    After the first pass, you should be able to answer the following questions:

    -
      -
    • Category: What type of paper is this? (e.g. measurement, analysis)
    • -
    • Context: Which other papers is it related to? What is the theoretical background?
    • -
    • Correctness: Do the assumptions appear to be valid?
    • -
    • Contributions: What are the paper's main contributions?
    • -
    • Clarity: Is the paper well written?
    • -
    -

    Second Pass

    -

    Grasp the content of the paper, but not necessarily the details. Read the whole thing, but ignore things like proofs. It helps to also annotate/take notes during this pass.

    -

    Pay special attention to diagrams, and also mark relevant unread references. This should take about an hour.

    -

    After this pass, you should be able to summarize the paper in a few sentences to a peer. If you still don't understand, you can do one of three things:

    -
      -
    • Set the paper aside and hope
    • -
    • Come back to the paper later after looking up references you didn't understand
    • -
    • Continue to the third pass anyways 😢
    • -
    -

    Third Pass

    -

    Understand the paper in depth. You should virtually re-implement the paper, following all reasoning and challenging every assumption. Also try to think about how you would have presented the information differently. You should note potential follow-up work during this pass. This should take up to 5 hours, and at least 1 hour for a well-written paper and well-read reader.

    -

    After this pass, you should be able to reconstruct the paper's structure from memory. You should also be able to critique the paper and pinpoint implicit assumptions, limitations, and potential improvements.

    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/internet-design-philosophy.html b/site/systems-research/internet-design-philosophy.html deleted file mode 100644 index 7124623..0000000 --- a/site/systems-research/internet-design-philosophy.html +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - Design Philosophy of DARPA Internet Protocols - - - - - -
    - -

    Design Philosophy of DARPA Internet Protocols

    -
    - Last modified: 2025-01-13 - Category: networking -
    -
    -

    source

    -
    Design Philosophy of the DARPA Internet Protocols
    -
    -

    What is the Problem?

    -

    Summary

    -

    Fundamental Goal: develop effective technique for multiplexed utilization of interconnected networks.

    -

    Multiplexing: single channel used by many communicating parties -- Circuit-switching: dedicated channel for each pair of communicating parties, i.e. point-to-point comms - - Predictable performance because resources are "reserved" for each connection - - Inefficient use of resources, number of connections limited by number of channels. With N parties, N(N-1)/2 channels needed. -- Packet-switching: packets from many parties share a single channel - - More efficient use of resources, but performance less predictable - - Packets can be lost, delayed, or delivered out of order

    -

    At a high level, packet switching needs to happen in order to take advantage of the redundancy in paths between any two hosts. Connection at the transport layer can be established and maintained regardless of the underlying network topology, so long as a path exists between the two hosts.

    -

    Another fundamental goal: connecting heterogeneous networks. The internet is a network of many different types of networks, each with its own protocols and addressing schemes. The internet protocols need to be able to connect these networks together/transmit data between them.

    -

    Secondary Goals:

    -
      -
    • Continue despite failure
    • -
    • Multiple types of communication
    • -
    • Variety of networks
    • -
    • Distributed management of resources
        -
      • Any centralized control would be a bottleneck
      • -
      • Each network should be able to manage its own resources
      • -
      -
    • -
    • Cost effective
    • -
    • Host attachment
        -
      • Hosts should be able to connect to the network without requiring changes to the network
      • -
      -
    • -
    • Accountability
        -
      • Hosts should be able to identify themselves to the network
      • -
      • Quality of service should be able to be enforced
      • -
      -
    • -
    -

    Datagrams

    -
      -
    • Connectionless service
        -
      • no state established ahead of time
      • -
      -
    • -
    • Key building block for switching
    • -
    • UDP is app-level interface to datagram service of the internet
        -
      • building block for other protocols (TCP)
      • -
      -
    • -
    • Each packet is independent
    • -
    -

    TCP vs. UDP

    -
      -
    • TCP: connection-oriented, reliable, in-order delivery
    • -
    • UDP: connectionless, unreliable, unordered delivery (loss is OK)
        -
      • No QoS guarantees in lower-level
      • -
      -
    • -
    -

    Supporting Variety of Networks

    -

    "Thin waist" of the internet/hourglass model: IP at the network layer, TCP/UDP at the transport layer.

    -
      -
    • IP: provides a common interface for all networks
    • -
    • TCP/UDP: provides a common interface for all applications
    • -
    -

    Abstraction hides details of lower layer, allowing whatever you want to happen at the lower level while the application remains unaware.

    -

    Unfortunately, can also lead to some problems -- Can't use hints directly from lower level for optimizations - - Workarounds: ECN, dpdk - - Parallels in storage, e.g. direct access, spdk -- Can't evolve interface of IP without changing everything above it

    -

    Fate Sharing

    -

    Move state to endpoints for survivability. If a network fails, the endpoints can reestablish the connection.

    -

    Strengths

    -
      -
    • Simple idea of datagrams
    • -
    • Scalable/distributed
    • -
    • It works!
    • -
    -

    Weaknesses

    -
      -
    • Narrow IP interface hurts innovation at IP level
    • -
    • Hiding secrets can hurt efficiency
    • -
    -

    Further Reading

    - -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/paper-review-template.html b/site/systems-research/paper-review-template.html deleted file mode 100644 index b6e09ed..0000000 --- a/site/systems-research/paper-review-template.html +++ /dev/null @@ -1,209 +0,0 @@ - - - - - - Paper Review Template - - - - - -
    - -

    Paper Review Template

    -
    - Last modified: 2025-01-12 - Category: research -
    -
    -

    source

    -
    Paper Title
    -
    -

    What is the Problem?

    -

    Summary

    -

    Key Insights

    -

    -

    -

    Notable Design Details/Strengths

    -

    -

    -

    Limitations/Weaknesses

    -

    -

    -

    Summary of Key Results

    -

    -

    -

    Open Questions

    -

    -

    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/sparsity-notes.html b/site/systems-research/sparsity-notes.html deleted file mode 100644 index 2934cc7..0000000 --- a/site/systems-research/sparsity-notes.html +++ /dev/null @@ -1,309 +0,0 @@ - - - - - - Sparsity Notes - - - - - -
    - -

    Sparsity Notes

    -
    - Last modified: 2025-01-14 - -
    -
    -

    Attention Sparsity Review

    -

    Faster Causal Self Attention

    -

    This paper presents an important advancement in making transformer attention mechanisms more efficient for processing long sequences. Here are the key points:

    -
      -
    1. -

      Core Innovation: The authors introduce Sparse Causal Flash Attention (SCFA), which extends the existing FlashAttention algorithm to handle irregular/sparse attention patterns while maintaining high computational efficiency.

      -
    2. -
    3. -

      Two Main Applications:

      -
    4. -
    5. Query/Key (QK) dropping: Selectively removing certain query and key pairs
    6. -
    7. -

      Hash-based attention: Using locality-sensitive hashing to group similar queries and keys together

      -
    8. -
    9. -

      Key Results:

      -
    10. -
    11. Achieves 2.0× speedup for sequences of 8,192 tokens
    12. -
    13. Achieves 3.3× speedup for sequences of 16,384 tokens
    14. -
    15. Maintains comparable perplexity to standard attention
    16. -
    17. -

      Outperforms previous approaches like Reformer in both speed and accuracy

      -
    18. -
    19. -

      Main Advantages:

      -
    20. -
    21. No computational complexity overhead compared to regular FlashAttention
    22. -
    23. Supports dynamic sparsity patterns rather than just static ones
    24. -
    25. Achieves exact computation (unlike some previous approaches that approximate)
    26. -
    27. -

      Works particularly well for longer sequences

      -
    28. -
    29. -

      Technical Innovation: -The key technical achievement is modifying FlashAttention to handle non-triangular causal masks, which enables more flexible attention patterns while maintaining the memory and computational benefits of the original FlashAttention algorithm.

      -
    30. -
    -

    This work is significant because it helps address one of the main bottlenecks in transformer models - the quadratic computational cost of attention with respect to sequence length - while maintaining exact computation and allowing for dynamic sparsity patterns.

    -

    Sparser is Faster: Long-Range Attention with Linear Complexity

    -

    Here's a summary of the key points from this paper about SparseK Attention:

    -

    Key Innovation: -- Introduces SparseK Attention, a novel sparse attention mechanism that offers both computational and memory efficiency for long sequences -- Uses a scoring network and differentiable top-k mask operator to dynamically select important key-value pairs for each query

    -

    Main Advantages: -1. Efficiency: -- Linear time complexity and constant memory footprint -- Better speed than previous sparse attention methods -- Efficient for both training and inference

    -
      -
    1. Performance:
    2. -
    3. Outperforms previous sparse attention approaches
    4. -
    5. Matches or exceeds full attention quality while being faster
    6. -
    7. -

      Can handle sequences up to 16,384 tokens effectively

      -
    8. -
    9. -

      Technical Features:

      -
    10. -
    11. Integrates with sliding window attention
    12. -
    13. Compatible with pre-trained LLMs through fine-tuning
    14. -
    15. Uses an IO-aware implementation based on Flash Attention
    16. -
    -

    Results: -- Language modeling tests show better perplexity than baseline methods -- Achieves 2.0× speedup for 8k sequences and 3.3× for 16k sequences -- Maintains performance while significantly reducing compute and memory requirements

    -

    Key Limitation: -- Currently validated only up to 1.1B parameter models and 16k token contexts due to computational constraints -- Only tested on decoder-only architectures and text tasks -- Some overhead for short sequences, though benefits increase with sequence length

    -

    The paper demonstrates that SparseK Attention can make transformer models more efficient for long sequences while maintaining or improving quality, offering a practical solution for scaling up context windows in language models.

    -

    MoA

    -

    This paper introduces MoA (Mixture of Attention), a novel method for compressing large language models (LLMs) by automatically optimizing sparse attention patterns. Here are the key points:

    -
      -
    1. Problem & Motivation:
    2. -
    3. LLMs struggle with long contexts due to quadratic memory and computation costs from attention
    4. -
    5. Existing sparse attention methods use uniform patterns across all attention heads, ignoring that different heads serve different purposes
    6. -
    7. -

      Current approaches fail to extend effective context length beyond their attention span

      -
    8. -
    9. -

      Key Innovation - MoA:

      -
    10. -
    11. Automatically discovers heterogeneous sparse attention patterns tailored to each attention head
    12. -
    13. Uses elastic rules that allow attention spans to scale differently with input length
    14. -
    15. -

      Maintains different patterns for different layers and heads based on their functions

      -
    16. -
    17. -

      Technical Approach:

      -
    18. -
    19. Profiles the influence of each attention position on model predictions using gradient-based analysis
    20. -
    21. Constructs a search space of various attention patterns and scaling rules
    22. -
    23. Uses calibration datasets with long-range dependencies
    24. -
    25. -

      Optimizes patterns automatically through a multi-objective framework

      -
    26. -
    27. -

      Key Results:

      -
    28. -
    29. Increases effective context length by 3.9× compared to baseline methods
    30. -
    31. Improves retrieval accuracy by 1.5-7.1× over uniform attention baselines
    32. -
    33. Reduces maximum performance drop from 9-36% to within 5% on benchmarks
    34. -
    35. Achieves 6.6-8.2× throughput improvement over FlashAttention2
    36. -
    37. -

      Reduces GPU memory usage by 1.2-1.4×

      -
    38. -
    39. -

      Limitations:

      -
    40. -
    41. Performance degrades under extremely low-density constraints
    42. -
    43. May benefit from dynamic attention patterns (left for future work)
    44. -
    45. Could explore non-linear elastic rules
    46. -
    -

    The paper demonstrates that automatically discovering heterogeneous attention patterns can significantly improve both the efficiency and capabilities of LLMs in handling long contexts, while maintaining model performance.

    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/strong-inference.html b/site/systems-research/strong-inference.html deleted file mode 100644 index 691fa0b..0000000 --- a/site/systems-research/strong-inference.html +++ /dev/null @@ -1,222 +0,0 @@ - - - - - - Strong Inference - - - - - -
    - -

    Strong Inference

    -
    - Last modified: 2025-01-12 - Category: research -
    -
    -

    source

    -

    Strong Inference

    -
    Certain systematic methods of scientific thinking may produce much more rapid progress than others.
    -
    -

    What is the Problem?

    -

    The process of doing research has become less standardized in some fields, especially when compared to the more structured approach of fields like molecular bio and high-energy physics.

    -

    When a formal scientific method isn't adhered to, especially when formulating hypotheses, the process can become less efficient and less effective, leading to fewer discoveries and slower progress over time.

    -

    Summary

    -

    John R. Platt critiques the application of the scientific method in modern research, arguing that the process has become less structured and less effective in some fields. He proposes a more structured approach to hypothesis formulation, which he calls "strong inference".

    -

    The paper goes on quite a bit about historical examples of the principles of strong inference working in practice, as well as a detailed breakdown of how to systematically apply the method to research.

    -
      -
    1. Devise alternative hypotheses - Generate multiple competing hypotheses that could explain the phenomenon being studied.
    2. -
    3. Devise a crucial experiment - Design an experiment that can unambiguously distinguish between the competing hypotheses, or at the very least, eliminate some or all of them.
    4. -
    5. Carry out the experiment - Conduct the experiment and analyze the results.
    6. -
    -

    Strong inference is essentially to carry out this process at every vertex of the logical tree of inquiry, and to do so in a systematic and structured way. He suggests keeping a notebook explicitly for this, and to pay particular attention to the process of hypothesis generation.

    -

    Key Insights

    -
      -
    • Have multiple competing hypotheses, and come up with the most efficient way to eliminate them via experimentation.
    • -
    • Systematically, explicitly, and regularly follow this process, from hypothesis generation to experimentation to analysis.
    • -
    • Ask yourself the two questions: "How would we know this hypothesis is wrong?" and "What hypothesis does this experiment disprove?"
    • -
    -

    Notable Design Details/Strengths

    -
      -
    • Deviations from strong inference only really manifest themselves as useless delays in the research process. Many scientists do a lot of busywork for no reason, which could be avoided had they spent more time formulating hypotheses.
    • -
    • Strong inference is strongly dependent on the actual induction being done when formulating hypotheses. This needs to be logically sound.
    • -
    • It is a system that works if done correctly, as it's essentially the minimum amount of work needed to make a discovery without just getting lucky.
    • -
    -
    - -
    - - \ No newline at end of file diff --git a/site/systems-research/unix-timesharing-system.html b/site/systems-research/unix-timesharing-system.html deleted file mode 100644 index 57335a1..0000000 --- a/site/systems-research/unix-timesharing-system.html +++ /dev/null @@ -1,240 +0,0 @@ - - - - - - The Unix Timesharing System - - - - - -
    - -

    The Unix Timesharing System

    -
    - Last modified: 2025-01-15 - Category: Operating Systems -
    -
    -

    source

    -
    The Unix Timesharing System
    -
    -

    What is the Problem?

    -

    A very unspecific problem. Essentially they were burnt by Multics and wanted to create a simpler and more general system.

    -

    Summary

    -

    Key design goal is simplicity. Everything is hierarchical. Everything is a file.

    -

    File System Implementation

    -
      -
    • Tree strucutre
    • -
    • FS mounted on a file
    • -
    • Sys table of i-numbers (i-list)
    • -
    • i-node contains metadata for each file
    • -
    • path names don't distinguish between files and directories
    • -
    • mount table for mounted file systems
    • -
    • buffering is built into kernel and transparent to user
        -
      • write-behind (flushes to disk when buffer is full)
      • -
      -
    • -
    -

    Storage Devices

    -
      -
    • block devices
        -
      • data is stored in fixed-size blocks
      • -
      • "free list" of blocks for allocation, Linked list of blocks
      • -
      • hard disks, usb drive, ssd, tape drives
      • -
      • early versions of ethernet
      • -
      -
    • -
    • character devices
    • -
    -

    Execution

    -
      -
    • Image is an execution env (parallel container)
    • -
    • Process is an instance/execution of an image
        -
      • Program text write-protected and shared between all instances of that process
      • -
      • Separate virtual address space for each process
      • -
      -
    • -
    • Kernel
        -
      • Mediator for accessing services/hardware/shared resources
      • -
      -
    • -
    -

    Key Insights

    -

    -

    -
    - -
    - - \ No newline at end of file diff --git a/site/tags/acyclic graphs.html b/site/tags/acyclic graphs.html deleted file mode 100644 index acd210c..0000000 --- a/site/tags/acyclic graphs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: acyclic graphs - - - - - -
    - -

    Tag: acyclic graphs

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: acyclic graphs

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/algorithm-analysis.html b/site/tags/algorithm-analysis.html deleted file mode 100644 index e711118..0000000 --- a/site/tags/algorithm-analysis.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: algorithm-analysis - - - - - -
    - -

    Tag: algorithm-analysis

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/algorithm.html b/site/tags/algorithm.html deleted file mode 100644 index 140b135..0000000 --- a/site/tags/algorithm.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: algorithm - - - - - -
    - -

    Tag: algorithm

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/algorithms.html b/site/tags/algorithms.html deleted file mode 100644 index 311bc14..0000000 --- a/site/tags/algorithms.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: algorithms - - - - - -
    - -

    Tag: algorithms

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/approximation.html b/site/tags/approximation.html deleted file mode 100644 index 53cbc83..0000000 --- a/site/tags/approximation.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: approximation - - - - - -
    - -

    Tag: approximation

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: approximation

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/asymptotic notation.html b/site/tags/asymptotic notation.html deleted file mode 100644 index 8dcf1a4..0000000 --- a/site/tags/asymptotic notation.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: asymptotic notation - - - - - -
    - -

    Tag: asymptotic notation

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/batch processing.html b/site/tags/batch processing.html deleted file mode 100644 index ca49108..0000000 --- a/site/tags/batch processing.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: batch processing - - - - - -
    - -

    Tag: batch processing

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/bipartite graphs.html b/site/tags/bipartite graphs.html deleted file mode 100644 index ebb8a0b..0000000 --- a/site/tags/bipartite graphs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: bipartite graphs - - - - - -
    - -

    Tag: bipartite graphs

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/bipartite matching.html b/site/tags/bipartite matching.html deleted file mode 100644 index fb0c135..0000000 --- a/site/tags/bipartite matching.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: bipartite matching - - - - - -
    - -

    Tag: bipartite matching

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/breadth-first search.html b/site/tags/breadth-first search.html deleted file mode 100644 index 3b238b6..0000000 --- a/site/tags/breadth-first search.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: breadth-first search - - - - - -
    - -

    Tag: breadth-first search

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/caching.html b/site/tags/caching.html deleted file mode 100644 index ee4a0b3..0000000 --- a/site/tags/caching.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: caching - - - - - -
    - -

    Tag: caching

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: caching

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/classification.html b/site/tags/classification.html deleted file mode 100644 index 8d3f98b..0000000 --- a/site/tags/classification.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: classification - - - - - -
    - -

    Tag: classification

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: classification

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/column-oriented storage.html b/site/tags/column-oriented storage.html deleted file mode 100644 index 87e9bb8..0000000 --- a/site/tags/column-oriented storage.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: column-oriented storage - - - - - -
    - -

    Tag: column-oriented storage

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: column-oriented storage

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/compatibility.html b/site/tags/compatibility.html deleted file mode 100644 index 9e2ea8e..0000000 --- a/site/tags/compatibility.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: compatibility - - - - - -
    - -

    Tag: compatibility

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/complexity analysis.html b/site/tags/complexity analysis.html deleted file mode 100644 index 462ff75..0000000 --- a/site/tags/complexity analysis.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: complexity analysis - - - - - -
    - -

    Tag: complexity analysis

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/complexity-analysis.html b/site/tags/complexity-analysis.html deleted file mode 100644 index f8753d0..0000000 --- a/site/tags/complexity-analysis.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: complexity-analysis - - - - - -
    - -

    Tag: complexity-analysis

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/connected components.html b/site/tags/connected components.html deleted file mode 100644 index df86433..0000000 --- a/site/tags/connected components.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: connected components - - - - - -
    - -

    Tag: connected components

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/connected graphs.html b/site/tags/connected graphs.html deleted file mode 100644 index 748d273..0000000 --- a/site/tags/connected graphs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: connected graphs - - - - - -
    - -

    Tag: connected graphs

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: connected graphs

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/data analysis.html b/site/tags/data analysis.html deleted file mode 100644 index 7a99134..0000000 --- a/site/tags/data analysis.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: data analysis - - - - - -
    - -

    Tag: data analysis

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/data modeling.html b/site/tags/data modeling.html deleted file mode 100644 index 8a4fa4d..0000000 --- a/site/tags/data modeling.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: data modeling - - - - - -
    - -

    Tag: data modeling

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/data replication.html b/site/tags/data replication.html deleted file mode 100644 index 99eb915..0000000 --- a/site/tags/data replication.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: data replication - - - - - -
    - -

    Tag: data replication

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/data serialization.html b/site/tags/data serialization.html deleted file mode 100644 index f98c1ba..0000000 --- a/site/tags/data serialization.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: data serialization - - - - - -
    - -

    Tag: data serialization

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/data structures.html b/site/tags/data structures.html deleted file mode 100644 index 5ff5391..0000000 --- a/site/tags/data structures.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: data structures - - - - - -
    - -

    Tag: data structures

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/data systems.html b/site/tags/data systems.html deleted file mode 100644 index cae867a..0000000 --- a/site/tags/data systems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: data systems - - - - - -
    - -

    Tag: data systems

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/deep learning.html b/site/tags/deep learning.html deleted file mode 100644 index b2c3ce0..0000000 --- a/site/tags/deep learning.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: deep learning - - - - - -
    - -

    Tag: deep learning

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: deep learning

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/depth first search.html b/site/tags/depth first search.html deleted file mode 100644 index 41aa0a4..0000000 --- a/site/tags/depth first search.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: depth first search - - - - - -
    - -

    Tag: depth first search

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: depth first search

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/depth-first search.html b/site/tags/depth-first search.html deleted file mode 100644 index a06a0e9..0000000 --- a/site/tags/depth-first search.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: depth-first search - - - - - -
    - -

    Tag: depth-first search

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/design.html b/site/tags/design.html deleted file mode 100644 index 8c61d33..0000000 --- a/site/tags/design.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: design - - - - - -
    - -

    Tag: design

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/distributed filesystems.html b/site/tags/distributed filesystems.html deleted file mode 100644 index 738a8b7..0000000 --- a/site/tags/distributed filesystems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: distributed filesystems - - - - - -
    - -

    Tag: distributed filesystems

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: distributed filesystems

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/document databases.html b/site/tags/document databases.html deleted file mode 100644 index 38c199d..0000000 --- a/site/tags/document databases.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: document databases - - - - - -
    - -

    Tag: document databases

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: document databases

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/dynamic-programming.html b/site/tags/dynamic-programming.html deleted file mode 100644 index fef9cb2..0000000 --- a/site/tags/dynamic-programming.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: dynamic-programming - - - - - -
    - -

    Tag: dynamic-programming

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/efficiency.html b/site/tags/efficiency.html deleted file mode 100644 index 76b41a1..0000000 --- a/site/tags/efficiency.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: efficiency - - - - - -
    - -

    Tag: efficiency

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/encoding formats.html b/site/tags/encoding formats.html deleted file mode 100644 index 26dbdd6..0000000 --- a/site/tags/encoding formats.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: encoding formats - - - - - -
    - -

    Tag: encoding formats

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/end-to-end.html b/site/tags/end-to-end.html deleted file mode 100644 index e02a90a..0000000 --- a/site/tags/end-to-end.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: end-to-end - - - - - -
    - -

    Tag: end-to-end

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: end-to-end

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/etl.html b/site/tags/etl.html deleted file mode 100644 index a384875..0000000 --- a/site/tags/etl.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: etl - - - - - -
    - -

    Tag: etl

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/exokernel.html b/site/tags/exokernel.html deleted file mode 100644 index 533c02d..0000000 --- a/site/tags/exokernel.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: exokernel - - - - - -
    - -

    Tag: exokernel

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/failover.html b/site/tags/failover.html deleted file mode 100644 index 521a6c1..0000000 --- a/site/tags/failover.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: failover - - - - - -
    - -

    Tag: failover

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/feedforward.html b/site/tags/feedforward.html deleted file mode 100644 index 4dbfb96..0000000 --- a/site/tags/feedforward.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: feedforward - - - - - -
    - -

    Tag: feedforward

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: feedforward

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/ford-fulkerson algorithm.html b/site/tags/ford-fulkerson algorithm.html deleted file mode 100644 index 3d8d181..0000000 --- a/site/tags/ford-fulkerson algorithm.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: ford-fulkerson algorithm - - - - - -
    - -

    Tag: ford-fulkerson algorithm

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: ford-fulkerson algorithm

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/gale-shapley.html b/site/tags/gale-shapley.html deleted file mode 100644 index e0ef85f..0000000 --- a/site/tags/gale-shapley.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: gale-shapley - - - - - -
    - -

    Tag: gale-shapley

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph coloring.html b/site/tags/graph coloring.html deleted file mode 100644 index dcd5b42..0000000 --- a/site/tags/graph coloring.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph coloring - - - - - -
    - -

    Tag: graph coloring

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph databases.html b/site/tags/graph databases.html deleted file mode 100644 index 91398c0..0000000 --- a/site/tags/graph databases.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph databases - - - - - -
    - -

    Tag: graph databases

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph fundamentals.html b/site/tags/graph fundamentals.html deleted file mode 100644 index 040edfd..0000000 --- a/site/tags/graph fundamentals.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph fundamentals - - - - - -
    - -

    Tag: graph fundamentals

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph properties.html b/site/tags/graph properties.html deleted file mode 100644 index 2ee8feb..0000000 --- a/site/tags/graph properties.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: graph properties - - - - - -
    - -

    Tag: graph properties

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph representation.html b/site/tags/graph representation.html deleted file mode 100644 index 213d7fc..0000000 --- a/site/tags/graph representation.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph representation - - - - - -
    - -

    Tag: graph representation

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph theory.html b/site/tags/graph theory.html deleted file mode 100644 index 843cb6e..0000000 --- a/site/tags/graph theory.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: graph theory - - - - - -
    - -

    Tag: graph theory

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph traversal.html b/site/tags/graph traversal.html deleted file mode 100644 index 6ae63ab..0000000 --- a/site/tags/graph traversal.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph traversal - - - - - -
    - -

    Tag: graph traversal

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph-theory.html b/site/tags/graph-theory.html deleted file mode 100644 index 4c2bfa0..0000000 --- a/site/tags/graph-theory.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph-theory - - - - - -
    - -

    Tag: graph-theory

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph-traversal.html b/site/tags/graph-traversal.html deleted file mode 100644 index befa449..0000000 --- a/site/tags/graph-traversal.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph-traversal - - - - - -
    - -

    Tag: graph-traversal

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/graph.html b/site/tags/graph.html deleted file mode 100644 index bb83e90..0000000 --- a/site/tags/graph.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: graph - - - - - -
    - -

    Tag: graph

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/greedy-algorithms.html b/site/tags/greedy-algorithms.html deleted file mode 100644 index 2a9cdf5..0000000 --- a/site/tags/greedy-algorithms.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: greedy-algorithms - - - - - -
    - -

    Tag: greedy-algorithms

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/independent set.html b/site/tags/independent set.html deleted file mode 100644 index 2bd97cb..0000000 --- a/site/tags/independent set.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: independent set - - - - - -
    - -

    Tag: independent set

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/index.html b/site/tags/index.html deleted file mode 100644 index 165d62f..0000000 --- a/site/tags/index.html +++ /dev/null @@ -1,298 +0,0 @@ - - - - - - Tags - - - - - -
    - -

    Tags

    -
    - Last modified: 2025-01-16 - -
    -
    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/indexing.html b/site/tags/indexing.html deleted file mode 100644 index 64b9b14..0000000 --- a/site/tags/indexing.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: indexing - - - - - -
    - -

    Tag: indexing

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/induction proofs.html b/site/tags/induction proofs.html deleted file mode 100644 index e4d199d..0000000 --- a/site/tags/induction proofs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: induction proofs - - - - - -
    - -

    Tag: induction proofs

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: induction proofs

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/induction.html b/site/tags/induction.html deleted file mode 100644 index 04fe347..0000000 --- a/site/tags/induction.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: induction - - - - - -
    - -

    Tag: induction

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/internet.html b/site/tags/internet.html deleted file mode 100644 index 4b4d681..0000000 --- a/site/tags/internet.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: internet - - - - - -
    - -

    Tag: internet

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/interval.html b/site/tags/interval.html deleted file mode 100644 index 192b876..0000000 --- a/site/tags/interval.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: interval - - - - - -
    - -

    Tag: interval

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/leader-follower model.html b/site/tags/leader-follower model.html deleted file mode 100644 index 57b7921..0000000 --- a/site/tags/leader-follower model.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: leader-follower model - - - - - -
    - -

    Tag: leader-follower model

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: leader-follower model

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/linear programs.html b/site/tags/linear programs.html deleted file mode 100644 index 1f4022b..0000000 --- a/site/tags/linear programs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: linear programs - - - - - -
    - -

    Tag: linear programs

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/linear systems.html b/site/tags/linear systems.html deleted file mode 100644 index d96e2ea..0000000 --- a/site/tags/linear systems.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: linear systems - - - - - -
    - -

    Tag: linear systems

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/machine learning.html b/site/tags/machine learning.html deleted file mode 100644 index 4529d15..0000000 --- a/site/tags/machine learning.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: machine learning - - - - - -
    - -

    Tag: machine learning

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/maintainability.html b/site/tags/maintainability.html deleted file mode 100644 index 6b9d11b..0000000 --- a/site/tags/maintainability.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: maintainability - - - - - -
    - -

    Tag: maintainability

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/mapreduce.html b/site/tags/mapreduce.html deleted file mode 100644 index 26bdfe0..0000000 --- a/site/tags/mapreduce.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: mapreduce - - - - - -
    - -

    Tag: mapreduce

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/matching.html b/site/tags/matching.html deleted file mode 100644 index ad31407..0000000 --- a/site/tags/matching.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: matching - - - - - -
    - -

    Tag: matching

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/max flow min cut.html b/site/tags/max flow min cut.html deleted file mode 100644 index 30dab85..0000000 --- a/site/tags/max flow min cut.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: max flow min cut - - - - - -
    - -

    Tag: max flow min cut

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/message passing.html b/site/tags/message passing.html deleted file mode 100644 index c235915..0000000 --- a/site/tags/message passing.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: message passing - - - - - -
    - -

    Tag: message passing

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/meta.html b/site/tags/meta.html deleted file mode 100644 index 9801cc0..0000000 --- a/site/tags/meta.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Tag: meta - - - - - -
    - -

    Tag: meta

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/multinomial logistic regression.html b/site/tags/multinomial logistic regression.html deleted file mode 100644 index bcc50b8..0000000 --- a/site/tags/multinomial logistic regression.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: multinomial logistic regression - - - - - -
    - -

    Tag: multinomial logistic regression

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: multinomial logistic regression

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/natural language processing.html b/site/tags/natural language processing.html deleted file mode 100644 index 5d1d959..0000000 --- a/site/tags/natural language processing.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: natural language processing - - - - - -
    - -

    Tag: natural language processing

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: natural language processing

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/networking.html b/site/tags/networking.html deleted file mode 100644 index 2d076e1..0000000 --- a/site/tags/networking.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: networking - - - - - -
    - -

    Tag: networking

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: networking

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/neural networks.html b/site/tags/neural networks.html deleted file mode 100644 index fc6470d..0000000 --- a/site/tags/neural networks.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: neural networks - - - - - -
    - -

    Tag: neural networks

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: neural networks

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/odd cycles.html b/site/tags/odd cycles.html deleted file mode 100644 index 2bc0d66..0000000 --- a/site/tags/odd cycles.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: odd cycles - - - - - -
    - -

    Tag: odd cycles

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/oltp vs olap.html b/site/tags/oltp vs olap.html deleted file mode 100644 index 436e535..0000000 --- a/site/tags/oltp vs olap.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: oltp vs olap - - - - - -
    - -

    Tag: oltp vs olap

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/operating systems.html b/site/tags/operating systems.html deleted file mode 100644 index 87c860b..0000000 --- a/site/tags/operating systems.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: operating systems - - - - - -
    - -

    Tag: operating systems

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/optimization.html b/site/tags/optimization.html deleted file mode 100644 index d9802f0..0000000 --- a/site/tags/optimization.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Tag: optimization - - - - - -
    - -

    Tag: optimization

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/paper.html b/site/tags/paper.html deleted file mode 100644 index e5eabcb..0000000 --- a/site/tags/paper.html +++ /dev/null @@ -1,200 +0,0 @@ - - - - - - Tag: paper - - - - - -
    - -

    Tag: paper

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/partitioning.html b/site/tags/partitioning.html deleted file mode 100644 index 3473852..0000000 --- a/site/tags/partitioning.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: partitioning - - - - - -
    - -

    Tag: partitioning

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/performance.html b/site/tags/performance.html deleted file mode 100644 index 13bf46b..0000000 --- a/site/tags/performance.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: performance - - - - - -
    - -

    Tag: performance

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/pigeonhole principle.html b/site/tags/pigeonhole principle.html deleted file mode 100644 index fd756a1..0000000 --- a/site/tags/pigeonhole principle.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: pigeonhole principle - - - - - -
    - -

    Tag: pigeonhole principle

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/problem-solving.html b/site/tags/problem-solving.html deleted file mode 100644 index 9f4cbf5..0000000 --- a/site/tags/problem-solving.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: problem-solving - - - - - -
    - -

    Tag: problem-solving

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/proof techniques.html b/site/tags/proof techniques.html deleted file mode 100644 index f0c26cb..0000000 --- a/site/tags/proof techniques.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: proof techniques - - - - - -
    - -

    Tag: proof techniques

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/query languages.html b/site/tags/query languages.html deleted file mode 100644 index f1a72e1..0000000 --- a/site/tags/query languages.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: query languages - - - - - -
    - -

    Tag: query languages

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/relational databases.html b/site/tags/relational databases.html deleted file mode 100644 index 76caa2d..0000000 --- a/site/tags/relational databases.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: relational databases - - - - - -
    - -

    Tag: relational databases

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: relational databases

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/reliability.html b/site/tags/reliability.html deleted file mode 100644 index 3ea06df..0000000 --- a/site/tags/reliability.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: reliability - - - - - -
    - -

    Tag: reliability

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/replication logs.html b/site/tags/replication logs.html deleted file mode 100644 index ecee0a4..0000000 --- a/site/tags/replication logs.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: replication logs - - - - - -
    - -

    Tag: replication logs

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/research.html b/site/tags/research.html deleted file mode 100644 index 27757f0..0000000 --- a/site/tags/research.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Tag: research - - - - - -
    - -

    Tag: research

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/resource management.html b/site/tags/resource management.html deleted file mode 100644 index c6bd2f1..0000000 --- a/site/tags/resource management.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: resource management - - - - - -
    - -

    Tag: resource management

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/review.html b/site/tags/review.html deleted file mode 100644 index 2e9aff7..0000000 --- a/site/tags/review.html +++ /dev/null @@ -1,200 +0,0 @@ - - - - - - Tag: review - - - - - -
    - -

    Tag: review

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/scalability.html b/site/tags/scalability.html deleted file mode 100644 index 482e92e..0000000 --- a/site/tags/scalability.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: scalability - - - - - -
    - -

    Tag: scalability

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/scaling.html b/site/tags/scaling.html deleted file mode 100644 index 97e9ff9..0000000 --- a/site/tags/scaling.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: scaling - - - - - -
    - -

    Tag: scaling

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: scaling

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/scheduling.html b/site/tags/scheduling.html deleted file mode 100644 index d42622f..0000000 --- a/site/tags/scheduling.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: scheduling - - - - - -
    - -

    Tag: scheduling

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/schema evolution.html b/site/tags/schema evolution.html deleted file mode 100644 index 19a0620..0000000 --- a/site/tags/schema evolution.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: schema evolution - - - - - -
    - -

    Tag: schema evolution

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/set cover.html b/site/tags/set cover.html deleted file mode 100644 index e4e019a..0000000 --- a/site/tags/set cover.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: set cover - - - - - -
    - -

    Tag: set cover

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: set cover

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/shortest-paths.html b/site/tags/shortest-paths.html deleted file mode 100644 index e560d86..0000000 --- a/site/tags/shortest-paths.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: shortest-paths - - - - - -
    - -

    Tag: shortest-paths

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/spanning trees.html b/site/tags/spanning trees.html deleted file mode 100644 index b678469..0000000 --- a/site/tags/spanning trees.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: spanning trees - - - - - -
    - -

    Tag: spanning trees

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/stable matching.html b/site/tags/stable matching.html deleted file mode 100644 index 23b150f..0000000 --- a/site/tags/stable matching.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: stable matching - - - - - -
    - -

    Tag: stable matching

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/synchronous vs asynchronous.html b/site/tags/synchronous vs asynchronous.html deleted file mode 100644 index 7abccd1..0000000 --- a/site/tags/synchronous vs asynchronous.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: synchronous vs asynchronous - - - - - -
    - -

    Tag: synchronous vs asynchronous

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: synchronous vs asynchronous

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/system design.html b/site/tags/system design.html deleted file mode 100644 index 869a6eb..0000000 --- a/site/tags/system design.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: system design - - - - - -
    - -

    Tag: system design

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: system design

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/systems.html b/site/tags/systems.html deleted file mode 100644 index 6ea50d6..0000000 --- a/site/tags/systems.html +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Tag: systems - - - - - -
    - -

    Tag: systems

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/template.html b/site/tags/template.html deleted file mode 100644 index 3b69546..0000000 --- a/site/tags/template.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: template - - - - - -
    - -

    Tag: template

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: template

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/time complexity.html b/site/tags/time complexity.html deleted file mode 100644 index 70f8429..0000000 --- a/site/tags/time complexity.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: time complexity - - - - - -
    - -

    Tag: time complexity

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/trees.html b/site/tags/trees.html deleted file mode 100644 index 41b74f9..0000000 --- a/site/tags/trees.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: trees - - - - - -
    - -

    Tag: trees

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/tags/unix.html b/site/tags/unix.html deleted file mode 100644 index b5a4ef7..0000000 --- a/site/tags/unix.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Tag: unix - - - - - -
    - -

    Tag: unix

    -
    - Last modified: 2025-01-16 - -
    -
    -

    Tag: unix

    - -
    - -
    - - \ No newline at end of file diff --git a/site/tags/vertex cover.html b/site/tags/vertex cover.html deleted file mode 100644 index 1a68bf4..0000000 --- a/site/tags/vertex cover.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - - - Tag: vertex cover - - - - - -
    - -

    Tag: vertex cover

    -
    - Last modified: 2025-01-16 - -
    - - -
    - - \ No newline at end of file diff --git a/site/teaching/modern-java/collections-and-records.html b/site/teaching/modern-java/collections-and-records.html deleted file mode 100644 index 1705f70..0000000 --- a/site/teaching/modern-java/collections-and-records.html +++ /dev/null @@ -1,319 +0,0 @@ - - - - - - Collections And Records - - - - - -
    - -

    Collections And Records

    -
    - Last modified: 2024-12-08 - -
    -
    -

    Creating Collections and Data Types in Modern Java

    -

    Motivation

    -

    Often while testing your code or implementing common algorithms, you'll want to specify an immutable collection of elements. The UW intro series as (as far as I'm aware) doesn't teach you some pretty useful java features that can make this a lot easier.

    -

    Furthermore, java can be a little verbose when it comes to defining new data types to hold structured data. This is where the record keyword comes in.

    -

    Arrays.asList

    -

    The most basic way to create a list in Java is to use the Arrays.asList method. This method takes a variable number of arguments and returns a fixed-size list backed by the specified array. This means that you can't add or remove elements from the list, but you can modify the elements themselves.

    -
    List<Integer> list = Arrays.asList(1, 2, 3, 4, 5);
    -
    -

    Java 9+ Factory Methods

    -

    Java 9 introduced a new way to create immutable collections using factory methods. These methods are available in the List, Set, and Map interfaces. Here are some examples:

    -
    List<Integer> list = List.of(1, 2, 3, 4, 5);
    -Set<Integer> set = Set.of(1, 2, 3, 4, 5);
    -Map<Integer, String> map = Map.of(
    -  1, "one",
    -  2, "two",
    -  3, "three"
    -);
    -
    -

    Records

    -

    Records are a new feature in Java 14 that allow you to define simple data classes with minimal boilerplate. On top of being far more concise than traditional classes, records also provide a toString, equals, and hashCode method by default.

    -
    // With classes
    -class Point {
    -  int x;
    -  int y;
    -
    -  Point(int x, int y) {
    -    this.x = x;
    -    this.y = y;
    -  }
    -
    -  public String toString() {
    -    return String.format("(%d, %d)", x, y);
    -  }
    -
    -  public boolean equals(Object o) {
    -    if (o == this) return true;
    -    if (!(o instanceof Point)) return false;
    -    Point p = (Point) o;
    -    return p.x == x && p.y == y;
    -  }
    -
    -  public int hashCode() {
    -    return Objects.hash(x, y);
    -  }
    -}
    -
    -// With records
    -record Point(int x, int y) {}
    -
    -

    Using this in Practice

    -

    Say we're implementing a poker game and we want to represent a card. With introductory Java knowledge, you might define a class like this:

    -
    enum Suit {
    -  HEARTS, DIAMONDS, CLUBS, SPADES
    -}
    -
    -class Card {
    -  private final Suit suit;
    -  private final int rank;
    -
    -  Card(Suit suit, int rank) {
    -    this.suit = suit;
    -    this.rank = rank;
    -  }
    -
    -  public Suit getSuit() {
    -    return suit;
    -  }
    -
    -  public int getRank() {
    -    return rank;
    -  }
    -
    -  public String toString() {
    -    return String.format("%d of %s", rank, suit);
    -  }
    -
    -  public boolean equals(Object o) {
    -    if (o == this) return true;
    -    if (!(o instanceof Card)) return false;
    -    Card c = (Card) o;
    -    return c.suit == suit && c.rank == rank;
    -  }
    -
    -  public int hashCode() {
    -    return Objects.hash(suit, rank);
    -  }
    -}
    -
    -

    With records, you can define the same class in a much more concise way:

    -
    enum Suit {
    -  HEARTS, DIAMONDS, CLUBS, SPADES
    -}
    -
    -record Card(Suit suit, int rank) {}
    -
    -

    We can even add methods to records, like so:

    -
    record Card(Suit suit, int rank) implements Comparable<Card> {
    -  public int compareTo(Card other) {
    -    return Integer.compare(rank, other.rank);
    -  }
    -}
    -
    -

    Now say we wanted to test our new Card object. We could do something like this:

    -
    public static void main(String[] args) {
    -  var cards = List.of(
    -    new Card(Suit.CLUBS, 4),
    -    new Card(Suit.DIAMONDS, 3),
    -    new Card(Suit.HEARTS, 2),
    -    new Card(Suit.SPADES, 5)
    -  );
    -
    -  var expected = List.of(
    -    new Card(Suit.HEARTS, 2),
    -    new Card(Suit.DIAMONDS, 3),
    -    new Card(Suit.CLUBS, 4),
    -    new Card(Suit.SPADES, 5)
    -  );
    -
    -  cards.stream()
    -    .sorted()
    -    .toList();
    -
    -  assert cards.equals(expected);
    -}
    -
    -
    - -
    - - \ No newline at end of file diff --git a/site/teaching/modern-java/lambdas-and-streams.html b/site/teaching/modern-java/lambdas-and-streams.html deleted file mode 100644 index cc116f5..0000000 --- a/site/teaching/modern-java/lambdas-and-streams.html +++ /dev/null @@ -1,335 +0,0 @@ - - - - - - Lambdas And Streams - - - - - -
    - -

    Lambdas And Streams

    -
    - Last modified: 2024-12-08 - -
    -
    -

    A Soft Introduction to Java Streams and Lambdas

    -

    Motivation

    -

    If you've been programming in Java for a while (perhaps you're coming out of the 12x/14x series at UW), you're probably familiar with the regular imperative style of programming using loops and conditionals. There's nothing wrong with this, but often times, you'll find yourself writing a lot of code for simple operations.

    -

    For example, consider the following code snippet:

    -
    List<User> getAdmins(List<User> users) {
    -    List<User> admins = new ArrayList<>();
    -    for (User user : users) {
    -        if (user.isAdmin()) {
    -            admins.add(user);
    -        }
    -    }
    -    return admins;
    -}
    -
    -

    At a high level, we iterate over the provided list of users, and filter for those that are admins.

    -

    Perhaps you're also trying to make sure that all users are admins:

    -
    boolean allAdmins(List<User> users) {
    -    for (User user : users) {
    -        if (!user.isAdmin()) {
    -            return false;
    -        }
    -    }
    -    return true;
    -}
    -
    -

    You've probably written similar code snippets hundreds of times. To reiterate, there's nothing wrong with this style of programming, but for a relatively simple operation, you're writing a lot of code.

    -

    What's arguably better is to use a more declarative style of programming. This is where Java Streams and Lambdas come in.

    -

    Java Streams

    -

    Java Streams are in some way similar to Iterators, but with a lot more functionality. They allow you to perform operations on a collection of elements in a more compositional way. For example, consider the following code snippet:

    -
    List<User> admins = users.stream()
    -    .filter(user -> user.isAdmin())
    -    .collect(Collectors.toList());
    -
    -// note while using collect(Collectors.to*()) is going to work in most cases,
    -// when terminating a stream to a list you can also use the toList() method
    -List<User> admins = users.stream().filter(User::isAdmin).toList();
    -
    -

    There's a lot going on here, I know. Let's break it down:

    -
      -
    • users.stream(): This converts the list of users into a Stream (java.util.stream.Stream).
    • -
    • user -> user.isAdmin(): This is a Lambda expression. It's a way of passing a function as an argument. In this case, we're passing a function that takes a User object and returns a boolean. Alternatively, you could write this as User::isAdmin, meaning we call the isAdmin method on the User object passed in as an argument.
    • -
    • filter(...): This is an intermediate operation. It takes a predicate (a function that returns a boolean) and filters out elements that don't satisfy the predicate.
    • -
    • collect(...): This is a terminal operation. It collects the elements of the Stream into a list.
    • -
    -

    Lambdas

    -

    Lambdas are a way of defining functions without having to explicitly define a class or method. They're particularly useful when you want to pass a function as an argument to another function. Here are some different ways you can use lambdas:

    -
    // A lambda that takes no arguments and returns void
    -Runnable r = () -> System.out.println("Hello, world!");
    -
    -// A lambda that takes two integers and returns an integer
    -BiFunction<Integer, Integer, Integer> add = (a, b) -> a + b;
    -
    -// A lambda that takes a string and returns a string
    -Function<String, String> toUpperCase = s -> s.toUpperCase();
    -
    -// A lambda that takes a string and returns a boolean
    -Predicate<String> isEmpty = s -> s.isEmpty();
    -
    -

    Luckily, you don't need to remember all of these interfaces. More often than not, you're going to define lambdas without ever assigning them to a variable. In cases where you do need to assign them to a variable, you can use the var keyword.

    -

    Common Operations

    -

    Here are some examples of iterative vs. stream-based operations:

    -

    Filtering

    -
    // Imperative
    -List<User> admins = new ArrayList<>();
    -for (User user : users) {
    -    if (user.isAdmin()) {
    -        admins.add(user);
    -    }
    -}
    -
    -// Declarative
    -List<User> admins = users.stream()
    -    .filter(User::isAdmin)
    -    .toList();
    -
    -

    Mapping

    -
    // Imperative
    -List<String> names = new ArrayList<>();
    -for (User user : users) {
    -    names.add(user.getName());
    -}
    -
    -// Declarative
    -List<String> names = users.stream().map(User::getName).toList();
    -
    -

    All match

    -
    // Imperative
    -boolean allAdmins = true;
    -for (User user : users) {
    -    if (!user.isAdmin()) {
    -        allAdmins = false;
    -        break;
    -    }
    -}
    -
    -// Declarative
    -boolean allAdmins = users.stream().allMatch(User::isAdmin);
    -
    -

    Any match

    -
    // Imperative
    -boolean anyAdmins = false;
    -for (User user : users) {
    -    if (user.isAdmin()) {
    -        anyAdmins = true;
    -        break;
    -    }
    -}
    -
    -// Declarative
    -boolean anyAdmins = users.stream().anyMatch(User::isAdmin);
    -
    -

    Sum

    -
    // Imperative
    -int sum = 0;
    -for (int i : numbers) {
    -    sum += i;
    -}
    -
    -// Declarative
    -int sum = numbers.stream().reduce(0, Integer::sum);
    -
    -// alternatively, you can use the sum() method
    -int sum = numbers.stream().mapToInt(Integer::intValue).sum();
    -
    -// or Collectors.summingInt()
    -int sum = numbers.stream().collect(Collectors.summingInt(Integer::intValue));
    -
    -
    -

    Grouping

    -
    // Imperative
    -Map<String, List<User>> usersByRole = new HashMap<>();
    -for (User user : users) {
    -    usersByRole.computeIfAbsent(
    -      user.getRole(), k -> new ArrayList<>()
    -    ).add(user);
    -}
    -
    -// Declarative
    -Map<String, List<User>> usersByRole = users.stream()
    -    .collect(Collectors.groupingBy(User::getRole));
    -
    -
    - -
    - - \ No newline at end of file diff --git a/site/test.html b/site/test.html deleted file mode 100644 index 02b26c1..0000000 --- a/site/test.html +++ /dev/null @@ -1,197 +0,0 @@ - - - - - - Test - - - - - -
    - -

    Test

    -
    - Last modified: 2025-01-12 - -
    -
    -

    Directory

    -

    Who am I to question Apple's choices, but here we go. Was opting to make Apple Intelligence operate locally the best decision, considering privacy and security concerns? Is it really the smartest decision? If AI doesn't turn into a fiasco, limiting it to what the hardware can support seems like it will always lag behind in a world where technology evolves daily. 🌍📈

    -

    Besides, that leaves people who don't have the latest iPhone models holding the bag. ❄️ I understand Apple's conservative approach, but when it comes to AI, is it wise to just play it safe? 🤷‍♂️ Wouldn't an iCloud+ option with AI be a better alternative? ☁️⚙️

    -
    - -
    - - \ No newline at end of file diff --git a/site/tmp/bench/spec.html b/site/tmp/bench/spec.html deleted file mode 100644 index 945dcb9..0000000 --- a/site/tmp/bench/spec.html +++ /dev/null @@ -1,358 +0,0 @@ - - - - - - Spec - - - - - -
    - -

    Spec

    -
    - Last modified: 2024-05-11 - -
    -
    -

    SWECCathon Project Spec

    -

    Bench I/O format

    -

    Preamble

    -
    %d(dimension_in), %s(out_key)
    -%s(in_key_1), %s(in_key_2), ..., %s(in_key_{dimension_in})
    -%s(base_1), %s(base_2), ..., %s(base_{dimension_in})
    -%s(max_1), %s(max_2), ..., %s(max_{dimension_in})
    -
    -

    Data

    -
    %d(in_1), %d(in_2), ..., %d(in_{dimension_in}), %d(out)
    -%d(in_1), %d(in_2), ..., %d(in_{dimension_in}), %d(out)
    -...
    -
    -

    Input format

    -
    %d(argc), %s(argv_1), %s(argv_2), ..., %s(argv_{argc})
    -%d(argc), %s(argv_1), %s(argv_2), ..., %s(argv_{argc})
    -...
    -
    -

    Comp

    -

    Compare any number of .bio files and generate a benchmarking plot.

    -
    comp [options] <file1> <file2> ...
    -
    -

    Comp Options

    - - - - - - - - - - - - - - - - - -
    FlagTypeDescriptionDefault
    -ostrOutput file"benchmark.png"
    -

    Gen

    -

    Generate a .input file

    -
    gen [options]
    -
    -

    Gen Options

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FlagTypeDescriptionDefault
    -dintDimension of input vectors (number of input keys)1
    -bint,(CSV string of int)Base values
    -mint,(CSV string of int)Max values
    -sstrstrategy for generating input"randint"
    -ostrOutput file"out.bio"
    -

    Built-in strategies

    -

    | Strategy | Description | Compatibility Notes | -|----------|----------------------| -| randint | Random integers | requires b, m, compatible with all d | -| randfloat | Random floats | requires b, m, compatible with all d | -| randstr | Random strings | requires b, m, compatible with all d | -| randbool | Random booleans | requires b, m, compatible with all d | -| drandint | Distinct Random integers | requires b, m, compatible with all d | -| drandfloat | Distinct Random floats | requires b, m, compatible with all d | -| drandstr | Distinct Random strings | requires b, m, need d = 2 for string length as well | -| linear | Linear sequence of integers| requires b, m, compatible with all d |

    -

    Bench

    -

    Benchmark a function with a given input file.

    -
    bench [options] <file>
    -
    -

    Bench Options

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FlagTypeDescriptionDefault
    -dintDimension of input vectors (number of input keys)1
    -sstrOutput key"Time"
    -istr, (CSV string of strings)Input keys"Size"
    -bint, (CSV string of int)Base values"1"
    -mint, (CSV string of int)Max values"10"
    -incint, (CSV string of int)Increment values"1"
    -ostrOutput file"out.bio"
    -
    - -
    - - \ No newline at end of file diff --git a/site/tmp/pair.py b/site/tmp/pair.py deleted file mode 100644 index c110764..0000000 --- a/site/tmp/pair.py +++ /dev/null @@ -1,148 +0,0 @@ -import dataclasses -import numpy as np -from typing import List, Tuple -from model import SignupRecord - - -def f(availability1, availability2): - return np.sum(availability1 & availability2) - - -def g(response1, response2): - return np.linalg.norm(response1 - response2) - - -def normalize(matrix): - mn, mx = matrix.min(), matrix.max() - if mn == mx: - return matrix - return (matrix - mn) / (mx - mn) - -def next_pair(F, unpaired): - mrm, mrmi = np.inf, 0 - for i in unpaired: - row_max = max(F[i, j] for j in unpaired if j != i) - if row_max < mrm: - mrm = row_max - mrmi = i - - mrmj = 0 - for j in unpaired: - if j != mrmi and F[mrmi, j] == mrm: - mrmj = j - break - return mrmi, mrmj - - -def calc_pairs(F, unpairable=set()): - - unpaired = set(range(len(F))) - unpaired -= unpairable - pairs = [] - while len(unpaired) > 1: - p1, p2 = next_pair(F, unpaired) - unpaired.remove(p1) - unpaired.remove(p2) - pairs.append((p1, p2)) - - return pairs, unpaired | unpairable - - -def pair_signups(signups: List[SignupRecord]) -> Tuple[List[Tuple[str, str]], List[str]]: - availabilities = np.array([s.availability for s in signups]) - F_time = np.zeros((len(signups), len(signups))) - F_valid = np.zeros((len(signups), len(signups))) - - for i in range(len(signups)): - for j in range(len(signups)): - F_time[i, j] = f(availabilities[i], availabilities[j]) - F_valid[i, j] = 1 if i != j and F_time[i, j] != 0 else 0 - - unpairable = set() - for i, Fvr in enumerate(F_valid): - if max(Fvr) == 0: - unpairable.add(i) - - scores = normalize(F_time * F_valid) - - pairs_indices, unpaired_indicies = calc_pairs(scores, unpairable) - pairs = [(signups[p1].user_id, signups[p2].user_id) for p1, p2 in pairs_indices] - unpaired = [signups[i].user_id for i in unpaired_indicies] - return pairs, unpaired - - -import random -import uuid -import datetime - - -def generate_realistic_availability(sparse_probability: float = 0.3) -> List[List[int]]: - availability = np.zeros((7, 24), dtype=int) - - # typical working hours (9 AM - 5 PM) - runlen = int(random.random() * 5) + 1 - for day in range(5): # Monday to Friday - for hour in range(9, 17): # 9 AM to 5 PM - if runlen > 0: - availability[day][hour] = 1 - runlen -= 1 - else: - availability[day][hour] = 0 - runlen = int(random.random() * 5) + 1 - - # randomly set availability for weekends - for day in range(5, 7): # Saturday and Sunday - for hour in range(24): # 24 hours - if ( - random.random() < sparse_probability - ): - availability[day][hour] = 1 - - return availability.tolist() # Convert NumPy array to list - - -def generate_fake_data(num_entries: int) -> List[SignupRecord]: - sparse_probabilities = [0.1, 0.3, 0.5, 0.7, 0.9] - fake_data = [] - for _ in range(num_entries): - created_at = datetime.datetime.utcnow().isoformat() + "+00:00" - form_id = random.randint(1, 2) - availability = generate_realistic_availability( - random.choice(sparse_probabilities) - ) - user_id = str(uuid.uuid4()) - - fake_entry = SignupRecord( - created_at=created_at, - form_id=form_id, - availability=availability, - user_id=user_id, - ) - - fake_data.append(fake_entry) - - return fake_data - - -if __name__ == "__main__": - students = generate_fake_data(500) - pairs, unpaired = pair_signups(students) - - print("paired: ") - for pair in pairs: - print(pair) - - print("unpaired: ") - for student in unpaired: - print(student) - - if any(p[0] == p[1] for p in pairs): - print("Error: a student is paired with themselves") - - seen = set() - - for p in pairs: - if p[0] in seen or p[1] in seen: - print("Error: a student is paired with multiple students") - seen.add(p[0]) - seen.add(p[1]) \ No newline at end of file diff --git a/site/tmp/pairing-algorithm.html b/site/tmp/pairing-algorithm.html deleted file mode 100644 index 979310b..0000000 --- a/site/tmp/pairing-algorithm.html +++ /dev/null @@ -1,296 +0,0 @@ - - - - - - Pairing Algorithm - - - - - -
    - -

    Pairing Algorithm

    -
    - Last modified: 2024-03-22 - -
    -
    -

    Formalizing the Pairing Algorithm for the SWECC Mock Interview Program

    -

    Given $s$ signups, form pairs for all the students based on the following constraints: -- Each student should be paired with exactly one other student. If there are an odd number of students, one student will be left unpaired. -- A valid pair must have at least two 1-hour time slots in common in their availability. The number of common time slots is factored into the score of the pair. -- Each student responds to a form while signing up. The responses are vectorized, and pairings are made based on the similarity of the responses.

    -

    Statement of the Problem

    -

    Let $T^{7 \times 24}i$ be a bit-matrix of 1-hour time slots in a week for student $i$. The value of $T$ is 1 if student $i$ is available at time slot $(j,k)$, and 0 otherwise.

    -

    $$ -T_i = \begin{bmatrix} - t_{1,1} & t_{1,2} & t_{1,3}& \cdots & t_{1,24} \ - t_{2,1} & t_{2,2} & t_{2,3}& \cdots & t_{2,24} \ - \vdots & \vdots & \vdots & \ddots & \vdots \ - t_{7,1} & t_{7,2} & t_{7,3}& \cdots & t_{7,24} -\end{bmatrix} -$$

    -

    Let $S$ be the set of students, and $A$ be the set of all possible pairings of students in $S$

    -

    $$ -A = { (i, j) \mid (i, j) \in S \times S, i < j } -$$

    -

    Now, enforcing a strict and repeatable ordering on the students through sorting, we have a mapping from $s_i \in S$ to $T_i$.

    -

    We use this ordering to define a scoring function $f: A \to \mathbb{R}$ that takes a pair of students and returns the number of common time slots in their availability, or $0$ if it is less than $2$. We then create a matrix $F_{time}$ of the scores of all possible pairings.

    -

    $$ -F_{time} = \begin{bmatrix} - f(s_1, s_2) & f(s_1, s_3) & \cdots & f(s_1, s_n) \ - f(s_2, s_1) & f(s_2, s_3) & \cdots & f(s_2, s_n) \ - \vdots & \vdots & \ddots & \vdots \ - f(s_n, s_1) & f(s_n, s_2) & \cdots & f(s_n, s_{n-1}) -\end{bmatrix} -$$

    -

    Next, we follow a similar pattern using the responses of the students. Let $R_i$ be the vectorized response of student $i$.

    -

    $$ -R_i = \begin{bmatrix} - r_{1} \ - r_{2} \ - \vdots \ - r_{dim(R_i)} -\end{bmatrix} -$$

    -

    We define a scoring function $g: A \to \mathbb{R}$ as the euclidean distance between the responses of the students. We then create a matrix $F_{response}$ of the scores of all possible pairings.

    -

    $$ -F_{response} = \begin{bmatrix} - g(s_1, s_2) & g(s_1, s_3) & \cdots & g(s_1, s_n) \ - g(s_2, s_1) & g(s_2, s_3) & \cdots & g(s_2, s_n) \ - \vdots & \vdots & \ddots & \vdots \ - g(s_n, s_1) & g(s_n, s_2) & \cdots & g(s_n, s_{n-1}) -\end{bmatrix} -$$

    -

    Finally, we define a binary matrix $F_{valid}$ of the valid pairings. A valid pairing is one where the students have at least two common time slots in their availability, or rather when $F_{time, i, j} \neq 0$.

    -

    In python, this would look like:

    -
    # student_i has availability matrix availabilities[i] and response vector responses[i]
    -availabilities = [ ... ] # list of availability matrices. Each matrix is a 7x24 bit-matrix
    -responses = [ ... ] # list of response vectors
    -
    -def f(availability1, availability2):
    -    count = 0
    -    for i in availability1:
    -        for j in availability2:
    -            count += np.sum(i & j)
    -
    -    return count if count >= 2 else 0
    -
    -def g(response1, response2):
    -    return np.linalg.norm(response1 - response2)
    -
    -F_time = np.zeros((len(students), len(students)))
    -F_response = np.zeros((len(students), len(students)))
    -F_valid = np.zeros((len(students), len(students)))
    -
    -for i in range(len(students)):
    -    for j in range(len(students)):
    -        F_time[i, j] = f(availabilities[i], availabilities[j])
    -        F_response[i, j] = g(responses[i], responses[j])
    -        F_valid[i, j] = 1 if F_time[i, j] != 0 else 0
    -
    -

    Finally, we normalize (min max) F_time and F_response and compute their sum's hadamard product with F_valid to get the final score matrix $F_{final}$.

    -

    $$ -F_{final} = F_{valid} \odot \left( \frac{F_{time} - \min(F_{time})}{\max(F_{time}) - \min(F_{time})} + \frac{F_{response} - \min(F_{response})}{\max(F_{response}) - \min(F_{response})} \right) -$$

    -
    def normalize(matrix):
    -    return (matrix - np.min(matrix)) / (np.max(matrix) - np.min(matrix))
    -F_final = F_valid * (normalize(F_time) + normalize(F_response))
    -
    -

    Finally, using max-min fairness, we repeatedly find the student with the lowest max score for all their pairs, and select that maximal pair. We then update the scores of both students in the pair to be $0$ for all their pairs, and repeat the process until all students are paired.

    -

    ```python -def next_pair(F_final): - max_scores = [np.max(F_final[i]) for i in range(len(F_final))] - s1, s2 = np.argmin(max_scores), np.argmax(F_final[student]) - for i in range(len(F_final)): - F_final[s1, i] = 0 - F_final[i, s1] = 0 - F_final[s2, i] = 0 - F_final[i, s2] = 0 - return s1, s2

    -

    def calculate_pairs(F_final): - pairs = [] - unpaired = set(range(len(students))) - while len(pairs) < len(students) // 2: - pairs.append(next_pair(F_final)) - unpaired.remove(pairs[-1][0]) - unpaired.remove(pairs[-1][1]) - return pairs - ```

    -
    - -
    - - \ No newline at end of file diff --git a/site/tmp/pairings.png b/site/tmp/pairings.png deleted file mode 100644 index bbe332f..0000000 Binary files a/site/tmp/pairings.png and /dev/null differ diff --git a/site/tmp/toc.py b/site/tmp/toc.py deleted file mode 100644 index 03299d1..0000000 --- a/site/tmp/toc.py +++ /dev/null @@ -1,22 +0,0 @@ -import argparse -import mistletoe -from mistletoe.ast_renderer import ASTRenderer -import json -from types import SimpleNamespace - -def table_of_contents(ast): - for child in ast.children: - if child.type == 'Heading': - print('\t' * (child.level - 1) + f'- {child.children[0].content}') - - -def main(): - parser = argparse.ArgumentParser(description='Generate a table of contents for a markdown file') - parser.add_argument('file', type=str, help='The markdown file to generate a table of contents for') - args = parser.parse_args() - - with open(args.file, 'r') as f: - md = f.read() - - ast = mistletoe.create_markdown(md) - table_of_contents(ast) \ No newline at end of file diff --git a/site/tmp/tst.html b/site/tmp/tst.html deleted file mode 100644 index 73924a1..0000000 --- a/site/tmp/tst.html +++ /dev/null @@ -1,129 +0,0 @@ - -

    Consistency

    -

    Consistency: the allowed semantics of operations that mutate a data store/shared object.

    -

    Consistency specifies the interface (as opposed to implementation) for behavior of your system. It is essentially the contract between the programmer and implementer. An anomaly is a violation of the consistency semantics of the system

    -

    Types of Consistency

    - - - - - - - - - - - - - - - - - - - - - -
    TypeDescription
    Strong ConsistencyThe system behaves as if there is a single server. Systems that maintain a single consistent log of operations are often strongly consistent.
    Weak ConsistencyDefinitions vary, but basically just not strong consistency.
    Eventual ConsistencyWeak consistency with any anomalies guaranteed to be temporary.
    -

    Coordinating through a KV Store

    -
    def Produce(key, lock, command):
    -  result = application.execute(command)
    -  storage.put(key, result)
    -  storage.put(lock, True)
    -
    -def Consume(key, lock):
    -  while storage.get(lock) is False:
    -    pass
    -  return storage.get(key)
    -
    -

    With strong consistency semantics, the above approach works fine. However, with eventual consistency, and particularly for any system without multi-key transactions, we might see the update for storage.get(done) before the update for storage.get(key), leading to unexpected behavior.

    -

    Formalization

    -

    Read here for more info/theory.

    -

    For a given RPC, the initial request starts at time $t$ and the reply returns at time $t + x$. We cannot be sure what happens during $(t, t + x)$, since the request/reply could be lost and retransmitted, and intermediate coordination sometimes has to take place.

    -

    With only a single server, you don't know precisely when the operation takes place, but we expect it to be some time in $(t, t + x)$. However, weaker consistency models relax this assumption, also sometimes allowing different readers to see different results concurrently.

    -

    We use different models because of the following tradeoffs:

    -
      -
    • Performance: Consistency requires coordination, so there is often a tradeoff between the level of consistency and the performance of the system
    • -
    • Availability: If some client is offline or some network failure occurs, we might be forced to abandon strong consistency
    • -
    • Programmability: Weaker consistency models are harder to reason about and program with
    • -
    -

    Lamport's Register Semantics

    -

    Registers hold a single value, and we define operations $r_i, $w(v)$ as the $i$th read, and a write to the register with value $v$. Each operation has some starting time and ending time.

    -
      -
    • A read is safe if it is not concurrent with any write, and thus obtains the previously written value.
    • -
    • A read is regular if it is either safe, or if concurrent with a write, obtains either the old or new value.
    • -
    • A read/write is atomic if operations are safe, or if reads and writes behave as if they occur in some definite order.
    • -
    - - - - - - - - - - - - - - - - - - - - - -
    SemanticsConstraints
    safe$r_1 o v_1$
    regular$r_1 o v_1 \land (r_2 o v_1 \lor r_2 o v_2) \land (r_3 o v_1 \lor r_3 o v_2)$
    atomic$r_1 o v_1 \land (r_2 o v_1 \lor r_2 o v_2) \land (r_3 o v_1 \lor r_3 o v_2) \land (r_2 o v_2 \implies t_3 o v_2)$
    -
                r1           r2     r3
    -          |----|       |----| |----|
    -   w(v1)                w(v2)
    -|------|             |---------|
    -
    -

    Linearizability

    -

    A linearizable system is one in which actions appear to occur in a single global order that is consistent with real time/causal order. Not all systems enforce linearizability.

    -

    To do linearizable reads in Paxos, you need to first verify that the leader is still the leader at the time of the read. Otherwise, its possible that some other leader took over and formed a majority without the old leader. This can be done by waiting for the leader to execute some other request, which will only go through if we are indeed still the leader.

    -

    Linearizable Sharding with Paxos

    -

    For linearizability with shards, we have the following requirements:

    -
      -
    • All operations from the same node occur in order
    • -
    • All operations to the same shard occur in order
    • -
    • All operations complete between the request send and response receive.
    • -
    -

    Parallelism/concurrency of batched requests becomes difficult in a sharded system, since breaking up operations of a batched request into a pipeline completely throws out the original order of the request. We can instead think of systems in terms of a weaker consistency model.

    -

    Sequential Consistency

    -

    Sequential Consistency is a weaker form of consistency that requires all operations to be executed in some order that is consistent with the order in which they were issued. However, S.C. doesn't always follow real-time order. This is also referred to as serializability in the context of transactions.

    -

    Simplistically, we can think of sequential consistency as a system where all operations are executed in some order that is consistent with the order in which they were issued, but not necessarily during their window of request/response timing. This allows stale reads, while still maintaining some order that is consistent with a prefix of the global state of the system.

    -

    Snapshot Reads

    -

    Gives us a consistent view of our global state across some set of views of the system. This requires all operations being serializable, but it is okay if reads return stale data.

    -
      -
    • All reads in a transaction must be from the same snapshot
    • -
    • Client can define how old is too old for their usecase
    • -
    -

    To implement this (without sharding) in conjunction with Paxos, we can do the following:

    -
      -
    1. Primary defines update order in log
    2. -
    3. Shadow replicas apply changes in that order
    4. -
    5. Each lag primary from some variable amount
    6. -
    7. Snapshot reads occur at a single replica
    8. -
    9. If a replica crashes during a transaction, restart transaction at another snapshot replica
    10. -
    -

    Causal Consistency

    -
      -
    • Causally related reads and writes (ordered by happens before relation) must occur in that order.
    • -
    • Concurrent writes can be seen in different orders on different nodes
    • -
    • Note that linearizability imples causality
    • -
    -

    Processor Consistency

    -
      -
    • Writes done by the same process are seen in that order.
    • -
    • Writes by different processors can be seen in different orders by different readers
    • -
    -

    Memory Barrier/Fence

    -
      -
    • Whenever consistency matters, you can insert a "fence" in a point of time that says all preceding operations happen before the fence, and all subsequent operations happen after
    • -
    • On either side of the fence, order is not enforced
    • -
    • If every operation is fenced, your system is linearizable
    • -
    -

    This is how POSIX files work. Also many mutli-cache systems use fences to enforce consistency.

    diff --git a/systems-research/development-of-the-dns.md b/systems-research/development-of-the-dns.md new file mode 100644 index 0000000..e93abb7 --- /dev/null +++ b/systems-research/development-of-the-dns.md @@ -0,0 +1,99 @@ +--- +title: Development of the Domain Name System +category: systems +tags: dns, domain-name-system, networking, systems +description: Paper review of the paper Development of the Domain Name System +--- + +# [source](https://courses.cs.washington.edu/courses/cse551/09sp/papers/dns.pdf) + +###### Development of the Domain Name System + +--- + +### What is the Problem? + +The original solution for naming was to share single file `HOSTS.TXT` that contained all the hostnames and their corresponding IP addresses. This originally worked fine, since the number of hosts was proportional to the number of timesharing systems. However, as the internet evolved to consist of many networks, each with many hosts, this solution became unscalable. Instead, a distributed database was needed to store the mappings of hostnames to IP addresses. This paper describes the development of the Domain Name System (DNS) to solve this problem. + +### Summary + +#### Design Requirements + +- Provide all current functionality of `HOSTS.TXT` +- Allow for distributed maintenance +- No obvious size limits for names, name components, data associated with names, etc. +- Interoperability with existing systems (DARPA Internet) +- Tolerable performance +- Independence from the underlying network topology, and ability to encapsulate other name spaces +- Avoid forcing a single OS/architecture/organization structure. Should support both large time-sharing systems and individual PCs + +#### Architecture + +Two main components: +- **name server**: repository of mappings of names to data, responsible for answering queries +- **resolver**: interface for client programs to query name servers + +The lines are blurred, and particular this architecture allows for organizations to maintain a centralized name server/resolver to be shared by all hosts in the organization, meaning PCs wouldn't need to run their own resolver to resolve names. + +#### Name space + +Domains are organized hierarchically, with all names sharing a common "null" root. DNS doesn't make any assumptions about the structure or presentation of names, but does suggest that domains should model the organization they represent. + +#### Data attached to names + +DNS allows for arbitrary data to be attached to names, but does organize data into types. Each name has corresponding **resource records** (RRs) that contain a type and class, as well as unstructured application data. Type represents the abstract resource, and class represents the protocol family/instance, or in some case functionality (e.g. a universal mail registry). + +Types and classes were originally expected to be often extended, but it is pretty rare in practice, and more bits than necessary were allocated to these fields. + +Interestingly, when responding to queries, a name server is free to return any data in wants in addition to the requested data. It can therefore anticipate future queries to cut down on the number of round trips. For example, root servers include both the host address, and name when passing back the name of a host. + +#### Database distribution + +##### Zones + +A complete description of a contiguous portion of the domain name space, along with some "pointers" to other zones. Zones can be either a single node, or the whole tree, but are typically a subtree. + +An organization can get control of a zone by persuading its parent to insert RRs in its zone to mark a zone division, e.g. the CS dept. got `cs.washington.edu` by having `washington.edu` insert RRs to mark a zone boundary between `washington.edu` and `cs.washington.edu`. + +An organization managing a zone should provide redundancy by having multiple name servers for the zone, and should maintain a "master file" and make it available within a "master" server. Then, secondary servers are either manually refreshed, or use a zone refresh algorithm which queries serial numbers and updates if it has increase. Zone transfers happen over TCP. + +NSs can support any number of zones, which may or may not be contiguous. In fact, they don't even need to be in corresponding zone. When an NS responds to a query without cached data, it marks that response as "authoritative". + +##### Caching + +Each RR has an associated TTL in seconds, which is the maximum time a resolver can reuse the cached record (zero TTL means no caching). The administrator of a zone sets the TTL for each RR as part of their zone definition. + +Cached answers should be "as good" as authoritative answers, and the design seeks to enable updates before TTL expiration. If both are available, the authoritative answer should be preferred. + +As am optimization, negative responses should also be cached, e.g. non existent domain, or domain with no data associated. + +#### Root servers + +Resolvers search "downward" from domains they already know. They usually also have "hints" pointing at root servers and the top of their local domain. Thus, if a resolver can find a root server, it can find any domain. On the other hand, if a resolver is partitioned, it can at least resolve names in its local domain. + +Root servers must be highly available and distributed geographically. + +### Key Insights + +- Seeing as how the internet is distributed in nature, a distributed database/namespace management system was inevitable. The predecessor, `HOSTS.TXT`, simply couldn't scale with the growth of the internet, both in terms of complexity and size. +- DNS was designed around very general requirements for functionality, but also struck a balance between flexibility and simplicity in order to be widely adoptive and performant. + +### Notable Design Details/Strengths + +- Using a hierarchical structure gave a very powerful way to model namespaces after organizations, while also giving very natural ways to distribute and manage data, ultimately proving to be an extremely effective interface for the problem at hand. +- Caching being such a central part of the design made it possible to achieve good performance and availability, while remaining simple in terms of policy and implementation. In particular, negative caching is a very general but effective optimization. + +### Limitations/Weaknesses + +- Decentralized management can lead to inconsistencies in cached answers, difficulty in pushing updates through the system in real-time, accountability issues, etc. +- The system doesn't expose versioning metadata,nor any way of tuning runtime performance, which could be useful for some applications/organizations with specific needs. + +### Summary of Key Results + +- DNS successfully replaced `HOSTS.TXT` with a far more scalable and extensible system, particularly in terms of growth in complexity and size of the internet. +- Although the underlying internet was far less performant than initially expected, DNS was able to cope with this by making the common case of cached answers very fast, especially considering the multiple levels of caching involved which drastically reduced the number of round trips needed to resolve a name after some initial queries. + +### Open Questions + +- What level of security was considered in the design of DNS, and how has this evolved over time? +- Are type/class fields still over-allocated? If so, could it be possible to introduce some sort of versioning within these fields? diff --git a/site/tmp/bench/gen.py b/systems-research/modules.md similarity index 100% rename from site/tmp/bench/gen.py rename to systems-research/modules.md diff --git a/systems-research/papers/darpa.pdf b/systems-research/papers/darpa.pdf new file mode 100644 index 0000000..95660bc Binary files /dev/null and b/systems-research/papers/darpa.pdf differ diff --git a/systems-research/papers/dns.pdf b/systems-research/papers/dns.pdf new file mode 100644 index 0000000..cc4946b Binary files /dev/null and b/systems-research/papers/dns.pdf differ diff --git a/systems-research/papers/endtoend.pdf b/systems-research/papers/endtoend.pdf new file mode 100644 index 0000000..7ac8fd8 Binary files /dev/null and b/systems-research/papers/endtoend.pdf differ diff --git a/systems-research/papers/exokernel.pdf b/systems-research/papers/exokernel.pdf new file mode 100644 index 0000000..330ce77 Binary files /dev/null and b/systems-research/papers/exokernel.pdf differ diff --git a/systems-research/papers/hints-for-computer-system-design.pdf b/systems-research/papers/hints-for-computer-system-design.pdf new file mode 100644 index 0000000..0ce6bd4 Binary files /dev/null and b/systems-research/papers/hints-for-computer-system-design.pdf differ diff --git a/systems-research/papers/how-to-give-a-bad-talk.pdf b/systems-research/papers/how-to-give-a-bad-talk.pdf new file mode 100644 index 0000000..da657ae Binary files /dev/null and b/systems-research/papers/how-to-give-a-bad-talk.pdf differ diff --git a/systems-research/papers/how-to-read-a-paper.pdf b/systems-research/papers/how-to-read-a-paper.pdf new file mode 100644 index 0000000..fe43605 Binary files /dev/null and b/systems-research/papers/how-to-read-a-paper.pdf differ diff --git a/systems-research/papers/strong-inference.pdf b/systems-research/papers/strong-inference.pdf new file mode 100644 index 0000000..e7be255 Binary files /dev/null and b/systems-research/papers/strong-inference.pdf differ diff --git a/systems-research/papers/unix-timesharing-system.pdf b/systems-research/papers/unix-timesharing-system.pdf new file mode 100644 index 0000000..e6f05b5 Binary files /dev/null and b/systems-research/papers/unix-timesharing-system.pdf differ diff --git a/systems-research/papers/xen.pdf b/systems-research/papers/xen.pdf new file mode 100644 index 0000000..3f0b815 Binary files /dev/null and b/systems-research/papers/xen.pdf differ