diff --git a/content/codex/updates/2025-07-21.md b/content/codex/updates/2025-07-21.md deleted file mode 100644 index 69915f5120..0000000000 --- a/content/codex/updates/2025-07-21.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: "2025-07-21 Codex weekly" -tags: - - "codex-updates" -date: 2025-07-28 -lastmod: 2025-07-28 -draft: false ---- - -## Week July 2025 -Updates on what the past couple of weeks of discussions and refocusing have brought so far. Directions Codex is taking + still ongoing conversations. - -### Documents and Knowledge -Codex has had a long-standing problem of documents and knowledge being locked into ad hoc places and contributor's personal accounts and getting lost. This week, we finally made a step towards addressing that. TL;DR, we'll have: - -* a [team HackMD](https://hackmd.io/team/codex-storage) (as opposed to personal accounts) for quick collaboration and ephemeral documents; -* git(hub) as the eventual place where everything "important" should be. - -This is good because: - -* team HackMD means we don't lose sight of documents anymore: it's full-text-searchable, it's markdown, supports github sync, and is very low friction for collaboration and quick-and-dirty docs; -* people who prefer Obsidian can use it, as the github repo is a just an Obsidian vault; -* github flows (PRs and comments on PRs) are still possible; -* the vault builds to a quartz website so it's also conveniently publicly visible/searchable; -* last, but not least, it sits well with the team's preferences. - -### Short-term Roadmap - -As part of the effort in re-steering the project towards Logos and faster releases, we have been intensely dicussing internal use cases to get our short-term roadmap going: - - - Discussion on Waku's requirements: https://github.com/waku-org/pm/pull/328 - - Thoughts on assumptions for quick wins: https://hackmd.io/OQaVT3wVScyNAyEO3UJQbQ - -Shout out to Václav, Franck, and the Waku team for being so welcoming and supporting to our efforts. We have also been talking to the Status team and should have more on that shortly. - -### Altruistic Mode - -**Can we build a meaningful Codex without Marketplace?** There is a hypothesis that thinking of Codex without a marketplace could make things simpler, maybe even revealing a path towards faster, incremental releases. But will it really? We are trying to find out. So far we don't a lot to share, but here goes some unfiltered brain dumps for you: - -1. Balazs' brain dump: https://hackmd.io/qgsdVzJ6SFC7ZVpDfWTmKQ -2. Giuliano's brain dump: https://hackmd.io/vuWTutZiSUalVZp5YRcImQ -3. Eric's brain dump: https://hackmd.io/ytUZ7Ik6QwuWbXmfcBe0Qg - -### Eco Dev/comms -- Update the Codex community regarding upcoming changes -- Workshop in Zanzalu where we are gonna use: - - Testnet for the work shop and get users in. We use it for feedback. And any changed roadmap will basically motivated by "after collecting feedback from users, we've decided to adjust the roadmap priorities" - - -### Focus for next week - -- Energy will continue in building an aligned/meaningful short-term roadmap, and figuring out the ins and outs of an altruistic mode. diff --git a/content/codex/updates/2025-07-28.md b/content/codex/updates/2025-07-28.md index 15abd1882a..0a3489c54e 100644 --- a/content/codex/updates/2025-07-28.md +++ b/content/codex/updates/2025-07-28.md @@ -2,45 +2,51 @@ title: "2025-07-28 Codex weekly" tags: - "codex-updates" -date: 2025-08-01 -lastmod: 2025-08-11 +date: 2025-07-28 +lastmod: 2025-07-28 draft: false --- -## Week 31 July 2025 +## Week 30 July 2025 +Updates on what the past couple of weeks of discussions and refocusing have brought so far. Directions Codex is taking + still ongoing conversations. -### nim libs +### Documents and Knowledge +Codex has had a long-standing problem of documents and knowledge being locked into ad hoc places and contributor's personal accounts and getting lost. This week, we finally made a step towards addressing that. TL;DR, we'll have: -* ABI encoder / decoder integration in web3 (https://github.com/status-im/nim-web3/pull/216) - * Complete decoding - * Fix dynamic data encoding - * Integration with nim-serialization - * Benchmark (https://github.com/2-towns/nim-abi-benchmark) -* extracted async iterators to a spearate Nim package - initial version, work in progress: https://github.com/codex-storage/nim-asynciterators +* a [team HackMD](https://hackmd.io/team/codex-storage) (as opposed to personal accounts) for quick collaboration and ephemeral documents; +* git(hub) as the eventual place where everything "important" should be. -### Altruistic mode +This is good because: -Use cases have been more or less understood through requirements gathering efforts. +* team HackMD means we don't lose sight of documents anymore: it's full-text-searchable, it's markdown, supports github sync, and is very low friction for collaboration and quick-and-dirty docs; +* people who prefer Obsidian can use it, as the github repo is a just an Obsidian vault; +* github flows (PRs and comments on PRs) are still possible; +* the vault builds to a quartz website so it's also conveniently publicly visible/searchable; +* last, but not least, it sits well with the team's preferences. -#### Waku requirements +### Short-term Roadmap -Final round of reviews went into the [FURPs requirements](https://github.com/waku-org/pm/pull/328), which were merged. After much discussion, the Waku use cases are summarised in https://hackmd.io/@codex-storage/B1DYfESPlg. +As part of the effort in re-steering the project towards Logos and faster releases, we have been intensely dicussing internal use cases to get our short-term roadmap going: -#### Status desktop requirements + - Discussion on Waku's requirements: https://github.com/waku-org/pm/pull/328 + - Thoughts on assumptions for quick wins: https://hackmd.io/OQaVT3wVScyNAyEO3UJQbQ -An initial meeting with Jo helped us get a better understanding of what was needed from Status Desktop. Plans to create FURPs the following week (spoiler: https://github.com/status-im/status-desktop/pull/18483) +Shout out to Václav, Franck, and the Waku team for being so welcoming and supporting to our efforts. We have also been talking to the Status team and should have more on that shortly. -#### New altruistic design ideas +### Altruistic Mode -* Rahul -- [using an altrustic version of Codex v1](https://hackmd.io/@codex-storage/H1uDkyLvxx) -* Eric -- [replication network](https://hackmd.io/Z3j1YBZ2SF6XKykRJ1ZF2g?both) +**Can we build a meaningful Codex without Marketplace?** There is a hypothesis that thinking of Codex without a marketplace could make things simpler, maybe even revealing a path towards faster, incremental releases. But will it really? We are trying to find out. So far we don't a lot to share, but here goes some unfiltered brain dumps for you: -#### Research efforts +1. Balazs' brain dump: https://hackmd.io/qgsdVzJ6SFC7ZVpDfWTmKQ +2. Giuliano's brain dump: https://hackmd.io/vuWTutZiSUalVZp5YRcImQ +3. Eric's brain dump: https://hackmd.io/ytUZ7Ik6QwuWbXmfcBe0Qg -* Explored DHT light client architectures and evaluated the JS-IPFS implementation. -* Investigating privacy-preserving approaches for message propagation in P2P overlay networks. -* [DHT scaling formulations](https://hackmd.io/@codex-storage/rkIWizSvee), including bandwidth estimates for content distribution, query traffic, maintenance, and repair. - * Generated traffic matrices across varying network sizes. -* Learning the details of multivariate zk protocols. [Draft writeup on sum-check](https://hackmd.io/_3xKeSMKQZ6P-kDxBEhueA) -* Started researching light-weight (off-chain) approaches to sampling proofs - KZG-based, FRI-based, and Ligero-based (mainly ZODA). Plan is to explore these more in the coming days for the altruistic mode. -* Started work on making the plonky2 proof aggregation tree as a standalone general-use repo that can possibly be used by other projects that requires N-to-1 proof tree aggregation. +### Eco Dev/comms +- Update the Codex community regarding upcoming changes +- Workshop in Zanzalu where we are gonna use: + - Testnet for the work shop and get users in. We use it for feedback. And any changed roadmap will basically motivated by "after collecting feedback from users, we've decided to adjust the roadmap priorities" + + +### Focus for next week + +- Energy will continue in building an aligned/meaningful short-term roadmap, and figuring out the ins and outs of an altruistic mode. diff --git a/content/codex/updates/2025-08-04.md b/content/codex/updates/2025-08-04.md index ab112e5581..5614e003ae 100644 --- a/content/codex/updates/2025-08-04.md +++ b/content/codex/updates/2025-08-04.md @@ -2,85 +2,45 @@ title: "2025-08-04 Codex weekly" tags: - "codex-updates" -date: 2025-08-08 -lastmod: 2025-08-11 +date: 2025-08-04 +lastmod: 2025-08-04 draft: false --- -## Week 32 August 2025 +## Week 31 July 2025 ### nim libs -* Complete the ABI encoder / decoder PR (https://github.com/status-im/nim-web3/pull/216) - * Fix offsets encoding issue - * Use `nim-fastream` api advance to avoid loading the whole buffer in memory for dynamic data - * Apply PR's feedback (comments, handling int overflow decoding...) - * Replace the old encode / decode functions by the new decoder in nim-web3 - * Test in eth1 (https://github.com/status-im/nimbus-eth1/pull/3554) - * Testing in eth2 and nwaku (in progress) +* ABI encoder / decoder integration in web3 (https://github.com/status-im/nim-web3/pull/216) + * Complete decoding + * Fix dynamic data encoding + * Integration with nim-serialization + * Benchmark (https://github.com/2-towns/nim-abi-benchmark) +* extracted async iterators to a spearate Nim package - initial version, work in progress: https://github.com/codex-storage/nim-asynciterators ### Altruistic mode -#### Status desktop requirements - -Status desktop requirements FURPs PR has been created: https://github.com/status-im/status-desktop/pull/18483) - -#### New altruistic design ideas - -* Eric -- [replication network alternative](https://hackmd.io/@codex-storage/BkN_REgulx) -* Eric -- [replication network alternative -- over simplified assumptions](https://hackmd.io/@codex-storage/S1DWaqW_ex) -* Balazs -- additions to the [altruistic musings](https://hackmd.io/jN6IN1qQQyaVSt9uGhHu9Q) - -### Specs - -Kicked off specs effort for existing modules. -Template: https://hackmd.io/@codex-storage/rkbytlbOxg -Kanban of components to spec: -https://github.com/orgs/codex-storage/projects/6/views/1 -Due 15 August -Specs will be contained in https://github.com/codex-storage/codex-docs-obsidian +Use cases have been more or less understood through requirements gathering efforts. -### Offsite +#### Waku requirements -Have more or less decided on mid-November for the offsite, to come after a -prototype is delivered, as there is not a lot of other convenient times due to -holidays and other availability considerations (eg paternity leave). Location -will be in Europe to keep costs down. Initial idea is Canary Islands but could -move to more central Europe to drop costs further. +Final round of reviews went into the [FURPs requirements](https://github.com/waku-org/pm/pull/328), which were merged. After much discussion, the Waku use cases are summarised in https://hackmd.io/@codex-storage/B1DYfESPlg. -### Roadmap - -We are working towards a roadmap that replaces bittorrent in status-go with -a prototype of the new Codex. - -```mermaid - -gantt - title Codex Prototype Roadmap 2025 - excludes weekends - dateFormat YYYY-MM-DD - - Bittorent replacement prototype :milestone, m01, 2025-11-15, 1d - - section 0.1 - Existing component specs :specs, 2025-08-11, 5d - Working groupathon :wg, after specs, 10d - New protocol specs :newspecs, after wg, 10d - Engineering :eng, 2025-09-08, 50d -``` +#### Status desktop requirements -1. Write existing component specs, 11-15 August -2. Create a new design validated by the requirements. Deliver roadmap. (working - groupathon), 18-29 August -3. Write specs for new protocol design, 1-12 Sept -4. Start engineering, deliver BT replacement *prototype* in `status-go`, 8 September-14 Nov +An initial meeting with Jo helped us get a better understanding of what was needed from Status Desktop. Plans to create FURPs the following week (spoiler: https://github.com/status-im/status-desktop/pull/18483) -### Solutions engineering +#### New altruistic design ideas -* Updates to constellations prototype: automatic support feature, stability improvements. +* Rahul -- [using an altrustic version of Codex v1](https://hackmd.io/@codex-storage/H1uDkyLvxx) +* Eric -- [replication network](https://hackmd.io/Z3j1YBZ2SF6XKykRJ1ZF2g?both) -### Research +#### Research efforts -* Completed hotspot calculations for Zipf-based query distribution. -* Researched on Dandelion++ (Stem & Fluff) based protocols -* [Altruistic musings](https://hackmd.io/jN6IN1qQQyaVSt9uGhHu9Q) +* Explored DHT light client architectures and evaluated the JS-IPFS implementation. +* Investigating privacy-preserving approaches for message propagation in P2P overlay networks. +* [DHT scaling formulations](https://hackmd.io/@codex-storage/rkIWizSvee), including bandwidth estimates for content distribution, query traffic, maintenance, and repair. + * Generated traffic matrices across varying network sizes. +* Learning the details of multivariate zk protocols. [Draft writeup on sum-check](https://hackmd.io/_3xKeSMKQZ6P-kDxBEhueA) +* Started researching light-weight (off-chain) approaches to sampling proofs - KZG-based, FRI-based, and Ligero-based (mainly ZODA). Plan is to explore these more in the coming days for the altruistic mode. +* Started work on making the plonky2 proof aggregation tree as a standalone general-use repo that can possibly be used by other projects that requires N-to-1 proof tree aggregation. diff --git a/content/codex/updates/2025-08-11.md b/content/codex/updates/2025-08-11.md index 581702edb9..d99fb4fac2 100644 --- a/content/codex/updates/2025-08-11.md +++ b/content/codex/updates/2025-08-11.md @@ -2,40 +2,85 @@ title: "2025-08-11 Codex weekly" tags: - "codex-updates" -date: 2025-08-15 -lastmod: 2025-08-19 +date: 2025-08-11 +lastmod: 2025-08-11 draft: false --- -## Week 33 August 2025 +## Week 32 August 2025 ### nim libs -* Update web3 in nimbus eth 2: https://github.com/status-im/nimbus-eth2/pull/7376 +* Complete the ABI encoder / decoder PR (https://github.com/status-im/nim-web3/pull/216) + * Fix offsets encoding issue + * Use `nim-fastream` api advance to avoid loading the whole buffer in memory for dynamic data + * Apply PR's feedback (comments, handling int overflow decoding...) + * Replace the old encode / decode functions by the new decoder in nim-web3 + * Test in eth1 (https://github.com/status-im/nimbus-eth1/pull/3554) + * Testing in eth2 and nwaku (in progress) -### Specs +### Altruistic mode + +#### Status desktop requirements -The majority of the team's effort this week have been in the production of -component and protocol specifications. +Status desktop requirements FURPs PR has been created: https://github.com/status-im/status-desktop/pull/18483) -Three component specs are ready for formalisation by the RFC team: +#### New altruistic design ideas -1. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Discovery.md -2. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Erasure%20Coding.md -3. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Purchase.md +* Eric -- [replication network alternative](https://hackmd.io/@codex-storage/BkN_REgulx) +* Eric -- [replication network alternative -- over simplified assumptions](https://hackmd.io/@codex-storage/S1DWaqW_ex) +* Balazs -- additions to the [altruistic musings](https://hackmd.io/jN6IN1qQQyaVSt9uGhHu9Q) -The remaining component and protocol specs are in review: +### Specs -1. [Sales module specification](https://github.com/codex-storage/codex-docs-obsidian/pull/11) -1. [Proving Module Specification](https://github.com/codex-storage/codex-docs-obsidian/pull/12) -1. [Codex Manifest Spec](https://github.com/codex-storage/codex-docs-obsidian/pull/13) -1. [Store Module Specifications](https://github.com/codex-storage/codex-docs-obsidian/pull/14) -1. [Slot building specification](https://github.com/codex-storage/codex-docs-obsidian/pull/15) -1. [initial draft - block exchange module](https://github.com/codex-storage/codex-docs-obsidian/pull/16) -1. [Codex protocol](https://github.com/codex-storage/codex-docs-obsidian/pull/17) +Kicked off specs effort for existing modules. +Template: https://hackmd.io/@codex-storage/rkbytlbOxg +Kanban of components to spec: +https://github.com/orgs/codex-storage/projects/6/views/1 +Due 15 August +Specs will be contained in https://github.com/codex-storage/codex-docs-obsidian ### Offsite -Dates settled November 17-22. -Location still TBD, but will be in Europe to keep costs down. +Have more or less decided on mid-November for the offsite, to come after a +prototype is delivered, as there is not a lot of other convenient times due to +holidays and other availability considerations (eg paternity leave). Location +will be in Europe to keep costs down. Initial idea is Canary Islands but could +move to more central Europe to drop costs further. + +### Roadmap + +We are working towards a roadmap that replaces bittorrent in status-go with +a prototype of the new Codex. + +```mermaid + +gantt + title Codex Prototype Roadmap 2025 + excludes weekends + dateFormat YYYY-MM-DD + + Bittorent replacement prototype :milestone, m01, 2025-11-15, 1d + + section 0.1 + Existing component specs :specs, 2025-08-11, 5d + Working groupathon :wg, after specs, 10d + New protocol specs :newspecs, after wg, 10d + Engineering :eng, 2025-09-08, 50d +``` + +1. Write existing component specs, 11-15 August +2. Create a new design validated by the requirements. Deliver roadmap. (working + groupathon), 18-29 August +3. Write specs for new protocol design, 1-12 Sept +4. Start engineering, deliver BT replacement *prototype* in `status-go`, 8 September-14 Nov + +### Solutions engineering + +* Updates to constellations prototype: automatic support feature, stability improvements. + +### Research +* Completed hotspot calculations for Zipf-based query distribution. +* Researched on Dandelion++ (Stem & Fluff) based protocols +* [Altruistic musings](https://hackmd.io/jN6IN1qQQyaVSt9uGhHu9Q) diff --git a/content/codex/updates/2025-08-18.md b/content/codex/updates/2025-08-18.md index 3f9377b32a..c14ddbcdd7 100644 --- a/content/codex/updates/2025-08-18.md +++ b/content/codex/updates/2025-08-18.md @@ -3,31 +3,39 @@ title: "2025-08-18 Codex weekly" tags: - "codex-updates" date: 2025-08-18 -lastmod: 2025-09-01 +lastmod: 2025-08-18 draft: false --- -## Week 34 August 2025 +## Week 33 August 2025 -This week was a dedicated “Groupathon” where the team split into 2 working groups and tackled: +### nim libs -1. the fastest way to replace Bittorrent with Codex for archival storage in status-go; -2. what the future of Codex might look like. +* Update web3 in nimbus eth 2: https://github.com/status-im/nimbus-eth2/pull/7376 -Problem statement, design goals and agenda can be found here: https://hackmd.io/JMuXL9AOSM6228KsgcTfXA +### Specs -### Outcomes +The majority of the team's effort this week have been in the production of +component and protocol specifications. -**Team Eurasia Submission** -[Presentation recording part 1](https://drive.google.com/file/d/19ntoDboaSEhBztkC5RC7r7Apy1U-Hthe/view) -[Presentation recording part 2](https://drive.google.com/file/d/1Sp-Rd0RhRtecmi4yHqSBI_4t4ehbh6CZ/view) -[Presentation slides](https://docs.google.com/presentation/d/1w1TdnTyJik2Ll31xQloR6V-6YyzeY_CswYQgDCQaABA/edit?slide=id.g158930d3f51_0_105#slide=id.g158930d3f51_0_105) +Three component specs are ready for formalisation by the RFC team: -**Team NL-BR** -[Team NL-BR Presentation Recording](https://drive.proton.me/urls/0D3VX0Q8VM#EWf9Kb5QPbXo) -[Team NL-BR Presentation](https://link.excalidraw.com/readonly/WENb1UXb9n2RYdvCKguP) -[Team NL-BR main document](https://hackmd.io/@codex-storage/BkCvWMVtlx) +1. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Discovery.md +2. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Erasure%20Coding.md +3. https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Component%20Specification%20-%20Purchase.md -Balazs proposal for longer term design: https://hackmd.io/6OGb1OKiRD-w6qxP-yDcJQ?view#A-possible-incremental-rollout-plan +The remaining component and protocol specs are in review: + +1. [Sales module specification](https://github.com/codex-storage/codex-docs-obsidian/pull/11) +1. [Proving Module Specification](https://github.com/codex-storage/codex-docs-obsidian/pull/12) +1. [Codex Manifest Spec](https://github.com/codex-storage/codex-docs-obsidian/pull/13) +1. [Store Module Specifications](https://github.com/codex-storage/codex-docs-obsidian/pull/14) +1. [Slot building specification](https://github.com/codex-storage/codex-docs-obsidian/pull/15) +1. [initial draft - block exchange module](https://github.com/codex-storage/codex-docs-obsidian/pull/16) +1. [Codex protocol](https://github.com/codex-storage/codex-docs-obsidian/pull/17) + +### Offsite + +Dates settled November 17-22. +Location still TBD, but will be in Europe to keep costs down. -**Tuesday 26th - Discussion and decision on a new design!** \ No newline at end of file diff --git a/content/codex/updates/2025-08-25.md b/content/codex/updates/2025-08-25.md index 811a6b16c3..ac758acb70 100644 --- a/content/codex/updates/2025-08-25.md +++ b/content/codex/updates/2025-08-25.md @@ -3,22 +3,31 @@ title: "2025-08-25 Codex weekly" tags: - "codex-updates" date: 2025-08-25 -lastmod: 2025-09-01 +lastmod: 2025-08-25 draft: false --- -## Week 35 August 2025 +## Week 34 August 2025 -After the discussion on the $26^{\text{th}}$, we have decided to follow through with [team NL-BR's approach](https://hackmd.io/@codex-storage/BkCvWMVtlx) and integrate Codex as a simple filesharing service. We will replace the use of torrent files with Codex CIDs, and at first roll out one archive per CID, with a simple bundling optimization in sight if that proves to be a problem. +This week was a dedicated “Groupathon” where the team split into 2 working groups and tackled: -We have scoped out and assigned most of the work for the integration PoC: -* improvements to NAT traversal: https://github.com/codex-storage/nim-codex/issues/904; -* a C-level API (with eyes on logos-core) that can be shimmed into Go bindings (which status requires): https://github.com/codex-storage/nim-codex/issues/1220; -* a wrap-up of the optimizations to the block protocol: https://github.com/codex-storage/nim-codex/issues/974; -* some back-of-the-envelope calculations for DHT overhead usage under load [here](https://hackmd.io/@codex-storage/rkIWizSvee) and [here](https://hackmd.io/ihMuAnWORI6yPChbwcRsZw). +1. the fastest way to replace Bittorrent with Codex for archival storage in status-go; +2. what the future of Codex might look like. -We are also carrying research and discussions on: -* what "low durability" mode should look like, and how it should fit within the broader Codex roadmap; -* what options do we have for scaling lookups for small content; -* what options do we have for privacy-preserving protocols in storage networks. +Problem statement, design goals and agenda can be found here: https://hackmd.io/JMuXL9AOSM6228KsgcTfXA +### Outcomes + +**Team Eurasia Submission** +[Presentation recording part 1](https://drive.google.com/file/d/19ntoDboaSEhBztkC5RC7r7Apy1U-Hthe/view) +[Presentation recording part 2](https://drive.google.com/file/d/1Sp-Rd0RhRtecmi4yHqSBI_4t4ehbh6CZ/view) +[Presentation slides](https://docs.google.com/presentation/d/1w1TdnTyJik2Ll31xQloR6V-6YyzeY_CswYQgDCQaABA/edit?slide=id.g158930d3f51_0_105#slide=id.g158930d3f51_0_105) + +**Team NL-BR** +[Team NL-BR Presentation Recording](https://drive.proton.me/urls/0D3VX0Q8VM#EWf9Kb5QPbXo) +[Team NL-BR Presentation](https://link.excalidraw.com/readonly/WENb1UXb9n2RYdvCKguP) +[Team NL-BR main document](https://hackmd.io/@codex-storage/BkCvWMVtlx) + +Balazs proposal for longer term design: https://hackmd.io/6OGb1OKiRD-w6qxP-yDcJQ?view#A-possible-incremental-rollout-plan + +**Tuesday 26th - Discussion and decision on a new design!** \ No newline at end of file diff --git a/content/codex/updates/2025-09-01.md b/content/codex/updates/2025-09-01.md index 8d6400c715..8ebeb8c82e 100644 --- a/content/codex/updates/2025-09-01.md +++ b/content/codex/updates/2025-09-01.md @@ -7,100 +7,18 @@ lastmod: 2025-09-01 draft: false --- -## Week 36 August 2025 +## Week 35 August 2025 -### [Replace bittorrent in status-go](https://github.com/codex-storage/nim-codex/milestone/8) +After the discussion on the $26^{\text{th}}$, we have decided to follow through with [team NL-BR's approach](https://hackmd.io/@codex-storage/BkCvWMVtlx) and integrate Codex as a simple filesharing service. We will replace the use of torrent files with Codex CIDs, and at first roll out one archive per CID, with a simple bundling optimization in sight if that proves to be a problem. -1. [Enable hole punching with circuit relay - v2](https://github.com/codex-storage/nim-codex/issues/904) - - **achieved**: - - investigated existing approaches to circuit relay and holepunching, - including Waku’s implementation - - **next**: - - create a small example with codex nodes communicated over circuit relay - - **blockers**: +We have scoped out and assigned most of the work for the integration PoC: +* improvements to NAT traversal: https://github.com/codex-storage/nim-codex/issues/904; +* a C-level API (with eyes on logos-core) that can be shimmed into Go bindings (which status requires): https://github.com/codex-storage/nim-codex/issues/1220; +* a wrap-up of the optimizations to the block protocol: https://github.com/codex-storage/nim-codex/issues/974; +* some back-of-the-envelope calculations for DHT overhead usage under load [here](https://hackmd.io/@codex-storage/rkIWizSvee) and [here](https://hackmd.io/ihMuAnWORI6yPChbwcRsZw). -2. [BlockExchange Performance and - reliability](https://github.com/codex-storage/nim-codex/issues/974) - - **achieved**: - - reviewed issue and requirements - - identified key areas to investigate (timeouts, retries, concurrency - handling) - - **next**: - - read through all existing work and documentation related to BlockExchange - - analyze current source code implementation - - **blockers**: +We are also carrying research and discussions on: +* what "low durability" mode should look like, and how it should fit within the broader Codex roadmap; +* what options do we have for scaling lookups for small content; +* what options do we have for privacy-preserving protocols in storage networks. -3. [Identify Swarm Participation Limits (DHT - Scaling)](https://github.com/codex-storage/codex-research/issues/204) - - **achieved:** - - Drafted an initial approach for estimating the performance of core DHT - operations. [Reference](https://hackmd.io/@codex-storage/HkGRPTTYlg). - - Reviewed the `codex-dht` implementation and identified key parameters - (e.g., message sizes, maintenance intervals, etc.) required for - performance calculations. - - Calculated the bandwidth requirements for routing table operations - - **next:** - - Calculate bandwidth requirements for core operations across different - network sizes, data sets, and churn rates. - - Explore additional scaling options and possible optimizations. - - **blockers:** - -4. [Implement a Codex library so that clients can use Codex directly from their - apps](https://github.com/codex-storage/nim-codex/issues/1220) - - **achieved:** - - Generated a C binding from a minimal Nim bare project - - Consumed the C binding in a minimal Go project - - Analyzed the nim-library-template and its integration with Chronos - - Opened a [PR](https://github.com/logos-co/nim-library-template/pull/6) to - fix issues in the template - - Started creating a Go binding to start a Codex node - - **next:** - - Clean up the Go binding foundations and request a review - - Begin exposing REST API features, starting with debug - - **blockers:** - -5. [Integration with - status-go](https://github.com/codex-storage/nim-codex/issues/1304) - - **achieved** - - gained some insights into the relation between status-go, status-backend, - and status-desktop to understand their corresponding dev environments - (especially testing) and how they map to our effort of replacing - bittorrent with codex (which bits and pieces do we need to get the work - done) - - Status desktop / status-go dev environment setup: - - built status-go and status-desktop from sources (on Arch Linux) - - **next:** - - Status desktop / status-go dev environment setup: - - checking building on macOS, so there are at least three different - clients on different machines for manual testing - - running functional tests (docker) for status-go - - light documenting of the findings on the dev-env relevant to the - codex-status-go integration - - initial design/code for the prototype - -### [Research future work](https://github.com/codex-storage/nim-codex/milestone/9) - -1. [DST Team sync](https://github.com/codex-storage/nim-codex/issues/1313) - - achieved: - - discussions - (https://www.notion.so/Deployment-Workflow-25d8f96fb65c80f7801ce9a06dede8a4) - - next: - - online design session, reviewing codebases - -2. **Altruistic Mode** - - achieved: - - **Document:** [The Assumptions of Durability under - Altruism](https://hackmd.io/hC4RAzJ1TBiyuek8nqAP_A), an attempt to - clarify the differences and similarities for system dynamics and storage - guarantees under altruism. - - **Document:** [Considerations on "Low - Durability"](https://hackmd.io/hC4RAzJ1TBiyuek8nqAP_A), an attempt to - understand what the "low durability" mode proposed during our working - groupathon actually achieves. - - - next: - - exploration of: - - recent studies on (e.g. IPFS) network dynamics; - - existing reputation systems; - - tradeoffs of other storage systems (e.g Swarm). diff --git a/content/codex/updates/2025-09-08.md b/content/codex/updates/2025-09-08.md index c9f4070bc9..0bc5a87c19 100644 --- a/content/codex/updates/2025-09-08.md +++ b/content/codex/updates/2025-09-08.md @@ -1,147 +1,106 @@ --- -title: "2025-09-15 Codex weekly" +title: "2025-09-08 Codex weekly" tags: - "codex-updates" date: 2025-09-08 -lastmod: 2025-09-15 +lastmod: 2025-09-08 draft: false --- -## Week 37 August 2025 - -### FURPs for replacing Bittorrent in `status-go` - -[FURPs -requirements](https://github.com/codex-storage/codex-pm/blob/master/requirements/bittorrent_statusgo_prototype.md) -have been published for the prototype of Codex replacing Bittorrent in status-go -for the Status Community History service. +## Week 36 August 2025 ### [Replace bittorrent in status-go](https://github.com/codex-storage/nim-codex/milestone/8) 1. [Enable hole punching with circuit relay v2](https://github.com/codex-storage/nim-codex/issues/904) - **achieved**: - - Investigated possible solutions for [dynamic multicodec/multihash - formats in - nim-libp2p](https://github.com/vacp2p/nim-libp2p/issues/1682) - - Investigated (default) limits for circuit-relays. Also in other - projects utilizing libp2p (IPFS). - - Stumbled into the fact that we run on branched-off version of - `nim-libp2p`. Explored possible solutions and decided to simply stay on - the branch until either "dynamic multicodec/multihash formats" - ([#904](https://github.com/codex-storage/nim-codex/issues/904)) is - implemented or we sunset the Poseidon2 hash function that is the reason - for the branch off. Till then we plan to bring updates from `master` to - the branch when needed. + - investigated existing approaches to circuit relay and holepunching, + including Waku’s implementation - **next**: - - Continue with initial basic implementation of the libp2p NAT traversal - components - [#1319](https://github.com/codex-storage/nim-codex/issues/1319) + - create a small example with codex nodes communicated over circuit relay - **blockers**: 2. [BlockExchange Performance and reliability](https://github.com/codex-storage/nim-codex/issues/974) - **achieved**: - - Analyzed BlockExchange architecture and identified performance - bottlenecks in streaming downloads - - Investigated bulk prefetch interference with individual block requests - through systematic testing - - Investigated peer coordination timing issues causing duplicate block - requests + - reviewed issue and requirements + - identified key areas to investigate (timeouts, retries, concurrency + handling) - **next**: - - Continue systematic investigation of peer discovery and block presence - coordination - - Research approaches to optimize individual block request latency + - read through all existing work and documentation related to BlockExchange + - analyze current source code implementation - **blockers**: 3. [Identify Swarm Participation Limits (DHT Scaling)](https://github.com/codex-storage/codex-research/issues/204) - **achieved:** - - Researched provider record liveliness and its affect in the - re-advertise rate in our use-case. - - Researched provider churn in peer-to-peer networks since it plays a - significant role in the query rate in our setting. - - Defined expected ranges for the core variables used in calculating - per-provider bandwidth based on prior research. - - Calculate bandwidth requirements for core operations across different - network sizes, data sets, and churn rates. - - Produced figures and tables summarizing these results. - - Finalised the research document in a - [PR](https://github.com/codex-storage/codex-research/pull/205). + - Drafted an initial approach for estimating the performance of core DHT + operations. [Reference](https://hackmd.io/@codex-storage/HkGRPTTYlg). + - Reviewed the `codex-dht` implementation and identified key parameters + (e.g., message sizes, maintenance intervals, etc.) required for + performance calculations. + - Calculated the bandwidth requirements for routing table operations - **next:** - - Review and refine the research document. - - Explore additional scaling options and possible optimizations. + - Calculate bandwidth requirements for core operations across different + network sizes, data sets, and churn rates. + - Explore additional scaling options and possible optimizations. - **blockers:** 4. [Implement a Codex library so that clients can use Codex directly from their apps](https://github.com/codex-storage/nim-codex/issues/1220) - **achieved:** - * First Go wrapper that starts and stops a Codex node. - * Added Codex version and revision API. - * Provided a better API with ACK and OK differentiation. - * Fixed an async issue from the Nim library template. + - Generated a C binding from a minimal Nim bare project + - Consumed the C binding in a minimal Go project + - Analyzed the nim-library-template and its integration with Chronos + - Opened a [PR](https://github.com/logos-co/nim-library-template/pull/6) to + fix issues in the template + - Started creating a Go binding to start a Codex node - **next:** - - Add more Codex API features to the wrapper (debug, spr, etc.). + - Clean up the Go binding foundations and request a review + - Begin exposing REST API features, starting with debug - **blockers:** 5. [Integration with status-go](https://github.com/codex-storage/nim-codex/issues/1304) - **achieved** - - Got better grip on the testing environment - - managed to run unit tests which is where the core history archive - functionality is tested, learning GO testing tooling, - - functional tests - more complex and less stable. Now under control. - Possibly not really crucial for the work to be done at this stage, - but it is good to know the system is behaving predictability. Made - some fine-grained scripts to run selected tests, all tests without - coverage, building status-go docker image, and more robust cleanup, - - improved reliability of functional test - selective running, - waiting for docker services to be healthy before running the tests - in order to avoid misleading re-runs and difficulties in cleanup, - - got some basics of nix and flake environment which is used by - status-go in CI. Decided not to use them during local development - though (adds lots of complexity, and feels like it makes the - machine dirty). - - documented how to setup dev/test environment (still wrapping up - some small bits, when ready the docs will be added via - [codex-docs-obsidian/pull/19](https://github.com/codex-storage/codex-docs-obsidian/pull/19) - and the relevant docs will be exposed via HackMd, see e.g.: - [Running functional tests in - development](https://github.com/codex-storage/codex-docs-obsidian/blob/4951eb792fa7920d7a6840be686701f152b36102/10%20Notes/Running%20functional%20tests%20in%20development.md), - - Draft PR to for the status-go-codex integration created (now only - containing the above mentioned testing scripts modifications): - [status-go/pull/6930](https://github.com/status-im/status-go/pull/6930) + - gained some insights into the relation between status-go, status-backend, + and status-desktop to understand their corresponding dev environments + (especially testing) and how they map to our effort of replacing + bittorrent with codex (which bits and pieces do we need to get the work + done) + - Status desktop / status-go dev environment setup: + - built status-go and status-desktop from sources (on Arch Linux) - **next:** - - working on the actual integration of Codex with status-go (using Codex - API calls) - - **blockers:** + - Status desktop / status-go dev environment setup: + - checking building on macOS, so there are at least three different + clients on different machines for manual testing + - running functional tests (docker) for status-go + - light documenting of the findings on the dev-env relevant to the + codex-status-go integration + - initial design/code for the prototype ### [Research future work](https://github.com/codex-storage/nim-codex/milestone/9) -1. **Altruistic Mode** - - **Document:** The first version of our [simple model for DHT - traffic](https://hackmd.io/ihMuAnWORI6yPChbwcRsZw) is complete. This - should allow us to keep discussions on whether or not the "DHT will scale" - to support a proposed mechanism more grounded. - - **Document:** Completed churn section for [Assumptions for Durability - under - Altruism](https://codex-storage.github.io/codex-docs-obsidian/10-Notes/The-Assumptions-for-Durability-under-Altruism). - This tries to convey the idea that durability under altruism is nuanced, - and put together the factors that might affect it. - - **Document:** [Internal - critique](https://hackmd.io/XKHNrjnHSRqqXc7Bd0RfQA) for why "Low - Durability" is a bad name for altruistic Codex, and why it is also a - symptom of a lack of proper problem understanding. - - **Document:** [Running list of known problems for altruistic - Codex](https://hackmd.io/xzxmkqgKQ6mq55NDL449yA); we will be populating - this every week. - -### Misc +1. [DST Team sync](https://github.com/codex-storage/nim-codex/issues/1313) + - achieved: + - discussions + (https://www.notion.so/Deployment-Workflow-25d8f96fb65c80f7801ce9a06dede8a4) + - next: + - online design session, reviewing codebases -1. **Testnet Decommission** +2. **Altruistic Mode** - achieved: - - exported testnet discord bot logs - - took testnet discord bot offline + - **Document:** [The Assumptions of Durability under + Altruism](https://hackmd.io/hC4RAzJ1TBiyuek8nqAP_A), an attempt to + clarify the differences and similarities for system dynamics and storage + guarantees under altruism. + - **Document:** [Considerations on "Low + Durability"](https://hackmd.io/hC4RAzJ1TBiyuek8nqAP_A), an attempt to + understand what the "low durability" mode proposed during our working + groupathon actually achieves. + - next: - - export record of testnet blockchain - - take testnet blockchain offline \ No newline at end of file + - exploration of: + - recent studies on (e.g. IPFS) network dynamics; + - existing reputation systems; + - tradeoffs of other storage systems (e.g Swarm). diff --git a/content/codex/updates/2025-09-15.md b/content/codex/updates/2025-09-15.md index 67f32a83b3..359e1deafb 100644 --- a/content/codex/updates/2025-09-15.md +++ b/content/codex/updates/2025-09-15.md @@ -1,72 +1,147 @@ --- -title: "2025-09-22 Codex weekly" +title: "2025-09-15 Codex weekly" tags: - "codex-updates" date: 2025-09-15 -lastmod: 2025-09-22 +lastmod: 2025-09-15 draft: false --- -## Week 38 September 2025 +## Week 37 August 2025 + +### FURPs for replacing Bittorrent in `status-go` + +[FURPs +requirements](https://github.com/codex-storage/codex-pm/blob/master/requirements/bittorrent_statusgo_prototype.md) +have been published for the prototype of Codex replacing Bittorrent in status-go +for the Status Community History service. ### [Replace bittorrent in status-go](https://github.com/codex-storage/nim-codex/milestone/8) -1. [BlockExchange Performance and reliability](https://github.com/codex-storage/nim-codex/issues/974) +1. [Enable hole punching with circuit relay + v2](https://github.com/codex-storage/nim-codex/issues/904) + - **achieved**: + - Investigated possible solutions for [dynamic multicodec/multihash + formats in + nim-libp2p](https://github.com/vacp2p/nim-libp2p/issues/1682) + - Investigated (default) limits for circuit-relays. Also in other + projects utilizing libp2p (IPFS). + - Stumbled into the fact that we run on branched-off version of + `nim-libp2p`. Explored possible solutions and decided to simply stay on + the branch until either "dynamic multicodec/multihash formats" + ([#904](https://github.com/codex-storage/nim-codex/issues/904)) is + implemented or we sunset the Poseidon2 hash function that is the reason + for the branch off. Till then we plan to bring updates from `master` to + the branch when needed. + - **next**: + - Continue with initial basic implementation of the libp2p NAT traversal + components + [#1319](https://github.com/codex-storage/nim-codex/issues/1319) + - **blockers**: + +2. [BlockExchange Performance and + reliability](https://github.com/codex-storage/nim-codex/issues/974) - **achieved**: - - Enhanced peer selection: Implemented multi-factor scoring algorithm considering peer load, exchange recency, success rate, and response time - - Improved load balancing: Distributed block requests across multiple peers in blockPresenceHandler instead of sending all to announcing peer - - Discovery engine optimization: Added peer count limits to prevent unbounded accumulation while maintaining redundancy, with least-active peer removal - - Memory optimization: Reduced default fetch batch size, implemented chunked onBatch processing to prevent memory accumulation, and added buffer reuse to reduce GC allocation churn + - Analyzed BlockExchange architecture and identified performance + bottlenecks in streaming downloads + - Investigated bulk prefetch interference with individual block requests + through systematic testing + - Investigated peer coordination timing issues causing duplicate block + requests - **next**: - - Continue memory consumption optimizations in block exchange pipeline - - Investigate memory usage patterns in block batching and peer context management - - Profile under concurrent load to identify presure points + - Continue systematic investigation of peer discovery and block presence + coordination + - Research approaches to optimize individual block request latency - **blockers**: -2. [Identify Swarm Participation Limits (DHT Scaling)](https://github.com/codex-storage/codex-research/issues/204) +3. [Identify Swarm Participation Limits (DHT + Scaling)](https://github.com/codex-storage/codex-research/issues/204) - **achieved:** - - DHT scaling research [document](https://hackmd.io/@codex-storage/HkGRPTTYlg) reviewed. + - Researched provider record liveliness and its affect in the + re-advertise rate in our use-case. + - Researched provider churn in peer-to-peer networks since it plays a + significant role in the query rate in our setting. + - Defined expected ranges for the core variables used in calculating + per-provider bandwidth based on prior research. + - Calculate bandwidth requirements for core operations across different + network sizes, data sets, and churn rates. + - Produced figures and tables summarizing these results. + - Finalised the research document in a + [PR](https://github.com/codex-storage/codex-research/pull/205). - **next:** - - Refine the document and address the review comments. - - Research existing DHT simulations and compare results. + - Review and refine the research document. + - Explore additional scaling options and possible optimizations. - **blockers:** -3. [Implement a Codex library so that clients can use Codex directly from their apps](https://github.com/codex-storage/nim-codex/issues/1220) +4. [Implement a Codex library so that clients can use Codex directly from their + apps](https://github.com/codex-storage/nim-codex/issues/1220) - **achieved:** - - Exposed core Codex APIs: version, revision, repo, peerId, spr, debug information. - - Added ability to change log level. - - Implemented peer connection support. - - Exposed the upload feature with two ways: - - Chunks: 4-step flow (init, upload chunk, complete, cancel) with pause/resume support. - - Filepath: 3-step flow (init, upload, cancel). + * First Go wrapper that starts and stops a Codex node. + * Added Codex version and revision API. + * Provided a better API with ACK and OK differentiation. + * Fixed an async issue from the Nim library template. - **next:** - - Extend Go wrapper API to allow cancellation of ongoing uploads. - - Implement download feature. + - Add more Codex API features to the wrapper (debug, spr, etc.). - **blockers:** -4. [Integration with status-go](https://github.com/codex-storage/nim-codex/issues/1304) +5. [Integration with + status-go](https://github.com/codex-storage/nim-codex/issues/1304) - **achieved** - - documented running running unit tests for status-go: [codex-docs-obsidian/pull/19](https://github.com/codex-storage/codex-docs-obsidian/pull/19) (direct link to the document: [Running Unit Tests for status-go](https://github.com/codex-storage/codex-docs-obsidian/blob/spec/codex-status-go-prototype/10%20Notes/Running%20Unit%20Tests%20for%20status-go.md)), - - added a small convenience script to start a docker containiner running a waku node that is suitable for some of the unit tests: [status-go/pull/6930](https://github.com/status-im/status-go/pull/6930) (`_assets/scripts/run_waku.sh in`), - - coding the actual integration + - Got better grip on the testing environment + - managed to run unit tests which is where the core history archive + functionality is tested, learning GO testing tooling, + - functional tests - more complex and less stable. Now under control. + Possibly not really crucial for the work to be done at this stage, + but it is good to know the system is behaving predictability. Made + some fine-grained scripts to run selected tests, all tests without + coverage, building status-go docker image, and more robust cleanup, + - improved reliability of functional test - selective running, + waiting for docker services to be healthy before running the tests + in order to avoid misleading re-runs and difficulties in cleanup, + - got some basics of nix and flake environment which is used by + status-go in CI. Decided not to use them during local development + though (adds lots of complexity, and feels like it makes the + machine dirty). + - documented how to setup dev/test environment (still wrapping up + some small bits, when ready the docs will be added via + [codex-docs-obsidian/pull/19](https://github.com/codex-storage/codex-docs-obsidian/pull/19) + and the relevant docs will be exposed via HackMd, see e.g.: + [Running functional tests in + development](https://github.com/codex-storage/codex-docs-obsidian/blob/4951eb792fa7920d7a6840be686701f152b36102/10%20Notes/Running%20functional%20tests%20in%20development.md), + - Draft PR to for the status-go-codex integration created (now only + containing the above mentioned testing scripts modifications): + [status-go/pull/6930](https://github.com/status-im/status-go/pull/6930) - **next:** - - have basic POC finished (without fixing all the relevant tests, but having things in place to confirm that POC is working) + - working on the actual integration of Codex with status-go (using Codex + API calls) - **blockers:** ### [Research future work](https://github.com/codex-storage/nim-codex/milestone/9) 1. **Altruistic Mode** - - - **Document:** [Proposal for outsourcing local erasure coding](https://hackmd.io/SazdX1SqSs6hNjtc-3B2BQ) (especially interesting together with file bundling). Next: prototype implementation - - **Document:** Updates to our [collection of the research questions/problems](https://hackmd.io/@codex-storage/rJeZeJC9gg) related to the altruistic mode. - - **Document:** Consolidated understanding of [Swarm networking](https://hackmd.io/sHpB5N39QVqLSJV-k_-jLQ) and ongoing work with [Swarm Incentives](https://hackmd.io/aSkvCy-LT8GjjdOzTrrLfw). - - Ongoing work in (re-)analysing Arweave, Sia, and Walrus based both on previous research done in the group (students and departed members), and more recent publications as we rebuild our knowledge on related approaches. - + - **Document:** The first version of our [simple model for DHT + traffic](https://hackmd.io/ihMuAnWORI6yPChbwcRsZw) is complete. This + should allow us to keep discussions on whether or not the "DHT will scale" + to support a proposed mechanism more grounded. + - **Document:** Completed churn section for [Assumptions for Durability + under + Altruism](https://codex-storage.github.io/codex-docs-obsidian/10-Notes/The-Assumptions-for-Durability-under-Altruism). + This tries to convey the idea that durability under altruism is nuanced, + and put together the factors that might affect it. + - **Document:** [Internal + critique](https://hackmd.io/XKHNrjnHSRqqXc7Bd0RfQA) for why "Low + Durability" is a bad name for altruistic Codex, and why it is also a + symptom of a lack of proper problem understanding. + - **Document:** [Running list of known problems for altruistic + Codex](https://hackmd.io/xzxmkqgKQ6mq55NDL449yA); we will be populating + this every week. ### Misc 1. **Testnet Decommission** - achieved: - - Merged docs update: [90](https://github.com/codex-storage/codex-docs/pull/90) + - exported testnet discord bot logs + - took testnet discord bot offline - next: - - Example of docs page with support for multiple versions: [WIP](https://github.com/codex-storage/codex-docs/pull/91) + - export record of testnet blockchain + - take testnet blockchain offline \ No newline at end of file diff --git a/content/codex/updates/2025-09-22.md b/content/codex/updates/2025-09-22.md new file mode 100644 index 0000000000..f1e596697f --- /dev/null +++ b/content/codex/updates/2025-09-22.md @@ -0,0 +1,72 @@ +--- +title: "2025-09-22 Codex weekly" +tags: + - "codex-updates" +date: 2025-09-22 +lastmod: 2025-09-22 +draft: false +--- + +## Week 38 September 2025 + +### [Replace bittorrent in status-go](https://github.com/codex-storage/nim-codex/milestone/8) + +1. [BlockExchange Performance and reliability](https://github.com/codex-storage/nim-codex/issues/974) + - **achieved**: + - Enhanced peer selection: Implemented multi-factor scoring algorithm considering peer load, exchange recency, success rate, and response time + - Improved load balancing: Distributed block requests across multiple peers in blockPresenceHandler instead of sending all to announcing peer + - Discovery engine optimization: Added peer count limits to prevent unbounded accumulation while maintaining redundancy, with least-active peer removal + - Memory optimization: Reduced default fetch batch size, implemented chunked onBatch processing to prevent memory accumulation, and added buffer reuse to reduce GC allocation churn + - **next**: + - Continue memory consumption optimizations in block exchange pipeline + - Investigate memory usage patterns in block batching and peer context management + - Profile under concurrent load to identify presure points + - **blockers**: + +2. [Identify Swarm Participation Limits (DHT Scaling)](https://github.com/codex-storage/codex-research/issues/204) + - **achieved:** + - DHT scaling research [document](https://hackmd.io/@codex-storage/HkGRPTTYlg) reviewed. + - **next:** + - Refine the document and address the review comments. + - Research existing DHT simulations and compare results. + - **blockers:** + +3. [Implement a Codex library so that clients can use Codex directly from their apps](https://github.com/codex-storage/nim-codex/issues/1220) + - **achieved:** + - Exposed core Codex APIs: version, revision, repo, peerId, spr, debug information. + - Added ability to change log level. + - Implemented peer connection support. + - Exposed the upload feature with two ways: + - Chunks: 4-step flow (init, upload chunk, complete, cancel) with pause/resume support. + - Filepath: 3-step flow (init, upload, cancel). + - **next:** + - Extend Go wrapper API to allow cancellation of ongoing uploads. + - Implement download feature. + - **blockers:** + +4. [Integration with status-go](https://github.com/codex-storage/nim-codex/issues/1304) + - **achieved** + - documented running running unit tests for status-go: [codex-docs-obsidian/pull/19](https://github.com/codex-storage/codex-docs-obsidian/pull/19) (direct link to the document: [Running Unit Tests for status-go](https://github.com/codex-storage/codex-docs-obsidian/blob/spec/codex-status-go-prototype/10%20Notes/Running%20Unit%20Tests%20for%20status-go.md)), + - added a small convenience script to start a docker containiner running a waku node that is suitable for some of the unit tests: [status-go/pull/6930](https://github.com/status-im/status-go/pull/6930) (`_assets/scripts/run_waku.sh in`), + - coding the actual integration + - **next:** + - have basic POC finished (without fixing all the relevant tests, but having things in place to confirm that POC is working) + - **blockers:** + +### [Research future work](https://github.com/codex-storage/nim-codex/milestone/9) + +1. **Altruistic Mode** + + - **Document:** [Proposal for outsourcing local erasure coding](https://hackmd.io/SazdX1SqSs6hNjtc-3B2BQ) (especially interesting together with file bundling). Next: prototype implementation + - **Document:** Updates to our [collection of the research questions/problems](https://hackmd.io/@codex-storage/rJeZeJC9gg) related to the altruistic mode. + - **Document:** Consolidated understanding of [Swarm networking](https://hackmd.io/sHpB5N39QVqLSJV-k_-jLQ) and ongoing work with [Swarm Incentives](https://hackmd.io/aSkvCy-LT8GjjdOzTrrLfw). + - Ongoing work in (re-)analysing Arweave, Sia, and Walrus based both on previous research done in the group (students and departed members), and more recent publications as we rebuild our knowledge on related approaches. + + +### Misc + +1. **Testnet Decommission** + - achieved: + - Merged docs update: [90](https://github.com/codex-storage/codex-docs/pull/90) + - next: + - Example of docs page with support for multiple versions: [WIP](https://github.com/codex-storage/codex-docs/pull/91)