ArneBab
Arne Babenhauserheide
Freenet 0.7.5 build 1498 is now available. [overview]
This release resolves the last blocker for Freenet / Hyphanet 0.8 by
providing an official Debian package. Additionally it optimizes the
networking and data transfer core and provides many improvements for
website authors and user experience.
Starting with this release, Freenet / Hyphanet has an official Debian
package built automatically via github actions. This was the most
important high-impact-task and the last release blocker of version
0.8 in our Roadmap. Big thanks go to DC*!
With this finally realized, the next step is to get in contact with
the many privacy focussed distributions which build on Debian to make
hyphanet-fred available where it is most important. Once this is
done, tools which build on Hyphanet — like FMS, but also jSite and
tools from pyFreenet — can be packaged to work out of the box, using
Hyphanet as an ordinary background service. That’s a step towards
Hyphanet as decentralized, privacy-preserving communication backend for
other applications.
Another step towards this is accepting the Schema hypha[net] to
simplify writing browser extensions that forward hypha:-links to
Hyphanet.
The networking layer was optimized significantly. Searching packet
types is often stopped early and common or cheaper checks are done
before less common or time-consuming checks. This gives significant
reductions of CPU load, especially for very fast nodes.
Juiceman fixed a bug limiting MTU to 1280 where not needed.
And recently failed and data not found cooldown times were reduced to
5 minutes and 3 minutes, reducing one of the big annoyances when
accessing a site quickly after upload.
On the data transfer layer, healing was optimized. After 1495 strongly
increased the amount of healing to keep large files available for
longer, 1498 specializes healing to keys close to the node location.
This reduces healing per file, but improves privacy, because healing
inserts are then more similar to forwarding — they mostly send data
close to the nodes location — and it reduces the network load of
healing, because the specialized healing inserts need fewer hops to
reach the optimal storage location in the network.
In addition to these changes deep down, there are a number of directly
visible improvements.
The plugins KeepAlive and Sharesite are updated (the latter now uses
the new Night Zen Garden style). The UPnP2 plugin is now visible in
simple mode. It can replace UPnP and should work better. On the
flipside the Library plugin is moved to advanced plugins, because it
does not work reliably enough.
The plugin list is easier to navigate by removing the defunct option
to download plugins from the clearnet and by adding better styling.
Downloading from the clearnet was an unnecessary privacy risk since
we’ve been bundling essential plugins with the installer for a few
years now.
The noderef for friend-to-friend connections is shown in simple mode
again, because it is robust enough with the changes in recent years.
This should remove a barrier to adding direct connections and enabling
fully confidential messages between friends.
There are new configuration options to allow connecting via local
services. That’s a step towards making it easy to add a second layer
of security, for example confining connections to a local network.
Thanks goes to s7r for these changes!
When bandwidth detection fails, the upload bandwidth now defaults to
160KiB/s. Also the NLM config is now disabled statically. This was a
testing setup which could still be active in old nodes, but it would
break connectivity nowadays.
The default bookmarks include the Opennet SeedNodes statistics,
the generate media site to create decentralized streaming sites, and
the high-impact-tasks. The bookmarks are also re-ordered to be a
better match for newcomers. Starting category: first steps, clean
spider, Index of Indexes. For the software category ordered by ease of
use from fproxy.
For website authors, more CSS elements, selectors and combinators
(:checked, word-wrap: anywhere, focus-within, ^=, $=, *=,
>, +, ~) and additional HTML elements (summary, details,
<meta name="Viewport"...>) are available. This strongly expands the
possibilities of websites authors in Hyphanet, because Javascript or
webassembly are no viable options in an environment where a privacy
breach could put people at risk. We’ve seen with Java applets, that
untrusted code will always break out of its containment. The CSS
improvements in contrast provide a safe way to enable limited
interactivity.
Streaming support via m3u lists was improved to allow accessing
segments of up to 200MiB.
And using -1 as version in a USK now properly finds version 0, if
this is the only existing version.
There were a number of Java 21 fixes, including all our tests (thanks
to Bombe!), and improvement to the github actions (thanks to
AHOHNMYC).
In addition to that there was a lot of polish. Bert Massop and
Veniamin Fernandes replaced our homegrown CurrentTimeUTC with modern
Java options. Alex fixed the pronoun used in strings. Bombe added
getters for all direct field access in the node. Hiina reduced logging
level of store warnings so no unneeded backtraces are created for node
with large stores and Juiceman updated code to use more modern
structures.
Time-dependence of compressor selection was removed. This caused
non-determinism for inserts and could cause keys to be
non-reproducible on systems with faster or slower network.
And finally the new exe signing workflow we built to fulfill the
requirements of SignPath, our new windows installer signing provider
for the upcoming releases, runs the verify-build script on every
release to ensure that the jar we release has actually been built from
the sources. This provides a second safety net, in addition to
anonymous users running the script and posting the results (thanks to
all who did this — please keep it up, otherwise people have to fully
trust github). The release is not yet byte-by-byte reproducible,
because the jar MANIFEST defines among other info the exact java
version used to compile it, and the java version available differs by
distribution and time, so it would get harder over time to verify the
build.
A special thanks goes to Bombe for many careful reviews!
Thank you for using Freenet!
- AB
Developer changelog:
2024-04-28
Changes in 1498:
high impact tasks
merge debian package as default build action thanks to DC*/desyncr! This resolves one of our high impact tasks.
Plugins
Update KeepAlive to commit 86e47a101f26fd1d3be0437681a043aa4ae3f22c
Update Sharesite to 0.5.1
Move UPnP2 to normal plugins. It does not seem broken, but UPnP does
Move Library plugin to advanced plugins because new users tend to get lost with it
💄 Add better styling to the plugin list in winterfacey to make it easier to understand at a glance — thanks to Bombe
🔥 Remove option to load plugins from central server — thanks to Bombe! This was an unnecessary privacy risk, since we’re already bundling essential plugins with the installer, and it made plugin handling harder to understand.
Bookmarks
Add high-impact-tasks to bookmarks
Add generate media site to the default bookmarks
Add Opennet SeedNodes stats site
Reorder starting bookmarks: FFS → clean spider → Index of Indexes
Reorder default software bookmarks by ease of use from fproxy
Disable activelink for Index of Indexes (workaround, because it fails)
Optimize networking and transfer layer
break early when condition is met — thanks to Juiceman
Check the HashCode before equals. This saves ~20% method-runtime.
Re-order or’ed MessageFilters so the most likely is checked first
specialize healing to keys close to the node
fix healing decision: do not divide 0-1 by MAX_VALUE — thanks to Bombe for the review!
Reduce recently failed and data not found wait times
Filters
CSS: Fix: checked only the first char of the key part of CSS selectors, Add test that would catch too lax filtering.
CSS: Support pseudo-element checked. This enables limited interactivity via CSS.
CSS: Support the attribute selectors ^= $= *=, Add tests.
CSS: Support Combinators > + and ~, add test for ~ and simplify the implementation
CSS: Support word-wrap: anywhere and CSS selector focus-within.
HTML: allow summary and details html element. Thanks to naejadu
HTML: accept <meta name="Viewport" ...>, thanks to torusrxxx
Constants
Show the noderef in basic-mode: it is now robust enough
Configuration
accepting localhost in NodeIPPortDetector and allowBindToLocalhost configurable — thanks to s7r!
Provide static methods for simpler boolean config creation
Increase default bandwidth to 160KiB upload, when detection fails
disable setting for new-load-management (NLM broke nodes)
add utility to disable a config option, thanks to Bombe
Misc
add m3u-player insertion test: is added at end of body
[CI] Update actions, fix actions cache
♻️ add and use getters and setters for access to node fields
Increase max transparent passthrough to 200MiB links in m3u-lists.
Remove time-dependence of compressor selection. This caused
non-determinism for inserts and could cause keys to be
non-reproducible.
improve date object construction in CurrentTimeUTC.get()
Support Schema hypha[net] to simplify writing browser extensions that forward hypha://-links to Hyphanet.
polish: show datastore size warning with GiB suffix
Remove hash generation to native big integer to reduce dependencies. This had come in when merging an old pull request and added a new dependency without need.
Polish
Replace indexOf with .contains()
Change more string comparisons into .isEmpty() checks
Capitalize L in literal longs
Add missing Global flag to GetFailed FCP message. Thanks to oishii and Bombe!
Fix links to mailing lists and IRC in CONTRIBUTING.md — thanks to Juiceman!
Deprecated custom datetime handling class CurrentTimeUTC
replace GregorianCalendar with java.time.OffsetDateTime in CurrentTimeUTC — thanks to Veniamin Fernandes
Refine ClientRequestSelectorTest, PersistentJobRunnerImplTest, SplitFileFetcherStorageTest, and SplitFileInserterStorageTest — thanks to Veniamin Fernandes
Change deprecated jcenter() maven repo to mavenCentral() — thanks to Veniamin Fernandes
Added tests for PebbleUtils — thanks to Bombe!
🐛 Fix NPE when subsets are not initialized — thanks to Bombe
reduce logging for too many excluded sub-arrays thanks to Hiina
fix the flag size of nepal — thanks to Percept0r@NYZkOs7eQ…!
Switch swiss flag to civil and state ensign — thanks to Percept0r@NY
Fixes
Thanks to Bombe all our tests work again on Java 21!
🐛 Fix JarClassLoader’s ability to work with ServiceLoader — thanks to Bombe!
gzip: replace test workaround by fixing the output of the compressor — thanks to Bombe for the SingleOffsetReplacingOutputStream!
fix: invalid max store size showed bytes with GiB suffix
Also a special thanks to Bombe for many careful reviews!
- AB
AHOHNMYC (1):
[CI] Update actions, fix actions cache
Alex (1):
Changed pronoun
Arne Babenhauserheide (174):
refactor the browser warning wizard page
clean up auto-refactored method
clean up more
Update links in CONTRIBUTING.md and add a contact-section in README.md
fix regression: default security level is normal
fix: invalid max store size showed bytes with GiB suffix
polish: also show datastore size warning with GiB suffix
Sort peer-addresses to try by hostname -> ipv6 -> ipv4
Use Collections.addAll instead of manual set adding
Prefer IPv6 over IPv4 for handshake address
avoid possible Null Pointer Exception
Prefer IPv6 over IPv4 in the InetAddressComparator
prefer reachable addresses
compare equal peer-instances by their addresses
only sort if at least 2 addresses (performance)
add two more logMINOR paths
when using a USK with -1 as version, also find version 0.
cache reachability for 100s, not only for 10s
prefer reachable LAN addresses over reachable globally reachable addresses
increase cache time to 5 minutes
use the default max ping time for reachability
prefer everything over broadcast addresses
whitespace
fix whitespace
start test for USK -1 to 0, WIP
gradle: undo change to disable preserving file timestamps
Also build debian package on next
update news
whitespace
build debian package from master
build debian package from debian-pkg for quick experimentation
disable -Zgzip
mark debian version as 1497
include new deps in debian package
add new required wrapper arguments for Java 11 in debian package
switch debian java to 17 (from debian stable)
build debian package from debian-package
typo
make debian/rules work on guix
Add debian/changelog entry for 1497
manually update changelog again
show debian diff
debian 1497
sudo
stay false
Summary: first patch
Debian init: name 1497
extract static PebbleUtils from WebTemlateToadlet for easier reuse
Show the noderef in basic-mode: it is now robust enough
Add generate media site to the default bookmarks
Re-order default software bookmarks by ease of use from fproxy
Reorder starting bookmarks: FFS → clean spider → Index of Indexes
bookmarks: Add Opennet SeedNodes stats
Update debian/freenet.init from the output of running on debian
Version without -1 (try to get github runner working)
revert the version change
fix day of week in changelog
hack: Do a local commit to get the package to work
hack-1: Do a local commit to get the package to work
noninteractive debian
more hackery
hackmore: format the patch
output format-patch directly
actually output the latest commit (HEAD^)
output hack
don’t try to get the patch
Switch swiss flag to civil and state ensign — thanks to Percept0r@NY
Remove no longer used showNoderef parameter
Show the unencoded IP addresses before the noderef
Provide default Key Type for File Insert
Fix by reference comparison for review
build from the current branch instead of hardcoded debian-package
Also build the debian package on every pull-request
fix: event pull-request → pull_request
use head ref with fallback to ref if not avail
use HEAD
keep the debian package as asset
remove trailing empty line
copy package to relative path
Use logging instead of backtrace for IPv6-addresses when unsupported
fix the flag size of nepal — thanks to Percept0r@NYZkOs7eQ…!
html-filter: allow summary and details html element. Thanks to naejadu
Add test for article, details, wbr, and summary tags
Fix whitespace
Set the new load management option to ignored.
Move Library to advanced plugins because new users tend to get lost
Move UPnP2 to normal plugins. It does not seem broken, but UPnP does
Fix merge error.
Open module java.util.zip for tests
gzip: replace test workaround by fixing the output of the compressor
Support word-wrap: anywhere
fix merge error
add word-wrap test
Add fallback for non-ArrayBuckets so every path fixes the OS byte
Use the much nicer SingleOffsetReplacingOutputStream by Bombe — 💟
Shorten description of sharesite freesite for english
Support the Schemas hypha: hyphanet: with web+ and ext+
Add tests for schema in FreenetURI
Update sharesite plugin to 0.5.1
Update KeepAlive to commit 86e47a101f26fd1d3be0437681a043aa4ae3f22c
update NEWS
note
Update Debian package to Version 1498
remove special casing of freenet: keys in WelcomeToadlet, because FreenetURI supports that already
Add focus-within to NEWS
specialize healing to keys close to the node
remove unused imports
refactor: extract variable to avoid long long line
Add high-impact-tasks to bookmarks
Disable activelink for Index of Indexes (workaround, because it fails)
Update NEWS
Re-order or’ed MessageFilters so the most likely is checked first
Call the match function with all arguments directly.
Check the HashCode before equals. This saves ~20% method-runtime.
Add TODOs with notes for optimization and commented out logging code.
Increase default bandwidth to 160KiB upload, when detection fails
Harmonize New wizard with old wizard
Remove hash generation to native big integer to reduce dependencies.
Do not divide 0-1 by MAX_VALUE — thanks to Bombe for the review!
Improve legibility
Pass in Bucket instead of RandomAccessBucket
Extract HealingDecisionSupplier — thanks to Bombes review!
Pass precise supplies to the HealingDecision instead of the node.
Remove no longer needed imports
Add test for healing in Darknet mode
Add negative tests for the healing decision supplier
Increase max transparent passthrough to 200MiB links in m3u-lists.
Also remove the NativeBigInteger part of HashTest
Revert "Add TODOs with notes for optimization and commented out logging code."
FIx indentation (match surrounding code)
Extend comment for review: last must match most frequently
Add more tests for review by Bombe
Add test around zero for review by Bombe
Fix IntelliJ indentation (reindent + tabify)
Extract helper methods for review by Bombe
Support CSS Combinators > + and ~
Support CSS the attribute selectors ^= $= *=
Support CSS pseudo-element checked
Add CSS test for ~ and simplify the implementation
Fix: checked only the first char of the key part of CSS selectors
Add tests for ^= $= and *=
Whitespace
Add test that would catch too lax filtering
remove trailing empty line
Remove redundancy in CSS Filter
Remove stray ;
tabify CSSTokenizerFilter
Fix: do not tabify in string
Only link indexes via the index of indexes.
Fix links to mailing lists and IRC in CONTRIBUTING.md — thanks to Juiceman!
Remove time-dependence of compressor selection
Remove translation keys for removed config
Improve Log message for wrong minimum percentage
Add missing Global flag to GetFailed FCP message. Thanks to oishii!
Java 21 fix: int used as return type but integer required
allow 127.0.0.1 in bindTo
Make allowBindToLocalhost configurable
Provide static methods for simpler boolean config creation
Use simpler static config creation
Fix static boolean callback creation method, only used specialized
Add test for static method BooleanCallback.from().
Add description of allowBindToLocalhost — thanks to s7r!
also make accepting localhost in NodeIPPortDetector configurable
Reduce recently failed and data not found wait times
Remove unused import
make instance variable private
Cleanup whitespace
Add getter test.
Shorten description
Remove duplication of value in documentation
add m3u-player insertion test: is added at end of body
Fix merge error
whitespace change reversal
check that m3u-filter-tag starts with <script and test in own method
Arne Babenhauserheide (freenet releases) (4):
Update default bookmark editions
Update default bookmark editions
Update default bookmark editions
Build 1498
Bert Massop (2):
Deprecate freenet.support.CurrentTimeUTC
Remove all Fred usages of deprecated CurrentTimeUTC
DC* (30):
Debian package continuation
Delete freenet.ini
Add functions to retrieve seednodes
Remove seednodes.fref
Update clean target
Lintian warning and error fix ups
Add missing dependencies for debian bookworm
Add github action
Build binary package only, add build dependencies
Use gbp build package
apt update
Custom action and install deps
Install build script dependencies
Missing sudo for apt-get on GH actions
Missing equivs dependency
Missing privileges in GH actions
Remove freenet deps .deb package
Cache steps and gradle configuration
Remove gradle.properties generation
Streamline clean up task
Update changelog
Update maintainer name
Update maintainer name
Add attribution to original work
Update package build descriptions
Update github action to test package generation only on tags and master
Remove unnecessary file entries
Lintian overrides
Build on debian-pkg branch
Update changelog
David ‘Bombe’ Roden (110):
🐛 Fix JarClassLoader’s ability to work with ServiceLoader
⬆️ Update Hamcrest dependency to 2.2
🐛 Fix NPE when subsets are not initialized
✅ Add test for L10nExtension
✅ Add test for PebbleUtils
✨ Allow registering of ignored options
🔥 Remove option to load plugins over HTTP(S) from web interface
🔥 Remove translation texts for removed options
🔥 Remove option that would always load plugins from central server
🔥 Remove code that has been obsoleted by removal of HTTP(S) downloads for official plugins
💄 Add a bit of styling to the plugin list (for winterfacey)
🔥 Remove http(s) download from user alert
🗑️ Add deprecated methods for legacy options
✨ Add output stream to fix other output streams
✅ Add more tests for writing buffers with offset != 0
✅ Add test for Global flag in GetFailed message
♻️ Add FCPConnectionHandler.getServer()
♻️ Add FCPConnectionHandler.getSocket()
♻️ Add FCPConnectionHandler.getOutputHandler()
♻️ Add FCPServer.getCore()
♻️ Add FCPServer.getNode()
♻️ Add FCPServer.isEnabled()
♻️ Add FCPServer.getGlobalRebootClient()
♻️ Use FCPServer.getGlobalForeverClient()
♻️ Add NodeClientCore.getBandwidthStatsPutter()
♻️ Add NodeClientCore.getUskManager()
♻️ Add NodeClientCore.getRequestStarters()
♻️ Add NodeClientCore.getFormPassword()
♻️ Add NodeClientCore.getTempFilenameGenerator()
♻️ Add NodeClientCore.getPersistentFilenameGenerator()
♻️ Add NodeClientCore.getTempBucketFactory()
♻️ Add NodeClientCore.getPersistentTempBucketFactory()
♻️ Add NodeClientCore.getClientLayerPersister()
♻️ Add NodeClientCore.getNode()
♻️ Add NodeClientCore.getNodeStats()
♻️ Add NodeClientCore.getRandom()
♻️ Add NodeClientCore.getAlerts()
♻️ Use NodeClientCore.getDirectTMCI()
♻️ Add NodeClientCore.getStoreChecker()
♻️ Add NodeClientCore.getClientContext()
♻️ Add Node.getNodeStats()
♻️ Add Node.getConfig()
♻️ Add Node.getGetPubKey()
♻️ Add Node.getIpDetector()
♻️ Add Node.isDisableProbabilisticHTLs()
♻️ Add Node.getTracker()
♻️ Use Node.getLocationManager()
♻️ Add Node.getPeers()
♻️ Use Node.getNodeDir()
♻️ Add Node.getRandom()
♻️ Add Node.getSecureRandom()
♻️ Add Node.getFastWeakRandom()
♻️ Use Node.getUSM()
♻️ Add Node.getDarknetCrypto()
♻️ Use Node.getOpennet()
♻️ Add Node.getExecutor()
♻️ Add Node.getPacketSender()
♻️ Use Node.getTicker()
♻️ Add Node.getDNSRequester()
♻️ Add Node.getDispatcher()
♻️ Add Node.getUptimeEstimator()
♻️ Add Node.getOutputThrottle()
♻️ Add Node.isThrottleLocalData()
♻️ Add Node.isEnableARKs()
♻️ Add Node.isEnablePerNodeFailureTables()
♻️ Add Node.isEnableULPRDataPropagation()
♻️ Add Node.isEnableSwapping()
♻️ Add Node.isEnableSwapQueueing()
♻️ Add Node.isEnablePacketCoalescing()
♻️ Add Node.getCollector()
♻️ Add Node.getClientCore()
♻️ Add Node.getFailureTable()
♻️ Add Node.getLastVersion()
♻️ Add Node.getNodeUpdater()
♻️ Add Node.getSecurityLevels()
♻️ Use Node.getPluginManager()
♻️ Add Node.getFreenetLocalhostAddress()
♻️ Add Node.getArkFetcherContext()
♻️ Add Node.getLastBootId()
♻️ Add Node.getBootId()
♻️ Add Node.getStartupTime()
♻️ Add Node.getNonPersistentClientBulk()
♻️ Add Node.getNonPersistentClientRT()
♻️ Add Node.getOldPK()
♻️ Add Node.getOldPKCache()
♻️ Add Node.getOldPKClientCache()
♻️ Add OpennetManager.getNode()
♻️ Add OpennetManager.getCrypto()
♻️ Add OpennetManager.getAnnouncer()
♻️ Add OpennetManager.getSeedTracker()
♻️ Add NodeUpdateManager.getNode()
♻️ Add NodeUpdateManager.getRevocationChecker()
♻️ Add NodeUpdateManager.getUpdateOverMandatory()
♻️ Add NodeUpdateManager.getByteCounter()
♻️ Add NodeCrypto.getPacketMangler()
♻️ Add NodeCrypto.isOpennet()
♻️ Add NodeCrypto.getSocket()
♻️ Add NodeCrypto.getPortNumber()
♻️ Add NodeCrypto.getMyIdentity()
♻️ Add NodeCrypto.getIdentityHash()
♻️ Add NodeCrypto.getIdentityHashHash()
♻️ Add NodeCrypto.getEcdsaPubKeyHash()
♻️ Add NodeCrypto.getMyARK()
♻️ Add NodeCrypto.get/setMyARKNumber()
♻️ Add NodeCrypto.getConfig()
♻️ Add NodeCrypto.getDetector()
♻️ Use NodeCrypto.getAnonSetupCipher()
♻️ Add PluginManager.getSingleUpdaterRequestClient()
♻️ Add SimpleToadletServer.getPushDataManager()
🐛 Fix JarClassLoaderTest to work with newer Java versions
Hiina (2):
Change logging for "too many excluded sub-arrays" from normal to minor.
Change logging for "too many excluded sub-arrays" from normal to minor.
Juiceman (20):
Add text further describing IPv6 limitations.
Fix bug limiting MTU to 1280
Remove security lowering kludge.
Change string comparison into a .isEmpty() check
Remove now unnecessary parenthesis
Apply sanity check for size
Improve text of log message
Fix some spelling and typos
Capitalize L in literal longs
Fix typo and remove extra unary operator
Replace deprecated tag
Typo in comment
Change more string comparisons into a .isEmpty() checks
Replace indexOf with .contains()
Remove unnecessary semicolons
Use Math.min and Math.max
Revert "Typo in comment"
Use switch statements
break early when condition is met
Remove parenthesis
Veniamin Fernandes (17):
Update junit to version 4.13.2
Change deprecated jcenter() maven repo to mavenCentral()
remove unused code from ClientRequestSelectorTest
make inner classes to be static in PersistentJobRunnerImplTest
improve SplitFileFetcherStorageTest and SplitFileInserterStorageTest
Improve other tests in the freenet.client package
Minor improvements and code formatting according to #826 review
Replace charset string parameters with constants from StandardCharsets
restore the Util.hashString() method
Fix indentation according to code review comments in #836
Use StandardCharsets.UTF_8 in the FileLoggerHook
replace GregorianCalendar with java.time.OffsetDateTime in CurrentTimeUTC
fix typo in the javadoc freenet.support.CurrentTimeUTC#getInMillis
improve date object construction in CurrentTimeUTC.get()
Replace OffsetDateTime with LocalDate in CurrentTimeUTC
improve method deprecation guidelines in MasterKeys and DatabaseKey
restore checked exception in the FileLoggerHook.logString() method signature