OTP-28 readiness

[ferd]

Good news: the compiler works and all CT tests pass
Bad news: we got some Dialyzer fixing to do.

Here's the error log from running on OTP-28-rc1:

λ [vps] rebar3 → main → ./rebar3 ct                                                                                                                                                                                  ===> Verifying dependencies...                                                                                                                                                                                       ===> Analyzing applications...                                                                                                                                                                                       ===> Compiling meck                                                                                                                                                                                                  ===> Analyzing applications...                                                                                                                                                                                       ===> Compiling cth_readable                                                                                                                                                                                          ===> Compiling bbmustache                                                                                                                                                                                            ===> Compiling cf
===> Compiling erlware_commons
===> Compiling getopt
===> Compiling providers
===> Compiling eunit_formatters
===> Compiling ssl_verify_fun
===> Compiling relx
===> Compiling certifi
===> Compiling rebar
===> Running Common Test suites...
%%% rebar_alias_SUITE: .........
%%% rebar_as_SUITE: ...........
%%% rebar_compile_SUITE: ...................................................................................
%%% rebar_compile_SUITE ==> always_recompile_when_erl_compiler_options_set: SKIPPED                                                                                                                                  %%% rebar_compile_SUITE ==> {tc_user_skip,"compile:env_compiler_options/0 available"}
.....
%%% rebar_compiler_dag_SUITE: ...........
%%% rebar_compiler_epp_SUITE: ......
%%% rebar_compiler_format_SUITE: ..                                                                                                                                                                                  %%% rebar_completion_SUITE: ...
%%% rebar_cover_SUITE: .............
%%% rebar_ct_SUITE: .........................................................
%%% rebar_deps_SUITE: ..............................
%%% rebar_dialyzer_SUITE: ..............
%%% rebar_dir_SUITE: .......................
%%% rebar_disable_app_SUITE: .
%%% rebar_discover_SUITE: ....
%%% rebar_dist_utils_SUITE: ....
%%% rebar_edoc_SUITE: ...
%%% rebar_escriptize_SUITE: .....
%%% rebar_eunit_SUITE: .........................................
%%% rebar_file_utils_SUITE: ......................
%%% rebar_hooks_SUITE: .............
%%% rebar_install_deps_SUITE: .......................................
%%% rebar_lock_SUITE: ....
%%% rebar_manifest_SUITE: .....
%%% rebar_namespace_SUITE: ...............
%%% rebar_new_SUITE: .......
%%% rebar_opts_parser_SUITE: ..
%%% rebar_parallel_SUITE: ....
%%% rebar_paths_SUITE: ....
%%% rebar_pkg_SUITE: ...........
%%% rebar_pkg_alias_SUITE: ....
%%% rebar_pkg_repos_SUITE: ..............
%%% rebar_plugins_SUITE: .............
%%% rebar_profiles_SUITE: .........................
%%% rebar_release_SUITE: ............
%%% rebar_resource_SUITE: ...
%%% rebar_src_dirs_SUITE: ............
%%% rebar_templater_SUITE: ..
%%% rebar_unlock_SUITE: ....
%%% rebar_upgrade_SUITE: ..............................................................
%%% rebar_uri_SUITE: ..
%%% rebar_utils_SUITE: .............................
%%% rebar_xref_SUITE: ....
Skipped 1 (1, 0) tests. Passed 637 tests.
λ [vps] rebar3 → main → rm -rf _build
λ [vps] rebar3 → main → rebar3 as dialyze dialyzer
===> No entry for profile dialyze in config.
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling bbmustache
===> Compiling cf
===> Compiling erlware_commons
===> Compiling cth_readable
===> Compiling getopt
===> Compiling providers
===> Compiling eunit_formatters
===> Compiling ssl_verify_fun
===> Compiling relx
===> Compiling certifi
===> Compiling rebar
===> Dialyzer starting, this may take a while...
===> Add debug_info to compiler options (erl_opts) if Dialyzer fails to load Core Erlang.
===> Updating plt...
===> Resolving project files...
===> Updating base plt...
===> Resolving base files...
===> Building with 221 files in /home/ferd/.cache/rebar3/rebar3_28.0-rc1_plt...
===> Copying /home/ferd/.cache/rebar3/rebar3_28.0-rc1_plt to /home/ferd/code/self/rebar3/_build/dialyze/rebar3_28.0-rc1_plt...
===> Checking 221 files in _build/dialyze/rebar3_28.0-rc1_plt...
===> Adding 455 files to _build/dialyze/rebar3_28.0-rc1_plt...
===> Doing success typing analysis...
===> Resolving project warning files...
===> Analyzing 173 files with _build/dialyze/rebar3_28.0-rc1_plt...

apps/rebar/src/rebar_compiler_xrl.erl
Line 38 Column 10: Unknown function leex:file/2
apps/rebar/src/rebar_compiler_yrl.erl
Line 45 Column 10: Unknown function yecc:file/2
/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:458:9: Body yields the type
          {'error', {'rebar_prv_common_test', {_, _}}} which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:459:9: Body yields the type
          {'error', {'rebar_prv_common_test', {_, _}}} | dict:dict(_, _) which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:467:9: Body yields the type
          {'error', {'rebar_prv_common_test', {_, _}}} which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:468:9: Body yields the type
          {'error', {'rebar_prv_common_test', {_, _}}} | dict:dict(_, _) which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:478:9: Body yields the type
          {'error', {'rebar_prv_common_test', {_, _}}} which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_common_test.erl:479:9: Body yields the opaque type
          dict:dict(_, _) whose opacity is broken by the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_eunit.erl:286:9: Body yields the type
          {'error', {'rebar_prv_eunit', {_, _}}} which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_eunit.erl:287:9: Body yields the type
          {'error', {'rebar_prv_eunit', {_, _}}} | dict:dict(_, _) which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_eunit.erl:295:9: Body yields the type
          {'error', {'rebar_prv_eunit', {_, _}}} which violates the opacity of the other clauses.

/home/ferd/code/self/rebar3/apps/rebar/src/rebar_prv_eunit.erl:296:9: Body yields the opaque type
          dict:dict(_, _) whose opacity is broken by the other clauses.


vendor/cth_readable/src/cth_readable_compact_shell.erl
Line 114 Column 50: The pattern "OK" can never match the type [65 | 68 | 69 | 70 | 73 | 76,...]
Line 117 Column 50: The pattern "OK" can never match the type [65 | 68 | 69 | 70 | 73 | 76,...]

vendor/cth_readable/src/cth_readable_failonly.erl
Line 90 Column 24: The created fun has no local return
Line 112 Column 53: The call logger:add_handler('cth_readable_failonly', 'cth_readable_failonly', #{'max_events':=_, 'sasl':='true', _=>_}) breaks the contract (HandlerId, Module, Config) -> 'ok' | {'error',term()} when HandlerId :: logger_handler:id(), Module :: module(), Config :: logger_handler:config()
Line 116 Column 53: The call logger:add_handler('cth_readable_failonly', 'cth_readable_failonly', #{'max_events':=_, 'sasl':='false', _=>_}) breaks the contract (HandlerId, Module, Config) -> 'ok' | {'error',term()} when HandlerId :: logger_handler:id(), Module :: module(), Config :: logger_handler:config()
Line 344 Column 17: The pattern {'ok', {_, Cfg}} can never match the type {'error',{'not_found',_}} | {'ok',#{'config'=>_, 'filter_default'=>'log' | 'stop', 'filters'=>[{atom(),{fun((#{'level':='alert' | 'critical' | 'debug' | 'emergency' | 'error' | 'info' | 'notice' | 'warning', 'meta':=#{'domain'=>[atom()], 'file'=>string(), 'gl'=>pid(), 'line'=>non_neg_integer(), 'mfa'=>{atom(),atom(),non_neg_integer()}, 'pid'=>pid(),'report_cb'=>fun((...) -> binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | char(),binary() | []) | {atom() | binary() | string(),[any()]}), 'time'=>integer(), atom()=>_}, 'msg':={atom() | binary() | string(),binary() | maybe_improper_list(any(),binary() | []) | map()}},_) -> 'ignore' | 'stop' | #{'level':='alert' | 'critical' | 'debug' | 'emergency' | 'error' | 'info' | 'notice' | 'warning', 'meta':=#{'domain'=>[atom()], 'file'=>string(), 'gl'=>pid(), 'line'=>non_neg_integer(), 'mfa'=>{atom(),atom(),non_neg_integer()}, 'pid'=>pid(), 'report_cb'=>fun((...) -> binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | char(),binary() | []) | {atom() | binary() | string(),[any()]}), 'time'=>integer(), atom()=>_}, 'msg':={atom() | binary() | string(),binary() | maybe_improper_list(any(),binary() | []) | map()}}),_}}], 'formatter'=>{atom(),#{atom()=>_}}, 'id'=>atom(), 'level'=>'alert' | 'all' | 'critical' | 'debug' | 'emergency' | 'error' | 'info' | 'none' | 'notice' | 'warning', 'module'=>atom()}}
Line 446 Column 5: Unknown function lager_app:start_handler/3

vendor/cth_readable/src/cth_readable_lager_backend.erl
Line 56 Column 9: Unknown function lager_util:config_to_mask/1
Line 89 Column 8: Unknown function lager_util:config_to_mask/1
Line 102 Column 10: Unknown function lager_util:is_loggable/3

vendor/erlware_commons/src/ec_cmd_log.erl
Line 248 Column 1: The pattern <State, Color, 'true', Msg> can never match the type <_,99 | 103 | 109 | 114,'false',_>
Line 253 Column 1: The pattern <State, Color, 'true', Msg> can never match the type <_,99 | 103 | 109 | 114,'false',_>

vendor/relx/src/relx.erl
Line 130 Column 2: Invalid type specification for function relx:build_relup/4. The success typing is (_,_,_,_) -> none() but the spec is (rlx_release:name(),rlx_release:vsn(),rlx_release:vsn(),rlx_config:t() | rlx_state:t()) -> {'ok',rlx_state:t()} | {'error',term()}
Line 132 Column 1: Function build_relup/4 has no local return

vendor/relx/src/rlx_relup.erl
Line 9 Column 2: Invalid type specification for function rlx_relup:do/4. The success typing is (_,_,_,_) -> none() but the spec is (atom(),string(),string() | 'undefined',rlx_state:t()) -> {'ok',rlx_state:t()} | relx:error()
Line 10 Column 1: Function do/4 has no local return
Line 37 Column 1: Function make_upfrom_script/4 has no local return
===> Warnings written to _build/dialyze/28.0-rc1.dialyzer_warnings
===> Warnings occurred running dialyzer: 29

So far our list of items to fix:

• Dialyzer in rebar3 xrl
• Dialyzer in rebar3 yrl
• Dialyzer in rebar3 CT
• Dialyzer in rebar3 Eunit
• Dialyzer in cth_readable
  • cth_readable needs a new release and to be updated
• Dialyzer in erlware_commons
  • erlware_commons needs a new release and to be updated
• Dialyzer in relx
  • relx needs a new release and to be updated

[ferd]

I also checked the changelog stating new compiler errors are emitted with suggestions on how to fix things, and they compose nicely with our recent richer output:

===> Compiling apps/rebar/src/rebar_compiler_xrl.erl failed
    ┌─ apps/rebar/src/rebar_compiler_xrl.erl:
    │
 42 │              rebar_compiler:ok_tuple(Source, Ws, Config, Opts);
    │                                              ╰── variable 'Ws' is unbound, did you mean 'W'?


    ┌─ apps/rebar/src/rebar_compiler_xrl.erl:
    │
 41 │          {ok, _Mod, W} ->
    │                     ╰── variable 'W' is unused

[garazdawi]

Hello!

As you may have seen the are working on some archive improvements in erlang/otp#9386. This PR changes how escript archives work quite a bit and we have found that it break rebar3 a lot. What the PR does that breaks rebar3 is this:

• Changes the layout of archives so that escript:extract/1 returns a different structure.
• Loads all modules of the archive when the escript is started.
• Disables interactive loading of modules in the archive.

The first two are easy enough to work around and only require small patches to rebar3, if any at all. But the last one turned out to be quite hard has various parts of rebar3 purges+deletes and mock/unmocks code within the archive, expecting it to be loaded again upon usage.

To fix this, we were thinking of adding an escript:reload/0 function, that would reload all beam files in the current archive. We tried adding it, but could not figure out which places to put it in and if it would be enough of a solution for rebar3 to work again.

So, we were wondering if you had any ideas about what could be done to make things easier for rebar3? We want to make things as easy as possibly for you, while also making things simpler/faster on our side.

I'm posting this in this thread as at first we aimed to have this in Erlang/OTP 28, but given how much it effects rebar3 I think we may have to postpone it.

cc @bjorng @lucioleKi

[ferd]

So, we were wondering if you had any ideas about what could be done to make things easier for rebar3? We want to make things as easy as possibly for you, while also making things simpler/faster on our side.

Most of our path handling and reloading has been isolated in the rebar_paths module.

It contains only 3 calls:

• set_paths([target(), ...], state()) -> ok.
• unset_paths([target), ...], state()) -> ok.
• clashing_apps([target), ...], state()) -> [{target(), [binary()]}].

the target() type is any of deps | plugins | runtime. This lets us do things such as figure out we're executing plugins, such that we will load [plugins, deps] and if there are any clashes between app, those in the plugin path will take over others. If you check out this code search you'll see that runtime is likely exclusively used by xref, whereas other targets may or may not be overlaid depending on the task.

then there's maybe the rebar3 shell, which like usual does some weird hacky magic. In general, if the path loading is properly constrained within rebar_path though, my expectation is that everything else should be alright.

Plugins are an unknown though, I have no idea what they respect or not.

[garazdawi]

We did get a bit further by fixing some things in rebar_paths, mostly in the clashing_apps functionality, but there is also the rebar_utils that does some things with modules and also rebar_agent as you pointed out.

Either we did not catch all the places in rebar_paths that we needed to, or there are more that needs to be fixed.

We'll make another attempt att fixing things, only focusing on rebar_paths and see if we can make that work.

[ferd]

Huh oh yeah there's some stuff in there too...

rebar3/apps/rebar/src/rebar_utils.erl

Lines 825 to 891 in 9311637

	%% Replace code paths with new paths for existing apps and
	%% purge code of the old modules from those apps.
	update_code(Paths) -> update_code(Paths, []).
	update_code(Paths, Opts) ->
	lists:foreach(fun(Path) ->
	Name = filename:basename(Path, "/ebin"),
	App = list_to_atom(Name),
	application:load(App),
	case application:get_key(App, modules) of
	undefined ->
	code:add_patha(Path),
	ok;
	{ok, Modules} ->
	%% replace_path causes problems when running
	%% tests in projects like erlware_commons that rebar3
	%% also includes
	%code:replace_path(App, Path),
	code:del_path(App),
	code:add_patha(Path),
	case lists:member(soft_purge, Opts) of
	true ->
	[begin code:soft_purge(M), code:delete(M) end || M <- Modules];
	false ->
	[begin code:purge(M), code:delete(M) end || M <- Modules]
	end
	end
	end, Paths).
	remove_from_code_path(Paths) ->
	lists:foreach(fun(Path) ->
	Name = filename:basename(Path, "/ebin"),
	App = list_to_atom(Name),
	application:load(App),
	case application:get_key(App, modules) of
	undefined ->
	application:unload(App),
	ok;
	{ok, Modules} ->
	application:unload(App),
	[case erlang:check_process_code(self(), M) of
	false ->
	code:purge(M), code:delete(M);
	_ ->
	?DEBUG("~p can't purge ~p safely, doing a soft purge", [self(), M]),
	code:soft_purge(M) andalso code:delete(M)
	end || M <- Modules]
	end,
	code:del_path(Path)
	end, lists:usort(Paths)).
	%% @doc Revert to only having the beams necessary for running rebar3 and
	%% plugins in the path
	-spec cleanup_code_path([string()]) -> true | {error, term()}.
	cleanup_code_path(OrigPath) ->
	CurrentPath = code:get_path(),
	AddedPaths = CurrentPath -- OrigPath,
	%% If someone has removed paths, it's hard to get them back into
	%% the right order, but since this is currently rare, we can just
	%% fall back to code:set_path/1.
	case CurrentPath -- AddedPaths of
	OrigPath ->
	_ = [code:del_path(Path) || Path <- AddedPaths],
	true;
	_ ->
	code:set_path(OrigPath)
	end.

I think rebar_utils:update_code and rebar_utils:remove_from_code_path are no longer used on our side (maybe in plugins, hence why it trails in utils).

We do have trailing usages of cleanup_code_path in cover and the bare compiler, also exposed in the API for plugins.

I think that based on the doc, cleanup_code_path should be possible to stub out and replace with rebar_paths:unset_paths([deps, plugins], State) and/or rebar_paths:set_paths([plugins], State). This stuff is unfortunately really finnicky and hard to test, so I don't how how risky it would be. I'm not too worried about the bare compiler, but the cover code could be messy because I don't know how good the new code is at dealing with in-memory modules and flushing them out.

[ferd]

oh god I've just noticed I ran the issue's command is rebar3 as dialyze dialyzer instead of rebar3 as dialyzer dialyzer.

We're OTP-28 ready, aside from the path issues the OTP team is working on.

@garazdawi on Tuesday the 4th, we're having the build and packaging meeting on the EEF slack if someone from the OTP team wants to chat about it there.

Otherwise if there's any sort of setup that reproduces issues in there maybe I can help track the weirdness in some ways