Option -hw_cache_consistency not working on AArch64

[egrimley-arm]

If you run the program from #2055 with a debug build of a2beedc you get:

Internal Error: DynamoRIO debug check failure: /home/egrimley/dr-upstream/dynamorio/core/unix/signal.c:4118 get_sigcxt_stolen_reg(sc) != (reg_t)*get_dr_tls_base_addr()

The same happens (with a different line number) with release_8.0.0-1, release_9.0.1 or release_10.0.0 so the option has perhaps never worked on AArch64.

[egrimley-arm]

It runs like this:

main_signal_handler
main_signal_handler_C
  get_thread_private_dcontext
  local_heap_protected
  is_safe_read_ucxt
  compute_memory_target
  is_on_dstack
  is_on_alt_stack
  is_on_initstack
  safe_is_in_fcache
  check_for_modified_code
    was_executable_area_writable
      d_r_read_lock
      executable_areas_match_flags
    d_r_mutex_lock
    fragment_pclookup
      fcache_fragment_pclookup
    recreate_app_pc
    handle_modified_code
    transfer_from_sig_handler_to_fcache_return

The comment

dynamorio/core/unix/signal.c

Lines 4110 to 4112 in e571733

	/* We do not have to set dr_reg_stolen in dcontext's mcontext here
	* because dcontext's mcontext is stale and we used the mcontext
	* created from recreate_app_state_internal with the original sigcontext.

claims that recreate_app_state_internal was used but this is untrue: in this case recreate_app_state_internal was called with just_pc so the rest of the register state must come from somewhere else.

Does anyone (e.g. @derekbruening) know if -hw_cache_consistency has ever worked on non-Intel?

[derekbruening]

Does anyone (e.g. @derekbruening) know if -hw_cache_consistency has ever worked on non-Intel?

I think it has only ever been used on x86.

[egrimley-arm]

It seems that that bit of code which is only for Arm and RISC-V (

dynamorio/core/unix/signal.c

Lines 4109 to 4133 in e571733

	#if defined(AARCHXX) || defined(RISCV64)
	/* We do not have to set dr_reg_stolen in dcontext's mcontext here
	* because dcontext's mcontext is stale and we used the mcontext
	* created from recreate_app_state_internal with the original sigcontext.
	*/
	/* We restore dr_reg_stolen's app value in recreate_app_state_internal,
	* so now we need set dr_reg_stolen to hold DR's TLS before sigreturn
	* from DR's handler.
	*/
	ASSERT(get_sigcxt_stolen_reg(sc) != (reg_t)*get_dr_tls_base_addr());
	/* Preserve the translated value. */
	dcontext->local_state->spill_space.reg_stolen = get_sigcxt_stolen_reg(sc);
	/* Now put DR's base in the sigcontext. */
	set_sigcxt_stolen_reg(sc, (reg_t)*get_dr_tls_base_addr());
	# ifdef RISCV64
	os_set_app_tls_base(dcontext, TLS_REG_LIB, (void *)get_sigcxt_tp_reg(sc));
	/* Now put host tp in the sigcontext. */
	set_sigcxt_tp_reg(sc, (reg_t)read_thread_register(TLS_REG_LIB));
	# endif
	# ifdef ARM
	/* We're going to our fcache_return gencode which uses DEFAULT_ISA_MODE */
	set_pc_mode_in_cpsr(sc, DEFAULT_ISA_MODE);
	# endif
	#endif

) needs to also be conditional on if (is_kernel_xfer). But that's not the only change needed to make -hw_cache_consistency work on AArch64.

[egrimley-arm]

@derekbruening Is it possible that the "sandbox" mechanism is not working on Arm, or some component of it is not implemented? I have a slightly modified DynamoRIO and a test program that calls a self-modifying block of code twice. There is a only a single SIGSEGV, and only the first code change takes effect. However, if I insert .align 12 so that the code doing the modification and the code being modified are in different pages (though it's still a single block) then there are two SIGSEGVs and both changes take effect. What should I see in the logs if the sandbox mechanism detects that the source code has changed? Is there a command-line option for disabling the sandbox mechanism, for a comparison?

[derekbruening]

Is the sandbox mangling code not ported beyond x86?
E.g. insert_selfmod_sandbox()