Skip to content

reverse-engineering.md

Latest commit

 

History

History
143 lines (115 loc) · 7.23 KB

reverse-engineering.md

File metadata and controls

143 lines (115 loc) · 7.23 KB
Raw
title Reverse Engineering
type technique
tags
reverse-engineering
ctf
ghidra
radare2
angr
malware
binary
phase exploitation
date_created 2026-06-16
date_updated 2026-07-14
sources
hacktricks-binexp

What it is

Recovering program logic from compiled binaries (or bytecode) without source: understanding control flow, finding hidden checks, extracting flags/keys, and locating vulnerabilities to feed [[binary-exploitation]]. Core CTF category and the basis of malware analysis.

How it works

Static analysis disassembles/decompiles to recover structure; dynamic analysis runs under a debugger to observe real behaviour; symbolic execution explores many paths automatically. Most CTF RE is a hidden comparison (input vs transformed flag) you reverse or solve.

Attack phases

Exploitation (challenge solving; vulnerability discovery for exploit dev; malware triage).

Prerequisites

  • The binary. Determine arch/format first; matching disassembler/emulator for the target.

Methodology

1. Triage

file ./bin;  checksec --file=./bin        # arch, PIE, NX, canary, RELRO
strings -n6 ./bin | less;  nm -C ./bin;  ldd ./bin
rabin2 -zzqq ./bin                        # strings + sections (radare)
detect-it-easy / die ./bin                # packer/compiler ID; UPX -> upx -d

2. Static analysis

  • Ghidra (free, decompiler): import -> auto-analyze -> read decompiled main; rename vars; follow the comparison that gates success.
  • radare2 / Cutter: r2 -A ./bin -> afl (list funcs) -> pdf @main (disasm) -> VV (graph). See [[radare2]].
  • IDA if available. Look for: strcmp/memcmp against flag, XOR/transform loops, system/execve, format strings.

3. Dynamic analysis

ltrace ./bin;  strace ./bin               # lib + syscalls (catch strcmp args, opens)
gdb-gef ./bin -> break *main+N; run; x/s $rdi   # inspect compare operands. See [[gdb-gef]]

Set a breakpoint on the compare, read both operands -> one is the expected flag.

4. Symbolic / automated

import angr, claripy
p = angr.Project("./bin"); s = p.factory.entry_state()
sm = p.factory.simulation_manager(s)
sm.explore(find=0x<win_addr>, avoid=0x<fail_addr>)
print(sm.found[0].posix.dumps(0))         # input that reaches "win"

Good for many-branch input checks and key constraint solving (z3 backend).

5. Bytecode / managed

  • Python .pyc -> decompyle3 / uncompyle6; PyInstaller -> pyinstxtractor.
  • Java/.NET -> jadx, dnSpy, ilspy. JS -> deobfuscate (de4js, AST).

Bypasses and variants

  • Anti-debug: patch ptrace check (ret 0), or gdb set follow-fork; LD_PRELOAD a fake.
  • Packed/obfuscated: dump from memory after unpack (gdb at OEP); VM-based -> reverse the bytecode interpreter then write a disassembler.
  • Encoded flag: trace the transform loop, invert it (XOR/add/swap) in Python.

Detection and defence

Strip symbols, control-flow obfuscation, packing, anti-debug/anti-VM, server-side checks for license/flag logic.

Tools

[[ghidra]], [[radare2]] / Cutter, IDA, [[gdb-gef]], angr, ltrace/strace, pwntools, jadx/dnSpy, binwalk ([[binwalk]]). Pairs with [[binary-exploitation]], [[malware-analysis]] and [[fuzzing]].

Anti-debug / anti-VM and shellcode analysis

Execution guards to expect (and neutralize) in samples:

  • Debugger checks: IsDebuggerPresent, PEB BeingDebugged flag, CheckRemoteDebuggerPresent, timing (rdtsc/GetTickCount deltas). Patch the check to return 0, or LD_PRELOAD a fake on Linux ptrace.
  • Anti-VM: CPUID hypervisor bit, BIOS/disk-model strings, MAC OUI, IN (VMware backdoor).
  • Sandbox/emulator canaries: scan own process for Defender emulator exports (MpVmp32Entry, VFS_*, ThrdMgr_*); if found, sleep 10-30 min to time out analysis.
  • Locale gate: GetKeyboardLayout/GetUserDefaultLangID abort on CIS locales before any IOC.
  • Argument gate: run only if a benign switch like /i:--type=renderer is present. Hunting: a process that queries several locale/keyboard/anti-VM APIs early then exits with no activity is gated malware.

Shellcode / packed sample workflow:

scdbg.exe -f sc.bin -r          # which WinAPIs the shellcode calls, self-decode?
scdbg.exe -f sc.bin -d          # dump decoded shellcode
# BlobRunner / jmp2it: allocate the blob, print/loop, attach x64dbg at the address, step
upx -d packed.exe               # trivial packers; else dump from memory at OEP
detect-it-easy sample           # packer/compiler ID

MBA-obfuscated arithmetic: keep the bit-width, verify each rewrite with Z3; CoBRA simplifies (cobra-cli --mba "(x&y)+(x|y)" --verify -> x+y). Movfuscator (all-mov) -> demovfuscator.

Word / VBA macro malware analysis

Maldocs hide their real logic behind obfuscation, so extract and read the VBA rather than running it. Common tricks: junk code guarded by an always-false If to bloat the listing, and GetObject reading data out of UserForm controls (text boxes nested inside text boxes) so the payload string never appears in the module body.

Static extraction (oletools, safe, no execution):

olevba --decode sample.doc        # dump macros + auto-deobfuscate common encodings
oledump.py sample.doc             # list streams; -s <n> -v to dump a macro stream
mraptor sample.doc                # flag auto-exec + write + exec = likely malicious

Look for auto-exec triggers (AutoOpen, Document_Open, Workbook_Open), string building via Chr()/concatenation/StrReverse, Shell/WScript.Shell.Run, and form-field reads. Rebuild the decoded command by hand from the transform rather than trusting the visible strings, and pull payload text out of the embedded forms.

Sources

Wired sub-techniques

  • [[crash-analysis]]
  • [[exploit-development]]
  • [[seh-exploitation]]
  • [[windows-exploit-development]]

Thick-client / game -> backend API

A shipped desktop game/app is a THIN CLIENT for a server API: the real authority (and often the flag) is server-side, and any on-client "win", cheat-console, or vault animation is a decoy. Decompile the client to recover the whole protocol, then replay it directly.

  • Managed builds decompile cleanly: Godot-Mono / Unity / .NET -> ilspycmd <asm>.dll (or ILSpy / dnSpy); Android -> jadx. Pull out: the base URL, every endpoint + JSON shape, the request-signing key (hardcoded HMAC/secret), and any client-side role/permission derivation the client computes but never sends (e.g. an unused DeriveStaffRole() that yields a privileged claim the UI won't make).
  • Forge what the UI won't: with the recovered signing key you can sign arbitrary requests, including the privileged role/params the client hardcodes away from you. That is the intended bypass, not a game hack; client-side gates (cheat code, on-screen "vault opens") are lies.
  • Server gotchas, read them from the JSON error bodies (always read 4xx): step time-gates (too_fast / need / got -> sleep then retry = "proof of play"), strict step order (wrong_order / expected), and a per-step rotating token/nonce returned in each reply that must be threaded into the next request + its signature (reusing the session's first token -> bad_token).