If you used a standard compiler setup, the entry point is not your main function. It is a hidden function injected by Microsoft called mainCRTStartup. This is the C Runtime (CRT).
Most Significant Bit (MSB) - Least Significant Bit (LSB)
The primary difference is that signed integers can represent both positive and negative values, while unsigned integers can only represent non-negative values (zero and positive).
KERNEL32 is the name of the 64-bit library.
Operands = Parameters
In assembly, the square brackets [] act exactly like the dereference operator (*) in C or C++. They change the destination of the operation from a CPU register to a physical address out in system RAM.
the stack grows downward, subtracting from RSP creates empty space in memory.
Windows demands that every time you call an API, you must leave exactly 32 bytes (four 8-byte slots) of empty space on the stack. The Windows API uses this as a scratchpad to back up the fastcall registers if it gets interrupted. If you don’t leave this space, the API will overwrite your other memory.
The stack must always be a multiple of 16 bytes before an API executes. When your main function started, the CPU pushed an 8-byte return address onto the stack, throwing the alignment completely off. By subtracting an extra 8 bytes, you push the stack pointer back onto a perfect 16-byte boundary.
x86/x64 Architecture Evolution Timeline
A historical mapping of Intel and AMD microarchitectures, tracking the introduction of critical memory models, instruction sets, and register expansions.
The 32-Bit Foundation (IA-32 / x86)
1985 (80386): First 32-bit Intel processor. Crucial introduction of 32-bit registers/addresses, the optional flat memory model, and paging.
2013 (Intel Haswell): 4th Gen Core series. Introduced AVX2 and FMA instructions.
Modern Processors (Intel & AMD): Standardization on AVX2 and the massive AVX-512 instruction sets.
CPU Operating Modes
The CPU transitions through different hardware states that dictate memory addressing limits and security ring enforcement.
1. Legacy Mode (The 32-Bit & 16-Bit Era)
This branch represents the older architecture styles before AMD64 took over.
Real Mode (CPU Reset)
The Architecture : When the CPU first powers on, it starts here. It is a 16-bit environment with a maximum of 1MB of addressable memory.
The Maldev Context : There is zero security. There are no privilege rings (Ring 0 and Ring 3 do not exist), no memory protections, and no paging. Every program has raw, direct access to the hardware. Advanced bootkits (like BlackLotus) target the boot sequence here or early UEFI phases to compromise the system before the OS or EDR ever loads.
Protected Mode
The Architecture : The jump to 32-bit. This is where modern operating system concepts were born. It introduced Virtual Memory (Paging) and hardware-enforced privilege rings (Ring 0 for Kernel, Ring 3 for User-land).
The Maldev Context : If you write a 32-bit exploit, you are dealing with Protected Mode. Memory is segmented and protected. You cannot just read another program’s memory without passing through the kernel (using APIs like ReadProcessMemory), which EDRs hook and monitor.
Virtual 8086 Mode
The Architecture : A hardware sandbox inside Protected Mode designed to run legacy 16-bit DOS applications safely without crashing the 32-bit OS.
The Maldev Context : Mostly a historical artifact today, but in the past, hackers used to find vulnerabilities here to execute “VM Escapes,” breaking out of the 16-bit sandbox to gain Ring 0 execution in the 32-bit host.
2. IA32e / Long Mode (The Modern 64-Bit Kingdom)
64-Bit Mode (Focus of the course)
The Architecture : The native 64-bit environment. It utilizes massive registers (RAX, RBX) and a massive flat memory address space.
The Maldev Context : Security is brutal here. Microsoft introduced Kernel Patch Protection (PatchGuard), SMEP, and SMAP to prevent you from easily hooking the kernel. You have to rely on advanced BYOVD (Bring Your Own Vulnerable Driver) attacks to disable telemetry, because you can no longer just overwrite kernel structures directly like you could in 32-bit Protected Mode.
Compatibility Mode
The Architecture : This is how a 64-bit Windows OS runs older 32-bit applications. Windows uses a subsystem called WoW64 (Windows 32-bit on Windows 64-bit) to translate 32-bit API calls into 64-bit system calls under the hood.
The Maldev Context : This is an elite attack surface. EDRs often place their detection hooks in the 32-bit memory space of a WoW64 process. Exploit developers use a technique called Heaven’s Gate to manually force the CPU to switch from Compatibility Mode into 64-Bit Mode mid-execution. By doing this, the malware completely steps over the EDR’s 32-bit hooks, executing raw 64-bit syscalls completely undetected.
3. System Management Mode (SMM)
The Architecture : Often referred to as “Ring -2”. It is a special, highly privileged operating mode used by the hardware firmware (BIOS/UEFI) to handle hardware control like thermal management or power states.
The Maldev Context : SMM execution is completely invisible to the operating system, the hypervisor, and the EDR. If an APT or state-sponsored group manages to write a rootkit into SMM, they have “God Mode.” The OS cannot audit it, and the malware survives complete hard drive wipes and OS reinstallations.
NASM vs. MASM
While both assemblers use Intel syntax (mov destination, source), they process variables, memory references, and directives entirely differently.
1. Memory Dereferencing (The Bracket Rule)
NASM: Brackets []always mean “read/write the data at this memory address.” Without brackets, it always means the immediate address pointer.
MASM: Brackets are ambiguous and often optional. MASM tracks variable types internally and infers memory access.
; --- NASM Syntax ---mov rax, my_var ; Moves the ADDRESS (pointer) of my_var into RAXmov rax, [my_var] ; Moves the DATA stored inside my_var into RAX; --- MASM Syntax ---mov rax, OFFSET my_var ; Requires 'OFFSET' to get the ADDRESSmov rax, my_var ; Moves the DATA stored inside (brackets optional)
2. Variable Definitions
NASM: Labels are defined with the data allocation allocation size (db, dw, dd, dq) placed after the label name. Case-sensitive.
MASM: Data allocations (DB, DWORD, QWORD) are placed between the variable name and the value. Case-insensitive.
; --- NASM ---msg db "Hello World", 0val dq 0x1122334455667788; --- MASM ---msg DB "Hello World", 0val QWORD 01122334455667788h ; Note the trailing 'h' for hex in MASM
3. High-Level Constructs & Macros
NASM: Pure, bare-metal assembly. It does not support high-level programming logic loops natively. You must manually construct loops and branches using loops (cmp, jmp, conditional jumps like je, jne).
MASM: Includes built-in high-level compiler directives that mimic C logic, allowing for faster structured coding.
NASM (-f bin): The absolute king for crafting position-independent shellcode. It can output completely raw, flat binary blobs without adding standard OS headers (PE/ELF), which you can feed directly into memory allocation loaders.
MASM (ml64.exe): Designed to link smoothly with the Microsoft C/C++ compiler (cl.exe). If you want to embed custom standalone assembly procedures natively inside a Visual Studio C++ project, MASM is seamless.
The Low-Level Execution Pipeline: From Code to RAM
When writing assembly or low-level exploit code, you are dealing directly with the bare metal. To turn plain text into bytes executing on a physical CPU, your code must pass through this exact four-stage automated pipeline.
1. The Compiler (The Architect)
What it does: Translates high-level language (like C or C++) into low-level CPU assembly language.
Why it matters: The hardware cannot process abstract logic like if (x == 5). The compiler translates this logic into the hardware’s specific instruction set architecture (ISA).
Output: Assembly Source File (.asm or .s).
Example: Translates int x = 5; into mov eax, 5.
2. The Assembler (The Translator)
What it does: Translates assembly mnemonics (text instructions) into raw binary machine code bytes (opcodes). This is where tools like NASM or MASM operate.
Why it matters: The CPU doesn’t understand words like mov or push. It only executes raw binary bytes (hex representation). The assembler parses your instruction text, maps it to the CPU’s opcode matrix, and writes the bytes out.
Output: Object File (.obj on Windows, .o on Linux).
Crucial Note: An object file contains raw machine code but is not runnable. It lacks an execution header, OS-level structure, and resolved external memory references.
3. The Linker (The Construction Crew)
What it does: Collects multiple object files, merges them, resolves external function references, and packages the entire bundle into a structure the OS recognizes.
Why it matters: If your code calls a Windows API (like MessageBoxA or VirtualAlloc), your object file only has an empty placeholder. The linker looks up the OS import libraries (.lib), links your placeholders to the actual DLL exports, and adds the OS executable structure (the Portable Executable / PE Header).
Output: Executable File (.exe or .dll).
4. The Loader (The Gatekeeper)
What it does: A built-in OS kernel component that reads the executable from disk, carves out a virtual memory address space (RAM), loads the segments, and points the CPU to the entry point.
Why it matters: Executables must reside in physical/virtual memory to run. The loader parses the PE header, maps the code/data sections with explicit memory protections (Read, Write, Execute / RX, RW), performs necessary base relocations, and points the CPU’s Instruction Pointer register (RIP/EIP) directly to your program’s entry point.
Data Types & Little-Endian Memory Layout
When writing custom shellcode, understanding how the CPU stores variables in RAM is mandatory to prevent access violations.
The Mapping:
Byte = 8 bits (char in C, db in ASM)
Word = 16 bits (short in C, dw in ASM)
Dword = 32 bits (int in C, dd in ASM)
Qword = 64 bits (long long in C, dq in ASM)
The Offensive Context (Little-Endian): x64 architecture stores data with the Least Significant Byte (LSB) at the lowest memory address. If you need to push the DWORD 0x11223344 onto the stack for an API call, it physically sits in memory as 44 33 22 11. If you fail to reverse your bytes when writing raw opcodes, your API arguments will be garbage and the thread will crash.
The Core Data Types (MASM)
Byte (8-bit): Used for single ASCII characters or small opcodes. Hex: 0xFF
Word (16-bit): Used for UTF-16 characters or short offsets. Hex: 0xFFFF
Dword (32-bit): The standard 32-bit integer. Hex: 0xFFFFFFFF
Qword (64-bit): Used for 64-bit memory addresses and pointers. Hex: 0xFFFFFFFFFFFFFFFF
The Two’s Complement Vulnerability
Memory holds bits, not intent. The hex value 0xFFFFFFFF in a 32-bit register translates to:
4,294,967,295 if read by an unsigned instruction.
-1 if read by a signed instruction (because the highest bit is set to 1).
The Exploit: Attackers pass massive hex values (like 0xFFFFFFFF) into bounds-checking functions. If the developer used a signed integer check, the system reads it as -1 (passing the check). When passed to memory allocation APIs that expect unsigned integers, it results in massive allocations and heap corruption.
Hexadecimal & Memory Addressing
Base 16 (Hex) is mandatory for exploit development because exactly two hex characters equal one byte of memory. All memory offsets, shellcode arrays, and pointer arithmetic must be calculated in Hex to ensure alignment.
Register Aliasing & Null-Byte Evasion
General Purpose Registers (like RAX) are 64-bit containers, but they are sliced into smaller, directly accessible sub-registers (EAX, AX, AH, AL).
The Offensive Context: This slicing is the primary mechanism for writing position-independent shellcode that avoids null-byte (0x00) restrictions.
The Trap: Compiling mov rax, 0x5 generates the opcodes 48 c7 c0 05 00 00 00. String-copy vulnerabilities will terminate execution the moment they hit those zeros.
The Solution: Zero the register first with xor rax, rax, then write only to the 8-bit sub-register: mov al, 0x5. This generates b0 05 clean, tiny, and zero null bytes.
The Windows x64 Calling Convention
In the 32-bit era, arguments were pushed to the stack. In 64-bit Windows development, this is obsolete. The Application Binary Interface (ABI) dictates strict hardware usage.
The Rule: When writing loaders that call Windows APIs (like VirtualAlloc), the first four arguments must be loaded into specific registers in this exact order:
RCX (Arg 1)
RDX (Arg 2)
R8 (Arg 3)
R9 (Arg 4)
The Maldev Context: Any additional arguments are pushed to the stack. If you mess up this exact sequence, the kernel will read the wrong pointers, the API call will fail, and your payload will not execute.
x64 Architecture Registers
In low-level engineering, hardware registers are your absolute source of truth. Using the wrong register doesn’t just throw a compiler warning; it causes an access violation and crashes the thread.
1. The Core Volatile Registers (x64 Calling Convention)
If you are building a custom loader and calling Windows APIs (like VirtualAlloc or CreateThread), the hardware strictly enforces where your arguments go.
RCX, RDX, R8, R9: These handle arguments 1 through 4. Any additional arguments get pushed to the stack. If you misalign these, the kernel drops the execution.
RAX (The Accumulator): The scratchpad. Most importantly, when a Windows API finishes executing, it always drops its return value (like a new memory address or an error code) right here.
2. The Execution Pointers
These registers control the thread’s memory state and execution flow. Hijacking these is the core of exploit development.
RIP (Instruction Pointer): Points to the exact memory address of the next executing opcode. In shellcode, you never hardcode memory addresses. You use RIP-relative addressing to dynamically locate your variables, making your payload 100% Position Independent.
RSP (Stack Pointer): Points to the current top of the stack. If an EDR blocks standard execution, you overwrite the stack and manipulate RSP to force the CPU to bounce between existing, legitimate memory addresses (Return-Oriented Programming / ROP).
RBP (Base Pointer): Traditionally used to set up local stack frames for variables, but modern optimized C compilers often just use it as another general storage register.
3. Index & General Storage
RSI & RDI (Source & Destination Index): The heavy lifters for raw memory manipulation. If your loader is decrypting a payload and moving it into a new memory segment, you use these registers with string instructions to blast bytes from point A to point B rapidly.
RBX, R10 - R15: General storage. These are often “non-volatile,” meaning if a function uses them, it is required to save their original values to the stack and restore them before returning.
4. Specialized Hardware
RFLAGS: Stores the mathematical state of the CPU (Zero Flag, Carry Flag, Sign Flag). When you write an if statement or a loop in C, the compiler translates it into a cmp instruction that updates RFLAGS, followed by a conditional jump (je, jne) that reads those flags.
XMM / YMM (SIMD Registers): The massive 128-bit and 256-bit data containers. Red teamers hijack these specifically for rapid, stealthy payload encryption/decryption in memory, bypassing standard Windows cryptographic APIs.
RFLAGS
The RFLAGS register controls conditional branching and hardware-level execution states. Understanding how to manipulate these bits is mandatory for binary patching, reverse engineering, and anti-analysis tradecraft.
1. Conditional Branching Flags
These flags are dynamically updated after arithmetic or comparison (cmp, test) instructions.
Zero Flag (ZF): Set to 1 if the result of an operation is exactly zero. This is the primary target for binary instrumentation. By flipping ZF in a debugger at runtime, an analyst can invert the outcome of authorization checks or EDR telemetry branches (e.g., forcing a jne to act like a je).
Sign Flag (SF): Set to 1 if the result of an operation is negative. Used heavily in loop decrements.
Carry Flag (CF): Set to 1 if an operation results in an overflow of the destination register’s physical bit capacity. Frequently manipulated in custom, bitwise shellcode encryption routines (shifts/rotates).
2. Hardware Control Flags
These flags directly alter how the CPU processes instructions.
Direction Flag (DF): Dictates the memory traversal direction for high-speed string operations (movsb, stosb).
cld sets DF to 0 (Increment/Forward).
std sets DF to 1 (Decrement/Backward).
Maldev Rule: Always execute cld before writing payloads into memory via assembly. Failing to do so can result in backwards memory corruption if the thread inherited a dirty flag state.
Trap Flag (TF): The hardware debugging flag. When set to 1, the CPU fires an INT 1 exception after every single opcode execution. Used maliciously by rootkits and loaders to detect the presence of Ring-3 debuggers (x64dbg) or Ring-0 kernel debuggers (WinDbg).
Sign vs. Zero Extension
When a compiler casts a smaller data type (like an 8-bit char) into a larger one (like a 32-bit int), the CPU must fill the newly created empty bits. The compiler’s choice of instruction dictates how the raw memory is interpreted.
Zero Extension (movzx)
Used when the source variable is defined as unsigned. The CPU pads the new empty space on the left entirely with zeros.
0xFF (255) becomes 0x000000FF (255).
Sign Extension (movsx)
Used when the source variable is defined as signed. To prevent a negative number from being corrupted into a massive positive number, the CPU copies the Most Significant Bit (the leftmost sign bit) across all the new empty space.
0xFF (-1) becomes 0xFFFFFFFF (-1).
Instruction Operands Cheat Sheet
Operands specify where an assembly instruction retrieves or stores its data.
The Three Operand Targets
Immediate: Hardcoded value embedded directly within the instruction bytes (e.g., mov rax, 0x20). Max execution speed.
Register: Data stored inside the CPU’s on-silicon storage cells (e.g., mov rax, rcx). High-speed execution.
Memory: Data stored in system RAM, denoted by brackets [] (e.g., mov rax, [rbx]). Acts as a pointer dereference (*ptr). Requires a bus transaction to fetch.
SIB Addressing Math
The syntax [base + index * scale] allows real-time calculation of array offsets at the hardware level.
Example:[rbx + rsi * 8] translates to: Start at base address RBX, skip forward RSI elements, where each element is 8 bytes wide (a 64-bit pointer or QWORD).
Pointer Size Enforcement
When moving data from an immediate value into a memory address, the destination size must be explicitly declared to avoid ambiguity:
byte ptr [rax]→ 1 Byte (8 bits)
word ptr [rax]→ 2 Bytes (16 bits)
dword ptr [rax]→ 4 Bytes (32 bits / DWORD)
qword ptr [rax]→ 8 Bytes (64 bits / QWORD)
The MOV Instruction Family & Data Movement
In x86/x64 assembly, moving data is not as simple as an assignment operator (=). The CPU utilizes distinct instructions and physical pathways depending on the size, location, and endianness of the data being moved.
Core Rules & Constraints
Memory-to-Memory Restriction: The CPU cannot move data directly from one RAM address to another RAM address in a single instruction (e.g., mov [rax], [rbx] is illegal). A general-purpose register must always act as the intermediary.
64-Bit Immediate Exception:MOV is the only instruction capable of loading a literal, hardcoded 64-bit value directly into a register (e.g., loading a full 8-byte virtual memory address).
Size Matching: The source and destination operands must be identical in size unless an explicit extension instruction is used.
Extended MOV Variations
MOVZX (Move with Zero Extension): Copies a smaller unsigned value into a larger register, padding the upper bits with 0s.
MOVSX / MOVSXD (Move with Sign Extension): Copies a smaller signed value into a larger register, copying the Most Significant Bit (sign bit) across the upper bits to preserve negative numbers. MOVSXD specifically targets a 32-bit source moving into a 64-bit destination.
MOVBE (Move Big Endian): Moves data while simultaneously reversing the byte order.
MOVD / MOVQ (Move Doubleword/Quadword to/from XMM): Moves 32-bit or 64-bit data between standard general-purpose registers and 128-bit vector/SIMD registers (XMM).
The Offensive Engineering Context
Network Order Realignment (MOVBE): While x64 processors operate using Little Endian, network protocols require Big Endian (Network Byte Order). Instead of parsing complex bit-shifting math functions in C to align a port or IP structure for a socket connection, offensive operators use MOVBE to reverse data on the fly.
Register Evasion (MOVD / MOVQ): Memory scanners, AV, and EDR solutions frequently monitor general-purpose registers (RAX, RBX, etc.) during execution to catch plaintext strings or decryption keys. Vector registers (XMM) are often ignored by legacy or performance-constrained hooks. Staging payloads or multi-byte cryptographic keys inside XMM registers via MOVQ hides them from basic inspection.
ADD, SUB, and Flag Mechanics Cheat Sheet
ADD and SUB alter the destination register directly and dynamically update the system’s status flags based on the resulting mathematical state.
Immediate Value Sign-Extension
Arithmetic operations cannot accept raw 64-bit immediate values. If the destination is a 64-bit register, a 32-bit immediate value is automatically sign-extended by the processor before execution.
Hardcoded 5 becomes 0x0000000000000005
Hardcoded -5 becomes 0xFFFFFFFFFFFFFFFB
Multi-Register Arithmetic Chain
To perform math on data structures wider than 64 bits (such as cryptographic payloads):
ADC (Add with Carry):Dest = Dest + Source + CarryFlag
SBB (Subtract with Borrow):Dest = Dest - Source - CarryFlag
Key Status Flags Affected
Zero Flag (ZF): Automatically set to 1 if the result of the operation is exactly 0. Frequently targeted by conditional jumps (JE/JNE) in control-flow flattening obfuscation.
Carry Flag (CF): Set to 1 if an unsigned operation overflows (in ADD) or requires a borrow (in SUB).
CALL & RET
The CPU does not inherently understand “functions.” It relies entirely on the Stack and the Instruction Pointer (RIP) to branch execution and find its way back.
The Core Mechanics
CALL <address>: Executes two hardware steps in a single cycle:
Pushes the address of the next sequential instruction (the return address) onto the top of the stack.
Overwrites the RIP register with the target <address>, forcing execution to jump there.
RET (Return): The exit strategy.
Pops the 64-bit value sitting at the top of the stack ([RSP]) directly into the RIP register.
Execution instantly snaps back to the restored address.
RET [value] (e.g., ret 8): Pops the return address, then immediately adds [value] to the Stack Pointer (RSP). Used in specific calling conventions to automatically clean up function arguments from the stack without requiring extra instructions.
Args 5+ (Overflow): Pushed onto the stack from right to left. They reside directly above the shadow space.
C++ Class Context: The implicit this pointer for structural object objects always takes priority in RCX.
Stack Preservation Boundaries
Shadow Space: A mandatory 32-byte management block allocated by the caller immediately before execution, reserved strictly for the API’s internal register spills.
Volatile Registers (Unsafe across calls):RAX, RCX, RDX, R8-R11. Can be altered at any time by the invoked function.
Non-Volatile Registers (Safe/Must Preserve):RBX, RBP, RSI, RDI, R12-R15. Must be pushed to the stack and popped back if modified within custom procedures.
