BSides Mumbai CTF — Reverse Engineering Challenge Author Writeups

Hello guys, 5h1kh4r here again! This (and the next) post is (are?) going to be slightly different from my usual CTF writeups because instead of writing about challenges I solved, I’ll be covering some of the reverse engineering challenges I made for BSides Mumbai CTF.

The three challenges covered here are:

Camlpocalypse
Revenant
PSE ESP this one had zero solves so…

I may have gone slightly overboard with some of these :)

Anyways, without wasting much time let’s get into the challenges.

1. Camlpocalypse

Category: Rev
Author: 5h1kh4r

Have you heard of Camels? They’re really nice animals. If you know me, I really love love Camels. If you played BSidesMumbaiCTF25 and BlitzCTF, you’re already familiar with this ;)

Players are given a single stripped x86-64 ELF called camlpocalypse.

Running file on it gives us:

$ file camlpocalypse
camlpocalypse: ELF 64-bit LSB executable, x86-64, dynamically linked, stripped

Now normally with a rev challenge you throw the binary into IDA/Ghidra, find main, follow the flag checker, invert some transformations and boom flag.

I did not want that to happen here.

The first slightly cursed thing about this binary is that it is not purely C. There is an entire OCaml 5 native runtime embedded inside it.

If we run strings we can immediately see stuff such as:

caml_darken_cont
caml_ephe_await_key
OCAML_RUNTIME_EVENTS_DIR
OCAMLRUNPARAM
caml_plugin_header

The executable was built using something similar to:

ocamlopt -output-complete-obj

which lets us embed a complete native OCaml program inside another executable.

So the real structure of the binary is approximately:

              +-------------------+
stdin ------> | C entry / wrapper |
              +---------+---------+
                        |
                        v
              +-------------------+
              | C engine / VMs    |
              +---------+---------+
                        |
              +---------+---------+
              |                   |
              v                   v
       OCaml callbacks        Custom VMs
              |
              v
      effects / closures /
      runtime generated logic

The input representation

The challenge operates on a 48-byte candidate.

Those bytes first go through a reversible byte layer containing:

byte XOR masks
byte rotations
a permutation

After this we effectively end up with six 64-bit logical words:

state[0]
state[1]
state[2]
state[3]
state[4]
state[5]

The very important thing here is that the first five words are modified throughout the verifier while:

state[5]

acts as a carrier and remains logically preserved.

This carrier is probably the biggest foothold in the challenge.

A lot of the challenge behaviour depends on it, including what instructions are executed, what order they execute in and how several pieces of data are decoded.

Basically:

wrong candidate
     |
     v
wrong carrier
     |
     v
different program

So even dynamic analysis becomes annoying because running:

AAAAAAAAAAAAAAAA...

doesn’t necessarily show you the execution path that the real flag takes.

Candidate-dependent schedule

There are eight encoded stage identifiers.

However the bytes aren’t directly:

0 = stage A
1 = stage B
...

The runtime combines them with:

the carrier
a SplitMix-derived value
phase-specific constants

to recover the actual stage order.

The correct execution alternates between several different implementations.

There are normal C transformations, custom virtual machines and OCaml callbacks.

So the challenge becomes less:

find the check

and more:

reconstruct the reversible computation that produced the final state.

The C stages

Two stages are implemented as normal native C code.

Each contains 38 reversible ARX/MBA-like operations.

The instruction order itself is shuffled using a Fisher-Yates permutation derived from:

carrier + stage salt

The operations use things like:

addition
subtraction
XOR
rotations
odd multiplication

For example one forward instruction may conceptually perform something like:

state[d] += rol(state[a] ^ k1, r) ^ state[b] * k2

Because every operation was deliberately made reversible we can simply invert it:

state[d] -= rol(state[a] ^ k1, r) ^ state[b] * k2

Of course the actual functions contain several slightly uglier variations.

The important part is that we have to first reconstruct the permutation and then process it in reverse.

Something like:

order = recover_order(carrier, salt)

for idx in reversed(order):
    state = inverse_instruction(state, ops[idx])

The custom VMs

Then we get to VM-A and VM-B.

Before executing the VM, each byte in words 0..4 is passed through a generated 256-byte S-box.

The instructions themselves are stored as encrypted fixed-size records.

And this is where another mildly evil thing happens.

The opcode byte is not actually the opcode.

At this point even the opcode was wearing a disguise. I may have given the disassembler trust issues.

Its meaning depends on:

encrypted token
program counter
carrier
runtime state

So you cannot just recover a table like:

0x01 = ADD
0x02 = XOR
0x03 = ROL

because no global opcode table exists.

On top of this, the encrypted VM program has a checksum and that checksum participates in the arithmetic performed by every instruction.

And after executing an instruction, its record is overwritten with a new encoding.

So modifying the VM program itself also modifies the state transformations later.

Static patching therefore becomes quite annoying.

To reverse a VM stage the basic process is:

regenerate instruction semantics
        ↓
process instructions backwards
        ↓
invert arithmetic operation
        ↓
invert generated S-box

The completely useless VM

There is also another VM-like loop.

This one evolves candidate-dependent junk state and this junk state reaches the final equality.

So slicing backwards from the final comparison makes the VM appear important.

Except the value reaches both sides of the final comparison through two different masks which cancel each other out.

So after spending time reversing it you discover that it contributes exactly nothing to the actual flag recovery.

sorry :)

Then there is OCaml

The six logical words aren’t directly handed to OCaml either.

C first converts them into a separate wire representation.

Word 5 has its own reversible encoding, and once decoded it becomes the key material used to recover the other words.

The OCaml side uses a few features I specifically wanted to experiment with:

algebraic effects
deep effect handlers
captured continuations
runtime-created closures
lazy blocks
OCaml objects and method dispatch
XOR-shared values
a tiny functional AST evaluator

So instead of something friendly like:

let check x = ...

the transformation is distributed across runtime-created structures.

Each callback eventually performs another reversible transformation on the logical state and re-encodes it before returning to C.

Again, the important observation is that although the implementation is deliberately messy, the actual mathematical state transformations are reversible.

Reversing the full verifier

Once all the constants and stage information have been extracted from the ELF, the actual reversing logic becomes surprisingly small.

The final state is known, so we simply walk the correct schedule backwards:

st = target[:]

for stage in reversed(intended):
    st = inverse_stage(stage, st)

The inverse stage dispatcher roughly looks like:

invstage = {
    0: inverse_c_stage_0,
    1: inverse_vm_a,
    2: inverse_c_stage_1,
    3: inverse_vm_b,
    4: inverse_ocaml_0,
    5: inverse_ocaml_1,
    6: inverse_ocaml_2,
    7: inverse_ocaml_3,
}

After all stages have been undone we recover the initial six transformed words.

We pack them back into 48 bytes:

mixed = b''.join(
    int(x).to_bytes(8, "little")
    for x in st
)

and then reverse the original byte permutation, XOR and rotation layer:

for i in range(48):
    src[perm[i]] = ror8(mixed[i], brot[i]) ^ bmask[i]

which finally gives us:

BSides_Mumbai{C4mL_1s_the_futur3!!!!!}

So the flag for Camlpocalypse becomes:

BSides_Mumbai{C4mL_1s_the_futur3!!!!!}

The hard part of this challenge was never writing the final inverse solver. The hard part was figuring out what actually needed to be inverted, extracting the runtime-generated constants and not getting baited by all the unrelated execution paths.

2. Revenant

Category: Rev
Author: 5h1kh4r, ChatGPT

The challenge description for this one was:

IDK about the name, chatgpt suggested it. I mean the whole challenge is AI generated (except for some twists)… should be easy to solve honestly.

Very trustworthy challenge description.

Players get another stripped x86-64 ELF:

$ file revenant
revenant: ELF 64-bit LSB executable, x86-64, dynamically linked, stripped

This one uses Go + Rust.

The Go side handles the majority of the verifier while selected operations cross into a native Rust component.

The rough architecture looks like:

              candidate
                  |
                  v
        +------------------+
        | byte preprocessing|
        +--------+---------+
                 |
                 v
          six 64-bit words
                 |
                 v
      +----------------------+
      | XOR shared Go state  |
      +----------+-----------+
                 |
        12 worker goroutines
                 |
        +--------+--------+
        |                 |
        v                 v
      Go VM            Rust ABI
        |                 |
        +--------+--------+
                 |
                 v
          evolving VM key

The carrier

Just like Camlpocalypse, one word acts as a carrier.

The candidate first passes through a streaming byte transformation:

for i, ch in enumerate(flag):
    p = i % 48
    r = (i * 5 + 3) & 7
    buf[p] ^= rol8(ch ^ byte_mask(i), r)

The resulting 48 bytes are unpacked into six little-endian 64-bit words.

Word five is transformed into the carrier:

carrier = rol(word5 ^ CARR_X, 17) + CARR_ADD

and the other five words are whitened using carrier-derived values.

Again this means the candidate controls the execution environment itself.

XOR-shared state

The Go program does not normally keep a logical state word directly.

Instead there are two arrays:

A[i]
B[i]

where:

logical[i] = A[i] XOR B[i]

Between worker hops the shares are changed while the logical value remains identical.

Meaning that if you watch memory during execution you can see:

A = completely different
B = completely different

while:

A XOR B = exactly the same state

This makes normal memory tracing significantly noisier.

The worker graph

There are twelve persistent goroutines communicating using channels.

Every VM record enters a candidate-dependent first worker and can move through another one to three workers before the actual operation happens.

Each hop does four things:

reshare logical state
run decoy Go VM
call Rust decoy mixer
modify atomic shadow state

And yes, the shadow value also reaches the final check.

And yes, it cancels there.

I clearly like useless state.

The integrity seal

There is also an embedded 256-byte seal blob.

It gets hashed before the VM starts:

ih = seal_hash(seal)

and becomes part of the initial VM key:

vmkey = mix(
    carrier ^
    0x510E527FADE682D1 ^
    ih
)

This isn’t implemented as:

if (hash != expected)
    exit();

Changing the blob instead changes the interpreter state itself.

144 encrypted instructions

The verifier contains 144 encrypted records.

Each record is decrypted with a stream depending on:

carrier
round number

But even once the record is decrypted, its opcode still doesn’t have one fixed meaning.

There are four execution families:

0 -> Go native reversible operations
1 -> VM-A / Feistel operations
2 -> Rust operations
3 -> VM-B / generated S-box operations

Inside each family there are six possible semantics.

Which semantic an encrypted opcode means depends on:

carrier
current VM key
round
Rust seed output

depending on the family.

So once again:

opcode 3

might be one operation in one round and a completely different operation in another.

The VM key changes after every instruction

This is probably the central part of Revenant.

After every instruction:

vmkey_after = vmkey_before ^ mix(
    output_word ^
    ((round + 1) * constant)
)

At first this looks annoying because we need vmkey_before to interpret the opcode.

But while reversing we already know the post-instruction destination word.

So we simply rearrange it:

vmkey_before = vmkey_after ^ mix(
    post_state[d] ^
    ((round + 1) * constant)
)

This is the trick that makes the entire VM reversible.

Once vmkey_before has been recovered we can regenerate the round’s opcode permutation and figure out which operation happened.

VM-A

VM-A uses reversible operations including:

64-bit rotations
modular addition
XOR
source-dependent masks
a three-round 32-bit Feistel network

A Feistel construction is nice here because it is trivially invertible once the round function is known.

VM-B

VM-B is slightly more annoying.

Every applicable instruction generates a fresh 256-byte permutation using:

Rust seed + round constants

and applies the S-box independently to each byte of a 64-bit word.

The solver therefore generates the S-box again:

p = sbox(seed)

builds the inverse:

q = invperm(p)

and applies it while processing the instruction backwards.

Rust stage

The Rust library exports only a tiny interface conceptually similar to:

seed(carrier, phase)
mask(index)
step(state, ...)
decoy(...)

Internally the seed implementation uses two different trait-object implementations.

The Rust seed also affects VM-B opcode permutations, so unfortunately we cannot simply go:

haha Rust, ignore

and continue reversing Go.

Recovering the final state

Seven final values are XOR-shared between:

Go data
Rust-generated mask
embedded tail data

We combine them using:

target[i] = (
    go_share[i]
    ^ rust_mask(i)
    ^ tail[i]
)

giving:

state[0..5]
final_vmkey

Now we have everything necessary to reverse all 144 rounds.

The central reversing loop becomes:

for r in range(len(program) - 1, -1, -1):

    fam, enc, d, a, b, rot, mode, k1, k2, k3 = unpackrec(program[r])

    vmkey_before = vmkey ^ mix(
        state[d] ^
        ((r + 1) * 0x6A09E667F3BCC909)
    )

    apply_record(
        state,
        program[r],
        carrier,
        r,
        vmkey_before,
        reverse=True
    )

    vmkey = vmkey_before

Once all records are reversed we verify that our recovered initial VM key matches the integrity-seal-derived key.

Then we undo the whitening and input layer.

And finally:

BSides_Mumbai{g0_rus7_sh4d0w_vm_m3t4m0rph!!!!!!}

So the final flag is:

BSides_Mumbai{g0_rus7_sh4d0w_vm_m3t4m0rph!!!!!!}

This challenge was basically an experiment in making dynamic analysis as unhelpful as possible while still keeping the verifier mathematically reversible.

3. PSE ESP

Category: Rev
Author: 5h1kh4r

Do you know what microcontrollers Espressif systems produce? You do? Good. Solve this

Now this one is probably the most difficult one (according to the players) out of the 3. I intended for this to be the EASIEST one out of the 3 but it turned out otherwise.

Unlike the previous two challenges, players don’t receive one executable.

They receive:

flash.bin
efuse.bin
crash.bin
trace.bin
traffic.pcapng

The idea was that this should feel more like recovering the state of an actual embedded system instead of simply loading one program into a decompiler.

There is intentionally not enough information in any one artifact to recover the flag.

You need all of them.

The flash

The first thing to notice is the ESP-style partition table around:

0x8000

Parsing it reveals:

nvs
otadata
phy_init
factory
ota_0
ota_1
rtc_ulp
vault
jit_ir
coredump

OTA metadata tells us which application slot is current.

But the older OTA image still matters later.

This becomes important because normally you might immediately discard the outdated firmware image.

Reconstructing the eFuse material

efuse.bin contains:

device MAC
secure version
two 32-byte data blocks
CRC metadata

The bootloader contains the routine used to derive the root material.

For each byte:

root[i] =
    blockA[i]
    XOR ROL8(blockB[i], i & 7)
    XOR ((29*i + 0x53) & 0xff)

So now we can recover a 32-byte root secret used throughout the rest of the boot chain.

Looking at the PCAP

The traffic capture contains small UDP relay records beginning with:

Q3

There are three important packet types.

They leak:

device MAC
nonce A
nonce B
session counter
challenge
sync salt

We can verify the MAC against the eFuse dump which helps confirm that we parsed the packets correctly.

Decrypting trace.bin

trace.bin has a small header followed by encrypted eight-byte records.

The bootloader contains the xorshift-based stream generator used to decrypt it.

The seed combines:

eFuse root
nonce A

After decrypting, every record is:

struct {
    uint16_t delta;
    uint8_t channel;
    uint8_t edge;
    uint32_t sample;
};

The firmware folds all of these records together to create:

trace_seed

and then derives another 32 timing bytes from it.

These timing values end up infecting practically the entire remaining challenge:

NVS decryption
runtime IR
S-box generation
scheduler
VM key evolution
input permutation

So we definitely need this value correct.

NVS, except stale data matters

The flash has custom NVS3 pages with both current and stale generations.

The active page contains a 32-byte encrypted device secret.

Decrypting it depends on:

eFuse root
trace seed
CRC32(old OTA image)

Notice the last part.

Even though ota_0 is obsolete, its CRC is still cryptographically involved in the active boot state.

The stale NVS pages are also not actually useless.

They contain masked chunks of another structure we need later.

The RTC ULP

The rtc_ulp partition contains a separate RV32IMC payload.

The relevant ULP calculation consumes:

RTC words from crash.bin
nonce A
nonce B
current NVS secret
counter
trace seed

and produces:

ULP token

So now we have state being combined across:

flash
eFuse snapshot
network traffic
trace
crash dump
ULP code

which was basically the main idea behind the challenge.

Reconstructing the runtime IR

The VM instruction semantics do not exist statically.

Instead we first have to recover a tiny runtime intermediate representation.

Three 768-byte shares are hidden in:

factory partition
active OTA partition
stale NVS data

The stale NVS share must first be unmasked using:

old OTA permutation table
timing bytes
eFuse root

Then another stream is generated from:

ULP token
trace seed
eFuse root
capture challenge

and all three shares are combined.

What comes out is:

96 records × 8 bytes

of microcode IR.

There is actually a second copy of the same IR inside the jit_ir partition, but that one is:

compressed
masked
encrypted-ish

Recovering it gives us an independent check that our first IR reconstruction is correct.

Compiling the VM ISA

The active OTA image contains sixteen generic descriptor structures.

The 96 IR records mutate these structures.

Things such as:

semantic type
constants
rotate count
S-box selector
routing byte
mask

are modified or swapped.

After running all 96 records we have sixteen completely generated VM instruction descriptors.

In other words, there is no static instruction set. The firmware literally synthesizes the VM ISA at runtime.

We also calculate an IR digest which is used later.

Generating S-boxes

Using:

compiled descriptors
eFuse root
trace data
ULP token
old OTA table
IR digest

the firmware builds four 256-byte permutation S-boxes.

Some generated instructions use these tables, which means the final VM isn’t even purely ARX.

Finding the target state

The final VM state isn’t stored in the vault.

Instead it is XOR-shared across four completely different places:

active NVS raw trailer
PHY calibration region
crash.bin
trace.bin

Combining the four shares gives:

12 x uint32 state words
uint64 carrier
uint64 final VM key
uint32 scheduler state
uint32 IR digest

Before doing anything else we compare the embedded IR digest to our calculated one.

If it doesn’t match, something earlier in the reconstruction is wrong.

Finally… the vault

The vault partition contains encrypted twelve-byte VM records.

They are split into shuffled chunks, so first we reconstruct their original logical order.

There are 176 VM instructions.

Each record is decoded using the current/post-instruction VM key.

A record contains approximately:

virtual opcode
destination register
source register
rotation
immediate
scheduler symbol

But the virtual opcode still isn’t the actual instruction.

The real descriptor is selected through a sixteen-way permutation generated from:

carrier
previous VM key
previous scheduler state
trace seed
program counter

So reversing each instruction requires several steps.

Reversing one VM round

Starting at instruction 175 and moving backwards:

1. decrypt record using current key

2. recover previous scheduler/DFA state

3. recover previous VM key from
   the post-instruction destination value

4. regenerate the round's opcode permutation

5. map virtual opcode -> generated descriptor

6. inspect descriptor semantic family

7. invert operation

8. move to previous instruction

The core loop roughly becomes:

for pc in range(nins - 1, -1, -1):

    record = decrypt_record(
        bytecode[pc],
        nextkey,
        trace_seed,
        pc
    )

    idx, d, s, rot, imm, symbol = parse(record)

    prevaut = reverse_dfa(
        nextaut,
        symbol,
        trace_data,
        legacy_table,
        root
    )

    prevkey = evolve_key_backwards(
        nextkey,
        state[d],
        pc,
        nextaut
    )

    perm = opcode_permutation(
        carrier,
        prevkey,
        prevaut,
        trace_seed,
        pc
    )

    descriptor = descriptors[perm[idx & 15]]

    undo_instruction(
        state,
        d,
        s,
        descriptor,
        imm
    )

    nextkey = prevkey
    nextaut = prevaut

The generated instruction set contains eight reversible families including:

add + rotated mixing
XOR + rotated mixing
nonlinear rotate/XOR
Feistel transformations
word swaps
generated S-box substitution
old-OTA-table-dependent operations
state-dependent rotations

After reversing all 176 records we can verify the recovered initial VM key and scheduler state.

Undoing the final input layer

The remaining 48 bytes are still transformed using another input permutation involving:

trace timing
compiled descriptors
eFuse root
legacy OTA table
ULP token
IR digest

Once that is inverted we finally get:

BSides_Mumbai{r1scv_j1t_tr4ce_efus3_vault!!!!!!}

and recomputing the 64-bit carrier from the recovered flag gives the exact carrier obtained from the target shares.