Skip to content

fix(ww3d2): Prevent undefined behavior when writing dynamic vertices and reading screenshot pixels - #3671

Merged
xezon merged 1 commit into
TheSuperHackers:mainfrom
CryoTheRenegade:refactor/vertex-buffer-casts
Nov 16, 2026
Merged

xezon merged 1 commit into
TheSuperHackers:mainfrom
CryoTheRenegade:refactor/vertex-buffer-casts

Conversation

@CryoTheRenegade

@CryoTheRenegade CryoTheRenegade commented Nov 10, 2026 •

Copy link
Copy Markdown

Follow-up to #3670, from xezon's "Are there more of these?" on #3580. This collects the last pointer casts of this kind in the renderers and the screenshot writer into one PR.

  • SegLineRendererClass::Render and StreakRendererClass::RenderStreak cast the locked dynamic vertex buffer to unsigned char*, then write each vertex through Vector3*, unsigned int* and Vector2* casts at offsets taken from FVFInfoClass. The dynamic vertex buffer always has the dynamic_fvf_type format, and Get_Formatted_Vertex_Array already returns it as VertexFormatXYZNDUV2* and asserts that format. The loops now write the fields of that struct, as dazzle.cpp and dynamesh.cpp already do:

    -				Vector3 *vertex = reinterpret_cast<Vector3 *>(vb + verticesOffset);
    -				vertex->X = vArray[i].x;
    ...
    -				*reinterpret_cast<unsigned int *>(vb + diffuseOffset) = vArray[i].diffuse;
    ...
    -				vb += vbSize;
    +				vb[i].x = vArray[i].x;
    ...
    +				vb[i].diffuse = vArray[i].diffuse;
    +				vb[i].u1 = vArray[i].u1;
    +				vb[i].v1 = vArray[i].v1;
  • screenshotThreadFunc reads the copied surface, a new unsigned char[] array, through const unsigned int* and const unsigned short* casts. Reading unsigned char objects as unsigned int is undefined behavior. Each pixel is now copied out with memcpy.

The reinterpret_cast<long*> in dx8webbrowser.cpp stays. It hands the device pointer to the BrowserEngine COM interface, whose Initialize is declared with a long* parameter, and nothing in this code reads through it.

Evidence

  • Disassembly: I compared seglinerenderer.obj, streakRender.obj and W3DScreenshot.obj for both games against a build of main. In every changed function, the float arithmetic and the call targets are the same as on main.
    • VS 2026 (ClangCL) Release: in Render and RenderStreak, the vertex stores use fixed struct offsets instead of the FVFInfoClass offsets. In screenshotThreadFunc, the pixel loops load the color bytes directly, so they use fewer shifts and masks.
    • VC6 Release: Render and RenderStreak lose the offset locals, so their stack frames shrink by 4 bytes. That changes the x87 scheduling of the transform math before the vertex loop: the same multiplications and additions, in a different register order, with one operand kept on the x87 stack instead of reloaded from memory. This is render-only code, so it does not affect game logic. screenshotThreadFunc is unchanged, because the 4- and 2-byte memcpy compile to the same loads.

AI Disclosure

Claude Opus 5.5 was used to aid in implementation and to compare the disassembly.

… pointer casts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Screenshot conversion now reads 32-bit and 16-bit pixels with memcpy. Two rendering paths now assign vertex position, color, and UV data through named fields in VertexFormatXYZNDUV2.

Changes

Pixel and Vertex Data Access

Layer / File(s) Summary
Screenshot pixel reads
Core/GameEngineDevice/Source/W3DDevice/GameClient/W3DScreenshot.cpp
The 32-bit and 16-bit conversion loops read pixel values from byte-addressed rows with memcpy. RGB extraction remains unchanged.
Typed vertex assignments
Core/Libraries/Source/WWVegas/WW3D2/seglinerenderer.cpp, Core/Libraries/Source/WWVegas/WW3D2/streakRender.cpp
Both rendering paths assign vertex position, diffuse color, and UV values through VertexFormatXYZNDUV2 fields instead of FVF-offset-based byte writes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested labels: Rendering, Fix, Stability, Minor, Platform

Suggested reviewers: bobtista, xezon

Merge Risk: Minimal · up to ddcf7

No actionable merge-blocking risk was identified in the pixel and vertex access changes.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the changes to dynamic vertex writes and screenshot pixel reads.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description explains the changes to dynamic vertex writes and screenshot pixel reads, and provides relevant implementation and testing context.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 124b4b5e-fd35-440d-a2e4-b7d31594ceef
Commits

Reviewing files that changed from the base of the PR and between 998f337 and ddcf743.

Files selected for processing: 3

  • Core/GameEngineDevice/Source/W3DDevice/GameClient/W3DScreenshot.cpp
  • Core/Libraries/Source/WWVegas/WW3D2/seglinerenderer.cpp
  • Core/Libraries/Source/WWVegas/WW3D2/streakRender.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

@CryoTheRenegade
CryoTheRenegade marked this pull request as ready for review November 12, 2026 23:47
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T23:50:17.320757Z ddcf743 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Replaces pointer casts with typed vertex writes and byte-safe screenshot reads.

  • Line and streak renderers write vertices through their typed fields.
  • Screenshot pixels are copied from byte storage before conversion.

Reviews (1) · Last reviewed commit: "fix(ww3d2): Write dynamic vertices and r..." · Reviewed by Greptile

@xezon xezon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@xezon xezon added Minor Severity: Minor < Major < Critical < Blocker Fix Is fixing something, but is not user facing labels Nov 15, 2026
@xezon
xezon merged commit f1cf26d into TheSuperHackers:main Nov 16, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Fix Is fixing something, but is not user facing Minor Severity: Minor < Major < Critical < Blocker

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants