Real-Time Game Video Analysis and Overlay System

I’m hiring an experienced developer to build a **real-time computer-vision system for game video analysis and interactive overlays**, initially supporting NBA 2K27, Call of Duty, Fortnite, and Rainbow Six Siege.

The system will analyze a live gameplay stream externally using computer vision rather than direct game-memory access. I want a reusable CV platform with game-specific profiles, real-time detection/tracking, visual overlays, radar-style visualization, FOV filtering, timing analysis, prediction, smoothing, and detailed performance telemetry.

CORE PIPELINE

Game / PS5 / PC
↓
Remote Play / Capture Card
↓
Helios / Video Capture
↓
Live Frame
↓
CVPython / GCVWorker
↓
YOLO / Roboflow / Custom CV
↓
Detection + Classification
↓
Tracking / Filtering / FOV / Prediction
↓
Game-Specific Analysis
↓
ESP-Style Overlay / Radar / Meter Analysis
↓
GUI + Telemetry

The critical requirement is that the software processes the **actual live video stream**. I do not want simulated detections, placeholder numbers, or a GUI that only reports that the system is running.

NBA 2K27 METER ANALYSIS

One of the main components is a CV system that can recognize and analyze the NBA 2K27 shooting meter from live gameplay.

It should be able to:

• Detect when the meter appears
• Locate the meter
• Track its position/state frame by frame
• Identify the relevant release/green region
• Detect timing/state transitions
• Handle different meter presentations where possible
• Handle different resolutions/scaling
• Provide live timing telemetry
• Record detection results for testing

The architecture should allow additional meter styles and visual configurations to be added later.

SHOOTER VISUAL ANALYSIS

For Call of Duty, Fortnite, Rainbow Six Siege, and future supported games, the CV platform should support:

• Player detection
• Head detection
• Bounding boxes
• Target markers
• Radar-style visualization
• FOV visualization
• Object/class detection
• Target tracking
• Target prioritization
• Prediction
• Smoothing
• Confidence filtering

The exact classes and behavior should be controlled through game profiles and model configuration.

ESP-STYLE OVERLAY

The overlay should visualize actual CV results in real time.

Possible information includes:

• Player boxes
• Head points
• Object markers
• Confidence
• Tracking state
• FOV boundaries
• Target/crosshair relationship
• Other model-provided information

The rendering system should remain lightweight and update independently from inference whenever possible.

RADAR

Radar should use the same underlying detection and tracking data rather than running a duplicate detector.

It should provide a simplified 2D representation of detected objects/players and allow different coordinate interpretations for different games.

TRACKING

Detection determines where an object is in one frame.

Tracking determines whether detections across multiple frames represent the same object.

The tracker should reduce:

• Jitter
• Flickering
• Target switching
• Temporary detection loss
• Unstable positions

Tracking parameters should be configurable and independent from the detection model.

FOV / FILTERING

The system should support runtime-adjustable FOV and filtering.

Useful parameters include:

• FOV size
• Confidence threshold
• Detection classes
• Target priority
• Head/body priority
• Tracking persistence
• Prediction
• Smoothing

FOV should determine which detections are considered relevant around the screen center.

PREDICTION / SMOOTHING

Prediction may estimate future object position from recent movement.

Example:

Frame 1: X=800
Frame 2: X=810
Frame 3: X=820

Estimated next position ≈ X=830.

Prediction must be configurable.

Smoothing should reduce visual jitter while allowing the user to balance stability versus responsiveness.

REAL-TIME FRAME PROCESSING

A major requirement is a **latest-frame architecture**.

For example, if the capture stream is 60 FPS but inference is 30 FPS, the application should not process an ever-growing queue of old frames.

Preferred design:

New frame arrives
→ replace previous unprocessed frame
→ process newest frame
→ discard stale frame

This keeps the analysis synchronized with current gameplay.

Measure:

• Capture FPS
• Inference FPS
• Tracking FPS
• GUI FPS
• Inference time
• Frame age
• Dropped frames
• Processing latency
• Detection confidence

At 60 FPS the frame interval is approximately 16.67 ms; at 120 FPS it is approximately 8.33 ms.

LATENCY

The system should measure actual CV latency instead of displaying arbitrary values.

The developer should distinguish CV processing latency from the total video/input path, which may include:

Controller
→ Console
→ Remote Play/network
→ Encoding
→ Decoding
→ Capture
→ CV
→ Tracking
→ Overlay
→ Display

I want real measured telemetry for each stage where practical.

HELIOS / CVPYTHON / GCVWORKER

Expected integration:

Helios Frame
→ CVPython
→ GCVWorker
→ CV Processing
→ Model
→ Results

The developer must verify that the worker is genuinely receiving and processing live frames.

The application should detect and report:

• CV runtime startup failure
• GCVWorker failure
• Capture disconnect
• Invalid frame
• Model initialization failure
• Inference stall
• Missing dependency
• Network/API failure where applicable

YOLO / ROBOFLOW

Support YOLO and/or Roboflow models/workflows.

Possible classes include:

• Player
• Head
• Body
• NPC
• Vehicle
• Meter
• Game-specific objects

Game-specific models should be supported where they provide better results.

For Roboflow Workflows, use the actual Workflow definition and output schema rather than assuming field names or response formats.

Pipeline:

Live Frame
→ Preprocess
→ Model/Workflow
→ Parse
→ Filter
→ Track
→ Game-Specific Analysis
→ Visualization

IMAGE PROCESSING

Support:

Capture
→ ROI/crop
→ Resize
→ Color conversion
→ Normalization
→ Inference
→ Postprocessing

ROI should be configurable where it improves performance.

Input sizes such as 640×640 and 960×960 should be benchmarked for the target hardware.

GPU OPTIMIZATION

Where supported, evaluate:

• NVIDIA CUDA
• FP16
• TensorRT
• GPU preprocessing
• Asynchronous inference
• Efficient memory transfer
• CPU/GPU synchronization

Use measured benchmarks rather than unnecessary optimization.

GUI

Create a modern Windows GUI that controls the real system.

Display:

• Game/profile
• Capture status
• CV status
• Model status
• Detection status
• FPS
• Inference time
• Frame age
• Detection count
• FOV
• Confidence
• Tracking
• Prediction/smoothing
• Meter state/timing
• Overlay settings
• Radar settings
• Logs/errors

Every control must actually affect the live application.

GAME PROFILES

Support separate configurable profiles for:

• NBA 2K27
• Call of Duty
• Fortnite
• Rainbow Six Siege
• Future games

A profile should define:

• Model/Workflow
• Detection classes
• Capture settings
• Resolution
• Confidence
• FOV
• Tracking
• Prediction
• Smoothing
• Overlay settings
• Radar settings
• Game-specific analysis

Adding another game should require a new profile/model rather than rewriting the core CV engine.

RELIABILITY

Detect and recover where practical from:

• Capture disconnect
• Helios failure
• CVPython failure
• GCVWorker crash
• Model failure
• Workflow/API failure
• Invalid frames
• Inference stalls

All failures should be timestamped and logged.

CODE REQUIREMENTS

• Modular architecture
• No hard-coded Windows/user paths
• No embedded secrets/API keys
• Configuration/environment variables
• Clear module boundaries
• Error handling
• Logging
• Type hints where practical
• Build instructions
• PyInstaller support where practical
• Documentation for adding future games/models

DELIVERABLES

1. Complete source code
2. Working live CV pipeline
3. Helios → CVPython/GCVWorker integration
4. NBA 2K27 meter detection/timing analysis
5. YOLO/Roboflow integration
6. Player/head/object detection
7. Real-time overlay
8. Radar visualization
9. Tracking/FOV/filtering
10. Prediction/smoothing
11. Live telemetry/logging
12. Modern Windows GUI
13. Modular game profiles
14. Buildable Windows release
15. Setup/troubleshooting documentation
16. Developer/API documentation
17. Performance benchmark/report
18. Demonstration using real gameplay footage

ACCEPTANCE CRITERIA

The system is complete when:

• Actual live gameplay frames are processed
• GCVWorker/CVPython is genuinely executing
• Real model inference is occurring
• NBA 2K27 meter detection works on live gameplay
• Shooter object/player detection works on live gameplay
• Overlay/radar use real detection data
• Tracking operates across consecutive frames
• FOV and configuration settings affect live processing
• Telemetry reports real measurements
• Stale-frame accumulation is prevented
• Runtime failures are detected and logged
• The application remains stable during extended testing
• Performance is benchmarked
• Additional game profiles can be added without rewriting the core engine

APPLICANTS

Please apply only if you have practical experience with:

• Real-time computer vision
• YOLO / Roboflow
• Python / C++
• Windows video capture
• CVPython / GCVWorker-style runtimes
• GPU inference optimization
• Object tracking
• Overlay rendering
• Low-latency video processing
• Game-specific CV systems

Please provide examples of comparable systems you personally developed and explain which components you implemented.