Image-to-Instruction AI Prototype

I’m leading the APAC Visual Language Action initiative and need a working prototype that can look at photos of everyday objects or scenes, understand what it sees, and return clear, step-by-step instructions the user can follow. Think of a system that can glance at a kitchen counter and tell someone exactly how to prepare the ingredients in front of them, or view a cluttered workbench and explain the safest way to organise the tools.

Scope of work
The core challenge is marrying robust computer-vision object recognition with natural-language generation so the model can first reason about the scene and then translate that understanding into actionable guidance. I already have a small seed set of labelled photos but I’m open to your recommendation on augmenting or synthetically expanding the data. A lightweight web demo or callable REST endpoint will be needed to showcase end-to-end performance.

Deliverables
• End-to-end model (vision encoder + instruction generator) with reproducible training/inference code
• Inference interface (simple web app or API) that accepts an image upload and returns ordered instructions
• Documentation covering setup, model architecture, datasets used, and how to retrain or fine-tune
• Brief evaluation report with accuracy metrics and example outputs on unseen everyday-scene photos

Acceptance criteria
The system must reliably identify key objects in diverse household or outdoor scenes and output instructions that are logical, coherent, and directly relevant to the input photo. Latency under five seconds per image on a standard GPU instance is preferred.

Feel free to recommend specific frameworks—PyTorch, TensorFlow, OpenAI CLIP, BLIP-2, or any multimodal transformer you trust—as long as everything is containerised and easy to reproduce. I’m ready to start as soon as we agree on the milestones and timeline.