AI-Controlled XYZ Plotter for Touchscreen
Budget / Salary$1,500–3,000
TypeFreelance project
LocationRemote
Posted1 hour ago
AI-Controlled XYZ Writing Plotter + Camera System
Project Goal
I am looking for a developer to build a reliable, plug-and-play system that allows an AI/LLM running on a Windows PC to control an XYZ writing/drawing plotter and view a tablet through a camera.
The goal is for the plotter to use a capacitive stylus to physically interact with a tablet touchscreen.
The system should be capable of:
Tapping specific locations
Swiping
Selecting buttons/icons
Typing using an on-screen keyboard
The intended workflow is:
User Command → AI/LLM → Camera Observation → Plotter Action → Camera Verification
I should be able to give the AI a normal-language command. The AI should interpret the command, view the tablet, determine the necessary actions, physically perform those actions using the plotter, and verify the result through the camera.
Planned Hardware
The planned hardware is:
XYZ writing/drawing plotter
ESP32-S3 Sense camera
Windows 10/11 PC
Capacitive stylus
This is the planned configuration, not an absolute restriction.
If different or additional equipment would make the system more accurate, reliable, or easier to implement, I am open to discussing recommendations before the hardware is finalized.
Any hardware changes or additional purchases should be discussed and approved before purchase or implementation.
Hardware and shipping costs will be paid separately from the development cost.
XYZ Plotter Control
The Windows PC should be able to send high-level commands to the plotter, including:
Move to a position
Stylus up/down
Tap
Swipe
Home
The touchscreen and plotter should be calibrated so positions on the screen can be translated into accurate plotter movements.
The system should include appropriate homing and movement limits so the plotter can reliably establish its position and remain within its intended working area.
The plotter may use its normal USB connection for operation.
If practical, I would also like a Wi-Fi control option so the plotter can receive commands wirelessly from the PC/server. The exact method can be determined based on the selected hardware.
Reliability is more important than requiring a specific wireless implementation.
Camera & Mounting
The ESP32-S3 Sense camera should be securely mounted to the plotter/system in a fixed position with a clear view of the tablet touchscreen and working area.
The mounting position should remain consistent so camera-to-screen and plotter calibration remains accurate.
The camera should:
Connect to Wi-Fi automatically
Automatically discover/connect to the PC server
Provide JPEG snapshots on request
Automatically reconnect if Wi-Fi or the server connection is lost
The AI should be able to request images whenever needed to observe the screen and verify its actions.
Windows PC Server
Create a Windows application/server that provides the communication and control layer between the AI/LLM, plotter, and camera.
The server should expose simple controls for the plotter and camera and store required configuration/calibration settings.
Settings should automatically restore when the system restarts.
AI / LLM Control
The purpose of the AI/LLM integration is to allow me to give the AI normal-language commands and have the AI interpret and physically carry out those commands using the plotter.
For example:
“Open the browser and search for something.”
The AI should be able to:
Receive and interpret the command.
Request an image from the camera to see the tablet screen.
Determine what needs to be tapped, swiped, or typed.
Send the appropriate commands to the plotter.
Request another camera image to see the result.
Continue taking actions as needed to complete the command.
An existing AI/LLM may be used. The developer has discretion to select the AI/LLM and integration method they believe is best suited for the project.
The developer does not need to build or train an AI model from scratch.
The developer's responsibility is to build and integrate the software and hardware control system necessary for the AI/LLM to receive a command, view the tablet through the camera, and control the plotter to carry out that command.
The completed system must demonstrate the full AI-controlled workflow, not only manual control of the plotter from the PC.
Plug-and-Play Operation
After the initial setup and calibration, normal operation should be as automatic as possible:
Power On → Devices Connect → PC Detects Devices → Ready
I do not want to manually enter IP addresses or reconfigure devices each time the system starts.
Required firmware/software should already be installed and configured.
Normal use should not require reflashing or programming the devices.
Future Expansion
The architecture should allow additional compatible plotters and cameras to be added later.
Ideally:
Install/Flash Firmware → Power On → Automatically Discovered → Ready
Each device should have its own identity so the PC/AI can determine which plotter and camera it is communicating with.
The completed system only needs to implement and test one plotter and one camera, but the core software should be designed so additional compatible devices can be added later without rebuilding the system.
Physical Development & Testing
The developer must be able to physically work with and test the hardware.
The agreed hardware will be provided or purchased for the project and shipped to the developer.
The developer will:
Assemble/integrate the system as needed
Securely mount the camera
Install and configure the required firmware/software
Calibrate the plotter, camera, and touchscreen
Test the camera and plotter together
Demonstrate reliable touchscreen tapping
Demonstrate touchscreen swiping
Demonstrate interaction with an on-screen keyboard
Demonstrate AI/LLM command interpretation and execution
Test automatic startup/reconnection behavior
Perform final system testing
After completion, the tested hardware should be securely packaged and shipped back to me.
Deliverables
Completed and tested physical system, including the plotter, mounted camera, and Windows PC used for the project
XYZ plotter control software/firmware
ESP32-S3 Sense camera firmware
Windows PC server/application
Working AI/LLM integration
Plotter and camera control API/interface
Automatic device discovery/reconnection
Touchscreen/camera/plotter calibration
Source code and compiled firmware/software
Buildable project files
Saved configuration/calibration
Basic setup and usage instructions
Video demonstration of the completed system
Remote final test before completion
Completed/tested hardware securely packaged and shipped back to me
End Goal
The end goal is a reliable, plug-and-play physical system that allows an AI to receive a normal-language command, interpret it, see a touchscreen, and physically interact with the touchscreen using a stylus to carry out the command.
The completed system should be stable, accurate, and practical for regular use, while allowing additional compatible devices to be added later.
Project Goal
I am looking for a developer to build a reliable, plug-and-play system that allows an AI/LLM running on a Windows PC to control an XYZ writing/drawing plotter and view a tablet through a camera.
The goal is for the plotter to use a capacitive stylus to physically interact with a tablet touchscreen.
The system should be capable of:
Tapping specific locations
Swiping
Selecting buttons/icons
Typing using an on-screen keyboard
The intended workflow is:
User Command → AI/LLM → Camera Observation → Plotter Action → Camera Verification
I should be able to give the AI a normal-language command. The AI should interpret the command, view the tablet, determine the necessary actions, physically perform those actions using the plotter, and verify the result through the camera.
Planned Hardware
The planned hardware is:
XYZ writing/drawing plotter
ESP32-S3 Sense camera
Windows 10/11 PC
Capacitive stylus
This is the planned configuration, not an absolute restriction.
If different or additional equipment would make the system more accurate, reliable, or easier to implement, I am open to discussing recommendations before the hardware is finalized.
Any hardware changes or additional purchases should be discussed and approved before purchase or implementation.
Hardware and shipping costs will be paid separately from the development cost.
XYZ Plotter Control
The Windows PC should be able to send high-level commands to the plotter, including:
Move to a position
Stylus up/down
Tap
Swipe
Home
The touchscreen and plotter should be calibrated so positions on the screen can be translated into accurate plotter movements.
The system should include appropriate homing and movement limits so the plotter can reliably establish its position and remain within its intended working area.
The plotter may use its normal USB connection for operation.
If practical, I would also like a Wi-Fi control option so the plotter can receive commands wirelessly from the PC/server. The exact method can be determined based on the selected hardware.
Reliability is more important than requiring a specific wireless implementation.
Camera & Mounting
The ESP32-S3 Sense camera should be securely mounted to the plotter/system in a fixed position with a clear view of the tablet touchscreen and working area.
The mounting position should remain consistent so camera-to-screen and plotter calibration remains accurate.
The camera should:
Connect to Wi-Fi automatically
Automatically discover/connect to the PC server
Provide JPEG snapshots on request
Automatically reconnect if Wi-Fi or the server connection is lost
The AI should be able to request images whenever needed to observe the screen and verify its actions.
Windows PC Server
Create a Windows application/server that provides the communication and control layer between the AI/LLM, plotter, and camera.
The server should expose simple controls for the plotter and camera and store required configuration/calibration settings.
Settings should automatically restore when the system restarts.
AI / LLM Control
The purpose of the AI/LLM integration is to allow me to give the AI normal-language commands and have the AI interpret and physically carry out those commands using the plotter.
For example:
“Open the browser and search for something.”
The AI should be able to:
Receive and interpret the command.
Request an image from the camera to see the tablet screen.
Determine what needs to be tapped, swiped, or typed.
Send the appropriate commands to the plotter.
Request another camera image to see the result.
Continue taking actions as needed to complete the command.
An existing AI/LLM may be used. The developer has discretion to select the AI/LLM and integration method they believe is best suited for the project.
The developer does not need to build or train an AI model from scratch.
The developer's responsibility is to build and integrate the software and hardware control system necessary for the AI/LLM to receive a command, view the tablet through the camera, and control the plotter to carry out that command.
The completed system must demonstrate the full AI-controlled workflow, not only manual control of the plotter from the PC.
Plug-and-Play Operation
After the initial setup and calibration, normal operation should be as automatic as possible:
Power On → Devices Connect → PC Detects Devices → Ready
I do not want to manually enter IP addresses or reconfigure devices each time the system starts.
Required firmware/software should already be installed and configured.
Normal use should not require reflashing or programming the devices.
Future Expansion
The architecture should allow additional compatible plotters and cameras to be added later.
Ideally:
Install/Flash Firmware → Power On → Automatically Discovered → Ready
Each device should have its own identity so the PC/AI can determine which plotter and camera it is communicating with.
The completed system only needs to implement and test one plotter and one camera, but the core software should be designed so additional compatible devices can be added later without rebuilding the system.
Physical Development & Testing
The developer must be able to physically work with and test the hardware.
The agreed hardware will be provided or purchased for the project and shipped to the developer.
The developer will:
Assemble/integrate the system as needed
Securely mount the camera
Install and configure the required firmware/software
Calibrate the plotter, camera, and touchscreen
Test the camera and plotter together
Demonstrate reliable touchscreen tapping
Demonstrate touchscreen swiping
Demonstrate interaction with an on-screen keyboard
Demonstrate AI/LLM command interpretation and execution
Test automatic startup/reconnection behavior
Perform final system testing
After completion, the tested hardware should be securely packaged and shipped back to me.
Deliverables
Completed and tested physical system, including the plotter, mounted camera, and Windows PC used for the project
XYZ plotter control software/firmware
ESP32-S3 Sense camera firmware
Windows PC server/application
Working AI/LLM integration
Plotter and camera control API/interface
Automatic device discovery/reconnection
Touchscreen/camera/plotter calibration
Source code and compiled firmware/software
Buildable project files
Saved configuration/calibration
Basic setup and usage instructions
Video demonstration of the completed system
Remote final test before completion
Completed/tested hardware securely packaged and shipped back to me
End Goal
The end goal is a reliable, plug-and-play physical system that allows an AI to receive a normal-language command, interpret it, see a touchscreen, and physically interact with the touchscreen using a stylus to carry out the command.
The completed system should be stable, accurate, and practical for regular use, while allowing additional compatible devices to be added later.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.