AWS SageMaker Algorithm Accuracy Validation

I need rigorous algorithm-level accuracy tests run on my existing AI model and want the entire validation workflow built and executed inside AWS, with Amazon SageMaker at the core. The model is already trained; what’s missing is a repeatable validation pipeline that produces clear accuracy metrics, confidence intervals and an exportable report I can hand to stakeholders.

Here’s how I picture the engagement flowing:

• You spin up or reuse SageMaker resources, import the current model artefact from my S3 bucket and design a validation script (Python preferred) that measures precision, recall, F1 and any additional metrics you recommend.
• The job must finish with an automated notebook or processing job that I can trigger again whenever the dataset updates, plus a concise HTML/PDF summary generated at the end of each run.

Optional but welcome is guidance on linking the output to a Lambda function or dashboard; however, the immediate priority is the SageMaker-based accuracy testing itself. If you normally use Azure ML, that background could be helpful later, yet for this phase everything stays within AWS.

Please tell me about similar validation pipelines you have already built and your typical turnaround time. Once we agree on the evaluation methodology, I’ll provide the model binary and a sample of the labelled validation set so you can get started straight away.