Numeric Data Cleaning Automation

I have several datasets coming in from different systems where the numeric fields are stored as plain ASCII files in some cases and UTF-8 exports in others. My goal is to run solid, reproducible data analysis on them, but first they need to be cleaned and standardised.

Here is what I need from you:

• Build a repeatable cleaning pipeline in Python (Pandas, NumPy) that detects the file encoding, converts everything to UTF-8 where necessary, and validates that all numeric columns are truly numeric—no stray characters, misplaced separators, or locale-specific quirks.
• Where the raw files already sit in SQL Server, create T-SQL or Python-based routines that achieve the same checks and corrections in-database so I can schedule them in an Agent job.
• Document each step clearly so I can port the logic later to R or even a small C/C++ utility if performance becomes critical.

Acceptance criteria
– The script ingests a folder full of mixed-encoding files and leaves a clean UTF-8 version plus a detailed log of any corrections made.
– A sample table in SQL Server is fully cleansed using your routine, with before/after counts matching exactly.
– Code is delivered via Git with concise README and in-line comments.

If you are comfortable working with numeric data, encoding issues, and the usual Python + SQL toolkit, I’d love to see how you’d approach this.