Data Engineer Needed to Organise & Clean 4.8M Records Across 230 Excel Files

Data Engineer Needed to Organise & Clean 4.8M Records Across 230 Excel Files

I have a large business dataset containing approximately 4.8 million records spread across 230 Excel files.

The data has been collected from multiple sources and is currently not consistently organised. I need an experienced freelancer/data engineer to consolidate, clean, standardise and structure the entire dataset so that it can be reliably used for further processing, filtering, analysis and database import.

Current Situation

* ~4.8M total records
* 230 Excel files
* Multiple files/sources
* Different column structures and naming conventions
* Duplicate records are expected
* Missing fields are expected
* Formatting is inconsistent
* Some records may have incomplete or malformed information
* Data needs to be preserved accurately — no accidental record loss

Scope of Work

You will be responsible for:

1. Audit the existing files

* Analyse all 230 Excel files
* Identify different structures/formats
* Identify columns that represent the same information under different names
* Identify inconsistencies between files
* Provide an initial data-quality report

2. Consolidate the data

* Combine the 230 files into a structured master dataset
* Map equivalent columns into standard fields
* Preserve useful source information
* Maintain source/file tracking where possible

3. Standardise the data

Standardise fields such as:

* Company name
* First name / Last name
* Email
* Telephone
* Website
* Street / Address
* PLZ / Postal Code
* City
* State/Region
* Country
* Industry/category
* Source
* Any other relevant fields discovered during the audit

Examples of standardisation may include:

* Consistent telephone formatting
* Consistent country codes
* Correct postal-code formatting
* Normalised company names
* Consistent casing
* Removing unnecessary whitespace
* Standardising URLs
* Identifying malformed emails
* Handling empty/null values consistently

4. Deduplication

Identify and remove/flag duplicates using appropriate matching logic.

This should not rely only on exact company-name matching.

Where appropriate, use combinations such as:

* Email
* Telephone
* Website/domain
* Company name + address
* Company name + city
* Other reliable identifiers

I want both:

Clean dataset
and
Duplicate/removed records dataset

so that no data is permanently lost.

5. Data quality classification

Create clear classifications such as:

* Complete records
* Records with email
* Records without email
* Records with website
* Records missing website
* Invalid emails
* Invalid/duplicate phone numbers
* Missing company information
* Potential duplicates
* Unusable records

The exact classification can be finalised after the initial audit.

6. Final structured output

The final dataset should be delivered in a format that can easily be:

* Imported into a database
* Processed using Python
* Filtered by different criteria
* Used for future deduplication
* Used for enrichment
* Exported into smaller Excel/CSV files when required

Because of the size of the dataset, do not assume that Excel should be the final storage format. If CSV, Parquet, SQLite, PostgreSQL, MySQL or another format is more appropriate, recommend it.

Important Requirements

I am not looking for someone who simply opens the Excel files and manually combines them.

You should be comfortable with:

* Python
* Pandas / Polars
* Large datasets
* Excel/CSV processing
* Data cleaning
* Data normalisation
* Deduplication
* Data validation
* Database-ready data structures

Experience handling millions of records is strongly preferred.

Data Integrity Is Critical

The most important requirement is:

Do not lose data accidentally.

Before making destructive changes, the process should create backups/intermediate outputs and maintain counts.

For example:

Original records → Cleaned → Deduplicated → Validated

At every stage, I want record counts and a summary of what changed.

Deliverables

I expect:

1. Master cleaned dataset
2. Duplicate records dataset
3. Invalid/problematic records dataset
4. Data-quality report
5. Column mapping / schema documentation
6. Processing script/code
7. Processing logs/statistics
8. Instructions for running the process again on future files

The code should be reusable because I may receive additional datasets in the future.

Before Hiring

Please provide:

* Examples of large datasets you have processed
* Maximum dataset size you have handled
* Your proposed technical approach
* Whether you would use Pandas, Polars, SQL, or another approach
* How you would prevent data loss
* Estimated timeline
* Fixed project price or milestone-based pricing

Do not quote purely based on the number of Excel files. The dataset contains approximately 4.8M records, so processing volume and data quality are the main considerations.

I can provide a small sample of the files to the shortlisted freelancer before starting.

Budget: Open to reasonable proposals based on experience and proposed approach.

Please start your proposal with: 4.8M DATA so I know you have read the complete requirement.