1. Project Overview

The Data Redundancy Removal System is a Python command-line application that solves a critical data quality problem: when new records are being added to a cloud database, how do you ensure no duplicates, near-duplicates, or false positives contaminate the dataset?

The system reads incoming CSV data, validates each record against existing database records using two detection methods (hash-based and similarity-based), classifies each record into one of four categories, and only inserts records that are genuinely new and unique.

2. Objectives

• Design a system that identifies and classifies data as redundant or false positive
• Implement a validation mechanism to check new data against existing data
• Prevent duplicate data from being added into the cloud database
• Append only unique and verified data entries to the database
• Ensure database accuracy and efficiency by removing or avoiding redundancy

3. System Architecture

Input: new_customers.csv
          |
          v
[VALIDATION ENGINE]
          |
          |-- Step 1: Generate SHA-256 hash (email + phone)
          |-- Step 2: Check hash against database
          |-- Step 3: Check email against database
          |-- Step 4: Calculate name/email/phone similarity scores
          |-- Step 5: Classify the record
          |
          v
[CLASSIFICATION]
          |
    UNIQUE -----> INSERT into database
    FALSE_POSITIVE --> INSERT into database
    EXACT_DUPLICATE --> REJECT + LOG
    NEAR_DUPLICATE --> REJECT + LOG
          |
          v
[REPORT GENERATOR]
          |
          v
Output: reports/redundancy_report_TIMESTAMP.csv

4. Tech Stack

5. Phase 1 — VS Code Project Setup

Goal: Create the full project folder structure in VS Code.

5.1 Open VS Code terminal