Skip to main content

Posts

Showing posts with the label data locality

How to remove duplicates from data?

Removing Duplicates from Production Data in Real-Time Using SQL Handling duplicates in production data requires efficient strategies to maintain data integrity and avoid system performance issues. Here’s a structured approach to achieve this:   --- ### **1. Prevention: Use Unique Constraints**   The best way to deal with duplicates is to prevent them. Ensure your database schema is designed to enforce uniqueness:   - **Primary Key**: Define a primary key to prevent identical rows.   - **Unique Constraints**: Apply unique constraints to columns or combinations of columns that should not contain duplicate values.   **Example:**   ```sql ALTER TABLE my_table  ADD CONSTRAINT unique_constraint_name UNIQUE (column1, column2); ``` --- ### **2. Identifying Duplicates**   Before removing duplicates, identify them using `GROUP BY` and `HAVING`:   **Example:**   ```sql SELECT column1, column2, COUNT(*) AS duplicate_count FROM my_table GROUP BY colu...

Data Locality in Hadoop

What is data Locality in Hadoop? We can say that, Data locality improves the overall execution of the system and makes Hadoop faster. It reduces the network congestion. There are two benefits of data Locality in Hadoop. i. Faster Execution In data locality, the program is moved to the node where data resides instead of moving large data to the node, this makes Hadoop faster. Because the size of the program is always lesser than the size of data, so moving data is a bottleneck of network transfer. ii. High Throughput Data locality increases the overall throughput of the system. Hope this helps.