Remove duplicates from an array of objects in JavaScript

Remove duplicates from an array of objects in JavaScript

I have an object that contains an array of objects.
things = new Object();

things.thing = new Array();

things.thing.push({place:”here”,name:”stuff”});
things.thing.push({place:”there”,name:”morestuff”});
things.thing.push({place:”there”,name:”morestuff”});

I’m wondering what is the best method to remove duplicate objects from an array. So for example, things.thing would become…
{place:”here”,name:”stuff”},
{place:”there”,name:”morestuff”}

Solutions/Answers:

Solution 1:

Let’s see … a primitive one would be:

var obj = {};

for ( var i=0, len=things.thing.length; i < len; i++ )
    obj[things.thing[i]['place']] = things.thing[i];

things.thing = new Array();
for ( var key in obj )
    things.thing.push(obj[key]);

Ok, I think that should do the trick. Check it out, Travis.

EDIT
Edited the code to correctly reference the place (former id) property .

Solution 2:

How about with some es6 magic?

things.thing = things.thing.filter((thing, index, self) =>
  index === self.findIndex((t) => (
    t.place === thing.place && t.name === thing.name
  ))
)

Reference URL

A more generic solution would be:

const uniqueArray = things.thing.filter((thing,index) => {
  return index === things.thing.findIndex(obj => {
    return JSON.stringify(obj) === JSON.stringify(thing);
  });
});

Stackblitz Example

Solution 3:

If you can use Javascript libraries such as underscore or lodash, I recommend having a look at _.uniq function in their libraries. From lodash:

_.uniq(array, [isSorted=false], [callback=_.identity], [thisArg])

Basically, you pass in the array that in here is an object literal and you pass in the attribute that you want to remove duplicates with in the original data array, like this:

var data = [{'name': 'Amir', 'surname': 'Rahnama'}, {'name': 'Amir', 'surname': 'Stevens'}];
var non_duplidated_data = _.uniq(data, 'name'); 

UPDATE: Lodash now has introduced a .uniqBy as well.

Related:  Passing parameters on JQuery .trigger

Solution 4:

I had this exact same requirement, to remove duplicate objects in a array, based on duplicates on a single field. I found the code here: Javascript: Remove Duplicates from Array of Objects

So in my example, I’m removing any object from the array that has a duplicate licenseNum string value.

var arrayWithDuplicates = [
    {"type":"LICENSE", "licenseNum": "12345", state:"NV"},
    {"type":"LICENSE", "licenseNum": "A7846", state:"CA"},
    {"type":"LICENSE", "licenseNum": "12345", state:"OR"},
    {"type":"LICENSE", "licenseNum": "10849", state:"CA"},
    {"type":"LICENSE", "licenseNum": "B7037", state:"WA"},
    {"type":"LICENSE", "licenseNum": "12345", state:"NM"}
];

function removeDuplicates(originalArray, prop) {
     var newArray = [];
     var lookupObject  = {};

     for(var i in originalArray) {
        lookupObject[originalArray[i][prop]] = originalArray[i];
     }

     for(i in lookupObject) {
         newArray.push(lookupObject[i]);
     }
      return newArray;
 }

var uniqueArray = removeDuplicates(arrayWithDuplicates, "licenseNum");
console.log("uniqueArray is: " + JSON.stringify(uniqueArray));

The results:

uniqueArray is:

[{"type":"LICENSE","licenseNum":"10849","state":"CA"},
{"type":"LICENSE","licenseNum":"12345","state":"NM"},
{"type":"LICENSE","licenseNum":"A7846","state":"CA"},
{"type":"LICENSE","licenseNum":"B7037","state":"WA"}]

Solution 5:

One liner using Set

var things = new Object();

things.thing = new Array();

things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});

// assign things.thing to myData for brevity
var myData = things.thing;

things.thing = Array.from(new Set(myData.map(JSON.stringify))).map(JSON.parse);

console.log(things.thing)

Explanation:

  1. new Set(myData.map(JSON.stringify)) creates a Set object using the stringified myData elements.
  2. Set object will ensure that every element is unique.
  3. Then I create an array based on the elements of the created set using Array.from.
  4. Finally, I use JSON.parse to convert stringified element back to an object.
Related:  Is there a version of setTimeout that returns an ES6 promise?

Solution 6:

If you can wait to eliminate the duplicates until after all the additions, the typical approach is to first sort the array and then eliminate duplicates. The sorting avoids the N * N approach of scanning the array for each element as you walk through them.

The “eliminate duplicates” function is usually called unique or uniq. Some existing implementations may combine the two steps, e.g., prototype’s uniq

This post has few ideas to try (and some to avoid 🙂 ) if your library doesn’t already have one! Personally I find this one the most straight forward:

    function unique(a){
        a.sort();
        for(var i = 1; i < a.length; ){
            if(a[i-1] == a[i]){
                a.splice(i, 1);
            } else {
                i++;
            }
        }
        return a;
    }  

    // Provide your own comparison
    function unique(a, compareFunc){
        a.sort( compareFunc );
        for(var i = 1; i < a.length; ){
            if( compareFunc(a[i-1], a[i]) === 0){
                a.splice(i, 1);
            } else {
                i++;
            }
        }
        return a;
    }