Removing duplicates without sorting

Content Authoring
1

(imported topic written by SystemAdmin)

Is it possible to remove duplicates without automatically sorting the data?

“unique values” and the string set properties automatically sort.

2

(imported comment written by cstoneba)

You could use the BigFix Computer Remover utility. This will remove duplicate computers.

http://support.bigfix.com/bes/install/downloadutility.html#bescomputerremove

3

(imported comment written by SystemAdmin)

I am speaking about removing duplicates from a substring. When you use unique values, it automatically sorts the data. It looks like the elements function of the sets does the same thing.

For example (unique values of it):

tiger

bear

tiger

I want my result to be:

tiger

bear

I don’t want:

bear

tiger

4

(imported comment written by JasonO91)

The only way that I know of would be to use a custom perl or awk script to accomplish this. The logic behind unique values implies a sort or else it will not remove all duplicates.

You don’t give any examples of what you’re trying to accomplish, but I think you’ll need to send the “output” to a file and then execute a script against the output for unique values, and then you can use the values as you wish.

awk is available natively on unix machines, and you can download a version for Windows.

awk ’ !x

$0

++’ filetosort

Jason

5

(imported comment written by SystemAdmin)

I am working with relay affiliation doing a mapping of subnets to sites. In some cases people have multiple adapters, which result in multiple sites/regions. These sites/regions can be the same or different depending on subnet. So if I use unique values I lose the ordering I am trying to achieve. The order is accomplished by using the higher CIDR number as preferred.

So I may have a list

site1, site2, site3, region1, region2, region3

I need to do this duplicate removal in relevance.

6

(imported comment written by BenKus)

Hey Kevin,

Both the “unique value” and the “set” inspector implicitly sort the values alphabetically… but you shouldn’t rely on the order of the results returned in relevance as a general rule… Can you post the relevance you are working with and maybe we can help you find a better way…

Ben

7

(imported comment written by SystemAdmin)

Hello Ben,

The relevance code is putting the our site codes and regions in a specific order based on high CIDR subnet notation taking precedence (this is for relay affiliation). Example /24 takes precedence over /16. Here is an example of a relevance result where I would like to remove the duplicates, but keep the order. Site code first, then region second.

il99-dmz;il97;il97;americas;americas;americas;FAILOVERS;*

If you would like to see the relevance, I would probably need to setup a time to go over it with you, plus I would not want to post it to the forum.

  • Kevin
8

(imported comment written by BenKus)

Hey Kevin,

I can’t see any reasonable mechanism to do this except by writing this list to a file and using a simple vbscript to process it to remove duplicates…

Ben

9

(imported comment written by SystemAdmin)

Thank you Ben… Can we throw in an enhancement request for “unique values” so that we can have the option of sorting vs. non sorting?

10

(imported comment written by NoahSalzman)

FWIW, that enhancement request is already in our bug tracker as issue #10283.