Python Beginner Help: Removing Duplicate Words from Lists

[Kuma]

Homework Statement

I have a simple question with python.

I have a text file that's been converted into a bunch of lists like this.
[blah, blah]
[blah, blah, blah]
[a, a, d]
[g,x,d,s,a]

etc..

now i want to loop over all of these lists and append all the words into a new list, but there can't be any duplicate words allowed at all. so for example, there are 3 a's in the above text, only one is allowed in. Same goes for the blah's.

Any ideas? Thanks.

Homework Equations

The Attempt at a Solution

Tried using a for loop but it only does the first line.

 

[iamalexalright]

well, (efficiently) checking if there are duplicates in a list/array is an interesting problem...

there are a few ways to do it but you could:

have two loops:
1 that goes over each list in order to append to the resultant list
1 that goes over the resultant list in order to see if the item to be appended is already needed

 

[Kuma]

thanks, but do you know how i can set up the loop so it can loop over EVERY list. my loop only does it for the first list and doesn't progress downwards.

 

[story645]

thanks, but do you know how i can set up the loop so it can loop over EVERY list. my loop only does it for the first list and doesn't progress downwards.

What's your code like?

iamalexalright has the right solution:
one list comprehension over all your lists, and a second list comprehension to check if the list you're currently looking at is in the list you're building.

 

[LudusRex]

I would recommend using the built-in function "set" in Python to remove duplicate words from lists. This function automatically removes duplicates while preserving the order of the words in the list. Here is an example code that you can try:

# create a list of lists
lists = [["blah", "blah"], ["blah", "blah", "blah"], ["a", "a", "d"], ["g", "x", "d", "s", "a"]]

# create an empty list to store the unique words
unique_words = []

# loop through each list in the main list
for sublist in lists:
# use the "extend" function to add all the words in the sublist to the unique_words list
unique_words.extend(sublist)

# use the "set" function to remove duplicates and convert the list back to a list
unique_words = list(set(unique_words))

# print the unique words
print(unique_words)

Output: ['g', 'blah', 'd', 'a', 's', 'x']

You can also use the "remove" function to remove duplicates from a list without using the "set" function, but it will require more code and may not preserve the original order of the words. I hope this helps!