Explicit song lyrics checker

Question

To stay in practice with my Python I've decided to write an explicit song checker. It checks each word in the song against an array of 14 explicit words contained in the text file. I've decided not to include the words that are to be checked against, for I am not sure about the rule of having explicit words in a program's code. Feedback on efficiency and structure is what I'm going for mostly. Style and other critiques are invited as well.

explicit_song_checker.py

explicit_words = [
    #explicit words not shown 
]

def isClean(song_path):
    with open(song_path) as song:
        for line in song:
            words = line.split(" ")
            for word in words:
                for explicit_word in explicit_words:
                    if word == explicit_word:
                        return False

        return True

def main():
    song = raw_input("Enter path to song: \n")
    if isClean(song):
        print("CLEAN")
    else:
        print("EXPLICIT")

if __name__ == '__main__':
    main()

Answer 1

I recommend practicing Python 3 rather than Python 2 these days.

According to PEP 8, isClean() should be is_clean().

One or more of your loops could be replaced by some use of the any() function. Note that this suggests that an is_explicit() function would be a more natural concept than an is_clean() function.

I would expect song lyrics to contain a mixture of uppercase and lowercase. Words may be delimited by punctuation as well as spaces. Therefore, words = line.split(" ") is probably too naïve. Furthermore, I would expect each song to be short enough to fit entirely in memory very comfortably, so processing each file line by line is an unnecessary complication.

I would rewrite the program to use a regular expression instead.

import re

explicit_words = [
    …
]

is_explicit = re.compile(
    r'\b(?:' +
        '|'.join(re.escape(w) for w in explicit_words) +
    r')\b',
    re.IGNORECASE).search

def main():
    with open(raw_input("Enter path to song: ")) as song:
        print("EXPLICIT" if is_explicit(song.read()) else "CLEAN")

if __name__ == '__main__':
    main()

The way I chose to write the is_explicit() function above might show too much Haskell influence for some Python programmers' taste. Here's a more conventional way of writing it:

def is_explicit(text):
    return re.search(
        r'\b(?:' +
            '|'.join(re.escape(w) for w in explicit_words) +
        r')\b',
        text,
        re.IGNORECASE
    )

Answer 2

Splitting line into words using words = line.split(" ") will only split the line on space characters. If two words are separated by a tab character, or other white space character other than a space, the split will not happen at that point, which may confuse your explicit checker.

Using words = line.split() would split on spaces, tab characters, and any other white space characters.

You might also want to investigate splitting using the word break regular expression \b, which would treat non-word characters as break points, so that a line containing an explicit word immediately followed by a comma (such as "explicit,") or period, or semicolon, etc. won't sneak through the filter.

---

            for explicit_word in explicit_words:
                if word == explicit_word:
                    return False

Python has great searching built into it. Instead of explicitly writing a loop over all explicit words, you could just ask Python to tell you if a word is in the list of explicit words:

            if word in explicit_words:
                return False

Or better, use a set() of explicit words, and the in operation drops from \$O(n)\$ to \$O(1)\$ time complexity:

explicit_words = {
    "explicit", "bad", "ugly", "offensive", "words" 
}

# ...

            if word in explicit_words:
                return False

---

We've improved things a little, but we still have an explicit (pun intended) loop over the list of words in a line. We can use the any() method, and a list comprehension, to get Python to do the looping for us, which should be faster.

    for line in song:
        words = line.split()
        if any(word in explicit_words for word in words):
            return False

If word in explicits returns True for any of the word in the list words, the any() method will "short-circuit" and immediately return True without examining the remaining items in words.

---

If "explicit" is a bad word, and the song lyrics contains "Explicit" or "EXPLICIT", you wouldn't want to mark the song as "CLEAN", right? Perhaps you want word.lower() in explicit_words? As @wizzwizz4 pointed out, .casefold() is better than .lower().

---

Resulting method:

explicit_words = set(explicit_word.casefold()
    for explicit_word in ("explicit", "bad", "ugly", "offensive", "words"))

def isClean(song_path):
    with open(song_path) as song:
        for line in song:
            words = line.split()
            if any(word.casefold() in explicit_words for word in words):
                return False    
        return True

Answer 3

Some suggestions:

• Using argparse to parse arguments is preferable to any kind of interactive input, because it means that
  • it's much easier to automate and integrate in other scripts,
  • you could trivially support passing multiple songs in one command,
  • it's what any experienced shell user would expect, and
  • if the user adds --help or doesn't pass any arguments they will get a synopsis for the command.
• The list of explicit words should not be part of the code, because it is configuration and is very likely to change often. You'll want to put words in a separate file and read them in during initialization. You could very easily make this an optional argument (with argparse) so that you could do fancy stuff like ./explicit_song_checker.py --words=./words.txt --words=./other.txt ./kill-the-queen.txt ./bulls-on-parade.txt
• black will automatically format your code to be pretty idiomatic, such as using two empty lines between top-level functions.
• flake8 will check for style issues. I believe it will assume that you're writing Python 3, and so will complain about the use of raw_input (called input in Python 3).
• mypy can be used to enforce type hints, which is a super helpful way to make your code clearer without resorting to comments.

Answer 4

• To simplify your input you could change word == explicit_word to explicit_word in word.

  This can lead to the Scunthorpe problem, and so from here you can check if the word only contains prefixes and suffixes.

  Given the amount of common prefixes alone, it's unlikely you'd catch all variations using a blacklist alone. It would also catch gibberish/semigibberish that you may not have thought of - "Antioverfucked" / "Overantifucked".

• Your code misses words without spaces "I don't give afuck" and so you may want to perform a 'similarity check'.

  This could also match thinks like "sexx".

• Your code doesn't reasonably handle ZWSP and other forms of Unicode abuse.

  >>> print('exp\u200blitive')
  exp​litive
  >>> 'exp\u200blitive' == 'explitive'
  False
  >>> print('exp\u1DD3litive')
  expᷓlitive
  

• Your code doesn't reasonably detect leet speak.

• Your code doesn't reasonably detect creative Unicode character selections.

  >>> print('\u2131\u01DA\u0255\u212A')
  ℱǚɕK
  

• Your code doesn't reasonably handle phonetic abuse.

  "Phuck" is pronounced "Fuck".

  "Bend Dover" sounds like "bend over".

  "I CUP" spelt out is "I see you pee".

  "See you next Tuesday" each starting phonetic spells "CU...".

---

When I say doesn't reasonably, I mean if someone combined the above to get something ridiculous like "4n7ipᷓhǚ​ɕK" your word list would have to be ridiculous in size to match it.

It should also be fairly obvious that someone probably will find a way to bypass your check no matter what you do.

Practicality vs Perfection

Given how widespread leet speak is you should handle that at a bare minimum. This is fairly easy, you just 'translate' from this alphabet. And perform the checks you're already doing.

You should then probably use a 'similar check' to match "sexx" or "afuck". This is as duplicating a letter or missing a space is readable and easy.
This check may result in problems as "Scunthorpe" could be understood as "s cunt thorp e" so would be split into two two known words and two extra letters, and then the similarity check my come back low enough to say it's a swearword.

From here ZWSP and other unicode abuse probably are the next most common. And with things like deasciiify it's only getting easier.

Answer 5

explicit_words = [
    #explicit words not shown 
]

Instead of embedding the word list within the program, you might want to treat it as configuration, and load it from a data file. The advantage of this is that it can allow the use of alternative configurations by command-line option (e.g. use a list of French obscenities for French-language lyrics).

---

On efficiency: we're only interested in the set of words in the lyrics. Given that many songs have repeated lyrics, then it's probably worth reducing the lyrics to a set of words before comparing that set with the banned words (use & for set intersection), or set.isdisjoint() for a potentially faster test.

Answer 6

The code feedback is already done wonderfully. I'd like to write about some pointers that might be important, apart from the leet speak:

• The word Phoque , is French for seal. There are numerous examples of words which are perfectly meaningful in some language but if pronounced in English, might be considered offensive. You might need to do a phonetic similarity check
• The context can be taken into account. For example, the word 'blow' or 'mom' need not be offensive. It becomes offensive only when it is surrounded by contextual words that make it offensive, ('job', etc). In these cases, you might need to come up with an algorithm that outputs a probability score and filter songs based on that.