Skip to main content

Merge ans deduplicate domains in a blocklist

Use WGET to download lists, then combine them into a single large file, and finally create a new file with no duplicates by using “awk '!visited[$0]++'” 

wget URL1 URL2 URL3 cat *.txt > all.txt (This overwrites all.txt)

awk '!visited[$0]++' all.txt > no_duplicates.txt

Or use chatgptlol