Wordlists and Wordlist manipulation - Part 2

Edit
====
A wordlist revisited post post can be found here ;
http://adaywithtape.blogspot.com/2011/07/wordlist-manipulation-revisited.html



Some tools for creating / editting wordlists;
Crunch
======
Crunch is a pretty easy yet powerful dictionary generator with general usage;
pentest/password/crunch [from length] [to length] [charset] > filename.txt

If you want a 4 character password list with just numbers the code is;
/pentest/passward/crunch 4 4 0123456789 > pass1.txt
If you want a 6 character password list with lowercase and numbers, the code is;
/pentest/password/crunch 6 6 abcdefghijklmnopqrstuvwxyz0123456789 > pass2.txt

You can also fix parts of the passwords; if for instance you are think the password will always start off with for instance "pass" followed by numbers, you can use crunch to do the work for you.
/pentest/password/crunch 8 8 0123456789 -t pass@@@@ > password.txt



That still results in a file with 10000 possible combinations though.. can check the number of lines with ;
cat password.txt | wc -l




The syntax for crunch gets slightly more complicated when dealing with special characters.
If for instance you wanted to make a five character wordlist with all possible special characters, you would need to 'escape' certain special characters using backslash \

If you wanted to fix certain characters, using the -t function, then again, you would need to escape certain characters, ie ;

/pentest/password/crunch 5 5 "\`\~\!@#$%^&*()-_=+[{]};:'\"\|,<.>/?" -t "@@\"\\@"


Using SED
========
Sed is short for StreamEditor, and although extremely powerful.. not easy to use and definately too complicated for me.. So herewith just an example;

You can copy the contents of a webpage with a simple 'select all' and 'copy', paste this into a txt file, save txt file (web.txt) ;





Transform a space into a new line;
sed 'se[[:space:]]e\neg' -i web.txt

Remove empty lines;
sed '/^$/d' -i web.txt

Then sort alphabetically and exclude duplicates;
cat web.txt | sort | uniq > web_sorted_uniq.txt



So with just a Ctrl + A, Copy & Paste and 3 lines of code you have a wordlist of all words on
a specific webpage.
Obviously some websites are better suited for this than others, however it is still a quick and dirty way to get a decently focussed wordlist and you can then clean it up further with sed commands and password inspector (see lower down in the post).

To remove any periods from the front of the words;
sed 's/^[.]//' -i web.txt

To remove any comma from the end of the words;
sed 's/[,]$//' -i web.txt

Appending characters to each word in wordlist (such as '123');
sed 's/$/123/' wordlist.txt > wordlist123.txt

To delete lines in file containing certain character (containing "?");
sed '/?/ d' -i wordlist.txt

or to create a new file with those changes;
sed '/?/ d' wordlist.txt > wordlist1.txt


Some good information on SED usage can be found here.


TR
==
The 'tr' command is handy as well, for instance to change upper to lower case or vice versa;
tr [:upper:] [:lower:] <> wordlist_lower.txt

Information can be found here.


Using Wget & Wyd
==============
This is much more refined way of getting words from a website, even going down several layers in the website.

First we make a folder and move to it;
mkdir tr
cd tr

Then we start wget to grap all from a site, specifying how deep we want to go (-l)
wget -r -l 1 -nd http://www.theregister.co.uk

Then to go to wyd and use it to extract all words from the downloaded files.
cd /pentest/password/wyd
perl wyd.pl -n -o ~/wordlistTR.txt ~/TR/

Head back to root
cd ~/
cat wordlistTR.txt | sort | uniq > TR_sorted_uniq.txt

So now we have a txt file with all words from the 1st level of theregister.co.uk in alphabetical order without duplicates.

Its handy to remember that the 'sort' function bases the sorting on the order as defined in the ASCII table and so will sort ABCabc instead of AaBbCc.
To get a real alphabetical sorting order, use the -f command;

cat wordlistTR.txt | sort -f | uniq > TR_sorted_uniq.txt


Password Inspector
===============You can use Password inspector to tidy up wordlist files based on minimum and maximum password length and which character set you want it to contain.

cat TR_sorted_uniq.txt | pw-inspector -m 4 -M 15 > TR_optimised.txt


So the above some ways to get wordlists and how to manipulate them to your liking !