NOTE: When cracking WPA/WPA2 passwords, make sure you check gpuhash.me first incase it's already been processed.

Home - General Discussion - HUGE word-lists duplicate remover and merge tool

WARNING!
Due to the number of SCAMS going on in the PAID forum, PLEASE ask an ADMIN or MODERATOR to verify ALL found passwords to ensure you are not being SCAMMED.
DO NOT PAY until an ADMIN or MOD has verified them for you!


243 Results - Page 3 of 9 -
1 2 3 4 5 6 7 8 9
Author Message
Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Fri, 17 Oct 2014 @ 14:33:13

Ah, nice to see you around again h0wler


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Tue, 21 Oct 2014 @ 10:16:22

WARNING! User is BANNED and maybe a SCAMMER.

blandyuk i have a suggestion
instead of writing all files name


remove duplicate from all from files in a folder

i means if i have 50 wordlist & i want to remove duplicate from them

i have to write names of all 50 files
so instead of this just select the folder which contain all wordlist


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Tue, 21 Oct 2014 @ 10:23:37

You can already specify a folder already supported as shown in initial post:

App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

Try it...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Fri, 24 Oct 2014 @ 01:42:27

G:\Cracking\Wordlists\best>App.Merge.exe o="output-file.txt" t=8 min=4 max=20 "output_duperemoved.txt"

I used that and i ended up with

http://gyazo.com/c49c484d80f692cdf27a9aee1af376fc

Why are 20+ length still in the list O.o

nvm seems the left ones are hex converted characters


2 x R9 290

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Fri, 24 Oct 2014 @ 15:43:20

Yes $HEX[...] are encoded. This format is supported by hashcat so will not cause it a problem.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Fri, 24 Oct 2014 @ 16:09:27

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
Milzo
Administrator
Status: Elite
Joined: Sat, 29 Dec 2012
Posts: 3176
Team:
Reputation: 5009 Reputation
Offline
Fri, 24 Oct 2014 @ 16:22:36

h0wler said:

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?


Ditto
https://forum.hashkiller.co.uk/topic-view.aspx?t=6644&m=45746#45746


1CrqbgYU63zfLjwKVagyiTYP9XGMgyFAVm

Forum Rules
https://i-disclose.net/ (Open for Registration, Limited Time)
Discord - Milzo#6567

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Sun, 26 Oct 2014 @ 15:56:28

WARNING! User is BANNED and maybe a SCAMMER.

hey does this tool support non english words ?



Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Sun, 26 Oct 2014 @ 23:54:35

h0wler said:

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?

bump


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Tue, 28 Oct 2014 @ 05:50:04

WARNING! User is BANNED and maybe a SCAMMER.

yhi said:

hey does this tool support non english words ?



bump


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Tue, 28 Oct 2014 @ 07:44:11

h0wler

I support $HEX[...] so np there.

yhi

Yes, it supports non-English words.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Tue, 28 Oct 2014 @ 11:29:20

blandyuk said:

h0wler

I support $HEX[...] so np there.

yhi

Yes, it supports non-English words.

purrrrfect


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
alotdv

Status: Trusted
Joined: Thu, 10 May 2012
Posts: 1353
Team:
Reputation: 1478 Reputation
Offline
Fri, 30 Jan 2015 @ 19:19:25

I got error when trying with file 3.6Gb:

Code:
Unhandled Exception:
System.IO.IOException: Reading more than 2GB with this call is not supported
  at System.IO.File.ReadAllBytes (System.String path) [0x00000] in (filename unknown):0 
  at App.Merge.Program.sortFile (System.Object o) [0x00000] in (filename unknown):0 
  at System.Threading.Thread.StartInternal () [0x00000] in (filename unknown):0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Reading more than 2GB with this call is not supported
  at System.IO.File.ReadAllBytes (System.String path) [0x00000] in (filename unknown):0 
  at App.Merge.Program.sortFile (System.Object o) [0x00000] in (filename unknown):0 
  at System.Threading.Thread.StartInternal () [0x00000] in (filename unknown):0


Team Hashcat

BTC: 16s8xjDmAxvt2JtGKZbdrFdbZko8hjtw9D
bitcoincash: qrzu70gg2he89s6l2gd2v4ypjfw3gc24lgjuaqqxq3

XMPP: alotdv@xmpp.jp

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Sun, 08 Feb 2015 @ 17:41:39

I hate to say this but sort64lm, part of the Unified List Manager utils does a ~150MB file in 20 seconds.

I actually tried sorting a 90GB file with this and it did it on 6 hours...


I'm on a shitty laptop too.

Sorry for bashing your work blandy :P


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Sun, 08 Feb 2015 @ 22:11:22

Haha yes, I've not looked at this for quite some time and I need to revisit it. I've got an awesome idea for doing this now and it will be a lot faster

I'll post the update here soon...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Sun, 08 Feb 2015 @ 23:46:50

blandyuk said:

Haha yes, I've not looked at this for quite some time and I need to revisit it. I've got an awesome idea for doing this now and it will be a lot faster

I'll post the update here soon...

Ah great, thanks for that.


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Mon, 09 Feb 2015 @ 00:04:19

OK, I've been busy lol. It's insane fast now so give v0.4 a go:

http://home.btconnect.com/md5decrypter/App.Merge.zip

Let me know results...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Mon, 09 Feb 2015 @ 00:07:08

OK 169MB word-list:

Code:
Merge complete to: C:\Temp\passwords.txt
Total words  : 15851422
Words skipped: 0
Duplicates removed: 0
Total time: 13.988623 seconds
Press any key...

Fast enough?


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Mon, 09 Feb 2015 @ 09:47:37

blandyuk said:

OK 169MB word-list:

Code:
Merge complete to: C:\Temp\passwords.txt
Total words  : 15851422
Words skipped: 0
Duplicates removed: 0
Total time: 13.988623 seconds
Press any key...

Fast enough?

Damn, you did that quick. And yeah, fast enough haha Thanks.


Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Mon, 09 Feb 2015 @ 10:52:45

Merging files into a 14GB wordlist, 50 minutes... not bad.


Merge complete to: compiled.txt
Total words : 1267042082
Words skipped: 55
Duplicates removed: 467395
Total time: 2965.8396822 seconds
Press any key...


Avatar
Sph1nx

Status: n/a
Joined: Wed, 11 Feb 2015
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Wed, 11 Feb 2015 @ 14:49:54

Just want to ask what is more effective format in wpa-wordlist. The text written in horizontal or vertical ones.
Horizontal that fiil up every space:

sidiefifgfghgghgjhhmgmmmmmmmmmmmjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
vfvfvdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddfvfvfvfv
xddefe1229rfdivjfbglnlhiycdyweriktmgbgblgnoh679u6yblblblght9h9ty945yo5ytkooityh

Vertical:

sasiee
rtrt6y6
sdr4rr4
e3e33
bnnjj
cxsfprgo
cdododifor

I know horizontal text can be read by aircrack as sidie, fifgf, etc.



Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Sat, 14 Feb 2015 @ 11:34:33

Merge complete to: KSKF_CLEAN
Total words : 4435297543
Words skipped: 19
Duplicates removed: 330649
Total time: 3 hrs 11 mins 46.794 secs

Not bad for a 47GB list


2 x R9 290

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Sat, 14 Feb 2015 @ 14:58:37

It's still in BETA so still have some coding to do yet


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Sun, 15 Feb 2015 @ 23:20:37

Is there anyway we can dupe remove multiple wordlists without having to merge them all , I normally run my wordlist attacks in order so my list i run first are just primarily cracked hashes which gets great results. Makes things easier when hitting hard algo's


2 x R9 290

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 14:52:05

I think there is a problem in the program because if I run twice or more continues to remove duplicate entries. For example:
AppMerge.exe o="output2.txt" "wordlist.txt"

AppMerge.exe o="output3.txt" "output2.txt"



Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Sun, 22 Feb 2015 @ 15:03:34

Yes, it's the way it processes the files but have just uploaded a new version so try it now.

I run it on two word-lists which were around the 2.8GB mark each. Worked OK.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 15:07:38

Thank you, but where I can get the new version?

http://home.btconnect.com/md5decrypter/App.Merge.zip

Not Found [CFN #0005]


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3093
Team: HashKiller
Reputation: 4140 Reputation
Offline
Sun, 22 Feb 2015 @ 15:30:44

Ah! I uploaded a RAR file. I have sorted so try it again...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 15:37:10

Thank you again!
Now it's working fine

Merge complete to: output5.txt
Total words : 12705321
Words skipped: 0
Duplicates removed: 0
Total time: 0 hrs 0 mins 32.932 secs


Avatar
megazoid

Status: n/a
Joined: Sun, 18 Jan 2015
Posts: 13
Team:
Reputation: 0 Reputation
Offline
Mon, 23 Feb 2015 @ 00:55:17

Merge complete to: password.txt
Total words : 1298208425
Words skipped: 259
Duplicates removed: 80025921
Total time: 7 hrs 31 mins 35.545 secs
took a while but worked a treat


Btc = 1KE7ovy85wyzJNFX7rrii6H2PReGAxNkA4


243 Results - Page 3 of 9 -
1 2 3 4 5 6 7 8 9

We have a total of 170534 messages in 21359 topics.
We have a total of 19372 registered users.
Our newest registered member is SeveralAbility.