Home - General Discussion - HUGE word-lists duplicate remover and merge tool


215 Results - Page 3 of 8 -
1 2 3 4 5 6 7 8
Author Message
Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Fri, 17 Oct 2014 @ 14:33:13

Ah, nice to see you around again h0wler


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Tue, 21 Oct 2014 @ 10:16:22

WARNING! User is BANNED and maybe a SCAMMER.

blandyuk i have a suggestion
instead of writing all files name


remove duplicate from all from files in a folder

i means if i have 50 wordlist & i want to remove duplicate from them

i have to write names of all 50 files
so instead of this just select the folder which contain all wordlist


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Tue, 21 Oct 2014 @ 10:23:37

You can already specify a folder already supported as shown in initial post:

App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

Try it...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Fri, 24 Oct 2014 @ 01:42:27

G:\Cracking\Wordlists\best>App.Merge.exe o="output-file.txt" t=8 min=4 max=20 "output_duperemoved.txt"

I used that and i ended up with

http://gyazo.com/c49c484d80f692cdf27a9aee1af376fc

Why are 20+ length still in the list O.o

nvm seems the left ones are hex converted characters


2 x R9 290

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Fri, 24 Oct 2014 @ 15:43:20

Yes $HEX[...] are encoded. This format is supported by hashcat so will not cause it a problem.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Fri, 24 Oct 2014 @ 16:09:27

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
Milzo
Administrator
Status: Elite
Joined: Sat, 29 Dec 2012
Posts: 2889
Team:
Reputation: 4443 Reputation
Offline
Fri, 24 Oct 2014 @ 16:22:36

h0wler said:

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?


Ditto
https://forum.hashkiller.co.uk/topic-view.aspx?t=6644&m=45746#45746


Secure Paste - https://i-disclose.net/paste/ | Forum Rules
BTC - 12MpH1WGdbyzbVNsonXYgv8qawwDs6zxJF
XMPP - milzo@xmpp.jp

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Sun, 26 Oct 2014 @ 15:56:28

WARNING! User is BANNED and maybe a SCAMMER.

hey does this tool support non english words ?



Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Sun, 26 Oct 2014 @ 23:54:35

h0wler said:

Speaking $HEX are you converting them when we upload them to the found areas? Or do I need to convert them back to ascii?

bump


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
yhi

Status: Banned
Joined: Tue, 05 Nov 2013
Posts: 875
Team:
Reputation: 365 Reputation
Offline
Tue, 28 Oct 2014 @ 05:50:04

WARNING! User is BANNED and maybe a SCAMMER.

yhi said:

hey does this tool support non english words ?



bump


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Tue, 28 Oct 2014 @ 07:44:11

h0wler

I support $HEX[...] so np there.

yhi

Yes, it supports non-English words.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
h0wler

Status: Elite
Joined: Tue, 08 Nov 2011
Posts: 309
Team:
Reputation: 257 Reputation
Offline
Tue, 28 Oct 2014 @ 11:29:20

blandyuk said:

h0wler

I support $HEX[...] so np there.

yhi

Yes, it supports non-English words.

purrrrfect


BTC: 16c3rG8EwyNXHDKtCWPtediC3NrVUQhu7M

Avatar
alotdv

Status: Trusted
Joined: Thu, 10 May 2012
Posts: 1268
Team:
Reputation: 1387 Reputation
Offline
Fri, 30 Jan 2015 @ 19:19:25

I got error when trying with file 3.6Gb:

Code:
Unhandled Exception:
System.IO.IOException: Reading more than 2GB with this call is not supported
  at System.IO.File.ReadAllBytes (System.String path) [0x00000] in (filename unknown):0 
  at App.Merge.Program.sortFile (System.Object o) [0x00000] in (filename unknown):0 
  at System.Threading.Thread.StartInternal () [0x00000] in (filename unknown):0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Reading more than 2GB with this call is not supported
  at System.IO.File.ReadAllBytes (System.String path) [0x00000] in (filename unknown):0 
  at App.Merge.Program.sortFile (System.Object o) [0x00000] in (filename unknown):0 
  at System.Threading.Thread.StartInternal () [0x00000] in (filename unknown):0


Team Hashcat

BTC: 16s8xjDmAxvt2JtGKZbdrFdbZko8hjtw9D
bitcoincash: qrzu70gg2he89s6l2gd2v4ypjfw3gc24lgjuaqqxq3

XMPP: alotdv@xmpp.jp

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Sun, 08 Feb 2015 @ 17:41:39

I hate to say this but sort64lm, part of the Unified List Manager utils does a ~150MB file in 20 seconds.

I actually tried sorting a 90GB file with this and it did it on 6 hours...


I'm on a shitty laptop too.

Sorry for bashing your work blandy :P


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Sun, 08 Feb 2015 @ 22:11:22

Haha yes, I've not looked at this for quite some time and I need to revisit it. I've got an awesome idea for doing this now and it will be a lot faster

I'll post the update here soon...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Sun, 08 Feb 2015 @ 23:46:50

blandyuk said:

Haha yes, I've not looked at this for quite some time and I need to revisit it. I've got an awesome idea for doing this now and it will be a lot faster

I'll post the update here soon...

Ah great, thanks for that.


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Mon, 09 Feb 2015 @ 00:04:19

OK, I've been busy lol. It's insane fast now so give v0.4 a go:

http://home.btconnect.com/md5decrypter/App.Merge.zip

Let me know results...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Mon, 09 Feb 2015 @ 00:07:08

OK 169MB word-list:

Code:
Merge complete to: C:\Temp\passwords.txt
Total words  : 15851422
Words skipped: 0
Duplicates removed: 0
Total time: 13.988623 seconds
Press any key...

Fast enough?


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Mon, 09 Feb 2015 @ 09:47:37

blandyuk said:

OK 169MB word-list:

Code:
Merge complete to: C:\Temp\passwords.txt
Total words  : 15851422
Words skipped: 0
Duplicates removed: 0
Total time: 13.988623 seconds
Press any key...

Fast enough?

Damn, you did that quick. And yeah, fast enough haha Thanks.


Avatar
Jake

Status: n/a
Joined: Sat, 06 Dec 2014
Posts: 24
Team:
Reputation: 0 Reputation
Offline
Mon, 09 Feb 2015 @ 10:52:45

Merging files into a 14GB wordlist, 50 minutes... not bad.


Merge complete to: compiled.txt
Total words : 1267042082
Words skipped: 55
Duplicates removed: 467395
Total time: 2965.8396822 seconds
Press any key...


Avatar
Sph1nx

Status: n/a
Joined: Wed, 11 Feb 2015
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Wed, 11 Feb 2015 @ 14:49:54

Just want to ask what is more effective format in wpa-wordlist. The text written in horizontal or vertical ones.
Horizontal that fiil up every space:

sidiefifgfghgghgjhhmgmmmmmmmmmmmjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
vfvfvdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddfvfvfvfv
xddefe1229rfdivjfbglnlhiycdyweriktmgbgblgnoh679u6yblblblght9h9ty945yo5ytkooityh

Vertical:

sasiee
rtrt6y6
sdr4rr4
e3e33
bnnjj
cxsfprgo
cdododifor

I know horizontal text can be read by aircrack as sidie, fifgf, etc.



Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Sat, 14 Feb 2015 @ 11:34:33

Merge complete to: KSKF_CLEAN
Total words : 4435297543
Words skipped: 19
Duplicates removed: 330649
Total time: 3 hrs 11 mins 46.794 secs

Not bad for a 47GB list


2 x R9 290

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Sat, 14 Feb 2015 @ 14:58:37

It's still in BETA so still have some coding to do yet


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
Lust

Status: Dumper
Joined: Thu, 11 Sep 2014
Posts: 72
Team:
Reputation: 110 Reputation
Offline
Sun, 15 Feb 2015 @ 23:20:37

Is there anyway we can dupe remove multiple wordlists without having to merge them all , I normally run my wordlist attacks in order so my list i run first are just primarily cracked hashes which gets great results. Makes things easier when hitting hard algo's


2 x R9 290

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 14:52:05

I think there is a problem in the program because if I run twice or more continues to remove duplicate entries. For example:
AppMerge.exe o="output2.txt" "wordlist.txt"

AppMerge.exe o="output3.txt" "output2.txt"



Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Sun, 22 Feb 2015 @ 15:03:34

Yes, it's the way it processes the files but have just uploaded a new version so try it now.

I run it on two word-lists which were around the 2.8GB mark each. Worked OK.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 15:07:38

Thank you, but where I can get the new version?

http://home.btconnect.com/md5decrypter/App.Merge.zip

Not Found [CFN #0005]


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2916
Team: HashKiller
Reputation: 3911 Reputation
Offline
Sun, 22 Feb 2015 @ 15:30:44

Ah! I uploaded a RAR file. I have sorted so try it again...


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
guardiao

Status: n/a
Joined: Sat, 04 Oct 2014
Posts: 3
Team:
Reputation: 0 Reputation
Offline
Sun, 22 Feb 2015 @ 15:37:10

Thank you again!
Now it's working fine

Merge complete to: output5.txt
Total words : 12705321
Words skipped: 0
Duplicates removed: 0
Total time: 0 hrs 0 mins 32.932 secs


Avatar
megazoid

Status: n/a
Joined: Sun, 18 Jan 2015
Posts: 13
Team:
Reputation: 0 Reputation
Offline
Mon, 23 Feb 2015 @ 00:55:17

Merge complete to: password.txt
Total words : 1298208425
Words skipped: 259
Duplicates removed: 80025921
Total time: 7 hrs 31 mins 35.545 secs
took a while but worked a treat


Btc = 1KE7ovy85wyzJNFX7rrii6H2PReGAxNkA4


215 Results - Page 3 of 8 -
1 2 3 4 5 6 7 8

We have a total of 148426 messages in 18357 topics.
We have a total of 18219 registered users.
Our newest registered member is OrlandoX.