NOTE: When cracking WPA/WPA2 passwords, make sure you check gpuhash.me first incase it's already been processed.

Home - General Discussion - HUGE word-lists duplicate remover and merge tool

WARNING!
Due to the number of SCAMS going on in the PAID forum, PLEASE ask an ADMIN or MODERATOR to verify ALL found passwords to ensure you are not being SCAMMED.
DO NOT PAY until an ADMIN or MOD has verified them for you!


239 Results - Page 8 of 8 -
1 2 3 4 5 6 7 8
Author Message
Avatar
payknight

Status: Cracker
Joined: Wed, 13 Apr 2016
Posts: 502
Team: just4fun
Reputation: 349 Reputation
Offline
Thu, 20 Apr 2017 @ 12:54:06

ima repeat my self

1, is that possible raise the capped over 3gb of ram? most of the system today have atleast 8gb+, that will be usefull for high end computers/servers that have lots of ram perhaps that will reduce the timing? (maybe maxing it to 32gbs of ram? or it does not matter??)


+rep if i helped
BTC : 1PAyKniGHt7yyCb8HdsziTHBEFX6zkGSHz

Avatar
Hxsh

Status: n/a
Joined: Thu, 29 Dec 2016
Posts: 26
Team:
Reputation: 10 Reputation
Offline
Mon, 24 Apr 2017 @ 17:45:58

Please, you have to help me.
I've been merging my master wordlist for about 7 days now (around 500 gbs) and as it was finishing up (removing dupes and etc) a storm knocked my power out ruining all that progress. Is there any way to resume from where it starts removing dupes and merging?



Im new :c

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3036
Team: HashKiller
Reputation: 4061 Reputation
Offline
Mon, 24 Apr 2017 @ 19:25:33

Hxsh, there is no way to resume I'm afraid

With regards to RAM usage, I'm looking at it


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
uknites

Status: n/a
Joined: Sun, 16 Apr 2017
Posts: 23
Team:
Reputation: 0 Reputation
Offline
Tue, 25 Apr 2017 @ 21:05:56

if the great admins can share some of the best wordlists with no duplicates please


Avatar
oayz

Status: n/a
Joined: Wed, 07 Feb 2018
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Wed, 07 Feb 2018 @ 00:48:03

blandyuk, thanks for the great tool! Any chance you can add option to specify TMP directory? I believe today it's in INPUT folder but separating input and tmp may help to speed sorting. Thanks!


Avatar
wakawaka

Status: n/a
Joined: Tue, 31 Jul 2018
Posts: 8
Team:
Reputation: 10 Reputation
Offline
Sun, 05 Aug 2018 @ 03:52:31

I keep getting the below error, when merging 2x 10gb wodlists, system ram is 16GB, 8 core processor,
can anyone help


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
app.merge.exe o=merge.txt t=4 c=3000 0001.txt 0002.txt

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

File: 61c.txt ~ Sort Time: 358.01 m/s ~ Duplicates: 135493150
File: 61e.txt ~ Sort Time: 304.31 m/s ~ Duplicates: 135580161

Unhandled Exception: File: 61f.txt ~ Sort Time: 284.85 m/s ~ Duplicates: 135754541
File: 620.txt ~ Sort Time: 342.29 m/s ~ Duplicates: 135820653
System.IndexOutOfRangeException: Index was outside the bounds of the array.
at System.UnSafeCharBuffer.AppendString(String stringToAppend)
at System.String.Join(String separator, String[] value, Int32 startIndex, Int32 count)
at App.Merge.Program.sortFile(Object vv)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)


Avatar
Purpleninja225

Status: n/a
Joined: Thu, 05 Jul 2018
Posts: 122
Team:
Reputation: 212 Reputation
Offline
Fri, 17 Aug 2018 @ 09:00:45

Is the source on github possibly? For forking and such? Maybe some help expanding the max mem limit? Would love to help out on this project cause I use it all the time.


+rep if I helped. GTX 750 Ti & GTX 550
Github: https://github.com/PurpleNinja225/Hash-Cracking Discord: PurpleNinja225 #6785

Tipz Jar:
BTC 321aVnFwQrhZcHoCoPzp1Vh46rUiQmExzp
ETH 0xF5ab8429F6991f0232Dd4A0eB8318a4e172b1282

Avatar
essquireo0o

Status: n/a
Joined: Mon, 20 Mar 2017
Posts: 128
Team:
Reputation: 55 Reputation
Offline
Wed, 29 Aug 2018 @ 19:19:34

This is sick BTW, you the man


Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Wed, 29 Aug 2018 @ 23:49:02

Hi sorry if it has been posted. What syntax do you type to remove anything below 5 characters.


Cheers Kev

Still Learning


Avatar
Purpleninja225

Status: n/a
Joined: Thu, 05 Jul 2018
Posts: 122
Team:
Reputation: 212 Reputation
Offline
Thu, 30 Aug 2018 @ 04:46:04

kevtheskin said:

Hi sorry if it has been posted. What syntax do you type to remove anything below 5 characters.


Cheers Kev

Still Learning

PS L:\CMIYC stuff\Tools> .\App.Merge.exe --help
Merge Tool by BlandyUK v0.50

Command format:
App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

o=[out-file] - Output file.
t=[threads] - Used to speed up sorting only.
c=[mem] - Memory / RAM to use in MB. Default is 1024 MB.
min=[num] - Minimum word length. Default = 1
max=[num] - Maximum word length. Default = 4096.
--export-dups - Export duplicates into separate output filename.
--remove-spaces - Removes spaces from words in word-lists.

To do a report analysis:

App.Merge.exe r="word-list1.txt"

You are looking for --min=5


+rep if I helped. GTX 750 Ti & GTX 550
Github: https://github.com/PurpleNinja225/Hash-Cracking Discord: PurpleNinja225 #6785

Tipz Jar:
BTC 321aVnFwQrhZcHoCoPzp1Vh46rUiQmExzp
ETH 0xF5ab8429F6991f0232Dd4A0eB8318a4e172b1282

Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Fri, 31 Aug 2018 @ 08:01:50

Purpleninja225 said:

kevtheskin said:

Hi sorry if it has been posted. What syntax do you type to remove anything below 5 characters.


Cheers Kev

Still Learning

PS L:\CMIYC stuff\Tools> .\App.Merge.exe --help
Merge Tool by BlandyUK v0.50

Command format:
App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

o=[out-file] - Output file.
t=[threads] - Used to speed up sorting only.
c=[mem] - Memory / RAM to use in MB. Default is 1024 MB.
min=[num] - Minimum word length. Default = 1
max=[num] - Maximum word length. Default = 4096.
--export-dups - Export duplicates into separate output filename.
--remove-spaces - Removes spaces from words in word-lists.

To do a report analysis:

App.Merge.exe r="word-list1.txt"

You are looking for --min=5


Thanks Dude


Avatar
DarkDeath25

Status: n/a
Joined: Tue, 04 Sep 2018
Posts: 78
Team:
Reputation: 92 Reputation
Online
Sat, 15 Sep 2018 @ 12:54:02

Merge complete to: AllMerged.txt
Total words : 10158302131
Words skipped: 350
Duplicates removed: 1336772882
$HEX[...] conversions: 3538531
Total time: 7 hrs 5 mins 6.425 secs

From 111go to 98.7go
-----------

Removed word under 5 char

STUCK at since 2hours

Position: 22,38 % of AllMerged.txt
- Words: 2031091961 ~ Skipped: 0 ~ Mem: 1523 MB


Rep+ is appreciated
BTC if i helped : 125jYmWxjJGtHJSfhoTQCjDnYNZEdbEHKV
---
1*GTX 1070
Goal : H370 Mining Master and 20*gtx1080

Avatar
seomelon06

Status: n/a
Joined: Wed, 19 Sep 2018
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Wed, 19 Sep 2018 @ 08:41:46

Thanks for the good ideas to bring it. I know a lot more. ufabet :J


Avatar
CrypticError

Status: Cracker
Joined: Tue, 03 Apr 2018
Posts: 241
Team: Hashdog
Reputation: 323 Reputation
Online
Wed, 03 Oct 2018 @ 21:00:29

Thank you so much blandy! Managed to tidy up 50gb+ of different word list files into one very quickly.

Merge complete to: dedupe.txt
Total words : 837866660
Words skipped: 12
Duplicates removed: 323044732
$HEX[...] conversions: 2190
Total time: 0 hrs 8 mins 16.607 secs


+rep if I helped, thanks. Means a lot when I'm using power in China to crack hashes!

Hardware: 5x Titan X, 1x i7 4770k

Avatar
Tenner

Status: n/a
Joined: Wed, 03 Oct 2018
Posts: 5
Team: Save Ferris
Reputation: 0 Reputation
Offline
Wed, 03 Oct 2018 @ 21:48:05

CrypticError said:

Thank you so much blandy! Managed to tidy up 50gb+ of different word list files into one very quickly.

Merge complete to: dedupe.txt
Total words : 837866660
Words skipped: 12
Duplicates removed: 323044732
$HEX[...] conversions: 2190
Total time: 0 hrs 8 mins 16.607 secs

Total words : 837866660

I need this word list!!


Avatar
successhalf

Status: n/a
Joined: Thu, 20 Sep 2018
Posts: 12
Team:
Reputation: 0 Reputation
Offline
Thu, 04 Oct 2018 @ 03:40:07

please help me
I have a file big 90gb
how to split

input 90gb.txt
split 1.txt / 2.txt.........3,4,5,6


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3036
Team: HashKiller
Reputation: 4061 Reputation
Offline
Thu, 04 Oct 2018 @ 11:43:52

successhalf

To do a report analysis:

App.Merge.exe r="word-list1.txt"

From here, look at what you have. If your wordlist contains brute-force keyspaces, you might as well remove them as pointless. Hashcat can process them fast in brute-force attack mode than reading from a wordlist and also uses zero space.

Once you have established what you can remove, I suggest you remove them using the likes of "min=5" which will remove all words upto 4 chars. Maybe even use "min=6".

If you still need to split the list, use my RegEx Tool here:

https://forum.hashkiller.co.uk/topic-view.aspx?t=7645&m=55993#55993

You'll have to decide how you want to split based on a regular expression. There are numerous ways.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
DarkDeath25

Status: n/a
Joined: Tue, 04 Sep 2018
Posts: 78
Team:
Reputation: 92 Reputation
Online
Thu, 04 Oct 2018 @ 13:49:19

Succesfully got to remove all word under 5 char in my 98gb file

here you go : https://paste.hashkiller.co.uk/0sb-osfTEeiA_ECNXEjIzQ


Rep+ is appreciated
BTC if i helped : 125jYmWxjJGtHJSfhoTQCjDnYNZEdbEHKV
---
1*GTX 1070
Goal : H370 Mining Master and 20*gtx1080

Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3036
Team: HashKiller
Reputation: 4061 Reputation
Offline
Thu, 04 Oct 2018 @ 14:41:09

Nice one way of splitting very large wordlists up is by length and once you have a report, you can calculate how large each section will be. You can also combine lengths to make up smaller ones.

You can use my RegEx Tool to do this quickly as it does not need much RAM to do it as it processes in chunks and also purges every n MB / GB which you specify. Only bottleneck would be the mechanical drive if you are using one due to slower write speeds.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
successhalf

Status: n/a
Joined: Thu, 20 Sep 2018
Posts: 12
Team:
Reputation: 0 Reputation
Offline
Sat, 06 Oct 2018 @ 16:43:40

help me set the command line with the instructions below

bruteemail.exe [OPTIONS] [TARGETS]

Where [TARGETS] are one or more mail address

Options:

-i, --in FILE
Read addresses from FILE, one address per line. If FILE is "-" then stdin is read.

-o, --out FILE
Append login to FILE, one per line.


bruteemail.exe ......?


Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Tue, 23 Oct 2018 @ 20:26:17

blandyuk said:

successhalf

To do a report analysis:

App.Merge.exe r="word-list1.txt"

From here, look at what you have. If your wordlist contains brute-force keyspaces, you might as well remove them as pointless. Hashcat can process them fast in brute-force attack mode than reading from a wordlist and also uses zero space.

Once you have established what you can remove, I suggest you remove them using the likes of "min=5" which will remove all words upto 4 chars. Maybe even use "min=6".

If you still need to split the list, use my RegEx Tool here:

https://forum.hashkiller.co.uk/topic-view.aspx?t=7645&m=55993#55993

You'll have to decide how you want to split based on a regular expression. There are numerous ways.

Hi there Blandyuk,
When you run r= how do you work out what keyspace is getting used? Most of the dictionarys I have I cant even view in windows or edit? Also can you tell me what syntax to use to remove doubles

Thanks Kev

Doh found it hahahaha. Is there anyway that you can reduce the keyspace number list to about max 20?


Avatar
Purpleninja225

Status: n/a
Joined: Thu, 05 Jul 2018
Posts: 122
Team:
Reputation: 212 Reputation
Offline
Fri, 26 Oct 2018 @ 04:24:27

app.merge automatically removes dupes. you don't have to add syntax.


+rep if I helped. GTX 750 Ti & GTX 550
Github: https://github.com/PurpleNinja225/Hash-Cracking Discord: PurpleNinja225 #6785

Tipz Jar:
BTC 321aVnFwQrhZcHoCoPzp1Vh46rUiQmExzp
ETH 0xF5ab8429F6991f0232Dd4A0eB8318a4e172b1282

Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Thu, 01 Nov 2018 @ 21:02:47

Hello there,

Is there anyway to remove spaces at the start of a word ? I have a word list that has spaces for 4 then letters.

Cheers Kev


Avatar
freeroute
Moderator
Status: Trusted
Joined: Sat, 16 Jul 2016
Posts: 2346
Team:
Reputation: 7915 Reputation
Online
Thu, 01 Nov 2018 @ 21:10:57

kevtheskin said:

Hello there,

Is there anyway to remove spaces at the start of a word ? I have a word list that has spaces for 4 then letters.

Cheers Kev

sed command:
"sed -r 's/^\s+//' word_list_beginning_spaces.txt > word_list_without_spaces.txt"

https://paste.hashkiller.co.uk/LsmuTN4bEeiA_kCNXEjIzQ

perl command:
"perl -ne 'print if s/^\s+//' word_list_beginning_spaces.txt > word_list_without_spaces.txt"

awk command:
"awk 'sub ("^\\s+", "") {print $0}' word_list_beginning_spaces.txt > word_list_without_spaces.txt"


If I helped a +rep is appreciated!

: 13hDMK85KhVnPb2eTFBacHD6kDjKYFLudb
XMPP: freeroute@xmpp.jp

Avatar
Batmanx

Status: Banned
Joined: Tue, 25 Sep 2018
Posts: 50
Team:
Reputation: 64 Reputation
Offline
Mon, 12 Nov 2018 @ 14:31:47

WARNING! User is BANNED and maybe a SCAMMER.

Great work. Thank you.

Merge complete to: done.txt
Total words : 1343193633
Words skipped: 3
Duplicates removed: 445387310
$HEX[...] conversions: 141466
Total time: 0 hrs 32 mins 22.364 secs


Avatar
Batmanx

Status: Banned
Joined: Tue, 25 Sep 2018
Posts: 50
Team:
Reputation: 64 Reputation
Offline
Mon, 12 Nov 2018 @ 17:47:32

WARNING! User is BANNED and maybe a SCAMMER.

Merge complete to: othermail.txt
Total words : 585068183
Words skipped: 133
Duplicates removed: 205640425
$HEX[...] conversions: 993
Total time: 0 hrs 31 mins 39.745 secs


Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Sat, 17 Nov 2018 @ 20:36:00

Hi there,
Can appmerge be used to remove spaces in wordlist. I noticed in one of my list big spaces instead of words which I cant remove in windows because the wordlist is to big to edit. I am only viewing usin Glogg.

Cheers Kev
Ps using windows 10


Avatar
dipeperon

Status: n/a
Joined: Tue, 03 Apr 2018
Posts: 192
Team:
Reputation: 281 Reputation
Online
Sat, 17 Nov 2018 @ 21:17:42

kevtheskin said:

Hi there,
Can appmerge be used to remove spaces in wordlist. I noticed in one of my list big spaces instead of words which I cant remove in windows because the wordlist is to big to edit. I am only viewing usin Glogg.

Cheers Kev
Ps using windows 10

You can use "unified list manager" with a replace expression

It's freeware


My haschat stuff (rules, scripts): https://github.com/theherp/Hashcat-stuff

Avatar
kevtheskin

Status: n/a
Joined: Wed, 21 Feb 2018
Posts: 162
Team:
Reputation: 58 Reputation
Offline
Mon, 19 Nov 2018 @ 19:21:49

dipeperon said:

kevtheskin said:

Hi there,
Can appmerge be used to remove spaces in wordlist. I noticed in one of my list big spaces instead of words which I cant remove in windows because the wordlist is to big to edit. I am only viewing usin Glogg.

Cheers Kev
Ps using windows 10

You can use "unified list manager" with a replace expression

It's freeware


Hello dipeperon,
How are you. Thanks for getting back to me. Would you mind explaining how to use this tool to view large wordlist and edit .

Thanks Kev



239 Results - Page 8 of 8 -
1 2 3 4 5 6 7 8

We have a total of 163685 messages in 20542 topics.
We have a total of 19308 registered users.
Our newest registered member is WeeJobbieMilzo.