NOTE: Why not use our List Manager to crack your lists? Its easy and enables better management.

NOTE: When cracking WPA/WPA2 passwords, make sure you check gpuhash.me first incase it's already been processed.

Home - General Discussion - HUGE word-lists duplicate remover and merge tool


249 Results - Page 9 of 9 -
1 2 3 4 5 6 7 8 9
Author Message
Avatar
dipeperon

Status: n/a
Joined: Tue, 03 Apr 2018
Posts: 362
Team:
Reputation: 406 Reputation
Online
Sun, 10 Feb 2019 @ 11:49:17

successhalf said:

Please help me
How to delete spaces?

input.txt
1 C K X c


output.txt
1CKXc

Use unified list manager


My haschat stuff (rules, scripts): https://github.com/theherp/Hashcat-stuff

Avatar
successhalf

Status: n/a
Joined: Thu, 20 Sep 2018
Posts: 14
Team:
Reputation: 0 Reputation
Offline
Sun, 10 Feb 2019 @ 12:23:21

dipeperon said:

successhalf said:

Please help me
How to delete spaces?

input.txt
1 C K X c


output.txt
1CKXc

Use unified list manager

I know that software you can take me a picture
with user manual
thank you


Avatar
dipeperon

Status: n/a
Joined: Tue, 03 Apr 2018
Posts: 362
Team:
Reputation: 406 Reputation
Online
Sun, 10 Feb 2019 @ 13:01:09

successhalf said:


I know that software you can take me a picture
with user manual
thank you

Use a regex (regular expression) replace for a space character, and replace it with an empty string

You can find it in the tools menu at the top


My haschat stuff (rules, scripts): https://github.com/theherp/Hashcat-stuff

Avatar
zer0day

Status: Banned
Joined: Thu, 06 Oct 2016
Posts: 7
Team:
Reputation: 0 Reputation
Offline
Sat, 09 Mar 2019 @ 19:42:27

WARNING! User is BANNED and maybe a SCAMMER.

What does this program do?

1.txt =

1
2
3
4
5
6
7
8
9
0

2.txt =

a
b
c
d
e
f
1
2
3

App.Merge.exe o="2.txt" t=4 "1.txt"

=

6
2
9
7
4
1
0
3
8
5

App.Merge.exe o="1.txt" t=4 "2.txt"

=

2
1
d
a
e
f
c
3
b

Author are you a degenerate?


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3199
Team: HashKiller
Reputation: 4152 Reputation
Online
Sat, 09 Mar 2019 @ 20:09:52

Clearly your the "degenerate" due to the fact that you have over-written your original "2.txt" file with your "1.txt! file lol

App.Merge.exe o="2.txt" t=4 "1.txt"

Input file: 1.txt
Output file: 2.txt

Learn how it fucking works properly before writing a stupid fucking topic! Idiot.

Correct is:

App.Merge.exe o="merged.txt" t=4 "1.txt" "2.txt"


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 9x GTX 1070 + 4x GTX 1080

Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 4
Team:
Reputation: 0 Reputation
Offline
Sun, 17 Mar 2019 @ 20:27:12

been using this for a while and its amazing,you should make more programs for huge files


Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 4
Team:
Reputation: 0 Reputation
Offline
Fri, 05 Apr 2019 @ 14:21:40

(perfect example of what i mean attached as a gif below)If anyone would like to do individual files 1 by 1 automatically without merging them into 1 file, make a .bat file and then put this:

FOR %%f IN (*.txt) DO App.Merge.exe o="deduped_%%f" c=3072 t=8 "E:/dbs/temp/%%f"

but change E:/dbs/temp/ to whatever directory you want. Make sure the .bat and App.Merge.exe are both in the directory you put then run it.

also works with the App.Regex.exe tool

set /p regex="Regex: "
FOR %%f IN (*.txt) DO App.RegEx.exe "E:/dbs/temp/%%f" x="%regex%" o="regexed_%%f"

same as before, change E:/dbs/temp/ to directory you want and make sure App.Regex.exe and batch file in the directory


Attachments: Login to view attachments.
Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 4
Team:
Reputation: 0 Reputation
Offline
Thu, 18 Apr 2019 @ 04:03:00

blandyuk said:

OK, so I've started writing my own word-list merge and duplicate remover already done 2 word-lists which are nearly 2 GB each. It obviously trades the use of your local HD instead of memory but it does chunk the data in memory before flushing it to files on local HD which improves performance massively. You will need to make sure you have enough local HD space = SUM(all word-lists).

Processing files are sorted in: [BaseDirectory]\tmp

Finished file is not sorted properly but are sorted in 256 chunks, 00 to ff based on HEX value of plain-text.

Example:

passwords.txt
Original Size: 184 MB
Converted Size: 169 MB
Duplicates Removed: 28
Run-time: 129 seconds

Obviously we can merge multiple files but I'm just processing 1 file for the above example. Takes awhile but I'm working on improving it. Anyone interested in trying it, please see below:

http://home.btconnect.com/md5decrypter/App.Merge.zip [21.4 KB]

Command format:

App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

For a report analysis of a word-list:

App.Merge.exe r="word-list1.txt"

Double-quotes required for path / file names which contain spaces. You can also specify directory paths if you wish to merge / sort whole directories.

o=[out-file] - Output file.
t=[threads] - Used to speed sorting up only.
c=[mem] - Used to control how much RAM memory to use in MB. Default is 1024. Capped at 3072.
min=[num] - Minimum word length. Default = 1
max=[num] - Maximum word length. Default = 4096.

Words containing control characters will be converted into the Hashcat HEX format: $HEX[...]

is there any chance you could add a -r option that recursively goes through a directory to your app.merge.exe and app.regex.exe programs?(through folders and subfolders)


Avatar
12hashman5

Status: n/a
Joined: Tue, 12 Mar 2019
Posts: 27
Team:
Reputation: 37 Reputation
Offline
Wed, 15 May 2019 @ 04:29:21

@blandyuk would sort -u *wordlists on linux do the same thing?



249 Results - Page 9 of 9 -
1 2 3 4 5 6 7 8 9

We have a total of 188974 messages in 23444 topics.
We have a total of 21239 registered users.
Our newest registered member is Triskebab.