NEW: We have a Discord server now. Click here to go there now!

NOTE: Why not use our List Manager to crack your lists? Its easy and enables better management.

NOTE: When cracking WPA/WPA2 passwords, make sure you check gpuhash.me first incase it's already been processed.

Home - General Discussion - HUGE word-lists duplicate remover and merge tool


253 Results - Page 9 of 9 -
1 2 3 4 5 6 7 8 9
Author Message
Avatar
dipeperon

Status: Member
Joined: Tue, 03 Apr 2018
Posts: 431
Team:
Reputation: 431 Reputation
Online
Sun, 10 Feb 2019 @ 11:49:17

successhalf said:

Please help me
How to delete spaces?

input.txt
1 C K X c


output.txt
1CKXc

Use unified list manager


My haschat stuff (rules, scripts): https://github.com/theherp/Hashcat-stuff

Avatar
successhalf

Status: n/a
Joined: Thu, 20 Sep 2018
Posts: 14
Team:
Reputation: 0 Reputation
Offline
Sun, 10 Feb 2019 @ 12:23:21

dipeperon said:

successhalf said:

Please help me
How to delete spaces?

input.txt
1 C K X c


output.txt
1CKXc

Use unified list manager

I know that software you can take me a picture
with user manual
thank you


Avatar
dipeperon

Status: Member
Joined: Tue, 03 Apr 2018
Posts: 431
Team:
Reputation: 431 Reputation
Online
Sun, 10 Feb 2019 @ 13:01:09

successhalf said:


I know that software you can take me a picture
with user manual
thank you

Use a regex (regular expression) replace for a space character, and replace it with an empty string

You can find it in the tools menu at the top


My haschat stuff (rules, scripts): https://github.com/theherp/Hashcat-stuff

Avatar
zer0day

Status: Banned
Joined: Thu, 06 Oct 2016
Posts: 7
Team:
Reputation: 0 Reputation
Offline
Sat, 09 Mar 2019 @ 19:42:27

WARNING! User is BANNED and maybe a SCAMMER.

What does this program do?

1.txt =

1
2
3
4
5
6
7
8
9
0

2.txt =

a
b
c
d
e
f
1
2
3

App.Merge.exe o="2.txt" t=4 "1.txt"

=

6
2
9
7
4
1
0
3
8
5

App.Merge.exe o="1.txt" t=4 "2.txt"

=

2
1
d
a
e
f
c
3
b

Author are you a degenerate?


Avatar
blandyuk
Administrator
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3205
Team: HashKiller
Reputation: 7744 Reputation
Offline
Sat, 09 Mar 2019 @ 20:09:52

Clearly your the "degenerate" due to the fact that you have over-written your original "2.txt" file with your "1.txt! file lol

App.Merge.exe o="2.txt" t=4 "1.txt"

Input file: 1.txt
Output file: 2.txt

Learn how it fucking works properly before writing a stupid fucking topic! Idiot.

Correct is:

App.Merge.exe o="merged.txt" t=4 "1.txt" "2.txt"


Please read the forum rules | Please read the paid section rules

Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 16
Team:
Reputation: 12 Reputation
Offline
Sun, 17 Mar 2019 @ 20:27:12

been using this for a while and its amazing,you should make more programs for huge files


Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 16
Team:
Reputation: 12 Reputation
Offline
Fri, 05 Apr 2019 @ 14:21:40

(perfect example of what i mean attached as a gif below)If anyone would like to do individual files 1 by 1 automatically without merging them into 1 file, make a .bat file and then put this:

FOR %%f IN (*.txt) DO App.Merge.exe o="deduped_%%f" c=3072 t=8 "E:/dbs/temp/%%f"

but change E:/dbs/temp/ to whatever directory you want. Make sure the .bat and App.Merge.exe are both in the directory you put then run it.

also works with the App.Regex.exe tool

set /p regex="Regex: "
FOR %%f IN (*.txt) DO App.RegEx.exe "E:/dbs/temp/%%f" x="%regex%" o="regexed_%%f"

same as before, change E:/dbs/temp/ to directory you want and make sure App.Regex.exe and batch file in the directory


Attachments: Login to view attachments.
Avatar
$

Status: n/a
Joined: Wed, 30 Jan 2019
Posts: 16
Team:
Reputation: 12 Reputation
Offline
Thu, 18 Apr 2019 @ 04:03:00

blandyuk said:

OK, so I've started writing my own word-list merge and duplicate remover already done 2 word-lists which are nearly 2 GB each. It obviously trades the use of your local HD instead of memory but it does chunk the data in memory before flushing it to files on local HD which improves performance massively. You will need to make sure you have enough local HD space = SUM(all word-lists).

Processing files are sorted in: [BaseDirectory]\tmp

Finished file is not sorted properly but are sorted in 256 chunks, 00 to ff based on HEX value of plain-text.

Example:

passwords.txt
Original Size: 184 MB
Converted Size: 169 MB
Duplicates Removed: 28
Run-time: 129 seconds

Obviously we can merge multiple files but I'm just processing 1 file for the above example. Takes awhile but I'm working on improving it. Anyone interested in trying it, please see below:

https://hashkiller.co.uk/downloads/App.Merge.zip [21.4 KB]

Command format:

App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

For a report analysis of a word-list:

App.Merge.exe r="word-list1.txt"

Double-quotes required for path / file names which contain spaces. You can also specify directory paths if you wish to merge / sort whole directories.

o=[out-file] - Output file.
t=[threads] - Used to speed sorting up only.
c=[mem] - Used to control how much RAM memory to use in MB. Default is 1024. Capped at 3072.
min=[num] - Minimum word length. Default = 1
max=[num] - Maximum word length. Default = 4096.

Words containing control characters will be converted into the Hashcat HEX format: $HEX[...]

is there any chance you could add a -r option that recursively goes through a directory to your app.merge.exe and app.regex.exe programs?(through folders and subfolders)


Avatar
12hashman5

Status: n/a
Joined: Tue, 12 Mar 2019
Posts: 27
Team:
Reputation: 37 Reputation
Offline
Wed, 15 May 2019 @ 04:29:21

@blandyuk would sort -u *wordlists on linux do the same thing?


Avatar
damian101

Status: n/a
Joined: Fri, 12 Jul 2019
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Fri, 12 Jul 2019 @ 13:54:22

http://home.btconnect.com/md5decrypter/App.Merge.zip just returns message: "Not Authorized to View This Page [CFN #0004]". I tried multiple browsers.
https://hashkiller.co.uk/downloads/App.Merge.zip works.


Avatar
blandyuk
Administrator
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 3205
Team: HashKiller
Reputation: 7744 Reputation
Offline
Fri, 12 Jul 2019 @ 16:49:52

Ah yes, I've been through and changed them all. Thanks for letting me know.


Please read the forum rules | Please read the paid section rules

Avatar
egorthik

Status: n/a
Joined: Tue, 13 Aug 2019
Posts: 2
Team:
Reputation: 0 Reputation
Offline
Tue, 13 Aug 2019 @ 22:31:38

@blandyuk


Avatar
egorthik

Status: n/a
Joined: Tue, 13 Aug 2019
Posts: 2
Team:
Reputation: 0 Reputation
Offline
Tue, 13 Aug 2019 @ 22:44:06

blandyuk said:

OK, so I've started writing my own word-list merge and duplicate remover already done 2 word-lists which are nearly 2 GB each. It obviously trades the use of your local HD instead of memory but it does chunk the data in memory before flushing it to files on local HD which improves performance massively. You will need to make sure you have enough local HD space = SUM(all word-lists).

Processing files are sorted in: [BaseDirectory]\tmp

Finished file is not sorted properly but are sorted in 256 chunks, 00 to ff based on HEX value of plain-text.

Example:

passwords.txt
Original Size: 184 MB
Converted Size: 169 MB
Duplicates Removed: 28
Run-time: 129 seconds

Obviously we can merge multiple files but I'm just processing 1 file for the above example. Takes awhile but I'm working on improving it. Anyone interested in trying it, please see below:

https://hashkiller.co.uk/downloads/App.Merge.zip [21.4 KB]

Command format:

App.Merge.exe o="output-file.txt" t=4 [options] ... "word-list1.txt" "word-list2.lst" "directory1" ...

For a report analysis of a word-list:

App.Merge.exe r="word-list1.txt"

Double-quotes required for path / file names which contain spaces. You can also specify directory paths if you wish to merge / sort whole directories.

o=[out-file] - Output file.
t=[threads] - Used to speed sorting up only.
c=[mem] - Used to control how much RAM memory to use in MB. Default is 1024. Capped at 3072.
min=[num] - Minimum word length. Default = 1
max=[num] - Maximum word length. Default = 4096.

Words containing control characters will be converted into the Hashcat HEX format: $HEX[...]

Hello. I do not speak English, so sorry my errors.

I found 2 errors

1)
if inputfile have <= 3 strings
when System.NullReferenceException: in App.Merge.Program.Main(String[] args)
///-----------------------
input file:
asdas
dsfdbsddfd
fgdhbfgfg
///-----------------------
______________________________________________

2) error if file write in UTF8 with boom ( 1 str have 0xef, 0xbb, 0xbf prefix )
because of this first str getting inuque. ( scrin and input file )


if u have question for me, plz, write me in discord egorthik#8692


Attachments: Login to view attachments.

253 Results - Page 9 of 9 -
1 2 3 4 5 6 7 8 9

8 users online in the last hour
mrniz1, splx, techtrip, xssfile, Reconsniper, str0nGh0ld, Plum, Allesanddro,

We have a total of 216670 messages in 26493 topics.
We have a total of 23369 registered users.
Our newest registered member is Iddob.