Home - General Discussion - Tips for creating a local database (MariaDB, ElasticSearch, etc.) of hashes!


5 Results - Page 1 of 1 -
1
Author Message
Avatar
drego85

Status: n/a
Joined: Mon, 25 Sep 2017
Posts: 1
Team:
Reputation: 0 Reputation
Offline
Thu, 08 Mar 2018 @ 04:23:56

Hi guys, I was thinking of creating a local Hash database (mainly SHA256).
Considering that I bought the Dazzlepod list, I think I get a database of 60Gb of data.
What is the best storage system?

Personally I thought of MariaDB, but maybe it's too slow...HashKiller which database use? Elastic? Microsoft Azzure?

Any suggestions?

Thank you so much


Avatar
blandyuk
Admin / Owner
Status: Trusted
Joined: Tue, 05 Jul 2011
Posts: 2902
Team: HashKiller
Reputation: 3880 Reputation
Offline
Thu, 08 Mar 2018 @ 10:06:58

Hi, I wrote my own flat-file database as its the only way to do it and use zero memory overhead. You will need to think about your own indexes for this also. If done correctly, your lookup times should be < 50ms even with say a trillion hashes. You would need WAY more than 60GB for a trillion thou also, I store all generated key-spaces in 4-byte uint.

For wordlist based lookups, just think about the structure and indexing.


Please read the forum rules | Please read the paid section rules
I accept private hash lists, with forum donations only.
BTC: 15qF9WUeFUD63ishxyAMiEgGqTcYzk4j9b
GPU Power: 7x GeForce GTX 1070 and My Brain

Avatar
payknight

Status: Cracker
Joined: Wed, 13 Apr 2016
Posts: 293
Team: just4fun
Reputation: 117 Reputation
Offline
Mon, 12 Mar 2018 @ 08:21:44

drego85 said:

Hi guys, I was thinking of creating a local Hash database (mainly SHA256).
Considering that I bought the Dazzlepod list, I think I get a database of 60Gb of data.
What is the best storage system?

Personally I thought of MariaDB, but maybe it's too slow...HashKiller which database use? Elastic? Microsoft Azzure?

Any suggestions?

Thank you so much

maybe u should try https://snusbase.com/

this is what pub db use


+rep if i helped
BTC : 1PAyKniGHt7yyCb8HdsziTHBEFX6zkGSHz

Avatar
s3in!c

Status: Elite
Joined: Wed, 10 Apr 2013
Posts: 769
Team:
Reputation: 1092 Reputation
Online
Mon, 12 Mar 2018 @ 10:34:02

payknight said:

maybe u should try https://snusbase.com/

this is what pub db use


I don't think that this helps creating an own database system.

Hashes.org is running with Apache Cassandra.


+Rep if I helped

Hashes.org - Powerful Community Password Recovery
Hashtopolis - Hashcat distributed cracking tool

BTC: 1VTuiKrNoytU1PvYrF3J7VapQ9oPg93Jn

Avatar
StuW

Status: n/a
Joined: Sun, 04 Jun 2017
Posts: 55
Team:
Reputation: 150 Reputation
Offline
Mon, 12 Mar 2018 @ 20:08:07

I would suppose it would depend on what you wanted to do with the data. Checking hashes one by one wouldn't be too stressful on a Mysql/MariahDB server, the most time taken would be adding the data even with bulk inserts.



5 Results - Page 1 of 1 -
1

We have a total of 145836 messages in 18008 topics.
We have a total of 17977 registered users.
Our newest registered member is urbanHC.