sindresorhus/hasha

Add support for doing the hashing in a worker thread #16

sindresorhus posted onGitHub

Could maybe be useful for hashing large files, as the crypto Node.js module is synchronous, so it could block the main thread for some time.

https://nodejs.org/api/worker_threads.html

Thoughts?


I like it. It's usually not super expensive, but a 1gb file shasum takes ~4s on my macbook 2015 2.8 GHz Intel Core i7

I would actually prefer a C++ binding (that I'd guess would be faster, since in OS X for example, running openssl sha1 :file takes 4x less time than the crypto approach in node.

posted by tomasdev about 6 years ago

I would actually prefer a C++ binding (that I'd guess would be faster, since in OS X for example, running openssl sha1 :file takes 4x less time than the crypto approach in node.

Not interested in doing native bindings. That comes it its own set of horribleness.

posted by sindresorhus almost 6 years ago
posted by sindresorhus almost 6 years ago

Can easily be done for file hashes(and will work perfectly). For strings/buffers, I don't see a way to go further than is done in https://github.com/sindresorhus/crypto-hash/pull/7 so I'd just use it. As far as my understanding goes, we will have to always clone(not transfer) bytes for arbitrary streams, and as chunks are expected to be pretty small, the chance clone+send task overhead will be be bigger than just hashing in place is quite high. Needs some benchmarks, but I'm pretty sure it's not viable.

Speaking of performance comparison with openssl, it sounds strange to me, as node is using openssl underneath. If such issue actually exists it could use some serious debugging on libuv/nodejs/app level.

posted by stroncium almost 6 years ago

@issuehunt has funded $80.00 to this issue.


posted by IssueHuntBot almost 6 years ago

On it. @sindresorhus Should I add crypto-hash as a dependency or duplicate code?

posted by stroncium almost 6 years ago

Does crypto-hash expose enough functionality? Would be good to share code if possible, yes.

posted by sindresorhus almost 6 years ago

@sindresorhus Need to check it, but should be enough.

posted by stroncium almost 6 years ago

@sindresorhus Never mind. I didn't notice there was no async version of string/buffer hashing in hasha. Without it there is almost no code intersection with crypto-hash.

posted by stroncium almost 6 years ago

I didn't notice there was no async version of string/buffer hashing in hasha.

Sorry if the issue was not clear enough on this, but the intention is to add it now that it's possible.

posted by sindresorhus almost 6 years ago

@sindresorhus Well, then there are some questions:

  • Should it be a breaking change(to promise returning function) or additional method?
  • crypto-hash doesn't support md5, should we add it just for nodejs? (And while at it, should we add a method you pass algorithm string to for consistency?)
  • If not, should we use crypto-hash at all? Most code will need to be duplicated here to implement md5 anyhow.
posted by stroncium almost 6 years ago

Should it be a breaking change(to promise returning function) or additional method?

Additional method.

crypto-hash doesn't support md5, should we add it just for nodejs? (And while at it, should we add a method you pass algorithm string to for consistency?)

Nah

If not, should we use crypto-hash at all? Most code will need to be duplicated here to implement md5 anyhow.

Probably not worth using crypto-hash at all then.

posted by sindresorhus almost 6 years ago

@sindresorhus has rewarded $72.00 to @stroncium. See it on IssueHunt

  • :moneybag: Total deposit: $80.00
  • :tada: Repository reward(0%): $0.00
  • :wrench: Service fee(10%): $8.00
posted by issuehunt-app[bot] over 5 years ago

Fund this Issue

$80.00
Rewarded

Rewarded pull request

Other pull requests

Recent activities

stroncium was rewarded by sindresorhus for sindresorhus/hasha# 16
over 5 years ago
pro-src submitted an output to  sindresorhus/ hasha# 16
over 5 years ago