2014-03-04 23:45:44 +0000 <hyperthunk> (I don't know if we're using binary badly or something, but I doubt it)
2014-03-04 23:46:04 +0000 <hyperthunk> making serialisation fast(er) would be an enormous boost
2014-03-04 23:46:06 +0000 <carter> i've some thoughts on that, but they aren't "fast fix ones"
2014-03-04 23:46:18 +0000 <carter> hyperthunk: hav eyou asked dcouts about his pending new version of binary?
2014-03-04 23:46:41 +0000 <hyperthunk> carter: nope, but every time I talk to him about CH he has more ideas how we can improve it :)
2014-03-04 23:47:12 +0000 <hyperthunk> scouts did mention that we could use the new(ish) internal buffering stuff to speed up small message serialization
2014-03-04 23:47:33 +0000 <hyperthunk> though at the moment no messages are small, because there's like a 200% overhead
2014-03-04 23:47:48 +0000 <hyperthunk> the wire format sucks, and I've not had time to even understand it let alone think about making improvements
2014-03-04 23:48:00 +0000 <hyperthunk> (it sucks only in verbosity I mean)
2014-03-04 23:49:01 +0000 <carter> have you tried compression?
2014-03-04 23:49:12 +0000 <carter> like fastlz or whatever?
2014-03-04 23:49:28 +0000 <hyperthunk> carter: edsko thinks there's big improvements to make before we even get into that
2014-03-04 23:49:36 +0000 <carter> just try it
2014-03-04 23:49:42 +0000 <carter> its a localized change that doesn't require touching anythign else
2014-03-04 23:49:50 +0000 <carter> ideas not measured are ideas that dont matter
2014-03-04 23:50:28 +0000 <hyperthunk> carter: sure can do
2014-03-04 23:50:29 +0000 <hyperthunk> 1 sec
2014-03-04 23:51:01 +0000 <carter> http://hackage.haskell.org/package/lz4
2014-03-04 23:52:09 +0000lukexi(~lukexi@c-67-188-105-141.hsd1.ca.comcast.net)
2014-03-04 23:52:25 +0000asmyers(~quassel@c-50-181-115-44.hsd1.md.comcast.net)
2014-03-04 23:53:00 +0000 <hyperthunk> so
2014-03-04 23:53:02 +0000 <hyperthunk> Small CH messages have a large overhead. For instance, consider sending `(Nothing :: Maybe ProcessId)`. The binary encoding of `Nothing` is one byte, but the total message sent looks like
2014-03-04 23:53:02 +0000 <hyperthunk> <channel ID> 4 bytes
2014-03-04 23:53:04 +0000 <hyperthunk> <message length> 4 bytes
2014-03-04 23:53:05 +0000 <hyperthunk> <message type fingerprint> 16 bytes
2014-03-04 23:53:05 +0000 <hyperthunk> <payload> 1 byte
2014-03-04 23:53:09 +0000 <hyperthunk> For an overhead of 2400%
2014-03-04 23:53:17 +0000 <hyperthunk> we could...
2014-03-04 23:53:20 +0000 <hyperthunk> Use a single byte for the first 255 connection IDs
2014-03-04 23:53:31 +0000 <hyperthunk> Use a single byte for message length <= 255
2014-03-04 23:53:37 +0000 <carter> ok
2014-03-04 23:53:39 +0000 <hyperthunk> Use a single byte to index into some cache for commonly used fingerprints (since most applications will probably send only a handful of different types of messages across the network, this might be very effective)
2014-03-04 23:53:43 +0000 <carter> OR just do compression
2014-03-04 23:53:47 +0000 <hyperthunk> As (another) optimisation, we could take advantage of recent improvements to the binary package that allow us to pre-allocate a buffer and re-use it for serializing small messages.
2014-03-04 23:53:49 +0000 <hyperthunk> carter: