Roman Leventov
1 min readFeb 5, 2018

--

You can define optimum slightly differently, depending on how complicated you are ready to make the source code in exchange to efficiency improvements (and how substantial are those improvements). But anyway, the difference between absolute possible peak efficiency and “reasonable” peak (while code is still sane) is not so big, maybe not more than 10–20% by any efficiency metrics.

Talking specifically about Druid’s ingestion, it may produce less garbage, use better in-memory format to require less heap space, move more data off-heap, share dictionaries between ingestion “parts”, improve concurrency, etc. I think 50–100% improvements by some dimensions are still possible. For example, I’m currently working on a change that should reduce heap footprint significantly, it is referenced in this PR.

It’s just about in-JVM state of things, on higher level it is much more inefficient. I mentioned some of them (but not all) in this thread.

--

--

Roman Leventov
Roman Leventov

Written by Roman Leventov

Writing about systems, technology, philosophy.

No responses yet