Discussion:
Static index, fastest way to do forceMerge
Jerven Bolleman
2018-11-02 10:12:52 UTC
Permalink
Dear Lucene Devs and Users,

First of all thank you for this wonderful library and API.

forceMerges are normally not recommended but we fall into one of the few
usecases where it makes sense.

In our use case we have a large index (3 actually) and we don't update
them ever after indexing. i.e. we index all the documents and then never
ever add another document to the index, nor are any deleted.

It has proven beneficial for search performance to always foreMerge down
to one segment. However, this takes significant time. Are there any
suggestions on what kind of merge scheduler/policy settings will utilize
the most of the available IO, CPU and RAM capacity? Currently we end up
being single thread bound, leaving lots of potential cpu and bandwidth
not used during the merge.

e.g. we are looking for a MergeEvertyThing use all hardware policy and
scheduler.

We are currently on lucene 7.4 but nothing is stopping us from upgrading.

Regards,
Jerven

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Erick Erickson
2018-11-02 16:56:46 UTC
Permalink
The merge process is rather tricky, and there's nothing that I know of
that will use all resources available. In fact the merge code is
written to _not_ use up all the possible resources on the theory that
there should be some left over to handle queries etc.

Yeah, the situation you describe is indeed one of the few where
merging down to 1 segment makes sense. Out of curiosity, what kind of
performance gains to you see?

This applies to the default TieredMergePolicy (TMP):

1> there is a limit to the number of segments that can be merged at
once, so sometimes it can take more than one pass. If you have more
than 30 segments, it'll be multi-pass. You can try (and I haven't done
this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
to see if it helps. That only takes effect when you forceMerge.
There's a trick bit of reflection that handles this, see the very end
of TieredMergePolicy.java for the parameters you can set.

2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
from automatically merging down to 1 segment to respecting
"maxMergedSegmentMB" (default 5G). You will have to explicitly pass
maxSegments=1 to get the old behavior.

Best,
Erick
On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
Post by Jerven Bolleman
Dear Lucene Devs and Users,
First of all thank you for this wonderful library and API.
forceMerges are normally not recommended but we fall into one of the few
usecases where it makes sense.
In our use case we have a large index (3 actually) and we don't update
them ever after indexing. i.e. we index all the documents and then never
ever add another document to the index, nor are any deleted.
It has proven beneficial for search performance to always foreMerge down
to one segment. However, this takes significant time. Are there any
suggestions on what kind of merge scheduler/policy settings will utilize
the most of the available IO, CPU and RAM capacity? Currently we end up
being single thread bound, leaving lots of potential cpu and bandwidth
not used during the merge.
e.g. we are looking for a MergeEvertyThing use all hardware policy and
scheduler.
We are currently on lucene 7.4 but nothing is stopping us from upgrading.
Regards,
Jerven
---------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Dawid Weiss
2018-11-02 18:19:34 UTC
Permalink
We are faced with a similar situation. Yes, the merge process can take
a long time and is mostly single-threaded (if you're merging from N
segments into a single segment, only one thread does the job). As
Erick pointed out, the merge process takes a backseat compared to
indexing and searches (in most cases), so it's not a priority, but
it's definitely something people like you (and me) could utilize, if
given the opportunity.

I actually don't see any reasons why merging of individual parts of a
segment can't be done in parallel (this would be a start; later on a
splittable strategy of merging single things could make use of things
like the fork-join executor). I'd love to work on this at some point,
but I honestly don't see any time soon this could be happening. If you
have a spare cycle, take a look at how index writer merges a single
segment; there are quite trivial ways this could be split into
parallel subtasks and executed with, for example, the system fork-join
executor (even without forkable tasks).

https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L2999-L3007

As a side note, you may want to make absolutely sure your merge
scheduler (if it's the CMS) is not using any I/O throttling -- this is
theoretically self-adjustable, but in practice if you only care about
the wall-clock end of a single merge, it's better to turn it off.

Dawid
Post by Erick Erickson
The merge process is rather tricky, and there's nothing that I know of
that will use all resources available. In fact the merge code is
written to _not_ use up all the possible resources on the theory that
there should be some left over to handle queries etc.
Yeah, the situation you describe is indeed one of the few where
merging down to 1 segment makes sense. Out of curiosity, what kind of
performance gains to you see?
1> there is a limit to the number of segments that can be merged at
once, so sometimes it can take more than one pass. If you have more
than 30 segments, it'll be multi-pass. You can try (and I haven't done
this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
to see if it helps. That only takes effect when you forceMerge.
There's a trick bit of reflection that handles this, see the very end
of TieredMergePolicy.java for the parameters you can set.
2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
from automatically merging down to 1 segment to respecting
"maxMergedSegmentMB" (default 5G). You will have to explicitly pass
maxSegments=1 to get the old behavior.
Best,
Erick
On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
Post by Jerven Bolleman
Dear Lucene Devs and Users,
First of all thank you for this wonderful library and API.
forceMerges are normally not recommended but we fall into one of the few
usecases where it makes sense.
In our use case we have a large index (3 actually) and we don't update
them ever after indexing. i.e. we index all the documents and then never
ever add another document to the index, nor are any deleted.
It has proven beneficial for search performance to always foreMerge down
to one segment. However, this takes significant time. Are there any
suggestions on what kind of merge scheduler/policy settings will utilize
the most of the available IO, CPU and RAM capacity? Currently we end up
being single thread bound, leaving lots of potential cpu and bandwidth
not used during the merge.
e.g. we are looking for a MergeEvertyThing use all hardware policy and
scheduler.
We are currently on lucene 7.4 but nothing is stopping us from upgrading.
Regards,
Jerven
---------------------------------------------------------------------
---------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Jerven Tjalling Bolleman
2018-11-02 19:30:27 UTC
Permalink
Hi Dawid, Erick,

Thanks for the reply. We are using pure lucene and currently this is
what I am doing

int processors = Runtime.getRuntime().availableProcessors();
int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
cms.setMaxMergesAndThreads(processors,processors);
cms.disableAutoIOThrottle();

config.setMergeScheduler(cms);

TieredMergePolicy tms = new TieredMergePolicy();
int merges = Math.max(2, processors);
tms.setMaxMergeAtOnce(merges);
tms.setMaxMergeAtOnceExplicit(merges);
tms.setSegmentsPerTier(merges * 2);
config.setMergePolicy(new TieredMergePolicy());
IndexWriter writer = new IndexWriter(this.dir, config);
writer.forceMerge(1, true);

Regarding, seeing a performance decrease without merging. Yes
significant.
That the one time we tried we broke every SLA and haven't tried in long
time.
Of course now our largest index is more than 200 million documents so
perhaps we
should retry this.

38G _583u.fdt
25M _583u.fdx
13K _583u.fnm
47G _583u_Lucene50_0.doc
54G _583u_Lucene50_0.pos
30G _583u_Lucene50_0.tim
413M _583u_Lucene50_0.tip
2.1G _583u_Lucene70_0.dvd
213 _583u_Lucene70_0.dvm

If we did such a max resource merge code would there be interest to have
this merged?

Or should we maybe do something like this assuming 64 cpus

writer.forceMerge(64, true);
writer.forceMerge(32, true);
writer.forceMerge(16, true);
writer.forceMerge(8, true);
writer.forceMerge(4, true);
writer.forceMerge(2, true);
writer.forceMerge(1, true);

Regards,
Jerven
Post by Dawid Weiss
We are faced with a similar situation. Yes, the merge process can take
a long time and is mostly single-threaded (if you're merging from N
segments into a single segment, only one thread does the job). As
Erick pointed out, the merge process takes a backseat compared to
indexing and searches (in most cases), so it's not a priority, but
it's definitely something people like you (and me) could utilize, if
given the opportunity.
I actually don't see any reasons why merging of individual parts of a
segment can't be done in parallel (this would be a start; later on a
splittable strategy of merging single things could make use of things
like the fork-join executor). I'd love to work on this at some point,
but I honestly don't see any time soon this could be happening. If you
have a spare cycle, take a look at how index writer merges a single
segment; there are quite trivial ways this could be split into
parallel subtasks and executed with, for example, the system fork-join
executor (even without forkable tasks).
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L2999-L3007
As a side note, you may want to make absolutely sure your merge
scheduler (if it's the CMS) is not using any I/O throttling -- this is
theoretically self-adjustable, but in practice if you only care about
the wall-clock end of a single merge, it's better to turn it off.
Dawid
Post by Erick Erickson
The merge process is rather tricky, and there's nothing that I know of
that will use all resources available. In fact the merge code is
written to _not_ use up all the possible resources on the theory that
there should be some left over to handle queries etc.
Yeah, the situation you describe is indeed one of the few where
merging down to 1 segment makes sense. Out of curiosity, what kind of
performance gains to you see?
1> there is a limit to the number of segments that can be merged at
once, so sometimes it can take more than one pass. If you have more
than 30 segments, it'll be multi-pass. You can try (and I haven't done
this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
to see if it helps. That only takes effect when you forceMerge.
There's a trick bit of reflection that handles this, see the very end
of TieredMergePolicy.java for the parameters you can set.
2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
from automatically merging down to 1 segment to respecting
"maxMergedSegmentMB" (default 5G). You will have to explicitly pass
maxSegments=1 to get the old behavior.
Best,
Erick
On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
Post by Jerven Bolleman
Dear Lucene Devs and Users,
First of all thank you for this wonderful library and API.
forceMerges are normally not recommended but we fall into one of the few
usecases where it makes sense.
In our use case we have a large index (3 actually) and we don't update
them ever after indexing. i.e. we index all the documents and then never
ever add another document to the index, nor are any deleted.
It has proven beneficial for search performance to always foreMerge down
to one segment. However, this takes significant time. Are there any
suggestions on what kind of merge scheduler/policy settings will utilize
the most of the available IO, CPU and RAM capacity? Currently we end up
being single thread bound, leaving lots of potential cpu and bandwidth
not used during the merge.
e.g. we are looking for a MergeEvertyThing use all hardware policy and
scheduler.
We are currently on lucene 7.4 but nothing is stopping us from upgrading.
Regards,
Jerven
---------------------------------------------------------------------
---------------------------------------------------------------------
---------------------------------------------------------------------
--
Jerven Tjalling Bolleman
SIB | Swiss Institute of Bioinformatics
CMU - 1, rue Michel Servet - 1211 Geneva 4
t: +41 22 379 58 85 - f: +41 22 379 58 58
***@sib.swiss - http://www.sib.swiss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Dawid Weiss
2018-11-02 19:52:29 UTC
Permalink
Post by Jerven Tjalling Bolleman
int processors = Runtime.getRuntime().availableProcessors();
int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
cms.setMaxMergesAndThreads(processors,processors);
See the number of threads in the CMS only matters if you have
concurrent merges of independent segments. What you're doing
effectively forces an eventual X -> 1 merge, which is done by a single
thread (regardless of the max processors above).
Post by Jerven Tjalling Bolleman
38G _583u.fdt
25M _583u.fdx
13K _583u.fnm
47G _583u_Lucene50_0.doc
54G _583u_Lucene50_0.pos
30G _583u_Lucene50_0.tim
413M _583u_Lucene50_0.tip
2.1G _583u_Lucene70_0.dvd
213 _583u_Lucene70_0.dvm
Merging segments as large as this one requires not just CPU, but also
serious I/O throughput efficiency. I assume you have fast NVMe drives
on that machine, otherwise it'll be slow, no matter what. It's just a
lot of bytes going back and forth.
Post by Jerven Tjalling Bolleman
If we did such a max resource merge code would there be interest to have this merged?
I think so. Try to experiment locally first though and see if what you
can find out. Hacking that code I pointed at shouldn't be too
difficult. see what happens.
Post by Jerven Tjalling Bolleman
Or should we maybe do something like this assuming 64 cpus
writer.forceMerge(64, true);
writer.forceMerge(32, true);
writer.forceMerge(16, true);
writer.forceMerge(8, true);
writer.forceMerge(4, true);
writer.forceMerge(2, true);
writer.forceMerge(1, true);
No, this doesn't make much sense. If your goal is 1 segment then you
want to read from as many of them as once as possible and merge into a
single segment. Doing what you did above would only bump I/O traffic a
lot.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Toke Eskildsen
2018-11-02 20:24:41 UTC
Permalink
Post by Dawid Weiss
Merging segments as large as this one requires not just CPU, but also
serious I/O throughput efficiency. I assume you have fast NVMe drives
on that machine, otherwise it'll be slow, no matter what. It's just a
lot of bytes going back and forth.
We have quite a lot of experience with creating fully merged 900GB indexes. On our plain-SSD (Samsung 840) equipped machine this took ~8 hours with a single CPU-core at 100%. On our 7200 RPM spinning drive machines (same CPU class) it took nearly twice as long. Back of the envelope says reading & writing 900GB in 8 hours is 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our SSD machine, but even with SATA II this is only ~1/5th of the possible fairly sequential IO throughput. So for us at least, NVMe drives are not needed to have single-threaded CPU as bottleneck.

And +1 to the issue BTW. It does not matter too much for us now, as we have shifted to a setup where we build more indexes in parallel, but 3 years ago our process was sequential so the 8 hour delay before building the next part was a bit of an annoyance.

- Toke Eskildsen

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Dawid Weiss
2018-11-02 21:17:03 UTC
Permalink
Thanks for chipping in, Toke. A ~1TB index is impressive.

Back of the envelope says reading & writing 900GB in 8 hours is
2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
SSD machine, but even with SATA II this is only ~1/5th of the possible
fairly sequential IO throughput. So for us at least, NVMe drives are
not needed to have single-threaded CPU as bottleneck.

The mileage will vary depending on the CPU -- if it can merge the data
from multiple files at ones fast enough then it may theoretically
saturate the bandwidth... but I agree we also seem to be CPU bound on
these N-to-1 merges, a regular SSD is enough.
Post by Toke Eskildsen
And +1 to the issue BTW.
I agree. Fine-grained granularity here would be a win even in the
regular "merge is a low-priority citizen" case. At least that's what I
tend to think. And if there are spare CPUs, the gain would be
terrific.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Erick Erickson
2018-11-03 20:45:09 UTC
Permalink
Do you really need exactly one segment? Or would, say, 5 be good enough?
You see where this is going, set maxsegments to 5 and maybe be able to get
some parallelization...
Post by Dawid Weiss
Thanks for chipping in, Toke. A ~1TB index is impressive.
Back of the envelope says reading & writing 900GB in 8 hours is
2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
SSD machine, but even with SATA II this is only ~1/5th of the possible
fairly sequential IO throughput. So for us at least, NVMe drives are
not needed to have single-threaded CPU as bottleneck.
The mileage will vary depending on the CPU -- if it can merge the data
from multiple files at ones fast enough then it may theoretically
saturate the bandwidth... but I agree we also seem to be CPU bound on
these N-to-1 merges, a regular SSD is enough.
Post by Toke Eskildsen
And +1 to the issue BTW.
I agree. Fine-grained granularity here would be a win even in the
regular "merge is a low-priority citizen" case. At least that's what I
tend to think. And if there are spare CPUs, the gain would be
terrific.
Dawid
---------------------------------------------------------------------
Jerven Tjalling Bolleman
2018-11-02 20:36:20 UTC
Permalink
Post by Dawid Weiss
Post by Jerven Tjalling Bolleman
int processors = Runtime.getRuntime().availableProcessors();
int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
cms.setMaxMergesAndThreads(processors,processors);
See the number of threads in the CMS only matters if you have
concurrent merges of independent segments. What you're doing
effectively forces an eventual X -> 1 merge, which is done by a single
thread (regardless of the max processors above).
Post by Jerven Tjalling Bolleman
38G _583u.fdt
25M _583u.fdx
13K _583u.fnm
47G _583u_Lucene50_0.doc
54G _583u_Lucene50_0.pos
30G _583u_Lucene50_0.tim
413M _583u_Lucene50_0.tip
2.1G _583u_Lucene70_0.dvd
213 _583u_Lucene70_0.dvm
Merging segments as large as this one requires not just CPU, but also
serious I/O throughput efficiency. I assume you have fast NVMe drives
on that machine, otherwise it'll be slow, no matter what. It's just a
lot of bytes going back and forth.
Yup, it's now cloud so optimizing for quick index and then merge to one
has become financially interesting. Now it's too much cpu and ram being
idle. Nor even maxing out the disk io (about 25% of max rate).
Post by Dawid Weiss
Post by Jerven Tjalling Bolleman
If we did such a max resource merge code would there be interest to have this merged?
I think so. Try to experiment locally first though and see if what you
can find out. Hacking that code I pointed at shouldn't be too
difficult. see what happens.
Yeah, before I left I started with an experiment to have one running
without the
merge scheduler being involved at all.

Will try a few more experiments next week.
Post by Dawid Weiss
Post by Jerven Tjalling Bolleman
Or should we maybe do something like this assuming 64 cpus
writer.forceMerge(64, true);
writer.forceMerge(32, true);
writer.forceMerge(16, true);
writer.forceMerge(8, true);
writer.forceMerge(4, true);
writer.forceMerge(2, true);
writer.forceMerge(1, true);
No, this doesn't make much sense. If your goal is 1 segment then you
want to read from as many of them as once as possible and merge into a
single segment. Doing what you did above would only bump I/O traffic a
lot.
Thanks, I always thought so but wasn't sure anymore.

Have a nice weekend everyone!
Post by Dawid Weiss
Dawid
---------------------------------------------------------------------
--
Jerven Tjalling Bolleman
SIB | Swiss Institute of Bioinformatics
CMU - 1, rue Michel Servet - 1211 Geneva 4
t: +41 22 379 58 85 - f: +41 22 379 58 58
***@sib.swiss - http://www.sib.swiss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-***@lucene.apache.org
For additional commands, e-mail: java-user-***@lucene.apache.org
Loading...