Discussion:
Issue with FacetFields.addFields() throwing ArrayIndexOutOfBoundsException
(too old to reply)
Matthew Petersen
2014-01-16 22:33:33 UTC
Permalink
I’m having an issue with an index when adding category paths to a document.
They seem to be added without issue for a long period of time, then for
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the same
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix the
index because according to lucene it is not broken. Could this be a bug in
lucene? Below is the stack trace when the exception occurs:


Lucene v4.4.0


java.lang.ArrayIndexOutOfBoundsException: -65535

at java.util.ArrayList.elementData(ArrayList.java:371)

at java.util.ArrayList.get(ArrayList.java:384)

at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)

at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)

at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)

at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)

at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)

at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)

at org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)

at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)

at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)

at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)

at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)

at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)

at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)

at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)

at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)

at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)

at org.junit.runners.ParentRunner.run(ParentRunner.java:300)

at org.junit.runner.JUnitCore.run(JUnitCore.java:157)

at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)

at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)

at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Michael McCandless
2014-01-17 13:36:19 UTC
Permalink
Are you sure you're using 4.4?

Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.

Mike McCandless

http://blog.mikemccandless.com
I’m having an issue with an index when adding category paths to a document.
They seem to be added without issue for a long period of time, then for
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the same
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix the
index because according to lucene it is not broken. Could this be a bug in
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
at org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Matthew Petersen
2014-01-17 16:29:22 UTC
Permalink
I'm sure. I had seen that issue and it looked similar but the stack trace
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem seems
to go away. I'm working right now on running a test that will prove this
but it takes a while as the cache needs to get very large. If this proves
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.

Thanks for the response.
Matt


On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to a
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time, then for
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the same
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix the
index because according to lucene it is not broken. Could this be a bug
in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
---------------------------------------------------------------------
Matthew Petersen
2014-01-17 21:59:57 UTC
Permalink
I've confirmed that using the LruTaxonomyWriterCache solves the issue for
me. It would appear there is in fact a bug in the Cl20TaxonomyWriterCache
or I am using it incorrectly (I use it as default, no customization).
Post by Matthew Petersen
I'm sure. I had seen that issue and it looked similar but the stack trace
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem seems
to go away. I'm working right now on running a test that will prove this
but it takes a while as the cache needs to get very large. If this proves
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.
Thanks for the response.
Matt
On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to a
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time, then for
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the same
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix the
index because according to lucene it is not broken. Could this be a
bug in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
---------------------------------------------------------------------
Shai Erera
2014-01-17 22:27:24 UTC
Permalink
Do you have a test which reproduces the error? Are you adding categories
with very deep hierarchies?

Shai
Post by Matthew Petersen
I've confirmed that using the LruTaxonomyWriterCache solves the issue for
me. It would appear there is in fact a bug in the Cl20TaxonomyWriterCache
or I am using it incorrectly (I use it as default, no customization).
Post by Matthew Petersen
I'm sure. I had seen that issue and it looked similar but the stack
trace
Post by Matthew Petersen
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem seems
to go away. I'm working right now on running a test that will prove this
but it takes a while as the cache needs to get very large. If this
proves
Post by Matthew Petersen
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.
Thanks for the response.
Matt
On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to a
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time, then
for
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the
same
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix
the
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
index because according to lucene it is not broken. Could this be a
bug in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Post by Matthew Petersen
Post by Michael McCandless
---------------------------------------------------------------------
Matthew Petersen
2014-01-17 22:40:23 UTC
Permalink
I do have a test that will reproduce. I'm not adding categories with very
deep hierarchies, I'm adding 129 category paths per document (all docs have
paths with same label) with each path having one value. All of the values
are completely random and likely unique. It's basically a worst case test
for our app but the condition has been seen in the field (the error has
been encountered at less than worst case scenario). The test i have
reproduces it very quickly, Only have to index ~330K docs.
Post by Shai Erera
Do you have a test which reproduces the error? Are you adding categories
with very deep hierarchies?
Shai
Post by Matthew Petersen
I've confirmed that using the LruTaxonomyWriterCache solves the issue for
me. It would appear there is in fact a bug in the
Cl20TaxonomyWriterCache
Post by Matthew Petersen
or I am using it incorrectly (I use it as default, no customization).
Post by Matthew Petersen
I'm sure. I had seen that issue and it looked similar but the stack
trace
Post by Matthew Petersen
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem
seems
Post by Matthew Petersen
Post by Matthew Petersen
to go away. I'm working right now on running a test that will prove
this
Post by Matthew Petersen
Post by Matthew Petersen
but it takes a while as the cache needs to get very large. If this
proves
Post by Matthew Petersen
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.
Thanks for the response.
Matt
On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to a
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time, then
for
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
some unknown reason the addition fails with an ArrayIndexOutOfBounds
exception. Subsequent attempts to add category paths fail with the
same
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot fix
the
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
index because according to lucene it is not broken. Could this be a
bug in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Post by Matthew Petersen
Post by Michael McCandless
---------------------------------------------------------------------
Shai Erera
2014-01-18 04:24:03 UTC
Permalink
Can you open an issue and attach the test there?
Post by Matthew Petersen
I do have a test that will reproduce. I'm not adding categories with very
deep hierarchies, I'm adding 129 category paths per document (all docs have
paths with same label) with each path having one value. All of the values
are completely random and likely unique. It's basically a worst case test
for our app but the condition has been seen in the field (the error has
been encountered at less than worst case scenario). The test i have
reproduces it very quickly, Only have to index ~330K docs.
Post by Shai Erera
Do you have a test which reproduces the error? Are you adding categories
with very deep hierarchies?
Shai
Post by Matthew Petersen
I've confirmed that using the LruTaxonomyWriterCache solves the issue
for
Post by Shai Erera
Post by Matthew Petersen
me. It would appear there is in fact a bug in the
Cl20TaxonomyWriterCache
Post by Matthew Petersen
or I am using it incorrectly (I use it as default, no customization).
Post by Matthew Petersen
I'm sure. I had seen that issue and it looked similar but the stack
trace
Post by Matthew Petersen
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem
seems
Post by Matthew Petersen
Post by Matthew Petersen
to go away. I'm working right now on running a test that will prove
this
Post by Matthew Petersen
Post by Matthew Petersen
but it takes a while as the cache needs to get very large. If this
proves
Post by Matthew Petersen
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.
Thanks for the response.
Matt
On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Jan 16, 2014 at 5:33 PM, Matthew Petersen <
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to a
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time,
then
Post by Shai Erera
Post by Matthew Petersen
for
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
some unknown reason the addition fails with an
ArrayIndexOutOfBounds
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. Subsequent attempts to add category paths fail with
the
Post by Shai Erera
Post by Matthew Petersen
same
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. I’ve run CheckIndex on both the index and the taxonomy
directory and both come back as clean with no issues. I cannot
fix
Post by Shai Erera
Post by Matthew Petersen
the
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
index because according to lucene it is not broken. Could this
be a
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
bug in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
---------------------------------------------------------------------
Matthew Petersen
2014-01-21 15:35:42 UTC
Permalink
Will do. I need to sanitize the unit test a bit so it might be a few days.
Post by Shai Erera
Can you open an issue and attach the test there?
Post by Matthew Petersen
I do have a test that will reproduce. I'm not adding categories with
very
Post by Matthew Petersen
deep hierarchies, I'm adding 129 category paths per document (all docs
have
Post by Matthew Petersen
paths with same label) with each path having one value. All of the
values
Post by Matthew Petersen
are completely random and likely unique. It's basically a worst case
test
Post by Matthew Petersen
for our app but the condition has been seen in the field (the error has
been encountered at less than worst case scenario). The test i have
reproduces it very quickly, Only have to index ~330K docs.
Post by Shai Erera
Do you have a test which reproduces the error? Are you adding
categories
Post by Matthew Petersen
Post by Shai Erera
with very deep hierarchies?
Shai
Post by Matthew Petersen
I've confirmed that using the LruTaxonomyWriterCache solves the issue
for
Post by Shai Erera
Post by Matthew Petersen
me. It would appear there is in fact a bug in the
Cl20TaxonomyWriterCache
Post by Matthew Petersen
or I am using it incorrectly (I use it as default, no customization).
Post by Matthew Petersen
I'm sure. I had seen that issue and it looked similar but the
stack
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
trace
Post by Matthew Petersen
is slightly different. I've found that if I replace the
Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem
seems
Post by Matthew Petersen
Post by Matthew Petersen
to go away. I'm working right now on running a test that will
prove
Post by Matthew Petersen
Post by Shai Erera
this
Post by Matthew Petersen
Post by Matthew Petersen
but it takes a while as the cache needs to get very large. If this
proves
Post by Matthew Petersen
to solve the problem then I'd say there is still a bug in the
Cl2oTaxonomyWriterCache implementation.
Thanks for the response.
Matt
On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless <
Post by Michael McCandless
Are you sure you're using 4.4?
Because ... this looks like
https://issues.apache.org/jira/browse/LUCENE-5048 but that was
supposedly fixed in 4.4.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Jan 16, 2014 at 5:33 PM, Matthew Petersen <
Post by Matthew Petersen
I’m having an issue with an index when adding category paths to
a
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
document.
Post by Matthew Petersen
They seem to be added without issue for a long period of time,
then
Post by Shai Erera
Post by Matthew Petersen
for
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
some unknown reason the addition fails with an
ArrayIndexOutOfBounds
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. Subsequent attempts to add category paths fail with
the
Post by Shai Erera
Post by Matthew Petersen
same
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
exception. I’ve run CheckIndex on both the index and the
taxonomy
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
directory and both come back as clean with no issues. I cannot
fix
Post by Shai Erera
Post by Matthew Petersen
the
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
index because according to lucene it is not broken. Could this
be a
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
bug in
Post by Matthew Petersen
Lucene v4.4.0
java.lang.ArrayIndexOutOfBoundsException: -65535
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Matthew Petersen
Post by Shai Erera
Post by Matthew Petersen
Post by Matthew Petersen
Post by Michael McCandless
Post by Matthew Petersen
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
---------------------------------------------------------------------
Loading...