public class OfflineSorter
extends java.lang.Object
sort(String)
Modifier and Type | Class and Description |
---|---|
static class |
OfflineSorter.BufferSize
A bit more descriptive unit for constructors.
|
static class |
OfflineSorter.ByteSequencesReader
Utility class to read length-prefixed byte[] entries from an input.
|
static class |
OfflineSorter.ByteSequencesWriter
Utility class to emit length-prefixed byte[] entries to an output stream for sorting.
|
(package private) static class |
OfflineSorter.FileAndTop |
class |
OfflineSorter.SortInfo
Sort info (debugging mostly).
|
Modifier and Type | Field and Description |
---|---|
static long |
ABSOLUTE_MIN_SORT_BUFFER_SIZE
Absolute minimum required buffer size for sorting.
|
private SortableBytesRefArray |
buffer |
private Counter |
bufferBytesUsed |
private java.util.Comparator<BytesRef> |
comparator |
static java.util.Comparator<BytesRef> |
DEFAULT_COMPARATOR
Default comparator: sorts in binary (codepoint) order
|
private Directory |
dir |
static long |
GB
Convenience constant for gigabytes
|
static int |
MAX_TEMPFILES
Maximum number of temporary files before doing an intermediate merge.
|
private int |
maxTempFiles |
static long |
MB
Convenience constant for megabytes
|
static long |
MIN_BUFFER_SIZE_MB
Minimum recommended buffer size for sorting.
|
private static java.lang.String |
MIN_BUFFER_SIZE_MSG |
private OfflineSorter.BufferSize |
ramBufferSize |
(package private) OfflineSorter.SortInfo |
sortInfo |
private java.lang.String |
tempFileNamePrefix |
private int |
valueLength |
Constructor and Description |
---|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix)
Defaults constructor.
|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix,
java.util.Comparator<BytesRef> comparator)
Defaults constructor with a custom comparator.
|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix,
java.util.Comparator<BytesRef> comparator,
OfflineSorter.BufferSize ramBufferSize,
int maxTempfiles,
int valueLength)
All-details constructor.
|
Modifier and Type | Method and Description |
---|---|
java.util.Comparator<BytesRef> |
getComparator()
Returns the comparator in use to sort entries
|
Directory |
getDirectory()
Returns the
Directory we use to create temp files. |
protected OfflineSorter.ByteSequencesReader |
getReader(ChecksumIndexInput in,
java.lang.String name)
Subclasses can override to change how byte sequences are read from disk.
|
java.lang.String |
getTempFileNamePrefix()
Returns the temp file name prefix passed to
Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files. |
protected OfflineSorter.ByteSequencesWriter |
getWriter(IndexOutput out)
Subclasses can override to change how byte sequences are written to disk.
|
(package private) void |
mergePartitions(Directory trackingDir,
java.util.List<java.lang.String> segments)
Merge the most recent
maxTempFile partitions into a new partition. |
(package private) int |
readPartition(OfflineSorter.ByteSequencesReader reader)
Read in a single partition of data
|
java.lang.String |
sort(java.lang.String inputFileName)
Sort input to a new temp file, returning its name.
|
protected java.lang.String |
sortPartition(TrackingDirectoryWrapper trackingDir)
Sort a single partition in-memory.
|
private void |
verifyChecksum(java.lang.Throwable priorException,
OfflineSorter.ByteSequencesReader reader)
Called on exception, to check whether the checksum is also corrupt in this source, and add that
information (checksum matched or didn't) as a suppressed exception.
|
public static final long MB
public static final long GB
public static final long MIN_BUFFER_SIZE_MB
public static final long ABSOLUTE_MIN_SORT_BUFFER_SIZE
private static final java.lang.String MIN_BUFFER_SIZE_MSG
public static final int MAX_TEMPFILES
private final Directory dir
private final int valueLength
private final java.lang.String tempFileNamePrefix
private final OfflineSorter.BufferSize ramBufferSize
private final Counter bufferBytesUsed
private final SortableBytesRefArray buffer
OfflineSorter.SortInfo sortInfo
private int maxTempFiles
private final java.util.Comparator<BytesRef> comparator
public static final java.util.Comparator<BytesRef> DEFAULT_COMPARATOR
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix) throws java.io.IOException
java.io.IOException
OfflineSorter.BufferSize.automatic()
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator) throws java.io.IOException
java.io.IOException
OfflineSorter.BufferSize.automatic()
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator, OfflineSorter.BufferSize ramBufferSize, int maxTempfiles, int valueLength)
valueLength
is -1 (the default), the length of each value differs; otherwise,
all values have the specified length.public java.lang.String getTempFileNamePrefix()
Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext)
to generate temporary files.public java.lang.String sort(java.lang.String inputFileName) throws java.io.IOException
java.io.IOException
protected java.lang.String sortPartition(TrackingDirectoryWrapper trackingDir) throws java.io.IOException
java.io.IOException
private void verifyChecksum(java.lang.Throwable priorException, OfflineSorter.ByteSequencesReader reader) throws java.io.IOException
java.io.IOException
void mergePartitions(Directory trackingDir, java.util.List<java.lang.String> segments) throws java.io.IOException
maxTempFile
partitions into a new partition.java.io.IOException
int readPartition(OfflineSorter.ByteSequencesReader reader) throws java.io.IOException
java.io.IOException
protected OfflineSorter.ByteSequencesWriter getWriter(IndexOutput out) throws java.io.IOException
java.io.IOException
protected OfflineSorter.ByteSequencesReader getReader(ChecksumIndexInput in, java.lang.String name) throws java.io.IOException
java.io.IOException
public java.util.Comparator<BytesRef> getComparator()