Lucene++ - a full-featured, c++ search engine
API Documentation


Loading...
Searching...
No Matches
Public Member Functions | Static Public Member Functions | Protected Member Functions | Protected Attributes | Static Protected Attributes
Lucene::CharTokenizer Class Referenceabstract

An abstract base class for simple, character-oriented tokenizers. More...

#include <CharTokenizer.h>

+ Inheritance diagram for Lucene::CharTokenizer:

Public Member Functions

 CharTokenizer (const ReaderPtr &input)
 
 CharTokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
 
 CharTokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
 
virtual ~CharTokenizer ()
 
virtual String getClassName ()
 
boost::shared_ptr< CharTokenizershared_from_this ()
 
virtual bool incrementToken ()
 Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token.
 
virtual void end ()
 This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.
 
virtual void reset (const ReaderPtr &input)
 Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.
 
- Public Member Functions inherited from Lucene::Tokenizer
virtual ~Tokenizer ()
 
boost::shared_ptr< Tokenizershared_from_this ()
 
virtual void close ()
 By default, closes the input Reader.
 
virtual int32_t correctOffset (int32_t currentOff)
 Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOffset, else returns currentOff.
 
virtual void reset ()
 Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).
 
- Public Member Functions inherited from Lucene::TokenStream
virtual ~TokenStream ()
 
boost::shared_ptr< TokenStreamshared_from_this ()
 
- Public Member Functions inherited from Lucene::AttributeSource
 AttributeSource ()
 An AttributeSource using the default attribute factory DefaultAttributeFactory.
 
 AttributeSource (const AttributeSourcePtr &input)
 An AttributeSource that uses the same attributes as the supplied one.
 
 AttributeSource (const AttributeFactoryPtr &factory)
 An AttributeSource using the supplied AttributeFactory for creating new Attribute instances.
 
virtual ~AttributeSource ()
 
boost::shared_ptr< AttributeSourceshared_from_this ()
 
AttributeFactoryPtr getAttributeFactory ()
 returns the used AttributeFactory.
 
template<class ATTR >
boost::shared_ptr< ATTR > addAttribute ()
 This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.
 
void addAttribute (const String &className, const AttributePtr &attrImpl)
 Adds a custom Attribute instance.
 
bool hasAttributes ()
 Returns true if this AttributeSource has any attributes.
 
template<class ATTR >
bool hasAttribute ()
 Returns true, if this AttributeSource contains the passed-in Attribute.
 
template<class ATTR >
boost::shared_ptr< ATTR > getAttribute ()
 Returns the instance of the passed in Attribute contained in this AttributeSource.
 
void clearAttributes ()
 Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation.
 
AttributeSourceStatePtr captureState ()
 Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource.
 
void restoreState (const AttributeSourceStatePtr &state)
 Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).
 
virtual int32_t hashCode ()
 Return hash code for this object.
 
virtual bool equals (const LuceneObjectPtr &other)
 Return whether two objects are equal.
 
virtual String toString ()
 Returns a string representation of the object.
 
AttributeSourcePtr cloneAttributes ()
 Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using AttributeSource(AttributeSource))
 
Collection< AttributePtrgetAttributes ()
 Return a vector of attributes based on currentState.
 
- Public Member Functions inherited from Lucene::LuceneObject
virtual ~LuceneObject ()
 
virtual void initialize ()
 Called directly after instantiation to create objects that depend on this object being fully constructed.
 
virtual LuceneObjectPtr clone (const LuceneObjectPtr &other=LuceneObjectPtr())
 Return clone of this object.
 
virtual int32_t compareTo (const LuceneObjectPtr &other)
 Compare two objects.
 
- Public Member Functions inherited from Lucene::LuceneSync
virtual ~LuceneSync ()
 
virtual SynchronizePtr getSync ()
 Return this object synchronize lock.
 
virtual LuceneSignalPtr getSignal ()
 Return this object signal.
 
virtual void lock (int32_t timeout=0)
 Lock this object using an optional timeout.
 
virtual void unlock ()
 Unlock this object.
 
virtual bool holdsLock ()
 Returns true if this object is currently locked by current thread.
 
virtual void wait (int32_t timeout=0)
 Wait for signal using an optional timeout.
 
virtual void notifyAll ()
 Notify all threads waiting for signal.
 

Static Public Member Functions

static String _getClassName ()
 
- Static Public Member Functions inherited from Lucene::Tokenizer
static String _getClassName ()
 
- Static Public Member Functions inherited from Lucene::TokenStream
static String _getClassName ()
 
- Static Public Member Functions inherited from Lucene::AttributeSource
static String _getClassName ()
 

Protected Member Functions

virtual bool isTokenChar (wchar_t c)=0
 Returns true if a character should be included in a token. This tokenizer generates as tokens adjacent sequences of characters which satisfy this predicate. Characters for which this is false are used to define token boundaries and are not included in tokens.
 
virtual wchar_t normalize (wchar_t c)
 Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, eg., lowercase tokens.
 
- Protected Member Functions inherited from Lucene::Tokenizer
 Tokenizer ()
 Construct a tokenizer with null input.
 
 Tokenizer (const ReaderPtr &input)
 Construct a token stream processing the given input.
 
 Tokenizer (const AttributeFactoryPtr &factory)
 Construct a tokenizer with null input using the given AttributeFactory.
 
 Tokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
 Construct a token stream processing the given input using the given AttributeFactory.
 
 Tokenizer (const AttributeSourcePtr &source)
 Construct a token stream processing the given input using the given AttributeSource.
 
 Tokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
 Construct a token stream processing the given input using the given AttributeSource.
 
- Protected Member Functions inherited from Lucene::TokenStream
 TokenStream ()
 A TokenStream using the default attribute factory.
 
 TokenStream (const AttributeSourcePtr &input)
 A TokenStream that uses the same attributes as the supplied one.
 
 TokenStream (const AttributeFactoryPtr &factory)
 A TokenStream using the supplied AttributeFactory for creating new Attribute instances.
 
- Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr getAttribute (const String &className)
 The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it.
 
bool hasAttribute (const String &className)
 Returns true, if this AttributeSource contains the passed-in Attribute.
 
void computeCurrentState ()
 
- Protected Member Functions inherited from Lucene::LuceneObject
 LuceneObject ()
 

Protected Attributes

int32_t offset
 
int32_t bufferIndex
 
int32_t dataLen
 
CharArray ioBuffer
 
TermAttributePtr termAtt
 
OffsetAttributePtr offsetAtt
 
- Protected Attributes inherited from Lucene::Tokenizer
ReaderPtr input
 The text source for this Tokenizer.
 
CharStreamPtr charStream
 
- Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr factory
 
MapStringAttribute attributes
 
AttributeSourceStatePtr currentState
 
- Protected Attributes inherited from Lucene::LuceneSync
SynchronizePtr objectLock
 
LuceneSignalPtr objectSignal
 

Static Protected Attributes

static const int32_t MAX_WORD_LEN
 
static const int32_t IO_BUFFER_SIZE
 

Detailed Description

An abstract base class for simple, character-oriented tokenizers.

Constructor & Destructor Documentation

◆ CharTokenizer() [1/3]

Lucene::CharTokenizer::CharTokenizer ( const ReaderPtr input)

◆ CharTokenizer() [2/3]

Lucene::CharTokenizer::CharTokenizer ( const AttributeSourcePtr source,
const ReaderPtr input 
)

◆ CharTokenizer() [3/3]

Lucene::CharTokenizer::CharTokenizer ( const AttributeFactoryPtr factory,
const ReaderPtr input 
)

◆ ~CharTokenizer()

virtual Lucene::CharTokenizer::~CharTokenizer ( )
virtual

Member Function Documentation

◆ _getClassName()

static String Lucene::CharTokenizer::_getClassName ( )
inlinestatic

◆ end()

virtual void Lucene::CharTokenizer::end ( )
virtual

This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Reimplemented from Lucene::TokenStream.

◆ getClassName()

virtual String Lucene::CharTokenizer::getClassName ( )
inlinevirtual

◆ incrementToken()

virtual bool Lucene::CharTokenizer::incrementToken ( )
virtual

Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use captureState to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to addAttribute(Class) and getAttribute(Class), references to all Attributes that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in incrementToken().

Returns
false for end of stream; true otherwise

Implements Lucene::TokenStream.

◆ isTokenChar()

virtual bool Lucene::CharTokenizer::isTokenChar ( wchar_t  c)
protectedpure virtual

Returns true if a character should be included in a token. This tokenizer generates as tokens adjacent sequences of characters which satisfy this predicate. Characters for which this is false are used to define token boundaries and are not included in tokens.

Implemented in Lucene::LetterTokenizer, and Lucene::WhitespaceTokenizer.

◆ normalize()

virtual wchar_t Lucene::CharTokenizer::normalize ( wchar_t  c)
protectedvirtual

Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, eg., lowercase tokens.

Reimplemented in Lucene::LowerCaseTokenizer.

◆ reset()

virtual void Lucene::CharTokenizer::reset ( const ReaderPtr input)
virtual

Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Reimplemented from Lucene::Tokenizer.

◆ shared_from_this()

boost::shared_ptr< CharTokenizer > Lucene::CharTokenizer::shared_from_this ( )
inline

Field Documentation

◆ bufferIndex

int32_t Lucene::CharTokenizer::bufferIndex
protected

◆ dataLen

int32_t Lucene::CharTokenizer::dataLen
protected

◆ IO_BUFFER_SIZE

const int32_t Lucene::CharTokenizer::IO_BUFFER_SIZE
staticprotected

◆ ioBuffer

CharArray Lucene::CharTokenizer::ioBuffer
protected

◆ MAX_WORD_LEN

const int32_t Lucene::CharTokenizer::MAX_WORD_LEN
staticprotected

◆ offset

int32_t Lucene::CharTokenizer::offset
protected

◆ offsetAtt

OffsetAttributePtr Lucene::CharTokenizer::offsetAtt
protected

◆ termAtt

TermAttributePtr Lucene::CharTokenizer::termAtt
protected

The documentation for this class was generated from the following file:

clucene.sourceforge.net