61 using TokenStream::reset;
#define LUCENE_CLASS(Name)
Definition LuceneObject.h:24
A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text.
Definition TokenStream.h:44
A Tokenizer is a TokenStream whose input is a Reader.
Definition Tokenizer.h:20
CharStreamPtr charStream
Definition Tokenizer.h:48
Tokenizer()
Construct a tokenizer with null input.
virtual int32_t correctOffset(int32_t currentOff)
Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOf...
virtual void close()
By default, closes the input Reader.
Tokenizer(const ReaderPtr &input)
Construct a token stream processing the given input.
Tokenizer(const AttributeSourcePtr &source)
Construct a token stream processing the given input using the given AttributeSource.
Tokenizer(const AttributeFactoryPtr &factory)
Construct a tokenizer with null input using the given AttributeFactory.
ReaderPtr input
The text source for this Tokenizer.
Definition Tokenizer.h:47
virtual void reset(const ReaderPtr &input)
Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will ...
Tokenizer(const AttributeSourcePtr &source, const ReaderPtr &input)
Construct a token stream processing the given input using the given AttributeSource.
Tokenizer(const AttributeFactoryPtr &factory, const ReaderPtr &input)
Construct a token stream processing the given input using the given AttributeFactory.
Definition AbstractAllTermDocs.h:12
boost::shared_ptr< AttributeSource > AttributeSourcePtr
Definition LuceneTypes.h:520
boost::shared_ptr< Reader > ReaderPtr
Definition LuceneTypes.h:547
boost::shared_ptr< AttributeFactory > AttributeFactoryPtr
Definition LuceneTypes.h:519
boost::shared_ptr< CharStream > CharStreamPtr
Definition LuceneTypes.h:27