folly::gen::StreamSplitter< Callback > Class Template Reference

#include <String.h>

Public Member Functions
	StreamSplitter (char delimiter, Callback &&pieceCb, uint64_t maxLength=0, uint64_t initialCapacity=0)
bool	flush ()
bool	operator() (StringPiece in)

Private Attributes
IOBuf	buffer_
char	delimiter_
uint64_t	maxLength_
Callback	pieceCb_

Detailed Description

template<class Callback>
class folly::gen::StreamSplitter< Callback >

Outputs exactly the same bytes as the input stream, in different chunks. A chunk boundary occurs after each delimiter, or, if maxLength is non-zero, after maxLength bytes, whichever comes first. Your callback can return false to stop consuming the stream at any time.

The splitter buffers the last incomplete chunk, so you must call flush() to consume the piece of the stream after the final delimiter. This piece may be empty. After a flush(), the splitter can be re-used for a new stream.

operator() and flush() return false iff your callback returns false. The internal buffer is not flushed, so reusing such a splitter will have indeterminate results. Same goes if your callback throws. Feel free to fix these corner cases if needed.

Tips:

• Create via streamSplitter() to take advantage of template deduction.
• If your callback needs an end-of-stream signal, test for "no trailing delimiter **and** shorter than maxLength".
• You can fine-tune the initial capacity of the internal IOBuf.

Definition at line 196 of file String.h.

Constructor & Destructor Documentation

template<class Callback>

folly::gen::StreamSplitter< Callback >::StreamSplitter ( char  delimiter, Callback &&  pieceCb, uint64_t  maxLength = 0, uint64_t  initialCapacity = 0  )

inline

Definition at line 198 of file String.h.

      : buffer_(IOBuf::CREATE, initialCapacity),
        delimiter_(delimiter),
        maxLength_(maxLength),
        pieceCb_(std::move(pieceCb)) {}

Member Function Documentation

template<class Callback >

bool folly::gen::StreamSplitter< Callback >::flush

(

)

Consume any incomplete last line (may be empty). Do this before destroying the StreamSplitter, or you will fail to consume part of the input.

After flush() you may proceed to consume the next stream via ().

Returns false if the callback wants no more data, true otherwise. A return value of false means that this splitter must no longer be used.

Definition at line 128 of file String-inl.h.

References buffer_, and folly::gen::detail::ch().

                                     {
  CHECK(maxLength_ == 0 || buffer_.length() < maxLength_);
  if (!pieceCb_(StringPiece(detail::ch(buffer_.data()), buffer_.length()))) {
    return false;
  }
  // We are ready to handle another stream now.
  buffer_.clear();
  return true;
}

template<class Callback >

bool folly::gen::StreamSplitter< Callback >::operator()

(

StringPiece

in

)

Consume another piece of the input stream.

Returns false only if your callback refuses to consume more data by returning false (true otherwise). A return value of false means that this splitter must no longer be used.

Definition at line 139 of file String-inl.h.

References folly::Range< Iter >::advance(), folly::Range< Iter >::begin(), buffer_, folly::gen::detail::consumeBufferPlus(), folly::gen::detail::consumeFixedSizeChunks(), folly::Range< Iter >::data(), folly::Range< Iter >::empty(), min, prefix(), folly::Range< Iter >::size(), folly::gen::detail::splitPrefix(), and uint64_t.

                                                        {
  StringPiece prefix;
  // NB This code assumes a 1-byte delimiter. It's not too hard to support
  // multibyte delimiters, just remember that maxLength_ chunks can end up
  // falling in the middle of a delimiter.
  bool found = detail::splitPrefix(in, prefix, delimiter_);
  if (buffer_.length() != 0) {
    if (found) {
      uint64_t num_to_add = prefix.size();
      if (maxLength_) {
        CHECK(buffer_.length() < maxLength_);
        // Consume as much of prefix as possible without exceeding maxLength_
        num_to_add = std::min(maxLength_ - buffer_.length(), num_to_add);
      }

      // Append part of the prefix to the buffer, and send it to the callback
      if (!detail::consumeBufferPlus(pieceCb_, buffer_, prefix, num_to_add)) {
        return false;
      }

      if (!detail::consumeFixedSizeChunks(pieceCb_, prefix, maxLength_)) {
        return false;
      }

      found = detail::splitPrefix(in, prefix, delimiter_);
      // Post-conditions:
      //  - we consumed all of buffer_ and all of the first prefix.
      //  - found, in, and prefix reflect the second delimiter_ search
    } else if (maxLength_ && buffer_.length() + in.size() >= maxLength_) {
      // Send all of buffer_, plus a bit of in, to the callback
      if (!detail::consumeBufferPlus(
              pieceCb_, buffer_, in, maxLength_ - buffer_.length())) {
        return false;
      }
      // Post-conditions:
      //  - we consumed all of buffer, and the minimal # of bytes from in
      //  - found is false
    } // Otherwise: found is false & we cannot invoke the callback this turn
  }
  // Post-condition: buffer_ is nonempty only if found is false **and**
  // len(buffer + in) < maxLength_.

  // Send lines to callback directly from input (no buffer)
  while (found) { // Buffer guaranteed to be empty
    if (!detail::consumeFixedSizeChunks(pieceCb_, prefix, maxLength_)) {
      return false;
    }
    found = detail::splitPrefix(in, prefix, delimiter_);
  }

  // No more delimiters left; consume 'in' until it is shorter than maxLength_
  if (maxLength_) {
    while (in.size() >= maxLength_) { // Buffer is guaranteed to be empty
      if (!pieceCb_(StringPiece(in.begin(), maxLength_))) {
        return false;
      }
      in.advance(maxLength_);
    }
  }

  if (!in.empty()) { // Buffer may be nonempty
    // Incomplete line left, append to buffer
    buffer_.reserve(0, in.size());
    memcpy(buffer_.writableTail(), in.data(), in.size());
    buffer_.append(in.size());
  }
  CHECK(maxLength_ == 0 || buffer_.length() < maxLength_);
  return true;
}

Member Data Documentation

template<class Callback>

IOBuf folly::gen::StreamSplitter< Callback >::buffer_

private

Definition at line 231 of file String.h.

template<class Callback>

char folly::gen::StreamSplitter< Callback >::delimiter_

private

Definition at line 232 of file String.h.

template<class Callback>

uint64_t folly::gen::StreamSplitter< Callback >::maxLength_

private

Definition at line 233 of file String.h.

template<class Callback>

Callback folly::gen::StreamSplitter< Callback >::pieceCb_

private

Definition at line 234 of file String.h.

---

The documentation for this class was generated from the following files:

• proxygen/folly/folly/gen/String.h
• proxygen/folly/folly/gen/String-inl.h