topic "Huge"; [i448;a25;kKO9;2 $$1,0#37138531426314131252341829483380:class] [l288;2 $$2,2#27521748481378242620020725143825:desc] [0 $$3,0#96390100711032703541132217272105:end] [H6;0 $$4,0#05600065144404261032431302351956:begin] [i448;a25;kKO9;2 $$5,0#37138531426314131252341829483370:item] [l288;a4;*@5;1 $$6,6#70004532496200323422659154056402:requirement] [l288;i1121;b17;O9;~~~.1408;2 $$7,0#10431211400427159095818037425705:param] [i448;b42;O9;2 $$8,8#61672508125594000341940100500538:tparam] [b42;2 $$9,9#13035079074754324216151401829390:normal] [2 $$0,0#00000000000000000000000000000000:Default] [{_}%EN-US [ {{10000@(113.42.0) [s0; [*@7;4 Huge]]}}&] [s3;%- &] [s1;:Upp`:`:Huge`:`:class:%- [@(0.0.255)3 class][3 _][*3 Huge]&] [s2; This class is intended for dealing of large raw binary data. The main use is supposed to be in serialization loading, where it is used to preload large binary data (e.g. String or Image) as means for protection against invalid input `- Huge is storing data in 1MB chunks, so is likely to run to the end of stream without allocating invalid amounts of data.&] [s2; To explain further, the problem we are trying to solve here is&] [s2; &] [s2; [C void Foo`::Serialize(Stream`& s)]&] [s2; [C `{]&] [s2; [C int len;]&] [s2; [C s % len;]&] [s2; [C if(s.IsLoading()) `{]&] [s2; [C ptr `= new byte`[len`];]&] [s2; [C s.GetAll(ptr, len);]&] [s2; [C `}]&] [s2; [C .....]&] [s2; [C `}]&] [s2; &] [s2; now if input stream is invalid, the allocation is likely to fail. We could prevent this by examining the amount of remaining data in the stream, but for certain class of streams (e.g. compressed ones) we do not have that info. [* Huge] solves the problem by serving as temporary buffer for loaded data:&] [s2; &] [s2; [C void Foo`::Serialize(Stream`& s)]&] [s2; [C `{]&] [s2; [C int len;]&] [s2; [C s % len;]&] [s2; [C if(s.IsLoading()) `{]&] [s2; [C -| Huge h;]&] [s2; [C s.GetAll(h, len);]&] [s2; [C ptr `= new byte`[h.GetSize()`];]&] [s2; [C h.Get(ptr);]&] [s2; [C `}]&] [s2; [C .....]&] [s2; [C `}]&] [s2;C &] [s2; Now if stream remaining length is less than [C len], [* Huge] performs allocation just in 1MB chunks and the end of stream will be reached without allocating too much data. GetAll will then fail with LoadingError. The downside of this step is the fact that data get copied one more time and that for particular element, twice as much memory is required to load it, however, usually these are negligible.&] [s0;i448;a25;kKO9;:noref:@(0.0.255)%- &] [ {{10000F(128)G(128)@1 [s0; [* Public Method List]]}}&] [s3;%- &] [s5;:Upp`:`:Huge`:`:CHUNK:%- [@(0.0.255) enum]_[* CHUNK]_`=_[@3 1024]_`*_[@3 1024]_&] [s2; This is the size of allocation chunk for [* Huge], currently at 1MB.&] [s3;%- &] [s4;%- &] [s5;:Upp`:`:Huge`:`:AddChunk`(`):%- [_^Upp`:`:byte^ byte]_`*[* AddChunk]()&] [s2; Adds a chunk and returns a pointer to it. Client code must respect the [* CHUNK] size.&] [s3;%- &] [s4;%- &] [s5;:Upp`:`:Huge`:`:Finish`(int`):%- [@(0.0.255) void]_[* Finish]([@(0.0.255) int]_[*@3 last`_ chunk`_size])&] [s2; After adding a last chunk, [* Huge] is `'closed`' with calling this method with the size of its [%-*@3 last`_chunk`_size]. Usually, [* Huge] gets data from some source (like Stream) which returns full sized chunks until the last one.&] [s3; &] [s4;%- &] [s5;:Upp`:`:Huge`:`:GetSize`(`)const:%- [_^size`_t^ size`_t]_[* GetSize]()_[@(0.0.255) cons t]&] [s2; Returns the number of bytes stored.&] [s3;%- &] [s4;%- &] [s5;:Upp`:`:Huge`:`:Get`(void`*`,size`_t`,size`_t`)const:%- [@(0.0.255) void]_[* Get]([@(0.0.255) v oid]_`*[*@3 t`_], [_^size`_t^ size`_t]_[*@3 pos], [_^size`_t^ size`_t]_[*@3 sz])_[@(0.0.255) c onst]&] [s2; Reads data into target pointer.&] [s3; &] [s4;%- &] [s5;:Upp`:`:Huge`:`:Get`(void`*`):%- [@(0.0.255) void]_[* Get]([@(0.0.255) void]_`*[*@3 t])&] [s2; Same as Get(t, 0, GetSize()).&] [s3; &] [s4;%- &] [s5;:Upp`:`:Huge`:`:Get`(`)const:%- [_^Upp`:`:String^ String]_[* Get]()_[@(0.0.255) const]&] [s2; Returns data as String. There must be less than 2GB of data or operation fails with Panic.&] [s3;%- ]]