Optimizing TOAST Handling and Expanded In-Memory Representation

byte alignment anyway) then it's possible to avoid some of the overhead of <function>PG_DETOAST_DATUM</function>. You can use <function>PG_DETOAST_DATUM_PACKED</function> instead (customarily hidden by defining a <function>GETARG_DATATYPE_PP</function> macro) and using the macros <function>VARSIZE_ANY_EXHDR</function> and <function>VARDATA_ANY</function> to access a potentially-packed datum. Again, the data returned by these macros is not aligned even if the data type definition specifies an alignment. If the alignment is important you must go through the regular <function>PG_DETOAST_DATUM</function> interface. </para> <note> <para> Older code frequently declares <structfield>vl_len_</structfield> as an <type>int32</type> field instead of <type>char[4]</type>. This is OK as long as the struct definition has other fields that have at least <type>int32</type> alignment. But it is dangerous to use such a struct definition when working with a potentially unaligned datum; the compiler may take it as license to assume the datum actually is aligned, leading to core dumps on architectures that are strict about alignment. </para> </note> <para> Another feature that's enabled by <acronym>TOAST</acronym> support is the possibility of having an <firstterm>expanded</firstterm> in-memory data representation that is more convenient to work with than the format that is stored on disk. The regular or <quote>flat</quote> varlena storage format is ultimately just a blob of bytes; it cannot for example contain pointers, since it may get copied to other locations in memory. For complex data types, the flat format may be quite expensive to work with, so <productname>PostgreSQL</productname> provides a way to <quote>expand</quote> the flat format into a representation that is more suited to computation, and then pass that format in-memory between functions of the data type. </para> <para> To use expanded storage, a data type must define an expanded format that follows the rules given in <filename>src/include/utils/expandeddatum.h</filename>, and provide functions to <quote>expand</quote> a flat varlena value into expanded format and <quote>flatten</quote> the expanded format back to the regular varlena representation. Then ensure that all C functions for the data type can accept either representation, possibly by converting one into the other immediately upon receipt. This does not require fixing all existing functions for the data type at once, because the standard <function>PG_DETOAST_DATUM</function> macro

The text delves into optimizing TOAST handling by discussing `PG_DETOAST_DATUM_PACKED` as an alternative to `PG_DETOAST_DATUM` when data alignment is not critical. It warns about potential issues with older code that declares `vl_len_` as `int32` instead of `char[4]` when dealing with unaligned data. Furthermore, the text introduces the concept of an expanded in-memory data representation for complex data types, which allows for a more convenient format for computation than the flat storage format. To utilize expanded storage, data types need to define an expanded format and functions to convert between the flat and expanded representations.