Home Explore Blog CI



postgresql

30th chunk of `doc/src/sgml/perform.sgml`
f47688e77451554aef636a1d960ad3da9cbca9eb2c97ee160000000100000fa8
 </para>

   <para>
    Note that loading a large number of rows using
    <command>COPY</command> is almost always faster than using
    <command>INSERT</command>, even if <command>PREPARE</command> is used and
    multiple insertions are batched into a single transaction.
   </para>

   <para>
    <command>COPY</command> is fastest when used within the same
    transaction as an earlier <command>CREATE TABLE</command> or
    <command>TRUNCATE</command> command. In such cases no WAL
    needs to be written, because in case of an error, the files
    containing the newly loaded data will be removed anyway.
    However, this consideration only applies when
    <xref linkend="guc-wal-level"/> is <literal>minimal</literal>
    as all commands must write WAL otherwise.
   </para>

  </sect2>

  <sect2 id="populate-rm-indexes">
   <title>Remove Indexes</title>

   <para>
    If you are loading a freshly created table, the fastest method is to
    create the table, bulk load the table's data using
    <command>COPY</command>, then create any indexes needed for the
    table.  Creating an index on pre-existing data is quicker than
    updating it incrementally as each row is loaded.
   </para>

   <para>
    If you are adding large amounts of data to an existing table,
    it might be a win to drop the indexes,
    load the table, and then recreate the indexes.  Of course, the
    database performance for other users might suffer
    during the time the indexes are missing.  One should also think
    twice before dropping a unique index, since the error checking
    afforded by the unique constraint will be lost while the index is
    missing.
   </para>
  </sect2>

  <sect2 id="populate-rm-fkeys">
   <title>Remove Foreign Key Constraints</title>

   <para>
    Just as with indexes, a foreign key constraint can be checked
    <quote>in bulk</quote> more efficiently than row-by-row.  So it might be
    useful to drop foreign key constraints, load data, and re-create
    the constraints.  Again, there is a trade-off between data load
    speed and loss of error checking while the constraint is missing.
   </para>

   <para>
    What's more, when you load data into a table with existing foreign key
    constraints, each new row requires an entry in the server's list of
    pending trigger events (since it is the firing of a trigger that checks
    the row's foreign key constraint).  Loading many millions of rows can
    cause the trigger event queue to overflow available memory, leading to
    intolerable swapping or even outright failure of the command.  Therefore
    it may be <emphasis>necessary</emphasis>, not just desirable, to drop and re-apply
    foreign keys when loading large amounts of data.  If temporarily removing
    the constraint isn't acceptable, the only other recourse may be to split
    up the load operation into smaller transactions.
   </para>
  </sect2>

  <sect2 id="populate-work-mem">
   <title>Increase <varname>maintenance_work_mem</varname></title>

   <para>
    Temporarily increasing the <xref linkend="guc-maintenance-work-mem"/>
    configuration variable when loading large amounts of data can
    lead to improved performance.  This will help to speed up <command>CREATE
    INDEX</command> commands and <command>ALTER TABLE ADD FOREIGN KEY</command> commands.
    It won't do much for <command>COPY</command> itself, so this advice is
    only useful when you are using one or both of the above techniques.
   </para>
  </sect2>

  <sect2 id="populate-max-wal-size">
   <title>Increase <varname>max_wal_size</varname></title>

   <para>
    Temporarily increasing the <xref linkend="guc-max-wal-size"/>
    configuration variable can also
    make large data loads faster.  This is because loading a large
    amount of data into <productname>PostgreSQL</productname> will
    cause checkpoints to occur more often than the normal checkpoint
    frequency (specified by the <varname>checkpoint_timeout</varname>
    configuration

Title: Optimizing Bulk Data Loading in PostgreSQL
Summary
To improve performance when loading large amounts of data into PostgreSQL, consider techniques such as removing indexes and foreign key constraints, increasing maintenance_work_mem, and increasing max_wal_size, as these can significantly reduce overhead and improve loading speed, but be aware of the trade-offs in terms of data integrity and database performance for other users.