To optimize insert speed, combine many small operations into a single large operation. Ideally, you make a single connection, send the data for many new rows at once, and delay all index updates and consistency checking until the very end.
The time required for inserting a row is determined by the following factors, where the numbers indicate approximate proportions:
Sending query to server: (2)
Parsing query: (2)
Inserting row: (1 × size of row)
Inserting indexes: (1 × number of indexes)
This does not take into consideration the initial overhead to open tables, which is done once for each concurrently running query.
The size of the table slows down the insertion of indexes by
N, assuming B-tree indexes.
You can use the following methods to speed up inserts:
If you are inserting many rows from the same client at the
same time, use
statements with multiple
to insert several rows at a time. This is considerably
faster (many times faster in some cases) than using
statements. If you are adding data to a nonempty table,
you can tune the
variable to make data insertion even faster. See
Section 5.1.4, “Server System Variables”.
MyISAM table, you can use
concurrent inserts to add rows at the same time that
SELECT statements are
running, if there are no deleted rows in middle of the
data file. See Section 8.8.3, “Concurrent Inserts”.
With some extra work, it is possible to make
INFILE run even faster for a
MyISAM table when the table has many
indexes. Use the following procedure:
Optionally create the table with
Use myisamchk --keys-used=0 -rq
This removes all use of indexes for the table.
Insert data into the table with
INFILE. This does not update any indexes and
therefore is very fast.
If you intend only to read from the table in the future, use myisampack to compress it. See Section 220.127.116.11, “Compressed Table Characteristics”.
Re-create the indexes with myisamchk -rq
This creates the index tree in memory before writing
it to disk, which is much faster than updating the
INFILE because it avoids lots of disk seeks.
The resulting index tree is also perfectly balanced.
INFILE performs the preceding optimization
automatically if the
MyISAM table into
which you insert data is empty. The main difference
between automatic optimization and using the procedure
explicitly is that you can let
myisamchk allocate much more temporary
memory for the index creation than you might want the
server to allocate for index re-creation when it executes
tbl_nameDISABLE KEYS; ALTER TABLE
To speed up
operations that are performed with multiple statements for
nontransactional tables, lock your tables:
LOCK TABLES a WRITE; INSERT INTO a VALUES (1,23),(2,34),(4,33); INSERT INTO a VALUES (8,26),(6,29); ... UNLOCK TABLES;
This benefits performance because the index buffer is
flushed to disk only once, after all
INSERT statements have
completed. Normally, there would be as many index buffer
flushes as there are
statements. Explicit locking statements are not needed if
you can insert all rows with a single
Locking also lowers the total time for multiple-connection tests, although the maximum wait time for individual connections might go up because they wait for locks. Suppose that five clients attempt to perform inserts simultaneously as follows:
Connection 1 does 1000 inserts
Connections 2, 3, and 4 do 1 insert
Connection 5 does 1000 inserts
If you do not use locking, connections 2, 3, and 4 finish before 1 and 5. If you use locking, connections 2, 3, and 4 probably do not finish before 1 or 5, but the total time should be about 40% faster.
DELETE operations are very
fast in MySQL, but you can obtain better overall
performance by adding locks around everything that does
more than about five successive inserts or updates. If you
do very many successive inserts, you could do a
LOCK TABLES followed by an
TABLES once in a while (each 1,000 rows or so)
to permit other threads to access the table. This would
still result in a nice performance gain.
To increase performance for
tables, for both
INSERT, enlarge the key
cache by increasing the
variable. See Section 8.9.2, “Tuning Server Parameters”.