Assuming a good portion of the data on your MySQL server remains unchanged over time, you can increase the speed and reduce the required storage space for your regular backups by backing up not all the data on the server each time, but only the changes to the data which have taken place over time. In order to that, after making first a full backup that contains all data, you can do one of the following:
Performing a series of differential backups. Each differential backup includes all the changes made to the data since the last full backup was performed. To restore data up to, for example, time
t, you simply restore first the full backup, and then, on top of it, the differential backup taken for time
Perform a series of incremental backup. Each incremental backup only includes the changes since the previous backup, which can itself be a full or incremental backup. The first backup in an incremental series is always then a differential backup; but after that, each incremental backup only contains the changes made since that last incremental backup. Each subsequent incremental backup is thus usually smaller in size than a differential backup, and is faster to make; that allows you to make very frequent incremental backups, and then enables you to restore the database to a more precise point in time when necessary. However, restoring data with incremental backups might take longer and more work: in general, to restore data up to, for example, time
t, you start with restoring the full backup, and then restore the incremental backups one by one, until you are finished with the incremental backup taken for time t.
MySQL Enterprise Backup supports both incremental and differential backups. You should decide on which backup strategy to adopt by looking at such factors like how much storage space you have, how quickly you have to be able to restore data, and so on.
MySQL Enterprise Backup treats differential backup as a special case of incremental backup that has a full backup as its base. To create a differential backup, simply follow the instructions below for performing incremental backups, and make sure you specify a full backup as the base of your incremental backup using the methods we describe below; you should also ignore any instructions that only apply to the handling of multiple incremental backups.
See Section 14.7, “Incremental Backup Options”, for
descriptions of the mysqlbackup options used
for incremental backups. An Incremental backup is enabled with
one of the two options:
Creating Incremental Backups Using Only the Redo Log for
When creating an incremental backup, you have to indicate to
mysqlbackup the point in time of the previous
full or incremental backup. For convenience, you can use the
--incremental-base option to
automatically derive the necessary log
sequence number (LSN) from the metadata stored in a
previous backup directory or on the server. Or, you can specify
an explicit LSN value using the
--start-lsn option, providing to
mysqlbackup the ending LSN from a previous
full or incremental backup (see
Other Considerations for Incremental Backups on some
limitation that applies when using the
To prepare the backup data to be restored, you combine all incremental backups with an original full backup. Typically, you perform a new full backup after a designated period of time, after which you can discard the older incremental backup data.
Creating Incremental Backups Using Only the Redo Log
might offer some benefits over the
--incremental option for creating an
The changes to InnoDB tables are determined based on the contents of the
InnoDBredo log. Since the redo log files have a fixed size that you know in advance, it can require less I/O to read the changes from them than to scan the InnoDB tablespace files to locate the changed pages, depending on the size of your database, amount of DML activity, and size of the redo log files.
Since the redo log files act as a circular buffer, with records of older changes being overwritten as new DML operations take place, you must take new incremental backups on a predictable schedule dictated by the size of the log files and the amount of redo data generated for your workload. Otherwise, the redo log might not reach back far enough to record all the changes since the previous incremental backup, in which case mysqlbackup will quickly determine that it cannot proceed and will return an error. Your backup script should be able to catch that error and then perform an incremental backup with the
To calculate the size of the redo log, issue the command
SHOW VARIABLES LIKE 'innodb_log_file%'and, based on the output, multiply the
innodb_log_file_sizesetting by the value of
innodb_log_files_in_group. To compute the redo log size at the physical level, look into the
datadirdirectory of the MySQL instance and sum up the sizes of the files matching the pattern
The InnoDB LSN value corresponds to the number of bytes written to the redo log. To check the LSN at some point in time, issue the command
SHOW ENGINE INNODB STATUSand look under the
LOGheading. While planning your backup strategy, record the LSN values periodically and subtract the earlier value from the current one to calculate how much redo data is generated each hour, day, and so on.
Prior to MySQL 5.5, it was common practice to keep the redo logs fairly small to avoid a long startup time when the MySQL server was killed rather than shut down normally. With MySQL 5.5 and higher, the performance of crash recovery is significantly improved, as described in Optimizing InnoDB Configuration Variables, so that you can make your redo log files bigger if that helps your backup strategy and your database workload.
This type of incremental backup is not so forgiving of too-low
--start-lsnvalues as the standard
--incrementaloption. For example, you cannot make a full backup and then make a series of
--incremental-with-redo-log-onlybackups all using the same
--start-lsnvalue. Make sure to specify the precise end LSN of the previous backup as the start LSN of the next incremental backup; do not use arbitrary values.Note
To ensure the LSN values match up exactly between successive incremental backups, it is recommended that you always use the
--incremental-baseoption when you use the
To judge whether this type of incremental backup is practical and efficient for a particular MySQL instance:
Measure how fast the data changes within the InnoDB redo log files. Check the LSN periodically to decide how much redo data accumulates over the course of some number of hours or days.
Compare the rate of redo log accumulation with the size of the redo log files. Use this ratio to see how often to take an incremental backup, in order to avoid the likelihood of the backup failing because the historical data are not available in the redo log. For example, if you are producing 1GB of redo log data per day, and the combined size of your redo log files is 7GB, you would schedule incremental backups more frequently than once a week. You might perform incremental backups every day or two, to avoid a potential issue when a sudden flurry of updates produced more redo than usual.
Benchmark incremental backup times using both the
--incremental-with-redo-log-onlyoptions, to confirm if the redo log backup technique performs faster and with less overhead than the traditional incremental backup method. The result could depend on the size of your data, the amount of DML activity, and the size of your redo log files. Do your testing on a server with a realistic data volume and a realistic workload. For example, if you have huge redo log files, reading them in the course of an incremental backup could take as long as reading the InnoDB data files using the traditional incremental technique. Conversely, if your data volume is large, reading all the data files to find the few changed pages could be less efficient than processing the much smaller redo log files.
Other Considerations for Incremental Backups
The incremental backup feature is primarily intended for InnoDB tables, or non-InnoDB tables that are read-only or rarely updated. Incremental backups detect changes at the level of pages in the InnoDB data files, as opposed to table rows; each page that has changed is backed up. Thus, the space and time savings are not exactly proportional to the percentage of changed InnoDB rows or columns.
For non-InnoDB files, the entire file is included in an incremental backup if that file has changed since the previous backup, which means the savings for backup resources are less significant when comparing with the case with InnoDB tables.
You cannot perform incremental backups with the
When making an incremental backup that is based on a backup
(full or incremental) created using the
--no-locking option, use the
--skip-binlog option to skip the
backing up of the binary log, as binary log information will be
unavailable to mysqlbackup in that situation.
No binary log files are copied into the incremental backup if
--start-lsn option is used. To
include binary log files for the period covered by the
incremental backup, use the
--incremental-base option instead,
which provides the necessary information for
mysqlbackup to ensure that no gap exists
between binary log data included in the previous backup and the
current incremental backup.
Examples of Incremental Backups
This example uses mysqlbackup to make an
incremental backup of a MySQL server, including all databases
and tables. We show two alternatives, one using the
--incremental-base option and the
other using the
--incremental-base option, you do not
have to keep track of LSN values between one backup and the
next. Instead, you can just specify the location of the previous
directory backup (either full or incremental), and
mysqlbackup figures out the starting point
for this backup based on the metadata of the earlier one.
Because you need a known set of directory names, you might want
to use hardcoded names or generate a sequence of names in your
own backup script, rather than using the
$ mysqlbackup --defaults-file=/home/dbadmin/.my.cnf --incremental \ --incremental-base=dir:/incr-backup/wednesday \ --incremental-backup-dir=/incr-backup/thursday \ backup ...many lines of output... mysqlbackup: Backup created in directory '/incr-backup/thursday' mysqlbackup: start_lsn: 2654255717 mysqlbackup: incremental_base_lsn: 2666733462 mysqlbackup: end_lsn: 2666736714 101208 17:14:58 mysqlbackup: mysqlbackup completed OK!
Note that if your last backup was a single-file instead of a
directory backup, you can still use
--incremental-base by specifying for
the location of the temporary directory you supplied with the
--backup-dir option during the full backup.
As an alternative to specifying
you can tell mysqlbackup to query the
end_lsn value from the last successful backup
as recorded in the
backup_history table on
the server using
(this required that the last backup was made with
mysqlbackup connected to the server).
You can also use the
--start-lsn option to
specify where the incremental backup should start. You have to
record the LSN of the previous backup reported by
mysqlbackup at the end of the backup:
mysqlbackup: Was able to parse the log up to lsn 2654255716
The number is also recorded in the
meta/backup_variables.txt file in the
folder specified by
during the backup. Supply then that number to
mysqlbackup using the
--start-lsn option. The incremental backup then
includes all changes that came after the
specified LSN. Since then the location of the previous backup is
not very significant then, you can use
--with-timestamp to create named subdirectories
$ mysqlbackup --defaults-file=/home/dbadmin/.my.cnf --incremental \ --start-lsn=2654255716 \ --with-timestamp \ --incremental-backup-dir=/incr-backup \ backup ...many lines of output... mysqlbackup: Backup created in directory '/incr-backup/2010-12-08_17-14-48' mysqlbackup: start_lsn: 2654255717 mysqlbackup: incremental_base_lsn: 2666733462 mysqlbackup: end_lsn: 2666736714 101208 17:14:58 mysqlbackup: mysqlbackup completed OK!
To create an incremental backup image instead, use the following
command, specifying with
--backup-dir a temporary directory
for storing the metadata for the backup and some temporary
$ mysqlbackup --defaults-file=/home/dbadmin/.my.cnf --incremental \ --start-lsn=2654255716 \ --with-timestamp \ --backup-dir=/incr-tmp \ --backup-image=/incr-backup/incremental_image.bi \ backup-to-image
In the following example though, because
--backup-image does not provide a
full path to the image file to be created, the incremental
backup image is created under the folder specified by
$ mysqlbackup --defaults-file=/home/dbadmin/.my.cnf --incremental \ --start-lsn=2654255716 \ --with-timestamp \ --backup-dir=/incr-images \ --backup-image=incremental_image1.bi \ backup-to-image
On how to restore your database using the incremental backups, see Section 5.2.3, “Restoring an Incremental Backup”
On a regular schedule determined by date or amount of database activity, take more incremental backups.
Optionally, periodically start the cycle over again by taking a full, uncompressed or compressed backup. Typically, this milestone happens when you can archive and clear out your oldest backup data.