WL#9838: Group Replication: Flow-control fine tuning
Affects: Server-8.0 — Status: Complete
EXECUTIVE SUMMARY 1. Current flow-control scheme Group Replication introduced flow control to avoid excessive buffering and to maintain group members reasonably close to one another. For several reasons, it's more efficient to keep buffering low but, as mentioned before, it is not a requirement that members are kept in sync for replication to work: once a slave becomes delayed it will just increase the number of pending transactions in the relay log. The flow control mechanism enters the scene to bound the back-log members can accumulate, in terms of transactions to certify and transactions to apply. Once those bounds are exceeded, and while they remain exceeded, the flow-control mechanism limits the throughput of writer members to adjust to the capacity of the slowest members of the group. By imposing conservative bounds it tries to make sure all members are able to keep-up and can return safely to within the bounds defined by the user. The flow-control system works asynchronously, the capacity to use in each period is calculated deterministically in each member, in each period, using the statistics that are regularly sent to the group by each member. The coordination is purposely kept coarse-grained, in order to reduce the number of messages in flight and to keep the system running as smoothly as possible. The implementation depends on two basic mechanisms: 1) monitoring the members to gather statistics on throughput and queue sizes of all group members to make educated guesses on what is the maximum write pressure each member can withstand and 2) throttling the members to avoid writing beyond its fair-share of the capacity available in each step. The following variables are used to control flow-control behaviour in Group Replication: - group-replication-flow-control-mode = QUOTA | DISABLED - group-replication-flow-control-certififier-threshold = 0..n - group-replication-flow-control-applier-threshold = 0..n Flow-control can be enabled and disabled by selecting a flow control algorithm with the group-replication-flow-control-mode option. QUOTA is the default flow-control mode, and only one for now, and it calculates a write quota for the group in terms of commits/period. That quota, after being divided by the number of members that tried to write in the last period, will be the maximum allowance the client threads have to spend during the next flow-control period. Flow control takes into account two work queues: the certification queue and the group replication applier queue, so a user may choose 1) doing flow control at the certifier or at the applier level, or both and 2) what is the trigger size on each queue. The options group-replication-flow-control-certififier-threshold and group-replication-flow-control-applier-threshold specify the size of the queues that when exceeded will force flow-control to throttle the throughput on the writer members. More information about how these options work is available in the user's manual here: https://dev.mysql.com/doc/refman/5.7/en/group-replication-options.html 2. Limitations of current scheme Current flow-control mechanism used by GR depends on heuristics that leave little control to the user, other then enabling/disabling it and setting the flow-control thresholds. However, there are several built-in values that could benefit from user input in some situations, and that is presently not possible. 3. User stories - as a user, I would like to be able to configure flow-control in order to optimize the lag between member members, while extracting the highest throughput of the underlying system and running within what I consider safe operational limits; - as a user, I would like to be able to configure flow-control to keep the throughput as stable as possible independently of the number of clients and of the demands of the workload; - as a user, I would like to be able to configure flow-control so that all members of the group have similar opportunity to submit transactions on the cluster independently of the number of clients and of the demands of the workload; - as a user, I would like the perform management tasks on some members without reducing the throughput of the rest of the cluster in some cases; - as a user, I would like to adapt flow-control policies to workloads which take a long time to commit as well as it does to shorter transactions; - as a user, I would like to assign different tasks to some members of the group and be allowed to configure flow-control accordingly. 4. Areas requiring more user control This worklog introduces user control over: 1. The minimum flow-control quota that can be assigned to a member: presently it is set to 5% of the lowest flow-control thresholds, but these are unrelated variables and it makes little sense for them to be dependent. 2. The minimum recovering flow-control quota: different from the minimum flow-control quota in that it applies only to members entering the group, so that the members entering the group don't excessively affect throughput. Some users complained that flow-control should not be used when a member is joining, but one of the original goals was precisely to avoid having members that never join because they cannot keep up with the group workload while joining. 3. The maximum cluster commit quota: in order to keep the group operating within a safe throughput range (to improve on predictable behaviour and balance between members). 4. The proportion of the quota that is assigned to each member: when flow-control is active the quota of each writer member is similar, which may be sub-optimal if some members are expected to write more than others. This can also be used to assign more than 100% of the calculated quota (total), so the user can over/under-subscribe the estimations done be the system, either because the calculations are too conservative, or because the user prefers to let some lag to pile up, but slower then without flow control. It will also allow the user to set a maximum quota por member, as a part of the maximum commit quota. 5. The flow-control period: currently flow-control messages are sent once per second, a fixed rate that depends on the unfortunately also fixed rate of the garbage collection in GR, which can be inconvenient if the commit rate is very low and not enough transactions are executed per second to give the heuristics enough data to work properly. 6. The percentage of the quota that is reserved for catch-up and when flow-control is released: presently constants in the code, 10% and 50% of the quota respectively, require better tunning those properly reduce variations that are presently seem in the throughput when the system is close to exhausting its free capacity. In addition, participation in flow-control should be optional, so that some members can be ignored by other members if they are doing some maintenance tasks for instance. That would allow the members to remain in the group, but would not try to make them in sync, at the cost of having the garbage collection in GR become less effective if the members stop applying the transactions that are being sent to the group. Flow-control should remain local to the writer, and control the local member only. If there is a mismatch between the configurations, if unintentional, they need to be adressed by changing the variables, something that should continue to be doing dynamically even if Group Replication is running.
FUNCTIONAL REQUIREMENTS FR1. When a member has group_replication_flow_control_mode=DISABLED the other members should not be delayed even if it does not apply any transactions, or has a large back-log. FR2. The computed commits/period quota assigned to each member of the group shall not be lower then group_replication_flow_control_min_recovery_quota (new option). This option will replace group_replication_flow_control_min_quota when the only reason for the flow-control to throttle the master is due to some members being in recovery mode, if other members exceed the queue sizes due to other reasons group_replication_flow_control_min_quota will be applied. FR3. When flow-control is active the computed commits/period quota assigned for each member shall not be: - lower than group_replication_flow_control_min_quota (new option) - higher than group_replication_max_quota (new option), multiplied by group_replication_member_quota_percent and divided by 100 if group_replication_member_quota_percent is not zero. FR4. The total commits/period quota of the cluster, when flow control is active, shall be assigned to a member according to the following policy: - divided equaly by the number of members that wrote in the last period if group_replication_member_quota_percent=0 (new option) - assigned a quota = group_replication_member_quota_percent * total quota if group_replication_member_quota_percent is between 1% and 100% (the user is responsible for the quota over-subscription). FR5. The flow-control messages and execution of heuristics to set the quota shall occur every group_replication_flow_control_period (new option) seconds. The period specified does not need to be the same on all nodes, but members that have periods exceeding 10 times that of other members will be ignored in some iterations, as stats information about remote members is discarded if those members do not update them at least once every 10 periods. FR6. When flow-control is active, the member commits/period quota shall be reduced by group_replication_flow_control_hold_percent (new option) percent to allow the back-log to decrease gradualy. FR7. When flow-control becomes inactive, the total cluster commits/period quota shall be increased by group_replication_flow_control_release_percent percent in each flow control period to allow the cluster the increase throughput gradually and avoid large spikes. To disable this option the user can set group_replication_flow_control_release_percent=0, in which case the quota shall be released immediately. NON-FUNCTIONAL REQUIREMENTS NF1. The user should be able to configure the flow-control settings to allow smooth execution over time, even when the cluster is near its capacity. NF2. The flow-control settings should not reduce the cluster throughput significantly unless the user specifies it.
APPROACH Change current flow-control code to introduce further boundaries on the quotas assigned. To implement this worklog one needs to create 7 new options: - group_replication_flow_control_min_quota= X commits/s - group_replication_flow_control_min_recovery_quota= X commits/s - group_replication_flow_control_max_commit_quota= X commits/s - group_replication_flow_control_member_quota_percent= Y % - group_replication_flow_control_period= Z seconds - group_replication_flow_control_hold_percent= Y % - group_replication_flow_control_release_percent= Y % In addition, an existing option needs to have its behaviour better defined, so that a member which has: - group_replication_flow_control_mode= DISABLED it will be excluded from the flow-control calculations by other members. DETAILS * New system variables to allow user control over the flow-control: - Name: group_replication_flow_control_min_quota - Input Values: [0, MAX_INT] - Default: 0 - Units: commits per flow-control period - Description: This option controls the lowest flow-control quota that can be assigned to a member, independently of the calculated minimum quota executed in the last period. A value of 0 implies that there is no minimum quota. Cannot be larger than group_replication_flow_control_max_commit_quota. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_min_recovery_quota - Input Values: [0, MAX_INT] - Default: 0 - Units: commits per flow-control period - Description: This option controls the lowest quota that can be assigned to a member because of another recovering member in the group, independently of the calculated minimum quota executed in the last period. A value of 0 implies that there is no minimum quota. Cannot be larger than group_replication_flow_control_max_commit_quota. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_max_quota - Input Values: [0, MAX_INT] - Default: 0 - Units: commits per flow-control period - Description: This option defines the maximum flow-control quota of the cluster, or the maximum available quota for any period while flow-control is enabled. A value of 0 implies that there is no maximum quota set. Cannot be smaller than group_replication_flow_control_min_commit_quota and group_replication_flow_control_min_recovery_quota. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_member_quota_percent - Input Values: [0, 100%] - Default: 0 - Description: This option defines the percentage of the quota that a member should assume is available for itself when calculating the quotas. A value of 0 implies that the quota should be split equally between members that were writers in the last period. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_period - Input Values: [1, 60] - Default: 1 - Units: seconds between flow-control periods - Description: This option defines how many seconds to wait between flow-control iterations, in which flow-control messages are sent and flow-control managements tasks are ran. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_hold_percent - Input Values: [0, 100%] - Default: 10% - Description: This option defines what percentage of the cluster quota should remain unused to allow a cluster under flow-control to catch-up on backlog. A value of 0 implies that no part of the quota will be reserved for catching up on the work backlog. - Dynamic: yes. - Scope: Global - Type: Integer - Name: group_replication_flow_control_release_percent - Input Values: [0, 1000%] - Default: 50% - Description: This option defines how the cluster quota should be released when flow-control no longer needs to throttle the writer members, with this percentage being the quota increase per flow-control period. A value of 0 implies that once the flow-control thresholds are within limits the quota will be released in a single flow-control iteration. The range allows the quota to be released at up to 10 times current quota, as that allows a greater degree of adaptation, mainly when the flow-control period is large and the quotas are very small. - Dynamic: yes. - Scope: Global - Type: Integer All the variables are not only dynamic, but must also be changeable while Group Replication is running. * Observability - This worklog relies on the flow-control mechanism that we have in place already, only the commit quotas change. * Cross-version group replication - The flow-control decisions impacted by this worklog are local and should not affect cross-version replication. * Rolling-upgrades - The upgrade procedures do not change as a results of this worklog.
The main heuristics of the flow-control logic are executed in the Flow_control_module::flow_control_step(). Other then adding the new options, the changes in this worklog consist on introducing boundaries on the quota calculations performed there, according to the specifications above. The changes in the flow-control interval are also controlled inside this function, which continues to be called every second but only allows flow-control to run when enough seconds have passed since it last executed. In order to ignore members that have flow-control-mode=DISABLED the flow-control the flow control messages will add a char to share the flow-control-mode of each member.
Copyright (c) 2000, 2019, Oracle Corporation and/or its affiliates. All rights reserved.