WL#2669: Slave Cluster recovery should be automated

If either the slave cluster is being started from a restore of a master cluster
dump, or being recoverd from an outage, the slave should have a way of knowing
exactly where to start processing from in the master bin log without a user
having to complete 4 - 5 step process. (i.e. should have a --recover flag that
automates these steps.

Conversation from IRC:

jeb_phone	tomas: and why can't this all be automated?
tomas_break	well the whole point of it is that it is in the same transaction   
           
                as the epoch... so it will tell you exactly which epoch was the   
                last successfull
tomas_break	we have problems convincing brian that epochs are general enough 
                and that is the way to go...
tomas_break	the whole replicaition is built around binlog positions...
jeb_phone	so we use the manually instead? Not sure that make sense. ;-)
tomas_break	but it would be "trivial" to automate...
tomas_break	almost...
tomas_break	the binlog_index and apply status are cluster addons... that 
                does not interfere with replication of innodb or myisam...
tomas_break	if we were to integrate this functionality more tightly it would 
                affect the other storage engines...
tomas_break	jeb_phone, ^
jeb_phone	Just seems that we should be able to tell the slave process to get the
last epoch and fing that position in the bin log. That way if we do a restore or
have a crash, the slave would know exactly where to take off from without
depending on users. <-tomas
jeb_phone	the more you can remove user intervention the better a product you
have :-)
tomas_break	jeb_phone, but I would have to extend the syntax somehow...
tomas_break	right now the whole interface is manual... no automation...
tomas_break	what do do if there is a conflict between apply_status and positions
in the master info?
tomas_break	override? What is the user _want's_ to go with the position in the
master info?
tomas_break	s/is/if
jeb	tomas: my guess would be to back down the epoch until there is a match in
the master.
tomas_break	I could have a flag
update_log_postion_automatically_according_to_apply_status ...
tomas_break	that would be checked on SLAVE START and correct the log position if
there is a discrepancy...
tomas_break	I only have one issue that does not make it simple... I don't have
access to the binlog_index on the master side from the slave...
tomas_break	jeb, ^
jeb	sounds good to me, can you have that before you leave today? Why don't we
replicate the binlog_index?
jeb	tomas ^
tomas	jeb, it conflicts with the binlog_index on the slave...
jeb	Then could we have a master_binlog_index that gets the replicated items
pumped into it?
jeb	Might should get Mats into this conversation.
tomas	jeb, yes but we are making this more complex than it should be... the
replication _should_ support epochs directly... not as an addon... and we enough
work to fix what we have already implemented...
tomas	we need to give people a readon to upgrade to 5.2 :)
tomas	s/readon/reason