WL#3147: MyISAM repair code cleanup

Affects: Server-7.1   —   Status: Un-Assigned   —   Priority: Medium

Remove code duplication and fix possible numeric overflow in the MyISAM repair code.

In the course of analyzing BUG#11527 (Setting myisam_repair_threads to >1 leads
to corruption), I found two areas of code duplication and a possible integer
overflow with very big tables (in addition to the overflow that caused the bug).
Copyright (c) 2001-2007 by MySQL AB. All rights reserved.

All of this happens in myisam/sort.c.

thr_find_all_keys() duplicates code from _create_index_by_sort() and

I suggest to move the duplicated part of _create_index_by_sort() into
find_all_keys() and just call find_all_keys() from thr_find_all_keys().

In the duplicated code from _create_index_by_sort() we compute with the number
of records. If this is very big, a numeric overflow can happen. An additional
condition can check for this.

What I mean is this:

-    if ((my_off_t) (records + 1) * (sort_length + sizeof(char*)) <=
	 (my_off_t) memavl)

+   /*
+     With true varchar we could have records much shorter than key_length.
+     We could have so many of these that records * key_length overflows
+     my_off_t. So we take a two step decision here.
+   */
+   if ((records < memavl) &&
+       (my_off_t) (records + 1) * (sort_length + sizeof(char*)) <=
	 (my_off_t) memavl)

Also the variables 'maxbuffer' and 'skr' should be of type ' ha_rows' to be safe.