Chapter 5

Divide and conquer refers to a class of algorithmic techniques in which one breaks the input into several parts, solves the problem in each part recursively, and then combines the solutions to these subproblems into an overall solution. Analyzing the running time of a divide and conquer algorithm generally involves solving a recurrence relation that bounds the running time recursively in terms of the running time on smaller instances.

Many common divide and conquer algorithms follow the following template: divide the input into two pieces of equal size; solve the two subproblems on these pieces separately by recursion; and then combine the two results into an overall solution, spending only linear time for the initial division and final recombining.

For any algorithm that follows the template described above let T(n) denote its worst-case running time on input instances of size n. The algorithm spends O(n) time to divide the input into two pieces of size n/2 each; it then spends time T(n/2) to solve each one (since T(n/2) is the worst-case running time for an input of size n/2); and finally it spends O(n) time to combine the solutions from the two recursive calls. Thus, we have the follow relation:

(5.1) For some constant c, T(n) ≤ 2T(n/2) + cn (where n > 2), and T(2) ≤ c.

There are two ways to solve a recurrence relation:

• “Unroll” the recursion, accounting for the running time across the first few levels, and identify a pattern that can be continued as the recursion expands. Sum the running times over all levels of the recursion (i.e., until it “bottoms out” on subproblems of constant size, thereby arriving at a total running time.
• Start with a guess for the solution, substitute it into the recurrence relation, and check that it works. Formally, one justifies this plugging-in using an argument by induction on n.

Mergesort sorts a given list of numbers by first dividing them into two equal halves, sorting each half separately by recursion, and then combining the results of these recursive calls–in the form of the two sorted halves–using a linear-time algorithm for merging sorted lists (that we discussed in Chapter 2).

We can unroll the recurrence for Mergesort as follows:

At the first level of recursion, we have a single problem of size n, which takes time at most cn plus the time spent in all subsequent recursive calls. At the next level, we have two problems each of size n/2. Each of these takes time at most cn/2, for a total of at most cn, again plus the time in subsequent recursive calls. At the third level, we have four problems each of size n/4, each taking time at most cn/4, for a total of at most cn. Therefore, at level j of the recursion, the number of subproblems has doubled j times, so there are now a total of 2^j subproblems. Each has correspondingly shrunk in size by a factor of two j times, and so each has size n/2^j, and hence each takes time at most cn/2^j. Thus level j contributes a total of at most 2^j(cn/2^j) = cn to the total running time. We know that the number of times the input must be halved in order to reduce its size from n to 2 is log n. So summing the cn work over log n levels of recursion, we get a total running time of O(n log n).

If we have a guess for the running time that we want to verify, we can do so by plugging it into the recurrence as follows:

Suppose we believe that T(n) ≤ cn log n for all n ≥ 2, and we want to check whether this is indeed true. This clearly holds for n = 2, since in this case cn log n = 2c, and (5.1) explicitly tells us that T(2) ≤ c. Now suppose, by induction, that T(m) ≤ cm log m for all values of m < n, and we want to establish this for T(n). We do this by writing the recurrence for T(n) and plugging in the inequality T(n/2) ≤ c(n/2) log(n/2). We then simplify the resulting expression by noticing that log (n/2) = (log n) - 1. This establishes the bound we want for T(n), assuming it holds for smaller values m < n, and thus it completes the induction argument.

An interesting section that introduces a novel concept: recurrence relations. I have known for a while now that Mergesort is a “good” sorting algorithm that runs in O(n log n) time–but now I can actually understand why it runs in O(n log n) time. I mean, to be honest, I knew that you divide the original list into log n sublists, and sort each list in linear time, but now I have concrete proof of how the running time is actually derived–as opposed to a vague idea.

That said, I feel like, in general, recurrence relations look like they are difficult to understand or follow. I also felt like most of the class was not immediately comfortable with the idea and the notations. I guess this is again one of those things that are difficult to get used to, but once you do get used to them, they're super intuitive.

In terms of how interesting this section was to me, I'd say about an 8/10.

We now consider a subset of recurrence relations that generalizes the relation in (5.1), and show how to solve the recurrences in this subset. This more general class of algorithms is obtained by considering divide and conquer algorithms that create recursive calls on q subproblems of size n/2 each and then combine the results in O(n) time. Thus we have:

(5.3) For some constant c, T(n) ≤ qT(n/2) + cn (where n > 2), and T(2) ≤ c.

We can unroll the recurrence for the case where q > 2 as follows:

At the first level of recursion, we have a single problem of size n, which takes time at most cn plus the time spent in all subsequent recursive calls. At the next level, we have q problems, each of size n/2. Each of these takes time at most cn/2, for a total of at most (q/2)cn, again plus the time in subsequent recursive calls. The next level yields q^2 problems of size n/4 each, for a total time of (q^2/4)cn. Since q > 2, we see that the total work per level is increasing as we proceed through the recursion. At an arbitrary level j, we have q^j distinct instances, each of size n/2^j. Thus the total work performed at level j is q^j(cn/2^j) = (q/2)^j cn.

There are log n levels of recursion, and the total amount of work performed is the sum over all the levels. This sum turns out to be a geometric series, and solving the series yield a running time of O(n^log q) (see page 216 of the textbook).