Nowhere monotone functions and functions of nonmonotonic type

We investigate the relationships between the notions of a continuous function being monotone on no interval, monotone at no point, of monotonic type on no interval, and of monotonic type at no point. In particular, we characterize the set of all points at which a function that has one of the weaker properties fails to have one of the stronger properties. A theorem of Garg about level sets of continuous, nowhere monotone functions is strengthened by placing control on the location in the domain where the level sets are large. It is shown that every continuous function that is of monotonic type on no interval has large intersection with every function in some second category set in each of the spaces pn, Cn, and Lip1. 1. NONMONOTONICITY PROPERTIES In a series of interesting papers [4], [5], [6], [7], [8], Garg investigated level set structures and derivate structures of continuous functions. This investigation was continued in a paper by Bruckner and Garg [2]. These articles considered several notions that measure degrees of pathology in the class of continuous, nowhere monotone functions. In this section we further study the relationships among these properties. We use C and BV to denote the collections of functions from [0, 1] into R, the reals, that are continuous and of bounded variation, respectively. Df (x) and Df (x) denote the lower and upper (two-sided) Dini derivates, respectively, of a function f at a number x (see [1]). We use standard terms such as perfect sets, first category sets, sets with the Baire property, etc., whose definitions may be found in [9]. Following Bruckner and Garg [2], we say that a function f is nondecreasing at x if f(t)-f(x) > 0 for all t $& x in some neighborhood of x. That f is nonincreasing t-xat x is defined with the obvious modification. If f is either nonincreasing at x or nondecreasing at x, then we say that f is monotone at x. That f is nonmonotone at x means that f is not monotone at x. If f is a function and m E R then, following Garg [8], denote by f+m and f-r the functions defined by f+r(x) = f (x) + mx and f-m(x) = f(x) mx. Inclusion of the "+" and "-" avoids confusion with Received by the editors August 20, 1996 and, in revised form, May 7, 1997. 1991 Mathematics Subject Classification. Primary 26A48; Secondary 26A24.