Question:

How to find the second derivative of this function according to the limit definition?

Alice: 26 June 2022

I want to find the second derivative of function $f(x)=-(x-1)^{2}+(x-1)^{3} \sin \left(\frac{1}{(x-1)^{2}}\right)$ at x = 1 according to the definition of limit:

f[x_] := -(x - 1)^2 + (x - 1)^3 Sin[1/(x - 1)^2]
Limit[((f[h + k + x] - f[k + x])/h - (f[h + x] - f[x])/h)/k /. 
  x -> 1, {h -> 0, k -> 0}]

Limit[((f[h + k + x] - f[k + x])/h - (f[h + x] - f[x])/h)/k, {h -> 0, 
  k -> 0}]
Limit[((f[h + k + x] - f[k + x])/h - (f[h + x] - f[x])/h)/
  k, {h, k} -> {0, 0}]

But the result of the above code is very strange, I want to know what the correct method should be?

Answer:
Aurora: 26 June 2022

This is a variation on the theme of Example 2 from Ch.3 in B. Gelbaum and J. Olmsted, Counterexamples in Analysis (https://rads.stackoverflow.com/amzn/click/com/0486428753), where a differentiable function having the discontinuous derivative is constructed.

First, you should to redefine the function at $x=1$ by

f[x_] := Piecewise[{{-(x - 1)^2 + (x - 1)^3 Sin[1/(x - 1)^2], 
x != 1}, {0, x == 1}}]

Second, you need not two gains $h$ and $k$ to find the second derivative at $x=1$, only $h$ is enough (see Wiki (https://en.wikipedia.org/wiki/Finite_difference))

Limit[(f[1 + h] - 2*f[1] + f[1 - h])/h^2, h -> 0]
(*-2*)

The second derivative is discontinuous at $x=1$ as

Plot[f''[x], {x, 0, 2}]

enter image description here shows.

Addition. Unfortunately, nobody notices an inaccuracy in my answer so I must indicate it on my own. The limit

Limit[(f[1 + h] - 2*f[1] + f[1 - h])/h^2, h -> 0]

equals f''[1] only if that second derivative exists. However, in the case under consideration the derivative f'[x] is discontinuous at x==1 as the results of

MaxLimit[f'[x], x -> 1]
(*2*)

and

MinLimit[f'[x], x -> 1]
(*-2*)

prove. Therefore, f'[x] does not have the derivative at x==1, being discontinuous at that point.