Finite difference

Numerical approximation for the derivative of a numerical function

Why

  • not always possible to analytically differentiate a function
  • sometimes the "function" is a complicated procedure

Method

Use Taylor expansion:

$$f(x_0+\Delta x) \approx f(x_0) + f'(x_0)\Delta x$$

Solve for the derivative:

$$f'(x_0) \approx \frac{f(x_0+\Delta x) - f(x_0)}{\Delta x}$$

I need 2 evaluations of the function $f$ to estimate the derivative. For the derivative at location $x$ I have three options:

  • forward difference (use $x$ and $x+\Delta x$) $$f'(x_0) \approx \frac{f(x_0+\Delta x) - f(x_0)}{\Delta x}$$
  • backward difference (use $x-\Delta x$ and $x$) $$f'(x_0) \approx \frac{f(x_0) - f(x_0-\Delta x)}{\Delta x}$$
  • central difference (use $x-\Delta x$ and $x+\Delta x$) $$f'(x_0) \approx \frac{f(x_0+\Delta x) - f(x_0-\Delta x)}{2\Delta x}$$

Which one is best depends on the situation:

  • interval where $f$ is defined (evaluating $f(x_0+\Delta x)$ or $f(x_0-\Delta x)$ might not always be possible)
  • if evaluation of $f$ is expensive and we are interested in the value at $x_0$ forward or backward difference might be better

Error estimate for forward difference

Use Taylor expansion again: $$\frac{f(x_0+\Delta x) - f(x_0)}{\Delta x} = \frac{f(x_0)+\Delta x f'(x_0)+\frac12\Delta x^2f''(x_0)+\mathcal{O}(\Delta x^3) - f(x_0)}{\Delta x} $$

$$= f'(x_0)+\frac12 f''(x_0)\Delta x +\mathcal{O}(\Delta x^2)$$

The error is (approximately) proportional to $\Delta x$

Error estimate for backward difference

Similarily for the backward difference: $$\frac{f(x_0) - f(x_0-\Delta x)}{\Delta x} = \frac{f(x_0)- \left( f(x_0) -\Delta x f'(x_0)+\frac12\Delta x^2f(x_0)+\mathcal{O}(\Delta x^3) \right)}{\Delta x} $$ $$= f'(x_0)-\frac12 f''(x_0)\Delta x +\mathcal{O}(\Delta x^2)$$

The error is (approximately) also proportional to $\Delta x$

Error estimate for central difference

The central difference is better: $$\frac{f(x_0+\Delta x) - f(x_0-\Delta x)}{2\Delta x} = \frac{\left(f(x_0) +\Delta x f'(x_0)+\frac12\Delta x^2f''(x_0)+\mathcal{O}(\Delta x^3)\right) - \left( f(x_0) -\Delta x f'(x_0)+\frac12\Delta x^2f''(x_0)+\mathcal{O}(\Delta x^3) \right)}{2\Delta x} $$ $$ = f'(x_0)+\mathcal{O}(\Delta x^2)$$ The error has a higher polynomial dependency in the small parameter $\Delta x$.

Assignment 1

We have covered now the material for the coding in the first assignment.

It's worth having a go at it now to help you understand the material in the lectures with a real example.