In a previous post, I mentioned subfunctions in Octave and Matlab.
Nested functions are an interesting new beast. I can see situations where they can be useful as explained in this short video, and this longer, more detailed blog.
They are currently not featured in GNU Octave. Here's their reasoning:
There was discussion in Octave of having these even prior to Matlab, and the decision was made not to have these in Octave at the time for compatibility.
Now that Matlab has recently introduced nested functions, Octave will probably have them soon as well. Until then nested functions in Octave are treated as sub-functions with the same scoping rules as sub-function.
The authors of Octave consider the nested function scoping rules of Matlab to be more problems than they are worth as they introduce diffiult to find bugs as inadvertantly modifying a variable in a nested function that is also used in the parent is particularly easy.