I will try to explain. When I came to emacs as a user, a "buffer" seemed just a weird way of saying "tab" without displaying actual tabs. The word choice only started making more sense once I wrote non-trivial amounts of emacslisp. The buffer is the fundamental emacs data structure, like the file is the fundamental unix data structure, so it's really important to understand it well.
The definition is really simple: a buffer is just a bunch of text from some unknown source, but in fact it is a very powerful abstraction, for two reasons. First, every time you do something in Emacs when simply editing files, you are really executing emacslisp functions on the buffer that is currently displayed. A lot of things you learn in Emacs can be used in three ways: interactively in editing sessions, when automating editing tasks with emacslisp, and finally when doing programming in general, even when it has nothing to do with editing files.
Second, a buffer can represent anything, it can of course contain the text of a file, but also of only fragments of a file, or a directory file listing, and, what's really great, Emacs binds TCP/FTP/HTTP/... network connections and external processes to buffers as well, allowing asynchronous streaming of incoming data into buffers, so that a emacslisp callback is called every time new text arrives from the process or socket, you can transform it as you wish and insert into the buffer the part to be shown to the user or kept for future processing. Then there are lisp primitives implemented in C like save-excursion which let you execute functions on buffer off-screen, so the same commands that are used for editing can be used for processing HTTP requests or grep output without the user noticing the cursor jumping back and forth or anything of this kind. So, when I am processing HTTP requests in Emacs, I use the same commands I sometimes use in interactive editing sessions, for example I can use re-search-forward to check if the headers were sent completely already and to get number of bytes they contain (the location in the buffer of the end of the headers, in other words). A lot of the weird emacs word like point, mark etc. that seem like anachronistic words for interface elements really are powerful abstractions that apply as much to programming as to the interactive editing.
Then there is a lot of other stuff that makes this yet more powerful, you can have buffer-local variables, timers that check buffer input periodically, transaction queues for those asynchronous processes, modes that define buffer-specific key-bindings, menus and behaviours etc., it goes on an on. It's a lot like node.js or other asynchronous networking stacks. There is also lot of magic that makes all this work concurrently without Emacs having implemented threads, and that makes it not interfere with the user interface.
A lot of the cool things Emacs does follows from this architecture, like the SLIME package for interacting with Common Lisp interpreters (local or remote), Dired for editing directory contents, Tramp for editing files on remote servers, Comint for running interactive processes of various kind inside Emacs etc. It's a really great lesson in how naturally features flow out of powerful design ideas.
As for resources, I learnt by reading the GNU Emacs Lisp Reference [1], and have done a lot of M-x describe-function and source code reading in the process of writing some elisp more sophisticated than just small editing helpers. E.g. the process stuff is described here:
Thanks for taking the trouble to write all this (when all I asked for was a link). It gives a good flavor of what you're talking about. I used to read network news inside of emacs, using a mode called gnus, decades ago, and before I switched to vim. That's probably an example of a complex application that could be written within emacs because of the architecture that you describe. This extensibility is tempting me to try emacs again, because vim is so hard to extend, but I don't want to give up vim's pure editing efficiency. I guess I need to see how well evil (vim mode for emacs) works in practice.
IMHO, Evil Mode makes to use emacs doable OTOH, the fact that there are so much extensions and possibilities under the hood makes to use emacs mandatory! :P
I've been using emacs for a while (after vim for 6, 7... 8 years?) and have recently been looking at evil. I can't seem to find any really comprehensive tutorials for Evil mode, though I haven't put a lot of work towards it.
The great thing about evil is you can enable it in your .emacs and it mostly just works. But even better, because this is Emacs the vi layer is also easily extendable. Here's an example of adding other operators (adding `mit`, `mit>` etc): https://github.com/ZaneA/Dotfiles/blob/1858a02de25e25de20e96...
I'm tempted by emacs/evil as well, but everytime I try to make the switch I run into two problems:
1) intial emacs setup: it feels really tricky to just get basic editor sanity configured (auto-indent, syntax highlighting, etc.)
2) I worry that by using evil mode, I lose out on a lot of the goodness of emacs major and minor mode's. Do you remap everything to be more vi-ish for each mode that you use? I guess that is fine but the transition just seems daunting.
I second the idea of using Prelude to start with Emacs, along with the package manager which makes the initial setup and adding modes/stuff very easy (see http://melpa.milkbox.net).
I haven't seen any lost functionality by using evil; modes that define bindings continue to work, and the usual C-x/C-c/M-x/whatever works fine with evil. There is the odd major mode that doesn't work with evil out of the box (usually if it redefines j/k) but most have no problems. You can still use most (all?) regular Emacs movement commands with evil as well.
Getting the vim portions working in emacs was surprisingly easy. I didn't have a significantly modified vim so there wasn't much to port. Stuff like mapping jj to esc and Y to y$ was simple enough. I don't notice any difference between evil and vim.
I'm still trying to figure out the emacs portions of emacs. I don't have the work flow quite right; things like buffer management, saving sessions, shell in emacs, etc.
3. For 3 I just use built in shell-mode, but what is really neat is saving your shell session to a file "C-x C-w" every so often. Then if you reboot or crash or whatever, emacs-desktop will bring back all your open files and all your shells. I usually have 3, regular bash, a sql session, and maybe an ssh session to somewhere, all with a full record of what I've done, sometimes going back months.
Just wanted to recommend reading more Pratchett. :) I bought a couple more novels last week and couldn't help but finish one (Equal Rites) less than twenty four hours later.
This sounds intriguing. Do you have a pointer to something I could read that might expand on what you're saying here? (Aside from the emacs source.)
Amusing coincidence: the only Pratchet book I've read is The Fifth Elephant.