The problem of special characters (of which space is only one example) in file names is awful, and causes lots to hassles. It can be solved, but the solution requires being very careful and thorough.
To begin with: File names are not character strings that have the semantics of an encoding (such as UTF-8 or ISO8859-2), but are arrays of bytes. End users have to be careful to be consistent in their use of encodings: If you create a file (write the file name) using a UTF-8 name, then please do not look at the directory (read the file name) using an ISO encoding; you will see something that is "wrong" and confusing. My suggestion is to always use UTF-8, consistently in all applications.
File names can contain any byte other than "/" and NUL. As you said, spaces in file names are bad. Other characters are actually worse: newlines are very amusing (the output of ls becomes completely garbled), and other special characters can be better or worse. For example, it is legal to create a file whose name is the ANSI escape sequence for "print everything in blue", and then when doing ls, all text afterwards will be blue. More amusing is a file whose name is the escape sequence for "clear screen". The only defense against such things is discipline: Doctor, it hurts when I do this ... then just stop doing it. Personally, I try to avoid file names with spaces in them (but grudgingly allow them, they are just too useful), and non-printable special characters are completely out. My backup software warns me if it finds any file names that have characters outside of the range 0x21-0x7E, if they start with a minus sign, or contain a "#" sign (which many scripting languages take to mean "beginning of comment"), and when I see these warning, I clean up after myself.
If you process file names using "real programming languages" (perl, python, C++, ...), all this is not really a big problem, since all of these languages have a concept of a well-defined string, and strings can contain special characters. The real pain comes in shells, which are used both interactively and as shell scripts (which are really programs written in a pedestrian language). In interactive use, both quoting and escaping works, as pointed out above: cd New\ York or cd "New York". In scripts, only quoting is sensible. And there one has to be careful, and learn the quoting rules of the shell one is using: If $a is a variable that contains special nonsense (like quotes, dollar signs, and backticks), then '$a' and "$a" probably will work differently. I'm too lazy to look up the general rules and teach them right now, but from vague memory: in sh- and bash-scripts the single quote '$a' prevents expansion of nearly all special characters inside, while the double quote "$a" does not; but please read the documentation carefully before relying on that.
Lastly: I know nothing about vi. In emacs, you can set various line end modes to automatically suppress visualizing the carriage return (^M) at the end of the line; that is typically called DOS mode. But be careful: Just because you can't see the ^M doesn't mean it is not actually there.
