"Syntax error" feature request

gutiersa · May 28, 2020

Hello:

I have noticed a few files in FreeBSD which require a blank line at the very end, otherwise they give an error message such as:

Code:

Syntax error at ' '

or something similar.

Is it possible to:
1. Start a list somewhere, where users of FreeBSD can add the name and location of such files, and
2. Add something like this at the very top of the file, so that we (non-programmers and non-sysadmins) don't go bananas trying to figure out the problem?

Code:

# BEWARE
# this file requires an empty line at the very end!
# BEWARE

Thank you for your consideration.
Sandra

PS: so far: /etc/nsswitch.conf and /etc/pf.conf

ralphbsz · May 28, 2020

Do you mean blank line? Or do you mean a newline at the end of the last line? I just looked at my FreeBSD system, and it doesn't have a blank line at the end of either one (here's pf.conf, nssswitch is unmodified from the install default)

Code:

...
# Other than that, everything goes through unmolested.
pass all

If you are talking about newline at the end of the line: Yes, it would be nice if the parsers in all the tools would clearly identify that. But given that many tools have their own parsers, and writing parsers is often done with bad ad-hocery, that's unrealistic. Adding a comment to the top of the file that it needs a newline at the end is a bit over-the-top, since that's sort of assumed with text files. But what might be realistic would be to add a very simple syntax checker to the daily/weekly/monthly checks. It could have a list of files that are known to be binary or unicode in /etc/ and /usr/local/etc (the others must be straight text), and it could check that text files have no binary in them, unicode files are correctly encoded, and all but binary files end with a newline. That's something that could be proposed on the mailing list for inclusion in base, and someone could write a prototype and submit it. Hint hint blink blink: since you mentioned it, maybe you could volunteer?

zirias@ · May 28, 2020

ralphbsz said:
Adding a comment to the top of the file that it needs a newline at the end is a bit over-the-top, since that's sort of assumed with text files.

On a loosely related side note, ports-mgmt/portlint complains about newlines at the end of some port files and calls them "blank lines". I agree, ending each line with a newline, including the very last one, seems to be more common, but unfortunately, there is no real "standard" about that.

Given some experience, you will know about that particular problem, and adding a newline is one of the first things you try when some parser complains

Phishfry · May 28, 2020

This might using be off base but you are FreeBSD to edit those files correct?
Windows text editors can cause mayhem editing FreeBSD text files. Particularly <LF> and <CR>.
Notepad++ is able to edit FreeBSD files correctly.

memreflect · May 28, 2020

gutiersa said:
I have noticed a few files in FreeBSD which require a blank line at the very end, otherwise they give an error message such as:

Code:

Syntax error at ' '

or something similar.

Traditionally, a "line" of text ended with a newline sequence, whether that is line feed (LF) on Unix, carriage return (CR) on classic Macs, or CR LF on DOS and Windows. Many terminal-based editors will ensure the file ends with a newline sequence when writing the file for this reason. In most graphical text editors, this looks like a blank line at the end of the file because the insertion point must be placed after the last newline to be able to add more lines of text.

Unfortunately, many config file parsers rely on this definition of "line", some built using parser generators like yacc(1) with a grammar that requires a newline character to recognize a line for simpler parsing behavior:

Code:

config_file  : config_entry
             | config_file config_entry
             ;
config_entry : "\n"              /* empty line: do nothing */
             | expression "\n"
             ;
expression   : KEY "=" VALUE     { set_config(KEY, VALUE); }
             ;

A line that ends without a newline character is consequently considered a syntax error because there is no match for an expression without a newline at the end. This blank line at the end of files may be annoying, but it's just typical behavior of most parsers, and you should try to remember to keep/add the newline if your text editor doesn't already do it for you.

gutiersa · May 29, 2020

ralphbsz said:
Do you mean blank line? Or do you mean a newline at the end of the last line? I just looked at my FreeBSD system, and it doesn't have a blank line at the end of either one (here's pf.conf, nssswitch is unmodified from the install default)

Code:

... # Other than that, everything goes through unmolested. pass all

If you are talking about newline at the end of the line: Yes, it would be nice if the parsers in all the tools would clearly identify that. But given that many tools have their own parsers, and writing parsers is often done with bad ad-hocery, that's unrealistic. Adding a comment to the top of the file that it needs a newline at the end is a bit over-the-top, since that's sort of assumed with text files. But what might be realistic would be to add a very simple syntax checker to the daily/weekly/monthly checks. It could have a list of files that are known to be binary or unicode in /etc/ and /usr/local/etc (the others must be straight text), and it could check that text files have no binary in them, unicode files are correctly encoded, and all but binary files end with a newline. That's something that could be proposed on the mailing list for inclusion in base, and someone could write a prototype and submit it. Hint hint blink blink: since you mentioned it, maybe you could volunteer?

Hmmm, yes, it probably is my editor, I use ee.
Regarding volunteering, sure thing, guide me. Consider it done.
So how would I propose that in the mailing list? And which mailing list?
I am not sure about the prototype. I am not a programmer or a sys admin. I would like to learn python though.

Emrion · May 29, 2020

Whatever you call this, it's a bug. I remember experiencing that with my first try on pf. It's just stupid and easily correctable.

zirias@ · May 29, 2020

Emrion said:
Whatever you call this, it's a bug.

Absolutely not. The last line in a multi-line text file is an edge-case. A parser can either make "newline" part of the grammar, so that only something terminated with a newline is a valid line, or it can treat newline as a delimiter for lines. Both decisions are fine. Similar problems exist with other things, for example:

Lists in JSON are delimited by commas. There must not be a comma following the last list item.
Statements in C are terminated by semicolons. The very last statement in a block also needs one.

So, the problem here is at best there's no universally agreed standard about newlines in textfiles. The most user-friendly parser would just allow both. As most applications of textfiles would ignore empty lines, this is already the case with most parsers that don't require a terminating newline. But still, doing it differently is not a bug.

Emrion · May 29, 2020

I wrote a parser (C++), fall into this same problem and corrected it. No standard and no excuse for that.

gutiersa · May 29, 2020

Emrion said:
I wrote a parser (C++), fall into this same problem and corrected it. No standard and no excuse for that.

Well, can we use your parser for the
syntax checker above?

ralphbsz said:
ut what might be realistic would be to add a very simple syntax checker to the daily/weekly/monthly checks. It could have a list of files that are known to be binary or unicode in /etc/ and /usr/local/etc (the others must be straight text), and it could check that text files have no binary in them, unicode files are correctly encoded, and all but binary files end with a newline.

jmos · May 29, 2020

Zirias said:
So, the problem here is at best there's no universally agreed standard about newlines in textfiles.

I've learned for myself (and I'm telling it to others) that plain text config files should always end with a newline. Without it it may not work, with it it works … always. Or are there cases / software in which a newline isn't a good idea? Also: "cat file.txt" f.e. puts my cursor in an newline, while without a newline I'm unhappy with my xterm

I've wrote my own editor for programming - and it adds a newline by saving a file (if it's not there). Was that a bad idea?

Emrion · May 30, 2020

gutiersa said:
Well, can we use your parser for the
syntax checker above?

Lol. You're kind to me. I am by no mean a professional and I have no illusion on the quality of my code. So, it is certainly a bad idea. My point was just to say: if someone like me can do that, everyone can.

You can consider EOF as LF in all tests or, it's probably smarter, add LF at the end of data, if it is the last chunk of the file, and then parse it.

I don't know why they left this behaviour, but the result for a newbie is to lose minutes, if not hours, trying to figure out where is the error.

20-100-2fe · May 30, 2020

They probably left this behavior because when your job implies dealing with a great many files such as these, you want them to be easy to read.

A typical situation is when you edit a file (e.g. to diagnose a production incident), you often want to see the lines at its end and you type something like Shift-Ctrl-End. With most text editors, this key combination takes you at the end of the last line of the file. If it's long, you no longer see the rest of the file, not even the beginning of the line. And you curse the colleague who didn't care properly creating his file, because it adds a little more to an already high pressure on your shoulder in a difficult moment.

If this happens to you once a year, you don't care. If it happens to you 10 times a day, you very quickly learn to properly end ALL your lines, even the last one, believe me! And this has nothing to do with syntax errors, but only because it makes editing text files by a human being so inconvenient.

This is why such requests' priority is so low they are never taken into account: the time of volunteer developers is better employed fixing security issues or improving hardware support or performance, for instance. And don't forget the time they spend on FreeBSD is their spare time, it's not extensible and they have a life outside FreeBSD, a family, friends... And a daytime job that uses up a lot if not most of their energy.

I hope these explanations will help you better understand the situation and will also help you make more useful/relevant decisions as to your own needs.

Emrion · May 30, 2020

20-100-2fe said:
I hope these explanations will help you better understand the situation and will also help you make more useful/relevant decisions as to your own needs.

I have no special need. I never complained about that outside this thread because it's not important. That said, no one here gave a real argument why they left such a behaviour. And frankly, I don't know why people try.

This subject is closed for me.