Solved FreeBSD Handbook License

Hi guys,

I have a question, I am working upon the HTML documentation in order to have this available, locally, as GemText (GMI) file which is quite handy to read on a terminal with a Gemini Client.

I do wonder if the documentation is released in any permissive license that allow it to be published outside the FreeBSD main domains. Also I wonder if it can be published as a unofficial version while still preserves all the copyright/copyleft attributions to their respective owners.

Long story-short I would just have an online documentation available in GemText format.

freezr
 
drhowarddrfine this is true but this doesn't mean automatically are enforced some rights or restriction.

Also people are free to contribute and since the moment we are in the BSD space you may imply a certain extent of freedom is allowed.

Anyway I haven't noticed there is an entire section to explain how to contribute to the documentation from where I could find better information:

 
full disclosure: I haven't heard of Gemini/GemText before this thread and just glanced over a few search results;

I see there are a bunch of html/GemText converters on github, but from your wording I imply this won't produce a pleasantly readable form of GemText, but they might serve as a first step before manually optimizing the produced cleartext. OTOH, the original form of the handbook seems to be AsciiDoc (https://docs.freebsd.org/en/books/fdp-primer/asciidoctor-primer/), so you might want to use that as a source for automated conversions via a makefile (as it is done for other formats). The AsciiDoc syntax seems to be *very* close to GemText (or other Markup), so it might even be a simple case of replacing the AsciiDoc markup with the GemText equivalent.

With a privately maintained, "unofficial fork" of the full handbook, I see the risk of it getting out of sync rather quickly and/or abandoned at some time. So if it is feasible to produce GemText from the official AsciiDoc source via the same procedure as other formats are generated, and you think the GemText format might be beneficial for other people, you might want to write to the documentation mailinglist and propose GemText as an additional format besides html, pdf and epub.
If it gets rejected you can still maintain the conversion toolchain that came from this as your private project.
 
… handy to read on a terminal

Text​


Legible and searchable, but don't expect working links:
  1. sudo pkg install graphics/poppler-utils misc/freebsd-doc-en www/lynx
  2. pdftohtml -stdout -i /usr/local/share/doc/freebsd/en/books/handbook/handbook_en.pdf | lynx -stdin
1648170033160.png1648170074954.png

<https://unix.stackexchange.com/a/36202/13260>

GemText​


… Long story-short I would just have an online documentation available in GemText format.

I see there are a bunch of html/GemText converters on github, …

I didn't look for those specifically, but via <https://github.com/topics/gemtext>:
– and via the dioscuri page (off-topic, but interesting):
Universal Syntax Tree used by @unifiedjs

Conversions from HTML might be bugged by a severely misplaced table of contents, which freezr has seen already: <https://forums.freebsd.org/posts/560750>.
 

HTML​


… locally, … handy to read on a terminal …

The long (non-split) version is appealing. For example, saved by an extension to Firefox:

lynx /home/grahamperrin/Documents/IT/BSD/FreeBSD/docs/handbook-portal.html

1648173580323.png

Unfortunately, many links will fail to remain within the local file. FreeBSD Basics, for example.

HTML, made​


In a nutshell, after installing requirements:
  1. cd /usr/doc/documentation && make html
  2. lynx /usr/doc/documentation/public/en/books/handbook/book/index.html | <file:///usr/doc/documentation/public/en/books/handbook/book/index.html>
– or, if developing, and if a live view of changes will be required:
  1. cd /usr/doc/documentation && make run
  2. in a separate tab or window, lynx http://localhost:1313/en/books/handbook/book/ | <http://localhost:1313/en/books/handbook/book/>
I don't know whether the HTML is (or can be) stored anywhere after you allow the local server to stop. It's all quite new to me.

 
Thanks to all for your kind replies and interest!

Being more specific...

At sko your concerning are totally true; as a matter of fact mine procedure would be fixing manually the conversion from the single html version to GemText; I can see there quite mismatching but the worst scenario are tables; this latter aren't very suitable on GemText therefore must be editing by hand as preformatted test.

Someone with more skills and knowledge may try to convert AsciiDoc to GemText but this is not me, I don't go over than a "find & replace".

drhowarddrfine as I thought the copyright is quite permissive:

The FreeBSD Documentation License​

Copyright 1994-2022 The FreeBSD Project. All rights reserved.
Redistribution and use in source (AsciiDoc) and 'compiled' forms (HTML, PDF, EPUB and so forth) with or without modification, are permitted provided that the following conditions are met:
  1. Redistributions of source code (AsciiDoc) must retain the above copyright notice, this list of conditions and the following disclaimer as the first lines of this file unmodified.
  2. Redistributions in compiled form (Converted to PDF, EPUB and other formats) must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS DOCUMENTATION IS PROVIDED BY THE FREEBSD DOCUMENTATION PROJECT "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FREEBSD DOCUMENTATION PROJECT BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Well since this is solved, the only limitations are my skills which barely reach the 0.1%... ?‍♂️
 
I found that is possible converting from AsciiDoc to Plain Text:

5. Generating Plain Text Files​

AsciiDoc3 does not have a text backend (for most purposes AsciiDoc3 source text is fine), however you can convert AsciiDoc3 text files to formatted text using the AsciiDoc3 a2x3 toolchain wrapper utility.
Generate Plain Text
$ a2x3 -f text doc/test.txt

Maybe for me would be easier this procedure AsciiDoc Plain Text GemText... ?
 
… from AsciiDoc to Plain Text: …

Maybe not as plain as you'd like.

From <https://asciidoc-py.github.io/userguide.html#_generating_plain_text_files>, with added emphasis:

… you can convert AsciiDoc text files to formatted text …

Code:
% a2x
Usage: a2x [OPTIONS] SOURCE_FILE

A toolchain manager for AsciiDoc (converts Asciidoc text files to other file
formats)

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -a ATTRIBUTE, --attribute=ATTRIBUTE
                        set asciidoc attribute value
  --asciidoc-opts=ASCIIDOC_OPTS
                        asciidoc options
  --copy                DEPRECATED: does nothing
  --conf-file=CONF_FILE
                        configuration file
  -D PATH, --destination-dir=PATH
                        output directory (defaults to SOURCE_FILE directory)
  -d DOCTYPE, --doctype=DOCTYPE
                        article, manpage, book
  -b BACKEND, --backend=BACKEND
                        name of backend plugin
  --epubcheck           check EPUB output with epubcheck
  -f FORMAT, --format=FORMAT
                        chunked, epub, htmlhelp, manpage, pdf, text, xhtml,
                        dvi, ps, tex, docbook
  --icons               use admonition, callout and navigation icons
  --icons-dir=PATH      admonition and navigation icon directory
  -k, --keep-artifacts  do not delete temporary build files
  --lynx                use lynx to generate text files
  -L, --no-xmllint      do not check asciidoc output with xmllint
  -n, --dry-run         just print the commands that would have been executed
  -r PATH, --resource=PATH
                        resource file or directory containing resource files
  -m FILE, --resource-manifest=FILE
                        read resources from FILE
  --resource-dir=PATH   DEPRECATED: use --resource
  -s, --skip-asciidoc   DEPRECATED: redundant
  --stylesheet=STYLESHEET
                        HTML CSS stylesheet file name
  --safe                DEPRECATED: does nothing
  --dblatex-opts=DBLATEX_OPTS
                        dblatex options
  --backend-opts=BACKEND_OPTS
                        backend plugin options
  --fop                 use FOP to generate PDF files
  --fop-opts=FOP_OPTS   options for FOP pdf generation
  --xsltproc-opts=XSLTPROC_OPTS
                        xsltproc options for XSL stylesheets
  --xsl-file=XSL_FILE   custom XSL stylesheet
  -v, --verbose         increase verbosity
% a2x --format=text --destination-dir=/tmp /usr/doc/documentation/content/en/books/handbook/book.adoc
a2x: WARNING: --destination-dir option is only applicable to HTML and manpage based outputs

% du -h book.xml
2.4M    book.xml
% rm book.xml
% pwd
/tmp
% a2x --format=text --destination-dir=/tmp --lynx /usr/doc/documentation/content/en/books/handbook/book.adoc
a2x: WARNING: --destination-dir option is only applicable to HTML and manpage based outputs

% du -h book.text.html
2.2M    book.text.html
% head -n 10 book.text.html
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="AsciiDoc 10.1.3">
<title></title>
</head>
<body>
<p>---
title: FreeBSD Handbook
% lynx -dump ./book.text.html > /usr/home/grahamperrin/Documents/IT/BSD/FreeBSD/FreeBSD\ handbook.txt
% cd
% amfora Documents/IT/BSD/FreeBSD/FreeBSD\ handbook.txt
%

End result:

1648384981547.png
  • not excessively wide
  • there remain some chunks of text that should not be seen in a refined product, for example:
Code:
                --- title: Chapter 24. Updating and Upgrading FreeBSD part: Part III.       ▒
                System Administration prev: books/handbook/l10n next:                       ▒
                books/handbook/dtrace description: Information about how to keep a          ▒
                FreeBSD system up-to-date with freebsd-update or Git, how to rebuild        ▒
                and reinstall the entire base system, etc tags: ["updating",                ▒

Code:
   --- title: Chapter 24. Updating and Upgrading FreeBSD part: Part III.
   System Administration prev: books/handbook/l10n next:
   books/handbook/dtrace description: Information about how to keep a
   FreeBSD system up-to-date with freebsd-update or Git, how to rebuild
   and reinstall the entire base system, etc tags: ["updating",
   "upgrading", "documentation", "FreeBSD-STABLE", "FreeBSD-CURRENT",
   "Security Patches"] showBookMenu: true weight: 28 path:
   "/books/handbook/" ---
     __________________________________________________________________

                         Updating and Upgrading FreeBSD

   include::en/mailing-lists.adoc[] include::en/teams.adoc[]
   include::en/urls.adoc[]
     __________________________________________________________________

Synopsis

   FreeBSD is under constant development between releases. Some people
 
Lazy plain text (not from AsciiDoc):

View attachment 13467

freezr I see that the space bar presents a search field, however I could not see a way to search within the current page. Am I missing something?

The window is necessarily wide.


This is Amfora but my favorite TUI client is Telescope:


Anyway when you press space you open the URL bar so you can add an URL or a number of a link.
 

freezr maybe some of what's there can get better plainness. I'll leave that with you :) don't intend to run any of it for myself.

Many Gemini clients can open TXT files, in that case text is rendered as mono-space with quite layout limitations. Your file would changed a lot if just renamed as .GMI you would observed a different rendering with the text better flushing across the view-port. However Gemini has limited lines to render text which are:
  1. simple paragraphs;
  2. headings (#);
  3. unordered lists (*);
  4. quotations (>);
  5. preformatted (```);
  6. last but not least, links (=>).
The point is once you get the plain version you have to edit manually the file to make it a GemText file. This is not ideal but is the only strategy I have come out so far... ?‍♂️
 
Given the frequency of changes to the FreeBSD Hanbdook <https://cgit.freebsd.org/doc/log/documentation/content/en/books/handbook>, I'd avoid any workflow that will require manual edition …
exactly *this*

freezr
Why are you focusing on converting from HTML? Have you looked at the AsciiText source? The markup looks *very* close to GemText, so if you absolutely want/need GemText (for which there isn't even a reader available on FreeBSD?), start with the AsciiText as a baseline.
 
sko & grahamperrin

My goal is having GemText documentation that I (and everyone) can easily browse online whether is a terminal, Lagrange or Android.

The point of converting from HTML to GemText born because it exists a tool to do so, although AsciiDoc can be exported as plain text and therefore would be easier convert (manually) the plain version into GemText even though the documentation is HUGE!

I don't have the ability to parse the AsciiDoc file to strip everything off and converting the rest in GemText, I can do it only manually: I don't have any other means... ?
 
Back
Top