Other Help re web content download script


Well-Known Member

Reaction score: 11
Messages: 459

Greetings all,


for an upcoming trip, I would like to download publicly available maps from:
https://geonames.usgs.gov/apex/f?p=262:18:0::NO:RP,18:: .
As understood, upon entering the required fields, at least State, Scale, and Historical, the Search function forms an URL of pre-determined format, e.g., for Arizona, 24000, Historical, in the form:
Then one can download a *.csv file using the link Download to Excel (csv) File.

However, the size of the *.csv file is limited to 5000 element, thus the download must be accomplished by adding a letter, e.g., A, into the Map Name, upon which the URL is modified to:
Since the size of the *.csv file is now less than 5000, i.e., 319, the *.csv file can be downloaded.

Statement of a problem

There appears to be an interplay between the URL loading the page for the selected fields and the URL for the Download to Excel (csv) File, because the latter is always of the same form:

Since I know nothing about web design, my naive approach is to write a script that:

1. generates the URL loading the page for the selected fields;
2. loads the page using the URL, and confirms that the page loaded; and
3. "clicks" the URL for the Download to Excel (csv) File.

I already have a
#! /bin/sh
function for 1, but do not know how to accomplish 2-3. Any help in this matter would be appreciated.

Kindest regards,