@matigo No, I don't really need the content, as I have already scraped that.
I need metadata, if you will, about each page.
I found a thing called Scrapy [doc.scrapy.org] that seems like it ought to be able to do what I want, but it is way beyond my abilities to implement, and not really worth it for this one job. I doubt that there will be other jobs like it.
Maybe I should just pony up and pay the online service.
// @japchap