(Author here)

In an act of divine justice, my website is down.

https://web.archive.org/web/20210711201037/https://billpg.co...

(I'll send you a donation. Thank you!)

Lol I love that it was edited out right away. I probably agree with that decision but this was a really cool project. I have followed a few YouTube channels which make graphic visualizations for Generations and I think they would really appreciate if you shared this Tech with them

For anyone who is interested: I used to work with a guy named Richard Wang, who indexed Wikipedia as his training data set in order to do named entity recognition. He'd be a good person to talk for anyone pursuing this.

Here's a demo: https://www.youtube.com/watch?v=SyhaxCjrZFw

I have a basically English heritage. If I counted unrelated ancestors back through 45 generations I reckon I should have had 35,184,372,088,832! Since the population of England back in the 800s was probably a few million, what are the chances I am NOT a descendent of King Alfred?

I found that work very interesting. So I downloaded the data by OP (thanks for publishing it!) and compared it with Wikidata. The results are here:

http://simia.net/wiki/Main_Page

Loosely related: I had some fun working on a database with all events parsed from Wikipedia [0]. I parsed the entire wiki texts using a language model, extracting date/time/location and what happened then and there. The location data was then fed into a local openstreetmap instance to get the coordinates. I then built a front end to allow querying events near by, in time or in space. The UI is a bit clunky because my database is too large and I had little experience working with them. But building it was fun.

0: https://whataday.info/

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps (https://dumps.wikimedia.org) to the fully-featured API with a nice sandbox (https://en.wikipedia.org/wiki/Special:ApiSandbox) and Wikidata (https://wikidata.org).

Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...

Where's the profit?