There is one project[0] that accomplishes the same thing, but instead of searching for the titles, it searches for the links to online stores (like Amazon) and then checks if the links are indeed books or not. Might be relevant to you.

[0] http://hackernewsbooks.com/

I did this a while ago (www.hnreads.com, bookbot.io). Manually labelled ~2k comments, trained a NER system, then validated the titles via amazons api. Was pretty easy once I had manually labelled comments. Don't recall my F1/precision/recall scores - they were ok but lower than the state of the art reported in papers.

I would do something like this:

1. Get a few thousands book titles that are "long enough" so that the probability they appear in a sentence without meaning the book is low

2. Search these spans in HN comments and use this corpus to train Stanford's CoreNLP NER

3. Run the NER on all comments

4. Check on Openlibrary or another book DB that the extracted spans are real books titles

I built the simmilar stuff for Reddit using Amazon links. http://booksreddit.com

I'm in the process of adding Goodreads and other book websites to get better suggestions.(Using Amazon API is limiting on scale)

As for your question: I've been researching python natural language processing to get probable named entities and check them against Amazon API. I suggest https://spacy.io that has reasonable named entities extraction. However doing it at a scale might produce lot of books that are named as a common phrases.

I've found experimentally that doing a search on amazon for a book title, even if you're only close, almost always turns up that exact book. However, the product API has really restrictive terms.

I would look at the OpenLibrary dataset - you might be able to match titles in there to comments, or use it to validate the NER output, if you don't want to go the Amazon link regex route.

The entire dataset is available for download, or you can build a prototype with their API - I did this to map speakers to books with https://www.findlectures.com (you can see it if you hover over a name - e.g. https://www.findlectures.com/?p=1&speaker=-Barack%20Obama).

A pretty common structure in comments:

"name of book" by "author_first_name author_last_name"

If you could determine the most common words written before the title begins (read, liked, loved, recommend, etc.) you could probably parse out a lot of titles plus their authors.

You would probably be better served starting with an ISBN database and just looking for spans that match book titles. Not sure if there's one you can download for free, but I saw one on sale for $675.

NER will give you a lot of entities that you need to resolve against something to check if they're actually books anyway, depending on volume you may not find a free API tier.

One problem that you will run into is that X (the phrase in the comment) might be a permutation of a book title. Or perhaps even a title of a book being mentioned in a comment where the author did not intend on it being interpreted as such. This is mostly due to ambiguity & information density in language as well as how we assign names/titles to things.

Ex. If I write a comment saying "The number of our clients went from zero to one instantly", in this sentence, "Zero to One" will match Peter Theil's book.

If you are willing to put up with such "noise" in the output, then you don't need to train a thing or even use ML. Just chunk the given piece of text and look up all the tokens permutations of given length (from 1 up to N) in your SQL Entities Database.

You can generally find this kind of Entities Databases by aggregating a lot of datasets from all over the web or even use something like google books datasets: https://books.google.co.uk/

https://storage.googleapis.com/books/ngrams/books/datasetsv2... - This is an ngram set but there should be one with only book titles somewhere. Can't find it atm but you can search for it yourself!

You could even (if you are brave enough) try to use wikipedia's dumps to mine book titles from articles