Published the dataset. I'm on the front page of Kaggle!
update! its going super super well, the error logging i have in place works perfectly.
2014! 60mb file, 400k ish headlines.... roughly 1/8th of the way there. expected to finish in maybe 40 hrs or so?
started scraping today, it's going pretty well! (3/120 months done...)
the ramblings of a madman
fun thing ive noticed: the most viewed news sites have the most sane web design (NYtimes, CNN, etc) and then the more you go down, the worse the web design is to parse. (CNBC, Daily Mail, etc)
WITH THE EXCEPTION OF THE GUARDIAN THEY'RE THE ONLY ONES WHO THOUGHT OF USING <li> ITS SO CLEAN AND CONCISE
