Coding a website crawler...

[ForgottenCreature]

Hey guys,
I am working on programming a website crawler so I can quickly grab relevent articles from articlecity.com and goarticles.com.

I am having troubles thinking of a method to go about this. Can some please point me in the right diretion. Ideas you may have?

Thanks.

[IntellEJent]

Use PHP Cron Jobs to crawl the site every so often. :)

Use the PHP 5 DOM to look at the HTML. :idea:

[niallj]

Makes sense to me :D
The only thing is, if they aren't valid XHTML they'll give problems with the DOM extension, won't they?