Website Crawler 3 is a powerful application capable of crawling hundreds of thousands of pages looking for information.
This is source code. Once you purchase Website Crawler 3 it’s yours to modify, skin, sell – do whatever you want with it.
You should have the Pro or Dev version of Ubot Studio. You should also have the Heopas Plugin and File Management Plugin. Both are free. Links are included in the “Read Me First” file.
V1 users – how to upgrade.
Besides the search engine idea, you could rank the recipes and create a site of only the best recipes (like 4.5 stars or higher). You could create your own rating system based on the star rating, how many people made it, how many reviews etc. You could also rate the difficulty based on the number of ingredients and how long it takes to make. This would allow you to make a site with only the easiest recipes (and maybe the highest rated easy recipes).
- Capable of scraping 100k+ pages
- Extremely memory efficient
- Powerful internal URL detection
- Easy to modify to scrape whatever you want!
- Ability to save huge amounts of results without Ubot memory spiking
- Ability to crawl multiple sites back to back
- Add in proxies (supports username:password proxies too!)
- Whitelist filter (must contain) for URL, Title or Innertext
- Blacklist filter (must not contain) for URL, Title, or Innertext
- Ability to use both filters at once!
- Scrape by regex, you don’t even have to modify the code to use this
- Built-in regular expressions that you can modify for Emails, Phone Numbers, Files, Proxies
- Ability to remove query if you want (that http://website.com/?this_is=the_query)
- Powerful extension filter built in which filters pages that can cause problems, you can modify this right in the UI
- Set a custom user agent if you want
Upgrade Info (V1-2 users only):