A while back I wrote a little sample of a web crawler. It was not terribly sophisticated but has been the biggest draw to this blog through search engines for some time. This is likely because it ranks very high (why I am not sure) in Google's rankings. For example, my crawler sample page ranks #1 in Google for "c# web crawl" and #3 for "c# web crawler". Neat.
So the other day I dusted it off and started reworking it. It was such a simple sample and I thought some improvements could be made. Lo and behold, I just got an email about a different web crawler written in c#, which you can find here. I haven't downloaded and looked at the source yet, but I figured I would point it out.
Filed Under: .NET
Comments
- .NET (14)
- Ajax (16)
- ASP.NET (18)
- ASP.NET MVC (9)
- Books (22)
- C# (1)
- CSS (3)
- Development (13)
- Drawing (13)
- Entertainment (6)
- Flash (1)
- Fun (2)
- General (81)
- Greek (8)
- Html (1)
- IIS (1)
- Pylus (10)
- Silverlight (7)
- StructureMap (1)
- Tech Review (15)
- Visual Studio 2010 (1)
- Windows7 (1)
- Winforms (2)
- WPF (5)
- April 2006 (12)
- May 2006 (2)
- August 2006 (2)
- September 2006 (10)
- October 2006 (19)
- November 2006 (20)
- December 2006 (19)
- January 2007 (16)
- February 2007 (8)
- March 2007 (4)
- April 2007 (10)
- May 2007 (6)
- June 2007 (6)
- July 2007 (5)
- August 2007 (9)
- September 2007 (4)
- October 2007 (2)
- November 2007 (1)
- December 2007 (1)
- January 2008 (1)
- February 2008 (1)
- March 2008 (1)
- April 2008 (1)
- December 2008 (2)
- January 2009 (10)
- February 2009 (2)
- May 2009 (1)
- June 2009 (1)
- July 2009 (3)
- August 2009 (3)
- September 2009 (1)
- October 2009 (10)
- November 2009 (2)
- December 2009 (5)
- January 2010 (3)
I was trying to devise a method to crawl over the website like Yahoo mail by filling in the authentication details, never been much of a 'web' developer. The code helped me a lot to understand the details and see the 'Big Picture'.
Thanks!