Diez B. Roggisch deets at nospam.web.de
Tue Feb 7 12:16:51 EST 2006

• Previous message (by thread): Mark Hammond's pipe service
• Next message (by thread): Is Python good for web crawlers?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

Tempo wrote:

> Why do you say that the bottleneck of the crawler will always be
> downloading the page? Is it becasue there isn't already a modual to do
> this and I will have to start from scratch? Or a bandwidth issue?

Because of bandwidth - not necessarily yours directly, but the maximum flow
between your uplink and the site in question. It will always take at least
a fractioin of a second up to several seconds until the data is there - in
that time, lots of python code can run.

Diez

---

• Previous message (by thread): Mark Hammond's pipe service
• Next message (by thread): Is Python good for web crawlers?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

More information about the Python-list mailing list