Crawling via a proxy server
Posted: Mon Apr 21, 2025 3:56 am
If you want to use a proxy server for crawling, you can select "System → Proxy" in the Settings menu (Configuration). Here you can enter the proxy address and port. If you change these settings, it's important to restart Screaming Frog for the change to take effect.
Crawling via different “user agents”
If you want to perform crawling with a different user agent, there's also an option for this. Under Configuration → User Agent, you can select or enter other user agents.
Screaming Frog - User Agent
Summary: Configuration → User Agent
Crawling older websites or websites with crawl delay
In rare cases, some older servers may not be able to process the standard number of south africa phone number data requests per second. In this case, Screaming Frog offers the option to change the crawler speed. To do this, go to the "Configuration" menu and select "Speed." Here, you can set the maximum number of threads that should be executed simultaneously. You can also specify the maximum number of URLs requested per second.
Summary: Configuration → Speed
Let the spider crawl outside a specific folder
It's also possible to have the spider analyze additional pages located outside the starting folder. This allows the crawl to start at /leistungen and still crawl and analyze /blog. To do this, the "Crawl outside the starting folder" option must be enabled in the "Configuration" menu under the " SEO Spider" sub-item.
Crawling via different “user agents”
If you want to perform crawling with a different user agent, there's also an option for this. Under Configuration → User Agent, you can select or enter other user agents.
Screaming Frog - User Agent
Summary: Configuration → User Agent
Crawling older websites or websites with crawl delay
In rare cases, some older servers may not be able to process the standard number of south africa phone number data requests per second. In this case, Screaming Frog offers the option to change the crawler speed. To do this, go to the "Configuration" menu and select "Speed." Here, you can set the maximum number of threads that should be executed simultaneously. You can also specify the maximum number of URLs requested per second.
Summary: Configuration → Speed
Let the spider crawl outside a specific folder
It's also possible to have the spider analyze additional pages located outside the starting folder. This allows the crawl to start at /leistungen and still crawl and analyze /blog. To do this, the "Crawl outside the starting folder" option must be enabled in the "Configuration" menu under the " SEO Spider" sub-item.