Prevent crawlers from accessing deep forum pages

[sajal]

We have got a forum site that has a lot of threads/posts. Thus, if we go to the forum listing page, it has a "lot of" pages (In paging) of threads.

Now some crawlers access these deep forum pages, like 60000th page, 70000th page etc.

"GET /forum/latest-news/page-64099 HTTP/1.1"
"GET /forum/latest-news/page-65000 HTTP/1.1"

This creates a heavy load on the db because MySQL offset queries will be heavy to fetch this kind of content. And eventually, the site goes down in front-end for a...

Read more
Prevent crawlers from accessing deep forum pages

free plugins
minecraft paid for free
mcmarket
mc-market
free minecraft paid plugins
free minecraft plugins
spigotmc
SpigotMC
aac free
ewg free
Minecraft Premium Plugins
ewg free download
litebans free
epicworldgenerator free
free schematics
leaked schematics minecraft
schematics
schematics leaked
schematics free
minecraft schematics free
paid minecraft schematics free
paid minecraft schematics for free
free cracked plugins
free cracked minecraft plugins
plugins
featherboard download
featherboard plugin download
antiaura download
xenforo leaked
xenforo for free
xenforo addons