xml-sitemaps is crawling extensively my site, and stops at 500 page limit, though there are relatively small number of pages in the site.
Most of indexed pages comes in form of
New SEF is ON, meta tags for crawler is NO.
How can I stop from crawling eventsbyday/xxxx, etc... or this is bug in xml-sitemaps?
How can be sure that search engines are not indexing so many useless pages?
The link is correct - it just shows - No events on that date.
Most of indexed pages comes in form of
xxxxx/eventsbyday/2016/4/6/-
- note year 2016, I have no events in 2016.New SEF is ON, meta tags for crawler is NO.
How can I stop from crawling eventsbyday/xxxx, etc... or this is bug in xml-sitemaps?
How can be sure that search engines are not indexing so many useless pages?
The link is correct - it just shows - No events on that date.