Bulk-Testing Structured Data Markup Using Screaming Frog
I published a post yesterday on Moz.com about how to use Screaming Frog and Google’s Structured Data Testing Tool together. The concept allows us to test large sets of webpages for structured markup, such as the following:
- E-commerce product reviews and pricing
- Google+ Authorship
- Rel Publisher
- Event Listings
- Review and price markup on App Listings
- Music Snippets
- Recipes
- Business Reviews
- Just about anything referencing schema.org, data-vocabulary.org, and similar markup.
You can read the full post here: Quick Guide to Scaling Your Authorship Testing with Screaming Frog
Here’s the YouTube video that describes the process:
Adam
I’ve set up a similar thing on SF but sadly Google is very good at spotting bot usage and, even when crawling as slowly as possible, is throwing captchas at me after about 7 – 8 URLs.
The only way around this I can find it to change IP address using a VPN each time the crawler is ‘blocked’, which makes this a manual process that you need to watch over still.. I’m not done yet!
Kane Jamison
You’ll need to use a proxy service to fix this – probably one with built in rotation. It will be much easier than manually switching VPNs. Search “proxies screaming frog” and you should find a few services.
Brandon Lazovic
Nice article! When auditing websites, I often find improper use of structured markup that’s riddled with warnings and errors. I love using Google’s Structured Data Testing Tool to check and correct these issues, but unfortunately, Google is deprecating their tool in favor of the rich results tester, which isn’t as convenient because it only provides markup suggestions if the schema is eligible for rich results.
I talk about this in my latest post, but outside of Google’s Structured Data Testing Tool, Merkle’s Schema Tester is my second favorite for making sure my schema markup is set up properly: https://brandonlazovic.com/best-structured-data-testing-tools-for-schema-markup