Building Competitor SEO Profiles
When I worked agency-side, one of the first things we’d do for every new client was build a profile of all their main competitors across their top target keywords.
This kind of intelligence offered us a broad overview of the niche and how competitive it was, so we could budget effectively and pick our battles.
When I worked in-house I’d do this even more fervently, keeping a close eye on all my competitors and what they were up to throughout the months and years.
As it turns out, this sort of activity is where URL Profiler comes into its own.
Note: No additional paid tools or APIs are required, although you will have to set up a couple of API keys.
Building a Competitor SEO Profile Using URL Profiler
To give us an actual example to work through, we’ll pretend we are entering the fairly competitive space of online flower sales.
We’ll start by looking at the single keyword ‘online flowers’. A Google search (UK) gives us the top 10 that we’ll need to crack if we want to see any kind of organic progress. The first 3 look like this:
Using an extension like Linkclump we can quickly scrape all the links from this SERP and paste them straight into Excel. This gives us a list of links with which to populate URL Profiler. We’ll also select the metrics we wish to pull back, on a domain level and a URL level.
After you click to run the profile, you will very quickly get back a spreadsheet with loads of domain level and URL level data. With a bit of quick formatting, it looks like this:
This snapshot shows some of the domain metrics. Combine this with the URL metrics and you’ll start to get an idea of which sites are ranking and why.
Understanding The Data
To understand the data, it helps to separate the domain metrics from the URL metrics so you can start to see if sites are ranking on the strength of their domain, their URL, or both.
It is easier to see what’s going on if you implement a bit of conditional formatting (check out this guide from Tori Cushing on how to do this).
Domain Link Metrics
First we’ll look at the domain metrics:
This is currently ordered by Citation Flow, although ordering by Moz Domain Authority would yield similar results.
It is clear to see that there is one ultra powerful domain – marksandspencer.com – which currently ranks no. 2. It is interesting to see that the no. 1 ranked site interflora.co.uk isn’t top for any of the domain metrics we are looking at here. Similarly, clareflorist.co.uk does have some strong domain metrics, yet is only ranking at position 9.
URL Link Metrics
We can do the same with the URL metrics:
This time I have ordered the table by ranking, and you can see that the top pages in general have the most links to the URL – the exceptions of course being marksandspencer.co.uk and the second listing for serenataflowers.co.uk
In general within this niche it looks as though you are going to need a lot of good links or a very strong domain to rank in the top 1-5, but you might be able to break the top 10 with a dedicated flowers site and around a hundred decent links to your homepage.
You can look at social share counts in a similar way to see if they appear to have any impact or rankings correlation in your particular niche.
Screen Captures
The bulk screen capture function helps give you a visual reference of the state of the SERPs, which is particularly useful when you return to your benchmark or perform a similar analysis in a few months time.
Beyond The Initial Profile
There are two ways you could build out your data analysis beyond the initial profile above:
- Scale the process to a lot more keywords
- Dig deeper into each result to gain a better understanding of what’s working currently
For the purpose of this post we will just look at the first of these, but the second might be accomplished by analysing the link profile and velocity more comprehensively in Majestic (particularly using Topical Trust Flow) and checking out their historical performance in SEMrush.
Here’s how to scale up the process for lots of keywords:
Scraping The SERPs in Bulk
In addition to our flagship tool URL Profiler, we also offer a number of free tools for internet markets, all of which can be found here.
One of these is called the Simple SERP Scraper, and it will do exactly what it says on the tin: scrape the SERPs for you.
To scale up our profiling, we just need to download the software (free for Windows or Mac) and fire it up. Then add in all the main keywords we are targeting and hit scrape:
On the default settings it will begin a random delay between each scrape to avoid your IP getting blocked by Google. This will be sufficient unless you are looking for thousands of results, in which case you’ll need to use proxies.
The 100 results for my query took just a few minutes, with a spreadsheet output that looks like this:
We can then take each of these URLs and run them back through URL Profiler and perform the same analysis above.
Now, to combine the ranking results with the URL Profiler results you might be thinking it requires a bit of Excel wizardry.
Think again…
Import and Merge
Using our new import CSV and merge feature, you can push data back into URL Profiler and make it even easier to build your profile. You’ll need to save your scrape result as a CSV file, then in URL Profiler simply right click on the white URL box and select ‘Import from CSV and Merge data.’
Run your profile as before, and the spreadsheet you get back will show the rankings data on the left, with all the URL Profiler metrics on the right.
The resulting spreadsheet will look like this (I’ve sanitised the data a little to make this image clearer):
Scrolling right will give you all the rest of the URL Profiler data, which can then be nicely formatted and analysed as above (you may wish to split out each keyword before you do this).
Always Be Analysing
If you want to understand what’s working in your niche, you’ll need to be watching your competitors like a hawk. Performing this kind of competitor profile is only the beginning, you’ll need to revisit this often to assess what’s changed and why – I would typically do this every quarter.
As a final note, it is of course worth remembering that hundreds of factors go into Google’s ranking algorithm. Link analysis alone won’t tell the full story, but at the moment links are still such an important signal that this type of analysis can give you a great foundation.