How to Setup Site Audit Crawler in SEMrush

You have created your SEMrush account and now you want to use it properly for the SEO of your website. Here are the steps to do it.

Getting Started with the Tool: Site Audit Crawler Setup at the very beginning

Step 1:- Site Audit Crawler Setup

Open your account and go to the dashboard. There on the left-hand side, you will find Site Audit option. There you have to set up the following parameters.

Crawled Pages Limit:- 

• This is the first screen you will see. You may be asking, why do I want to limit the number of pages?
• Well, if you have an enterprise site, it's actually a better idea to break the site into sections and crawl a lower number of pages.
• We do this because it still brings up all the template level issues, without needing to wait all day to crawl a million pages.
• In SEO, it's all about cost/benefit analysis.
• You can specify if you want to crawl just a subdomain (like blog.domain.com or a subfolder).
• You can then choose what you want the bot to crawl from the site to the sitemaps.

Crawler Settings:-

• Next, we get a bit more technical. Most sites are fine crawling as SEMrush bot, however, if you are struggling to crawl due to CMS or technology restrictions, Googlebot is a good option.
• If you are on a hosted solution like Shopify or Squarespace, or your SysAdmin has really sensitive controls on your server, choose 1 URL per 2 seconds.

Allow and Disallow Certain URLs:-

• This bit is where we can really define our crawl.
• If there is a particular problem area on the site you want to investigate, you can define the exact paths you want to include and exclude.

URL Parameters:-

• If the site has lots of parameters caused by a search box or pagination, we want to exclude them from the crawl.
• The best way to check for them is to click this link to the search console to retrieve the parameters Google has already recognized.

Website Restrictions:-

• Next up is website restrictions. If you are crawling a staging server as you are migrating a site, typically, it's behind a login. You can add these details by clicking “crawling with your credentials”. If your robots file blocks 3rd party crawlers you can choose to ignore it.

Crawling Schedule:-

• Next up is the schedule. If you are going to be fixing things every week, I'd run it on a regular basis so you can see improvements over time.
• As a safety measure, I like to crawl 3 times a week - so if a rankings fluctuation happens I can easily rule out technical SEO as the cause.

Technical SEO Audit Strategy:-

• When you think about your website, think of it as your own musculoskeletal body. All of the parts are connected to each other and affect each other in some way.
• So just like when the doctor hits your knee with a hammer and your foot shoots out, similarly, when we change one aspect of the site, many pages will be affected.

Therefore, one of the most important factors in technical SEO is setting priorities. To do this we use the Eisenhower Matrix, which defines:

• Important and Urgent
• Things that need to be fixed right away
• Important and Not urgent
• Things we need to make a business case for and fix over time
• Not Important but Urgent
• Something to be briefed to a secondary team
• Not Important and Not Urgent

Also known as a “nice to have” - something that you will write down on a piece of paper and probably never do.

Within this decision-making matrix, we are going to approach key aspects on technical SEO theme by theme.

These include:

1. Hygiene. Basic technical checks that your site needs to fundamentally work with search engines.
2. Organization. How to best structure the content of your site for maximum visibility and user experience.
3. Presentation. Elements of your site that affect CTR and the way your search results look.
4. Page Power. How your pages interact and pass page rank to one another.
5. Code and Security. Organizing the technology on your site for a safe user experience.
6. Internationalization. How your site is perceived in different countries.
7. Performance. How your site works across different devices and internet connections.
8. Reporting. How to report on each of the issues and build a project management workflow.

How to Approach Issues, Warnings, and Notices

The first screen you see in the Site Audit tool is the overview

This presents all the top-level issues, warnings, and notices as well as thematic reports about the specific issues around crawlability, security, and internal linking.

The next tab is the Issues tab

This is a full list of every issue that we've been able to find. You can click into each of them to get a more detailed report.

The next tab is the crawled pages 

You can view this as a list or a tree structure.

The tree structure is particularly useful to get an overview of the content silos on your site and to see how you are organizing your information for the crawlers.

The next tab is the statistics

If you are an agency pitching for a new business or an SEO working in-house that needs stakeholder buy-in, the graph view is particularly useful for presentations.

The next tab is the compare crawls tab

This is useful for looking at progress over time and can be used to diagnose ranking fluctuations.

The progress tab is great for reporting

Now, when you first run your crawl and look at the dashboard and see all those Issues, Warnings and Notices, you are going to feel a couple of things.

Of these thousands of problems, lots of them can be fixed at a template level. This means that you are going to be able to make one fix and knock out several hundred issues at a time.

Errors

Errors are the highest priority fixes that we need to make on the site.

These are typically issues that are going to really hold your site back from making progress because Google is unable to access or understand what's happening on your site. It really should be fixed as quickly as possible, and when you fix it, make sure you use a permanent solution.

Warnings

Warnings are slightly less critical than errors.

Although they will have a direct impact on your performance, they won't take you out of the race altogether.

Imagine you are running a race at the Olympics but you need to carry a lot of weight with you.

You are still going to finish the race, but it's going to be way harder to win with the issues.

Notices

Notices are more like points of interest that you need to look into and diagnose.

A notice is a kind of like getting a pimple - it's probably nothing, but best to get it checked out just in case there are some underlying issues.

That's everything for lesson 1 on how to use SEMrush site audit for your technical SEO, we hope you've found it valuable.