When I first moved over to Hugo, I struggled to get the robots meta tag working. Note that I am using the Ananke theme and this may be different for other themes. Primer Honestly, I have not spent a lot of time in my career on SEO and did not have a deep understanding of how the robots meta-tag and robots.txt file work. Here is a quick primer. First, a page can include a meta-tag in the header that specifies that a page should be indexed by search engines or not.