After going through the initial pain of setting up a website, I thought it would be useful to compile a short checklist of all the steps I had to do to ensure that it shows up correctly in Google search.
Here are some of the gotchas that I have encountered:
1. If you are using nginx:
You HAVE to setup a default server that returns either a 410 or a 444 code, otherwise your website will be placed under some previously assigned domain name. Here is an example:
server {
listen *:80 default_server;
listen [::]:80 default_server ipv6only=on;
listen *:443 default_server;
listen [::]:443 default_server ipv6only=on;
server_name _;
return 410;
}
2. Setup HTTPS certificate quickly. You can just use this website: https://www.sslforfree.com/
3. Block every other port on your server that you are not going to use.
Ideally, you would only expose port 80 and 443 to the outside world and port 22 internally, limited only to your ssh key.
4. Setup robots.txt and sitemap.xml - also send a link to your sitemap.xml to Google!
Here you can find more about sitemap.xml: https://support.google.com/webmasters/answer/183668
And here is an example of a robots.txt file:
user-agent: *
disallow: /
allow: /index.html
...
sitemap: https://yourwebsite.com/sitemap.xml
5. Setup google webmaster account to confirm that you own your website!
6. Remove previous websites from google search: https://www.google.com/webmasters/tools/removals https://www.quora.com/How-do-I-permanently-remove-a-domain-from-Google-Search-results