Ten. Gzip Compression: Allow Gzip compression with your server to reduce the sizing of files transferred between the server along with the person’s browser. txt file is then parsed and can instruct the robotic as to which internet pages aren't for being crawled. For a search engine crawler may maintain https://www.youtube.com/watch?v=Tmal7B28Td0