Fetch as google problem?

[Lapan]

Hi.
I'm new here, found forum on Google, I need some help with Google Webmasters Tools.

My site is lapan.com.ba
Fetch as google keeps failling, I contact my host, they said everything is fine with site.
Even on subdomain forum.lapan.com.ba, fetch as google keeps refreshing and will fail later.
Tried adding new site with www, same, failed.
1 from 1000 added urls will succeed.

How do I check why this is happening, so I can fix it? :think:

This is my robots.txt now:

User-agent: *
Disallow: /external
Disallow: /external-embed
Disallow: /link
Disallow: /link_embed
Disallow: /oznaka
Allow: /oznaka/besplatni-filmovi
Allow: /oznaka/besplatno-gledanje
Allow: /oznaka/bosanski-prijevod
Allow: /oznaka/film online s prevodom
Allow: /oznaka/film-online-sa-prevodom
Allow: /oznaka/filmovi-onlajn
Allow: /oznaka/filmovi-online
Allow: /oznaka/filmovi-s-prevodom
Allow: /oznaka/filmovi-s-prevodom-na-bosanski-jezik
Allow: /oznaka/filmovi-s-prevodom-na-hrvatski-jezik
Allow: /oznaka/filmovi-s-prevodom-na-srpski-jezik
Allow: /oznaka/filmovi-sa-prevodom
Allow: /oznaka/hrvatski-prijevod
Allow: /oznaka/najbolji-filmovi
Allow: /oznaka/onlajn-filmovi
Allow: /oznaka/online-filmovi
Allow: /oznaka/online-filmovi-s-prevodom
Allow: /oznaka/online-filmovi-s-prevodom-na-bosanski-jezik
Allow: /oznaka/online-filmovi-s-prevodom-na-hrvatski-jezik
Allow: /oznaka/online-filmovi-s-prevodom-na-srpski-jezik
Allow: /oznaka/online-filmovi-sa-prevodom
Allow: /oznaka/prevedeni-filmovi
Allow: /oznaka/srpski-prevod
Allow: /oznaka/besplatne-serije
Allow: /oznaka/besplatno-gledanje
Allow: /oznaka/bosanski-prijevod
Allow: /oznaka/serije-online-s-prevodom
Allow: /oznaka/serije-online-sa-prevodom
Allow: /oznaka/serije-onlajn
Allow: /oznaka/serije-online
Allow: /oznaka/serije-s-prevodom
Allow: /oznaka/serije-s-prevodom-na-bosanski-jezik
Allow: /oznaka/serije-s-prevodom-na-hrvatski-jezik
Allow: /oznaka/serije-s-prevodom-na-srpski-jezik
Allow: /oznaka/serije sa prevodom
Allow: /oznaka/hrvatski-prijevod
Allow: /oznaka/najbolje-serije
Allow: /oznaka/onlajn-serije
Allow: /oznaka/online-serije
Allow: /oznaka/online-serije-s-prevodom
Allow: /oznaka/online-serije-s-prevodom-na-bosanski-jezik
Allow: /oznaka/online-serije-s-prevodom-na-hrvatski-jezik
Allow: /oznaka/online-serije-s-prevodom-na-srpski-jezik
Allow: /oznaka/online-serije-sa-prevodom
Allow: /oznaka/prevedene-serije
Allow: /oznaka/srpski-prevod
Disallow: /kategorija
Disallow: /video
Disallow: /author
Disallow: /search
Disallow: /epizoda
Disallow: /?s=
Disallow: /feed/
Disallow: /xmlrpc.php
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /2013
Disallow: /2014/01
Disallow: /2014/02

And .htaccess is

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

 

[Tommy]

Hello Lapan,

I got the same problem when I used robots.txt to block spiders crawl and indexing my site in the past. Your problem may be from robots.txt
As I saw code above then your .htaccess file is ok, base code generated from Wordpress cms and this is not reason cause fetch failed.

So I suggest you check your robots.txt file first, before checking your web system later.
Try to remove all your codes in robots.txt file, only put

User-agent: *
Allow: /

And go to Google fetch tool and try again, If you still got problem, let me know

If Google show your old robots.txt codes, you need to wait a little time for it or retry the process until Google show right code and status that you putted on robots.txt

Hope it helps!

 

[Lapan]

Thanks, Tommy.

I updated my robots.txt, now have to wait and see what will happen after GoogleBot re-download robots.txt file.

 

[vaneetagoswami]

Lapan in robort.txt try to insert only those thing that you want to not allow for search engine. and rest of one are automatically crawl by Google.

 

[Lapan]

Hi, vaneetagoswami.

Right now, I have problem with my hosting.
On some website I found that deleting and again registrating domain can resolve DNS glitch/error which cause GoogleBot to timeout, and when I wanted to re-add domain, I was not able to.

Now I encountered one more problem, I contacted host, waiting for reply.
I will see if that was what caused google to timeout with my site and I will update this topic.

 

[vinpay]

If you having problem with your hosting or domain provider then it could caused by DSN error or wrong configuration on your domain. Try to check it before using tool fetch as Google for your website.

 

[Rukowebsite]

I never use that tools at google webmastertool, i prefer onpage optimization