While trying to set up a new landing page for our maintenance services, I ran into a problem where the page would load just fine for me and on 3rd party tools like BrowserStack, but it was returning a 500 server error for Googlebot. It would also return a 500 server error for W3C Validation and Spider View checks. I would have never even noticed it had Google Adwords not disapproved the page URL repeatedly.
This was completely perplexing to me and took several hours to troubleshoot. In researching the issue, the only possible leads I found were some older forum posts mentioning both the core Joomla { loadposition } plugin for modules and the NoNumbers Modules Anywhere plugin. I was not using the Joomla loadposition and disabling Modules Anywhere did not change the error code.
I contacted our hosting company HostGator who found nothing from their end. In systemically disabling items, I found the culprit to be RSForms Pro by RSJoomla. It did not matter whether our signup form was loaded as a menu item, module, or in the content plugin (like { rsform 4 }), it would give the 500 server error for Googlebot. I have no idea why this is and would love feedback in the comments.
To solve this, I was able to disable RSForms, and load our ZenDesk form in a RokBox instead, but I think we could have better conversions if the user could see how short the form is. Again, I would love any alternative solutions that anyone can think of.
EDIT 2/19/2014: The problem within RSForms was a bad script in the PHP script section. I had pasted in sample code to use the ZenDesk API and then forgotten about it.
Here are some of the lessons I learned:
How to check whether Googlebot can access your site:
1. Login to Google Webmasters for that particular site.
2. From the left menu, click on CRAWL, then FETCH AS GOOGLEBOT.
3. Add the URL and hit FETCH.
4. Google will either tell you “Success” or “Unreachable.” If unreachable, it will show you the error messages.
Make sure URL is not Blocked by Robots:
1. Login to Google Webmasters for that particular site.
2. From the left menu, clickon CRAWL, then BLOCKED URLs.
3. Google will show you the currently loaded robots.txt file. Below it, is a part for “URLs: Specify the URLs and user-agents to test against.” Add the URL you want to test to that textarea, then hit TEST.
4. It will show the “Test Results” below.
How to View HTTP Live Headers for a website:
1. There is a Google Chrome extension that makes this easy. Go to https://chrome.google.com/webstore/detail/live-http-headers/iaiioopjkcekapmldfgbebdclcnpgnlo to download and install.
2. Once installed, click on the extension icon (little blue cloud), open a new tab, and then put in the URL you want to test.
