ivucica blog

Finally, ZATEMAS has been reindexed by Google. They took some time.

We had a horrible bug in ZATEMAS which was supposed to return 403 to search engines which accessed the “Profile” page. However it returned 403 for any request to the site. This means that we were removed from the search index (and rightfully so).

The bug is now fixed for more than 5 days, and Google is happily reindexing our site… except for the homepage! Go, try searching for ZATEMAS: since that’s the name of our service, we rightfully were ranked 1 as the search result. Not so anymore, the homepage is just ignored.

I have no idea how to remedy this. I’ve done some modifications to attempt to hint Google that, really, there’s no need to index 9000+ log pages for evaluator (which, frankly, I have no idea why they are public … we’ll look into that). But still it’s examining all the unnecessary pages.

I’m not keen on submitting for reconsideration because we’re actually serving a bit different pages to Google — unregistered users don’t get access to text for problems on our evaluator, and we’ve recognized several bots as “registered” users. Google may claim it to be cloaking. This was supposed to be sort-of a temporary measure until we allow non-registered users to view task text, but still…

Anyways, summary, Google and other bots are losing time crawling unnecessary pages,  while the one most important (homepage) is not indexed.

Seen this in your OpenID consumer implementation when logging on to Yahoo?

Warning: This website has not confirmed its identity with Yahoo! and might be fraudulent. Do not share any personal information with this website unless you are certain it is legitimate.

Over there at http://zatemas.zrs.hr/ we’ve had this for quite some time now. How we eradicated it? Here’s a few pointers. I’m not sure what exactly helped.

• More a generic tip than anything else: even Yahoo! crew recommends reading this blog post.
• We had a redirect from our root page / to /run.php?app=news. We eliminated it through some trickery (setting $_GET and including run.php … sick but works)
• Yahoo! discovers your XRDS document primarily through headers, just as aforementioned blog post says.
• So on your realm page send header X-XRDS-Location pointing to your XRDS document.
• And also include meta tags. ZATEMAS even added special kind of plugins so it can more easily send the special meta tag.
• By the way, in janrain’s php-openid library’s openid consumer samples, the ‘realm’ is generated by getTrustRoot()
• XRDS document should contain a pointer to your return-to page. (That’s the whole point.)
• To stay on the safe side, I also added the aforementioned X-XRDS-Location header and meta tag into our openid_finish.php page.
• Your XRDS document should return content-type application/xrds+xml
• Apache looks braindead here. While it’ll happily serve our xrds.xml.php under xrds.xml, it’ll also happily refuse to serve it in case it gets “Accept: application/xrds+xml”, like Yahoo! does, resulting in 406
• Yes, even with header(‘Content-type: application/xrds+xml’);
• Point is: don’t avoid .php extension. All works if request goes to /xrds.xml.php instead of /xrds.xml — and you won’t see the problem or difference with regular browser. Only hint is when you open access logs and you see non-browser requests for your file ending up in 406.
• Your realm page must not redirect.
• Your realm page, your trust root. In ZATEMAS we use http://zatemas.zrs.hr/ – this means both production and SVN version (in subfolder /SVN) can consume same OpenIDs. Otherwise our users would get different OpenIDs from at least Google.
• I’m not sure if it makes a difference to Yahoo!, but remember that forgetting slash in directory URLs means Apache performs a redirect. 
• Again, don’t mess up: if you change filename, make sure you change both X-XRDS-Location header and meta tag.
• Again, don’t forget: Apache won’t even open your PHP-generated XRDS document if only application/xrds+xml is accepted by the client, and you try to feed it by stripping PHP extension. Just keep the darned extension and serve it to the OpenID provider.

Hope my experiences help someone because this evening we finally got rid of Yahoo!’s warning.

One last note: if you’re a foreign visitor to ZATEMAS and you can’t speak Croatian, you can pass GET argument lang=en to make use of our automated Google-Translate-leeching string translation service. This only works in development version (subdirectory /SVN) because translation module is disabled on production site — it eats away too much page generation time

EDITED: This is value of Yahoo’s XRDS verifier Accept header: application/xrds+xml,text/html,text/plain