April 14, 2006
Google Sitemaps provides better, fresher crawls of Web pages
Free beta service helps you find and fix problems that hurt your search rankings
Recommend ThisWeb analytics products such as ClickTracks and HitsLink indicate how sites fare in search rankings, but they do little to help fix poor-performing Web properties. Google Sitemaps fills that void.
Currently in beta, Google Sitemaps allows you submit URLs to the Google index and inform Google when these pages change. Certainly, having Google index deep pages benefits many organizations; e-tailers, for example, could increase revenue because potential buyers find items that might ordinarily fall into an indexing crevasse.
Google Sitemaps is complemented by Sitemap Generator, a Python script for creating sitemap files. These contain the actual list of URLs in XML. Google also accepts sitemap files in OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) or text.
Sitemap Generator performed well in my tests. It works by scanning URL lists, access logs, or directory paths. Moreover, pages you submit can be for traditional Web sites plus those that serve content for mobile devices. Flexibility aside, Generator runs on the Web server and requires that you control it though a CLI -- a scenario some IT departments will not allow. However, Google lists various utilities and code snippets (many free) that could be used for building sitemaps remotely.
Sitemaps provides very helpful reports. I displayed the top queries on Google that returned pages from my site. Additionally, I saw how many of those results directed traffic to my site. Another report lists common words in your site content.
Equally helpful are troubleshooting reports, including the percentage of pages that were crawled successfully and pages that the Googlebot could not access. Turning to the robots .txt analysis, I quickly discovered that a certain section of my site was inadvertently blocked. Even better: You can simulate changes to the robots.txt file, see how Google crawlers will react, and make sure there are no errors before changing the robots.txt file on your site.
Similarly, Sitemaps’ error report lists pages and the specific problem Googlebot encountered. Fixing these issues -- such as broken redirects -- can quickly increase your crawl coverage.
Google Sitemaps Beta is available for free.
Mike Heck is a contributing editor of the InfoWorld Test Center.
White Paper
D2D Virtual Tape Library Replication Primer
This whitepaper explains the terminology and concepts behind Data Replication technologies and establishes some sizing rules through worked examples. Learn the new paradigm in disaster tolerance—protect data anywhere.
Download now »
White Paper
An Alternative to Virtualization for Datacenter Cost Savings
Server virtualization is a popular option for dealing with mounting datacenter costs. Another equally promising approach is the use of an Application Delivery Controller. Citrix NetScaler provides a low-cost way for organizations to reduce their server count and accrue cost savings from a reduction in space, cooling, power and personnel.
Download now »
White Paper
Why Your Firewall, VPN, and IEEE 802.11i Aren't Enough to Protect Your Network
The emergence of WLANs has created a new breed of security threats to enterprise networks.
Included in HP ProCurve WLAN solutions is security technology that alleviates threats from WLANs through:
* Monitoring wireless activity inside and out of the enterprise
* Classifying WLAN transmissions into harmful and harmless
* Preventing transmissions that pose a security threat to the enterprise network
* Locating participating devices for physical remediation
White Paper
Bringing the Edge to the Data Center
Effectively address data protection challenges, implementing solutions that help store and protect business–critical data while cutting costs and improving efficiency and reliability.
Download now »
- Tackling the Top Five Network Access Control Challenges | White Paper
- Opportunities and Challenges with the Convergence of Data Center Networks | White Paper
- Network Operating System Evolution | White Paper
- How Operating Systems Create Network Efficiency | White Paper
- Data Center Survival Guide | White Paper
Sign up to receive Data Management Resource Alerts
Maximize your IT Budget with the Fujitsu Dynamic CubeToday’s IT means business. New eBook shows you how to align your IT investments.A new level of interoperability. Make IT Work As One at novell.comLimited-time offer: Save up to $100 on Polycom phones. 64-page prescriptive guide to security, compliance, and IT operations.Drive innovation with Intel and Open Source softwareLinux Contributions and Support from IntelLearn about Foundations of Green IT, from CTO Marty PoniatowskiTDWI checklist helps define data readiness for analytics. Download report.<802.11n Drives an Architectural Evolution
See how AT&T can help protect your network. Make IT Work As One at novell.comHP ProLiant G6 Servers. Perform like a superstar, Save like an accountant.HP ProLiant G6 Servers. Perform like a superstar, Save like an accountant.Unified Communications: Thoughts, Strategies and Predictions. Join the discussionOptimize Linux* for high performance and energy efficiency on Intel® architectureSee how Intel is working across the Open Source ecosystem to help drive rapid technological innovationKeep your IT expertise up to date. Join the Intel Premier IT Professionals. Increase UPS efficiency without sacrificing protectionIn less than 45 minutes, Palisade Systems protects your data
