Networking Best Practices for Large Deployments
Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 www.google.com Part number: NETBP_GAPPS_3.5 October 15, 2013 © Copyright 2012 Google, Inc. All rights reserved. Google, the Google logo, Google Apps, Google Apps Mail, Google Docs, Google Calendar, Google Sites, Google Video, Google Talk, Gmail, Google Message Filtering, Google Message Security, Google Message Discovery, Postini, the Postini logo are trademarks, registered trademarks, or service marks of Google Inc.
Contents Chapter 3: Introduction....................................................................................... 5 About This Guide................................................................................................... 5 Target Audience.............................................................................................. 5 Benefits ........................................................................................................... 5 Level of Effort ........................
Network Addressing and Protocols ..................................................................... 19 Google IPv4 Addresses ................................................................................ 19 Google Host Names...................................................................................... 20 Google Global Cache.................................................................................... 20 Google Protocols......................................................................
Chapter 3 Introduction Chapter 3 About This Guide This document discusses best practices for optimizing your large-scale IP network for Google Apps for Business, Apps for Government, and Apps for Education. The recommendations and information in this guide have been gathered through our work with a variety of customers and partners in many network environments. We thank our customers and partners for sharing their insight and experience.
Level of Effort The level of effort needed to implement the recommendations in this guide will depend on your requirements, your current infrastructure, and the skills of your network team. The design principles and implementation best practices in this document are not industry- or technologyspecific. The principles in this document do not require specific technical expertise outside of an industry-standard network architecture and network engineering skill set.
Related Documentation For additional information and background about network best practices and related topics, refer to the following related resources: • Google Apps Technical Transition Guide: Overview of how to transition your organization to Google Apps from another messaging platform. • Google Apps Deployment Resources: A resource center for IT administrators to collect information on deploying Google Apps. Designed for large organizations. • Google Apps Partner Connect: Requires login.
Networking Best Practices for Large Deployments
Chapter 4 Network Action Checklist Chapter 4 About This Checklist This section contains a summary checklist of all action items in this guide. If you don't have the time to review this guide end-to-end, we suggest you start by reviewing this Network Action Checklist. Each topic is described in detail later in this guide. Network Evaluation Evaluate your current network and plan for capacity needs.
Network Configuration The following recommendations describe network configurations that will help provide your users with the best experience with Google Apps. These recommendations increase network availability and performance, and can reduce costs by simplifying the network equipment required to reach Google Apps.
If it’s not feasible to use a DNS resolver that’s close to the user, use a DNS server that supports the edns-client-subnet extension (Draft Proposal 2671)—such as Google’s DNS server or OpenDNS—which allows the resolver to pass part of the client’s IP address. Adhere to the advertised TTL value for all DNS record types. Set up firewall rules to allow unrestricted outbound HTTPS traffic to Google Apps.
Migration Google Apps deployments can involve migration traffic, either from local clients like Google Apps Migration for Microsoft Outlook, or from server-side clients like Google Apps Migration for Lotus Notes and Google Apps Migration for Microsoft Exchange. Migration of legacy data into Google Apps is typically a resource-intensive activity, but usually only happens once.
Chapter 5 Network Evaluation Chapter 5 Summary When planning for the implementation, you will achieve better results if you first understand your current network capacity and the expected amount of network load from Google Apps. The best way to predict how much load to expect is to benchmark your bandwidth usage in your network, and create a test environment to simulate how much capacity each user will require. This section discusses approaches to test your network environment in detail.
Below are the recommended steps to assess and test your network prior to deployment. • Conduct an inventory of all of your network locations, including location name, Internet access type (e.g., T1, VPN, DSL), and available Internet bandwidth. • Test DNS resolution from all network locations to Google Apps, to ensure that clients in your network can resolve Google Apps hostnames.
1. Open the sample list text file hosted on the apps-deployments code site. (Note that this is a sample only, not a complete list.) 2. Save the sample .txt file in the directory where you will be using the test commands: a. Click View raw file. b. Right-click the page, and choose Save As. 3. Run the following command to test DNS resolution: % dig +all +trace -f GoogleAppsDomains.txt ICMP Connectivity Test Ensure that clients in your network can reach the hostname mail.google.
% iperf -s Note: WAN Bandwidth tests are intrusive and can affect network performance. Run these tests during off-hours to gather data while causing the minimum possible the impact to your network. If you need to run these tests during business hours, be careful of possible effects this test may have on your network performance. Network Testing Tools You can obtain the tools discussed above from the following online sources: • Download the Hping packet analyzer tool from hping.org.
Collect the following data for each environment, while using Google Apps services available in your domain. For instance, open Gmail, Google Talk, Google Docs, and Google Calendar. • Average connections/sec • Peak connections/sec • Non-peak connections/sec Additionally, Google Apps, like many web-based applications that run in the cloud, keeps several connections open to the remote server to poll for new data.
• Chrome on Windows 7 Connections when entering URI: 1 Connections during initial load: 3 Connections during login: 6 Connections after a few minutes idle: 4 Connections when opening Calendar and Docs: 4 Connections when loading a document: 6 Average load: 3.6 connections Peak load: 6 connections Idle load: 3.
Chapter 6 Network Configuration Chapter 6 Summary This section includes details on how to optimize your network for Google Apps. This includes information on Google’s IPv4 addresses, protocols used, routing suggestions, proxy server configuration options, and DNS configuration. Use this information as a guide when configuring your network, and as a reference for what types of requests Google Apps clients will make to Google servers.
The second column in the result set is the TTL for the records in seconds. Based on these sample results, we can determine that the IPv4 addresses are valid for only about 2.5 minutes. Google IPv4 addresses for specific hostnames are not static. For example, do not assume mail.google.com will always be 74.125.225.245. If you need to configure your environment to accept mail from Google for a mail gateway, include all of the subnets from the _spf.google.com record per this Administrator Help Center article.
Google Protocols Google Apps services are mostly web-based or API-based. The table below shows common Google Apps services, and the protocol used for each. As shown in the table, the datagrams that Google Apps uses are almost always TCP-based, except as noted.
Google+ Hangouts Google+ Hangouts attempts to establish a connection between a participant and a Google server using a method similar to that of Google Talk Voice and Video. For details, see the Administrator Help Center. Firewall Configurations To provide users with the full capabilities (voice, video, and text) of Google Talk Voice and Video and Google+ Hangouts, allow UDP out from clients on your network. See the Help Center article “Optimize your Network for Hangouts” for more detail.
Traffic Prioritization You may be able to improve Google Apps performance with traffic prioritization, by giving Google Apps traffic priority over other network traffic to reduce network latency during congestion. Traffic prioritization is possible on the data link layer and the network layer; see the sections below for more information. You may wish to consider traffic prioritization to reduce potential latency if you have any of the following environments: • Hub and spoke network topologies.
Network Routing Tools • A variety of useful tools are available to generate detailed data regarding your Internet connection performance on the external website Measurement Lab. You can use these tools to measure your overall Internet access performance. • The site PeeringDB is a large worldwide public database with information about peering networks.
Content Inspection Avoid content inspection on your proxy server. When Google Apps is configured to run over HTTPS, which is common and recommended, proxy servers cannot inspect content or restrict access without a special proxy configuration. Filtering Google Apps traffic through a Proxy The vast majority of traffic originating from your users to Google Apps servers consists of HTTPS transactions. This type of traffic is preferred because it is secure and reliable.
return "DIRECT"; // All other URI requests should be routed through the proxy. else return "PROXY corporateproxy.domain.com:8080"; More examples for developing a proxy PAC file can be found on the external website FindProxyForURL. Proxy PAC file testing Implementing a functional proxy PAC file requires careful testing. Use a PAC file testing tool like pactester to test different JavaScript functions.
A common means of blocking access to web services is using a web proxy server to filter traffic directed at particular URIs or hostnames. This approach is ineffective in this case because all the URIs accessed between consumer and Google Apps accounts are the same. To only allow users to access Google services using specific Google accounts from your domain, you need the web proxy server to add an HTTP header to all traffic directed to *google.
Proxy Configuration Tools Download the following tools which may be helpful when configuring Proxy Servers: • Use pactester or a similar tool to validate PAC files for different URIs. Download pactester from the Google Code site. • Download HttpWatch or HttpFox (Firefox extension) to help you see what URIs are being requested by the browser prior to encryption. Other Network Services Google runs a sophisticated load-balancing system to ensure the best experience for the user.
DNS Resolution The diagram below shows a typical DNS resolution for a Google Apps user on an enterprise network. Google serves DNS A record queries dynamically to ensure users receive the best experience at the time they make their request. To ensure that this occurs properly, configure your DNS caching resolvers to adhere to the TTL values specified with each record.
Using a centralized DNS server architecture will obscure the user making the request from Google’s DNS servers, preventing Google from responding with an appropriate IPv4 address. If DNS queries are routed through a central server to resolve Internet hosts, users may not connect to the closest Google Apps servers. In extreme cases, this architecture can cause users in one continent to connect to servers in another, distant continent. The ideal solution is to place local DNS resolvers close to the users.
Mail Routing After you change your MX records to route mail traffic to Google Apps, your email is no longer delivered to your servers by SMTP. Instead, inbound email is directed to the Google Apps servers. This essentially eliminates inbound SMTP mail traffic on your network. Outbound Mail Connections Depending on your needs, you may have some outbound mail traffic that you wish to send from your own network, such as automated or batched communications from applications in your system.
Networking Best Practices for Large Deployments
Chapter 7 Client Configuration Chapter 7 Summary It is important to understand the effects that different clients can have on the performance of Google Apps. This section discusses elements of the user environment that can affect Google Apps performance, suggestions for setting up authentication for use with Google Apps, and advice for migrating your users’ data from an existing server into Google Apps.
For Google Apps, we support the latest version of Google Chrome (which automatically updates whenever it detects that a new version of the browser is available). We also support the current and some previous major releases of Mozilla Firefox, Microsoft Internet Explorer, and Apple Safari. Check the Administrator Help Center for more information about supported versions of browsers. Offline Access Offline access can dramatically affect overall network bandwidth.
Google Apps Connector for BlackBerry Enterprise Server Unlike Android and ActiveSync devices, BlackBerry Enterprise users do consume network resources when fetching mail from Google Apps, and when pushing data to the RIM network and the user’s BlackBerry device. Google Apps Connector for BlackBerry Enterprise Services acts as a bridge between Google Apps and the RIM network.
The largest amount of traffic between Google Apps and the Google Apps Connector for BlackBerry Enterprise Server occurs when a user is first added to the BlackBerry system via the BlackBerry admin panel. When a user is added to the system, the connector software will create a local cache of the user's email, calendar, and contacts. This local cache can be several hundred megabytes in size since it contains all user's data for the recent past (30 days by default).
If you plan to set up Single Sign-On authentication, consider the following suggestions: • Set up SSO servers in distributed network locations, rather than a central location. • Set up internal DNS servers to redirect SSO traffic to the nearest SSO server, and ensure that alternate SSO servers are in place for redundant service in case of disruptions that prevent users from accessing the SSO server in a particular location.
Single Sign-On for Selected Network Locations It is possible to create a conditional SSO system for your users that is based on a network subnet mask. This can be configured in the Google Apps control panel, under Advanced tools -> Set up single sign-on (SSO). This type of configuration is recommended as it can be configured to control whether users outside your network use your SSO system. A recommended setup is to configure Google Apps so that only users inside of your network require SSO authentication.
Server-side Migration Your migration servers should have a low-latency, high-bandwidth connection to your email server. Migration is traffic- and bandwidth-intensive, and you can expect significant network load between your email server and the migration server. Note: Do not install the migration software on the actual machines handling mail for your domain, since this will consume significant system resources.
Networking Best Practices for Large Deployments
Chapter 8 Network Monitoring Chapter 8 Summary After your network is set up to work with Google Apps and your users are enabled, you can maintain the quality of your users’ experience by monitoring the health of your network. To ensure the best user experience, follow these suggestions for monitoring tools and network traces. Monitoring Tools There are many commercial and open source tools to monitor various aspects of your network.
Type of Monitoring Tool Description Network multiping Helps monitor network latency, uptime, and route changes. Packet Capture Wireshark Performs packet captures. RTT latency wbox Attempts to measure RTT of web application latency using HTTP/TCP latency. Trace tcptrace Similar to traceroute but uses TCP packets rather than ICMP packets.
