Addendum to the White paper on Windows Time Sync Accuracy Contents Summary ....................................................................................................................................................... 2 Measuring Time Sync Accuracy..................................................................................................................... 3 Measurement Tools .................................................................................................................................. 3 Popcorn Filter ............................................................................................................................................ 5 Notes Server vs Client Terminology and Version information.................................................................. 6 Server 2016 Forest Test Topology ............................................................................................................ 6 Server2012R2 Forest Test Topology ......................................................................................................... 7 Methodology for AD Time Sync Accuracy Measurements ....................................................................... 8 Methodology for Simple NTP accuracy measurements ........................................................................... 9 Comparison between Server 2012R2 and Server 2016 ................................................................................ 9 Forest with a VMIC Root Time Source .................................................................................................... 10 Forest with a NTP Root Time Source ...................................................................................................... 14 Comparison of NTP in Server 2016 ............................................................................................................. 20 Server 2016 Forest using VMIC root time source (Scenario #1) ................................................................. 21 Root PDC (Server 2016)........................................................................................................................... 21 Root Domain Replica DC (Server 2016) .................................................................................................. 26 Root Domain Client #1 (Server 2016) ..................................................................................................... 31 Root Domain Client #2 (Server 2012R2) ................................................................................................. 36 Child Domain PDC (Server 2016)............................................................................................................. 38 Child Domain Client #1 (Server 2016) ..................................................................................................... 42 Server 2016 Forest using NTP Root time source (Scenario #2) .................................................................. 46 Root PDC (Server 2016)........................................................................................................................... 47 Root Domain Replica DC (Server 2016) .................................................................................................. 52 Root Domain Client #1 (Server 2016) ..................................................................................................... 56 Root Domain Client #2 (Server 2012R2) ................................................................................................. 60 Child Domain PDC (Server 2016)............................................................................................................. 62 Child Domain Client #1 (Server 2016) ..................................................................................................... 66 Server 2012 R2 Forest using VMIC Root Time Source (Scenario #3) .......................................................... 71 1 Root PDC (Server2012 R2) ...................................................................................................................... 71 Root Domain Replica DC (Server 2012 R2) ............................................................................................. 72 Root Domain Client #1 (Server 2016) ..................................................................................................... 73 Child Domain PDC (Server2012 R2) ........................................................................................................ 75 Child Domain Client #1 (Server2012 R2) ................................................................................................. 76 Child Domain Client #2 (Server 2016) ..................................................................................................... 78 Server 2012 R2 Forest using NTP Root Time Source (Scenario #4) ............................................................ 79 Root PDC (Server2012 R2) ...................................................................................................................... 80 Root Domain Replica DC (Server 2012 R2) ............................................................................................. 81 Root Domain Client #1 (Server 2016) ..................................................................................................... 83 Child Domain PDC (Server2012 R2) ........................................................................................................ 84 Child Domain Client #1 (Server2012 R2) ................................................................................................. 86 Child Domain Client #2 (Server 2016) ..................................................................................................... 87 Server 2016 NTP “Quarter mile” Test ......................................................................................................... 88 Scripts .......................................................................................................................................................... 91 Script #1 Setting up common high accuracy settings ......................................................................... 91 Script #2 Enable VMIC Provider Only .................................................................................................. 92 Script #3 Enable NTP Client Only with manual server ........................................................................ 92 Script #4 Enable NTP Client Only with Domain Sync .......................................................................... 93 Script #5 Enable NTP Server ................................................................................................................ 94 Index of Figures ........................................................................................................................................... 94 Summary This is an addendum referenced by the Windows 2016 Accurate Time article. The following documentation provides more details about our testing and measurement methodologies. Windows Time Service in Server 2012R2 and prior releases was mainly focused on keeping time in an Active Directory domain to within a 5-minute window to allow Kerberos authentication protocol to function. This was a pretty low bar to meet and once set up, the time synchronization worked smoothly. In case of any issues with time synchronization, customers could use the in-box tool W32tm.exe to run a set of diagnostic commands for troubleshooting. One of the commands is “stripchart”, which sent NTP pings to a specified time server and printed the time offset as its output (The command line format looked like this: w32tm.exe /stripchart /computer:<NTPServer> ….). This seemed to be reasonably accurate in a short time span and served its purpose in prior releases. 2 We have made significant improvements in the time synchronization accuracy in Server 2016 in order to achieve sub-millisecond time sync accuracy to meet the requirements in various new scenarios besides Kerberos. Sub-millisecond time synchronization can only be achieved under certain conditions. Accurately measuring the time synchronization correctly down to sub-millisecond values is not straight forward, due to noise that can creep in to the measurement process. This addendum describes the topologies we set up to measure the accuracy and the different techniques used to measure the very small time offsets. Next it describes a simple post-process filtering of data collected using NTP pings (by W32tm.exe “stripchart” or other means) in Server 2016, which helps simplify the measurement of time sync accuracy in any generic topology. Next several comparisons are made between time sync accuracy in Server2012R2 AD forest and Server2016 AD forest and one comparison that shows degradation of NTP accuracy in Server2016 due to increasing number of network hops. This should help the reader visualize the improvements that have been made in Server 2016 and to some extent, understand the impact the topology has on the accuracy. The rest of the document goes into details of measurements taken by 3 different ways and demonstrates the validity and effectiveness of the simple filtering of NTP ping data in Server2016 topologies. The data from these measurements has been used in the comparisons made in this document. Measuring Time Sync Accuracy Measurement Tools Our teammate Alan Jowett has published a set of tools to measure time sync accuracy on GitHub (https://github.com/Microsoft/Windows-Time-Calibration-Tools). Of these, we will be using the tools for TSC based offset in virtual machines in our measurements (specifically OsTimeSampler, Scripts and TimeSampleCorrelation). These tools are based on the idea that we can compute the time offset between a Hyper-V host and a VM guest over a period by recording the Time Stamp Counter (TSC) and System time in both machines simultaneously, recording the fixed difference between the TSC values in both machines and processing the data offline. The processing involves correlating TSC data points from both machines and interpolating the data within the sampling interval using LaGrange polynomial curve fitting. For this process to measure the time offset of the VM’s system clock reliably and accurately, the Hyper-V host system clock needs to be maintained reliably and accurately. The other means to measure time accuracy is using simple NTP pings. This functionality is conveniently available in the inbox tool W32tm.exe using the switch “stripchart”. Here is are examples of this command: W32tm.exe /stripchart /computer:time.windows.com /period:1 W32tm.exe /stripchart /computer:<YourNTPServer> /period:1 3 Comparing the time offset data from TSC with the time offset data gathered from W32time stripchart (or any other software that can send NTP pings) would help us ascertain the reliability of both methods and show us any limitations therein. The NTP pings could be made in either direction (from accurate time server to the test machine and vice versa) and the results can be compared in a similar manner as well. If the three data sets provide us with the same data, we can be very certain they are pointing at the same common truth. Note: Alan’s GitHub repository includes NTP ping based tools, but those were not used to generate the data presented here. The results from those tools should be very close to the results demonstrated here. The tests we ran are mostly geared to measure the accuracy of secure NTP in an AD Forest, though we have one test to measure the performance of simple/unsecured NTP as well. For our AD topology tests, we have used two Hyper-V hosts. These are server blades with lots of memory and processing power and multiple hard disks each and each is outfitted with Sync-n-Scale high accuracy time hardware/drivers. We built our AD topologies using guest VMs running on these host machines. The variety of topologies is endless. Our topologies were set up to demonstrate 2 strata of time servers in the domain as well as various scenarios for networking stack/OS versions etc., to give a peek into the potential of the time sync accuracy in Server 2016. We have one non-AD simple NTP test which used a generic physical machine connected to our corporate network. This did not involve collecting TSC data. Before collecting data on any computer being tested, we first configure it and let the time synchronization run on it for 2-3 hours to allow the computer’s time to converge with the reference time. Time offset/error measurements are taken after this period. We followed the same process for data collection on all computers in a complex topology, allowing 2-3 hours for the entire topology to converge. We collected the following four data streams simultaneously for measuring time on a VM: 1. TSC related data using OsTimeSampler tool (see github link above). a. TSC and System Time inside the VM, sampled every second b. TSC and System Time in the Hyper-V, host sampled every second 2. Ntp ping data using W32tm stripchart command, as measured inside the VM against the W32time NTP server running on its Hyper-V host. (eg: w32tm.exe /stripchart /computer:<hyper- v-host> /period:1) 3. Ntp ping data using W32tm stripchart command, as measured inside the Hyper-V host against the W32time NTP server running on the VM. (eg: w32tm.exe /stripchart /computer:<VMt> /period:1) We collected only the following data stream for measuring time inside a physical machine for the simple NTP test: 4. NTP ping data using W32tm stripchart command, as measured in the test machine against the accurate NTP server. (eg: w32tm.exe /stripchart /computer:<AccurateNTPServer> /period:1) 4 Note: In Server 2016, data streams (1a, 2) and (1b, 3) can be collected simultaneously using the new “rdtsc” switch for stripchart command. The command line looks like this: W32tm.exe /stripchart /computer:time.windows.com /rdtsc /period:1 However, this requires a modified TimeSampleCorrelation tool in order for it to be used with this data. The modified version of the tool published on GitHub shortly. The time offset data collected using NTP pings is typically noisy due to asymmetry in network transit times. W32time service uses complex filtering to reduce or eliminate this noise. One of the goals of this document is to demonstrate the effectiveness of a simple filter described next to eliminate noise in NTP measurements and show that the filtered data can be used for reliable time offset determination. Popcorn Filter The output of w32tm stripchart command is the raw time offset derived from NTP pings as specified in the SNTP protocol. You can write a custom tool to implement this function as well. (Alan’s github share has his custom NTP ping tool), NTP time offset computation assumes the request/response roundtrip times are symmetric. This does not always hold true, even in simple topologies. In the worst case, NTP can add up to half of the roundtrip time as noise to the offset measurement. This noise can be up to ~100-200 microseconds on average and it can spike up to several milliseconds in some cases. If we have two computers whose time is synchronized with each other within, say 200 microseconds, the noisy NTP measurements can be a significant compared to this offset. The readings we get from NTP will be incorrect if we include the noisy samples and the raw data cannot be used for measuring time offset reliably. Based on the research shared in this document, you can see that the NTP Round Trip Times (RTT) falls in a wide range with a pseudo-Gaussian distribution. We can see that there are some values that are way above the range of RTT seen “normally”. These samples may or may not contain valid time offset data and they are the noise in the raw data. They occur randomly during measurement, much like kernels starting to pop while making popcorn. This noise in NTP is being referred to here as the “Popcorn Noise” and the simple filter being described to eliminate the noise is named the “Popcorn filter”. Each data point measured with NTP pings should contain this information: (RTT, Offset). Once we collect a data set for an interval, we do the following: 1. Determine the minimum RTT value (Min_RTT) (which is the absolute minimum RTT value possible) 2. Determine the average RTT value (average of all RTT values in the set) (Avg_RTT). Based on our observations, it seems like It is very unusual for RTT to exceed (Avg_RTT + (Avg_RTT - Min_RTT)). In all cases with large RTTs, it is very likely the measured time offset is incorrect due to NTP noise. The Popcorn filter uses the hypothesis that legitimate NTP measurements come from this limited range: [Min_RTT, (Avg_RTT + (Avg_RTT – Min_RTT))] 5 and if an NTP ping sample has RTT outside the above range, it can be excluded from consideration without any loss of information. Note that we can apply this filter only to data from Server 2016 and not on data from Server2012R2 because the time offsets in Server2012R2 exceeds the roundtrip times in all cases. Applying this filter would have very little effect on Server 2012R2 data. Additionally, the RTT cannot be accurately computed on Server2012R2 because it is in the sub-millisecond range for our set up and the W32tm.exe stripchart utility on Server 2012R2 does not support this level of accuracy. Notes Server vs Client Terminology and Version information W32time implementation ships with NTP Server, NTP Client and VMIC providers in-box on both Server and Client SKUs. The improvements to W32time are available in both Server 2016 and Windows 10 Anniversary Update releases. The term “client” is being used in the topology to refer to a domain member/client, which in all cases is a Server 2016 machine in our tests. It should be noted that identical results could be obtained when using a Windows 10 Anniversary update domain client. Server 2016 Forest Test Topology The underlying server hardware machines have 8 processor cores and 16 GB of memory each. The VMs are each assigned 2 processor cores and 4GB of hardware each, with the remaining available for the Hyper-V host partition’s use. All the machines are collecting time related data described above and not doing much else. 6 Figure 1 Server 2016 Forest Test Topology Server2012R2 Forest Test Topology This topology was configured on the same hardware as the Server2016 topology, allocating identical resources to the VMs running therein. The data collection is identical to Server2016 topology as well. 7 Figure 2 Server 2012R2 Forest Test Topology Methodology for AD Time Sync Accuracy Measurements - We know that the Hyper-V hosts keep a very accurate time using the Sync-n-Scale hardware/software. - All the computers in the above topology have the recommended high accuracy time settings. (See Script #1 Setting up common high accuracy settings) - Two scenarios are tested out for each topology: o Root PDC syncing time over VMIC (See Script #2 Enable VMIC Provider Only) o Root PDC syncing time over NTP from its Hyper-V host (See Script #3 Enable NTP Client Only with manual server) 8 - In both scenarios, all computers in the forest besides the PDC are configured to sync from the AD hierarchy and not use VMIC. (See Script #4 Enable NTP Client Only with Domain Sync) - In both scenarios, all the computers in the forest have W32time NTP Server enabled to remotely query their time (See Script #5 Enable NTP Server) - The data streams specified earlier in the section are collected for each VM for a period of 1.5-2 days. - The data collected and processed offline to generate the graphs shown in this doc. o TSC data from Hyper-V host and VM are matched up and the time offset is computed using the TimeSampleCorrelation tool (see Github link above). o Unfiltered NTP Offsets measurements are shown in the first chart for each VM. o In Server 2016 VMs, a “Popcorn filter” is applied to the NTP offsets to eliminate noisy samples and the resulting data is shown along with TSC data in a separate graph for each VM. o The data from different sources is aligned using the timestamps (which are down to the second) and excess data in any source is trimmed to ensure we are looking at common data from all sources. Methodology for Simple NTP accuracy measurements The AD measurements Scenario #2 involves client PDC syncing time from its time source over simple NTP, crossing a single virtual network switch. This is the best-case scenario for NTP synchronization. We set up a physical machine to sync time from an accurate time source about a quarter mile away, across 6 network switches in a generic corporate environment. We used these scripts to set up this machine - Script #1 Setting up common high accuracy settings - Script #3 Enable NTP Client Only with manual server Although this is dissimilar to the above scenario in many respects, it demonstrates the loss of accuracy in NTP with increase in network distance. The measurement methodology in the both cases is the same. 1. Collect W32time stripchart data on the test machine, against its accurate time source. 2. Apply Popcorn filter on the NTP data and examine the resulting offset data. Comparison between Server 2012R2 and Server 2016 Before we dive into the nitty gritty, let us look at side-by-side comparison charts of time sync performance between Server 2012R2 and Server 2016. These will showcase the improvements made in Server 2016 in near-best case scenarios in both forests. There are many different comparisons that could be made even with the small test topologies. Only the interesting comparisons are being included here for clarity. The reader can perform other kinds of comparisons with the charts included in this document. Most comparisons shown here are made between TSC offset data collected from both topologies. 9 We are using TSC data for simplicity purposes. One could make nearly identical comparisons between NTP data in 2012R2 and filtered NTP data in 2016. The last comparison is between filtered NTP data, since we don’t have TSC data to compare in that case. Note that the data presented here was collected separately for each (topology + scenario) and not all at the same time. Forest with a VMIC Root Time Source This is a comparison of two scenarios – one with Server 2016 Forest and another with Server 2012R2 forest, where the Forest Root Domain PDCs are syncing time from an accurate Server 2016 Hyper-V host/time server through the VMIC time provider available inbox. Server2016 performs significantly better in the first three comparisons between identical nodes in both the topologies. The last comparison in this scenario illustrates the performance of Server2016 Client in a Server2012R2 domain and compare it with Server 2016 Domain/Server2016 Client. You can see the impact of an inaccurate time source on Server2016 deployment here. The following chart compares the time offset on 2016 Forest Root Domain PDC against 2012R2 Forest Root Domain PDC. (Stratum == 2) Figure 3 Root PDC Topology Reference 10