Mikrotik RouterOS Jitter Problem

Excessive Jitter on RouterOS

Updated for RouterOS 6.47.3 on 5-Sep-2020

Updated for RouterOS 6.46.7 on 14-Sep-2020

Willem A. Schreüder, Benjamin Matthews, Tracy Helhold, John Maxwell and Joey Stanford


Mikrotik RouterOS 6.46.2 introduced a change that has lead to excessive jitter. The problem persisted to various degree thoroughout the 6.46.3 to 6.47.1 verions, but appears to have been resolved in 6.47.2. The problem appears to reccur to a lesser extent in 6.47.3


Rocky Mountain Ham Radio (RMHAM) operates a network of more than 500 devices extending primarily over Wyoming, Colorado and New Mexico. The devices are primarily Mikrotik RB2011 routers, but larger cites have CCR1009 routers. Most microwave links are Netmetal 5HP radios operating in the 5GHz amateur band. The network is primarily used to connect amateur repeaters using AllStarLink for analog and DMR, P25, D-Star and Fusion digital repeaters.

Starting in the spring of 2020, RMHAM NetOps started to notice excessive jitter on the network when the network was upgraded RouterOS 4.46. The jitter was particularly noticable on the P25 repeaters. Reverting some of the microwave links to RouterOS 6.45.9 significantly reduced jitter.

Further investigation shows that in addition to increased jitter on the microwave links, there is also significantly increased jitter on routers connected by ethernet. The exact source of the jitter remains unknown but it appears to have been introduced in RouterOS 6.46.2 and persists to various degrees until 6.47.1. RouterOS 6.47.3 results appear similar to 6.47.1.

Routeros 6.46.7 replaced Routeros 6.45.9 as the new Long Term Stable release on September 14, 2020. The jitter observed in this version is similar to RouterOS 6.47.3 and except for VPN connections is lower than that of 6.45.9.

Materials and Methods

In order to eliminate variable traffic loads, the evaluation was performed on the RMHAM NetLab. The NetLab consists of 32 routers: one ten port RB2011 and thirty one RB750/RB751/RB951 five port routers. Each router is individually powered by 12V DC and routers are interconnected by short ethernet cables. The configuration mirrors the actual mountain top topology except that instead of microwave links the links are simply ethernet cables.

Figure 1. RMHAM Netlab

Figure 1 is a photograph of the NetLab. Purple cables represent direct connections between routers. Red cables represent indirect connections between routers via the RB2011 which represents commercial internet links. The NetLab is normally used for training, demonstration and testing.

Two Raspberry Pi 3B computers are connected part of the NetLab. One is used to provide NTP, DNS and similar services. The other is connected to the RB2011 and is used for penetration testing of firewalls and similar functions.

Every router in the NetLab was configured with the same version of RouterOS. The Routerboard firmware was updated to match to RouterOS version. All routing was performed using OSPF. A separate subnet was created for every link between routers. Routers connected via the RB2011 runs a PPTP VPN with a high OSPF weight. These VPN connections were not selected because the direct connections provided a lower cost. During the experiments, no Dijkstras were run on any of the routers because the networks was stable and no packets wer lost.

Timing was generated using the cping program. The cping program sends one ICMP packet per second to each router in the network and measures the round trip time. Measurements were made from a Raspberry Pi 3B computer connected near the center of the network. The identical hardware configuration was used for all measurements.

For each experiment, the appropriate NPK was downloaded to each router. All routers were then rebooted twice, once to install the appropriate packages, and once to update the Routerboard firmware. The identical configuration was used on each router for all experiments.

The last experiment attempted to use RouterOS 7.1beta2, but the system failed to route appropriately with the existing configuration.

Ping times were recorded over a six hour period commencing withing a few minutes of rebooting all NetLab routers. Individual ping times were recorded to a file for 21600 pings (one ping per second for 6 hours). No packets were lost during any of the experiments.

Experiments were run for RouterOS 6.45.9, the last long term stable verion of the RouterOS 6.45 series, all stable verions of the RouterOS 6.46 and 6.47 series. The NPK for each version of RouterOS was downloaded from the https://download.mikrotik.com/routeros/XXX/routeros-mipsbe-XXX.npk where XXX represents the version of RouterOS.


The results are shown in Figure 2. Individual ping times to each of the 32 routers are shown as blue lines. The arithmetic average ping time for all 32 routers computed for each one second interval is shown as a red line.

Figure 2a. ICMP ping times RouterOS 6.45.9

Figure 2b. ICMP ping times RouterOS 6.46

Figure 2c. ICMP ping times RouterOS 6.46.1

Figure 2d. ICMP ping times RouterOS 6.46.2

Figure 2e. ICMP ping times RouterOS 6.46.3

Figure 2f. ICMP ping times RouterOS 6.46.4

Figure 2g. ICMP ping times RouterOS 6.46.5

Figure 2h. ICMP ping times RouterOS 6.46.6

Figure 2h. ICMP ping times RouterOS 6.46.7

Figure 2i. ICMP ping times RouterOS 6.47

Figure 2j. ICMP ping times RouterOS 6.47.1

Figure 2k. ICMP ping times RouterOS 6.47.2

Figure 2l. ICMP ping times RouterOS 6.47.3

Average ping times are during normal operation is on the order of 1ms. The number of hops from the device recording the pings to each of the routers in the NetLab vary from 1 to 8.

The key result observed in Figure 2 is that starting with RouterOS 6.46.2, the jitter increased significantly. The median ping time remains about 1ms, but the at seemingly random intervals, ping times spike dramatically. No pings were lost, but times periodically spiked by one or two orders of magnitude.

To illustrate this behavior, Figure 3 shows a subset of the data from second 10000 for 10500.

Figure 3a. ICMP ping times RouterOS 6.45.9

Figure 3b. ICMP ping times RouterOS 6.46

Figure 3c. ICMP ping times RouterOS 6.46.1

Figure 3d. ICMP ping times RouterOS 6.46.2

Figure 3e. ICMP ping times RouterOS 6.46.3

Figure 3f. ICMP ping times RouterOS 6.46.4

Figure 3g. ICMP ping times RouterOS 6.46.5

Figure 3h. ICMP ping times RouterOS 6.46.6

Figure 3h. ICMP ping times RouterOS 6.46.7

Figure 3i. ICMP ping times RouterOS 6.47

Figure 3j. ICMP ping times RouterOS 6.47.1

Figure 3k. ICMP ping times RouterOS 6.47.2

Figure 3l. ICMP ping times RouterOS 6.47.3

Figure 3 shows that until RouterOS 6.46.1, the occasional ping is delayed by a few milliseconds, but on average the ping times remain around 1ms. However, starting with RouterOS 6.46.2, average ping times occasionally spike significantly into the tens of milliseconds to 100 millisecond range.

Another interesting observation is that this behavior does not start immediately on rebooting the router. Figure 4 shows the first 30 minutes of each experiment.

Figure 4a. ICMP ping times RouterOS 6.45.9

Figure 4b. ICMP ping times RouterOS 6.46

Figure 4c. ICMP ping times RouterOS 6.46.1

Figure 4d. ICMP ping times RouterOS 6.46.2

Figure 4e. ICMP ping times RouterOS 6.46.3

Figure 4f. ICMP ping times RouterOS 6.46.4

Figure 4g. ICMP ping times RouterOS 6.46.5

Figure 4h. ICMP ping times RouterOS 6.46.6

Figure 4h. ICMP ping times RouterOS 6.46.7

Figure 4i. ICMP ping times RouterOS 6.47

Figure 4j. ICMP ping times RouterOS 6.47.1

Figure 4k. ICMP ping times RouterOS 6.47.2

Figure 4l. ICMP ping times RouterOS 6.47.3

Figure 4 illustrates that in the problematic versions of RouterOS, the high jitter does not immediately start, but occurs after the experiment has been running for about 10 minutes. The start of the experiment was not timed precisely from the last reboot of the routers, but this result may suggest that there is some resource like memory that becomes exhausted leadingg to the observed increase in jitter.

The jitter was quantified by calculating the standard deviation of the ping times to each of the 32 routers. The results are shown in Figure 5.

Figure 5a. Standard Deviation of ICMP ping times for all versions.

Figure 5b. Standard Deviation of ICMP ping times for all LTS and latest verions.

Figure 5a shows that the dramatic increase in jutter for versions 6.46.2, 6.46.3 and 6.46.4 by router. Figure 5b shows just the two Long Term Stable versions 6.45.9 and 6.46.7 and the latest version 6.47.3. Figure 5b densonstartes that for many routers, teh standard deviation is significantly lower in version 6.46.7 than in 6.45.9. The Genoa router is connected over an SSTP VPN instead of a direct connection. This connection showed significantly more jitter than directly connected links, although the average ping time is essentially the same.

Detailed statistics are shown in Appendix 1.

Discussion and Conclusions

Although all versions of the RouterOS evaluated were from the Stable branch, the 6.46.2 through 6.46.5 introduced significant jitter. While no packets were lost, the jitter was sufficiently sever to create significant problems for UDP based radio linking systems. This was particularly sever for P25 systems, although all systems are effected to some degree.

A search of the Mikrotik forums showed no reports of the problem as detailed above. The source of the problem is not understood, but appears to have been resolved by RouterOS 6.47.2. The problem does appear to have resurfaced to a lesser extent in RouterOS 6.47.3 as it seems similar to 6.47.1.

The problem was readily apparrent when the production network was upgraded to RouterOS 6.46.6, but only by testing all versions in a controlled environment could this problem be isolated and evaluated.

Based on this experience, we recommend that the RMHAM network should only deploy the long term stable branch of RouterOS on the production network. The Long Term Stable branch receives security updates and therefore is as secure as later versions from the stable branch. There does not appear to be any advantages to using more recent versions of RouterOS 6.

The updated long term stable (LTS) RouterOS 6.46.7 appears to be well behaved and is an appropriate candidate for overwinter conditions 2020-2021.

Initial evaluation of RouterOS 7 did not go well. These problems may well be resolved by the time RouterOS 7 reaches Stable status, but significant testing should be performed before transitioning the network to RouterOS 7.

Appendix 1. Detailed Statistics