Jetty
  1. Jetty
  2. JETTY-93

SelectChannelConnector much to slow in windows

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.0rc2
    • Component/s: HTTP
    • Labels:
      None
    • Environment:
      jetty6 SVN head from 14:15 CEST 19. July 2006
      2 computers running winxp, connected via 100MBit/s Ethernet

    • Number of attachments :
      2

      Description

      SelectChannelConnector is extremely slow when client AND server run windows. Serving the same ~14MB jar file takes ~88 seconds (163 kB/s) using SelectChannelConnector and ~1 second (11,8 MB/s) using SocketConnector.

      When the client uses a smaller buffer than 32kB (see client example) , the download becomes a bit faster (up to 460 kB/s).
      (unfortunately, javaws 1.5 uses 32kB).
      When the server runs on freebsd and the client on windows, I get about 8-9 MB/s using the SelectChannelConnector.

      I will attach the following:

      • test server which serves the current directory using SelectChannelConnector on port 18080 and SocketConnector on port 18081
      • test client which connects to the server on both ports and measures time and speed
      • a big jarfile which is served

      To reproduce:

      • Insert server hostname in SlowClient.java
      • compile client and server
      • Put SlowServer.class, big.jar and jetty libraries on one windows box and SlowClient.class on another windows box, both being connected via 100MBit ethernet.
      • run server and client
      1. SlowClient.java
        1 kB
        Yug
      2. SlowServer.java
        0.8 kB
        Yug

        Issue Links

          Activity

          Hide
          Yug added a comment -

          test client

          Show
          Yug added a comment - test client
          Hide
          Yug added a comment -

          test server

          Show
          Yug added a comment - test server
          Hide
          Yug added a comment -

          jira won't let me attach the 14MB jarfile. But any other 14MB file called big.jar should do as well.

          Show
          Yug added a comment - jira won't let me attach the 14MB jarfile. But any other 14MB file called big.jar should do as well.
          Hide
          Greg Wilkins added a comment -

          Yug,

          as I don't have two windows machines... this is going to take me some time to look at.
          Actually... I'll see if Nik can have a crack at this.

          This is probably some sort of delay in noticing that the socket is writable?

          Yug, can you capture some traffic with ethereal as this may tell us where the
          delay is.

          Show
          Greg Wilkins added a comment - Yug, as I don't have two windows machines... this is going to take me some time to look at. Actually... I'll see if Nik can have a crack at this. This is probably some sort of delay in noticing that the socket is writable? Yug, can you capture some traffic with ethereal as this may tell us where the delay is.
          Hide
          nik gonzalez added a comment -

          I've replicated this already on 2 laptops running on winXP and ran ethereal to analyze what happens between the client and server. However, I can't pinpoint where exactly the selectchannel connector is taking long. I'm attaching the ethereal output files for both selectChannelconnector and socketconnector for anyone with better interpretting skills.

          Show
          nik gonzalez added a comment - I've replicated this already on 2 laptops running on winXP and ran ethereal to analyze what happens between the client and server. However, I can't pinpoint where exactly the selectchannel connector is taking long. I'm attaching the ethereal output files for both selectChannelconnector and socketconnector for anyone with better interpretting skills.
          Hide
          nik gonzalez added a comment -

          couldn't attach the file because it exceeded the 10 mb limit so i uploaded it here:

          http://www.yousendit.com/transfer.php?action=download&ufid=1C706A7D654FC034

          Show
          nik gonzalez added a comment - couldn't attach the file because it exceeded the 10 mb limit so i uploaded it here: http://www.yousendit.com/transfer.php?action=download&ufid=1C706A7D654FC034
          Hide
          nik gonzalez added a comment -

          i also tried to run the client on a linux machine. here is the ethereal file:
          http://rapidshare.de/files/28014771/slowserver-selectchannel18080-linux_client.zip.html

          It was slow as well. took 138 seconds but that may be partly due to the slow network here. nevertheless 138 seconds is a big number

          Show
          nik gonzalez added a comment - i also tried to run the client on a linux machine. here is the ethereal file: http://rapidshare.de/files/28014771/slowserver-selectchannel18080-linux_client.zip.html It was slow as well. took 138 seconds but that may be partly due to the slow network here. nevertheless 138 seconds is a big number
          Hide
          Greg Wilkins added a comment -

          Nik,

          Can you point to where I can get the capture file of the SocketConnector.

          What were the total times for the two captures and what is the file size you are using?

          Show
          Greg Wilkins added a comment - Nik, Can you point to where I can get the capture file of the SocketConnector. What were the total times for the two captures and what is the file size you are using?
          Hide
          Greg Wilkins added a comment -

          Looking at the trace from the server on windows vs one I did on my linux machine, there are
          three things I notice.

          1) There appears to be a smaller window on the windows box. IE acks are sent for the
          data packet pretty soon after they are sent. With the linux server, data is sent WAY in
          advance of acks being received.

          2) The windows server never fills up it's sliding window. The linux server sends data
          until the window is full and then waits a bit for acks. 3 to 10 ms is the typical wait.

          3) Even without a full window, the windows server stops sending data and then there
          is a long pause (200ms) until the client sends an ack. It looks like the client is waiting
          for either some return data or a full window before sending the ack.

          So there is something about how this data is being sent that is making the windows
          TCP/IP stack work really really badly.

          Nik - we need to do some experiments to see how we can affect this by making variations
          in the aspects discussed below.

          Because of the size of this file, it is probably being sent by the DefaultServlet
          using the Resource.writeTo() method, which eventually uses the
          IO.copy method. This has a buffer size of only 8192 for transfers. We should
          try a much larger buffer size ... perhaps 32k

          We can adjust the configuration of the DefaultServlet so that the file cache
          will accept this file size and use a memoryMapped buffer. This should be
          the fastest way to send data. Ifthis works, then we will have to review the
          concept of file cace with regards to memory mapped buffers... or perhaps
          have temporary mem mapped buffers for large files etc.
          Look at the code in DefaultServlet.getContent(String,Resource)

          Failing that - turn off useMemoryMapped buffers - so the whole massive
          file gets loaded into memory and then served from cache. This is not
          a good way to proceed, but will tell us if the problem is how we read the
          data or how we write the data.

          We should also see what the OS has set Socket.getSendBufferSize() at
          for the two types of connector. We should then try setting it to bigger
          values to see if that makes a difference. We should research how
          big this should be, and potentially we should either make it directly
          configurable or set it to the same size as the setResponseBufferSize
          field

          Nik - can you do the experiments - as I still don't have a good windows
          machine I can use for this.

          thanks

          Show
          Greg Wilkins added a comment - Looking at the trace from the server on windows vs one I did on my linux machine, there are three things I notice. 1) There appears to be a smaller window on the windows box. IE acks are sent for the data packet pretty soon after they are sent. With the linux server, data is sent WAY in advance of acks being received. 2) The windows server never fills up it's sliding window. The linux server sends data until the window is full and then waits a bit for acks. 3 to 10 ms is the typical wait. 3) Even without a full window, the windows server stops sending data and then there is a long pause (200ms) until the client sends an ack. It looks like the client is waiting for either some return data or a full window before sending the ack. So there is something about how this data is being sent that is making the windows TCP/IP stack work really really badly. Nik - we need to do some experiments to see how we can affect this by making variations in the aspects discussed below. Because of the size of this file, it is probably being sent by the DefaultServlet using the Resource.writeTo() method, which eventually uses the IO.copy method. This has a buffer size of only 8192 for transfers. We should try a much larger buffer size ... perhaps 32k We can adjust the configuration of the DefaultServlet so that the file cache will accept this file size and use a memoryMapped buffer. This should be the fastest way to send data. Ifthis works, then we will have to review the concept of file cace with regards to memory mapped buffers... or perhaps have temporary mem mapped buffers for large files etc. Look at the code in DefaultServlet.getContent(String,Resource) Failing that - turn off useMemoryMapped buffers - so the whole massive file gets loaded into memory and then served from cache. This is not a good way to proceed, but will tell us if the problem is how we read the data or how we write the data. We should also see what the OS has set Socket.getSendBufferSize() at for the two types of connector. We should then try setting it to bigger values to see if that makes a difference. We should research how big this should be, and potentially we should either make it directly configurable or set it to the same size as the setResponseBufferSize field Nik - can you do the experiments - as I still don't have a good windows machine I can use for this. thanks
          Hide
          nik gonzalez added a comment -

          I used a 15.77 MB big.jar.

          I adjusted the configuration of the DefaultServlet so that the file cache will accept this size as you suggested. Set it to 16MB. It worked! File transfer was much faster.

          Here's the ethereal file: http://rapidshare.de/files/28136625/selectchannelconnector-maxcachedfilesize-16MB.zip.html

          Show
          nik gonzalez added a comment - I used a 15.77 MB big.jar. I adjusted the configuration of the DefaultServlet so that the file cache will accept this size as you suggested. Set it to 16MB. It worked! File transfer was much faster. Here's the ethereal file: http://rapidshare.de/files/28136625/selectchannelconnector-maxcachedfilesize-16MB.zip.html
          Hide
          nik gonzalez added a comment -

          meanwhile, I'll continue with the other experiments.

          Show
          nik gonzalez added a comment - meanwhile, I'll continue with the other experiments.
          Hide
          nik gonzalez added a comment -

          Here's the ethereal file for the socketconnector:
          http://www.yousendit.com/transfer.php?action=download&ufid=D5F2E4BA413BEBB5

          I just tried setting Socket.setSendBufferSize() equal to responsebuffersize (32*1034) and this is what I got (still slow at 100984 milliseconds at 156 kb/s):
          http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html

          I tried setting setSendBufferSize to 64k and file transfer was much faster. about 1344 milliseconds at 11738 kb/s-is this an acceptable speed level? the selectsocketconnector had exactly the same speed readings as socketconnector.

          here's the ethereal file for that:
          http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html

          Show
          nik gonzalez added a comment - Here's the ethereal file for the socketconnector: http://www.yousendit.com/transfer.php?action=download&ufid=D5F2E4BA413BEBB5 I just tried setting Socket.setSendBufferSize() equal to responsebuffersize (32*1034) and this is what I got (still slow at 100984 milliseconds at 156 kb/s): http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html I tried setting setSendBufferSize to 64k and file transfer was much faster. about 1344 milliseconds at 11738 kb/s-is this an acceptable speed level? the selectsocketconnector had exactly the same speed readings as socketconnector. here's the ethereal file for that: http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html
          Hide
          Greg Wilkins added a comment -

          I believe the static content handle refactor has fixed this problem (not tested
          first hand, but has been reported as fixed).

          Yug - Can you please test 6.0.0rc2 and reopen this issue if it is not the case.

          Show
          Greg Wilkins added a comment - I believe the static content handle refactor has fixed this problem (not tested first hand, but has been reported as fixed). Yug - Can you please test 6.0.0rc2 and reopen this issue if it is not the case.
          Hide
          Yug added a comment -

          With the latest svn, the windows-windows transferrates are ok.

          SlowClient output windows server - windows client
          SelectChannelConnector
          http://crushinator:18080/big.jar
          bytes read: 14364739
          1609 ms
          speed: 8927 kB/s
          -----------------------------
          SocketConnector
          http://crushinator:18081/big.jar
          bytes read: 14364739
          1250 ms
          speed: 11491 kB/s

          For some odd reason, the SelectChannelConnector is still slow when the client runs on freebsd:

          server windows - client freebsd
          SelectChannelConnector
          bytes read: 14364739
          11177 ms
          speed: 1285 kB/s
          -----------------------------
          SocketConnector
          bytes read: 14364739
          1277 ms
          speed: 11248 kB/s

          Show
          Yug added a comment - With the latest svn, the windows-windows transferrates are ok. SlowClient output windows server - windows client SelectChannelConnector http://crushinator:18080/big.jar bytes read: 14364739 1609 ms speed: 8927 kB/s ----------------------------- SocketConnector http://crushinator:18081/big.jar bytes read: 14364739 1250 ms speed: 11491 kB/s For some odd reason, the SelectChannelConnector is still slow when the client runs on freebsd: server windows - client freebsd SelectChannelConnector bytes read: 14364739 11177 ms speed: 1285 kB/s ----------------------------- SocketConnector bytes read: 14364739 1277 ms speed: 11248 kB/s
          Hide
          Endre Stølsvik added a comment -

          .. but isn't it nevertheless strange that it still is noticable faster when using the SocketConnector?! And in particularly, obviously, on the second scenario windows/freebsd? (what about windows/linux?)

          Btw, this would probably be a 100 Mbps network, so that ~11.5 MBps basically is rather close to the theoretical limit (100/8=12.5, plus TCP/IP overhead)?
          It would be very interesting if one of you could repeat the tests using a setup where you couple those two machines together with a 1Gbps line, to see what it'd really max out at.
          Note that very many boxes these days have a 1Gbps network port, but still most office/home switches are 100Mbps: You could actually just "hotwire" the boxen together w/o using a hub/switch (use static ips). One should load the entire file into memory then, or else the disk will definately limit - and probably use a somewhat larger file too, or else it will go too fast to give reliable measurements.

          In a normal (not extremely-high-amount-of-users) environment, it seems like it would be smarter to still stick with the good'ol SocketConnector, rather than with the new'n'flash, obviously still not quite broken in SelectChannel stuff?

          Show
          Endre Stølsvik added a comment - .. but isn't it nevertheless strange that it still is noticable faster when using the SocketConnector?! And in particularly, obviously, on the second scenario windows/freebsd? (what about windows/linux?) Btw, this would probably be a 100 Mbps network, so that ~11.5 MBps basically is rather close to the theoretical limit (100/8=12.5, plus TCP/IP overhead)? It would be very interesting if one of you could repeat the tests using a setup where you couple those two machines together with a 1Gbps line, to see what it'd really max out at. Note that very many boxes these days have a 1Gbps network port, but still most office/home switches are 100Mbps: You could actually just "hotwire" the boxen together w/o using a hub/switch (use static ips). One should load the entire file into memory then, or else the disk will definately limit - and probably use a somewhat larger file too, or else it will go too fast to give reliable measurements. In a normal (not extremely-high-amount-of-users) environment, it seems like it would be smarter to still stick with the good'ol SocketConnector, rather than with the new'n'flash, obviously still not quite broken in SelectChannel stuff?

            People

            • Assignee:
              nik gonzalez
              Reporter:
              Yug
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: