Jetty

SelectChannelConnector much to slow in windows

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 6.0.0rc2
  • Component/s: HTTP
  • Labels:
    None
  • Environment:
    jetty6 SVN head from 14:15 CEST 19. July 2006
    2 computers running winxp, connected via 100MBit/s Ethernet

  • Number of attachments :
    2

Description

SelectChannelConnector is extremely slow when client AND server run windows. Serving the same ~14MB jar file takes ~88 seconds (163 kB/s) using SelectChannelConnector and ~1 second (11,8 MB/s) using SocketConnector.

When the client uses a smaller buffer than 32kB (see client example) , the download becomes a bit faster (up to 460 kB/s).
(unfortunately, javaws 1.5 uses 32kB).
When the server runs on freebsd and the client on windows, I get about 8-9 MB/s using the SelectChannelConnector.

I will attach the following:

  • test server which serves the current directory using SelectChannelConnector on port 18080 and SocketConnector on port 18081
  • test client which connects to the server on both ports and measures time and speed
  • a big jarfile which is served

To reproduce:

  • Insert server hostname in SlowClient.java
  • compile client and server
  • Put SlowServer.class, big.jar and jetty libraries on one windows box and SlowClient.class on another windows box, both being connected via 100MBit ethernet.
  • run server and client
  1. Java Source File
    SlowClient.java
    19/Jul/06 8:42 AM
    1 kB
    Yug
  2. Java Source File
    SlowServer.java
    19/Jul/06 8:42 AM
    0.8 kB
    Yug

Issue Links

is related to

Bug - A problem which impairs or prevents the functions of the product. JETTY-94 CruiseControl's (V2.5) Jetty collapse on processing big log files in "Test Results View"

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Yug added a comment -

test client

Show
Yug added a comment - test client
Hide
Permalink
Yug added a comment -

test server

Show
Yug added a comment - test server
Hide
Permalink
Yug added a comment -

jira won't let me attach the 14MB jarfile. But any other 14MB file called big.jar should do as well.

Show
Yug added a comment - jira won't let me attach the 14MB jarfile. But any other 14MB file called big.jar should do as well.
Hide
Permalink
Greg Wilkins added a comment -

Yug,

as I don't have two windows machines... this is going to take me some time to look at.
Actually... I'll see if Nik can have a crack at this.

This is probably some sort of delay in noticing that the socket is writable?

Yug, can you capture some traffic with ethereal as this may tell us where the
delay is.

Show
Greg Wilkins added a comment - Yug, as I don't have two windows machines... this is going to take me some time to look at. Actually... I'll see if Nik can have a crack at this. This is probably some sort of delay in noticing that the socket is writable? Yug, can you capture some traffic with ethereal as this may tell us where the delay is.
Hide
Permalink
nik gonzalez added a comment -

I've replicated this already on 2 laptops running on winXP and ran ethereal to analyze what happens between the client and server. However, I can't pinpoint where exactly the selectchannel connector is taking long. I'm attaching the ethereal output files for both selectChannelconnector and socketconnector for anyone with better interpretting skills.

Show
nik gonzalez added a comment - I've replicated this already on 2 laptops running on winXP and ran ethereal to analyze what happens between the client and server. However, I can't pinpoint where exactly the selectchannel connector is taking long. I'm attaching the ethereal output files for both selectChannelconnector and socketconnector for anyone with better interpretting skills.
Hide
Permalink
nik gonzalez added a comment -

couldn't attach the file because it exceeded the 10 mb limit so i uploaded it here:

http://www.yousendit.com/transfer.php?action=download&ufid=1C706A7D654FC034

Show
nik gonzalez added a comment - couldn't attach the file because it exceeded the 10 mb limit so i uploaded it here: http://www.yousendit.com/transfer.php?action=download&ufid=1C706A7D654FC034
Hide
Permalink
nik gonzalez added a comment -

i also tried to run the client on a linux machine. here is the ethereal file:
http://rapidshare.de/files/28014771/slowserver-selectchannel18080-linux_client.zip.html

It was slow as well. took 138 seconds but that may be partly due to the slow network here. nevertheless 138 seconds is a big number

Show
nik gonzalez added a comment - i also tried to run the client on a linux machine. here is the ethereal file: http://rapidshare.de/files/28014771/slowserver-selectchannel18080-linux_client.zip.html It was slow as well. took 138 seconds but that may be partly due to the slow network here. nevertheless 138 seconds is a big number
Hide
Permalink
Greg Wilkins added a comment -

Nik,

Can you point to where I can get the capture file of the SocketConnector.

What were the total times for the two captures and what is the file size you are using?

Show
Greg Wilkins added a comment - Nik, Can you point to where I can get the capture file of the SocketConnector. What were the total times for the two captures and what is the file size you are using?
Hide
Permalink
Greg Wilkins added a comment -

Looking at the trace from the server on windows vs one I did on my linux machine, there are
three things I notice.

1) There appears to be a smaller window on the windows box. IE acks are sent for the
data packet pretty soon after they are sent. With the linux server, data is sent WAY in
advance of acks being received.

2) The windows server never fills up it's sliding window. The linux server sends data
until the window is full and then waits a bit for acks. 3 to 10 ms is the typical wait.

3) Even without a full window, the windows server stops sending data and then there
is a long pause (200ms) until the client sends an ack. It looks like the client is waiting
for either some return data or a full window before sending the ack.

So there is something about how this data is being sent that is making the windows
TCP/IP stack work really really badly.

Nik - we need to do some experiments to see how we can affect this by making variations
in the aspects discussed below.

Because of the size of this file, it is probably being sent by the DefaultServlet
using the Resource.writeTo() method, which eventually uses the
IO.copy method. This has a buffer size of only 8192 for transfers. We should
try a much larger buffer size ... perhaps 32k

We can adjust the configuration of the DefaultServlet so that the file cache
will accept this file size and use a memoryMapped buffer. This should be
the fastest way to send data. Ifthis works, then we will have to review the
concept of file cace with regards to memory mapped buffers... or perhaps
have temporary mem mapped buffers for large files etc.
Look at the code in DefaultServlet.getContent(String,Resource)

Failing that - turn off useMemoryMapped buffers - so the whole massive
file gets loaded into memory and then served from cache. This is not
a good way to proceed, but will tell us if the problem is how we read the
data or how we write the data.

We should also see what the OS has set Socket.getSendBufferSize() at
for the two types of connector. We should then try setting it to bigger
values to see if that makes a difference. We should research how
big this should be, and potentially we should either make it directly
configurable or set it to the same size as the setResponseBufferSize
field

Nik - can you do the experiments - as I still don't have a good windows
machine I can use for this.

thanks

Show
Greg Wilkins added a comment - Looking at the trace from the server on windows vs one I did on my linux machine, there are three things I notice. 1) There appears to be a smaller window on the windows box. IE acks are sent for the data packet pretty soon after they are sent. With the linux server, data is sent WAY in advance of acks being received. 2) The windows server never fills up it's sliding window. The linux server sends data until the window is full and then waits a bit for acks. 3 to 10 ms is the typical wait. 3) Even without a full window, the windows server stops sending data and then there is a long pause (200ms) until the client sends an ack. It looks like the client is waiting for either some return data or a full window before sending the ack. So there is something about how this data is being sent that is making the windows TCP/IP stack work really really badly. Nik - we need to do some experiments to see how we can affect this by making variations in the aspects discussed below. Because of the size of this file, it is probably being sent by the DefaultServlet using the Resource.writeTo() method, which eventually uses the IO.copy method. This has a buffer size of only 8192 for transfers. We should try a much larger buffer size ... perhaps 32k We can adjust the configuration of the DefaultServlet so that the file cache will accept this file size and use a memoryMapped buffer. This should be the fastest way to send data. Ifthis works, then we will have to review the concept of file cace with regards to memory mapped buffers... or perhaps have temporary mem mapped buffers for large files etc. Look at the code in DefaultServlet.getContent(String,Resource) Failing that - turn off useMemoryMapped buffers - so the whole massive file gets loaded into memory and then served from cache. This is not a good way to proceed, but will tell us if the problem is how we read the data or how we write the data. We should also see what the OS has set Socket.getSendBufferSize() at for the two types of connector. We should then try setting it to bigger values to see if that makes a difference. We should research how big this should be, and potentially we should either make it directly configurable or set it to the same size as the setResponseBufferSize field Nik - can you do the experiments - as I still don't have a good windows machine I can use for this. thanks
Hide
Permalink
nik gonzalez added a comment -

I used a 15.77 MB big.jar.

I adjusted the configuration of the DefaultServlet so that the file cache will accept this size as you suggested. Set it to 16MB. It worked! File transfer was much faster.

Here's the ethereal file: http://rapidshare.de/files/28136625/selectchannelconnector-maxcachedfilesize-16MB.zip.html

Show
nik gonzalez added a comment - I used a 15.77 MB big.jar. I adjusted the configuration of the DefaultServlet so that the file cache will accept this size as you suggested. Set it to 16MB. It worked! File transfer was much faster. Here's the ethereal file: http://rapidshare.de/files/28136625/selectchannelconnector-maxcachedfilesize-16MB.zip.html
Hide
Permalink
nik gonzalez added a comment -

meanwhile, I'll continue with the other experiments.

Show
nik gonzalez added a comment - meanwhile, I'll continue with the other experiments.
Hide
Permalink
nik gonzalez added a comment -

Here's the ethereal file for the socketconnector:
http://www.yousendit.com/transfer.php?action=download&ufid=D5F2E4BA413BEBB5

I just tried setting Socket.setSendBufferSize() equal to responsebuffersize (32*1034) and this is what I got (still slow at 100984 milliseconds at 156 kb/s):
http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html

I tried setting setSendBufferSize to 64k and file transfer was much faster. about 1344 milliseconds at 11738 kb/s-is this an acceptable speed level? the selectsocketconnector had exactly the same speed readings as socketconnector.

here's the ethereal file for that:
http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html

Show
nik gonzalez added a comment - Here's the ethereal file for the socketconnector: http://www.yousendit.com/transfer.php?action=download&ufid=D5F2E4BA413BEBB5 I just tried setting Socket.setSendBufferSize() equal to responsebuffersize (32*1034) and this is what I got (still slow at 100984 milliseconds at 156 kb/s): http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html I tried setting setSendBufferSize to 64k and file transfer was much faster. about 1344 milliseconds at 11738 kb/s-is this an acceptable speed level? the selectsocketconnector had exactly the same speed readings as socketconnector. here's the ethereal file for that: http://rapidshare.de/files/28232735/selectchannelconnector-setsendbuffersize-32k.zip.html
Hide
Permalink
Greg Wilkins added a comment -

I believe the static content handle refactor has fixed this problem (not tested
first hand, but has been reported as fixed).

Yug - Can you please test 6.0.0rc2 and reopen this issue if it is not the case.

Show
Greg Wilkins added a comment - I believe the static content handle refactor has fixed this problem (not tested first hand, but has been reported as fixed). Yug - Can you please test 6.0.0rc2 and reopen this issue if it is not the case.
Hide
Permalink
Yug added a comment -

With the latest svn, the windows-windows transferrates are ok.

SlowClient output windows server - windows client
SelectChannelConnector
http://crushinator:18080/big.jar
bytes read: 14364739
1609 ms
speed: 8927 kB/s
-----------------------------
SocketConnector
http://crushinator:18081/big.jar
bytes read: 14364739
1250 ms
speed: 11491 kB/s

For some odd reason, the SelectChannelConnector is still slow when the client runs on freebsd:

server windows - client freebsd
SelectChannelConnector
bytes read: 14364739
11177 ms
speed: 1285 kB/s
-----------------------------
SocketConnector
bytes read: 14364739
1277 ms
speed: 11248 kB/s

Show
Yug added a comment - With the latest svn, the windows-windows transferrates are ok. SlowClient output windows server - windows client SelectChannelConnector http://crushinator:18080/big.jar bytes read: 14364739 1609 ms speed: 8927 kB/s ----------------------------- SocketConnector http://crushinator:18081/big.jar bytes read: 14364739 1250 ms speed: 11491 kB/s For some odd reason, the SelectChannelConnector is still slow when the client runs on freebsd: server windows - client freebsd SelectChannelConnector bytes read: 14364739 11177 ms speed: 1285 kB/s ----------------------------- SocketConnector bytes read: 14364739 1277 ms speed: 11248 kB/s
Hide
Permalink
Endre Stølsvik added a comment -

.. but isn't it nevertheless strange that it still is noticable faster when using the SocketConnector?! And in particularly, obviously, on the second scenario windows/freebsd? (what about windows/linux?)

Btw, this would probably be a 100 Mbps network, so that ~11.5 MBps basically is rather close to the theoretical limit (100/8=12.5, plus TCP/IP overhead)?
It would be very interesting if one of you could repeat the tests using a setup where you couple those two machines together with a 1Gbps line, to see what it'd really max out at.
Note that very many boxes these days have a 1Gbps network port, but still most office/home switches are 100Mbps: You could actually just "hotwire" the boxen together w/o using a hub/switch (use static ips). One should load the entire file into memory then, or else the disk will definately limit - and probably use a somewhat larger file too, or else it will go too fast to give reliable measurements.

In a normal (not extremely-high-amount-of-users) environment, it seems like it would be smarter to still stick with the good'ol SocketConnector, rather than with the new'n'flash, obviously still not quite broken in SelectChannel stuff?

Show
Endre Stølsvik added a comment - .. but isn't it nevertheless strange that it still is noticable faster when using the SocketConnector?! And in particularly, obviously, on the second scenario windows/freebsd? (what about windows/linux?) Btw, this would probably be a 100 Mbps network, so that ~11.5 MBps basically is rather close to the theoretical limit (100/8=12.5, plus TCP/IP overhead)? It would be very interesting if one of you could repeat the tests using a setup where you couple those two machines together with a 1Gbps line, to see what it'd really max out at. Note that very many boxes these days have a 1Gbps network port, but still most office/home switches are 100Mbps: You could actually just "hotwire" the boxen together w/o using a hub/switch (use static ips). One should load the entire file into memory then, or else the disk will definately limit - and probably use a somewhat larger file too, or else it will go too fast to give reliable measurements. In a normal (not extremely-high-amount-of-users) environment, it seems like it would be smarter to still stick with the good'ol SocketConnector, rather than with the new'n'flash, obviously still not quite broken in SelectChannel stuff?

People

  • Assignee:
    nik gonzalez
    Reporter:
    Yug
Vote (1)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: