Track Topic
: rss

Topic: 88 miles per hour, great scott Marty

posts 1–20 of 22 newer >
Page 1 · 2
?
50 posts

This in reply to something birchy said in a different thread but i thought should be discussed separatly. He mentioned internet speeds being the biggest bottleneck. I've done alot of profiling looking at call times for different functions. Im not sure where some of the the time spent. Basically a price call takes in order of 300ms to 400 ms to run (java + axis + soap). Hence on a single thread you can only do about 2x per second. You can obviously improve this but run into to interesting sync issues if you have 10 stagered threads running to get 20x. What im wondering is where the 400ms is spent.

As i see it,

1. Conversion from Object to soap msg

2. Transmission to bf

3. Conversion from soap msg into Object

4. Price function call in bf

5. Results converted to soap msg

6. Transmission from bf

7. Conversion from soap msg into price Object

I spose in jotting that out theres quite a lot to be done, however it concerns me that no matter how many threads you use, you are always looking at a market snapshot thats at least 400ms out of date. If you're trading off that then a place bet call means you are trading on a price that is at least 600ms (assuming the place bet takes 200ms to get the bf). This is also why i think the internal scalpers at betfair are so successful.

How long does it take to get prices via the screen scraping method?

?
128 posts
That seems painfully slow and i can't imagine that your code is THAT slow. Time your code minus the actual HTTP request. Fake the data or load a previously saved packet from a local file if you have to. I'd be surprised if your code is to blame unless you (or some of your imported libs) are using some crazy loops.

Assuming it turns out to be internet lag, there's not much you can do because 200ms each way would not be considered as a line fault or anything abnormal by your ISP or BT. However, there are a couple of interesting things you can play with:

http://www.kitz.co.uk/adsl/interleaving.htm
http://www.dslzoneuk.net/maxspeed.php
http://www.kitz.co.uk/adsl/linestats.htm
http://www.dslzoneuk.net/ (see what speeds others are getting on your ISP)

The top one is of particular importance. Interleaving affects ping speeds and getting it changed increased my bot speeds by 200%.... Wink
?
31 posts

Seems slow to me - by comparison you can run the gruss software at 10 refreshes a second (haven't timed my code recently)

?
50 posts

Yeah I am realising quickly that im doing something pretty stupid... not sure what yet though. I upped the log4j logging level and saw a few exceptions in the axis call when trying to create the service (ConfigurationException service not found). I know very little about axis so im sure a little reading should get to the bottom of it.

Ill post my investigation.

?
70 posts

I manage to get 60-70ms turnaround for getMarketPricesCompressed using java and axis2, even using xmlbeans and a crappy bt broadband connection. A client of mine gets around 30-40ms running the same code, presumably due to a better bband connection.

Make sure you have gzip compression enabled?

?
50 posts

I think I've missed the point of the binding stub, when i call the function stub.getMarketPricesCompressed i see in the axis logs that i get an exception saying that the service cannot be found. The function does however return the correct market prices (eventulaly).  Im however not sure what its doing internally. I know that the binding is essence the RPC proxy but im not sure how to set it up properly. I think the delay may actually be in it trying to search of the service properly. I need to do some more reading around what exactly a port/QName/service/Binding etc actually represents. Annoyingly im not really going to get any time to work on it until Saturday. The main thing is that I have a benchmark to work to now. 

?
50 posts

Ok so about this gzip thing. Theres a good article about it that makes it sound really easy

http://wso2.org/library/3007

Until you relaise that the hand hint,

//get the Options object used by your client
Option options = client.getOptions();
//set the property
options.setProperty(org.apache.axis2.transport.http.HTTPConstants.MC_GZIP_REQUEST, Boolean.TRUE);

Doesn't explain what the client object is or how you get a hold of it. Can you just set org.apache.axis2.transport.http.HTTPConstants.MC_GZIP_REQUEST as a System variable and get away with doing it that way?

?
70 posts

If you're using axis2 and xmlbeans binding it should look something like...

BFExchangeServiceStub stub = new BFExchangeServiceStub("https://api.betfair.com/exchange/v5/BFExchangeService");
Options o = stub._getServiceClient().getOptions();
o.setProperty(HTTPConstants.MC_ACCEPT_GZIP, Boolean.TRUE);
stub._getServiceClient().setOptions(o);

?
50 posts

Ok so i was using axis1 i have now rewritten all of my proxy code to use axis2, it all looks pretty healthy, still with compression on im still only getting a round trip time of 200ms.

Not sure where i can squeeze a little more performance out of it. Im wondering if there are still some more options that im missing from my current configurations

?
50 posts

Ok so i was using axis1 i have now rewritten all of my proxy code to use axis2, it all looks pretty healthy, still with compression on im still only getting a round trip time of 200ms.

Not sure where i can squeeze a little more performance out of it. Im wondering if there are still some more options that im missing from my current configurations

other
?
30 posts

What's your ping time for bdp.betfair.com? For me, it's pretty close to what a gMPC call takes. If it takes 200 ms for you to ping the developer's site, there's probably not much you can do to improve things.

?
128 posts
As i said above, i can't imagine ANY code being so bad that it takes >100ms to run. 1.01 that it's internet lag. 1.01 that changing interleaving to "fast path" improves it to 100ms or less...
?
70 posts

blackmagic ... have you read the axis2 1.3 docs on configuring the HTTP transport layer?

?
50 posts

austinpodhorzer i have not but its what im going to do next.

I have a beasty connection through BE and the ping time to bf is 30-40ms. So yeah should be getting a lot less ive noticed that all the calls are not rpc style but doc style to thinking that might be something to do with it. I need to do some reading around soap to work out exactly what its doing in general.

If you can think of anything else then any help is much appreciated.

?
128 posts
30-40ms is pretty smokey, so you have some seriously bad code running somewhere, or maybe a firewall interfering.
member
70 posts

I think the rpc/document thing is a red herring. I'm not a SOAP expert, but isn't that decided by the nature of the wsdl? I could be talking b0ll0x ... anyone know?

If you generate the stubs using the axis2 wsdl2java in xmlbeans mode it is fairly verbose in terms of the wrapper code you have to write, but performance shouldn't be too bad if you optimize the configuration of the HTTPTransport layer.
Does you code look something like?

GetMarketPricesCompressedResponseDocument resp = null;
GetMarketPricesCompressedReq req = GetMarketPricesCompressedReq.Factory.newInstance();
req.setHeader(header);
req.setMarketId(marketId);
GetMarketPricesCompressedDocument getMarketPricesCompressedDocument = GetMarketPricesCompressedDocument.Factory.newInstance();
GetMarketPricesCompressedDocument.GetMarketPricesCompressed getMarketPricesCompressed = GetMarketPricesCompressedDocument.GetMarketPricesCompressed.Factory.newInstance();
getMarketPricesCompressed.setRequest(req);
getMarketPricesCompressedDocument.setGetMarketPricesCompressed(getMarketPricesCompressed);
GetMarketPricesCompressedResponseDocument response = stub.getMarketPricesCompressed(getMarketPricesCompressedDocument);

member
30 posts

If your pings take 30-40 ms and your web service call takes 200 ms, there is something seriously wrong the way you're using the web service. Forget about looking for optimisations; there is a gross error or oversight somewhere. I've not used Axis (last time I fiddled with Java web services, I used jax-ws, wsimport, etc.), but are you sure you're not using some dynamic proxy? Perhaps you should post the code for your connection to the web service and subsequent pricing call. Maybe someone with Axis experience might be able to spot something.

member
50 posts

Nadat i think you right but theres not a whole load of code in the middle of the call, so i dont think there is that much i could have screwed up. (I think!)

So over the weekend i tried using Axis2 with ADB and Xmlbeans the ADB had serious problems working with the betdaq api so then rewrote it using xml beans. Betdaq calls take about 180ms and betfair 200ms.

As you can see the code the timing is around the proxy call so it should be pretty light. Any ideas? 

Here is the code,

protected BFExchangeServiceStub createExchangeBindingStub() throws ExchangeException {
    try {
        if (exchangeBindingStub == null) {
            exchangeBindingStub = new BFExchangeServiceStub(_sEndPointURLExchange);
            Options options = exchangeBindingStub._getServiceClient().getOptions();
            options.setProperty(HTTPConstants.MC_ACCEPT_GZIP, "true");
            exchangeBindingStub._getServiceClient().setOptions(options);
        }
        return exchangeBindingStub;
    } catch (AxisFault axisFault) {
        throw new ExchangeException("Axis is generally fucked and cant create exchange binding stub", axisFault);
    }
}

protected com.betfair.www.publicapi.types.exchange.v5.APIRequestHeader getHeaderExchange() {
    if (exchangeHeader == null) {
        exchangeHeader = com.betfair.www.publicapi.types.exchange.v5.APIRequestHeader.Factory.newInstance();
    }
    exchangeHeader.setClientStamp(new GregorianCalendar().getTimeInMillis());
    exchangeHeader.setSessionToken(sessionToken);
    return exchangeHeader;

public String getMarketPricesCompressed(Integer marketId) throws ExchangeException {
    //Set up service calls
    BFExchangeServiceStub stubUK = createExchangeBindingStub();

    //Initialise request object
    GetMarketPricesCompressedDocument doc = GetMarketPricesCompressedDocument.Factory.newInstance();
    GetMarketPricesCompressedDocument.GetMarketPricesCompressed sub = GetMarketPricesCompressedDocument.GetMarketPricesCompressed.Factory.newInstance();
    GetMarketPricesCompressedReq req = GetMarketPricesCompressedReq.Factory.newInstance();

     //Initialise Header
    req.setHeader(getHeaderExchange());

    req.setMarketId(marketId);

    sub.setRequest(req);
    doc.setGetMarketPricesCompressed(sub);
    try {
        //Make the call
        PerfTimer timer = new PerfTimer();
        GetMarketPricesCompressedResp resp = stubUK.getMarketPricesCompressed(doc).getGetMarketPricesCompressedResponse().getResult();
        logger.info("bf time:"+timer.getElapsedTime());

        //Maintain the session
        setHeader(resp.getHeader());

        //Check for errors & return result
        if (resp.getErrorCode() != GetMarketPricesErrorEnum.OK) {
            throw new ExchangeException("Service Call Error - " + resp.getErrorCode());
        } else {
            return resp.getMarketPrices();
        }
    } catch (RemoteException e) {
       throw new ExchangeException("Error calling getMarketPricesCompressed", e);
    }
}

member
50 posts

Also was just reading the docs is it worth doing something like this for the stub?

i.e. not just the gzip bit but the cached re-use of the httpClient objects?

String _sEndPointURLSecure = BmProperties.getInstance().getProperty(BetdaqPropertyKeys.BETDAQ_SECURE_ENDPOINT);
secureServiceStub = new SecureServiceStub(_sEndPointURLSecure);

MultiThreadedHttpConnectionManager conmgr = new MultiThreadedHttpConnectionManager();
conmgr.getParams().setDefaultMaxConnectionsPerHost(10);
HttpClient client = new HttpClient(conmgr);
secureServiceStub._getServiceClient().getLastOperationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, client);

Options options = secureServiceStub._getServiceClient().getOptions();
options.setProperty(HTTPConstants.MC_ACCEPT_GZIP, "true");
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, "true");

member
50 posts

Bingo, that did it, basically by using the REUSE_HTTP_CLIENT option then its working at roughly the ping speed. I think the extra 100ms was used to create a new HTTP client object which required a whole handshake of its own.

Thanks for all the help i hope this thread ends up being useful to anyone else trying to squeeze a little more juice from their axis engine.

posts 1–20 of 22 newer >
Page 1 · 2

This Topic Is Locked To Guest Posts

It's been a while since this topic was active, if you'd like to get it going again, please post as a registered member

join now