login join

blackmagic's Blog

Member For: 2 months, 1 week
Posts: 49

Mark, Male
Member of: Diybetfairbots Forum.

Re: Axis Dodgyness

August 22, 2008 by blackmagic

Erm... i was a little bit. However in my defence you would assume that if you are using a multi threaded connection manager it might think about locking the httpclient object while making the calls. Am i expecting too much?

This axis thing is a tough one to work out. Im not sure what it synchornisation policy is since now i have gotten rid of all the multi threaded bit but still get AXIS FAULTS and lots of errors like

Caused by: org.apache.axiom.om.OMException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1]
Message: Premature end of file.

I can only assume that this is because i have multiple threads accessing a single binding object(for place bets, cancel, get prices etc.). Surely there is some sort of locking policy around this? I am thinking maybe not since axis2 makes all that who har about asyncronus web service calls so i can only assume threads using the binding are handing off to some axis thread. Hence how the returns might get screwed up.

Today i am trying two things. Object pooling for bindings, and alternately giving every function call its own binding(Plan B). This is all i can really think of at the moment so if anyone thinks Im completely missing the point then any help is welcome.

If this doesnt work Im going to read the axis2 manual. lets just hope it doent come to that!!!

Axis Dodgyness

August 21, 2008 by blackmagic

Does anyone know why you get errors like this in axis2

Caused by: org.apache.axis2.AxisFault: java.io.IOException: expected '=', got ':'
line 1, char 62: ...<?xmlerssio"1.01.0" enco" enc="UTg="UTF-8"?><<soasoapenv:E:

looks like the cached httpclients are getting confused with each other by im not sure how to stop it.

Re: When Bots go wrong...

August 15, 2008 by blackmagic

transaction charge for what?

1+1=3

August 12, 2008 by blackmagic

Ok so this is another funky thing that ive seen with rounding.

I take it you do the summation over the order book to find net exposure by looking at each bet with full precision (double) and then round the returned value to the nearest penny.

I get the feeling that the algo in bf does the rounding per trade and summates them. I only noticed it when doing a large size via a shed load of small trades. I noticed at the end of the race our books did not match by a few pence event though all my calcs seem ok (well you get the point).

Has anyone else noticed this.

Re: 88 miles per hour, great scott Marty

August 11, 2008 by blackmagic

Bingo, that did it, basically by using the REUSE_HTTP_CLIENT option then its working at roughly the ping speed. I think the extra 100ms was used to create a new HTTP client object which required a whole handshake of its own.

Thanks for all the help i hope this thread ends up being useful to anyone else trying to squeeze a little more juice from their axis engine.

Re: 88 miles per hour, great scott Marty

August 10, 2008 by blackmagic

Also was just reading the docs is it worth doing something like this for the stub?

i.e. not just the gzip bit but the cached re-use of the httpClient objects?

String _sEndPointURLSecure = BmProperties.getInstance().getProperty(BetdaqPropertyKeys.BETDAQ_SECURE_ENDPOINT);
secureServiceStub = new SecureServiceStub(_sEndPointURLSecure);

MultiThreadedHttpConnectionManager conmgr = new MultiThreadedHttpConnectionManager();
conmgr.getParams().setDefaultMaxConnectionsPerHost(10);
HttpClient client = new HttpClient(conmgr);
secureServiceStub._getServiceClient().getLastOperationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, client);

Options options = secureServiceStub._getServiceClient().getOptions();
options.setProperty(HTTPConstants.MC_ACCEPT_GZIP, "true");
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, "true");

Re: 88 miles per hour, great scott Marty

August 10, 2008 by blackmagic

Nadat i think you right but theres not a whole load of code in the middle of the call, so i dont think there is that much i could have screwed up. (I think!)

So over the weekend i tried using Axis2 with ADB and Xmlbeans the ADB had serious problems working with the betdaq api so then rewrote it using xml beans. Betdaq calls take about 180ms and betfair 200ms.

As you can see the code the timing is around the proxy call so it should be pretty light. Any ideas? 

Here is the code,

protected BFExchangeServiceStub createExchangeBindingStub() throws ExchangeException {
    try {
        if (exchangeBindingStub == null) {
            exchangeBindingStub = new BFExchangeServiceStub(_sEndPointURLExchange);
            Options options = exchangeBindingStub._getServiceClient().getOptions();
            options.setProperty(HTTPConstants.MC_ACCEPT_GZIP, "true");
            exchangeBindingStub._getServiceClient().setOptions(options);
        }
        return exchangeBindingStub;
    } catch (AxisFault axisFault) {
        throw new ExchangeException("Axis is generally fucked and cant create exchange binding stub", axisFault);
    }
}

protected com.betfair.www.publicapi.types.exchange.v5.APIRequestHeader getHeaderExchange() {
    if (exchangeHeader == null) {
        exchangeHeader = com.betfair.www.publicapi.types.exchange.v5.APIRequestHeader.Factory.newInstance();
    }
    exchangeHeader.setClientStamp(new GregorianCalendar().getTimeInMillis());
    exchangeHeader.setSessionToken(sessionToken);
    return exchangeHeader;

public String getMarketPricesCompressed(Integer marketId) throws ExchangeException {
    //Set up service calls
    BFExchangeServiceStub stubUK = createExchangeBindingStub();

    //Initialise request object
    GetMarketPricesCompressedDocument doc = GetMarketPricesCompressedDocument.Factory.newInstance();
    GetMarketPricesCompressedDocument.GetMarketPricesCompressed sub = GetMarketPricesCompressedDocument.GetMarketPricesCompressed.Factory.newInstance();
    GetMarketPricesCompressedReq req = GetMarketPricesCompressedReq.Factory.newInstance();

     //Initialise Header
    req.setHeader(getHeaderExchange());

    req.setMarketId(marketId);

    sub.setRequest(req);
    doc.setGetMarketPricesCompressed(sub);
    try {
        //Make the call
        PerfTimer timer = new PerfTimer();
        GetMarketPricesCompressedResp resp = stubUK.getMarketPricesCompressed(doc).getGetMarketPricesCompressedResponse().getResult();
        logger.info("bf time:"+timer.getElapsedTime());

        //Maintain the session
        setHeader(resp.getHeader());

        //Check for errors & return result
        if (resp.getErrorCode() != GetMarketPricesErrorEnum.OK) {
            throw new ExchangeException("Service Call Error - " + resp.getErrorCode());
        } else {
            return resp.getMarketPrices();
        }
    } catch (RemoteException e) {
       throw new ExchangeException("Error calling getMarketPricesCompressed", e);
    }
}

Re: 88 miles per hour, great scott Marty

August 9, 2008 by blackmagic

austinpodhorzer i have not but its what im going to do next.

I have a beasty connection through BE and the ping time to bf is 30-40ms. So yeah should be getting a lot less ive noticed that all the calls are not rpc style but doc style to thinking that might be something to do with it. I need to do some reading around soap to work out exactly what its doing in general.

If you can think of anything else then any help is much appreciated.

Re: 88 miles per hour, great scott Marty

August 9, 2008 by blackmagic

Ok so i was using axis1 i have now rewritten all of my proxy code to use axis2, it all looks pretty healthy, still with compression on im still only getting a round trip time of 200ms.

Not sure where i can squeeze a little more performance out of it. Im wondering if there are still some more options that im missing from my current configurations

other

Re: 88 miles per hour, great scott Marty

August 9, 2008 by blackmagic

Ok so i was using axis1 i have now rewritten all of my proxy code to use axis2, it all looks pretty healthy, still with compression on im still only getting a round trip time of 200ms.

Not sure where i can squeeze a little more performance out of it. Im wondering if there are still some more options that im missing from my current configurations

Re: 88 miles per hour, great scott Marty

August 7, 2008 by blackmagic

Ok so about this gzip thing. Theres a good article about it that makes it sound really easy

http://wso2.org/library/3007

Until you relaise that the hand hint,

//get the Options object used by your client
Option options = client.getOptions();
//set the property
options.setProperty(org.apache.axis2.transport.http.HTTPConstants.MC_GZIP_REQUEST, Boolean.TRUE);

Doesn't explain what the client object is or how you get a hold of it. Can you just set org.apache.axis2.transport.http.HTTPConstants.MC_GZIP_REQUEST as a System variable and get away with doing it that way?

Re: 88 miles per hour, great scott Marty

August 7, 2008 by blackmagic

I think I've missed the point of the binding stub, when i call the function stub.getMarketPricesCompressed i see in the axis logs that i get an exception saying that the service cannot be found. The function does however return the correct market prices (eventulaly).  Im however not sure what its doing internally. I know that the binding is essence the RPC proxy but im not sure how to set it up properly. I think the delay may actually be in it trying to search of the service properly. I need to do some more reading around what exactly a port/QName/service/Binding etc actually represents. Annoyingly im not really going to get any time to work on it until Saturday. The main thing is that I have a benchmark to work to now. 

Re: Ticks size

August 6, 2008 by blackmagic

god damn buttons, eating my lunch on my keyboard!!!

What i was going to say was that kind of makes your way better than mine. A bold step, but one im not sure my ego will let me take!

Im still amazed at how much mileage we got out of this.

thanks again for the help.

Re: Ticks size

August 6, 2008 by blackmagic

touche my freind, touche...

unfortunatly

that

Re: Ticks size

August 6, 2008 by blackmagic

touche my freind, touche...

unfortunatly

Re: Ticks size

August 6, 2008 by blackmagic

touche my freind, touche...

unfortunatly

that

Re: Ticks size

August 6, 2008 by blackmagic

touche my freind, touche...

unfortunatly

Re: Ticks size

August 6, 2008 by blackmagic

JPL I think is still a problem if you truncate/round to 2-3d.p. If you are rounding or truncating to 2 or 3dp then you are loosing some digits that you need to do the tick rounding properly. If you look closely at the examples i gave you can see fence post errors you get for each of the round/truncate methods.

Re: 88 miles per hour, great scott Marty

August 6, 2008 by blackmagic

Yeah I am realising quickly that im doing something pretty stupid... not sure what yet though. I upped the log4j logging level and saw a few exceptions in the axis call when trying to create the service (ConfigurationException service not found). I know very little about axis so im sure a little reading should get to the bottom of it.

Ill post my investigation.

88 miles per hour, great scott Marty

August 5, 2008 by blackmagic

This in reply to something birchy said in a different thread but i thought should be discussed separatly. He mentioned internet speeds being the biggest bottleneck. I've done alot of profiling looking at call times for different functions. Im not sure where some of the the time spent. Basically a price call takes in order of 300ms to 400 ms to run (java + axis + soap). Hence on a single thread you can only do about 2x per second. You can obviously improve this but run into to interesting sync issues if you have 10 stagered threads running to get 20x. What im wondering is where the 400ms is spent.

As i see it,

1. Conversion from Object to soap msg

2. Transmission to bf

3. Conversion from soap msg into Object

4. Price function call in bf

5. Results converted to soap msg

6. Transmission from bf

7. Conversion from soap msg into price Object

I spose in jotting that out theres quite a lot to be done, however it concerns me that no matter how many threads you use, you are always looking at a market snapshot thats at least 400ms out of date. If you're trading off that then a place bet call means you are trading on a price that is at least 600ms (assuming the place bet takes 200ms to get the bf). This is also why i think the internal scalpers at betfair are so successful.

How long does it take to get prices via the screen scraping method?

Re: Ticks size

August 5, 2008 by blackmagic

Thanks you two for all the help, you've provided some good food for thought. I think ive improved on both your techniques though. See what you think.
So i did it using the mod way, which i think is better a) fast since it doesnt us a look up table b) allows a greater level of accuracy than jpl's method.

Basically the problem with jpls method is that 3.0 is rarely 3.0 but 2.999999999999999999999 (thanks IEEE) which works when (int)(2.99999 * 100.0) = 299 which will get you 3.0. However if you have 3.02499999 which should be 3.025, if you mult by 100 and floor it you get 302 which will round to 3.00 and not 3.05.

3.0265 --> 3.00  // wrong
3.0298 --> 3.00
3.0331 --> 3.05

So use round instead of floor i hear you cry. Well you get a different problem at the other end 2.9868 should round to 2.98 but 298.68 rounds to 299 which gives you 3.00

2.9835 --> 2.98 
2.9868 --> 3.00 // wrong
2.9901 --> 3.00

So where is the solution? Well you just need to do it with floor and look at an extra decimal place i.e. scale up by 1000 not 100 and then do the floor and integer calc.

Here is the code anyways. (It assumes you use jpl's lookup table for tick size)

public double roundToNearestTick(double value) {
        final double scaleup = 1000.0 ;
        int iValue = (int)(value * scaleup);
        int tickSize = (int) (getTickSize(value) * scaleup);

        int remainder = iValue % tickSize;
        if (remainder < (tickSize / 2.0)) return ((iValue - remainder) / scaleup);
        else return ((iValue + (tickSize - remainder)) / scaleup);
}

Rounding is a bitch.

Re: Ticks size

August 5, 2008 by blackmagic

but if you specify your rounding like that dont you have an epic lookup table? i.e. 1000 * 100 in size 

Re: Ticks size

August 5, 2008 by blackmagic

ok i posted that before seeing the rest of jpls post. Interesting. Thanks.

Re: Ticks size

August 5, 2008 by blackmagic

Thats how i generate the table in the first place.

Lets run with this though as it was also the last inteligent thought I had also. So if you know that you have a price 2.005 then you get from the increments that the tick size is 0.02. but then what? you can't add or subtract it, otherwise you get 1.985 or 2.025. The only other thing i can think of is rounding it up by 2dp then doing some sort of integer arithmetic based on the tick size using mod to work out the delta from the nearest tick and then adding or subtracting it and rounding it back. 

Ill try coding that up and see what happens. 

Ticks size

August 5, 2008 by blackmagic

Hi,

I am writting a function to do tick rounding in bf and although i have found a way to do it, the implementation just makes me feel a little dirty.  It is easy to take a valid tick i.e. 2.02 and get the next i.e. 2.04. But to know that 2.005 should be 2.0 and not 2.02 is not so simple. (i think).

Ok so the current implementation has a look up table (yeah i know) of all possible ticks. If you have a number, find the nearst two ticks, then working out which of the two is closer is simple. This procedure is O(log2N) using a binary search. The time sucks as the look up table is about 400 values and the search which is log2(400) is about 8-9. It works, but its crap.

is there a O(1) solution to this? Go team...

Betfair Events

July 29, 2008 by blackmagic

Hey,

Has anyone been invited to the betfair events that they put on? If so what are they like and what sort of things get discussed.

Harder, better, faster, stronger

July 28, 2008 by blackmagic

At the weekend I finished a cross market arb bot between betfair and betdaq. A beautiful bit of kit, it places bets, and rebalance the books should any bets get missed. I was not sure how many outright arb opportunities there would be, I actually have something slightly different in mind for it but Saturday was the first time i ran it properly.


I completely had my ass handed to me. When i say handed to me, i mean with ribbons. I now have my answer to why no one will touch the arb bot projects on rent a coder. Im only using the free betfair api at the moment and you dont stand a chance of lifting anything unless you're doing sub second calls. I'd like to know where the floor is where you can start making money, all i currently know is its not a 1000ms. It ran for about 2 races and managed to completely execute 2 of 26 opportunities. The rest were partial or completely unmatched in both markets. This was not totally unexpected but if you're thinking of doing something similar, make sure you wear your big boy pants because the people you're playing with are good at what they do.


Im sure that at least one of you is responsible for all of this, all i can say is touche.

Re: Exploring

July 21, 2008 by blackmagic

Ok, in light of their api being still a work in progress I supose I would reduce my inital rant to half flame. They are however still some way from having something functionally useable, at least enough to make me happy to trade a large amount of money.

I would like to point out that they are miles ahead of other markets such as WBX who seem to have just given up in their api testing phase. They replied to my query about it simply saying, "we dont know how long testing is going to take".

Thanks for being specific.

Exploring

July 19, 2008 by blackmagic

Ok, so I’ve been playing around with a few new markets to see what they were like.


The one I would like to personally like name and shame is IBetX. Their api is pretty easy to use and they have basically copied the betfair api with a limited degree of success but their design and testing is just outright sketchy.


The first point is more of an annoyance, IbetX seemed to have decided it would be clever to make orders assume one of three states Unmatched, Partially Matched and Matched. This is because they have no concept of transactions like in bf. This means you have to make 2 full getOrder calls just to see what happened to your unmatched orders, since if it’s not in the unmatched category and you haven’t cancelled it, it still could be in the partially matched category. Long story short you have to be quite careful if orders go unmatched.

The second point left me in absolute awe; you can go into debt. i.e you can place a trade when you have no money in your account. The balance will then be negative. This I have to say is one of the most spastic things I have ever come across. Im quite lucky I was running a bot which I know to be very stable but if you didn’t know about this and you ran one that accidently blew up in a liquid market you could easily piss £10,000 you don’t have up the wall.


Finally their crowning glory has to be their response to the fact that their api is still buggy. While writing an adaptor for the market I noticed an interesting thing, it was impossible to cancel bets. Now this I thought was me just being dim, but it turns out that the order id returned by the place bets function was wrong. Hence you cant cancel a bet that isn’t yours and you have no handle to the bet you just placed. On emailing them to raise this fairly crucial oversight to their testing they responded by just turning off the api. Safe I agree, but just generally gives an indication of their lack of professionalism as you'd expect that the loss of revenue from over 4 days of trading would maybe spur them on to you know... fix it or something. Obviously not.

So a word of warning to you all. If you are thinking of using IBetX. Be very, very careful as its amateur at best.

Re: New Cross-Matching Logic on Betfair

July 14, 2008 by blackmagic

Hmmm, this is an interesting one, so under a single order for a selection you will end up with a set of transactions for random sub set of market selections?

Thanks for the heads up. This is one to be careful of if you are trying to match transactions to orders.

Login to an existing account

Welcome back!
Want to register? click here

Create an account

in only 3 steps!
already registered?