jpl's Blog

I started with PHP (because I knew it, and because fred77 was kind enough to post sample code).

I've since switched over to Python, because it's such a nice language, and because of scipy, which is brilliant.

Unfortunately I've not yet had time to rewrite my main bot, so I'm running a rather messy hybrid where the data collection / bet execution are in PHP, communicating (via a file interface) with a separate scipy process that does the hard sums (minimising a Kelly utility function over an N-dimensional space of possible bets). 

But if I was starting again I'd go Python for sure.

Just bought a copy of 'Automatic Exchange Betting' by Colin Magee - it was waiting on the doormat when I got home.  All I need to do now is summon the energy to read it. 

Anyone else read it?  Any good?

Thanks.

It wouldn't do anything clever - it would just seek to get bets matched at the most competitive possible price, without knowing what the fair price is on any individual market.

Nonetheless, we know that (on average) the fair price for each market lies between the best back price on offer and the best lay price on offer.  So if a bot can get random lay bets matched at (or close to) the best back price on each market it's reasonable to expect it will be getting a better-than-fair price on average.

In the absence of the Premium Charge no-one would bother doing this, because any profit you make is likely to be swallowed up by commission.  But in the context of the Premium Charge a strategy that generates lots of commission could be very valuable, even if all of its profit is swallowed up in commission.

That's my theory anyway - I can't claim to have tried it out yet.

My thoughts on a workaround - which may be stupid, I haven't thought through the detail - are that you'd need a bot specificially designed to place lots of bets.

Let's suppose you've got a clever bot that makes £100k a year and pays very little commission. To shield that bot from the PC you need a second bot that generates £20k commission at minimum cost to yourself.

So let's say you write a bot that places a single £20 lay bet, at roughly even odds, on 50,000 randomly selected markets per year. If all the bets were placed at fair prices, the pre-commission profit would be zero. You'd win £500,000 on the bets that win, and lose £500,000 on the bets that lose. So if you were on 4% commission (for the sake of argument) you'd pay £20k on the winning bets.

Of course this hasn't helped at all - you've generated £20k commission at a cost of £20k.  But that's because we've assumed you placed the bets at a fair price.  What you need to do is take some of the spread for yourself, and lay the bets at a few ticks below the fair price.  Then you're potentially making a small pre-commission profit which offsets some or all of the £20k commission.

I've not looked into it in detail, but my gut feel is that it's not unrealistic to expect to be able to place small bets in a large range of markets at (on average) a few ticks below the fair price.  After all, the nature of an exchange is that those punters who want to be matched immediately pay the spread, and those with the patience to wait receive it.  Be interested to hear what others think though.

Unless your winnings were so huge as to directly affect future markets (e.g. by scaring off other punters) the fact that your bot has had a lucky streak doesn't make it any more likely to have an unlucky streak.  To quote from Wikipedia:

The gambler's fallacy, also known as the Monte Carlo fallacy or the fallacy of the maturity of chances, is the false belief that if deviations from expected behaviour are observed in repeated independent trials of some random process then these deviations are likely to be evened out by opposite deviations in the future. For example, if a fair coin is tossed repeatedly and tails comes up a larger number of times than is expected, a gambler may incorrectly believe that this means that heads is more likely in future tosses.^[1] Such an event is often referred to as being "due".

So just enjoy your extra winnings I'd say.

I suspect the new charges will be pretty easy to get round if you need to.   I've certainly never had any problem finding algorithms that generate lots of commission but are only marginally profitable.    If I ever start making 'too much' profit I'll just reactivate some more of those to make sure I'm feeding the beast enough commission.

I guess the intense reaction / bad publicity the new charges have generated is more of an emotional one - why should people have to change the way they bet to make sure they generate enough 'protection money' for BF.

Yes, I've been reading the abstractgenerator blog with interest.  The guy who writes it doesn't seem to have a betting background, and his thinking about bots doesn't seem to reference any of the normal betting concepts (overrounds, probabilities, values etc).  Instead his approach seems to be based purely on importing technical analysis techniques from the financial world.  It seems a bit crazy to me frankly, but refreshingly different, and I'm interested to see how he gets on.  (And look forward to stealing his ideas if they work!)

captain - I've put my Poisson code (and a brief explanation) here:

http://electricitymarket.co.uk/poisson.html

captain - I use some Poisson-related algorithms to pick out value bets on the Total Goals markets.  They normally seem to do quite well (although they lost a lot of money this weekend as it happens).

I'm happy to post my PHP class that calculates Poisson probabilities (like the Excel POISSON function) if that would be helpful.

This is more of an annoyance than a horror story, but I got back from work today to find I'd lost £150.  An algorithm that's supposed to stop trading 10 minutes before a football match starts had placed a bet fifty minutes into the game (after the first goal was scored) - and lost (unsurprisingly).

It's a new algorithm that I only put live a couple of weeks ago, and which I obviously should have tested more.  At first glance the code looked safe, because it was checking both the markettime and the betdelay.  But on closer inspection (I now find) it's not looking at the CURRENT markettime and betdelay, it's checking some values the bot stored the very first time it retrieved details for the market.

For this particular match (Ukraine v Poland) the markettime was recorded as 18:45, rather than the correct time of 17:00.  BF may have corrected it subsequently for all I know, but my bot was still using the firsr value it retrieved, which was wrong.  Truly I'm a f**kwit!  Could have been a lot worse though.  I suppose the £150 was worth it to find the bug before it caused worse damage. 

Thanks nadat - good point.  Still, my current bots don't ever get anywhere near the 1000 transactions per hour limit.  Let's suppose they only ever use 200, leaving me 800 free transactions that I can use for slicing up bets into chunks of just over £2 (the precise sum being chosen to maximise the rounding benefit).  Assuming I reserve the technique for bets that I expect to be matched (rather than speculative offers), and I can save 0.4p on each of the 800 slices, that's an extra £3.20 profit per hour just from rounding effects.  If I can manage to do that for 2000 hours each year that's £6k extra profit a year, just from making sure all the odd pennies blackmagic noticed on his statement start accrueing in the right direction.  Sounds well worthwhile coding up the necessary change to my bet execution routine.  Thanks for the tip blackmagic!

Of course to get that £6k extra I'd have to be turning over £1600 per hour in 2000 hours a year, which I'm not.  So probably it's my small turnover that's the limiting factor, not the transaction charges, at least for now.

Ok, I've now tried it out, and I can confirm that peteb was right.  The P&L shown on the website is calculated by rounding totals, but the actual P&L when the market is settled is based on rounding each individual transaction.

Perhaps I'm a bit sad, but I think this is quite interesting.  The effect of slicing a SINGLE bet into £2 fragments to maximise the rounding benefit is only going to be a few percent at most, but on a trade, where your profit might come from scalping a small difference in prices, the effect of slicing up each halfof the trade in the optimum way could easily add 20% to your profit.  I can feel some coding coming on

Thinking about it a tad more, perhaps I shouldn't be dismissing these rounding issues so flippantly.

One of my bots quite regularly lays bets for quite low odds.  For example, let's suppose it wants to lay £100 @ 1.04.  If it places a single bet it risks £4.

But now suppose it places 46 bets at £2.12 (plus a bet of £2.48 to make up the difference).  The payout on each of these £2.12 bets is 8.48p.  So if they're being rounded to the nearest penny individually, that's a saving of 22p.  Not that exciting I know, but it's a 5% reduction that could add up over time. And it seems perfectly legitimate - not in breach of BF's t&cs.

Might test that out tonight, and code it into my bet execution logic if it works.  Would have to watch out for transaction charges though.

Thanks peteb - what you say makes sense.  I think when I investigated this last time I was looking at the P&L figures displayed on the website, not the actual settled figures - I didn't take into account that they might be different.  Doh!

If the rounding is done at the transaction level, I guess the rational thing for a bot to do would be to tweak its bet sizes by a penny or two each time to make sure the rounding goes its way.  Otherwise (assuming £100 bets) it could be throwing away as much as 0.005% on every transaction! 

I remember looking into this before and convincing myself BF do rounding on the total, not per trade.

I was thinking about it because I was trying to understand a scam I've read about on the internet.  Apparently people put lots of 1p lay bets on events with odds of say 1.49, and can't lose money.  They either win 1p or lose 0p.

If BF rounded on a per trade basis, people could do this scam any number of times on a single market.  If they round on the total, they're limited to 1p profit per market.  (This isn't a scam I'd want to do myself you understand - I was just curious.)  Anyway, I came to the conclusion at the time that BF were rounded on the totals. not per trade.

Will look at it again though and see if I can post some evidence of what they actually do.

blackmagic - we may end up having to agree to differ on this one.

You flagged up 2.9868 as one that might round to 3.00 if rounded carelessly.  But if you just divide by the tick size, round and then multiply again it works:

2.9868 / 0.02 = 149.34, rounded to 149

149 * 0.02 = 2.98 (correct)

In summary, my view (which may be wrong of course) is that you don't need to round explicitly to 2 or 3 d.p. because rounding to the nearest tick will do that automatically.  If you try to round to 2 or 3 d.p first and then round to the nearest tick you'll get horrible errors (as blackmagic found it).  But if you keep it simple and just round to the nearest tick everything is very easy.

blackmagic - probably I'm missing something, but seems to me a lot of your rounding problems are self-created from deciding to round to 2 or 3 d.p. unncessarily.  What's wrong with:

public double roundToNearestTick(double value) {
        double tickSize = getTickSize(value);
        return (double)(round(value/tickSize)) * tickSize;
}

Apologies for any language mangling - not a Java programmer.

As per my previous post, surely you just have to divide by the tick size; round to the nearest integer; and then multiply by the tick size.

Taking your example of 2.005, we divide by 0.02 to get 100.25, round to 100, and multiply by 0.02 to get 2.00.

 (I think this only works if the tick sizes are of the form 1/N where N is an integer, but they all are, so that's fine).

Oops - managed to post that half-way through writing it.

Anyway, have a table of increments, and then round to the nearest increment:

tick=round(unroundedprice / increment) * increment

Not o(1), but it increases with the number of different increments (9 for Betfair), not the number of different ticks

Rather than have a table of all tick values, I guess it would be more efficient to have a table of increments i.e. the gap between ticks:

if($price<100) $tick=5;
            if($price<50) $tick=2;
            if($price<30) $tick=1;
            if($price<20) $tick=0.5;
            if($price<10) $tick=0.2;
            if($price<6) $tick=0.1;
            if($price<4) $tick=0.05;
            if($price<3) $tick=0.02;
            if($price<2) $tick=0.01;

dbowes - I can't really agree that what BF are doing is unfair.  Currently I've got a bot that puts offers at the head of the queue with the intention of hedging them against other offers (on other selections) that are already there.  So what it's doing is taking (typically rather small) amounts of money from customers who don't have the knowledge / inclination to faff about doing the maths.  Now BF have decided they'll cut my bot out of the loop - they'll do the maths, and pass most of the money on to the customer (perhaps keeping a bit due to rounding the prices of the hedging bets).  Not great news for me, but hard to argue with on principle (imo).

Clearly there's a trend for Betfair to take over any low-risk or zero-risk opportunities they identify:  last December it was Exchange Games, now it's arbitrage between selections within a market, I imagine next it will probably be arbitrage between different markets (e.g. Total Goals and Overs).  I think anyone who's about to invest time or money in a strategy needs to be thinking about how susceptible it is to being taken over.  And if I had a BF account manager (which I don't) I'd be telling them as little as possible about what I was up to.

I don't think the change will be visible in that way.  Suppose customer A places an order for selection 1, which is then cross-matched against customer B's order for selection 2 and customer C's order for selection 2.  Betfair will report to customer A that they've got a match on selection 1, customer B that they've got a match on selection 2, and customer C that they've got a match on selection 3.  The only way for a customer to know they were cross-matched rather than matched is that the total amount traded on their selection will go up by the size of their bet (not double the size of their bet, as it would for a normal match).

At least I'm assuming that's the way it will work - I've not tested it yet.

Yes, that was the case, but it's all changing today, according to the post BF Customer Services made yesterday:

As announced last week we’ll be making improvements to Betfair’s bet matching logic tomorrow, Monday 14th July. We’ve made a number of announcements to keep customers informed, but wanted to take the opportunity to explain the changes in detail.

What’s changing?
We’ve improved the code that matches bets. As well as matching backs against lays as we’ve always done, we’ll also try to match your bet against bets on other selections in the market. We‘ll give you an improvement over the price you‘ve requested where possible, and we‘ll match you against whichever bets get you the best price.

For example in a tennis market:

Roger Federer is 1.9 to back, 2.1 to lay.
Rafael Nadal is 1.8 to back, 2.0 to lay.

If you try to back Federer at 1.9 or less, previously we would have matched your bet against the customer looking to lay Federer at 1.9. Both bets would have been matched at 1.9, even if you‘d asked for a shorter price. In theory we could do even better than that though: we could match you against the customer trying to back Nadal at 2.0 (backing one player at 2.0 is of course the same as laying the other player at 2.0). Our new bet matching process will see which match gets you the better price. In this case we would get you 2.0 by matching you against the Nadal backer (who is offering a better price than the layer of Federer).

When placing a new bet you will only ever be matched by the new process if doing so gives you a better price than you would otherwise have got. We will match your bet at the best price possible that’s a valid increment on Betfair’s odds ladder, as we explained in our update of 6th June.

Does this only work for 2-runner markets like tennis?
No. The new matching logic works for any number of runners in a market. An example with a 2-runner market is probably easiest to understand, but the principle is the same for markets with 3 runners or more. For example if a football market looked like this:

Spain 2.3 to back, 2.5 to lay
Germany 3.9 to back, 4.0 to lay
The Draw 2.9 to back, 3.0 to lay

Then if you want to back Spain we could match you with customers looking to back Germany and the Draw at 4.0 and 3.0 respectively, which would result in you being matched at 2.4, a better price than you would have got had we matched you against Spain layers (who are only offering 2.3).

Which markets will this affect?
We’ll introduce the new code on Monday 14th July, but initially matching will be done exactly as before. As explained earlier in the year, introducing best execution across selections wasn’t possible without significant change to the existing code that matches backs and lays, so we will need verify that performance is as expected for the existing matching process before enabling the new functionality. All being well we’ll enable the new code for a small number of markets to ensure that everything is as it should be later on Monday. We’ll announce which markets on Monday. Again if all is well we’ll roll out to a wider range of markets on Tuesday.

We’d expect to match across selections on the same range of markets as we currently do:

Match Odds in Basketball, Boxing, Cricket, Ice Hockey, Rugby League, Rugby Union, Snooker, Tennis and Volleyball, Greyhounds win markets, Darts match odds, correct score and handicaps and Soccer match odds, HT/FT, correct score and unders/overs.

Horse racing will not be covered for now, due to the possibility of non-runners, and the new process isn’t applicable to markets where runners can be added (for example “Next manager” markets), where runners listed might not take part (e.g. First Goalscorer) or where the runners in a particular “market” are treated independently (e.g. Accumulators).

What about bets placed in error?
We’re aware of a concern that this change might make it more likely that customers would match bets placed in error, for example asking for 1.2 when you really wanted to back at 2.2. One consequence of the change we’re making is that any bet you place is more likely to get matched - making it easier to get a match is the whole idea. Being realistic though, if you had placed a bet in error like that in the past, in the vast majority of cases you would have been matched (against lays on that selection). There’ll now be far, far more circumstances where you would have been matched anyway , but instead you’ll now get a better price, than situations where your bet would have been unmatched and you might have had the chance to cancel. On average we would expect customers who place bets in error to be better off as a result.

On a related point, we’d also expect this change to make it more difficult for people who place “trap bets” to get matched (a trap bet is an offer that is only likely to be matched if another customer places a bet in error). While putting up “trap bets” is against Betfair’s terms and conditions and we close the accounts of persistent offenders, on an exchange where any customer can ask for any price it’s difficult to eradicate this practice. In most instances where a trap bet is the best price available on a selection, customers will in future be matched at better prices against bets on other selections rather than matching the trap bet.

How will the change affect liquidity?
We would expect the change to be beneficial to liquidity. Obviously if we have opposing customer bets in the system that could be matched, whether on the same selection or across different selections, the best thing for liquidity is to match them.