Remove this ad
avatar

birchy

Betfair Elite

Posts: 591 Member Since: May 11, 2008

Lead

April 28, 2010 12:47:36

Tags : :

OK, i'm a little out of touch with the whole betfair malarkey. As some of you will remember, a few years ago i was a die hard web scraper and it's good to see that my old VB6 code is still very popular in the Code Snippets section. I've just returned to sports botting after a break of around 18 months. Long story short, i'm no longer using Windows so this is my first crack at a Linux/Python bot. Have already got an API library that i coded up some time ago (but never used), but i'm just wondering what the state of play is regarding scraping and data charges?

What sort of refresh speeds can i get away with from scraping? I used to refresh at 25+ times a second but i guess those days are long gone unless you're willing to pay data charges. Have read betfair's data charges thing but can't make much sense of it. Can anyone that is still scraping the site give me some real-life figures please?

Quote    Reply   
Remove this ad
Remove this ad
avatar

fmaf

rookie botter

Posts: 3 Member Since:April 29, 2010

#1 [url]

April 29, 2010 01:47:58


Hello, I saw the code you have there on your webpage and it seemed very nice, congratulations.
It's a good question, I have been guided by information that is in http://bdp.betfair.com/index.php?option=com_content&task=view&id=36&Itemid=64
I have used these sample rates:

  • get_events; get_active_events => infinite sample rate
  • get_market => 12 seconds
  • get_maket_prices_compressed => 1 second
  • get_market_traded_volume => 1 second
I would like to hear more people here in the forum on the subject.

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#2 [url]

April 29, 2010 09:11:10

There's an interesting thread at bdp at moment to do with data charges and timing, I'm presuming it applies to scraping too http://forum.bdp.betfair.com/showthread.php?t=663

The scraper bot I maintain for a friend started getting data charges at 9 price requests per second, so I reduced it to 8. Looking at that bdp thread I reckon the charges were because of my ignorance of wall clock seconds. With scraping you should be able to retrieve prices up to 10 times per betfair wall clock second. Play it safe with 8 or 9 if you can't get the timing accurate. Once you start generating mountains of commission then you can consider increasing it a little bit.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#3 [url]

April 29, 2010 18:51:35

Hello, I saw the code you have there on your webpage and it seemed very nice, congratulations.

-fmaf

Thank you. It needs some heavy duty testing because there *will* be some bugs. I would like some feedback and bug fixes and suggestions, etc from people who have used the library so that i can improve it...but so far it's been downloaded over 1000 times and i've had no feedback what-so-ever.


Talk about tempting fate, i received an email today:

...Your data requests on 4/28/2010 would have incurred a charge of 2.72 GBP.  We are NOT applying the charge to your account for this incident. We want you to avoid being charged in future....

-betfair

First ever offence and i'm pretty sure it was due to me calling getMUBets with no market id inside a loop to which i had forgotten to add sleep(1). Was using the free API, so i guess most of those responses were "throttle exceeded" type errors. Only ran it for a minute or so while testing a trading function, so it just shows how easy it is to run up a big bill...


I think the biggest problem is allowing for internet lag times as it's such a variable rate?

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#4 [url]

May 8, 2010 20:46:04

Hmmmm, i've hit a problem with the old cross-matching logic...

When attempting to put some bets at the front of the queue, my bets are actually ending up in the 2nd or 3rd columns on the website. After checking my calculations many times, i discovered that the API was giving me the "wrong" prices. They are actually the RIGHT prices (i.e. actual bets), whereas the website is displaying the "virtual" prices calculated by the cross-matching algorithm. As you can imagine, it's a pain in the arse when you're trying to tickle some manual punters and your "column 1" bet actually ends up being the 2nd or 3rd best odds!

Is there a remedy for this or is it another +1 for web scraping?

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

talkbet

rookie botter

Posts: 10 Member Since:May 5, 2010

#6 [url]

May 10, 2010 10:43:43

Had to go through that pain myself, its an utter ripoff that Betfair can place their bots at the front of the queue and not tell you the genuine prices from the API (ie different to the prices the users see on the site). And we pay £200 a month for it ? FFS!
Got the logic going in the end but it's a right pain seeing if its working or not.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#7 [url]

May 11, 2010 11:51:40

Haven't attempted to code anything up yet just in case Google points me to something ready-made...

Seems a bit off not having the virtual bets in the API. I would of expected at least an option of which one we want the API to return, and preferably both so we could parse which ever one we want.

Was reading the BDP forum and came across THIS thread. Of particular interest is the final post:


You are correct, the additional % difference between the back and lay price is taken by Betfair.

-Mr Stokes


So basically, when we calculate a virtual bet and need to round it to a Betfair increment, Betfair are rounding Back bets DOWN and Lay bets UP and pocketing the difference. I know it's only a few % but i thought Betfair was an exchange, not a bookmaker!

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

denp

bot addict

Posts: 65 Member Since:June 14, 2008

#8 [url]

May 12, 2010 19:38:43

right - figured that might be happening.  Also note - fractional differences will also mean that even if you have the best back or lay price, many combinations of prices mean that you will never get matched if the virtual bet creates a price that looks the same at 2 decimal places, but is actually better at higher precision... so betfair's virtual bets always get matched first.  As evidence, i've entered the market with, say, £100 on the best lay price, then seen £500 matched at that price and I'm still not matched.  Could be another explanation... any suggestions?

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#9 [url]

May 12, 2010 21:06:12

Ohhhh Den! Kick a man while he's down. ha ha.

I'm kind of regretting having another go at sports bots. It's quite disappointing to see that Betfair are getting more and more greedy and becoming more like a traditional bookmaker. Do they still use that patronising slogan "winners welciome"?

My current strategy requires my bets to be on the front lines, so i'm checking my odds vs backing or laying all the other selections (i.e. calculating the virtual odds) and then placing my bet in relation to that.

If i could work out how to parse the event menus, i think i'd go back to scraping as it seems to be more informative than the API.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#10 [url]

May 12, 2010 22:19:31

I don't know why you just don't build your library to use API and web scraping, it's not like you have to code up lots of different calls. Have the best of both worlds. Have 10 price calls per second with the virtual prices included.

Incidentally, I adjusted my friend's hybrid bot, I modified its timing cycle to make a full 10 calls within each wall clock second as discussed in the BDP thread I mentioned above and apparently there's been no further charges (yet lol)

Quote    Reply   
Remove this ad
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#11 [url]

May 12, 2010 23:02:51

I don't know why you just don't build your library to use API and web scraping

-fred77


Because I'm a stubborn purist and for some reason a hybrid seems unclean. As you can imagine, I'm very unproductive.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#12 [url]

June 21, 2010 16:57:34

Has anyone worked out how to scrape the menu on the betfair site? To be more precise, is there an equivalent to the getAllMarkets API call? I've been playing around with the https://site.sports.betfair.com/Menu.do calls but not having much success. If i can emulate getAllMarkets via scraping then i won't need the API at all.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

ossi

rookie botter

Posts: 16 Member Since:June 12, 2008

#13 [url]

June 23, 2010 07:07:21

birchy,

http://site.sports.betfair.com/Menu.do?timeZone=Europe/London&region=GBR_IRL&locale=en_GB&brand=betfair" will give you the main event categories

from the string "var skeletonString" start splitting by "\|"
seperated by "~" you want to get $1,$id, $parent, $name and $type of event

and for each of these you call

'site.sports.betfair.com/menu/LoadMenuNodesAction.do?sReturnPath=parent.frames["menu"]&method=getMenuEvents&menuNodeId=' . $id .
               '&strArrayName=allSkeletonArray&iParentID='.$parent.'&layerName=allMarketsTreeContainer&strMenuPathArrayName=allPathArray&menuPathLayer=menuParents1&locale=en_GB'

it is much slower then the API call but it does work.

Enjoy
Ossi

Quote    Reply   
avatar

talkbet

rookie botter

Posts: 10 Member Since:May 5, 2010

#14 [url]

June 23, 2010 11:08:19

Sorry, just realised I posted this on another thread....

At work at the moment so not sure if it still exists but I always used to use the Betfair Lite site for stuff like scraping, far easier than the main site.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#15 [url]

June 23, 2010 12:26:28

OSSI: Yeah, i was playing around with similar code but was trying to find an "all in one" request but i guess there isn't one. Although reluctant to mix the API and scraping, it seems to be the best solution. The main benefit of scraping is the market refresh call which returns SO much info, including virtual prices, form figures/jockey names/trainer names/etc, P&L figures, links to jockey silks images, market parameters such as the turnInPlayMarket boolean, etc. To get the same info via the API requires at least 3 requests and even then we have to calculate the virtual prices (which is not easy!) and are missing all the form figures and other useful info. It's little things like this that make betfair hypocrites because they moan about data usage and then give us an inefficient platform. I think i'll email them, suggesting they add an API call that returns the same info as scraping getFullMarketData.

*birchy steps off his soap box*

TALKBET: I did look at the Lite site a few years ago but i seem to recall that it was very limited on info (i.e. none of the detailed info i mention above) and i'm sure i read somewhere that the Lite site is relayed from the main site but only refreshed once a second or something? There was certainly an issue with caching but i forget the details. Could be different nowadays.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#16 [url]

June 23, 2010 21:40:24

I have poked a stick into a possible hornets' nest, will be interesting to see what develops:
http://forum.bdp.betfair.com/showthread.php?t=896
Have also emailed bdp along similar lines. Why on earth the web site returns better structured data than the official API is beyond me, but i guess a hybrid (as Fred suggested) is the only way to get maximum efficiency.
 

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#17 [url]

June 24, 2010 00:23:49

You'll not rouse any hornets nest there, most likely get some agreement judging by vossie's comments
Perhaps you can even get a free API subscription by sharing your scraper/api code

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#18 [url]

June 24, 2010 01:21:17

Vossie doesn't seem to be a BF employee, he (or she) always comes across as one of "us" that just happened to get lumbered with the title of "BDP Team". I'd be surprised if he or she gets any financial reward beyond a free API subscription. I'm not convinced that Vossie would agree though, as he has not actively promoted scraping other than the timeform thing. Regarding the API structure, yeah i did read some comments saying it wasn't as user friendly as it could be...but that was like 2 years ago and nothing has happened.

I really should look into betdaq again, they seem a bit more welcoming...

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

ossi

rookie botter

Posts: 16 Member Since:June 12, 2008

#19 [url]

June 24, 2010 12:27:37

Betdaq better? Not really. A year ago I was running scripts against betdaq and I got a letter from then telling me that my 20-30 hits per second on their API was too much and could I please stop :-)

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#20 [url]

June 24, 2010 20:14:29

I believe Betdaq have a 20x a sec limit but they don't actually hit you with data charges or such like. Besides, even at 20x a sec we'd still have betfair speeds without the £200 month cost. Liquidity is lower, but i don't think that would be a major concern for my particular needs.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
Remove this ad
Add Reply

Quick Reply

bbcode help