Remove this ad
avatar

inksmithy

Botting Guru

Posts: 184 Member Since: June 1, 2008

Lead

January 19, 2009 18:13:08

Tags : :

Reason I ask is this: I have just been given a subscription to the full betfair API as a gift. I have a couple of ideas on how best to take advantage of it, but nothing really specific. I don't want to do any manual trading (I suck at it) but I'm fairly happy with the idea of doing automated trading to set rules, ie:

if (currentPrice == blah)
{
    place backbet(currentPrice);
    place layBet(currentPrice - blah);
}

Do you blokes have any other ideas? Not looking for systems as such, more looking for inspiration really.

Oh and yes, my missus is pretty cool. I'm speechless really.

Quote    Reply   
Remove this ad
Remove this ad
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#1 [url]

January 19, 2009 18:59:49

Keep doing whatever it was you were doing to inspire such a gift :)

Build a bot that will automatically discover your favourite markets andrecord them in order to provide you with lots of test data.

Take your best performing bot and mutate it 10 different ways?

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#2 [url]

January 19, 2009 19:37:17

Build a bot that will automatically discover your favourite marketsandrecord them in order to provide you with lots of test data.

Hmm, strong in the force you are to come up with such suggestions. Thats a damn good idea.

At the moment, my main bot just looks at the Market Odds markets in football. At the moment I'm thinking along the lines of looking at the over/unders markets as well. I think it will take (yet another) complete rewrite of the program, but even still....its intruiging.

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#3 [url]

January 19, 2009 21:26:14

Build a bot that will automatically discover your favourite markets and record them in order to provide you with lots of test data

-fred77


We have discussed this several times before and generally come to the conclusion that while running our bots (which ultimately failed), we saved NO data whatsoever. Data mining is usually the last thing on my mind when bot testing, yet it *should* be a part of our standard libraries...

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

bennean

rookie botter

Posts: 40 Member Since:May 14, 2008

#4 [url]

January 19, 2009 21:44:08

S' funny you should mention it. My bot saves tons of data to a database(e.g. every price every second, for every UK horse race), but am stillworking on a way to analyze it all. I'm currently using a relationaldata model (happy to publish the model), but think I need a datawarehouse schema (because of the volume of data)...but haven't a clue what the fact tables are, letalone the dimensions and measures. All of the DW examples I've seen arealong the lines of your more typical products & sales...

Are there any BI gurus out there? It's got me stumped at the moment! Ithas to be akin to trend analysis in financial markets (but look at themess they got into!)...

Quote    Reply   
avatar

myrddin

bot addict

Posts: 56 Member Since:June 2, 2008

#5 [url]

January 20, 2009 00:44:52

I think I might be tempted to dabble in a small-stakes trading bot,especially if my ping times were reasonable - something that's not really feasible with the free API.

Also I would extend my existing bot to work on all current markets instead of just the next race due off, and up the rate from 1/sec to whatever I could pull - probably run it alongside the existing one to see if there is a significant difference in returns.

Storing all-you-can-eat data would be a priority as well, although as bennean says, even at 1/sec the tables get very big very quickly. I must say that I think that facts/dimensions data warehousing malarkey is a pile of pooh; a properly normalized and well-indexed relational database is the right way to do it - anything else is a bodge for data which is hard to put into a reasonable number of tables, or is done by people jumping on the latest bandwaggon without thinking things through. If you have a hundred management-types wanting instant access to agregated data then, yes, it has a place - if you are a one man band running complicated analysis algorithms I think it's better to keep the data clean and properly organized.

Quote    Reply   
avatar

bennean

rookie botter

Posts: 40 Member Since:May 14, 2008

#6 [url]

January 20, 2009 09:19:20

I know what you mean, there is a lot of hoo-ha about BI, trouble is, I'm trying to use an OLAP tool to analyze my data (ad-hoc slice and dice, trend analysis etc.) and it has to be told what the dimensions and measures are...I'm new to OLAP but I'll work it out! My rationale is that once data is in a cube, it's a lot easier to play with. But I guess vanilla SQL would do the trick if you know what you're looking for (but I don't - yet!)

Quote    Reply   
avatar

peteb

rookie botter

Posts: 44 Member Since:May 14, 2008

#7 [url]

January 20, 2009 10:25:52

Definitely record lots of test data - and then divide it into different sets - as randomly as you can.
Then you can develop a bot against one set, and then test against the others before launch, to make sure you haven't just optimised your bot for the first set of data.
Build a simulator if you don't already have one, that can simulate when / whether your bets are really likely to get matched, and how much.

As far as volumes go, I only record prices when they change - reduces volumes enormously. I put a "last refresh interval" on the record - this is the time since you last had an unchanged record, so if your refreshes are working normally, it will be short, and let you know you captured the price change when it happened. If you lose connection for 5 minutes, then it will be long, and you can't say when the price change happened.

Are you under any pressure to make back the API costs???

Quote    Reply   
avatar

peteb

rookie botter

Posts: 44 Member Since:May 14, 2008

#8 [url]

January 20, 2009 11:29:49

Oh and the other thing you can do is diversify, as you'll have the bandwidth to do so.
In generally, running two or three moderately successful strategies at once will give better results than running any one of them - as long as the types of conditions that cause each of them to suffer are different.

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#9 [url]

January 20, 2009 13:28:38

Are you under any pressure to make back the API costs???

None at all, except for what pressure I put myself under.

So far the ideas coming up are to build a data mining app and to sort myself out with a system which will allow me to diversify the markets I'm monitoring, as well as possibly implementing a new strategy or two.

To be honest, I hadn't thought about setting myself up with lots of test data, but now that it is mentioned, its not a bad idea at all. I have been toying with the idea that certain prices are 'stickier' than others - for example 1.05 (in football) can take a while to break through, as can 1.1, but 1.13 just blows past as if it never existed, as does 1.19. If I can come up with something which finds out whether thats true or not, and if true, which prices have those properties, there could be an edge in there. Certainly something worth looking at anyway, and once the data is in, the problem becomes how to analyse it rather than trying to figure out how to store it.

The simulator idea is a good one - the way I generally write a bot is to build a simulator mode into it, which factors in delay times and price changes after the bet has been placed but before it has been matched. Once I have everything running smoothly with that, I'll start work on the live mode. It seems to work so far, particularly because it will make the bot go through the entire process, showing up bugs and problems on the way.

Alright, so far I think my priorities are as follows:

  • bodge up my current bots to accept more markets under their current rules;
  • Set up some sort of db for prices;
  • possibly combine the database app with a simulator which will allow me to test different strategies - I can feel a framework coming on.
  • Identify new strategies.
Right then. If you guys have any other ideas, weigh in!

cheers,

Alan

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

myrddin

bot addict

Posts: 56 Member Since:June 2, 2008

#10 [url]

January 20, 2009 17:56:26

A quick heads-up: there is an article on  data warehousing and visualizing large data sets on slashdot today - small world, innit?

Quote    Reply   
Remove this ad
avatar

denp

bot addict

Posts: 65 Member Since:June 14, 2008

#11 [url]

January 21, 2009 11:12:06

Ithas to be akin to trend analysis in financial markets (but look at themess they got into!)...

-bennean

bennean - some of the trend followers have done pretty well - here's a presentation from Winton who earned 18% last year and average 19%... but they have got a team of 90 PhDs working on it!

18% a year.  Hmmm.  0.05% per day.  :-(

Quote    Reply   
avatar

peteb

rookie botter

Posts: 44 Member Since:May 14, 2008

#12 [url]

January 21, 2009 12:29:47

That still helps for setting targets - so with a £1000 bank I'd love to make £10 per day, but if I can make 50p a day (and compound) I'm already out-performing figures a hedge fund is proud of!
At 5% risked, that's one tick on one market, and I'm done!
(still I'd want to do a bit better than that for the work I've put in so far - I don't have a PhD - but that's not going to stop me trying!)

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#14 [url]

January 22, 2009 07:20:49

May i suggest that you don't actually build a "simulator" as such? You're much better off building your "live" bot so that you can switch it between simulation and live mode. The basic theory is that you'll eliminate any worries about whether or not your live copy of the strategy is the same as your simulation one, plus you'll only have to write the code once. By far the easiest way is to save the "raw" response xml (or html if scraping) and then feed it straight into your functions as though it came down the wire...




www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#15 [url]

January 22, 2009 07:34:39

You know I hadn't even thought about doing it that way - thats a damn good idea.

I'm in the process at the moment of recoding my main bot to handle a few more markets. The way I had written it was pretty silly on the whole - looking at it six months after I had written the code, it looks naive, if anything. I reckon a couple of weeks will see it sorted so it can deal with multiple markets, then I can start in on some new projects to properly take advantage of it.

I've bodged the main bot up so it can handle the over/unders markets as well as the match odds, which was just a matter of creating a new ArrayOfRunnerPrices object with dummy data in it to simulate a third runner, because you guessed it - I hadn't allowed for any more or less than three runners in a market.

Still, its working for the moment, which gives me a bit of breathing space.

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

peteb

rookie botter

Posts: 44 Member Since:May 14, 2008

#16 [url]

January 22, 2009 09:55:50

Why are hedge fund called hedge funds btw? Surely they're gambling funds?


-austinpodhorzer


I think it's just historical - pension funds and mutual funds are "long only" i.e. buy-and-hold effectively, and have to ride the ups and downs of the market as a whole. Hedge funds are allowed to short sell, so in their early days you tended to get 80:20 funds with 80% long holdings and 20% short - so if you think a stock is going to outperform its sector, you can buy the stock, and short-sell another leader of the sector, so that you should be hedged against moves in the sector and only exposed to the movement of the stock relative to the sector.
Of course since then they do whatever they can to grow the fund as aggressively as possible - e.g. if you aren't still desperately buying at the end of a bubble, then your fund underperforms everyone else, and investors pull out and you don't get your bonus -but if you are still buiying then you lose everyone's money again in the crash. So yes gambling fund!

Quote    Reply   
avatar

peteb

rookie botter

Posts: 44 Member Since:May 14, 2008

#17 [url]

January 22, 2009 10:02:32

May i suggest that you don't actually build a "simulator" as such? You're much better off building your "live" bot so that you can switch it between simulation and live mode. The basic theory is that you'll eliminate any worries about whether or not your live copy of the strategy is the same as your simulation one, plus you'll only have to write the code once. By far the easiest way is to save the "raw" response xml (or html if scraping) and then feed it straight into your functions as though it came down the wire...

-birchy

Agreed - that's kind of what I meant - definitely don't want two sets of bot code, but you need to be able to run bot code in simulation mode. Need to make the switch idiot-proof though, and make sure everything responds to it - I had a problem the other day where in simulation mode it was still placing live bets, but then simulating what happened to them - oooops!

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#18 [url]

January 25, 2009 11:52:15

Well what I was thing was to utilise the computers I have available to me. At the moment I have a linux machine on all the time (fourteen days uptime at the moment - not much really, but not too shabby)  which I could use to for gathering a database full of data. Looking at the football matches for example, a GetAllMarkets call with a from and to date of three days will net me a fair few markets to quiz. If I then set it on a fairly slow call rate - say one or two per second - I'll still be gathering enough data to be able to spot changes in the prices over the pre-match period. Granted it won't catch the full volatility of the pre-match trading, but I'll still get some fairly broad statistical curves to play with.

Once the market goes in play though, I think I would have to up the rate on that machine to do a price call every two seconds or so for every in-play market which interests me. The reason for the fairly slow refresh time is because my actual betting bot will be making several calls per second as well, and I need to think about how many calls my account is making in total.

So if I set the surveyor bot to dump the raw XML into a database - using peteb's idea of the refresh time as an indicator of whether to store the XML or not, I could well end up with a useful database of prices to both analyse and model for statistical advantages.

Again, if you think I have the wrong idea there, let me know, please! What I'm doing at the moment is trying to train myself to think outside the limits of the free API - meaning the calls which I hadn't had access to while using it. I'm also struggling with concurrency in Java - I have a thread on the bdp forum which addresses that, if you have any ideas on that basis, please let me know.

On a slightly different note, I'm rewriting my main bot to be a bit (LOT) more efficient and I'm posting various classes onto the bdp forum as I write them, if anyone isn't a member there and they want something, post something here and I'll upload them to the code forum. Again, feel free to criticise them - any improvement is good.

Thanks guys.

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

austinpodhorzer

bot addict

Posts: 131 Member Since:May 11, 2008

#19 [url]

January 25, 2009 22:59:57

On the threading ...

Basically your soap tookit should be dealing with the thread management for the async calls.
ie. when you call loginAsync(request, asyncHandler) the soap toolkit will eventually call the  AsyncHandler.handleResponse method on one of it's own threads.

If you're seeing very high thread rumbers it sounds like the soap toolkit could be creating a new thread temporarily for each callback, which would be a bit wasteful. It might be that it can be configured to use a ThreaPool to avoid that, but to be honest it's probably not a big deal.

And you don't have to wrap you entire handleResponse code in SwingUtilities.invokeLater(...)
You only need to do this for bits of code that are making Swing related calls.

Also, you do NOT need to EVER use Thread.yield(). The vm handles all the switching and scheduling fine on it's own.

Quote    Reply   
avatar

austinpodhorzer

bot addict

Posts: 131 Member Since:May 11, 2008

#20 [url]

January 26, 2009 11:36:31

Hi inksmithy,

Just been through some of my library code to see how I go about it.

Instead of using the asynch methods provided by the Service stubs and relying on the soap toolkit to manage it's own threads I use the synchronous methods and manage the threads myself with a simple thread pool implementation.

Check out the class below. I create a ThreadPool in the main app when it starts up and then when I want to make a call to the API I call addJob(Runnable). Put all the api code in the run method of the runnable but using the synchronous methods.
When procesing the response wrap the final bit of code that updates the Swing GUI into the SwingThread using invokeLater.

Also, it's worth re-using the service/port stubs rather than constructing new ones for every call. (Certainly when using axis2 anyway which is what i use)

public class ThreadPool {

  private Set<Thread> threads = new HashSet();
  private List<Runnable> jobs = new LinkedList();

  public ThreadPool(String name, int threadCount) {
    for(int i = 0; i < threadCount; i++) {
      Thread t = new WorkerThread("ThreadPool-"+name+"["+i+"]");
      threads.add(t);
      t.start();
    }
  }

  public void addJob(Runnable job) {
    synchronized(jobs) {
      jobs.add(job);
      jobs.notify();
    }
  }

  class WorkerThread extends Thread {

    WorkerThread(String name) {
      super(name);
    }

    public void run() {

      while(true) {
        Runnable job = null;
        synchronized(jobs) {
          if(jobs.size() > 0) {
            job = jobs.remove(0);
          }
          else {
            try {
              jobs.wait();
            }
            catch(InterruptedException ie) {}
          }
        }
        if(job != null) job.run();
      }
    }
  }
}

Quote    Reply   
Remove this ad
Add Reply

Quick Reply

bbcode help