myrddin's Blog

Hi cluckhead - If you need technical help with building a bot, you've come to the right place, but as you point out, "system" - type info is something we all prefer to keep close to our chests.

Having said that, my general advice is to go 'live' as early as you can, even if it's just for minimum stakes. The main reason for this is that dummy systems quite often think that they are 'on' at a price, whereas if you place a live bet, especially in a trading-style system such as yours, you will find that someone got there first and snapped it up.

That brings up the second major piece of advice I would give someone building a trading app - speed is of the essence, so do all you can to reduce your latency. If you read back through this forum you will find a few threads covering this in some detail, especially regarding choice of ISP and hosted services.
Not sure how much of your prototype system is in excel, but if it makes heavy use of worksheet functions and data stored in cells you might want to move as much as you can into vba to speed things up.
Never used gruss, but their flash-based website looks a bit amateurish. That doesn't mean their VB.NET stuff will be as bad, but I would suggest benchmarking it for fetching prices and striking bets to make sure it's not a serious bottleneck. Other than that, all I can say is good luck and a Happy New Year.

Yeah, the LWP and HTTP modules I listed in the second post of this thread handle all the gzip/https/cookie-type stuff, and the HTML::TokeParser module is pretty much the low-level tag-handling parser you are planning on writing in Python - like I say, still pretty handraulic, but a lot quicker than raw string functions.

I've never used WWW::Mechanize, but a quick look suggests it is based on LWP::UserAgent, so it would include all those gzip/https/cookie handlers. It (Mechanize) looks pretty useful for navigating sites - thanks for bringing it to my attention.

I can't believe there aren't equivalent libraries floating around for Python, although they do say that the greatest strength of Perl is the number of good-quality modules available through CPAN.

Not sure what you mean, birchy, but the way I deal with the page is:

* Loop over all the div tags, if the class is "race_idx_hdr", get the meeting name

* Until the next "race_idx_hdr", loop though the div tags with class "racecard_link"

* Get the race details and pull the race number ($SportingLifeId) out of the java href with a regexp

* Get the page "http://horses.sportinglife.com/Racecards/0,12495,".$SportingLifeId.",00.html"

* Parse the desired info out of the racecard page.

As far as I can tell, the racecard url always looks like the above, with the only variable being the race number which I call $SportingLifeId, so no need to mess with the actual JavaScript at all.
BTW if you need a specific date for the meetings index, rather than just today, the url is of the form : "http://horses.sportinglife.com/Meetings/0,12496,30-12-2008,00.html", and they keep about a year's worth.

The page works fine on my laptop - Ubuntu 8.04, Firefox 3.0.4, AdBlock Plus and NoScript - So like you say probably a settings thing. Does it happen on any other pages? Have you got any weird settings for javascript/ asp pages in Preferences -> Applications ?
What is the little icon in the bottom right hand corner, next to the NoScript snake? If it's some sort of DOM inspector or debugger, have you tried disabling it?
Well that's me out of ideas - (turn it off and back on again? clear the cache?) If you need to check any settings, either in Preferences or about:config, let me know. I was going to post my entire about:config, but it's probably not a good idea in a public forum

Back in the 1970's there was a weekly publication called the Sporting Chronicle Handicap Book (later taken over by Raceform). I used to get it for the detailed results section, but the back pages were given over to a letters forum on betting systems. Bearing in mind that this was before computers were widely available, most of these 'systems' were fairly rudimentary in nature, but a good proportion were based on the string of form figures. I never saw one which worked then, and haven't found one since.
The basic reason is that raw form figures encode a very small amount of very low-quality information about a horse's ability - almost any other commonly available indicator is better (official handicap mark, speed figures, newspaper ratings, PostMark and especially Timeform rating).
I have seen all sorts of clever methods for manipulating form strings, but the basic rule of 'garbage in, garbage out' holds good - not only is the encoded information low-quality, but it is subsumed into the other measures anyway (OK, straight speed figures are an exception to that). This means that by the time a market forms, any information contained in the form string is fully incorporated into the price - if a manipulation of the string seems to suggest otherwise, it is because the price incorporates additional, probably higher grade, information.
This argument holds true for all the other ability measures as well, of course - just to a lesser degree.

Although most distros can be made to do any job, all tend to have strengths in different areas. For a general purpose desktop machine, Ubuntu seems to have all the momentum at the moment - I'm using it on one old laptop (8.04 lts) and one desktop (8.10 with all the compiz bells and whistles). It's certainly not perfect, but it is easy (as far as installing apps and such goes). I see it as a Windows replacement, with all the focus being on the 'end user' experience.

My other machine runs Slackware, and I basically use it as an all-purpose server. It runs my gateway, firewall, MySql server, apache, CUPS server and printer, NFS server with a couple of big external drives and all my botting stuff. Slackware is ideal for this sort of thing - the focus is on 'under the bonnet' stuff, and the config files are a joy, especially compared to the tangled mess which is the Ubuntu /etc directory. Package management is a bit basic, and some less common stuff may require building from source (although this is incredibly easy compared to Ubuntu).

I used Red Hat for a while in 2001, before they split it up into RHEL and Fedora. That was fine back in the day, but when I got serious with linux in 2005 I tried Fedora and it was a dog - the 'stable' version felt very 'beta'-ish. This may have changed since, but the feeling I get from the web is that it hasn't.

gentoo isn't an operating system, it's a religion. About a year ago there was a load of internecine strife, and it's very existence seemed to be in doubt. It seems to have come through that, but it's reputation as a 'serious' distro has suffered. I haven't actually tried it, so it may be fine, but I don't think custom compiling *everything* is worth the extra effort. (Kernels are a different matter, but you can do that on most distros).

As far as support from forums goes, Ubuntu is pretty crap - they are friendly enough, but I think that is part of the problem. Too many threads asking the same questions and too many 'helpful' newbies replying on topics they don't really understand. I had a problem with the Bluetooth stack in 8.04 which turned out to be quite common - half a dozen threads, twenty different theories and no solution. I found the answer in five minutes on a debian forum because the problem was upstream, but if you have an Ubuntu-specific glitch then you are shafted.

Slackware forums, especially http://www.linuxquestions.org/questions/slackware-14/ , are great. You get replies from people who know what they are talking about, quite often from people involved with the distro (Pat never replies himself, but Alien Bob does, usually in great detail)

64-bit distros seem pretty good for server apps, but they all seem to have a lot of niggly problems on the desktop (hardware support, firefox, flash etc.). It seems like the 32-bit linux desktop situation of five years ago - most things have a fix, but you may have to jump through hoops. (No personal experience here, I have one 64-bit AMD machine, but it's only running 32-bit Ubuntu)

Lol yeah, learning a new programing language is a piece of piss compared to learning emacs!

Birchy, I'm not sure what you mean by 'set it up' - the default install comes with a built-in python mode which should load automatically when you open a file with a '.py' extension or a python shebang line. I had a quick look at it - as I say I don't know any python so I just mucked around with a temperature conversion script I found on the web. Standard autocomplete in emacs (M-/) only works on stuff it finds in open buffers, but python-mode has the kind you want that does imported libraries. Unfortunately, the default key binding is M-TAB, which in gnome will switch to another window. (M-<key> in emacs basically means ALT+<key>, the 'M' comes from META, which was a key on VT-100 keyboards or somesuch) You can get round this by using the full 'M-x python-complete-symbol' command or remapping the key bindings in your .emacs file. This command just autocompletes, but there is another one ( M-x python-describe-symbol) which gives you the required parameters and usage; you would probably want to remap that as well.

A good .emacs file is important to tailor emacs to your needs - mine is heavily perl-centric and made up of bits I found on the web written by people much cleverer than me. Just google for examples by people who code mainly in python.

There seems to be a couple of other 3rd party python modes for emacs which some people rave about, so they might be worth trying. Personally I use cperl-mode rather than the default perl-mode because it has a lot more features, but I know a lot of people hate it - as always, suck it and see is the best policy.

Lynx is not a bad text-based browser, but I found Links a little better and w3m is better than both. (It also has an emacs mode, which is handy for building scraping apps).

I guess you've googled hundreds of comparisons between the various languages over the last couple of weeks, and if you can't find any deal-braking differences between them I suggest you just try them all - write a few simple apps in each and see which one floats your boat.
I was attracted to C because of all the stuff I'd read about it over the years, but when I actually started using it I found it a bit of a grind. Perl was just one of the scripting languages I tried when I moved to linux (php and ruby were the others), and I fell in love with it early on, just through using it. I'm not saying you will feel the same way, but perhaps one of them will just make you think "yes, that's the one". Unless you want to learn a specific language for employment reasons, why not go for the one you actually look forward to coding in?

I think the idea of abstracted HTML/XML handling code is great in theory, the problem is that most real-world websites don't obey the rules. I find I have to write new bits of code for each site, and things are prone to break without warning when the webmaster tweaks the formatting every five minutes (I'm looking at you, Sporting Life results geezer!). Now we all know that sort of stuff should be done in the css rather than the html, but 'should be' and 'is' are very different things. Half my web parsing code is made up of sanity checks to make sure the data I'm getting out is what I'm expecting - when it fails I need to re-code.

There are libraries for perl (and I assume all the other languages) which are fairly low-level and just parse the html tokens from a page. This still makes for labour-intensive code, but it's a lot better than using string-handling functions on the raw text.

As for regexes - all I can say is try them. All languages have them nowadays (although obviously perls is still the best:-). I read pages about them without really 'getting' them, but once you use them a few times you realize how much more flexible, robust and powerful they are than simple string-handling functions. I used to use Access VBA a lot ten years ago, and parsing data from web pages and VAX or Stratus report files was always a headache using string functions - with regexes it would have been a tiny fraction of the work.

Oh, and all IDEs suck - emacs ftw :-)

Do the methods you were using in VB6 not work in gambas, or are you looking for something less handraulic?

The standard way to create GUIs in perl is to use the Tk toolkit, although I couldn't say, hand on heart, that it is either 'good' or 'reasonable' - 'adequate' would be as far as I would go, and certainly nowhere near as easy as VB5 (never tried VB6). Like you, I've never really 'got' Python, but I understand that GUIs and Widgets are pretty straightforward, certainly better than perl.
Eclipse seems to be the default IDE on Ubuntu, and is good for pretty much any language - definitely worth learning if you are going to stick with linux.
Personally I use emacs in a terminal rather than an IDE. It does all that syntax highlighting, debugging and autocomplete nonsense, as well as running a MySQL client and web browser. It's a steep learning curve though, and bugger all use if you want a GUI.
I'm happier with text based interfaces (ncurses if I'm feeling flash), mainly because they are simpler to use over a network and you don't need to run XWindows on your bot machine, but I realise this sort of bare bones approach isn't everyone's cup of tea.

Denp: like I say, I don't really know a lot about Python, but from what I hear it would probably be something very practical, efficient and well-engineered - perhaps one of those Bahco adjustable spanners, or a good German pair of pliers?

perl is pretty easy for scraping, especially for 'old school' html websites (although it's not so hot for web 2.0 - type sites, but perhaps these are trickier in most languages).
I use the 'standard' perl libraries for this, most of which have been around for donkeys years. Nearly all of them ship with a standard perl install, and the ones that don't are available on CPAN. They tick all five of your boxes, so I thought I'd throw them up here.

Internet related libraries I use (for scraping AND betfair SOAP stuff) are:

LWP::UserAgent
LWP::Debug
LWP::Simple
HTTP::Request
HTTP::Cookies
XML::Simple
XML::XPath
HTML::TokeParser

Throw in the generic database library DBI and that is pretty much my entire 'toolkit'. Something along the lines of 'Gaffa Tape, WD40, Hammer (optional)', but it works for me.

Yep, that's what I got (eventually)

Original bet is a stake of S at price P, and required margin is m
So S = £3.00 P = 1.5 m = 0.05

New Price, P' = P/(1+m) = 1.5/(1.05) = 1.4285714

P' isn't a valid price, go to the next shorter valid price, so P" = 1.42

Revised Stake to Lay, S" = SP/P" = (£3.00 x 1.5)/1.42 = £3.169014 = £3.17
(rounded to the nearest penny), giving a revised margin of:

Revised margin, m" = (S"-S)/S = (£3.17 - £3.00)/ £3.00 = 0.0567

Arrgh - sorry Peteb, I was working on gross returns, not net - obviously if the horse wins, you make a profit equal to the difference in stakes (which is what we want). I'll extract my head from my arse and try again:

Original bet is a stake of S at price P, and required margin is m

New Price, P' = P/(1+m)
Stake to Lay, S' = (1+m)S

(which is what everyone else was saying anyway)

Obviously, if P' isn't a valid price, go to the next shorter valid price, say P". In this case, the stake to lay will become:

Revised Stake to Lay, S" = SP/P" , rounded to the nearest penny, giving a revised margin of:
Revised margin, m" = (S"-S)/S , or equivalently m" = (P-P")/P"

Sorry to have added confusion with my earlier posts, too many years working with bookmakers' prices:-)

Peteb, I think the confusion arises because your method just breaks even if the horse wins - Inksmithy was specifically after a formula to give an equal green outcome regardless of result.