Member For: 2 months, 2 weeks
Posts: 26
Yeah, time is always a problem. As I said, I've not had a lot of experience with Java applets; I wasn't aware of the port-must-be-on-the-same-server restriction. However, the Java applet security restrictions are there to protect the client, not the server. It shouldn't be too difficult to get the browser to run a modified JAR. Have a look at this, it might give you some ideas:
http://www.csnc.ch/static/download/misc/ObjectInspectorV1.0.pdf
Birchy, I found your Botchy posts on criticalsecurity.net. I've not had much to do with Java applets, but the first thing I'd do is try to enable the log4j logging they've left in there. You'd need to modify the log4j.properties file in pokerclient.jar to set the root category to DEBUG. I've not used log4j with an applet, so I don't know if or where it's able to write a log file; probably outputs to the console, though. After changing the properties in the JAR, you'll find out whether or not you can use a modified JAR on your HDD, or whether they check its hash or something. The logs might give you some ideas or they might be completely useless. Don't know, but it'd be my first step.
By the way, Birchy, if you're still tinkering with Betfair's Poker client and ReadProcessMemory, etc., you might want to have a look at this analysis of Skype:
http://www.blackhat.com/presentations/bh-europe-06/bh-eu-06-biondi/bh-eu-06-biondi-up.pdf
It covers a number of things that can be done to obfuscate an EXE. I found it pretty interesting.
This should get you started in Java:
http://parthian-shot.blogspot.com/2007/09/html-screen-scraping-easy-way.html
If you don't really want to use an HTML parsing library (what'll you save, a few hundred KB?) get a string from the connection's input stream - everything up to that point is 'native'.
If you change your mind and decide to use someone else's parsing code, there are plenty of options:
http://java-source.net/open-source/html-parsers
And, if you need cookie handling:
http://blogs.sun.com/CoreJavaTechTips/entry/cookie_handling_in_java_se
The usual answer to questions regarding threading is "if you have to ask, you should not be doing it." It's a somewhat smart-arse response, but it's not inappropriate. You will have to read the manuals; it's not safe to assume anything. Programming with multiple threads is hard.
You're not experimenting with multiple threads, are you?
Does it really depend upon your chosen OS? MySQL and SQLite are both open source, cross platform, documented and well tested. However, one is a client-server DBMS and the other is a 500 KB library. I'd suggest it depends upon your application's requirements.
MySQL was overkill for me. You might want to have a look at SQLite [http://www.sqlite.org/]. Afraid I don't know much about scraping with PHP, but Fred might be able to help you there.
Also, in what language have you been doing your API development?
As far as tools go, this is one of the best:
http://www.fiddler2.com/fiddler2/
If your pings take 30-40 ms and your web service call takes 200 ms, there is something seriously wrong the way you're using the web service. Forget about looking for optimisations; there is a gross error or oversight somewhere. I've not used Axis (last time I fiddled with Java web services, I used jax-ws, wsimport, etc.), but are you sure you're not using some dynamic proxy? Perhaps you should post the code for your connection to the web service and subsequent pricing call. Maybe someone with Axis experience might be able to spot something.
What's your ping time for bdp.betfair.com? For me, it's pretty close to what a gMPC call takes. If it takes 200 ms for you to ping the developer's site, there's probably not much you can do to improve things.
You're having lunch? That's a relief; I thought you might have been touche-ing yourself!
123, I have no idea how you should modify it, as HttpRequestHelper is not a class in the .NET framework - it's some other coder's creation. However, if it was working for you before Betfair changed their site, it must be managing the cookies post-login. The basic gist of what you need to do is request "http://www.betfair.com" first, to collect some cookies before performing the login request. If you don't know how cookies are handled in .NET, read this: http://menet.name/blog/christophe/archive/2008/05/29/sending-receiving-cookies-with-httpwebrequest-httpwebresponse.aspx
When Fred closed his forum, I deleted the C#, Ruby and Python examples I'd made available on Google Code. I've undeleted them; they are here:
http://code.google.com/p/betfairclient/
http://code.google.com/p/betfairrubyclient/
http://code.google.com/p/betfairpythonclient/
Yes, that's my code. I'm pleased you found it useful. It was something I put together out of curiosity - it was the first thing I wrote in Python. Actually, it's the only thing I've written in Python. My brief look at Python and Ruby convinced me that duck-typing is not my thing; there are too many runtime errors that I'm used to fixing at compile time.
Sounds like that last sentence could have been a little shorter: "She sadly had no idea".
I guess it depends upon the way the VB and MFC HTTP implementations deal with cookies. You can make Firefox fail by clearing all of the Betfair cookies before clicking the login button.
Fixed it. My framework wasn't working because I was calling GetRequestStream - and writing the username and password - before I set the request's CookieContainer, so the request didn't include any cookies. Obvious, really. It looks like the website's only significant change is that "LoginAction.do" requires some of the cookies from the URLs mentioned in my previous post.
I've done some testing with Fiddler and I now have a simple, working .NET harness, but I've not yet managed to get my framework working. Fortunately, it doesn't appear to have anything to with the speed-trap bits. The harness submits requests for "http://www.betfair.com/", ".../Header.do?target=portal", ".../Frameset.do", ".../NavBar.do?productName=portal", ".../PortalHomePage.do" - and collects the cookies - before submitting the "LoginAction.do" request. This seems to work every time in my harness, but does not work in my framework.
I'm not convinced it's the speed-trap bits that are causing the problem, either, but I haven't ruled it out. I'll do some more investigating today and will post what I find.
Yes, http://www.betfair.comnull is the redirection 'Location' in LoginAction.do's response. I wasn't using it; I'd hard-wired the LoginView.do URL in my code and was LoginView.do that was returning the abovementioned message.
Anyone else here scraping the Betfair website? If so, did your logins stop working, late on Tuesday evening? My scraping code can no longer login, receiving a 'You are not authorised to perform this action' message in the HTML response. A cookie I've not noticed before seems to be generated when the response to SpeedTrapScript.do (as in http://speed-trap.com) is processed by the browser. There's a massive amount of somewhat arcane JavaScript in that response, and if the cookie is what's preventing the login, it's looking like I'll have to sign up for the paid API.
