Topic: Website Scraping Login Failure

Hi,

Ive had a problem too, although i got the exception "cannot resolve http://www.betfair.comnull". My bot has been logging succesfully since the charges started. I first noticed the problem this morning!

Hadnt had chance to look into it yet.

Anyone had any luck resolving the issue?

Dan

Perhaps this is the problem but im not convinced from that article that this is casuing the login problems.

A couple of friends of mine use MFC and seem to be unaffected. Im using .net, can anyone else successfully login using the HTTPWebRequest object?

Yes, http://www.betfair.comnull is the redirection 'Location' in LoginAction.do's response. I wasn't using it; I'd hard-wired the LoginView.do URL in my code and was LoginView.do that was returning the abovementioned message.

I'm not convinced it's the speed-trap bits that are causing the problem, either, but I haven't ruled it out. I'll do some more investigating today and will post what I find.

I've done some testing with Fiddler and I now have a simple, working .NET harness, but I've not yet managed to get my framework working. Fortunately, it doesn't appear to have anything to with the speed-trap bits. The harness submits requests for "http://www.betfair.com/", ".../Header.do?target=portal", ".../Frameset.do", ".../NavBar.do?productName=portal", ".../PortalHomePage.do" - and collects the cookies - before submitting the "LoginAction.do" request. This seems to work every time in my harness, but does not work in my framework.

Fixed it. My framework wasn't working because I was calling GetRequestStream - and writing the username and password - before I set the request's CookieContainer, so the request didn't include any cookies. Obvious, really. It looks like the website's only significant change is that "LoginAction.do" requires some of the cookies from the URLs mentioned in my previous post.

Good stuff Nadat, will try that tonight, thanks! It would explain why logging in via the website continued to work as normal but our apps didnt.

What is strange is that friends of mine who use MFC have been unaffected. They just post the LoginAction.Do as they always have.

It seems only people who use .net are affected. Can't get my head around this at the moment but if your approach works, thats good enough for me.

I guess it depends upon the way the VB and MFC HTTP implementations deal with cookies. You can make Firefox fail by clearing all of the Betfair cookies before clicking the login button.

A good test Nadat (clearing the cache before actually logging in). I have now got my app working. I just retrieved a response from http://www.betfair.com and passed the cookie back to my LoginAction.Do request and it worked fine.

I think its something to do with the way .Net handles empty (or null) cookie containers when dealing with Betfair. Anyway panic over. Thanks again.

gentlemen,
my code is
string requeststr = "https://www.betfair.com/account/login/LoginAction.do";
string lResponseString = HttpRequestHelper.PostForm(requeststr, "username=" + Program.BetSettings.CurrentUser.UserName
+ "&password=" + Program.BetSettings.CurrentUser.Password);
if (lResponseString.ToLower().Contains("invalid username or password"))
return false;
else
...
How should I modify code in order to deal with this unfortunate cookies?
thanks in advance

123, I have no idea how you should modify it, as HttpRequestHelper is not a class in the .NET framework - it's some other coder's creation. However, if it was working for you before Betfair changed their site, it must be managing the cookies post-login. The basic gist of what you need to do is request "http://www.betfair.com" first, to collect some cookies before performing the login request. If you don't know how cookies are handled in .NET, read this: http://menet.name/blog/christophe/archive/2008/05/29/sending-receiving-cookies-with-httpwebrequest-httpwebresponse.aspx

thanks Nadat - problem solved