Remove this ad
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#21 [url]

December 5, 2008 20:33:32

emacs?  world of pain, world of pain...every single time I have tried to get a grip on emacs I've fallen by the wayside, nursing bleeding fingers and wondering why text editing has to be made such a challenge.

That said, it is yet another thing on my list of things I need to learn. I think the best way to do it is like any other awkward project. Just use it exclusively until it becomes second nature to you. The learning curve might be steep, but with emacs it is probably worthwhile. I've tried to use vi and vim a couple of times, but that ended up being worse than emacs.

Try being forced to use visudo - as a fairly new linux user who didn't know about setting variables and exporting them to PATH - on a system which has just lost X and you hadn't got around to setting yourself up with the su command. I ended up reinstalling because I didn't even have the internet and thus, no lynx browser (which has saved my arse a couple of times - if you haven't installed it yet, do it. Real lifesaver at times).

Matter of interest, for python editors, have you had a look at Eric?

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
Remove this ad
avatar

myrddin

bot addict

Posts: 56 Member Since:June 2, 2008

#22 [url]

December 6, 2008 01:52:08

Lol yeah, learning a new programing language is a piece of piss compared to learning emacs!

Birchy, I'm not sure what you mean by 'set it up' - the default install comes with a built-in python mode which should load automatically when you open a file with a '.py' extension or a python shebang line. I had a quick look at it - as I say I don't know any python so I just mucked around with a temperature conversion script I found on the web. Standard autocomplete in emacs (M-/) only works on stuff it finds in open buffers, but python-mode has the kind you want that does imported libraries. Unfortunately, the default key binding is M-TAB, which in gnome will switch to another window. (M-<key> in emacs basically means ALT+<key>, the 'M' comes from META, which was a key on VT-100 keyboards or somesuch) You can get round this by using the full 'M-x python-complete-symbol' command or remapping the key bindings in your .emacs file. This command just autocompletes, but there is another one ( M-x python-describe-symbol) which gives you the required parameters and usage; you would probably want to remap that as well.

A good .emacs file is important to tailor emacs to your needs - mine is heavily perl-centric and made up of bits I found on the web written by people much cleverer than me. Just google for examples by people who code mainly in python.

There seems to be a couple of other 3rd party python modes for emacs which some people rave about, so they might be worth trying. Personally I use cperl-mode rather than the default perl-mode because it has a lot more features, but I know a lot of people hate it - as always, suck it and see is the best policy.

Lynx is not a bad text-based browser, but I found Links a little better and w3m is better than both. (It also has an emacs mode, which is handy for building scraping apps).

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#23 [url]

December 15, 2008 17:10:32

How's it going with the python birchy? I've been giving it a go as it seems to be a good compromise between ease of programming and speed of execution.

Today I've been experimenting using the regex module to extract values from XML. Not really something that is supposed to be done but it's helping me grasp the language anyways. Using a like for like comparison with PHP (actually using the same regex pattern) and a similar amount of error checking, 1 million runs took 30 seconds with python and 48 seconds with PHP.

From the performance point of view I shouldn't really need to switch, it's just that as my number of bots is increasing my PHP bottlenecks and memory leaks are becoming an issue. I could rewrite all my stuff to be more streamlined but making a language switch for my worst offending bots and any future bots might actually be better. No doubt I'll have to sample quite a few libraries / modules before I find out if the grass is greener.

One thing I haven't fully investigated yet is if python code can be compiled to native code.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#24 [url]

December 15, 2008 17:32:39

Sounds like you've got further than me Fred. I'm not really "botting" at the moment, so i'm just dicking about (hence my recent stint of playing with various Linuxs' and Grub installs). I've printed a hard copy of "Dive into Python" (last time i was working night shifts)...and i'm back on nights tonight, so i'm hoping to get some serious studying done (assuming nobody breaks their machines).

I believe there are several ways of compiling Python code, one that springs to mind is py2exe or something like that, but i believe there are others. What i like about Python is that the modules/libraries are written in C for speed and are O.S. independent...so it achieves the same as Java with less(???) bloat. As stated elsewhere, Python and Perl seem to be favourites with Ubuntu.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#25 [url]

December 15, 2008 19:21:11

One thing that did surprise me during my tests, python had the larger memory footprint than php, I was expecting it to be the other way round as PHP has so many built in functions.

Regarding modules/libraries (I don't actually know the difference yet), I've seen that some are indeed written in C but that many (most?) are written in python.

One of the reasons I'm looking at the nuts and bolts (e.g. parsing XML / html using various techniques) is that it will hopefully make a future transition to an even lower level language easier for me without having to rely on big libraries. My current server is under significant load - but I imagine it would be under 0.01 load if everything was in C lol

The python indention thing is really weird to me, it just doesn't seem compatible with the way I cobble stuff together!

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#26 [url]

December 16, 2008 17:25:42

Well i read a fair bit of the book last night, and must admit i find the Python syntax a bit weird. The whole dictionary/lists chapters confused me...unless i'm mistaken, they're just a standard array with one bonus...you can index by names as well as numbers...so why so much writing to explain that??

The memory footprint is not a surprise as Python has more built in functions than PHP.

I must admit the chapters on xml/html/http were more interesting (and made more sense!) and the next chapter (which i'll hopefully read tonight), is dedicated to SOAP protocols.

So now i'm not sure what to do. Pure Python code such as For loops are actually pretty damned slow compared to other languages i have played with...but most of them aren't scripting languages. I much prefer the PHP syntax because it's similar to C/C++ and Java. And then we have to bear in mind that we don't actually NEED 25,000 built in functions (i like lean n' mean..hence why i'm resisting Java). The only stuff we're REALLY interested in is http scraping and/or soap. Soooo, i'm thinking i'd be better off writing my own http library in C (unless there's already one that suits my needs).

As far as GUI is concerned, it doesn't come any simpler than Gambas' drag n' drop, and there are other GUI builders based on wxWidgets which i've not tried yet. At the end of the day, a gui is only a fancy front end for the real workhorse code. And now i'm wondering if i actually need to write any code at all. With all the libraries available to Linux, i wouldn't be at all surprised if i could cobble a bot together with nothing more than a few shell commands. Hmmmm, now there's a crazy idea...

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#27 [url]

December 16, 2008 22:59:38

I thought you'd made your decision, sounds like you're getting cold feet now lol That whole book is confusing, I reckon the only way to make sense of things is to jump in and try write something.

The reason I found the memory usage odd was because I'd assumed that python didn't have loads of functions built in, but dynamically loaded groups of functions as modules at runtime.

I've been experimenting again today, getting into the swing a bit though I still don't really understand much, it's very much weird and wonderful, and the way some things can be strung together is amazing. I'm still taking an awkward course, generating requests by hand in order to dirty my hands quicker. I managed a login and keepAlive to the betfair API earlier today, not a fantastic acheivement but I'm pleased with my progress. I think my best upcoming learning test will be solving gMPC decompression (without peeking at nadat's code!)

Quote    Reply   
avatar

denp

bot addict

Posts: 65 Member Since:June 14, 2008

#28 [url]

December 17, 2008 08:38:17

The for loops are pretty slow, but you can generally avoid them using much faster "list comprehension" and lamdba functions - this makes your code much concise too. take a look at http://docs.python.org/tutorial/datastructures.html#list-comprehensions. re html processing, you should be able to find what you need in one of the prebuilt libs like sgmllib. But it's never going to be as fast or as light as C.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#29 [url]

December 18, 2008 21:40:43

The for loops are pretty slow, but you can generally avoid them using much faster "list comprehension" and lamdba functions - this makes your code much concise too

-denp

GULP...sounds complicated...yet reading your link, it's just a glorified for-each array??

I quite like the look of the built in http/html stuff.

Now i am wondering how good these HTML parsers REALLY are? Do they REALLY make it easier to parse html or can manually parsing the info you need be faster and easier to debug?

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#30 [url]

December 20, 2008 19:06:57

Fred, this may be of interest: http://stackoverflow.com/questions/2933/an-executable-python-app#2937
It shows various methods of making a native app from Python code for Linux, Mac and Windows. It also mentions multi platform GUI libs.
There is also an option with Python to compile to Java bytecode so that it runs on a JVM...i know it's bloaty but it *should* be fully cross platform?

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
Remove this ad
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#31 [url]

December 20, 2008 19:31:43

I've also done some research into scraping with Python. It seems that Python 3 will be a fairly radical update, cutting away some of the dead wood and doing away with having multiple built-in functions that achieve the same thing. So my findings are that urllib, urllib2 and sgml parser are NOT the way to go if you want future-proof code. Although using a couple of 3rd party libs, THIS recent (Nov 2008) tutorial does scraping the "correct" way.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#32 [url]

December 21, 2008 13:09:41

Thank you, that looks interesting birchy re:compiling. I didn't get around to looking into it myself as I've found Python to be fast and I don't currently need any obfuscation. Rewriting my bots in a new language is forcing me to think differently, I've even been writing chunks of code several different ways and timing them to see what works best, and even though I barely understand Python I'm now halfway towards a finished spaghetti test bot that should hopefully use less than quarter the resource usage of the PHP bot it's replacing. I think I could do a lot better by dropping my dependence on MySQL too and instead just keeping things in memory with occasional syncing to disk. I am kludging it somewhat now as I want it finished yesterday lol.

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#33 [url]

December 22, 2008 18:30:46

fred, if you really want to keep a db (which is a good idea for analysis) have a look at SQLite - you are probably no stranger to it, but from my understanding, python and SQLite go together like two halves of something.

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#34 [url]

December 23, 2008 02:15:12

Cheers Alan. I've looked at SQLite before, to use from PHP. I've preferred sticking to MySQL so far as I'm familiar with it, it does what I want, it has the rather convenient phpmyadmin and also because it's rock steady and more or less looks after itself.

I think my MySQL performance problems are basically down to me using inappropriate storage methods for one of my oldest bots while simultaneously hammering MySQL from all of my other bots that use it for market sequencing and state persistence.

In my Python test bot I don't need any long-term storage, I've chosen to use a global object to hold markets and states and just pickle it every so often - in normal operation the data will never be needed to be reloaded - I may even disable the pickling once long-term reliability is established, it's just handy during testing to have persistence between runs.

Quote    Reply   
avatar

inksmithy

Botting Guru

Posts: 184 Member Since:June 1, 2008

#35 [url]

December 24, 2008 14:10:48

No worries Fred, the reason I suggested it is because its something I've been looking at recently myself - storing prices and so on for later analysis, see if I can come up with trends (apart from the obvious ones) and so on. Cos I'm in Java, the obvious one to look at is Derby, which is reasonably good as far as it goes - it even has an embedded variant which means you can download and run the app and it will create the database and associated tables for you, so long as they are defined in the persistence modules.

Thing is though, I have come to recognise that I am stuck in something of a rut with Java and I'm a couple of months away from my next Java course at the Open Uni, so I wouldn't mind having somthing of a brain sweep by looking at something different. So looking at python and databases is a useful way of doing that.

Just as a matter of interest, how are you guys going about getting started on learning python? I've had a look at the documents and the very last thing I want to do is go through an analysis of 'Hello World' again. Are you blokes using a tutorial site or a book?

Slashdot. It's like Digg on slow, but sensible.

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#36 [url]

December 24, 2008 16:51:34

Personally, i'm a bit of a perfectionist so i have spent several weeks trying out various Python text editors and IDE's. You see, i'm used to the Visual Studio type environment. Code completion and call-tips are essential for me...if done properly, you can have the whole Python manual at your finger tips. Yes, it's lazy, but nobody can deny that it is also highly productive and easy to learn. The problems i've found with Python editors is that most of them cannot complete library functions and self-made objects OUTSIDE of an interactive shell. For instance:
____________________________________________________________________________________
import urllib2

pkt = urllib2.Request('http://www.google.com')
pkt.add_header('Accept-Encoding', 'gzip')
____________________________________________________________________________________

Most editors will give you a list of possibilities when you type "pkt = urllib2." but that's it. Some of the better ones will show the completion AND the function parameters (call-tips) and the description/doc string. Where most editors seem to struggle is when you type "pkt." on your self declared object. I've not really had any great success finding an editor or IDE that can satisfy this. I believe Emacs and Vim will do it, but how the hell did they manage to turn simple text editing into a steep learning curve? So having gone round and round in circles, I think i'll probably settle with Editra. IDLE is pretty good but it just looks so shit on Linux. I'm sure some of the other editors can offer the completions and call-tips, but if i couldn't get them working with a few hours, i dismissed them. Scite looks quite nice and supposedly supports completions and call-tips, but i couldn't get it working.

As for you, Alan, you probably have a decent Python editor built into Netbeans? Regarding learning Python, although i'm pissing about finding editors and reading tutorials, it's one of those languages that you can just jump in and start coding by trial and error. You shouldn't really dismiss starting from scratch because Python IS significantly different to Java, etc. For starters, you don't have to declare any data types (which i found hard to accept) and then you have things like lists and dictionaries. There are tutorials everywhere but Dive Into Python is pretty good for people who already know some programming and i also like THIS one. And we mustn't overlook the official Python documentation: http://www.python.org/doc

Now i bet you think i've become a Python wizz already? Actually, no. I've barely written ANY Python code yet.

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#38 [url]

December 25, 2008 02:19:05

Just as a matter of interest, how are you guys going about getting started on learning python?

-inksmithy


I started out looking at the manual and beginners tutorial. The same as birchy, "dive into python" put me off a bit to start with. I didn't actually "learn" anything until I attempted to do something myself for real (playing with some bf api xml). I've found it quite hard grasping tuples, lists, and dictionaries, and I still don't fully understand scope. I've been using google a lot, coming across various tutorials, mini how-tos etc. I've found it awkward pinning down available methods and usable attributes for some built-in functions and drawn a few blanks when looking into optional modules, must be some obvious resource I'm missing. I've just been using my usual text editor (gedit) to play with Python - the syntax errors have been most extensive! It's a good job I don't program for a living lol

Quote    Reply   
avatar

birchy

Betfair Elite

Posts: 591 Member Since:May 11, 2008

#39 [url]

December 25, 2008 20:11:35

Hey Fred, have you tried Geany? When i was tinkering in PHP, i found it nice and lightweight but excellent at code completion. I found a plugin for gedit that supported Python but it wasn't what i'm looking for.

Anyway, now that i've finally settled on an editor, i've decided i'm going to need a decent scraping library. Requirements are:
  • can handle http and https
  • can handle GET and POST
  • automatically handles cookies
  • automatically handles gzip decompression
After hours spent Googling, i can't find anything that supports all of the above, so it looks like i'll have to write my own....unless one of you Python guys can suggest a lib??

www.bespokebots.com

"This time next year Rodney, we'll be millionaires!"

Quote    Reply   
avatar

fred77

Botting Guru

Posts: 199 Member Since:May 14, 2008

#40 [url]

December 26, 2008 18:02:29

I have Geany installed, I started using that for PHP as it had code folding which I thought would be convenient, though I just tend to use whatever editor opens when I open a file lol (currently gedit). I haven't tried it with Python, I will give it a go.

If you find a perfect scraping library do let us know! I reckon for best performance a mix of different modules is needed but I haven't come across a site that shows a suitable high performance combination working nicely together.

Quote    Reply   
Remove this ad
Add Reply

Quick Reply

bbcode help