Brian Ray's Blog : Python: IPython as primary Shell

Brian Ray's Blog : Python: IPython as primary Shell: "IPython , a Python interactive shell developed for the scientific computing community by Fernando Perez is such a powerful tool, I just may replace my default login settings from just bash [1] directly to IPython over bash.

How may people do this? I mean really! Is it practical to cross this line?

If you are new to IPython, there is a decent intro article from ONLamp. Although, I fail to see how this article fits into the LAMP [2] world. Still, it unveils some need features like: magic: Quickly assess special IPython features. Tab Completion: auto-complete code attributes, classnames, methods, and more Introspection: Easy access to Doc Strings and internal documentation History: store a dumpable session of commands Debugger Access: internal access to pdb Run: a safe way to run a file and ignore the __name__ == '__main__', trick Macros: capture command history into a macro

All great stuff. Also mentioned in the article is System Shell Access. For example, you can use cd, pwd, and ls directly in IPython. To escape to the shell use '!' or '!!'.

The IPython Manual notes in the section IPython as system shell. I loose job control if I use iPython this way. Although, I haven't had many issues so far.

Starting IPython with 'ipython -p pysh', creates a special envirom"

I found this interesting because of recent ideas I've been drawing up for a Python Embedded Shell. The Python syntax for `foo` being taken as repr(foo) is to be removed, so I thought, "Why not have a special interpretter to expand `foo` to execute a program named `foo`?"

I went into it more in my mind, so I'll lay out the ideas here.

• foo=10 would be taken as normal
• print $foo would be used to print it to the terminal
• foo would just run the program, foo (I updated from the original thoughts)
• $foo would run a program stored in the string foo by name
• if, while, def, class, elif, else, and for all work as they currently do.
• That $ syntax is only needed where it would be otherwise confused for a program name.
• `foo` would resolve to a special object representing a new process running by executing foo. This process object would have stdin, stdout, and stderr file-like object attributes. It would also have kill, pause, and continue methods.
• New operators exist for piping. Such as foo->bar to pipe stdout of foo to stdin of bar. foo!->bar pipes stderr, instead. null is known as a built-in process object, which just pipes everthing to /dev/null or an equivalent.

Very basic, not well polished. Besides, I think I like Monad, the Microsoft Shell, which is still in Beta, a lot better than my idea or any command line I've ever used. Sorry, my Linux buddies, but a good idea is a good idea, no matter who comes up with it.

Dynamic Linking for Module Attributes

In much Python code, foo.bar refers to some attribute of the imported module "foo". There is only ever one instance of any module, so could attributes of modules be given pointers somewhere in the module object (internally) and thus importing scripts can be compiled such that foo.bar would be resolved to some new bytecode that says "load the object pointed to at this location". The location would be null when compiled, but importing would consult a table mapping the module's attribute references in the script to the locations in the bytecode of those null pointers. This would effectively work like dynamic linking in C/C++, and would reduce a lot of attribute lookups to a pair of pointer dereferences.

Focus, Direction, and Real Life

My wife and I have a baby on the way, my best friend moved in with us after running into some hard times, co-workers are causing my daily headaches, and my car broke down in my driveway. Where do I find time to code?

As it is I seem to barely have time for my code, and I wasn't making the progress I'd like even when I was impressed with the progress I was making. Factor in my continuing frustrations over the lack of writing and art I've been able to spend time on, and my output just looks dismal. Real Life, the good and the bad, are cutting into the things I always though real life would be made of, for me.

I pretty much know that until I can support myself and my family independantly, I don't have time for coding, writing, and art. I only have time for one. It forces me to look at it from a bussiness aspect, of which would bring me money quick enough to allow me to resurrect my other hobbies? It also instills growing fears of loss if I fail at any of them. Not only will I continue to work a dead job, but I'll continue to have no time for creativity.

Where's a grant when you need one?

Python Coroutines Schedulers

Just wanted to post a quick thought, mostly so I would remember it. I want to try to write a small decorator to turn any function into a coroutine, associated with a particular scheduler. When called, the coroutine would return a handler, and the scheduler would continue normally. it would be able to tell when a coroutine was called by another, and handle that appropriately, allowing the yields of the inner call to be yielded by the handler as an iterator. This would allow coroutines to be called pretty much just like functions, with everything being scheduled in the background pretty nicely.

PyPy and More Platforms

PyPy is coming along nicely. It gets closer to matching CPython for speed every day, but it still has its problems. It brings more questions to my mind about the matter of duplicated efforts with all of these Python implementations.

What we really need is a standardized from of representing Python code in Python code. Things like a Statement class and Expression instances, for example. This would allow people to easily write code to optimize and analyze code parsed by any front-end and compiled to any back-end.

Helping a Friend

A very good friend of mine has moved in with us this weekend. Some hard times, which were severely drawn out, lead to a four-hundred mile relocation back home, so I've cleaned out my office and set up the spare bed, to make him feel at home. Hopefully, things will get better in fast time. At least it all doesn't seem to be getting to him. Maybe, I'll get back to my old gamer-self with someone to thrash on in Halo, again.

My Take on Python Concurrency

The mailing lists and blogs have been abuzz lately with discussions, arguments, and good and bad ideas about how to deal with concurrency in Python. There is a general distaste against using vanilla threads, due to the difficulty in managing shared resources. Generators are an increasingly popular option, but they don't offer any pre-emptive nature, if you are into that sort of thing. And, of course, the Global Interpreter Lock is bashed and many pray for its removal, but that will only happen if a real concurrency mechanism is added not just as some library, but into the library itself.

Now, off the bat I'll say that I don't hate threads, but I really wish there was a better alternative. I do want to see the GIL go away. Finally, I don't like having to explicitly suspend my task every so often to allow something else to process for a while. This is tedious and does not always reflect the nature of the task well.

Active Objects seems like the most Pythonic approach, and like they could work well with existing, non-concurrent code. With Active Objects, you have objects designed to work concurrently. Calls to their methods are typically queued so that only one method of one active object will be executing at a single time. Thus, the active object does not need to syncronize on its internal state data. Extended loops in a method could get greedy, however, but that is a small matter.

The biggest issue is dealing with shared objects, which happens so much in Python's pass-everything-by-reference world. Methods can be crafted to be safer, such as by taking only basic types (integers and strings) and containers of those or containers (lists of strings and tuples of dictionaries mapping strings to booleans, etc.). But, inevitably, you will have to pass something more to a method.

One approach could be to copy objects before passing them, even to deep copy them all. Another would be to create special shallow objects that represent the state of the original without copying all of it. But, perhaps the best choice is to simply use more active objects. If we pass one active object to the method of another, and are accessing all data through their "active methods", then syncronization is generally secured. Perhaps it would also be beneficial to define a common locking API for all active objects.

Python is a wonderful, dynamic language, but there is some speculation that the very dynamic rules that make Python great, are also making it so difficult to settle on a good concurrency mechanism and to allow safe, concurrent code. Many Pythoneers scuff at the idea of true protected and private attributes, but maybe that's just what we need.

The ability to modify everything may be useful, but maybe it should be controlled a little more. We should be able to execute code in the context of protections on certain objects, like giving read-only versions of modules or write-locked lists. This becomes faulty when you realize you can just call object.__getattribute__, but I have a possible solution: the protected class. It would be a new built-in class, and anything inheriting it would be given special abilities: object's methods would treat is specially and following its access rules (properties would be immutable, exception through the functions defined for them, for example) even when using object methods directly, or other introspection mechanisms. Combining this with crafted globals and built-ins for 'jailed' code, could be a sucessful way to execute multiple lines of code in parellel without worrying they will step on each others' toes.

Python Object Space Demo

I've whipped up a little test of object spaces. You can create global contexts and run code within that context. You can also queue functions to be called (the API forbids multiple threads per space), and set response functions to be called at the return of a function called in context of the space. It is very simple, but effective, I like to think. The initial code is available here.

Branching out with Blogging

I may be creating some new blogs. I hold back posting some things, because I want to post on so many topics. I'm considering keeping this blog for general ranting, political commentary, and the like. I may also create the following blogs:

• Ranting and Raving about Games and Movies
  
• Ranting and Raving about Software
• Ranting and Raving about Art

The last one is likely not to be created, although I may resume my art and end my very long hiatus from Deviant Art (see my DA page here). This will allow me to more easily and openly blog about more specific things more often, without worrying that no one will read a blog so mixed with topics so unrelated.

Weekend Involgo Plans

With some luck and determination, I'll be able to make some good progress over the weekend. I've had overtime all week, and I still have to come in for a full shift on Saturday, but I'll have some time before work then and again on Sunday, and I'll have a prototype of my Involgo database working by the end of the weekend, I believe. Depending on the exact conditions of the prototype at that point, I may or may not be posting a link to an initial alpha release. We'll see.

Will XForms eliminate XUL?

Will the introduction of a complete XForms implementation in the Mozilla platform eliminate the need for XUL? I wouldn't be surprised if the current implementation uses XUL for the actual rendering of the forms, but wouldn't it make sense to work toward eliminating XUL from the Mozilla platform, in favor of a standard alternative? On a related note, it seems there could be plans to incorporate OpenGL or some similar API to the new canvas element, which would mean that if you allowed document rendering to a canvas element (XML is finally looking cool) you could have the entire window just be a big canvas element, render the browser's own XForm document to it (which would render the pages in their own canvas elements!) and all of this, being drawn with a clean and proper API, could be hardware accelerated, where such is available on Mac OS X, Windows with Avalon, and decent configurations of X.org.

Pandora Take Two

Since my last post about the service, I got a comment from Tom Conrad, the CTO of Pandora. I've had more time to think about the service, its good qualities, and its drawbacks. I'm sure Tom keeps a custom blog aggregator to find anyone posting about Pandora, but I'm still honored for that to be my first post (not counting the Blog Comment Spam I had to delete). I'm surprised I don't see any posts by Tom on Ned's blog, though, I guess Tom just peeks around on Blogspot and doesn't catch the independents.

Anyway, Tom did say to feel free to continue blogging about Pandora, which I will do, because I have some more thoughts I'd like to share, especially knowing that he is listening.

Pandora is great. Its the best online music service I've found, and possible the greatest source of music I have, next to listening to Nights with Alice Cooper on the nights at work when I get the desk with the radio. When I say something is the best I've found, that doesn't really say a lot. It does in Pandora's case, but I'll clear it all up here.

What Pandora does right:

• Deployment as a web app
• Multiple stations
• More fine grained than genre and year
• Finds new music
• Dirt cheap subscription ($36/year or $12/quarter)
  

What Pandora does wrong:

• Can't listen by genre and/or year
• Think computers are the best at organizing music
• Pass up on the greatest source of music listening habits since iTunes. Maybe better than iTunes.

The last point is where I'm going, so bare with me. Let me go over the good in more detail. I'm giddy over the pricing. I can't believe people really pay $.99 a song from Apple, and I'm sickened that the music execs actually think that isn't enough.

I love when Pandora plays a song, and I think, "Hey, this is pretty good, who are these guys?" Of course, I also get Tivo flashbacks when I think, "Hey, why are you playing that?" It does find some nice tunes and I would be out of my league to acquire and organize a playlist of that quality and size. Now, I'm a Linux user (though more and more distastefully), so I'm happy Pandora went the web app route. It makes it so much easier to refer the service to my friends, who can just go to www.pandora.com and start listening, just like that. And, thank god they don't just group by year and genre, because I might be in the mood often for 80's music, but lots of music not made in the 80's will fit that class of music.

Sometimes, I just want to listen to 80's music, and how do I tell Pandora that? I can't seem to search for shared stations by name so I can just find anyone's "80's Music Mix" station. The redefinition of the very meaning of a station is a god-send, I must say. I can listen to the same station as thousands of people, and skip over any song I don't like. That freaking rules, because I hate when the radio plays something I don't like.

Now, what is the biggest thing that Pandora is missing? First of all, let me say this is speculation on some level, because if they take advantage of this, it must be completely on the internal level. They have thousands of listeners voting songs in and out of various groupings and all of this is stored into a wonderful database of golden information. So, Tom Conrad, I ask you, where is the "Play what other's like to listen to with this" option? The Music Genome Project may be a great venture and successful at what it does, but nothing beats a fellow human with a common taste in music. How about mining that database for common listening patterns, generating collective stations?

I'm sure they may find this a scary idea, because in essence it seems to go against the very beginnings and roots of Pandora itself, but it is also quite possibly the biggest value in the service for the company behind it, and to not take advantage of this would be, to put it simply, kind of stupid. (Sorry, Tom)

C#, .Net, and Mono

So, I've been doing some tutorials on MSDN on the C# language. It seems like Java without the emphasis on a virtual machine. Or, like C++ with emphasis on portability of useful code. Either way, the gun is in a lock box instead of just having the safety on or being taken away completely. I'm going to be looking into Python.Net and IronPython very soon, as well, and the Boo programming language, which is a Python-ish .Net language.

So, from the get-go, I admit that I do like C# for what it is. I've been looking for a compiled language for those cases when I would care to have one, and maybe it would be a good way to diversify my skill set. I've even considered an eventual implementation of Involgo in C#. With Python.Net I can wrap a .Net assembly for Involgo into the original Python API, and let any existing code still work, but likely faster.

C# looks like it would let me do a lot of the things that I like Python for, but without making them too difficult to manage like some other compiled languages. Delegates are nifty for passing function-like objects. I can pass objects without a specific type when I really need to, or have static typing when I need/want/think I want it. And, the automatic usability from so many languages is just "to die for".

I do wonder, however, what the consensus among the .Net community is for the Mono project. Most .Net users are definitely Microsoft users, so what do they think about us "others"? In the FS/OS world, the community is king. In their world, the community is surely strong, but it is far different. I wonder often about this rift between geeks. We're all after the same thing, we just think there are different ways to go about it.

Lepers in America

Before Leprosy was cured, all the lepers would be sent to live together, away from the rest of the world. It might seem cruel by todays standards, but a horrible, killing disease is what it is. And, someone infected might as well be that disease walking.

Today we have several horrible diseases plaguing the planet. We have HIV spreading across the globe, for example. We don't put people with HIV into special camps. No, instead we just tell them, "remember to wrap it before you tap it." Meanwhile, the little AIDS demons continue to laugh at the world they may some day cripple.

Now, I'm not suggesting that we lock every with an STD up, but shouldn't there be some accountability? If you are walking around able to cause such an ammount of death and destuction, you'd think someone would say, "Hey, you better own up to this."

Crazy people are still punished for killing people, but is it a crime to know you have HIV and sleep around anyway? If you knock someone up, you have to support the child, but do you have to pay for their HIV medications if you infect them? People in this country won't even where protective masks when they have a bad cold, so why should I expect anything reasonable?

Pandora makes me want to spend money on music again

I love music, and I hate to buy a CD. To think how much I'm paying for that CD, with the full knowledge of how much it really costs to make, and how much the bands I love even get from the sales. I'd rather buy Ogg Vorbis streams directly from the band members, but that won't happen while the RIAA is still running this country.

Still, Pandora has given me the desire to spend money on music again, by giving me actual value in its service. Thanks, Ned, for blogging about this.

Is there room in the bubble?

There never really was 'a' bubble, and so 'the' bubble didn't really burst. There were bubbles, and there were a lot of them and a lot of them got big, and a lot of them burst. There are still a lot of bubbles.

Do any of these bubbles have room for me? Can I make my own little bubble and hope I don't pop it myself?

I'm evaluating my options for a start-up. I'm considering costs, estimating revenues, and trying to decide the timetable in which I could make it actually work. I'm doing my best to balence a quick turn around with a solid investment of my money and time.

Lots of geeks have done this, and I'd like to think I could be one of them who succeeds. And, I do think that I will, too.

Involgo Progress from Scratch

I made plans, I designed things, and I spent lots of time making very little progress. Finally, thanks to a suggestion on IRC, I just dove in, scrapping the ill-fating code I already had, and just making things work. My little object database now works, kind of. It doesn't store data between sessions, it just keeps them as Python objects in memory. I'm able to easily and quickly add in all the features, debug the database, and iron out my ideas. When the major features are generally operational, I'll add some basic persistance for more testing. Dive in: procrastination is the disease of the failures.

C-Pound is Looking Sharp

I took the time to view the video interview about the LINQ project and C# 3.0's first-class query mechanisms. I was actually impressed with it, and it solved some problems I have with Python. It also made me think how the same solutions could be applied in Python, or any such dynamic language.

What is most interesting about LINQ, I couldn't be sure of from the video. Does it pull in all the data and apply the anonymous functions to it for conditional filtering, or does it take the where clause and actually generate that into a request to the server? Now, this doesn't matter for XML documents and especially direct queries on CRL objects, but with relational tables it really is important.

This is what Python needs: a generalized way to generate patterns to match, using existing operators. There are some tricks available, like allowing sometype==value to return a callable that can be passed an instance of sometype and return True if it equals value. This can be used to generate lots of pattern matchers. The problem is that some things aren't so easily redefined, like and and or operators, and it isn't easy to define that operation if you also want == for instances of sometype. That requires defining an instance method and class method of the same name for one type. Possible, but very tricky.

It would be very useful to add first-class pattern matching to Python. I'm thinking of a syntax along the lines of ?==x to generate a pattern object representing the pattern needed. What is important, however, is that it would create more than an object that can be called to test if something matches the pattern, but other code would be able to introspect the pattern, merge patterns together, generate SQL code from them, etc.

I wonder how well the community would respond to this. It would probably be best to promote a common and hopefully standard pattern package, and to propose syntax integration of the module at a later date. That would also give time for the pattern matchings to evolve. Hopefully this will happen, because Python is improving every day, but if it doesn't do so in the right places, it will fall from its rising seat of glory.

(More examples of the Pattern Syntax)
(All comparison operators work as expected, such as `?>10' to test for values greater than 10)

`?foo' to test for an instance of foo

`?is foo' to test is something is foo

`?in foo' to test is something is in foo

`?contains foo' to test if foo is something

`?hasattr foo' to test if something has the attribute 'foo'

`?attr foo ?==10' to test if something's 'foo' attribute is equal to 10

`bar?foo' to test if bar is of type foo, where bar is the name of an iterated object, such as in a for loop or list comprehension. This can be used to clarify when there are more than one object per loop to test.

`?[4]?=="four"' to test if the 4th item in a list-like object is equal to the string "four".

`?foo,bar' test if something is of type foo or type bar

`?all foo,bar' test if something is of both type foo and type bar

Any thoughts on this?

Fan-Tab-ulous!

I've always found myself to be a proponent of tabs in favor of spaces in source code. It makes more sense, because the reader can choose whatever tab-width they want, so it makes the source code more accessable, with a properly configured editor. Lately, though, I find myself using two spaces instead of a tab character. I still feel tabs are better, so why am I using spaces?

On Game Ratings, Companies, and Retailers

There is a three-fold source to the recent debates and arguments surrounding the video game world. Most people who care, and plenty who don't, have heard about the scandles with Grand Theft Auto: San Andreas and the Hot Coffee mod. That isn't what I want to focus on, because that has already been blogged to hell and back.

The system is broken. While most studies show the average gamer as being 34 years old, the industry is still trying to sell games to kids and teenagers. Now, I have no trouble with giving a game an AO rating, but why do retailers have a problem selling them? They sure are cutting out a huge section of their consumer base. Besides, Wal-Mart actually does card for M games, last I checked, so why not sell AO games and card them as well?

The bottom line is this: if most of the people buying games are adults, why should they always aim for a rating that lets teens buy it (T or M is usually the highest rating they want)? The answer is that the stores don't carry AO games, but that just brings back the questions about the average gamer age.

In the end, we just need to work to make everyone realize that AO games aren't bad. You can carry them in your store and card anyone who looks too young, just like smokes. Only AO games don't kill people, they help stressed out adults relax after a long day of working as a productive member of society.