Squeak People diary for mikevdg

3 Mar 2008

Mon, 03 Mar 2008 02:23:58 -0700

Yes, I'm still working on Namespaces. I'm heading up to a new release which will include the work I've done on Packages as well.

Some documentation, which I'll touch up a bit before the release, is available at http://gulik.pbwiki.com/Namespaces.

The Packages work has a lot of potential. I'm going to separate source code from binary code; the Package will be binary and the PackageVersion will contain the source code. Packages have a unique UUID each, and dependencies between packages use these. Combined with the fact that multiple versions of the same package can be loaded into Squeak at the same time, it should - in theory - ensure that any code will always run using exactly the same dependencies that it was orginally installed with.

After this work, I'll be looking at making a Kernel and Collections package so I can start refactoring it! I'll be making the Squeak Kernel completely namespaced. I'll remove a lot of junk out of it, and add Dominion support so that runaway processes cannot crash the image (yay!).

4 Sep 2007

Tue, 04 Sep 2007 23:48:00 -0700

My NamespaceTools are slowly becoming usable. You can now define a new Package, Namespace and Class from scratch using the tools, and the import list is automatically populated... well at least it is when you define your class.

I spent most of last night tracing through the compiler trying to work out where it was still looking at the SystemDictionary for literals. I almost gave up, except for when I realised that I could put a breakpoint in Namespace>>bindingOf:. Before that, I couldn't insert breakpoints because the debugger uses the compiler (Aaargh!). A few minutes later... I realised that Class>>bindingOf: invokes itself on it's superclass as well (!), which was Object (the original Object class, that is, which is still in the SystemDictionary), who would then look up variable bindings in the SystemDictionary... then I went to bed.

So tonight my task is simple: rewrite Class>>bindingOf: to *not* call superclass>>bindingOf:. That behaviour is stupid, and I can only imagine it is so that the superclass's pooled and class-pool variables can be used by subclasses.

I don't intend for these tools to be rock-solid, production-quality code. I will probably just stop work on them once they work well enough for my purposes. These tools are just so that there's an initial development environment that allows people to work with Namespaces. In the long term, I hope that somebody else will come along and replace them with something that is much better designed.

Next up once Namespaces are working: replicate the buggers! I'm quite excited to see if I can finally get DPON to do remote class loading. After that I hope to make a few demos. After that, I'll probably look at making a completely secure Namespaced image (i.e. the goal of SecureSqueak).

Oh, and there's a write-up about Namespaces here: http://gulik.pbwiki.com/Namespaces.

SecureSqueak is described here: http://gulik.pbwiki.com/SecureSqueak

26 Jul 2007

Thu, 26 Jul 2007 01:37:10 -0700

I'm still chugging along with Namespaces. I had a brief diversion with REPLServer (a backdoor Telnet server for Squeak) - I found a couple of bugs. It's now been released on the PackageUniverse again.

In other news, I also made a release of Namespaces in the PackageUniverse for Squeak 3.10 as well. This required newer versions of ToolBuilder and a release of the PlusTools as well. The PlusTools aren't really ready for general consumption, but neither is Namespaces. They're only there for curious people to look at.

Next up is a PackageManager tool. This gives you a list of all packages installed locally. It is intended only for developers; my great end-goal is to have package management done completely transparently to the user such that imported Objects "just work". The PackageManager shows a list of packages available. Each Package can be edited (with the NamespaceBrowser) or removed. The PackageManager is used by the NamespaceBrowser and other tools to search for particular classes and to do automatic import list management.

The PackageManager also allows somebody to import several versions of the same Package into an image. The reason behind this is that Packages depend on particular versions of other Packages. If a user installs a Package, all dependancies (i.e. particular versions of other Packages) will also be installed. In this way, code will run in exactly the same environment it was written and tested in. Again, this is also up there with the great goal of having imported Objects "just work".

6 Jun 2007

Wed, 06 Jun 2007 00:02:44 -0700

I lied about BitBlt being slow on the Jornada. Sure, it's slower than a PC, but I wrote a small demo to move a form across the screen following the mouse. It was... comparable to moving solid windows on a new PC, and probably fast enough to make an arcade game! I always thought that re-drawing a window from a display list would be faster than copying large bit maps around, but it appears to not be the case, probably because bit maps are done using primitives?

I've made a few minor changes to StandardSystemView - a one-liner will make moving and selecting windows run at bearable speeds by re-enabling caching of bits.

Anyway... onwards. I'm working on ReactiveUI on the Jornada when I'm commuting or waiting for something. This is going to be a simple UI framework specifically targetted at... 16MB ARM-based Jornadas :-). If I can make it fast and fully functional then I'll release it and start using it instead of Morphic or MVC. I've always wanted a UI that reacts to your mouse clicks before the next screen refresh!

Meanwhile, on my desktop PC at home, Namespaces are coming along nicely. I've just gotten some better conversion of the SystemDictionary categories to Namespaces working, so that pooled variables and method literals are converted. I need to write some tests for this. Next I'll be working on the tools again as I start converting (umm... pick from the list...) Collections to a Namespaced Package.

So what "Packages" would a basic Namespaced Squeak kernel need? Kernel, Collections, Compiler, and a UI... maybe Pavel's console or REPLServer.

23 May 2007

Wed, 23 May 2007 04:44:50 -0700

I got myself a new (second-hand) Jornada 820. It's a nice little machine in theory: ARM processor, 640x480 screen, 10 hours battery life, 16MB RAM.

Unfortunately, Squeak doesn't run on it that great. After some initial hiccups, I finally got Squeak to go - the two issues were (1) Find a VM that run on this machine and find an image that's under 10MB. Chuck in a 1GB CF card and I have a NZ$80 laptop! Less than half the price of the OLPC :-).

However, it's slow. BitBlt is damn slow - I can watch individual Blts in progress. I've found a GAPI (i.e. fast graphics lib) DLL for this machine, so when I get some free time at home I might have a crack at making a VM that uses that. The LCD controller is built into the ARM processor and uses the system bus for screen refreshes meaning that the processor runs slower than it could if the screen wasn't there. There's a greyscale mode which I might try playing with - not only will it make BitBlts faster, but it theoretically might make the CPU faster by using less bandwidth on the bus.

It's too slow to run Morphic, and removing Morphic makes an image under 10MB, so that was the first step. MVC is slow but usable. There's a bug in MVC on 3.9 and 3.10 where half the mouse-clicks get lost, so I used a 3.8 image and stripped it myself. There are also some spots such as opening a debugger that needs some speeding up to to make the machine more usable.

At the moment, I'm working on Namespaces on this machine. I don't think I'll be able to run the Morphic-based Namespaces browser on it, so instead I'll be porting Squeak to Namespaces at home, and probably working on a new, simpler GUI framework on this small machine. When that GUI framework works, I'll port my Namespaces development tools to that.

7 May 2007

Mon, 07 May 2007 00:12:03 -0700

I gave up on modifying Monticello.

This happened when I finished writing the code to export a package to disk. Then I tried importing it. Then I realised that Monticello has this fairly non-trivial import system which actually creates a patch between the package it is importing and the current version in the image, and then applies the patch. I didn't want to have to deal with that: all I want to do is file in, and file out. No more, no less.

In summary, Monticello is hard-wired for the singleton SystemDictionary.

Most of the functionality that Monticello provides will be also provided by my Packages, with the exception that while Monticello objects need to be loaded into the image before they can be worked with, Packages are live objects that are already in an image and ready to be used. You can create instances of classes contained in a Package. What is more, you can have two versions of the same Package loaded in an image at the same time and make instances from both. You can copy Packages, merge Packages (when I implement that...), find differences between them, make them read-only (i.e. make a snapshot), save them to disk or throw them across a network (when I get DPON working again). So Monticello is stuck without Namespaces. The source code is available, but it's not complete and I have no intention of finishing it.

In other news, I've discovered a very interesting garbage collection algorithm: back-tracing. Essentially, you give the garbage collector the possibility to cheaply find all references to an object, and you trace backwards hoping to discover if that object is hooked up to the root set or not. You can mark an object as "white" (reachable from the root set), "grey" (probably reachable) and "black" (not reachable / is garbage), and by using the right state transitions and searching, it's possible to quickly discover if an object is garbage or not. I'll spare the details here; go ask Google for papers on back-tracing GC algorithms.

Unfortunately, the main way to cheaply find references to an object is to store a list of those references with that object. This gets very expensive; I am hoping that using a block-based VM will get around this. By using near-refs within a 64k block (which get GCed using standard mark-sweep) and far-refs between blocks (using the back-tracing GC algorith), it should be possible to contain most of the overhead.

This approach has several nice features. Firstly, it's concurrent. A near-ref GC is contained within a 64-k block meaning that the rest of the VM isn't affected by that GC. The back-tracing GC alg could probably be made concurrent as well. Secondly, the back-tracing GC alg doesn't touch all of memory, meaning that you can page blocks to disk. Nice!

30 Apr 2007

Mon, 30 Apr 2007 04:26:55 -0700

Lately I've been working on modifying Monticello to export Namespaces. The main change is a change to the format of the chunk-based fileOut. Standard fileOuts have a syntax something like:

!SomeClass subclass: 'SomeSubClass' ... !

Instead, I've added a CodeBuilder class which is used by the fileIns:

CodeBuilder current inNamespace: 'Some.Namespace' createClass: 'SomeClass' ...

This way, the fileOuts are still standard Smalltalk, but now no longer assume that classes have access to the compiler.

Monticello is a good piece of software, but I'm not so convinced by its architectural merits. It creates a kind of mirror of the classes and methods (referred to as a "snapshot") it is about to write out. My personal choice of architecture would have been to blatently deep-copy the classes a package comprises and use that as the snapshot. That way, minor modifications to the standard tools (or PlusTools!) could have been made to view and edit snapshots.

Nonetheless, I've modified Monticello as it stands to be able to create a snapshot of a package with Namespaces.

Doing so also made me come to a realisation: my original vision of Namespaces was that there would be a global root Namespace under which all other namespaces would have their hierarchies. I've now decided that a better approach would be to use a package manager (in this instance, Monticello) from which tools can be opened to edit each package. A Package is then a subclass of Namespace, and forms the root Namespace of a hierarchy of classes in that package. Packages can then refer to each other using Namespace's import lists.

5 Feb 2007

Mon, 05 Feb 2007 01:11:29 -0700

Last night I've been playing with ToolBuilder and PlusTools. ToolBuilder is a UI building framework that lets you write a user interface that will run on MVC, Morphic and Tweak. PlusTools are the Squeak development tools that have been developed using ToolBuilder.

And they're fantastic! PlusTools isn't finished yet, but it's very functional. The Browser, Inspector(s) and Workspace worked fine, so that's most of the ground covered, but some of the other less-often used tools acted a bit odd. But the most exciting part was using them in Morphic - and then firing up an MVC project and using the same tools there! I downloaded a Tweak image and they worked there too! I assume that PlusTools for wxSqueak will eventually exist if it doesn't already - I didn't go that far.

So what are my plans? Well, I need tools to work with my Namespaces. So far I've been modifying the old Morphic-based Browser to work with Namespaces, but now I'm going to chuck all that out and use PlusTools instead. That way, my work will transcend Morphic, and I think that ToolBuilder has been designed "the right way".

My plans are to:

Modify the Browser: take the functionality out of the Browser class and put it in classes for each of the individual panes.
Make a Namespace pane to replace the SystemCategories pane in the standard browser.
Add an import management pane, which can either be in it's own window or in the Browser... somewhere...

While I'm at it, I'll see if I can start fixing up PlusTools - that's really a project which should be in base Squeak replacing the cruddy old Browser there.

28 Nov 2006

Tue, 28 Nov 2006 11:47:17 -0700

REPLServer is now officially useful! I've just debugged a problem using it.

When I tried to start up DPON, Morphic popped up recursive debuggers. Normally, that's it: Fried image, chuck it out and make a new one. But wait... I had the REPLServer running from my SecureSqueak project, so I telnetted in to my image to the REPLServer running at InterruptPriority. Yay! Perfectly responsive, even though effectively the image was running a fork bomb.

So then I did this (I typed things after the colon):

(\ /) @ @ Squeak3.8 of '5 May 2005' update 6665 == o == Type 'help' for help.

: utils allProcesses a ProcessList( 1: 3400: the timer interrupt watcher 2: 448: the user interrupt watcher 3: 1752: the event tickler 4: 2479: the low space watcher 5: 1598: the WeakArray finalization process 6: 1906: the inactive Morphic UI process 7: a REPLServer: Semaphore>>waitTimeoutMSecs: 8: a REPLServer connection: the UI process 9: 1796: the idle process )

...except that the process list was actually about 60 processes longer. Then I ran my fantastic stop-all-forkbombs script:

: utils suspendAllProcesses
 a REPLUtils

This command I wrote specially to suspend all Processes in Squeak running at user priority. Damn handy for fork bombs. Unfortunately, it appears to also have suspended Morphic, so then I did this:

: utils allProcesses
 a ProcessList( 
 	1:  3400: the timer interrupt watcher 
 	2: REPLServer connection: the UI process 
 	3: REPLServer: Semaphore>>waitTimeoutMSecs: 
 	4:   608: the user interrupt watcher 
 	5:   706: the event tickler 
 	6:  2767: the low space watcher 
 	7:   893: the WeakArray finalization process 
 	8:  1906: the inactive Morphic UI process 
 	9: Domain kicker: [] in BlockContext>>newProcess {[self value.  Processor terminateActive]} 
 	10:  1444: the idle process 
 ) 
: p := last at: 8
  1906 
: p class
 Process 
: p priority
 40 
: p resume
  1906

Yay! So now I can actually get to one of those debuggers and work out what is going on. Oh... I could also just have typed in this new command of mine:

: utils rebootMorphic

When Morphic is fried, this will get it up and running again in a new Project (with an ugly white background).

Am I happy? Hell yes. I have a killer new tool that lets me get into broken images. Oh, and for the curious, the cause of my accidental fork bomb was that I overrode #at: on a Dictionary subclass but stuck a #halt in there for debugging purposes. When my Dictionary subclass was asked to do a #printOn: by the debugger, it started doing recursive halts. Next project: fixing the stupid debugger to not call methods on the objects that it's debugging.

Do you want to use REPLServer? Go get it from my SecureSqueak project! Oh, and while you're there, add yourself as an admirer!

9 Nov 2006

Thu, 09 Nov 2006 00:20:06 -0700

Currently, I'm working on REPLServer (originally by Avi Bryant), which is a command-line shell for Squeak that can be accessed over telnet. In my opinion, this is the best way to have a "back-door" to a running image. If your image is running as a network server and it starts acting screwy, a "back-door" will let you get in and fix stuff. Also, if Morphic crashes or freezes (which happens too often to me), you will be able to get in using a telnet client and do "Morphic reboot" to reset your screen and debug the problem. I plan to implement security by only allowing connections from the local host - use ssh to the host and then telnet to localhost, port 4445 :-).

Currently I'm adding small enhancements. I've added a "help" command which prints some useful commands. I'm currently wedging an "InteractiveStream", which is essentially JLine or getline for Smalltalk that allows things like tab completion and history. I'm writing this to be re-usable for other projects.

Next I'll be adding another class: TelnetSocket, which is a subclass of Socket that understands the telnet protocol. I might also add an UnbufferedSocketStream which is a leaner version of SocketStream.

This code is available under my SecureSqueak project, where I'm writing and gathering components for a secure, uncrashable version of Squeak.