SqueakMap today

Posted 20 Jul 2004 by gokr (Master)

SqueakMap is to most people "the tool that can install stuff"- but how does it work? And is it more than just a list of stuff?

I will try to first explain some history. the goals of SqueakMap, then move on to an architecture overview into explaining the SqueakMap domain model and defending it a bit, and finally explain how to install stuff in load scripts.

Note: This article is written in "sloppy mode". Fast and furious. :)

SqueakMap is most visible to Squeakers as the tool you get when selecting "open..." -> "SqueakMap Package Loader" in the World menu. Now, let's start from the beginning...

History

Now, SqueakMap was born during the 3.3Modules-era - a time of confusion and frustration. After Squeak 3.2 there was a hard push to move Squeak in a Big Leap into a modularized world. Henrik Gedenryd got deeply involved and wrote a lot of code with a vision of embracing the Environment experiment from Dan Ingalls and extending it further.

Henrik and I met at OOPSLA and I was intrigued by his ideas and the vision. I wanted to do "my part" and I realized that there were things missing from the planned implementation - the catalog on top. So I started building SqueakMap as a complement to the stuff Henrik was working on.

Many of the design ideas in 3.3Modules were sound, and a lot of the code was very good (I think so) but there were some issues that still made it go belly up - and we learned some hard lessons at the time.

First of all, some of the concepts were fuzzy and probably overly complex. Secondly, 3.3Module-land was hard to "live in", and that was probably the death blow. People didn't move over from 3.2, because 3.3 wasn't backwards compatible - at least not in a friendly way.

The lesson learned: Squeak must stay "livable". New things must come gradually. Otherwise people will not live there, and then the new things will die. Inevitable. New things need developers to be there and live inside it - it is like water for plants.

So unfortunately 3.3Modules died and we lost a few things. First of all we lost Henrik. He burned out a bit over the project - it was hard for us to follow his charge into his vision, and there was a lot of code to grasp in order to be able to help. When 3.3 was abandoned Henrik left the community, and that was a big loss IMHO. Henrik is a brilliant developer, some of the things he has done are very, very cool.

Ok, back to SqueakMap. SqueakMap didn't die. It was not dependent on 3.3Modules in that way. It lived on, but not very visible at first.

SqueakMap goals

SqueakMap is not only meant as a "package tool". From the beginning it has been meant to work as our "community memory", not only would you register packages but we could register web sites, information about people, custom Squeak images etc. The idea was to have a piece of object memory that we all *shared* and could maintain together.

Now, if we have such a memory - it is natural to stuff information there about available pieces of code - it is after all what we share the most. But I emplore everyone to remember that *that* is not the only thing we can have in SqueakMap. Today for example we have the names, developer initials and emails of Squeak developers in there - which is very useful.

So the goal was to establish a domain model which could be used to collect this information and which people could build tools on top of. And this leading to newer ways to collaborate.

My prediction is that this will become much more apparent during 3.8 when the currently dorman TFNR project will awaken and continue - it aims to "stake out" the base image in areas that can be registered on SM with responsbile maintainers and all. This is done using PackageInfo - our fledgling way of carving out a specific area of the image codebase.

The end result? Well, the most obvious result is that we can finally have people responsible for all parts of the image. This will mean that you can select a method or a class in the browser, bring up the menu and choose "send email to maintainers". Just a trivial example.

Architecture

The current architecture of SqueakMap is simplistic - but the domain model is not.

On the single master server running on map1.squeakfoundation.org there is an image running with the master copy of the map. The map is an instance of SMSqueakMap. It is the same map as you all have in your images and which you can get using "SMSqueakMap default" - that method simply returns the singleton held in a class var.

Currently, the only way to modify the map is to do it using a web UI that is layered on top of the domain model and which is running on map1.squeakfoundation.org. That web UI isn't good looking - but it works. It is built with <project>HttpView</project>, a more simplistic web app framework than Seaside, but still kinda neat.

So the maintainers log in on the web UI and modifies information in the map. The server takes snapshots of the SMSqueakMap instance using ImageSegments.

When you run "SMSqueakMap default loadUpdates" (which you can do using the menu in the Package Loader tool - "load updates from the net") your image will connect to the master server using HTTP, ask it if there is a newer snapshot of the image, and if there is - download it compressed and install it into your image. And save it locally in the "sm" directory too - those are the .sgz files with numbers. The number is the snapshot number. The older files can safely be deleted - in fact all those snapshots can be deleted because they are simply copies of the map from the master server - they do not contain any local information. If you delete them, a new file wil appear if you do "SMSqueakMap default loadUpdates".

And then there are some UI tools on top of the domain model, currently I think all use the standard Package Loader - the other tools haven't been updated with the new stuff in SM.

So as you can see - the architecture is simplistic:

...but on the other hand this makes SM easy for use developers to evolve. There is no special file format to keep up to date - ImageSegments removes that issue. There is a plan for a more advanced architecture but that is out of scope of this article.

The domain model - heart of SM

The domain model of SM is the only thing that really is important. The web UI will be gone in the future, the synch mechanism will be replaced with something much better - but the domain model will simply evolve.

Let's look at the model. SMSqueakMap, the sole instance which is the map, holds instance of SMObject. And some other silly less interesting things. All things in the map are thus instances of subclasses of SMObject.

SMObject hierarchy: 

SMObject #('id' 'map' 'created' 'updated' 'name' 'summary' 'url')
	SMCategorizableObject #('categories' 'resources')
		SMPackageRelease #('publisher' 'automaticVersion' 'version' 'note' 'downloadUrl' 'package' 'repository')
		SMRootedObject #('homeMap')
			SMAccount #('initials' 'email' 'signature' 'password' 'newPassword' 'advogatoId' 'objects' 'coObjects' 'isAdmin')
			SMPersonalObject #('owner' 'maintainers' 'rss')
				SMDocument #('description' 'author')
					SMPackage #('releases' 'packageInfoName')
					SMResource #('subject' 'version')
						SMEmbeddedResource #('content')
						SMExternalResource #('downloadUrl')
	SMCategory #('mandatory' 'subCategories' 'parent' 'objects')
Here we can learn a lot of stuff. First of all, each object has an id - an instance of UUID. We also keep track of when it was created and updated (timestamps), and they all have a name, a oneline summary and an optionally associated url. Not all these things are exposed in the UIs for SqueakMap.

Then we see that there are two subclasses - SMCategory and SMCategorizableObject. Things that can be categorized and the categories themselves. The SMCategory instances form a strict hierarchy, each category knows its parent and its subCategories. It also knows all categorizable objects that belong in it. An SMCategorizableObject can be in many different categories.

SMCategory also has a collection called "mandatory" which holds the classes (typically SMPackage or SMPackageRelease) for which at least one subcategory *must* be chosen. An example would be the category "License", inspect it like this: 

	SMSqueakMap default categoryWithNameBeginning: 'License'
You can also explore the category tree with alt-I on this: 
	SMSqueakMap default topCategories
Someone complained once and said it was a "forrest" but whatever. :) The top categories are the ones without a parent category, slightly odd design - I know, but whatever.

If we turn our attention back to SMCategorizableObject we can see that all objects currently are categorizable. The classes used today are SMAccount, SMPackage and SMPackageRelease. But we haven't exposed categorization of SMAccounts yet - but we could easily do that. So we could have a top category called "Community roles" and then have subcategories like "Guide", "Harvester", "Developer", "User", "VM port maintainer" etc.

You can also see that categorizable objects have another collection called "resources". This is a new thing that is coming soon. The idea is to be able to "attach" things to categorizable objects. It is a mechanism that will be used when adding dependency information to the model.

Going further - what the heck is SMRootedObject? Well, this is something (possibly) enters the picture when the map moves from being a single master thing to a tree of maps. Again, out of scope of article - ignore right now.

Then we have SMAccount. That is you. A Squeaker. You have developer initials, email, signature (not used yet - perhaps someone with better knowledge in crypto could help here?), password, newPassword (a parallell one if you forgot the original - both are stored as SHAs - also not a very good security solution but will do for now), advogatoId (connects to SqP), objects (all SMObjects owned by this account), coObjects (all SMObjects co maintained that someone else owns), isAdmin (hehe).

If we want to maintain more info about Squeakers - this is where we add it. The most interesting instvar is the "objects" and "coObjects". Two collections which today holds only SMPackage instances. But again, it could be other things.

SMPersonalObject means an object "owned by an SMAccount". Such an object knows its owner and co-maintainers (maintainers) as well as has an RSS field - not used today AFAIK.

Currently the only things we own are SMDocuments - meaning a series of bits. :) Currently we have two kinds, but only SMPackage is used today. SMPackage knows a packageInfoName - this is a crucial field to tie the SM packages into the image and was the thing I referred to earlier when I made my prediction for Squeak 3.8. Noteworthy is that this is on SMPackage and not per release - I think that having this on package level is sound - confusion would be far too possible otherwise.

An SMPackage more importantly has an OrderedCollection of releases. They are chronologically added there, but can form a tree (branches) - though this is not exposed in any UI yet. The relation to other releases is through the automaticVersion instvar.

I will leave SMResource for now - it isn't used at all yet.

This leaves us with SMPackageRelease - an odd little bird sitting directly beneath SMCategorizableObject. The reason for this is that releases are owned by SMPackages which in turn are owned by SMAccounts, so no need for an SMPackageRelease to be an SMPersonalObject. The single inheritance model is slightly limiting here - a typical example where Traits probably would make it better, but no big deal.

An SMPackageRelease has an instvar called "publisher". What is that? We already know which package it belongs to (instvar "package") and packages are owned by an SMAccount! Well, but we also have co-maintainers... :) So when a maintainer - either the owner or one of the co-maintainers - mark a release as "published" we actually set the publisher instvar to refer to that SMAccount. It is a good thing to know who published it. One thing we don't know in this model is who created the release - and that is a lack that has been born out of the introduction of co-maintainers - will see if we should remedy that.

Now we have 5 instvars left. :) "version" is the manually edited field and can be whatever the maintainer likes, "note" is the release note. DownloadUrl is the identifier that lets us fetch this package - we could use other kinds of identifiers and ways to fetch it - but currently we only use URLs. Repository is a new field that let's us point to a place where we can find the "bleeding edge" - typically a Monticello repository. This was very recently added.

And then we have "automaticVersion". This is an instance of VersionNumber which is a nice class written by Stephen Pair. It is in a separate package of the same name. A VersionNumber looks like "1.3.4", a simple classic multi numbered version string - read the class comment for details. One nice thing with VersionNumber is that it handles branches. And one nice thing with automaticVersion (compared to "version") is that this field is immutable and can not be modified by the maintainer.

Phew.

Defense of single map

Now, there has been lots of discussion on the list about being able to have "multiple maps". IMHO there is no such need, at least not in that sense. Remember - SqueakMap is our collective community memory - why would we want to split it up? There are things in there that we want to keep canonical - SMAccount, SMCategory and SMPackage being prime examples. We don't want those scattered around in ill maintained copies!

It is all about normalization of data. One enlightening example is the reason for having SMCategory in the first place. Why can't people just type "GPL" in a field and be done with it? Because then some will of course type "Gnu Public License" or "GPL v2" etc - and suddenly we lost the identity property of the category.

Anyway, the arguments for multiple maps has been centered around the fact that different Squeak images (3.5, 3.6, 3.7 etc) work with different releases and the fact that different releases have different characteristica, like being "stable" or "bleeding edge" etc.

But why would we need to split things to handle this? SMPackageRelease is categorizable. Mark each release appropriately. And even if the current SM doesn't use it - VersionNumber makes sure we can handle the release tree with branches. This means you can easily have this (excuse my ASCII art, "3.5" means categorized with "Squeak 3.5" etc, pub means published): 

1.0----(3.5, stable, pub)----1.1 (3.5, 3.6, stable, pub)---1.2 (3.6, stable)---1.3 (3.7, alpha)
                                  \ 
                                   \----1.1.1 (3.5, stable, pub)---1.1.2 (3.5, alpha)
Ok, if we are in 3.5, we can see that the latest published stable release for Squeak 3.5 is 1.1.1. Obviously the 1.1.1 branch was made to cater for 3.5-images.

For 3.6 it is 1.1 that is the safe bet, 1.2 isn't published yet - but it is marked as stable so what the hell, we could give it a shot. The maintainer hasn't told us yet that he wants us to depend on it and he might delete that release tomorrow. Possibly he just forgot marking it as published - because it is after all considered "stable". Perhaps he did just a few bug fixes and *thinks* it is generally stable, but he hasn't tested it yet.

Anyway, I hope you see what I mean. Note also that this categorization of packages and releases with Squeak versions is something that possibly will be up for discussion with the dependencies coming. Another way to do it would be to be able to register Squeak *images* as an SMObject (typically new subclass) and then use the dependency model to express this information.

Installing stuff

Hehe, ok - if you have read this far you should get a reward. :) How the heck do I write load scripts that uses SM? Load scripts can either just be code snippets you use locally to install stuff or build your images - and they can also be registered on SM as a package! This is possible since the .st (and .cs) formats are essentially "doits". So this can be used to make what other people sometimes call "virtual packages" - a package that installs other packages and doesn't have any content itself.

It is also a "poor mans" dependency mechanism, but will still be useful in the future when we have dependencies - because a load script can also be considered to be a "Makefile" for producing an image.

Anyway, enough with the small talk :) and show us some code, here is a trivial load script which I categorize as "static" meaning that it will always load the same stuff, regardless if you run it now - or next week: 

	| map |
	map := SMSqueakMap default. "We will use it many times in the script, so why not a shorter name for it?"
	map loadUpdates. "This can possibly be skipped - but if the map is fresh it is fast, otherwise we might want to be sure it is."
	map installPackageReleaseWithId: '5a9d100b-35ba-4fb8-ab4d-3c5bc88ed15a' "Install this exact release of Shout."
Now, how did I get the UUID of the release? Well, look in the map! :) Explore this: 
	SMSqueakMap default packageWithNameBeginning: 'Shout'
...or print this perhaps: 
	((SMSqueakMap default packageWithNameBeginning: 'Shout') releaseWithAutomaticVersionString: '4') id
In the web UI and in the package loader you sometime see "r4" instead of "4". That was just me trying to introduce a syntax for these VersionNumbers that people will instantly recognize as such, and not as "regular" version names. We will see if it catches on.

Someone else may wonder - why this UUID? Some Squeakers have expressed strong feelings against UUIDs and I still defend them because they are a common mechanism in SM that makes sure there is always a unique identifier for everything, regardless what it is. It also opens up the possibility of distributed creation of SMObjects - that was also one of the original intents - and still is.

If you use a UUID like above (for the release of for the package) instead of the name of the package - you are sure this will work next year when "Shout" has been renamed to "Shout!". Now, let us do the same thing differently: 

	| map |
	map := SMSqueakMap default loadUpdates; yourself.
	map installPackageWithId: '1314cee7-e1a3-4ca5-a167-b6517bf16367' autoVersion: '4' "Shout".
This time we refer to the package instead of the release, and specify the release with the automaticVersion. This is simpler because we can easily just modify the automatic version number in the future instead of looking up a new UUID. A small comment showing which package it is, is always nice. :) This variant is the best IMHO.

Ok, we can do it in some other ways too, like: 

	| map |
	map := SMSqueakMap default loadUpdates; yourself.
	map installPackageNamed: 'Shout' autoVersion: '4'
This will also work - as long as the package isn't renamed. Now we have a few other variants that behave differently: 
	| map |
	map := SMSqueakMap default loadUpdates; yourself.
	map installPackageWithId: '1314cee7-e1a3-4ca5-a167-b6517bf16367'
	map installPackageNamed: 'SharedStreams'
Now, these two methods (name or id as before) don't specify an automaticVersion. Currently they then try to figure out a suitable release. Earlier they just took the *last* release, regardless if it was published or marked for the image Squeak version. Now, they instead rely on SMPackage>>lastPublishedReleaseForCurrentSystemVersion which will result in the *last* *published* release marked for the current image version. Read the implementation.

The experts notes that this code does not yet take into the account of branches - like I described above - that will be rewritten when branches are enabled.

Another note: Some of the packages today are load scripts that haven't been modified to take this new behaviour into account. This means some packages are currently broken. I apologize for that, but I still think this is better than what we had before, and hopefully it will keep the masses at bay waiting for dependencies a little while longer. :)

Now I am tired of typing and need lunch, so I end this with an excerpt from an email I sent earlier today.

Bill Schwab wrote: 

> My biggest wish list item is to have a way (appologies if it already exists<g>) to obtain a script to load the packages present in
> any given image, with a choice of getting the same versions, or latest available versions.

And I responded: 

> Ah, right. I have been thinking of adding that. Well, here you go:


 Transcript show:
(String streamContents: [:s |
	s nextPutAll: '| map |
map := SMSqueakMap default.
map loadUpdates.'; cr.
	SMSqueakMap default installedPackageReleases do: [:rel |
		s nextPutAll: 'map installPackageReleaseWithId: ', rel id printString
printString, '. "', rel printString, '"';cr]])


 Try playing with that - I haven't tested the result myself - would be
great if you could report that it works. :) Note that this doesn't
install things in the "correct" order - there might be ordering issues.
And also it only installs exactly the same releases. But you could then
open the package loader and use the new "upgrade all installed"
mechanism which is quite user friendly and will not do anything without
confirmation first.

Ok, over and out. Time for lunch.

regards, Göran

Very good article to understand SM!, posted 21 Jul 2004 by GermanArduino (Master)

Thanks You Göran by the useful article (for me at least) to help better understand SM.

SqueakPeople should be part of SqueakMap, posted 25 Sep 2004 by bkv (Journeyer)

The user accounts and trust network between accounts should be part of SqueakMap, IMO.

I suspect that the reason this hasn't happened yet is because there is not yet a UI within Squeak that really shows the "people" part of SqueakMap in a way that gets a users' attention.

[ Home | Articles | Login/Account | People | Projects ]