Squeak People diary for dway

15 Jul 2004

Thu, 15 Jul 2004 21:26:49 -0700

I added Whisker Browser as a project recently. If you use or contribute to the Whisker Browser, please indicate your relationship on the project page, so I can get a better idea of the user base. Thanks!

9 Feb 2004

Mon, 09 Feb 2004 08:31:38 -0700

I thought I should reply to Julian's arguments for keeping the tests in a separate package. Overall, it's definitely not a "no-brainer" decision either way, neither solution is perfect... the voting is pretty close. But my overall feeling is that maintaining the tests via the update stream is not really a major inconvenience.

"1. Forces all users to download a 300k update (into every image they need to update) even if they do not want the tests (and presumably the tests code will be getting larger)"

Mostly a non-issue relative to the overall volume of the update stream, IMO. Maintaining tests in the update stream might add 10-20% to the volume of the update stream, I guess.

"2. If we follow the suggestion of removing the tests from the update stream for release and putting them back in again for alpha, we force users to *repeatedly* download these large updates"

True, and I agree that adding/removing the tests for every release adds confusion, so I'd say we should leave the tests in for the Basic-final release to be consistent and that solves this.

"3. After the tests have been rewritten and rearranged a bunch of times, a user updating from a 3.6 image, say, to a 3.9 image still has to download the original 300k update even though much of it will be overwritten in later updates. This is even worse if we believe we will remove the tests later - you have to download them just to remove them again in a later update."

This is true for any code in the update stream. Anyway, slightly higher volume in the update stream doesn't seem like a big deal.

"4. Updates are put in the stream only every week or two - tests want to be updated more frequently, and could be updated as frequently as we like in an external package."

I actually disagree with this one, the tests will more likely be in sync with the regular code updates if they are also part of the update stream. Updating tests every 1-2 weeks seems reasonably often... I'm not sure the test package maintainer (Marcus) would update it more often than that anyway.

"5. If tests are in the update stream, one can't get the new tests without pulling in all the changes with it. Perhaps someone wants to pull the tests into a deployment image (that they don't want to move to alpha) to see which particular problems that image suffers from."

True... Although the tests might sometimes only work with the latest alpha code anyway. But that still could be useful.

"6. The update stream, being imperative, makes it much harder to have local changes to tests while you are working: loading an update could overwrite your local changes. This is made worse by the amount of time it takes for changes to make it into the stream: you end up having to keep a lot of changes locally until they get accepted; it's easy in this state to lose track of what has been accepted, which fixes are in which saved image, and which updates might overwrite changes in a particular image."

Good point here. More commonly, people would be writing tests for their own applications and wouldn't run into this as much, I think, since they'd just test their own classes. But this could be an issue if you're editing existing base image tests.

So overall, I think #6 and maybe #5 are the only real issues. :) I might even add a #7 to your list, though... one annoyance would be that all the base image tests would show up in the test runner, which would mean you'd have to get rid of them if you only wanted to run your own tests. But an improved test runner ui could address this too.

3 Feb 2004

Tue, 03 Feb 2004 22:05:48 -0700

Argument for including tests in the update stream

(Culled from my posts to squeak-dev...)

The proposal is to add the entire contents of the BaseImageTests package on SqueakMap to the 3.7alpha update stream. This adds 700+ tests to the image.

These are our alternatives:

1. Keep the base image tests in a separate package on SqueakMap as they are now, maintained by Marcus Denker.

2. (variations on #1) Keep the tests in a separate package on SM, and add a special UI shortcut to make it easier to load tests. Or possibly distribute the tests as part of a local cache.

3. Add the package to the update stream as one of those "in-image" packages (like SqueakMap itself), which means that the latest master copy would still be the package on SqueakMap, but periodically the version in the alpha image would be updated to the latest version on SqueakMap.

4. Add the contents of the package to the update stream and just maintain the code in the image. The package on SM would become obsolete.

I think #4 is probably the best approach.

The main reason is to keep the tests close to the code, so that they can serve as a sort of documentation. In the stable releases, beginners will have plenty of example tests to look at. In the alpha releases, the tests will be there for developers to easily use to double-check if anything has broken.

Another reason is that as we move toward splitting up the image into packages, many of the tests (e.g. AtomMorphTest) will be split off into new test packages paired with their corresponding code packages, and would no longer maintained in BaseImageTests anyway. So we're not really impeding the splitting up of the image by doing this. (A ways in the future, when we have only a Minimal kernel image left, that kernel image would have its own medium-smallish test package, testing only the things in the kernel.)

In other words, I think of tests as a somewhat special case... they should be split off from the image when the tested code is split off. They will be all split off eventually... we'd just do it in a different order.

I guess I agree with Andreas' sentiment in his earlier message, that we don't want to split things up too quickly before the infrastructure is there. It's getting better now with SM2, but we still need an (agreed upon) dependency scheme, having BFAV supporting adding fixes/enhancements to SM packages, etc. (Ok, this is only sort-of related to the issue of adding tests to the image, but...)

(Additional points about maintaining tests via the update stream...)

It will be less work overall to harvest tests via the update stream. Tests are often submitted to the BFAV along with the changes that are being submitted.

It will be (slightly) less work for submitters, since they could submit code and tests as part of the same changeset if they wanted, as long as they follow the test naming/category conventions. (I guess we would allow this?)

It will eliminate the extra manual work that the BaseImageTests package maintainer (Marcus) has to do, which is significant... for every batch of [updates] he has to find and add the appropriate tests associated with each update, if any. Although at some point this could be automated, we're not quite there yet. (Marcus could comment here if I haven't described this correctly.)

Tests are a special case in some ways. It's not like Balloon3D or VMMaker, which are cohesive subsystems and where it makes sense to have a lead maintainer. Tests are spread out throughout the system (as far as what is being tested), and are arguably best maintained by whoever maintains the tested code, not by one single Maintainer Of All Tests.

Finally, again note that this is a "temporary" solution, although it may mean that these tests are in the image for a year or two. Either way, eventually all of the tests will be split off into sister packages related to the code packages that are being tested.