SuperPilot(tm) DeFragger(tm) 
and 
DeFragger(tm) for the PalmPilot(tm) Connected Organizer

Features:
  FAST complete defragmentation
  Graphic display of your system memory usage
  Both Quick and Normal options
  Provides more statistical information than memory applet
  Usage statistics for both RAM and ROM
  Can be safely stopped at any time
  Free to SuperPilot users
  Available for all PalmPilot users

Operating Instructions
1. Using the PalmPilot desktop install tool, select dfrag10.prc or dfrag10s.prc for 
installation on your PalmPilot.
2. HotSynch to install the program.
3. Select the program to run by tapping on the TRG icon on your PalmPilot.
4. When the graphical interface appears, tap menu on your PalmPilot.
5. Under Defrag there are three choices, Normal, Quick and Fragment. Under any of 
these options you may safely stop the process by hitting any button.
  Normal will defragment all of your memory providing the maximum free 
space to continue to load either programs or databases.
  Quick will defragment a small portion of memory to allow your next load.
  Fragment will refragment your memory simulating the way the OS spreads 
chunks around.
6. Under View there are four choices, Heap Table, Heap Graph, RAM statistics and NV 
statistics.
  Heap table gives your numerical information about the use of your RAM. It 
lists maximum chunk available in each heap and the total available memory in 
each heap.
  Heap graph is the default view and shows a graphical representation of your 
memory usage. During defragmentation or fragmenting this is the view you 
see.
  Ram statistics show the amount of RAM used for programs and databases for 
all of the things you have loaded on your PalmPilot.
  NV statistics show the usage for programs and databases which are stored in 
non-volatile memory. This has specific  applicability to the users of SuperPilot 
Memory Board II who have the option of loading programs and databases into 
non-volatile memory.


Frequently Asked Questions

Who benefits from DeFragger?

All Pilot and PalmPilot users will benefit from DeFragger. Although the fragmentation 
problem is more pronounced with larger memory configurations, fragmentation still 
occurs with the smaller configurations. 

What is memory fragmentation?

See below for details.

When/how often should I run Defragger?

Even though DeFragger will defragment your memory, as you add/delete databases, the 
OS will fragment the remaining available memory. Therefore, you will occasionally need 
to rerun DeFragger to once again defragment memory, especially when the OS or 
HotSync claim there is no memory available even though there is still plenty left. In order 
to fill my 3M SuperPilot completely full, I had to run DeFragger 4-5 times. Every time 
HotSync aborted due to what it thought was a memory full condition, I reran DeFragger, 
then restarted HotSync.

What is the difference between yMFrag and DeFragger? 

yMFrag only compacts memory within a heap. This is only a partial fix to the memory 
fragmentation problem. It fixes fragmentation within a heap, but not the fragmentation 
caused by the 64K heap limit. DeFragger on the other hand, moves "chunks" of memory 
between heaps, leaving entire heaps free for allocation. 

Why do yMFrag and DeFragger disagree on the number of heaps?

yMFrag shows both RAM and ROM heaps, but does not differentiate between the two. 
yMFrag shows first the RAM heaps then the ROM heaps. DeFragger only shows the 
RAM heaps.

What is the difference between DeskFree Recycle and DeFragger? 

DeFragger can be safely stopped during the process at any time by hitting any key. 
Recycle can be scheduled to run at times other than while you are present which 
minimizes the impact of the much slower defragmentation process. DeFragger provides a 
quick option which allows you to quickly provide space to accommodate your next 
install. DeFragger shows you exactly what is happening and shows you the fragmentation 
in your system. If you choose, DeFragger provides more information than the memory 
applet standard on the PalmPilot.

Can I safely stop the defragment process?

YES. When you hit a button to stop DeFragger, the program simply stops and prompts 
you to hit a button to reset the Pilot. You probably will have a heap that's still partially 
full, but other than that you will have no problems. 

What memory chunks are not moved? 

Any memory chunk marked as unmovable or locked by the OS is not moved by 
DeFragger. Also, HackMaster itself and any hacks you have loaded on your Pilot are not 
moved. This includes both installed and uninstalled hacks. 

Can I run Defragger to simply see what the state of my memory is without 
defragmenting?

Yes, DeFragger's initial screen shows a graphical display of all the RAM heaps. 

Since unmovable/locked chunks of memory can't be moved, will my memory still be 
fragmented after running DeFragger?

When the operating system needs to allocate an unmovable a chunk of memory for itself, 
it allocates the memory at the upper end of an available heap which minimizes the 
fragmentation. However, hacks can be locked down in the middle of a heap causing heap 
fragmentation. The only time this would be an issue is when a hack would be locked 
down in the middle of the last heap and all the other heaps were completely full. 
Otherwise, DeFragger is going to completely free up at least one other heap for use. 
By the time you start getting close to fully filling up memory, you're bound to have quite 
a few little chunks of memory that are going to fit in the "cracks" caused by locked 
chunks. 

After running DeFragger most of the "filled" heaps still have approximately 1024 
bytes free. Why is this?

The initial version of DeFragger did completely fill up the heaps. However, there is a 
problem(bug/feature/anomaly) with the OS when the heaps become completely full. With 
over 2Megs free on the SuperPilot board, I could not load a 680 byte PRC file! The 
HotSync log said that I was probably out of memory. After leaving 1024 bytes free in 
each heap, the problem went away. The OS didn't actually use any of the 1024 bytes, 
rather it found space in another heap. The fact that before there were 0 bytes free in the 
heaps must have confused the OS. The OS will use the remaining 1024 bytes, so 
DeFragger may show less than 1024 in some of the heaps after using the Pilot for 
awhile. 

After running DeFragger many of the heaps show "1024 free, 1018 max chunk". 
Why is this?

Every chunk of memory(a record is simply a chunk) that is allocated has 6 bytes of 
overhead associated with it so the OS can keep track of it. This overhead contains such 
things as the chunk's size, whether or not its locked/unmovable, if the chunk is a free 
chunk, etc.

Is there a limit to the number of databases that can be loaded?

Having become very familiar with the memory system, I don't believe there is a limit. 
The data structures associated with Database and Memory Managers contain no 
restrictions on the number of databases or the number of records within a database. 
What's probably happening is the good ol' fragmentation problem. We have not been 
able to reproduce the problem here at TRG. We've created well over 300 databases on a 
SuperPilot Memory Board with no problems. 

Is there a performance hit after running DeFragger?

We have not seem one, other than the fact that we can now load more databases which 
causes the Pilot to slow down. However, more testing needs to be done.

What is PalmPilot Fragmentation? (a detailed description)

Many people who use TRG's SuperPilot memory board have reported memory full errors 
even though they have loaded less than 3MB of applications and/or data. This error is 
seen when attempting to transfer more applications to the PalmPilot via HotSync or when 
attempting to create or add records to a database. This note will attempt an explanation 
for this condition and will suggest several methods that users can employ to reduce the 
occurrence of this condition.

To understand the memory full problem we must first understand the memory 
architecture of the hardware and then how the PalmOS manages that memory. The 
PalmPilot is designed around the Motorola 68328 microcontroller which is basically a 
68000 core CPU with a bunch of peripherals attached to it all on one piece of silicon. The 
68000 core addresses memory through 32bit registers which means that it can directly 
address 2^32 or 4 Gigabytes of memory! Flat model. No Intel segment-offset jazz - just 
load an address in an address register and access the memory. Great. Beautiful.

Because it is unlikely that the PalmPilot will ever need to directly address (or ever 
possess) 4GB of memory the hardware engineers at Palm decided to use the eight upper 
address lines (A24-A31) to control the touch screen and various LCD hardware lines 
instead of wasting them as unused address pins. This re-definition of hardware signals 
from address bits to general purpose I/O is easily done using the 68328's internal control 
registers. Because of this design decision, the PalmPilot can directly address 2^23 or 16 
Megabytes of memory instead of the 4 Giga bytes that the processor itself can directly 
address. With these physical limitations defined, it is easy to see why the Palm literature 
declares 16MB to be the maximum amount of memory that can be present on any one 
memory CARD. Additionally it is apparent that 256 memory cards can be supported each 
containing 16MB for a total of 4GB. Of course, the PalmPilot only has 1 physical 
memory card slot but its nice to know the expansion capability is there if its needed. This 
is the extent of the hardware segmentation and, in my opinion, it seems pretty decent. 
Plenty of room for expansion and simple to use. If only the software was as simple. 

The first thing to notice is that the nice, flat, 16M card address space is cut up into 64K 
pieces named HEAPS. This division of free RAM into heaps happens soon after a 
hardware reset, just after the total RAM size is determined by the OS. Each heap is 
assigned a reference number starting at zero by the OS. According to the Palm literature, 
this was done so that memory within the heaps can be subdivided into variable-sized 
items called CHUNKS which the OS can then keep track of using 16 bit numbers within 
a heap (as opposed to 24 bit numbers which could directly address 16MB).

Chunks are allocated and handed out by the OS to application programs as they request it 
or are used by the OS itself for program storage etc. Since a chunk must be contained 
within a heap and since a heap is defined as being 64K or less in size, the largest chunk of 
memory that a program can be granted by the OS in any one request is 64K.

To recap, anytime the OS allocates a piece of memory - that piece of memory is referred 
to as a CHUNK which resides within a HEAP. Databases contain records which are 
simply a CHUNK a memory. Databases can, and usually do, contain numerous records 
which are physically located throughout all of the system's memory heaps.

When a program requests a chunk of memory from the OS it is given a virtual address to 
that chunk called a HANDLE by the PalmOS. Handles, as opposed to physical addresses, 
are used because the OS occasionally needs to re-arrange the physical location of chunks 
in memory without confusing programs that may have data stored in those chunks. 
Versions of the OS prior to 1.0.6 could only rearrange chunks within a heap. Later 
versions of the OS corrected this problem by enabling the OS to move a chunk from one 
heap to another if necessary but rarely does.

In a most unfortunate design decision, Palm coded the memory management portion of 
the OS with the objective of attempting to "balance" the amount of free memory across 
all of the available heaps. This was done under the assumption that resizing chunks would 
be a common request of applications to the OS. The thought process being that if there 
was always some free space within the heap containing the chunk to be resized, that 
resizing would be simpler and quicker. If you consider this decision for a moment, it is 
obvious why there is such a memory fragmentation problem. The OS fragments the 
memory on purpose!!! For example, if you have a 1M memory card, then you have 
roughly 1M / 64K or 16 heaps of 64K each. As mentioned earlier, if you have no memory 
used, the maximum sized chunk that an application can receive is 64K (actually a little 
less because of some heap overhead). Since applications programmers know about this 
limit they can write their programs in sub-divisions (resources) which are less than 64K 
in size and can request memory in small chunks. This is a pain but is livable. However, if 
you have 50% of your 1M memory used, free memory balancing between the heaps 
ensures that each of the 16 heaps is half filled, and now the largest chunk available is 
32KB instead of 64KB even though you have 512KB free!!. If your memory is 75% full, 
then 16K will be the biggest chunk available and so on.

It is now easy to explain why, when someone has used 2MB of their 3MB card, they may 
not be able to load a 25K application even though they have 1MB free. This situation is 
made even worse by a thing called chunk locking. An application can, if it so chooses, 
make a call to the OS and request that a record be "locked down". This causes the OS to 
set a bit in the chunks header that the memory manager looks at and honors when moving 
chunks around in memory during certain housekeeping routines. If a record is locked, the 
OS is not allowed to move that record around within memory. This situation is 
devastating in terms of the OS's ability to defragment memory, which we have already 
seen is, maximally fragmented. There are other complex situations involving the OS's 
ownership of certain unmovable chunks and limitations on the number of chunks within 
database directories etc. etc. of which I will spare you the explanation. Suffice it to say 
that the situation can be very complex and symptoms can easily appear and disappear 
even though great care is taken to precisely duplicate certain memory situations. This 
makes the problem very difficult to deal with on a case-by-case basis.

In conclusion use TRG's SuperPilot DeFragger to solve all your fragmentation problems.
