(wrote this up minutes before lecture; trying to run with presented ideas - characters -> bits information isn't very meaningful to me in a simple polyalphabetic cipher.)
Heuristic: group the cipher key into blocks of 5 letters- 40 characters requires eight 5-character blocks. While doing the cipher, note the lengths of each word and track block placement by adding the word length to last block placement value and taking mod 5. Keeping track of the block placement values could make it easier to keep track of where you are in the key while enciphering mentally. If you lose track of your placement mid-word, the block placement value could be used to get you back on track instead of having to recount everything.
If an error occurs while enciphering, there are two likely possibilities. The first of the likely possibilities is that an individual letter may be incorrectly converted to its ciphertext. The second likely possibility is that the person creating the ciphertext may have shifted the key placement; hopefully the block placement value will allow the shift to maintain some regularity, e.g., suppose you are decrypting (bare with me) a [7-letter word and are on the 3rd letter], from memory you know the current and upcoming blocks are RIJND AELCR YPTOG and that [you started on the 2nd character of the RIJND block]. The correct character to apply as a cipher key is N (value: +14), but you made a mistake and thought you were already using AELCR. You recognize the 4th ((2+3)-1) character of your block is C and continue on your merry way having skipped the 5 intervening characters NDAEL (N obviously included). Fortunately this can be fixed at the decoding phase without too much trouble and importantly doesn't effect the strength of the cipher. When the 7 letter word is finished, you would still observe that (2+7) mod 5 = 4 is the starting position in the continuing block.
On to memorization:
It takes hours (who would sit around memorizing pi for days??) to memorize a couple hundred digits of pi- a sequence of digits that serve no practical purpose in man's memory beyond perhaps 9 digits. My point is that a lot of people know plenty of digits of pi despite the inherent randomness of the digits.
Strictly relying on memorized strings of numbers could be worked into some sort of mental cryptosystem- however there are some considerations to make. Speaking strictly numerically, grouping the digits would seemingly offer a fairly easy solution to increasing the range at which a cipher can effect text. The process, as consistent with previous descriptions, would be an additive mental operation, but first between the grouped digits and secondly as the sum is applied to the text as a member of the crypto-key. Through cursory reasoning result in a problem as you can expect an inherent bias for the pairs to sum to 10. While 10 would be the most probable sum with 9% of all pairings yielding this value, followed by 9 and 11 then 8 and 12, etc. 0 and 18 occur the least frequently only once out of 100 blocks. This is sensible because the only way to sum to 0 with two digits is (0,0), and 18 also can only be reached by pairing (9,9). All other values 2-17 have at least two possible pairings, with 10 having the most through (1,9),(2,8),(3,7),(4,6),(5,5),(6,4),(7,3),(8,2) and (9,1). This would seemingly be a flaw in the strength of the cipher if the process were numerically based on this method. Although I'm not quite certain as to how much trouble that sort of thing would really cause as the key itself would still prove useful in lighthearted applications (of the sort my postings have predominantly intended). If you already have 200 digits of pi memorized, that can serve as a 100-character key!
Other methods can mediate the storage and retrieval of key-information; the pontifex method exploits a common deck of cards as such a medium. However, if people demonstrate the ability to retain accurate memories of numerical sequences (or otherwise), why not exploit this in a mental cryptographic system?
The memorization of non-numeric sequences could help avoid the aforementioned numerical bias (to 10) by increasing the number of values stored in each symbol - pairing would thus be unnecessary.
Heuristic idea #2: As 3rd graders memorize times-tables, the memorization of a substitution chart corresponding to each letter could help exploit rote memory and reduce overall cognitive load during mental enciphering. Practicing monoalphabetic ciphers would probably be a good way to learn because effectively each character in a key functions as a switch into a different monoalphabetic cipher (fortunately, there are only 25 monoalphabetic ciphers that produce unique ciphertext).
Getting accustomed to applying the monoalphabetic cipher mentally is easy for letters along the extrema of the range, if "A" is given as a key and represents a translation of 1 position, "A" onto "A" is always "B", "A" onto "D" is always "E", "A" onto "Z" is always "A". If all the letters can be learned this way then a sequence can be ciphered without much computation, just mental substitution. The "key" would be switching gears, treating your mind as a Turing machine and switching machine states dependent on the current symbol reached as one iteratively moves through the mental 'tape'-recording of a random sequence of letters. Furthermore, while developing the mental codes for automatic retrieval, conversion to numerical representations and a little quick algebra consistently works and is always an option as a mental checksum.
As for a long key, people (myself included) often pick up various >>useless<< alphanumeric sequences, e-mail accounts, phone numbers, microsoft windows (egh) serials. Memorization of sequences is often less an issue of memory limitations as it is a limit of motivation. The memorization of several hundred randomized characters with regular blocking seems a bit excessive however well within human capacity. I think a Vigenere cipher exploiting a memorized key of sufficient length could be effectively utilized in a mental cryptosystem. This is perhaps overly simplistic. Are there any other suggestions for going about an applied mental cryptosystem? Feel free to critique what I've written. Personally, I'm much more interested in simple ciphers that I could use when passing notes among friends, strictly for the hell of it, but I suppose if security were an issue the system could effectively be extended by more dramatic means.
EDIT: I tried mentally ciphering a sentence using a memorized 25-character key. I used my laptop's XP serial which I remember (for no good reason, i run slackware12) and considered all letters as numbers, went through the arithmetic mentally and managed to write out my message without showing work. Maintaining awareness of the key and the original message wasn't as difficult as one might suppose, there is a minimal amount of information that one is required to keep in focus to accomplish a simple cipher task as I have described. Maybe the use of a more sophisticated system could provide more security but the methods should be designed to function reasonably under limited human-attentional resources.