There is no hard and fast rule to measure security. There are two techniques that we normally look out for,
1. Adherence to shannon's principle of confusion and diffusion
2. Mathematical background for the encryption method (if there is one at all) should have its root in one of the NP Class problems.
have no idea how code breakers do their job so I cannot rate an innovative algorithm.
There are 5 possible attacks on an encryption.
1. ciphertext only attack
2. known plaintext attack
3. probable plaintext attack
4. chosen plaintext attack
5. chosen ciphertext attack
A very handy tool in such attacks is a] common sense b] statistics
You can pick up any good encryption book and read up on how each of the encryptions can be attacked. (many material is available even on net)
Also, does a substitution algorithm like the enigma used to work exist
DES uses substition and permutation as its basic transformations to introduce confusion and diffusion in the encryption method. Even AES uses substitution but its substitution is based on inverse modulo in GF(2^8) field.
-- AI