These topics are covered in numerous elementary/intermediate books. I will strongly recommend that you read the relevant books/notes(early chapters at least) available on the subject ("recursion theory" or "computability theory"). The questions you are asking are very basic (but neverthless the related concepts still take significant time to sink in) ... and to be honest what I have written is just about most of the stuff I know about this topic.
As far as kolmogorov complexity is concerned I think that it would make sense to try to understand it once you have some hold on the basics of recursion theory (I don't know about it anymore than it is related to "smallest programs" in various ways).
I think I begin to understand it:
There can be 3 kinds of languages (regarding of infinity):
(1): Finite number of elements (strings), all elements lengths are finite.
(2): Finite number of elements, some elements are infinite strings.
(3): Infinite number of elements, some (most of the) elemenst are infinite strings.
(+1): There is at least one undefined string in the language.
You are getting in completely wrong track with this. Just forget about "infinite strings" for now. As already mentioned before if you have "infinite string" like 11111111111..., you can just model it with a language:
{ e, 1, 11, 111, 1111, 11111, 111111, ...}
I understand that it is just like the set of natural numbers
To do computations (of the kind mentioned in this thread) you need a basic set of objects. The main condition is that objects can be counted easily. Natural numbers are definition of counting. Strings can also be counted quite easily. In some scenarios natural numbers are more convenient and in others strings take this role.
I have a feeling that any well defined language is decidable, is it right?
That is not the case. The undecidable languages are too numerous.
Wolud you please explain what you mean on " HALT is recursively enumerable but not recursive" exactly?
HALT usually refers to halting problem (in some variation). As it happens all recursive/decidable languages can easily (somewhat trivially) be shown to be r.e.(recursively enumerable) languages.
But the converse doesn't happen to be true and halting set is one of the most easily understandable examples of a language that is r.e. but not recursive/decidable.
"...given a language L⊆{0,1}*, we could let K(L) be the length of the shortest program that decides L..."
K supposed to be a specific (but only an imaginary) function that takes one finite string as input, and outputs an integer, the Kolmogorov complexity of that string.
K is supposed to be a function whose input is some sub-set of {0,1}* and whose output is a natural number (denoting length of program that decides it).
K doesn't take a string as input (I am going to drop "finite" because we are always talking about finite by default). It only takes a set of strings as input. If you meant that it can take a set which contains only one string, that is possible of course. For example, K could take {11111} as input but not 11111 (because that is not a set of strings).
How come to the picture the K(L), in which L is a language, a set of finite strings?
First note that the set of strings that number of strings in L need not be finite. For example, L could be {1,11,111,...} (or many other examples). As for how to picture it? Well it is not particularly simple. Here is the best informal picture I can think of (note that this is only relevant to how K(L) was defined in the quote you gave in the OP):
Basically you can think of
unique languages placed in each ordinal position. For all ordinals that are greater than or equal to ω the corresponding K(L) would be infinite. For ordinals less than ω at some points K(L) would be finite (denoting decidable languages) and at some points it would be infinite (denoting r.e. languages that are not recursive/decidable).
edit:
If we are talking about "recognising L" (instead of "deciding L") this picture would have to be adjusted a little. In that case you can imagine K(L) being infinite once again for ordinals that are greater than or equal to ω. However K(L) will be finite for all ordinals less than ω (denoting r.e. languages).For readers that are familiar with arithmetic hierarchy. I don't know arithmetic hierarchy well enough, but what I am thinking with this is that the first row (less than ω) contains r.e. languages. The second row (greater or equal than ω but less than ω.2) contains languages that are h-r.e. but not h-recursive ... and so on (obviously continuing in this manner AH itself will end pretty soon). Something of that sort basically.
