I think your understanding Feynman diagrams is incomplete. Many Feynman diagrams look very simple and intuitive but the theory of Feynman diagrams is deceptively subtle. I'm not up to writing a full treatise here, but maybe I can say a few things to point you in the right direction.
1. The first thing you have to understand is that a Feynman diagram represents a specific number. It's not exactly a diagram in the same way that a blueprint of a house is a diagram; it's much more akin to a variable. In the same way that sin(x) is just a number (if I fill in some value for x), the diagram for, say, Compton scattering is just a number -- it's a variable written graphically instead of using letters.
2. What is the number that a diagram represents? It's the quantum-mechanical matrix element, which stands for the amplitude for a process to happen a certain way (symbolized by that diagram). There are in general an infinite number of ways for any process to occur, and so an infinite number of diagrams. What do I mean by "a certain way?" Without too much math, the idea is that for some initial and final states, given by the bra and ket <f| and |i>, there are an infinite number of quantum-mechanical operators I can sandwich between <f| and |i> that give nonzero results; all of these contribute to the transition amplitude. These operators can be derived from the Lagrangian that describes whatever particle theory you're using.
The exact probability for a process to occur is the square modulus of the sum of all of the matrix elements. For realistic theories of interacting particles, there are no known exact, closed-form results -- the matrix elements get too complicated and it becomes impossible to add an infinite number of them. Fortunately, in many situations (but not all!) the more complicated terms are numerically smaller than the simpler terms, so it is possible to get very good approximations by just adding the first couple of diagrams. If you're familiar with Taylor series approximations, it's very much the same idea.
3. If it's all matrix multiplication, why do we use diagrams at all? Because of Richard Feynman. Before Feynman, people did calculate interaction matrices from scratch each time and sandwich them between the initial and final state vectors to calculate transition amplitudes. However, this process can be simplified. Feynman showed that the initial and final states could be pictured as lines going into an interaction, and lines going out. Each line stands for a
number (okay, or a spinor, or vector, or tensor...) which is a solution to the equations of motion for that particle when it is not interacting. The interaction operator sandwiched between <f| and |i> is symbolized by a drawing that connects the final-state lines with the initial-state lines. Furthermore, Feynman showed that the drawing in the middle has to consist of only a couple of available pieces; you're not allowed to draw just anything (the rules for what you are allowed to draw can be derived from the underlying laws of physics for that theory, but once you have the rules you can forget about the theory, because they work for all possible diagrams -- this was the great simplification). And these pieces are not just drawings; they represent numbers (or matrices or vectors or tensors or...). To get the matrix element, you multiply all the pieces of the diagram together, and that's it.
For example, consider this diagram:
Time flows from bottom to top. In words, an electron and a positron collide at the bottom vertex. They annihilate and emit a photon. The photon travels and emits an electron and a positron, which travel off. Each of the five lines and two vertices is a number, given by the Feynman rules (actually spinors, a tensor, and matrices), which are multiplied together. The result is the matrix element for this process.
That's not the only valid diagram for e
+ e
- goes to e
+ e
-, but it is one of them. And importantly, all the possible diagrams consist of only the pieces you see above: electron or positron lines, photon lines, and vertices where exactly two electron lines and one photon line meet. Although you can draw many different diagrams built of those pieces, the pieces themselves always represent the same thing. That is why Feynman diagrams are so useful.