Yes, and indeed it was a famous debate between Feynman and Bohr about this point at the Shelter Island conference (see Schweber, QED and the Men who Made It). Bohr didn't like the hand-wavy interpretation as if the diagrams could be interpreted as particle trajectories in Minkowski space (worldlines). Of course, one problem was, that at the time, Feynman had no clear mathematical derivation of his rules but got it right by a lot of intuition. Schwinger was the opposite case: He had a cumbersome formal scheme without intuitive tools like Feynman's diagrams. To the surprise of the participants of the conference they however got precisely the same results. Finally the issue was clarified by Dyson, who gave a derivation for Feynman's ingenious diagrams from the (operator) QFT formalism, pretty much as it is still done today in all textbooks on QFT. Of course, a lot is simplified today by Feynman's other great achievement, i.e., the formulation of QFT in terms of functional integrals (in this case it's not literary a path integral as you can formulate for non-relativistic QT, see Feynman&Hibbs or Kleinert, but a functional integral over field configurations).