The Poynting vector is a definition, that is supposed to represent the energy flow at each point. Unfortunately, the only observable effect caused by the Poynting vector is through the energy variation in a volume subject to an energy flux through its surface, that is, the Poynting theorem.
As a curl could be added to the Poynting vector without changing the Poynting theorem, it can not be decided by EM only that this should be the actual flow of energy at each point. Feynman, commenting...