When stars form from a collapsing cloud of material, they are produced in a whole range of masses. At the very lowest end of the mass function the stars are barely heavy enough to cause nuclear fusion. Stars that are just at or below this cusp are known as 'brown dwarves'. There may well be stars that are even smaller, somewhere between the mass of a very small star and a large planet like Jupiter. They would be hot due to the energy released in the gravitational collapse but they wouldn't burn any fuel in fusion reactions. We don't really know much about the lower end of the stellar mass function, since we can't see these kind of objects due to them not emitting much light.
You couldn't really call these 'planetary systems' though, since they would be an isolated object. As cristo explained, it takes the large central mass of a star to allow planets to form orbits around the star. A very low mass star/Big Jupiter wouldn't accrete the required material to form any companion planets, or be able to hold them in orbit.