Unfortunately not. As I said, this was a test sheet and I don't have access to the relevant textbook. Anyway I'd say it's obvious now that this is not standard notation, I'll check it on the book itself when it becomes available.
Edit: For all the curious here, I managed to obtain the textbook. A* is defined as the set of all the elements greater than all the elements in A. Similarly, A* is the set of all the elements smaller than all the elements in A.