I'm going to forget the productivity issue here and only talk about fairness. Fairness is not a good tax metric. It begs the question of what does fair mean? It means everything from taxing every single person the exact same dollar amount to taxing the rich and the rich only depending on one's political leanings.
To me an even bigger problem with fairness is: How can something inherently unfair ever be made fair? Taking my money isn't fair, period. The government legally steals money from those who make the money. I am not advocating getting rid of taxes. Modern society needs a fairly extensive government, and that means taxes. Taxes are a necessary evil of modern life.
An unfairness metric is much more measurable and quantifiable than is a fairness metric. It hurts when the government takes money from people. The pain of taxes should be spread around evenly, and (almost) everyone should participate. Taking $500 per year from someone who makes $10,000 per year hurts that person a lot even though the tax rate is a paltry 5%. Someone making that little money has no money to spare. That 5% comes from the basics of life.
Taking 5% from a millionaire doesn't hurt the millionaire much at all. The government needs to dig much, much deeper into the millionaire's bank account to make the millionaire feel the same pain as the poor person taxed at 5%. A sharply graduated tax makes a whole lot of sense when you look at taxes from the perspective of taxes being inherently unfair.