Among the many idols of our age, there is one that rules them all: John Stuart Mill’s harm principle, the belief that an individual’s choices may never be legitimately hampered, by anyone at all, except if he is harming others. Bizarrely, this idea, radical in 1860 when Mill published On Liberty, has now even been enshrined as the core principle of our Constitution, at least if you believe Anthony Kennedy and the majority of the Supreme Court. This book, of which you have probably never heard, was published in 1873 and is regarded as the best contemporaneous refutation of Mill. Maybe it is, but its refutation is too narrowly based and accepts far too many of Mill’s premises. It is a start to overthrowing the golden calf, but only a start.
James Fitzjames Stephen was a man of a type instantly recognizable and now wholly disappeared, the Englishman confident in English superiority. He was a lawyer, then judge, with a sideline in journalism, consisting mostly of political analyses masquerading as book reviews (a terrible practice, as everybody knows). For the most part he is forgotten today, but perhaps he is coming back, if not into fashion, at least into view. Oxford University Press, for example, has recently begun releasing annotated volumes of his works. Perhaps as post-liberalism gains traction, he will become more prominent. We will see.
Stephen has little in common with conservatives, of his time or our time, of any stripe. He is a religious skeptic, and he makes no claim that any moral system is better than another, even if he pretty obviously thinks English morality is the most expedient and the “Hindoos” and “Mahomaddans” several steps below. He is that nineteenth-century classical liberal commonplace, a convinced utilitarian, whose own heroes are Jeremy Bentham, for how to evaluate societal structures, and Thomas Hobbes, for how to view human nature. His argument with Mill is also strictly utilitarian; he does not accuse Mill of immorality or of leading innocents, or society, astray with his philosophy, but rather of inexpediency (which Stephen equates with injustice) and Pollyannaism. Not in whole—Stephen himself notes that he often agrees with Mill, on utilitarianism especially, and was impressed by all his writings until he reflected more on them. But on the core conclusions to be reached, especially about liberty, Stephen thinks Mill is wrong on the numbers, on the costs as weighed against the benefits.
Still, Stephen was, most of all, a realist, which is the dividing line between today’s Left (and their fellow travelers) and the effective Right. And at least in this book, which aspires to destroy the pillars of Mill’s thought in On Liberty, Utilitarianism, and The Subjection of Women, Stephen aligns with many conclusions of modern post-liberals. So, if no conservative, he may be a wingman, and that is all that one can really ask for in these days of looming battle.
Refuting Mill is an uphill climb, not because his arguments are good (in fact, they are often criticized by philosophers across the political spectrum as vague, contradictory, and unconvincing), but because they are sweet. Mill’s core desire was that not just law, but all of society, in no way disapprove of any behavior as immoral, and if a person did disapprove of another’s behavior, that he keep his mouth shut. This was self-dealing, of course, since Mill was unhappy that his own adultery was stigmatized by the society of the time, no doubt one of the core reasons he wrote On Liberty, and he was an eccentric, so he celebrated eccentricity. But a part of every person wants unbridled liberty, even if we know it’s bad for us. In a famous Simpsons sketch, when his wife tells Homer Simpson he needs to spend more time with the kids rather than on enjoying himself, he guzzles a jar of mayonnaise mixed with vodka, and responds: “That’s a problem for future Homer! Man, I don’t envy that guy!” Homer embodies the basic tenor of Mill’s views on liberty.
Throughout his book, Stephen goes to great lengths to precisely delineate Mill’s arguments and to be directly responsive. Not for Stephen any slippery sophistry or comfortable vagueness. In fact, since this is a reprint of the second edition, published in 1874, he extensively footnotes objections from his chief critics, and responds in detail to them. He starts by outlining his basic objection to “Liberty, Equality, Fraternity.” That was most famously the slogan of the French revolutionaries of 1789, but Stephen correctly takes it as much more, as “the creed of a religion,” the “Religion of Humanity,” manifested not only in Mill, but in Comte’s Positivism and already embedded, in Stephen’s time, into all aspects of European life. “It is one of the commonest beliefs of the day that the human race collectively has before it splendid destinies of various kinds, and that the road to them is to be found in the removal of all restraints on human conduct, in the recognition of a substantial equality between all human creatures, and in fraternity or general love.” Stephen rejects all of this. “[W]hen used collectively the words do not typify, however vaguely, any state of society which a reasonable man ought to regard with enthusiasm or self-devotion.” Proving this, that the costs of such a society outweigh its benefits, is the object of Stephen’s book.
Turning to each piece of the slogan in turn, Stephen begins by talking about Mill’s doctrine of liberty “in general.” After quoting Mill’s own definition of liberty, and noting Mill’s tendency to make assertions rather than offer proofs, he points out that the core of Mill’s belief is that it is always unjustified to attempt to modify another’s conduct by appealing to his fears, other than for self-protection. This, of course, is wholly opposed to the usual (until the modern day) view of morality, which is that fear of “disapprobation by neighbors” properly constrains a wide range of behavior, with fear of the criminal law playing a de minimis role. Stephen correctly points out that most human action is a form of self-protection, and that Mill’s extreme doctrine would “condemn every existing system of morals,” and also correctly points out that is Mill’s (unexpressed) goal. And while Mill explicitly denied that his principle applied to children or to the “backward races,” because they lack the necessary “superior wisdom” to guide themselves, he believed that for advanced societies, “free discussion” was all that was necessary for socially beneficial results to be universal, or nearly universal. Stephen objects; it is entirely clear that some grown Englishmen have superior wisdom to others, and human nature means free discussion unconstrained by stigma will result in large-scale bad behavior and conflict. The solution, embodied in every human society and proven to work, is compulsion.
Compulsion should be used, but when? Stephen examines this with a strictly utilitarian eye; he is not interested in specific morality, although he pays occasional lip service to Christianity. In his view, compulsion aimed at good objects, which is calculated to obtain those objects and at not too-great an expense, is both desirable and necessary. Not all compulsion is desirable, but Mill is wholly wrong that all compulsion is undesirable. He could only be right if he could show that “compulsion must always be a greater evil in itself than the absence of any objects which can possibly be obtained by it.” Mill, of course, makes no attempt to show any such thing; he offers only sonorous assertions and unsupported conclusions. Stephen then spends quite a few pages applying this principle to different specific situations, with real-life examples. He also discusses the imposition of moral beliefs derived from religion, with which Stephen has no quarrel, pointing out the obvious fact, made real in the present day, that “European morality is in fact founded upon religion, and the destruction of the one must of necessity involve the reconstruction of the other. Many persons in these days wish to retain the morality which they like, after getting rid of the religion in which they disbelieve. Whether they are right or wrong in disturbing the foundation, they are inconsistent in wishing to save the superstructure.” Social intolerance toward those who contradict the moral beliefs of society, from wherever those beliefs are derived, is beneficial and necessary.
Stephen rejects the obvious rejoinder, that he must therefore love “the Inquisition and the war which desolated the Netherlands and Germany for about eighty years,” and be a devotee of Joseph de Maistre. Rather, he believes that criminal punishments, “prison, the stake, and the sword” are usually the wrong tool. However, if a thought has real value, it will persist in the face of social disapprobation, and once triumphant, it will be far more strongly rooted than if it was simply adopted as sweeter. This distinction between criminal law and what would today be called stigma is core to Stephen’s thought—not that the criminal law cannot be used to enforce morality, but its use should be reserved for appropriate and limited purposes, due to the criminal law being a “rough engine,” and in any case the vast majority of moral compulsion can and does rely on social stigma, not the law. Thus, most of Stephen’s defenestration of Mill revolves around the defense of stigma as a socially beneficial set of feelings and actions. Anthony Kennedy would shudder in horror if he read any of this.
Stephen devotes a whole chapter to liberty as it relates to thought and discussion, and another as it relates to morals, seeing this as the core of his disagreement with Mill, since it appears (wrongly, he says) that thought and discussion cannot harm others. He lists Mill’s four reasons why freedom of thought and expression ought never to be constrained, and rejects them all, then excoriates Mill’s famous celebration of “individuality” and “experiments in life.” Mill thought those would lead to general excellence and massive increases in social benefit, not by coercion, but by the masses recognizing their superiors and voluntarily choosing to follow their “pointing the way.” Stephen thought it would leave the vast majority of men adrift and wallowing in vice. “Habitual exertion is the greatest of all invigorators of character, and restraint and coercion in one form or another is the great stimulus to exertion.” No coercion leads to stagnation and vice. Virtue does not create followers; “The way in which the man of genius rules is by persuading an efficient minority to coerce an indifferent and self-indulgent majority, which is quite a different process [from Mill’s].” Variety by itself is worthless; “Mr. Mill . . . confounds the proposition that variety is good with the proposition that goodness is various.” Look around the world of 2019 and you tell me who was right.
Stephen applies a similar analysis to coercion in morals. This is the heart of the book, at least for a present-day reader. But it is also the one that shows why we are at the pass we are today. Much of the chapter relies on reductio ad absurdum. How could it possibly be, Stephen asks, that public disapprobation could not attend a group of men who organize to seduce women, or not attach to fornicators, or adulterers, or public drunkards? In Mill’s mind, for example, public drunkenness is fine; it is only if the drunkard cannot pay his bills that he may be stigmatized. True, Stephen says, the “employment [of compulsion] is a delicate operation,” whether of the criminal law or of social stigma, but that does not mean it should never be employed, when the benefits exceed the costs. “The object is to make people better than they would be without compulsion.” We must keep in mind that people differ along many axes, we need neither indifference nor compulsion to total conformity, but “a compromise between the two.” If we hew too much to Mill, “one of the principal motives to do well and one of the principal restraints from doing ill would be withdrawn from the world.”
So far, so good. But then Stephen admits that which is true—not only should we guard against excessive “meddlesomeness” by “busybodies,” and be cautious about exercising compulsion without due evidence, but “You cannot punish anything which public opinion, as expressed by the common practice of society, does not strenuously and unequivocally condemn. To try to do so is a sure way to produce gross hypocrisy and furious reaction. To be able to punish, a moral majority must be overwhelming.” And here we see why partisans of the harm principle have been so successful, because movement toward their position is a one-way ratchet, a feedback loop ending in moral agnosticism (except of the greatest sin of all, denying the harm principle, since anything can be tolerated except intolerance).
At first glance Stephen seems to think the opposite, suggesting that the moral standard, and the law protecting it, “may gradually be increased in strictness as the standard rises.” He notes that England in his day punishes homosexual acts (i.e., “practices which in Greece and Rome went almost uncensured”) with “utmost severity,” and “it is possible the time may come when it may appear natural and right to punish adultery, seduction, or possibly even fornication, but the prospect is, at present, indefinitely remote, and it may be doubted whether we are moving in that direction.” But if you described the world of 2019 to Stephen I suspect he would not be surprised. Shocked, perhaps, but if you extend his own thoughts forward, where we are today is actually the logical endpoint. This is especially true if you note that Stephen pleads that “Legislation and public opinion ought in all cases whatever scrupulously to respect privacy.” What he meant by privacy was not what today’s Supreme Court says is privacy, which is merely the embodiment of Mill’s harm principle. Yet if you combine shifts in public opinion with the supremacy of actual privacy, it is difficult to conclude that any form of moral compulsion can survive over time, given human nature and the sweet taste of vice.
Ultimately, though, all this is unsatisfying. Not wrong, but incomplete. Stephen’s analysis is blinkered and very much a product of his time, which treated any arguments from tradition, or of morality unmoored from utility, as archaic. Thus, he ignores other avenues of attack on the harm principle, such as Jonathan Haidt’s, which revolves around the recognition that, for the vast majority of people, morality is based not on utilitarianism, and not on liberty or equality, but on other, arguably more important, bases, such as loyalty, authority, and sanctity. Moreover, in Haidt’s view, morality is primarily intuitive, rather than something that should, or can, be justified by its costs and benefits. Haidt exalts Emile Durkheim and overtly rejects Mill, even though Haidt is himself a modern progressive. But nothing of this can be found in Stephen.
Another possible line of attack on the harm principle is that only with ordered liberty can a human society flourish. This, of course, was the view of all Western philosophy up until the eighteenth century, but Stephen does not focus on flourishing in the positive sense, merely on controlling the grosser forms of vice. As Yoram Hazony has recently succinctly outlined, and as Stephen completely ignores, compulsion or constraint benefiting society can be driven by positive, as well as negative, forces. Traditionally, social disapprobation was the mirror image of the honor and prestige that accrued to the virtuous, who disciplined themselves in order to benefit the community, whether as mothers, soldiers, or simply as benefactors to others. To Stephen, with his jaundiced view of mankind, it does not occur that society as a whole can be governed except through compulsion, but this is a parched view that is not borne out by the history of the West. To be sure, it is a constant struggle, but the Hobbesian view has always been a self-fulfilling prophecy, the bitter fruits of which we find ourselves displeased to eat in the twenty-first century.
Stephen’s discussion of liberty takes up three-fifths of the book. The rest is taken up with attacks on Mill’s positions on equality and fraternity. As to equality, the main focus is on whether women should be treated the same as men. Stephen thinks women inferior, because men have “greater muscular and nervous force, greater intellectual force, [and] greater vigour of character,” but that is not his point. Rather, inequality is the natural state of humanity, and many of those differences are “of sufficient importance to influence the rights and duties which it is expedient to confer upon them.” Mill admits that as to age, but denies it otherwise, which is irrational. As with his discussion of liberty, Stephen parades horribles which are today’s reality: harm to women from no-fault divorce; women in the military; little boys being treated as if they were little girls. Rather than a false equivalence between men and women, Stephen (in some ways like Jordan Peterson) preaches we should “provide for the common good of the two great divisions of mankind who are connected together by the closest and most durable of bonds.” Even Mill admits, to his chagrin, that the vast majority of English marriages are happy; this is proof enough the current system works. Equality, that is to say, is a chimera, and only a fool would pretend that it exists, or can be made to exist without worse harm resulting.
And then fraternity. “So far from joining in Mr. Swinburne’s odd address to France, ‘Therefore thy sins which are many are forgiven thee because thou has loved much,” it appears to me that the French way of loving the human race is one of the many sins which it is most difficult to forgive.” Ha ha. (Stephen is full of such pithy put-downs.) Men are mostly indifferent or bad, and enmity and strife the natural state of man. Even good men must have enemies, in the natural course of things. Men will never be angels and the human race is not improving. The Golden Rule is all very well, but Stephen thinks it aspirational at best. The idea of Mill, and others, like Comte, that love of mankind will become the new religion is dumb. Not even Mill believes it, in fact. “Mr. Mill, for instance, never loses an opportunity of speaking with contempt of our present ‘wretched social arrangements,’ the low state of society, and the general pettiness of his contemporaries, but he looks forward to an age in which an all-embracing love of Humanity will regenerate the human race.” Most presciently, and embodying every vision of the Left for a hundred and fifty years, “A man to whom this ideal becomes so far a reality as to colour his thoughts, his feelings, his estimate of the present and his action towards it, is usually, as repeated experience has shown, perfectly ready to sacrifice that which living people do actually regard as constituting their happiness to his own notions of what will constitute the happiness of other generations.”
All in, Stephen is certainly right in the vast majority of what he says. But he does not inspire, because he offers neither license nor grandeur. It is no wonder that he has been forgotten; being right is not what makes one remembered, but the ability to pander to the desires of, or stir the hearts of, men. Still, reading Stephen in light of present debates, the first in a hundred years where large swathes of the thinking classes reject the harm principle as a bedrock principle, will leave you better informed, and more confident in rejecting the poisoned fruit that Mill successfully fed the English-speaking West.