Neither she nor her husband paid much heed to the royal menaces, for before a year elapsed they both returned to town, and took up their residence at Somerset House. Here, as Pepys records, she kept a great court, "she being visited for her beauty's sake by people, as the queen is at nights: and they say also she is likely to go to court again and there put my Lady Castlemaine's nose out of joint. God knows that would make a great turn." But to such proposals as were made regarding her return to Whitehall, her husband would not pay heed, and she therefore remained a stranger to its drawing-rooms for some time longer. And when two years later she appeared there, her beauty had lost much of its famed lustre, for meantime she was overtaken by smallpox, a scourge ever prevalent in the capital. During her illness the king paid her several visits, and was sorely grieved that the loveliness he so much prized should be marred by foul disease.
But on her recovery, the disfigurement she suffered scarce lessened his admiration, and by no means abated his love; which seemed to have gained fresh force from the fact of its being interrupted awhile.
This soon became perceptible to all, and rumour whispered that the young duchess would shortly return to Whitehall in a position which she had declined before marriage. And amongst other stories concerning the king's love for her, it was common talk that one fair evening in May, when he had ordered his coach to be ready that he might take an airing in the park, he, on a sudden impulse, ran down the broad steps leading from his palace gardens to the riverside. Here, entering a boat alone, he rowed himself adown the placid river now crossed by early shadows, until he came to Somerset House, where his lady-love dwelt; and finding the garden-door locked, he, in his impatience to be with her, clambered over the wall and sought her. Two months after the occurrence of this incident, the young duchess was appointed a lady of the bedchamber to the queen, and therefore had apartments at Whitehall. There was little doubt now entertained she any longer rejected his majesty's love; and in order to remove all uncertainties on the point which might arise in her husband's mind, the king one night, when he had taken over much wine, boasted to the duke of her complaisancy. Lord Dartmouth, who tells this story, says this happened "at Lord Townshend's, in Norfolk, as my uncle told me, who was present." Soon after his grace accepted an honourable exile as ambassador to Denmark, in which country he died.
During the absence of the Duchess of Richmond, my Lady Castlemaine, then in the uninterrupted possession of power, led his majesty a sorry life. Her influence, indeed, seemed to increase with time, until her victim became a laughing-stock to the heartless, and an object of pity to the wise. Mr. Povy, whose office as a member of the Tangier Commission brought him into continual contact with the court, and whose love of gossip made him observant of all that passed around him, in telling of "the horrid effeminacy of the king," said that "upon any falling out between my Lady Castlemaine's nurse and her woman, my lady hath often said she would make the king make them friends, and they would be friends and be quiet--which the king had been fain to do." Nor did such condescension on his majesty's part incline his mistress to treat him with more respect; for in the quarrels which now became frequent betwixt them she was wont to term him a fool, in reply to the kingly assertion that she was a jade.
The disturbances which troubled the court were principally caused by her infidelities to him, and his subsequent jealousies of her.
Chief among those who shared her intrigues at this time was Harry Jermyn, with whom she renewed her intimacy from time to time, without the knowledge of his majesty. The risks she frequently encountered in pursuit of her amours abounded in comedy.
Speaking of Harry Jermyn, Pepys tells us the king "had like to have taken him abed with her, but that he was fain to creep under the bed into the closet." It being now rumoured that Jermyn was about to wed my Lady Falmouth, the countess's love for one whom she might for ever lose received a fresh impulse, which made her reckless of concealment. The knowledge of her passion, therefore, coming to Charles's ears, a bitter feud sprang up between them, during which violent threats and abusive language were freely exchanged.
At this time my lady was far gone with child, a fact that soon came bubbling up to the angry surface of their discourse; for the king avowed he would not own it as his offspring. On hearing this, her passion became violent beyond all decent bounds. "God damn me, but you shall own it!" said she, her cheeks all crimson and her eyes afire; and moreover she added, "she should have it christened in the Chapel Royal, and owned as his, or otherwise she would bring it to the gallery in Whitehall, and dash its brains out before his face."After she had hectored him almost out of his wits, she fled in a state of wild excitement from the palace, and took up her abode at the residence of Sir Daniel Harvey, the ranger of Richmond Park. News of this scene spread rapidly through the court, and was subsequently discussed in the coffee-houses and taverns all over the town, where great freedom was made with the lady's name, and great sport of the king's passion. And now it was said the monarch had parted with his mistress for ever, concerning which there was much rejoicement and some doubt. For notwithstanding the king had passed his word to this effect, yet it was known though his spirit was willing his flesh was weak. Indeed, three days had scarcely passed when, mindful of her temper, he began to think his words had been harsh, and, conscious of her power, he concluded his vows had been rash. He therefore sought her once more, but found she was not inclined to relent, until, as Pepys was assured, this monarch of most feeble spirit, this lover of most ardent temper, "sought her forgiveness upon his knees, and promised to offend her no more.".