We do not know when Pitarrow House was built. It was clearly a very old castellated house, judging from the contemporary account of its demolition in 1802 (see below). Since it contained fine murals illustrating scenes from Rome, we can be reasonably certain that these were painted before the Reformation; indeed, they were apparently concealed - possibly at the time of the Reformation - suggesting that the house would have existed for some time before the 16th century.

Sir John Wischeard, who succeeded to the lands and barony of Pitarrow in the 16th century, was a strong Protestant and a leading member of the first congregation against Catholicism. He became a member of the Scottish parliament and the Privy Council in 1560, and distinguished himself against the Earl of Huntly at the battle of Corrichie in 1562.

However, according to John Knox, Queen Mary hated him because "he flattered her not in her dancing and other things". In 1565, he was denounced as a rebel for opposing the marriage of Queen Mary with Lord Darnley, and was forced to flee to England. The Pittarrow estate was thus forfeited, but restored in the following year after the murder of David Rizzio, when Sir John returned to Scotland and was pardoned.

Sir John Wischeard was the last important Laird of Pitarrow. After his death in about 1607, the estates of Pitarrow and Reidhall passed to his second son James. However, by 1631 James Wishart's affairs had become "embarrassed" and he sold the family's lands at Pitarrow, Carnebeg, Woodtoun and the Mill of Conveth to David, Lord Carnegie for 59,000 merks or £3,277 15s 6d sterling.

Pitarrow House was subsequently passed down through three generations of Carnegies until, in 1802, a George Carnegie ordered its demolition. It was a fine example of an ancient Scottish baronial castle with many small turrets, a large vaulted entrance and a fair share of ghosts. The great hall was said to have been adorned with paintings of religious subjects afterwards covered over with wooden panels. William Fraser's History of the Carnegies, 1867 gives the following account:

"When the old mansion-house of Pitarrow was pulled down in 1802, there were discovered on the plaster of the great hall, to which access was had by a flight of steps, some paintings in a state of high preservation, the walls having been wainscotted. The air and dust having thus been excluded, the colours in the paintings were as vivid as if they had been done only a year before. One of the paintings .. represented a grand procession going to St Peter's in Rome. The Pope, adorned with the tiara, in his full robes of State, and mounted on a horse or mule, led by some person of distinction, was attended by a large company of cardinals, all richly dressed, and all uncovered. At a little distance near to where the procession was to pass, and nearly in front of it, stood a white palfrey, finely caparisoned, held by some person, also dressed and uncovered. Beyond this was the magnificent cathedral of St Peter, the doors of which seemed to be open to receive the procession."