Declining years
Ferrers lived on for another ten years, during which he attempted to regain his estates, with little success, largely because the machinations at Cippenham had been quietly supported by the King and his council. Edmund, in any case, was absent at the crusades until 1273 and no legal redress could be sought.
Soon after Edmund's return, Ferrers seized his old Chartley Castle by force, but was soon ejected. He then took a more considered approach, enlisting the help of Gilbert de Clare, Earl of Gloucester. In 1274, when Edward, now King, returned to England, Ferrers pleaded that he had accepted the Kenilworth ruling, with its seven years' redemption period, but that Edmund had refused. Edmund's defence was the Cippenham 'agreement' and Ferrers's failure to meet its terms. Ferrers argued that the 'agreement' was made under duress, but it was held that chancellor Chishall's presence at the signing gave it full legal validity.
Ferrers's case was dismissed and, although in 1275, he was able to recover his manor at Chartley (but not the castle), it marked the end of the great position of what had been one of England's most powerful families.
His final years were spent in the company of his family. His first wife, Mary, had died sometime between 1266 and 1269, and the marriage had been childless. He married (2nd) 26 June 1269 Eleanor, daughter of Humphrey de Bohun, Knt., of Kimbolton, Huntingdonshire, and Eleanor de Braose, and granddaughter of Humphrey de Bohun, 2nd Earl of Hereford. Until 1275, when he recovered Chartley, the family appeared to have lived on his mother's dower lands in Northamptonshire. The couple had two children: John born at Cardiff, Wales 20 June 1271 (who later became 1st Baron Ferrers of Chartley), and Eleanor, wife of Robert Fitz Walter, Knt., 1st Lord Fitz Walter.
Sir Robert de Ferrers, sometime Earl of Derby, died shortly before 27 April 1279, and was buried at St Thomas's Priory at Stafford, in Staffordshire. In Michaelmas term 1279 his widow, Eleanor, sued Edmund the king's brother for dower in a third of Tutbury, Scropton, Rolleston, Marchington, Calyngewode, Uttoxeter, Adgeresley, and Newborough, Staffordshire, and Duffield, Spondon, Chatesdene, and nine other vills named in Derbyshire, as well as other prominent landowners including Henry de Grenley (see Grindlay family); Edmund appeared in court and stated he held nothing in Spondon or Chatesdene, and as regards the rest Eleanor had no claim to dower in them, because neither at the time Robert had married her nor any time afterwards had he been seised of them. About 1280, Eleanor petitioned the king for the restoration of the manor of Chartley, Staffordshire, stating it was part of the inheritance of her son, John de Ferrers, who is underage and in the king's keeping. In 1284 she sued Thomas de Bray in a plea regarding custody of the land and heir of William le Botiller. In 1286 a commission was appointed by the king to investigate the persons who hunted and carried away deer and felled and carried away trees in the park of Eleanor late the wife of Robert de Ferrers at Chartley, Staffordshire. In 1290 she and her brother, Humphrey de Bohun, Earl of Hereford, acknowledged they owed a debt of £200 to Robert de Tibetot and Matthew de Columbers, the king's butler. Eleanor, Countess of Derby, died on 20 February 1313/4, and was buried at Walden Abbey, Essex.