On July 2 1813, Wellington writes to the Earl of Bathurst. He is not happy at the loss of discipline that his men showed after the battle of Vitoria. He writes: ‘We have in the service the scum of the earth as common soldiers.”
Huarte, 2nd July, 1813.
My Dear Lord – I enclose the copy of a letter from the Governor of Vitoria, which shows how our men are going on in that neighbourhood. These men are detachments from the different regiments of the army who were sent to Vitoria the day after the battle, each under officers, in order to collect the wounded and their arms and accoutrements. It is quite impossible for me or any other man to command a British army under the existing system. We have in the service the scum of the earth as common soldiers; and of late years we have been doing every thing in our power, both by law and by publications, to relax the discipline by which alone such men can be kept in order. The officers of the lower ranks will not perform the duty required from them for the purpose of keeping their soldiers in order; and it is next to impossible to punish any officer for neglects of this description. As to the non-commissioned officers, as I have repeatedly stated, they are as bad as the men, and too near them, inpoint of pay and situation, by the regulations of late years, for us to expect them to do any thing to keep the men in order. It is really a disgrace to have any thing to say to such men as some of our soldiers are.
‘I now beg to draw your attention to the mode in which these irregularities affect our numbers. On the 17th June the total British and Portuguese force was 67,036 rank and file; on the 29th June it is 58,694 rank and file: diminution 8342 rank and file. The British on the 17th June were 41,547 rank and file; on the 29th June 35,650 rank and file: diminution 5897. The loss of British rank and file in the battle was 3164, including 200 missing; so that the diminution from irregularities, straggling, &c., since, for plunder, is 2733. The loss of Portuguese rank and file in the battle was 1022, including 73 missing; and their diminution from the same causes is 1423.
‘While we were pursuing the enemy by the valley of Araquil towards Pamplona, finding so many men straggling from their ranks, I ordered that an hospital might be established to receive them; and, although there are so many men absent from their regiments, there are only 160 in that hospital.The others are plundering the country in different directions.
Believe me, &c. ‘WELLINGTON.
pastnow 
Here’s a longer version of the same sentiment, from 1831;
11th Nov 1831: The Duke spoke strongly in favour of having a strong military punishment in reserve, were it only to give efficacy to the milder ones. I think he must have alluded to flogging. He gave us an account of the system of billing* up in the Guards. “Who would,” he said, “bear to be billed up, but for the fear of a stronger punishment? He would knock down the sentry and walk out !”
Do they beat them in the French army?
“Oh, they bang them about very much with ramrods and that sort of thing, and then they shoot them. Besides, a French army is composed very differently from ours. The conscription calls out a share of every class—no matter whether your son or my son—all must march ; but our friends—I may say it in this room—are the very scum of the earth. People talk of their enlisting from their fine military feeling—all stuff—no such thing. Some of our men enlist from having got bastard children—some for minor offences—many more for drink; but you can hardly conceive such a set brought together, and it really is wonderful that we should have made them the fine fellows they are. I have never known officers raised from the ranks turn out well, nor the system answer; they cannot stand drink.”
* That is, being confined to barracks, with extra drills.
‘Notes of conversations with the Duke of Wellington, 1831-1851’ by Philip Henry, 5th Earl Stanhope.
http://www.nelsonandhisworld.co.uk/forum/viewtopic.php?f=1&t=931&start=0
Thanks for this.