Charade no. 4

My first those who know it would willingly shun,
As destructive of pleasure, of health, and of peace;
But those who despise it are often undone,
And when we reject it its pow'rs we increase.
My second is by comparison little,
Yet joined to my first has surprising effect;
It subtracts by addition, and forms to a tittle
My whole which from puzzles like this you expect.