Absolutely not! And don't let anyone make you feel guilty about it.
My husband and I have a 19 month old son and have debated whether to
have a 2nd child--with everyone (mostly friends) telling us and asking us when we are having our 2nd child.
We had a very, very, difficult time with our son when he was a newborn. He had an extreme case of colic for the first 3 months of his life and went to many doctors about it, only to be told he was as healthy as could be and just to be patient and he would outgrow it (only someone that has been this can relate to what we went through).
We are now leaning towards not having a second child only because like you said I'm not sure if I could handle going through it all again (I'm sure the 2nd would not be like my son with the colic) but I find it's already a lot of work just taking care of my son, let alone a baby as well.
I have a younger sister and we fought like cats and dogs growing up and to this day as adults we are still not that close and there is a lot tension because my mother has always been easier on my sister and has favoured my sister because she says "we just get along better".
I don't want to be like my mom and have a favourite, even though most parents would deny it.
If indeed my son does end up being an only child, we have have already talked about getting him involved in many sports, activities etc so that he has many opportunities to make friends. We also plan to always invite a friend on trips, excursions etc so that my son is not lonely.
You and your husband should really discuss this and see how things go. Your daughter is still very young and you may change your mind and want another baby. Even though my son was difficult as a baby he is now great as a toddler. I really couldn't ask for a nicer little boy and I get emotional when I see how quickly he is growing up and sometimes I miss those baby days........you just never know. Just don't feel guilty. It's a decision between you and your husband/partner.