The lowest you can go is your essential body fat, which is a vital fat mass. Never go below 8-12% for women and 3-5% for men. Ideally, you want a little more than this unless you are an athlete or bodybuilder.
Fat has many critical roles in the body including hormonal balance and building the brain and nervous system. A lack of fat in the diet can lead to mood imbalances and for women when body fat gets too low it can lead to infertility and loss of periods.