It’s hard to really wrap our heads around this. Yet, I find—over and over again—that it’s true. Love doesn’t always just slip away; we push it away… actively. This may sound accusatory and dooming, but to my mind, it is one of the most optimistic realities about relationships. To the degree that we ourselves control the amount of love we will tolerate, we control our romantic destiny. While we may not realize it, in countless, quiet ways, we may be giving up on love.

Our tolerance for love is established early in our lives and is based on our unique childhood experiences. The specific ways we were hurt influence us and come to shape our capacity for closeness. As we grow older, we gravitate to what’s familiar. We may choose partners who hurt us in the same ways we’ve always felt hurt. Or, if we do find ourselves in a healthy and rewarding relationship, we may reach a level of intimacy that exceeds our internal limits, and at that point, we recoil.

Most of us enter a good relationship in a good place. Early on, we feel great, because we feel valued and seen. We find what we always said we wanted. Yet, this blissful process of caring so deeply for someone else is also an invitation to care more deeply about our lives, which is scary. At this point, as in so many moments in life, we face a choice without being even fully aware of it. Do we side with life and invest in love, or do we choose the path of a more self-protective and defended part of ourselves? This is the part of us that resists feeling. It avoids risks. It gravitates toward numbness, eludes connection, commitment, and, ultimately, love itself.

In my 30 years as a researcher and clinical psychologist, I often reference the Fear of Intimacy, a book by my father, Dr. Robert Firestone, that aims to explain people’s resistance to love. When I introduce the theory surrounding fear of intimacy to people, they often say, “That sounds exactly like my husband!” or “My girlfriend totally has that issue.” It’s a concept people have trouble recognizing in themselves at first, because most people think they want love and don’t consciously feel afraid. Instead, they go along happily in their relationships for a time, then slowly, without awareness, they start to pull back. Ultimately, they diminish their feelings of real love and replace it with anything from routine to petty arguments to complete deadness between themselves and their partner.

Ironically, what sparks this fear can be the reality of getting exactly what we want. So many positive things can set us in motion to pull back from love and intimacy. We may receive a certain acknowledgment from our partner, something that is unknown or uncomfortable, because it contradicts feelings we’ve long had about ourselves.

Each of us harbors an inner critic that never quite believes in our value or our happiness. Milestones like falling in love, getting married, or having a baby can symbolically go against these long-held negative feelings we have about ourselves or our lives. In addition, these life events can remind us of time passing. They can arouse existential fears or a sense that we are growing up and divorcing from familiarities of our past. Negative events can further perpetuate this fear. Anything from an actual loss to a painful movie can strike a chord in us and remind us of life’s fragility.

So, what happens when we get scared? In what ways do we pull back from our relationship? Naturally, these behaviors manifest themselves differently in each individual, and they’re usually based on a person’s particular past. We all have our own specific set of defenses. We may become withholding toward our partner. We may start to feel easily trapped or intruded on. We may become controlling, overly critical, or destructively jealous. Or we may simply become…distracted.

It is all too easy to let practical aspects of life take over, especially with so many to choose from. Careers and kids tend to be big justifications we offer up when we realize we’ve lost touch with our partner. These, of course, are important priorities, but we can use them to divert us from our own desires to love and be loved. Think about ways we use technology, our phones, or even our food as substitutes for real contact. We can even use healthy-seeming activities like work, sleep, or exercise in the service of our defenses. When we work so hard, we miss time with our partner. What about when sleep takes priority over sex or affection? Someone I know went as far as to refuse to schedule any trip with his wife for years because it interfered with his daily routine of biking 20 miles.

We turn to our defenses for distraction or to “unwind,” in other words, to disconnect and burrow into our own self-sustaining world. Our lives take on an inward focus and, on a certain level, become more about taking care of ourselves than about the give and take of a relationship. This is not to say we are being selfish. In fact, on a practical level, we may be filling our days meeting the needs of others. Yet, on a personal level, we may be withdrawing from close and loving interactions.

Maintaining an outward focus is part of living a vital life. When both partners withdraw, the relationship becomes a “fantasy bond,” where both people remain together, imagining they are in love, while there is little to no actual relating. Couples may morph into societal roles of husband, wife, mother, or father and give up vital parts of themselves in the process. While the experiences involved in being a spouse or parent can be the most fulfilling parts of life, we get into trouble when we focus on form over substance. For instance, we can get wrapped up in schedules, arrangements, and functions, allowing them to take up more energy than acts of real relating, affection, humor, openness, or attraction.

We can use our endless “to-do’s” to cut off from deeper emotions that connect us to feelings of love and liveliness. Think about how good we feel on vacation. It isn’t just because there is less to do. It’s because we allot ourselves a period of time to just be, to connect, to take advantage of being with the people we love most. We don’t need weeks off on a faraway island to forge these connections. We can do it on a daily basis in those quiet, little moments we often miss because we have our guard up: that precious half hour in bed with our partner before we fall asleep, that commute we make every day sitting in silence or on a device.

If we stop being open and available to our partner, we are likely to wake up one day feeling as if we are living with a stranger. Those feelings of love that haven’t been allowed to flourish may seem to have withered away. Resisting a fantasy bond means not giving in to our fears. It means going out on a limb and living out our own ideas of what makes up a happy and fulfilling life. It means staying vulnerable despite the inside and outside forces that harden us to the world.

It can feel difficult, or even painful, to really do this in the moment, to stick in there and remain patient and loving with our partner. Yet, if we don’t, the outcome is much more desolate. We can miss out on our own life. When her parents had reached their 70s, a friend of mine asked them if they were still in love. They looked at each other and one responded, “We may not love each other, but we are loyal.” The truth is, we don’t have to settle for loyalty. What good is loyalty when two people decide to spend their lives miserable, but together?

Many couples don’t give up on each other, but they give up on what drew them to each other in the first place: love. Yet, studies in neuroscience show that people can maintain the exhilarating feelings of romantic love for decades. That is why I encourage almost every couple I meet who ever felt they were once in love to stick in there. Take actions toward your partner that he or she would perceive as loving. Make eye contact. Be affectionate—even after 30 years, even in line at the airport. Slow down. Be present. Practice mindfulness, as it may help you reconnect to your most authentic self, your real feelings and desires, and to be attuned to your partner. Offer acts of kindness, large and small. Take part in activities you and your partner used to share and enjoy together. Be open to new activities, something we tend to resist as we get older, more self-protective, or further into routine.

In short, do a lot of the things you did when you first met and started to form deep feelings for your partner, even if you don’t feel like it! Studies show that engaging in loving acts heightens our feelings of being in love. So, be free in flaunting your romantic feelings. Connect with them on a daily basis. No matter what our inner critic tells us, there is nothing foolish about allowing ourselves to be lovesick. There may be more to lose, but there is also much more to live for.