Photo: Marc Andrew Deley/FilmMagic

Hilary Duff, Joel Madden

Hilary Duffis keeping it friendly with at least one ex.

TheHow I Met Your Fatherstar, 35, revealed onWatch What Happens Livethat she and ex-boyfriendJoel Maddenhave long been good friends, after “the whole world’s mind was blown for a hot second” at her and husbandMatthew Koma’s dinner with Madden and his wifeNicole Richie.

“By the way, I have a good relationship with their whole family. I think people were [blown away]. I don’t know. We’re neighbors. And we actually hang out all the time,” she explained.

She said now that they’re neighbors, “We get them to hike with all the time,” adding: “And you know what? It’s lovely.”

“My husband is a giant troll, it’s one of my favorite things about him,” Duff noted. “Right when they moved in, for some reason Matt Photoshopped a picture of Joel and [brotherBenji Madden] in a photo with me onValentine’s Daywith like, ‘Happy Valentine’s Day, Honey.’ "

Josh Abraham/Instagram

Nicole Richie, Joel Madden, Hilary Duff

“So, this was before we started hanging out, and I think Joel and Nicole were like, ‘Oh my God, they don’t like us and we just moved in here next to them.’ But no, they just know that Matt is a big troll and likes to have lots of fun,” said Duff.

The Disney Channel alum and the Good Charlotte frontman began dating in 2004. After working together on her 2005 compilation albumMost Wanted, they called it quits in November 2006.

Dufftied the knotwith Koma in December 2019, and they share daughtersBanks Violet, 4½, and 10-month-oldMae James. She also co-parents sonLuca Cruz, 10½, with retired NHL ex-husbandMike Comrie, to whom she was married from 2010 to 2016.

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Madden and Richiegot marriedin December 2010, and they share daughterHarlow, 15, and 13½-year-old sonSparrow.

source: people.com