Masjid Tanah was believed to have gotten its name from a mosque that was built from earth, by the riverbank at centre of the town. The original name of this town was Pekan Sungai Baru taken after the name of a river called Sungai Baru. The mosque was built by that river, right in the heart of the town. This mosque was built during the 1800s by Sheikh Khadi Abdul Latif who came from Gujerat, India, but was of Arabic Yemen descent.

A local figure, Haji Sulong bin Sibeng, also worked hard during the construction of this mosque. In the early days of its establishment, all parts of the mosque including its floors and walls were made from earth. Hence the name ‘Masjid Tanah’, which literally translates as ‘the mosque made from earth’. In 1951, this mosque was renovated where its foundations were repaired, and roofs replaced. Masjid Tanah was once a trading town for goods brought from and to the City of Melaka (Kota Melaka) and the northern districts of Melaka and Alor Gajah. This town was inhabited mainly by the Malays of various societal status. Being in the Alor Gajah district and near the Negeri Sembilan border, most of the people in Masjid Tanah and its surrounding villages converse in the Negeri Sembilan or Minang accent; also referred to as the ‘Hulu’ accent or ‘Cakap Hulu’ by the residents of Masjid Tanah. For those who are originally from Masjid Tanah, the town centre is called ‘Simpang’, referring to the old town centre which had a three-way intersection located in front of the Masjid Tanah clock tower.